In their zeal to collect as much operational data as possible, organisations hoping to gain an advantage through the use of big data will also need to rethink how they process, analyse and present that material.
“When all this information finally gets to the business, it is difficult for the business to understand what to glean out of the data,” said Sharmila Shahani-Mulligan, CEO and co-founder of big-data start-up ClearStory Data. “We know this has been a problem for several years now.”
Shahani-Mulligan was one of a number of speakers at the O’Reilly Strata + Hadoop World conference in New York to offer tips on making the move from data to big data. She suggested that the executive dashboard is giving way to emerging technique of interactive storytelling, which gives data more readily apparent context and meaning.
Meanwhile, organisations should watch Google closely, advised M.C. Srivas, chief technology officer of Hadoop distributor MapR Technologies. “Google, with its vast and varied infrastructure, can provide us with a glimpse into the future of where computing is going,” said Srivas, who worked at Google before co-founding MapR.
More data
One of the basic rules to pick up from Google is that “more data beats complex algorithms,” Srivas said. “This is something that Google has demonstrated again and again: The company that can process the most data will have an advantage over everybody else in the future.”
A number of MapR customers are following this principle, Srivas said.
Millennial Media, a leader in the mobile advertising market, collects up to about 4TB of mobile user data each day, combining with petabytes of data on hand to build profiles of mobile users.
Cisco collects data from its firewalls worldwide, aggregating about a million events per second, all to better detect security threats. Credit agency TransHuman collects data from multiple sources to provide real-time credit scores.
But once an organisation has committed to collecting more data, the question becomes what to do with it.
Visualisation is a handy tool, but picking the correct visualisation is vitally important, advised Miriah Meyer, an assistant professor in the University of Utah’s School of Computing.
The most challenging and important step in visualisation is “gaining an understanding of the user’s needs and then being able to translate that into a set of visualisation requirements,” Meyer said.
Meyer worked with one researcher who was comparing the human genome with that of lizards. The researcher tried off-the-shelf data visualisation tools, but found that they hid many pertinent details and were not intuitive to work with.
The tool Meyer helped create, called Mizzbee, allowed the researcher to get insights from the data that couldn’t be gleamed from generic visualisation software.
“When done well, visualisation has the potential not only to support science but to also influence it,” Meyer said. “We have to move beyond thinking that visualisation is just about pretty pictures and instead embrace that it is a deep investigation into sense making.”
Dashboard fall off
Dashboards are one form of visualisation that could be used less, Shahani-Mulligan said.
Organisations have been using dashboards for well over a decade and not much has changed with them over that time, Shahani-Mulligan said. While they are fine for capturing key performance indicators and basic performance metrics, they are too brittle for advanced and timely analysis of big data, she said.
Ltd
