Technology & Innovation

Data paralysis is a myth. Or is it?

July 03, 2012

Global

July 03, 2012

Global
Denis McCauley

Consultant

Denis McCauley contributes to EIU research published on a bespoke basis in Europe, the Middle East and Africa.

He works closely with the research directors and editors in each of these regions to improve the insightfulness, relevance and timeliness of their analysis.

Mr McCauley previously directed the company's global technology practice, with responsibility for managing research projects dealing with the impact of information and communications technology (ICT) on businesses and societies.

He is often interviewed by the media, including the BBC, CNBC and Financial Times, for his views on technology industry developments.

A lively discussion has been under way on one of our LinkedIn pages, the "EIU Opinion Leaders' Panel", about whether organisations are truly drowning under ever expanding waves of data.

A lively discussion has been under way on one of our LinkedIn pages, the "EIU Opinion Leaders' Panel", about whether organisations are truly drowning under ever expanding waves of data. Commonplace today are ominous-sounding analyst estimates of terabytes and even petabytes (definition: more than you can imagine) of data flowing into companies and clogging up their networks and storage systems. As are grumbles from some executives that the volumes are far too much for their organisation to handle. Apparently, many firms are not only leaving substantial value on the table by failing to analyse or act on their data, but many feel themselves paralysed under the sheer weight of it. Most participants in our panel discussion believe, however, that the issue is a good deal more complex than meets the eye. The problem, in their view, is less with the volume of data companies are having to handle but the manifold forms such data are coming in.

Recent research of our own – The deciding factor: Big data and decision-making – supports this view. In our survey of 607 senior business leaders, a majority of 63% indeed report that they're "struggling" with greater amounts of data, but an even larger share – 85% – say the problem is less the volume, but more their organisations' inability to analyse and act on the data in real time. And they distinguish between their ability to handle the structured kind of data – figures viewed and manipulated in spreadsheets or databases – and the unstructured kind, which comes in the form of audio, video, email and web pages, for example. Half of the survey respondents say they don't have enough of the former (only 7% report having "too much"), while 40% complain they have more of the unstructured variety than they can handle (compared with 28% who say they need more).

Why are many companies intimidated by unstructured data? That's because such things as video clips and images, beyond the fact that they are usually much larger than files of structured data, are extremely difficult for computers to interpret. Although consisting mainly of text, social media aren't straightforward either. Today’s business intelligence tools are very good at aggregating and analysing structured data, but there is a need for more advanced tools which can aggregate and summarise unstructured content. Which is why there is excitement in the data world about Hadoop, an open-source platform to support analytics tools that can manipulate extremely large amounts of unstructured data.

Of course, all the sophisticated data analysis in the world means nothing if organisations are unable to act upon it. As our aforementioned report and an earlier blog of mine point out, human intuition will remain an essential ingredient of many types of business decisions, even as more and more types of operational decisions become subject to automation. Nonetheless, all the research suggests that the business appetite for analysing and asking questions of data – whether structured or unstructured – in real time is growing. The fear of data paralysis is no myth, but companies' hope and optimism about Big Data means they should have little problem in overcoming their fear.

The views and opinions expressed in this article are those of the authors and do not necessarily reflect the views of The Economist Intelligence Unit Limited (EIU) or any other member of The Economist Group. The Economist Group (including the EIU) cannot accept any responsibility or liability for reliance by any person on this article or any of the information, opinions or conclusions set out in the article.

Enjoy in-depth insights and expert analysis - subscribe to our Perspectives newsletter, delivered every week