Report Summary

The deciding factor: Big Data and decision-making is report published by Capgemini and written by the Economist Intelligence Unit. The report examines how far down the road firms in different industries and regions are in utilising big data, and sheds light on the steps some organisations are taking to make data a critical success factor in the decision-making process.

When it comes to making business decisions, it is difficult to exaggerate the value of managers' experience and intuition, especially when hard data is not at hand. Today, however, when petabytes of information are freely available, it would be foolhardy to make a decision without attempting to draw some meaningful inferences from the data.

Anecdotal and other evidence is indeed growing that the intensive use of data in decision-making can lead to better decisions and improved business performance. One academic study cited in this report found that, controlling for other variables, firms that emphasise decision-making based on data and analytics have performed 5-6% better—as measured by output and performance—than firms that rely on intuition and experience for decision-making. Although that study examined “the direct connection between data-driven decision-making and firm performance”, it did not question the size of the data-sets used in decision-making. In fact, very little has been written about the use of "big data"—which is distinguished as much by its large volume as by the variety of media which generate it—for decision-making. This report is an attempt to address that shortfall.

The research confirms a growing appetite among organisations for data and data-driven decisions, despite their struggles with the enormous volumes being generated. Just over half of executives surveyed for the report say that management decisions based purely on intuition or experience are increasingly regarded as suspect, and two-thirds insist that management decisions are increasingly based on “hard analytic information”. Nine in ten of the executives polled feel that the decisions they've made in the past three years would have been better if they’d had all the relevant data to hand.

At the same time, practitioners interviewed for the report—all enthusiastic about the potential for big data to improve decision-making—caution that responsibility for certain types of decisions, even operational ones, will always need to rest with a human being.

Other findings from the research include the following:

• The majority of executives believe their organisations to be “data driven”, but doubts persist. Fully two-thirds of survey respondents say that the collection and analysis of data underpins their firm’s business strategy and day-to-day decision-making. The proportion of executives who say their firm is data-driven is higher in the energy and natural resources (76%), financial services (73%), and healthcare, pharmaceuticals and biotechnology sectors (75%). They may not be as data-savvy as their executives think, however: majorities also believe that big data management is not viewed strategically at their firm, and that they do not have enough of a "big data culture".
• Organisations struggle to make effective use of unstructured data for decision-making. Notwithstanding the heavy volumes, one-half of executives say they do not have enough structured data to support decision-making, compared with only 28% who say the same about unstructured data. In fact, 40% of respondents complain that they have too much unstructured data. Most business people are familiar with spreadsheets and relational databases, but less familiar with the tools used to query unstructured data, such as text analytics and sentiment analysis. A large number of executives protest that unstructured content in big data—with the partial exception of social media—is too difficult to interpret.
• Social media data are growing in importance. Social media tell companies not only what consumers like but, more importantly, also what they don’t like. They are often used as an early warning system to alert firms when customers are turning against them. Forty-three percent of respondents agree that using social media to make decisions is increasingly important. For consumer goods and retail, manufacturing, and healthcare and pharmaceuticals firms, social media provide the second most valued datasets after business activity data.  
• The job of automating decision-making is far from over. Automation has come a long way, but a majority of surveyed executives (62%) believe there are many more types of operational and tactical decisions that are yet to be automated. This is particularly true of heavy industry where regulation and technology have held automation back. There is, to be sure, a limit to the decisions that can be automated. Although technical limits are constantly being overcome, the increasing demand for accountability—especially following the financial crisis—means that important business decisions must ultimately rest with a human, not a machine. For less critical or risky decisions, however, there is still much scope for decision-automation. This is particularly true of machine-to-machine communication, where low-risk decisions, such as whether to replenish a vending machine or not, will increasingly be made without human intervention.
• Organisational silos and a dearth of data specialists are the main obstacles to putting big data to work effectively for decision-making. Data silos are a perennial problem, and one which the business process reengineering revolution of the 1990s failed to resolve. Regulation and the emergence of "trusted data aggregators" may help to break down today's application silos, however. Arguably a longer term challenge is the lack of skilled analysts. Technology firms are working with universities to help train tomorrow's data specialists, but it is unlikely that supply will meet demand soon. In the near future, there is likely to be a “war for talent” as firms try and outbid each other for top-flight data analysts.