The Decision Dilemma: How More Data Causes Anxiety And Decision Paralysis

Every business needs data to make decisions that drive growth, streamline operations and improve profits. But is there such as thing as too much data? A recent survey by Oracle suggests that the answer is a definitive “yes”.

After surveying more than 14,000 employees and business leaders across 17 countries, it found that “decision paralysis” is an increasingly common issue. Troublingly, the negative impact of this phenomena is causing difficulties in both our personal and business lives.

The 2023 Decision Dilemma global study found that while 83 percent of us agree that access to data is essential for helping businesses make decisions, 86 percent say that data makes us feel less confident, and 85 percent have struggled with “decision distress” – feelings of regret over decisions we’ve made in the past year. 72 percent of the people who took part in the survey said that data has stopped them being able to make a decision – leading to decision paralysis.

It's not a new problem – in my work I’ve found that it’s not uncommon for people to feel that they are “drowning in data”. However, this is the first study I’ve seen that attempts to quantify the doubt and anxiety that this is causing.

To attempt to identify solutions I spoke to Oracle’s James Richardson, VP for Product Strategy, Analytics.

He told me “It’s the anxiety that comes from having the wealth of data we have at our fingertips, but perhaps a limited amount of time to use it.

“I say that the limited resource when you’re making decisions is not data, it’s time … one of the interesting findings was a significant number of people said that at the point of making a decision, they were so overwhelmed by the data that they didn’t make the decision. So the question is, how do we deal with that?”

Clearly the answer isn’t to go back to the “bad old days” when information was a rare commodity, expensive to obtain and difficult to make sense of. Today, data is widely used across all areas of business to drive growth and streamline operations. And the emerging class of artificial intelligence (AI) tools aimed at helping businesses work with the data they have means we’re less reliant of guesswork and gut-instinct when it comes to making choices.

Some examples: Marketing departments rely on customer demographic data and behavioral data such as purchasing history to put the right products and services in front of the right consumers, at the right time.

In operations, data is used to optimize supply chains, identify bottlenecks and inefficiencies, and forecast demand to help with decisions about inventory and resource management.

In HR, employee data is used to create an efficient workforce as well as to identify areas where training and recruitment activity should be focused, and to measure and assess staff performance.

And in finance, data is used to forecast performance, evaluate the value of a business’s products and services, and identify where cost-cutting can improve efficiency.

A key insight from the report is that while we know data is critical to the success of business, we often don’t feel we have the tools and technologies in place to put it to the best use.

Robotic Decision-Making

One telling statistic is that 64 per cent of us – and this rises to 70 per cent if we only talk to business leaders – would sometimes prefer to simply leave these decisions in the hands of a robot or AI.

Richardson tells me “I think this is very interesting, it tells me that we’re getting more comfortable with the idea of machine assistance or intelligence.

“If you’re happy to put your physical safety in the hands of a self-driving car, which is increasingly happening, then I can see why people would be moving in that direction.”

But what this really tells us, he believes, isn’t that we want to abdicate all responsibilities for decision-making. Rather it’s that we’re desperate for assistance.

“What they are really saying is ‘help me’”, says Richardson. “Obviously we have a limited capacity as humans to deal with data, so why not use machinery?”

To me, this also highlights a growing need to establish a balance as we move on from an age where insights would traditionally be handed down from business analysts, to an era of “self service” analytics where, increasingly, anyone has the power to find the answers they need. In my experience, organizations that successfully make this transition are those that put resources into supporting workforces through these changes. This can include initiatives such as setting up data and analytics hubs where those who do have professional training in the technology and techniques can to some extent hand-hold those who don’t on their journey.

Beating Data Paralysis – The NHS

As an example of an organization that’s already taking pro-active steps to rectify this situation, he points to the UK’s NHS – one of the world’s largest employers with responsibility for providing healthcare for close to 70 million citizens.

In particular, the Business Services division has made it a priority to help stakeholders from clinicians to patients and business administrators to leverage the vast amount of data at their fingertips.

Setting out with a “modest target” of helping the organization save £1 billion ($1.2 billion), it quickly enabled the NHS commissioning bodies to identify around £100 million ($1.24 million) in potential savings. This was done purely by identifying outliers – statistics that didn’t seem to fit with general trends – around spending by dentists and pharmacists on prescription medication.

Awareness initiatives were then launched, aimed at informing the dentists and pharmacies where this anomalous spending was taking place of efficiencies that could be made. This might include reducing over-prescribing where patients were given more than one treatment for the same condition, or switching to providing generic rather than brand-name drugs. This proved to be effective at generating the savings.

The truth was these departments had the data necessary to make the decisions all the time, but it wasn’t until they were given a hand with working their way through it that the insights were unearthed and the benefits came about.

When I spoke to the NHS Business Insights team’s chief insight officer, Nina Monckton (who is now Head of DataHead of Data at Just Group plc), she told me that building on this success, her organization has three ongoing priorities focused on helping overcome this data paralysis. The first is placing “data advocates” within individual departments to help them understand their data.

Secondly, it will create systems of pushing insights directly to the decision-makers who can use it, but may not have the skills, time or inclination to seek them out for themselves.

“If it’s an email, text or whatever that might be”, she tells me, “because I can’t see GPs logging into a system and playing around with data – some of them will, but not all of them.”

The third is moving towards more data transparency and implementing “open data” systems and policies.

Monckton said “It is pushing as much of the information, with respect for people’s privacy, as we can, because the more people that pick up on the data and use it … the more great things they can do and the better everyone’s life is going to be.”

Richardson shares my confidence that eventually technology such as AI, along with the data and analytics that goes with it, will bring long-term benefits, both to businesses and us as individuals in our personal lives.

Long-Term Benefits

“I love the idea of a four-day working week for humans, and I think the way we get to that is to increase productivity through machinery,” he says.

“There’s a lot of work within managerial decision-making that could be automated, there’s been some fear around that, but it’s starting to happen … I guess it’s a phase-shift in human society. We’re getting used to living with data, but there’s anxiety and often people feel under pressure.”

It certainly makes sense that any industrial and societal change on this scale is likely to be accompanied by feelings of uncertainty as we start to find our feet. However, it’s my belief that with proper planning, a careful alignment of technology with strategic aims, and a focus on ensuring we’re working with the right, relevant data and tools, we will start treating information as an enabling ally rather than a cause of stress and anxiety.

You can click here to watch my full interview with to James Richardson, VP for Product Strategy, Analytics at Oracle, where we deep-dive into more of the issues covered by the report. And if you would like to read the full insights from the you can find them here: 2023 Decision Dilemma.