Written by

Bernard Marr

Bernard Marr is a world-renowned futurist, influencer and thought leader in the fields of business and technology, with a passion for using technology for the good of humanity. He is a best-selling author of 20 books, writes a regular column for Forbes and advises and coaches many of the world’s best-known organisations. He has over 2 million social media followers, 1 million newsletter subscribers and was ranked by LinkedIn as one of the top 5 business influencers in the world and the No 1 influencer in the UK.

Bernard’s latest book is ‘Business Trends in Practice: The 25+ Trends That Are Redefining Organisations’

View Latest Book

Follow Me

Bernard Marr ist ein weltbekannter Futurist, Influencer und Vordenker in den Bereichen Wirtschaft und Technologie mit einer Leidenschaft für den Einsatz von Technologie zum Wohle der Menschheit. Er ist Bestsellerautor von 20 Büchern, schreibt eine regelmäßige Kolumne für Forbes und berät und coacht viele der weltweit bekanntesten Organisationen. Er hat über 2 Millionen Social-Media-Follower, 1 Million Newsletter-Abonnenten und wurde von LinkedIn als einer der Top-5-Business-Influencer der Welt und von Xing als Top Mind 2021 ausgezeichnet.

Bernards neueste Bücher sind ‘Künstliche Intelligenz im Unternehmen: Innovative Anwendungen in 50 Erfolgreichen Unternehmen’

View Latest Book

Follow Me

Driving Success And Better Decisions Through Smart Business Experiments

2 July 2021

Everything useful that human beings have done has been based on some degree of experimentation. From the wheel to Penicillin, from the airplane to the iPhone, ideas have been conceived, tested and following failures improved and retested – and through this we have success. But what do we mean specifically when we talk about a Business Experiment?

In essence, a Business Experiment enables the applying of research tools and techniques to test different assumptions (or hypotheses) and subject these to measurement, validation and analysis, so that we can come to an evidence-based conclusion. If the conclusions are positive, then move forward with the idea/product/service. If not, then abandon, or restructure and retest.

Thomas Davenport made a particularly useful contribution to our understanding of creating “smart” Business Experiments in his article in the Harvard Business Review, in which he observed that the process should always begin with having a shared understanding of what constitutes a valid test. ‘Too many business innovations are launched on a wing and a prayer – despite the fact that it’s now reasonable to expect truly valid tests.’

A Testable Hypothesis

As Davenport rightly stresses, the process must begin with the creation of a testable hypothesis (it should be possible to pass or fail the test based on the measured goals of the hypothesis). For example, the hypothesis might be that “rolling out a new customer service training program to front-line staff will increase customer satisfaction and profit margins,” is a reasonable assumption and might well support the “hypothesis,” embedded in your strategy or plan on a page, but whether it is true or not has to be tested – not just assumed.

Design the Experiment

With the hypothesis agreed the next step is to design the test itself: that is, design the experiment. This means identifying sites or units to be tested, selecting the treatment groups and control groups (sometimes called A/B testing), and defining the test and control situations. So, with regard the customer service training example, do not train all staff at once: rather train some (the treatment group) and not others (the control groups, which become the baseline measure – what results do we get if we do nothing?).

Data Analysis and Action

Once the experiment has been completed, the data is then analysed to determine the results and appropriate actions. For customer service training the focus is on what is the difference between the performance of the treatment and control groups; did what we expected to happen (that is increase customer satisfaction and profit margins) actually take place within the treatment groups. What happened that was a surprise and what does that mean with regard to the fuller customer service training rollout or the need to improve other business processes or sub-processes; or perhaps the need for other, complementary experiments.

CKE Restaurant Case Example

Let’s look at an example. This one comes from the US-headquartered CKE Restaurants, which includes major US brands such as the Hardee’s and Carl’s Jr. quick-service restaurant chains. It applies Business Experiments during the introduction of new products.
Testing begins with brainstorming, in which several cross-functional groups develop a variety of new product ideas. Only some of them make it past the next phase, judgmental screening, during which a group of marketing, product development and operations people evaluate ideas based on experience and intuition.

Those that make the cut are actually developed and then tested in stores, with well defined measures, treatment and control groups. At that point, executives decide whether to roll out a product system-wide, modify it for retesting or kill the whole idea. CKE has attained an enviable hit rate in new product introductions – about one in four new products is successful, compared against one in 50 or 60 for consumer products – and executives say that their rigorous testing process is part of the reason why.

eBay Case Example

As a further example, at eBay, there is an rigorous process for making website changes, and randomised testing is a key component. Like other online businesses eBay benefits greatly from the fact that it is relatively easy to perform randomised tests of website variations. Its managers have conducted thousands of experiments with different aspects of its website, and because the site garners over a billion page views per day, they are able to conduct multiple experiments concurrently and not run out of treatment and control groups. Simple A/B experiments (comparing two versions of a website) can be structured within a few days, and they typically last at least a week so that they cover full auction periods for selected items. Larger, multivariate experiments may run for more than a month.

While broadly following the classic, and simple, Business Experiment process of hypotheses to analyse and learn, the company has also built its own application, called the eBay Experimentation Platform, to lead testers through the process and keep track of what is being tested at what times on what pages.

As with CKE’s new product introductions, however, this online testing is only part of the overall change process for eBay’s website. Extensive offline testing also takes place, including lab studies, home visits, participatory design sessions, focus groups and trade off analysis of website features – all with customers.

The company also conducts quantitative visual-design research and eye-tracking studies as well as diary studies to see how users feel about potential changes. No significant change to the website is made without extensive study and testing. This meticulous process is clearly one reason why eBay is able to introduce most changes with no backlash from its potentially fractious seller community.

Not Proving – But Testing

Given the wealth of data analytics tools now at our disposal, and the increasing connectivity with customers and other stakeholder groups, running Business Experiments is a relatively straightforward approach for testing proposed new products, or product or service enhancements and should be a key element of an overall approach to testing, analysis and improvement. But note, the idea is to test not to prove an assumption (an easy trap to fall into). Organisations that routinely punish failure would do well to heed the time-proven fact that getting things wrong is usually a vital stepping stone to getting things right.

Business Trends In Practice | Bernard Marr
Business Trends In Practice | Bernard Marr

Related Articles

Debunking The Top 5 Quantum Computing Myths

It makes sense that most people don’t understand quantum computing.[...]

Mastering Teamwork: Top 10 Strategies for Better Collaboration at Work

The nature of teams may be changing as more and more people work remotely, but the truth is businesses will always want people on their teams who can work well with others.[...]

Personalization Pitfalls: 5 Common Mistakes To Avoid For Effective Marketing

Targeted mass marketing was developed by direct mail businesses in the 1960s and 1970s to enable customers to be segmented by age, geography, or income.[...]

How To Upgrade From Data-Driven To AI-Driven Marketing Analytics

We’re told that data is the key to business success. But how do we go about turning data into money?[...]

How to Make AI Work in Your Organization

As the world continues to embrace the transformative power of artificial intelligence, businesses of all sizes must find ways to effectively integrate this technology into their daily operations.[...]

The 3 Biggest Digital Threats And How To Protect Yourself

Our digital footprints are bigger than ever. We bank online. We shop online. We order the Friday night takeaway from our phones.[...]

Stay up-to-date

  • Get updates straight to your inbox
  • Join my 1 million newsletter subscribers
  • Never miss any new content

Social Media

Yearly Views


View Podcasts