The Surprising Power of Online Experiments
In 2012, a Microsoft employee working on Bing had an idea about changing the way the search engine displayed ad headlines. Developing it didn't require much effort (it would take an engineer a few days), but it was one of hundreds of ideas put forward, and the program managers pushed it to the back of the priority list. So the idea languished for more than six months, until an engineer who realized the cost of writing the code would be low launched a simple online controlled experiment (an A/B test) to measure its impact. Within hours, the new headline type was generating abnormally high revenues and triggering a ''too good to be true'' alert. Such alerts usually indicate the existence of a bug, but not this time. An analysis revealed that the change had increased revenues by a staggering 12 percent without affecting key user experience metrics; on an annual basis, that translated to more than $100 million in the United States alone. This was the highest-yielding idea in Bing's history, but its value wasn't sufficiently appreciated until it was tested.
What a shame! This example highlights how difficult it can be to evaluate the potential of new ideas. Just as importantly, it demonstrates the benefit of running many tests at low cost and in parallel. This is a reality that many companies are beginning to recognize.
Today, Microsoft and many other leading companies (including Amazon, Booking.com, Facebook, and Google) each run more than 10,000 online controlled experiments per year, with many of them involving millions of users. Startups and non-digital-native companies such as Walmart, Hertz, and Singapore Airlines also run these studies regularly, but on a smaller scale. These organizations have realized that an ''experiment with everything'' approach yields surprisingly large returns. For example, it leads Bing to identify dozens of revenue-related changes each month, which collectively boost revenue by 10 to 25 percent every year. Along with hundreds of other changes that improve user satisfaction, these improvements are the main reason Bing has become profitable and has grown its share of personal computer searches in the United States from 8 percent when it launched in 2009 to 23 percent.
At a time when the web is vital to almost every business, careful online experimentation should be among standard operating procedures. If a company develops the software infrastructure and the organizational skills to use it, it can identify not just ideas for websites but also potential business models, strategies, products, services, and marketing campaigns; and quite cheaply. Controlled experiments can transform decision-making from an instinctive reaction into a scientific, evidence-based process. Without them, many groundbreaking innovations would never emerge, and many bad ideas would be implemented only to fail and waste resources.
Yet we've learned that most organizations, including large digital enterprises, are haphazard in their experimental approach, don't know how to apply careful scientific testing, or do too little experimentation.
Together, we have spent more than 35 years working on and applying experiments and consulting companies in a wide range of industries on this topic. In these pages we'll share the lessons we have compiled on how to design and implement them, ensure their accuracy, interpret their results, and address the challenges they can pose. While we'll focus on the simplest type of controlled experiment, the A/B test, our findings and recommendations are applicable to more complex experimental designs as well.
"APPRECIATE THE VALUE OF A/B TESTING"
In an A/B test, the experimenter prepares two experiences: the control group ''A'' is usually the existing system and is considered the ''champion''; the treatment group ''B'' is a modification that tries to fix something, the ''challenger.'' Users are randomly assigned to groups and key metrics are calculated and compared. (In contrast, single-variable A/B/C and A/B/C/D tests and multivariate tests evaluate more than one treatment or changes in different variables simultaneously.) This arrangement may be a new feature in the online environment, a change in the user interface (such as a new outline view), a back-end change (such as an improvement in an algorithm that recommends books at Amazon), or a different business model (such as a deal or free shipping). Whatever part of operations companies care about most (whether sales, repeat usage, click-through rates, or the time users spend on a site), they can use online A/B tests to learn how to optimize it.
Any company with at least a few thousand daily active users can implement these tests. The ability to automatically collect huge volumes of data on user interactions on websites and apps and to access large customer samples to run simultaneous experiments gives companies a unique opportunity to evaluate many ideas quickly, with extraordinary precision, and at a trivial cost per incremental experiment. This enables organizations to iterate quickly, fail fast, and recover.
Some leading technology companies, recognizing these advantages, dedicate entire groups to building, managing, and improving an experimentation infrastructure used by many product teams. Such a capability can be a significant competitive advantage, provided you know how to use it. Here are the points managers need to understand:
Small changes can have big effects. People often think the bigger the investment, the bigger the impact they'll see. But things rarely work that way in the online world, where success is more about getting many small changes right. While the business world celebrates big, disruptive ideas, in reality most of the progress comes from implementing hundreds or thousands of small improvements.
Consider the following example, again from Microsoft. (Although most of the examples in this article are from Microsoft, where Ron led the experimentation efforts, they also include lessons from many other companies.) In 2008, an employee in the UK made a seemingly trivial suggestion: whenever a user clicked the Hotmail link on the MSN homepage, automatically open a new tab to launch Hotmail instead of opening it in the same tab (in older browsers, open a new window). A trial was conducted with about 900,000 users from the UK, and the results were quite encouraging: engagement among users who opened Hotmail increased by an impressive 8.9 percent as measured by clicks on the MSN homepage. (Most engagement changes have less than a 1 percent effect.) Even so, the idea was controversial because at the time, many sites opened links in new tabs, so the change was only implemented in the UK.
The experiment was repeated in June 2010 in the US with 2.7 million users, with similar results, so the change was rolled out worldwide. Microsoft then explored whether to open results in a new tab for people who initiated a search on MSN, to see how this idea would perform elsewhere. In a US experiment with more than 12 million users, clicks per user increased by 5 percent. Opening links in new tabs is one of the best ways Microsoft has ever introduced to increase user engagement, and all it takes is changing a few lines of code. Today, many websites, including Facebook.com and Twitter.com, use this technique.
Microsoft's experience isn't particularly unique. For example, Amazon's experiments showed that moving credit card options from its homepage to the shopping cart page increased its annual profits by millions of dollars. It's clear that small investments yield large returns. By contrast, large investments either have very small returns or none at all. Enabling social media integration on Bing (so that content from Facebook and Twitter opened a third pane on the search results page) cost Microsoft more than $25 million and resulted in only small increases in engagement and revenue.
Experiments can guide investment decisions. Online tests can help managers understand how much investment in a potential improvement is appropriate. This was a decision Microsoft turned to when looking for ways to reduce Bing's search results display time. Of course, faster is better, but can the value of an improvement be measured quantitatively? Are three, 10, or 50 people needed to work on this enhancement? To answer these questions, the company ran a series of A/B tests in which artificial delays were added to study the effects of minute differences in load speed. The data showed that every 100-millisecond difference in performance had a 0.6 percent impact on revenue. Given that Bing's annual revenue exceeds $3 billion, a 100-millisecond speedup is worth $18 million in annual incremental revenue, enough to fund a sizable team.
The test also enabled Bing to make important tradeoffs, particularly about features that could improve the relevance of search results but slow down the software's response time. Bing wanted to avoid a situation where many small features would cumulatively cause a significant drop in performance. So the introduction of individual features that slowed response time by more than a few milliseconds was deferred until the team could improve its own performance or another unit's performance.
"BUILD A LARGE-SCALE CAPABILITY"
More than 100 years ago, the department-store owner John Wanamaker reportedly said what has become a marketing maxim: ''Half the money I spend on advertising is wasted; the trouble is, I don't know which half.'' We've noticed a similar situation with new ideas: most fail in experiments, and even experts often get it wrong predicting which ones will yield good results. At Google and Bing, about 10 to 20 percent of experiments produce positive results. At Microsoft as a whole, a third have come out positive, a third neutral, and a third negative. All of this shows that companies have to kiss many frogs (that is, run many experiments) before they find a prince.
Experimenting with everything is critical to make sure changes don't regress or have unintended effects. At Bing, about 80 percent of proposed changes are initially implemented as controlled experiments. (Some low-risk bug fixes and machine-level changes such as operating-system updates are not included.)
An infrastructure is needed to scientifically test almost every idea: instrumentation (to record clicks, mouse movements, and event times), data pipelines, and data scientists. Some third-party tools and services make experimentation easier, but if you want to scale, you need to embed the capability deeply into your processes. This will lower the cost and increase the reliability of each experiment. By contrast, a lack of infrastructure will keep marginal costs high and may discourage senior managers from wanting more experiments.
Microsoft offers an example of a non-trivial test infrastructure, but of course a smaller company or one less dependent on experimentation may get by with less. Microsoft's Analysis & Experimentation team can help run hundreds of controlled experiments on any given day, on a variety of products including Bing, Cortana, Exchange, MSN, Office, Skype, Windows, and Xbox. Each experiment introduces hundreds of thousands, sometimes tens of millions, of users to a new feature or change. The team carries out rigorous statistical analyses on all these tests and produces scorecards that automatically monitor hundreds of thousands of metrics and flag significant changes.
"The best data scientists swear by Twyman's law: Any figure that looks interesting or different is usually wrong."
A company's experimentation staff can be organized in three ways:
The centralized model. In this approach a team of data scientists serves the entire company. The advantage is that they can focus on long-term projects such as building better experimentation tools and developing more advanced statistical algorithms. A core problem is that the business units that draw on the group may have different priorities, which can lead to disagreements about the sharing of resources and costs. Another issue is that data scientists may feel isolated when dealing with companies and may not fully align with the unit's goals and domain knowledge, which can make it harder for them to connect the dots and share their perspectives on a topic. Moreover, data scientists may not have the credibility to persuade senior management to invest in the necessary tools or to convince corporate and in-house managers to trust the experiment's results.
The decentralized model. Another approach is to distribute data scientists across different business units. The advantage of this model is that data scientists can specialize in each business area. The main disadvantage is that it does not provide a clear career path for these professionals, who also miss out on the feedback and guidance from colleagues that helps them grow. Experiments in individual units may also not have enough mass to justify building the necessary tools.
The center-of-excellence model. A third option is to have data scientists in a central function and others in business units. (Microsoft uses this approach.) A center of excellence focuses more on the design, implementation, and analysis of controlled experiments. By establishing a company-wide experimentation platform and related tools, it significantly reduces the time and resources required for these tasks. It also spreads the best testing practices across the organization through classes, lab work, and conferences. The main downsides are that it's unclear what the center of excellence and the product teams own separately, who pays to hire more data scientists when various units scale up their experiments, and who is responsible for investing in warnings and controls that flag unreliable results.
There is no right or wrong model. Small companies usually start with the centralized model or use a third-party tool, and after growing, transition to one of the other models. In companies operating in multiple business areas, managers who view testing as a priority may not want to wait until corporate leaders develop a coordinated organizational approach. In such cases, a decentralized model may make sense, at least initially. If online experimentation is a corporate priority, a company may want to develop expertise and set standards in a central unit before deploying it across business units.
"DEFINE WHAT SUCCESS LOOKS LIKE"
Every business group needs to establish an appropriate (and usually multiple) evaluation metric aligned with its strategic objectives for experiments. This may sound simple, but identifying which short-term metrics best predict long-term outcomes is not easy. Most companies get this wrong. Getting it right (that is, coming up with an overall evaluation criterion, or OEC) requires careful understanding and usually extensive interviews within the company. There needs to be tight collaboration between senior managers who understand strategy and data scientists who understand metrics and tradeoffs. This is also not a one-off exercise: we recommend that the OEC be adapted annually.
As Bing's experience shows, an OEC is not arrived at directly. Long-term core goals are increasing search engine queries and advertising revenue. Interestingly, reducing the relevance of search results would lead users to make more queries (thus increasing the query rate) and click on more ads (thus increasing revenue). Such gains tend to be short-lived, however, because people eventually switch to other search engines. In OEC discussions, Bing's executives and data scientists decided that they wanted to minimize user queries per task or session and maximize the number of tasks or sessions users carry out.
It's also important to break an OEC into its components and track those components, because they often provide insight into why an idea succeeded. For example, if clicks are essential to the integrity of the OEC, it's important to measure which parts of a page are clicked. Looking at different metrics is critical, because it helps teams discover whether an experiment had an unanticipated effect in another area. For example, a team that changes the related search queries shown (say, a search for ''Harry Potter'' might surface queries for Harry Potter books, films, the casts of those films, and similar topics) may not realize it has altered the distribution of queries (by increasing searches for related queries), which can affect revenue positively or negatively.
The process of building and adapting the OEC and understanding the goals and effects gets easier over time. By running experiments, debugging the results (which we'll discuss shortly), and interpreting them, companies not only gain valuable experience about which metrics work best for certain types of tests but also develop new metrics. Over the years, Bing has created more than 6,000 metrics that experimenters can use, grouped into templates based on the areas tests cover (web searches, image searches, video searches, changes in advertising, and so on).
"AVOID LOW-QUALITY DATA"
If people don't trust the results of your experiments, it doesn't matter how good your evaluation criterion is. Getting numbers is easy; getting numbers you can trust is hard! You should devote time and resources to validating the experimentation system and to automated checks and safeguards. One method is to run careful A/A tests, that is, testing something against itself to make sure the system correctly fails to find a statistically significant difference in nearly 95 percent of cases. This simple approach has enabled Microsoft to detect hundreds of invalid experiments and incorrect applications of formulas (such as using a formula that assumes all metrics are independent when they aren't).
We've learned that the best data scientists are skeptics and swear by Twyman's law: any figure that looks interesting or different is usually wrong. Surprising results should be repeated, both to prove their validity and to alleviate people's doubts. For example, in 2013 Bing ran a series of experiments with the colors of various text on the search results page, including titles, links, and image captions. Although the color changes were imperceptible (see the diagram on the left), the results were unexpectedly positive: users who saw slightly darker blues and greens in titles and slightly lighter black in captions were more successful in their searches, and those who found what they wanted did so in significantly less time.
Because color differences are rarely perceptible, the results were understandably greeted with skepticism by some disciplines, including design experts. (Like many other companies, Microsoft long relied on expert designers, rather than the behavior of real users, to determine its corporate style model and colors.) So the experiment was repeated more comprehensively with 32 million users, and the results were similar. Analysis showed that, when offered to all users, the color changes could increase revenue by more than $10 million a year.
If you want the results to be reliable, you need to make sure high-quality data is used. Excluding outliers, detecting collection errors, and similar measures may be required. This issue is especially important in the online world for many reasons. Take internet bots. More than half the requests to Bing come from bots. This data can distort the results or add ''noise,'' making it harder to determine statistical significance. Another issue is the prevalence of outlier data indicators. For example, Amazon found that certain individual users were placing book orders large enough to skew an entire A/B test; it turned out they were library accounts.
Managers should also be cautious when some segments experience larger or smaller effects than others (a phenomenon some statisticians call ''heterogeneous treatment effects''). In some cases, a single good or bad segment can skew the average enough to invalidate the overall results. This happened in one of Microsoft's experiments: a segment made up of Internet Explorer 7 users couldn't click on Bing search results because of a JavaScript bug, and the otherwise positive overall results came out negative. An experimentation platform should detect such unusual segments; if it doesn't, experimenters looking for an average effect may dismiss a good idea as bad.
Results can also be affected when companies reuse control and treatment populations from one experiment to the next. This practice causes ''carryover effects,'' where people's experiences in one experiment change their future behavior. To prevent this, companies need to ''rotate'' users between experiments.
Another general check that Microsoft's experimentation platform runs is verifying that the percentages of users in the control and treatment groups in the actual experiment match the intended split. When these don't match, a ''sample ratio mismatch'' typically arises, which usually invalidates the results. For example, a ratio of 50.2/49.8 (821,588 versus 815,482 users) deviates so much from the expected 50/50 ratio that the probability it occurred by chance is less than one in 500,000. Such mismatches occur regularly (often weekly), and teams need to be diligent about understanding why and resolving them.
"AVOID ASSUMPTIONS ABOUT CAUSALITY"
Because of the excitement around Big Data, some managers mistakenly think that causality isn't important. They believe all they need to do is establish correlations and causality will be inferred. Wrong!
The following two examples explain why and also highlight the shortcomings of experiments without a control group. In the first example, two teams conducted separate observational studies of two advanced features in Microsoft Office. Both concluded that the new feature they were evaluating reduced churn. In fact, almost any advanced feature will show such a correlation, because those who try an advanced feature are usually heavy users, and they also have a low churn rate. So while a new advanced feature may be associated with a low churn rate, it doesn't necessarily cause it. Office users who receive error messages also have a low churn rate, because they tend to be heavy users as well. Does this mean that showing users more error messages will reduce churn? Not really.
The second example concerns Yahoo's research to evaluate whether display ads shown on its own sites increased searches for the brand name or related keywords. The observational portion of the study estimated that the ads increased searches by 871 percent, reaching 1,198 percent. However, when Yahoo ran a controlled experiment, the increase was only 5.4 percent. Without a control group, the company might have concluded that the ads had an incredible effect and failed to realize that the increase in searches was due to other variables during the observation period.
"Some managers mistakenly think that all they need to do is establish correlations and causality will be inferred. Wrong!"
Clearly, observational research cannot establish causality. This is well known in medicine; that is why the US Food and Drug Administration requires companies to conduct randomized clinical trials to prove that their drugs are safe and effective.
Including too many variables in tests also makes it harder to infer causality. Such tests yield results that are difficult to untangle and interpret. Ideally, an experiment should be simple enough that the cause-and-effect relationship can be easily understood. Another downside of complex designs is that they make experiments much more vulnerable to bugs. If there is a 10 percent chance that a test of a new feature triggers a catastrophic problem requiring it to be canceled, a change that includes seven new features has more than a 50 percent chance of having a fatal bug.
But what if you can determine that one thing causes another but you don't know why? Should you try to understand the underlying mechanism? The short answer is ''yes.''
Between 1500 and 1800, about 2 million sailors died of scurvy. Today we know that scurvy is caused by a vitamin C deficiency in the diet; sailors couldn't carry enough fruit supplies on long voyages. In 1747, Royal Navy surgeon Dr. James Lind decided to run a trial to test six potential remedies. On a voyage, he gave some sailors oranges and lemons, and others alternative remedies like vinegar. The experiment showed that citrus fruits could prevent scurvy, although no one knew why. Lind mistakenly thought the acidity of the fruit was the curative agent and tried to prepare a more shelf-stable remedy by heating citrus juice into a concentrate, thereby destroying the vitamin C. It was only 50 years later, when unheated lemon juice was added to sailors' daily rations, that the Royal Navy finally rid its crew of the scurvy scourge. If Lind had conducted a controlled experiment with heated and unheated lemon juice, the treatment would likely have been discovered much earlier and saved many lives.
So we should note that to take advantage of the ''what'' knowledge, you don't always have to have the ''why'' and ''how'' knowledge. This is especially true when it comes to the behavior of users, whose motivations are hard to predict. Some of Bing's breakthrough innovations have been made without an underlying theory. For example, while Bing could improve the user experience with imperceptible changes in text colors, there are no clear theories about color that would help us understand why. Here, evidence took the place of theory.
THE ONLINE WORLD is often perceived as turbulent and full of pitfalls, but controlled experiments can help us navigate it. When the answers aren't clear enough, or when people hold opposing views, or when the value of an idea is uncertain, they can point us in the right direction.
A few years ago, Bing was trying to decide whether to enlarge ads so that advertisers could add links pointing to specific landing pages. (For example, a lender could add links such as ''compare interest rates'' and ''about the company'' instead of just a single link to the homepage.) The downside was that larger ads would naturally take up more space on the screen and increase both user dissatisfaction and customer loss. Opinions on the idea were split. So the Bing team increased the size of the ads while keeping the overall screen real estate allotted to them fixed; this meant displaying fewer ads. The results showed that fewer but larger ads produced a major improvement. Revenue increased by more than $50 million annually, without harming key elements of the user experience.
If you really want to understand the value of an experiment, look at the difference between its expected outcome and its actual outcome. If you thought something would happen and it did, you didn't learn much. If you thought something would happen and it didn't, you learned something important. If you thought something insignificant would happen and the results were a big surprise and led to a breakthrough innovation, you learned something extremely valuable.
Your company can combine the power of software with the scientific rigor of controlled experiments to build a learning laboratory. The gains you'll reap (in the form of cost savings, new revenue, and improved user experience) can be tremendous. If you want a competitive advantage, your company must develop an experimentation capability and become proficient in the science of running online tests.
IN SUMMARY
THE NEED;When building websites and apps, many companies decide everything from new product features to look-and-feel to marketing campaigns based on subjective opinions rather than hard data.
THE SOLUTION; Companies should run online controlled experiments to measure the value of their ideas. Potential improvements should be tested rigorously, because large investments may not pay off, while some small changes can yield large returns and others can do surprising damage.
IMPLEMENTATION; Leaders should know how to properly design and run A/B tests and other controlled experiments, ensure their accuracy, interpret their results, and avoid their hidden pitfalls.
