Will Knight at MIT Technology Review: “On your way to this article, you probably took part in several experiments. You may have helped a search engine test a new way of displaying its results or an online retailer fine-tune an algorithm for recommending products. You may even have helped a news website decide which of two headlines readers are most likely to click on.
In other words, whether you realize it or not, the Web is already a gigantic, nonstop user-testing laboratory. Experimentation offers companies a powerful way to understand what customers want and how they are likely to behave, but it also seems that few people realize quite how much of it is going on.
This became clear in June, when Facebook experienced a backlash after publishing a study on the way negative emotions can spread across its network. The study, conducted by a team of internal researchers and academics, involved showing some people more negative posts than they would otherwise have seen, and then measuring how this affected their behavior. They in fact tended to post more negative content themselves, revealing a kind of “emotional contagion” (see “Facebook’s Emotion Study Is Just Its Latest Effort to Prod Users”).
Businesses have performed market research and other small experiments for years, but the practice has reached new levels of sophistication and complexity, largely because it is so easy to control the user experience on the Web, and then track how people’s behavior changes (see “What Facebook Knows”).
So companies with large numbers of users routinely tweak the information some of them see, and measure the resulting effect on their behavior—a practice known in the industry as A/B testing. Next time you see a credit card offer, for example, you might be one of a small group of users selected at random to see a new design. Or when you log onto Gmail, you may one of a chosen subset that gets to use a new feature developed by Google’s engineers.
“When doing things online, there’s a very large probability you’re going to be involved in multiple experiments every day,” Sinan Aral, a professor at MIT’s Sloan School of Management, said during a break at a conference for practitioners of large-scale user experiments last weekend in Cambridge, Massachusetts. “Look at Google, Amazon, eBay, Airbnb, Facebook—all of these businesses run hundreds of experiments, and they also account for a large proportion of Web traffic.”
At the Sloan conference, Ron Kohavi, general manager of the analysis and experimentation team at Microsoft, said each time someone uses the company’s search engine, Bing, he or she is probably involved in around 300 experiments. The insights that designers, engineers, and product managers can glean from these experiments can be worth millions of dollars in advertising revenue, Kohavi said…”