Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

Book by Ron Kohavi, Diane Tang, and Ya Xu: “Getting numbers is easy; getting numbers you can trust is hard. This practical guide by experimentation leaders at Google, LinkedIn, and Microsoft will teach you how to accelerate innovation using trustworthy online controlled experiments, or A/B tests. Based on practical experiences at companies that each run more than 20,000 controlled experiments a year, the authors share examples, pitfalls, and advice for students and industry professionals getting started with experiments, plus deeper dives into advanced topics for practitioners who want to improve the way they make data-driven decisions.

Learn how to use the scientific method to evaluate hypotheses using controlled experiments Define key metrics and ideally an Overall Evaluation Criterion Test for trustworthiness of the results and alert experimenters to violated assumptions. Build a scalable platform that lowers the marginal cost of experiments close to zero. Avoid pitfalls like carryover effects and Twyman’s law. Understand how statistical issues play out in practice….(More)”.

Like a moth to a flame, we’re drawn to metaphors to explain ourselves

Kenan Malik at The Guardian: “The selfish gene. The Big Bang. The greenhouse effect. Metaphors are at the heart of scientific thinking. They provide the means for both scientists and non-scientists to understand, think through and talk about abstract ideas in terms of more familiar objects or phenomena.

But if metaphors can illuminate, they can also constrain. In his new book, The Idea of the Brain, zoologist and historian Matthew Cobb tells the story of how scientists and philosophers have tried to understand the brain and how it works. In every age, Cobb shows, people have thought about the brain largely in terms of metaphors, drawn usually from the most exciting technology of the day, whether clocks or telephone exchanges or the contemporary obsession with computers. The brain, Cobb observes, “is more like a computer than like a clock”, but “even the simplest animal brain is not a computer like anything we have built, nor one we can yet envisage”.

Metaphors allow “insight and discovery” but are “inevitably partial” and “there will come a point when the understanding they allow will be outweighed by the limits they impose”. We may, Cobb suggests, be at that point in picturing the brain as a computer.

The paradox of neuroscience today is that we possess an unprecedented amount of data about the brain but barely a glimmer of a theory to explain how it works. Indeed, as the French neuroscientist Yves Frégnac has put it, making ample use of metaphor, it can feel as if “we are drowning in a flood of information” and that “all sense of global understanding [of brain function] is in acute danger of being washed away”.

It’s not just in science that metaphors are significant in shaping the ways in which we think. In 1980, the linguist George Lakoff and philosopher Mark Johnson set off the modern debate on this issue with their seminal work, Metaphors We Live By. Metaphors, they argued, are not linguistic flourishes but the fundamental building blocks of thought. We don’t simply talk or write with metaphors, we also think with them….(More)”

Handwashing Can Stop a Virus—So Why Don’t We Do It?

Michael Hallsworth at Behavioral Scientist: “Why don’t we wash our hands as much as we should?

Behavioral science can help identify some of the key barriers. It may also suggest what might make a difference for COVID-19 in the absence of a vaccine, recognizing that there is much we still do not know about this virus.

The first barrier may be a lack of awareness about the effectiveness of soap, water, and scrubbing. People may simply not realize how well specific handwashing actions can prevent the spread of infectious disease. This is why many public health agencies run educational campaigns, which may have varying effects based on how far they take evidence about behavior into account.

For example, last weekend the Behavioural Insights Team (BIT), the organization for which I work, ran a set of online trials with 3,500 U.K. adults to test the impact of various posters on people’s intended handwashing behavior. We found that posters seemed to have stronger effects on people who were already washing their hands more frequently. In other words, the more compliant people got more compliant. Obviously, this is a real problem for infection control.

One specific issue with COVID-19 may be that people’s attention is being drawn to something else instead: face masks. In many countries, face masks in public are uncommon. Therefore, people in these places are more likely to notice when others are wearing masks, since doing so is visible and novel—unlike washing of hands! This may create the perception that wearing a face mask is the priority for preventing infection.

There are benefits from face masks, but we still lack evidence about how they are used or whether they work if worn by people who are not yet infected. At least one study suggests that on their own they may be less effective than handwashing at preventing transmission. And given that there’s a limited supply, face masks need to be reserved for the people and situations where they can do the most good.

Perhaps the main concern is that people may have a risk thermostat, whereby taking protective measures in one area means that they feel greater license to take risks in another. Obtaining a face mask may make people feel more protected and could mean they make less of an effort to wash their hands adequately.

Awareness is unlikely to be enough on its own. We also need to consider availability. In some instances, there are practical barriers to handwashing—water, soap, and drying materials may not be available. People may be aware of what they should do but be unable to follow through. One obvious solution is to increase the provision of alcohol-based hand sanitizer dispensers at locations where handwashing is infeasible. Doing this has been shown to improve hand hygiene on its own.

However, behavioral science shows that not all “availability” is equal: even small increases in required effort may result in a hand sanitizer going unused. Therefore, those providing hand sanitizer should also consider whether they’ve made usage as convenient as possible. How can dispensers be located so people do not have to make detours to use them? How can the dispensers be made more prominent—like the use of color? Where do people normally have to pause, thus making them more open to usage—like waiting for an elevator?…(More)”.

Upstream: The Quest to Solve Problems Before They Happen

Book by Dan Heath: “So often in life, we get stuck in a cycle of response. We put out fires. We deal with emergencies. We stay downstream, handling one problem after another, but we never make our way upstream to fix the systems that caused the problems. Cops chase robbers, doctors treat patients with chronic illnesses, and call-center reps address customer complaints. But many crimes, chronic illnesses, and customer complaints are preventable. So why do our efforts skew so heavily toward reaction rather than prevention?

Upstream probes the psychological forces that push us downstream—including “problem blindness,” which can leave us oblivious to serious problems in our midst. And Heath introduces us to the thinkers who have overcome these obstacles and scored massive victories by switching to an upstream mindset. One online travel website prevented twenty million customer service calls every year by making some simple tweaks to its booking system. A major urban school district cut its dropout rate in half after it figured out that it could predict which students would drop out—as early as the ninth grade. A European nation almost eliminated teenage alcohol and drug abuse by deliberately changing the nation’s culture. And one EMS system accelerated the emergency-response time of its ambulances by using data to predict where 911 calls would emerge—and forward-deploying its ambulances to stand by in those areas.

Upstream delivers practical solutions for preventing problems rather than reacting to them. How many problems in our lives and in society are we tolerating simply because we’ve forgotten that we can fix them?…(More)”.

Nudge Regulation and Innovation Policy

Paper by Nissim Cohen and Hadar Yoana Jabotinsky: “Whilst there is widespread agreement among decision makers that fostering innovation should be a priority, there is far less consensus on how to achieve this objective. Given the fact that the effects of new technologies are often unknown, in the early stages of technological development, there might be insufficient information for conducting a cost-benefit analysis. Under uncertainty, using strict regulatory measures might kill the innovation before the market matures, resulting in inefficiency. Moreover, strict regulation can infringe on entrepreneurs’ right to conduct a business. In addition, using strict regulation without fully understanding the technology and the harm it might cause consumers might not provide them with the needed protection. We argue that when regulating new technologies, the use of nudges is a desirable policy tool, superior to most other policy tools available to regulators. Nudging leaves room for technological developments while allowing the regulators to rely on the Wisdom of the Crowd to move regulation in the most efficient direction….(More)”.

Nudge Theory and Decision Making: Enabling People to Make Better Choices

Chapter by Vikramsinh Amarsinh Patil: “This chapter examines the theoretical underpinnings of nudge theory and makes a case for incorporating nudging into the decision-making process in corporate contexts. Nudging and more broadly behavioural economics have become buzzwords on account of the seminal work that has been done by economists and highly publicized interventions employed by governments to support national priorities. Firms are not to be left behind, however. What follows is extensive documentation of such firms that have successfully employed nudging techniques. The examples are segmented by the nudge recipient, namely – managers, employees, and consumers. Firms can guide managers to become better leaders, employees to become more productive, and consumers to stay loyal. However, nudging is not without its pitfalls. It can be used towards nefarious ends and be notoriously difficult to implement and execute. Therefore, nudges should be rigorously tested via experimentation and should be ethically sound….(More)”.

Behavioral Public Administration: : Past, Present, and Future

Essay by Syon P. Bhanot and Elizabeth Linos: “The last decade has seen remarkable growth in the field of behavioral public administration, both in practice and in academia. In both domains, applications of behavioral science to policy problems have moved forward at breakneck speed; researchers are increasingly pursuing randomized behavioral interventions in public administration contexts, editors of peer‐reviewed academic journals are showing greater interest in publishing this work, and policy makers at all levels are creating new initiatives to bring behavioral science into the public sector.

However, because the expansion of the field has been so rapid, there has been relatively little time to step back and reflect on the work that has been done and to assess where the field is going in the future. It is high time for such reflection: where is the field currently on track, and where might it need course correction?…(More)”.

When a Nudge Backfires: Combining (Im)Plausible Deniability with Social and Economic Incentives to Promote Behavioral Change

Paper by G. Bolton, E. Dimant, and U. Schmidt: “Both theory and recent empirical evidence on nudging suggest that observability of behavior acts as an instrument for promoting (discouraging) pro-social (anti-social) behavior. We connect three streams of literature (nudging, social preferences, and social norms) to investigate the universality of these claims. By employing a series of high-powered laboratory and online studies, we report here on an investigation of the questions of when and in what form backfiring occurs, the mechanism behind the backfiring, and how to mitigate it. We find that inequality aversion moderates the effectiveness of such nudges and that increasing the focus on social norms can counteract the backfiring effects of such behavioral interventions. Our results are informative for those who work on nudging and behavioral change, including scholars, company officials, and policy-makers….(More)”

Under the Influence: Putting Peer Pressure to Work

Book by Robert H. Frank: “Psychologists have long understood that social environments profoundly shape our behavior, sometimes for the better, often for the worse. But social influence is a two-way street—our environments are themselves products of our behavior. Under the Influence explains how to unlock the latent power of social context. It reveals how our environments encourage smoking, bullying, tax cheating, sexual predation, problem drinking, and wasteful energy use. We are building bigger houses, driving heavier cars, and engaging in a host of other activities that threaten the planet—mainly because that’s what friends and neighbors do.

In the wake of the hottest years on record, only robust measures to curb greenhouse gases promise relief from more frequent and intense storms, droughts, flooding, wildfires, and famines. Robert Frank describes how the strongest predictor of our willingness to support climate-friendly policies, install solar panels, or buy an electric car is the number of people we know who have already done so. In the face of stakes that could not be higher, the book explains how we could redirect trillions of dollars annually in support of carbon-free energy sources, all without requiring painful sacrifices from anyone….(More)”.

Nudge and the European Union

Chapter by Alberto Alemanno: “Europe has largely been absent from the US-dominated debate surrounding the introduction of nudge-type interventions in policy-making. Yet the European Union and some of its Member States are exploring the possibility of informing their policy action with behavioural insights. While a great deal of academic attention is currently been paid to the philosophical, ethical and other abstract implications of behavioural-informed regulation, such as those concerning autonomy, dignity and moral development, this chapter charts and systematizes the incipient European Nudge discourse.

Besides a few isolated initiatives displaying some behavioural considerations (e.g. consumer rights, revised tobacco products directive, sporadic behavioural remedies in competition law), the EU – similarly to its own Member States – has not yet shown a general commitment to systematically integrate behavioural insights into policy-making. Given the potential of this innovative regulatory approach to attain effective, low-cost and choice-preserving policies, such a stance seems surprising, especially when measured against growing citizen mistrust towards EU policy action. At a time in which some EU countries are calling for a repatriation of powers and the European Commission promises to redefine – in the framework of its Better Regulation agenda – the relationships between the Union and its citizens, nudging might provide a promising way forward. In the aftermath of the Brexit vote, this promise has not only been shared by the 27 remaining Member State but also represents one of their major priorities . Yet with promises come challenges too.

The chapter proceeds as follows. Section 2 sets the scene by discussing the growing appeal of nudging among policymakers within and across Europe. Section 3 introduces the notion of behavioural policymaking and contrasts it with that of nudging. Section 4 describes the early and rather timid attempts at integrating behavioural insights into EU policymaking and identifies some domestic experiences. Section 5 discusses the institutional and methodological efforts undertaken by the EU and some of its member states to embrace behavioural policymaking. In turn, section 6 discusses the major difficulties of integrating behavioural insights into EU policymaking and offers some concluding remarks….(More)”