Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

Curated on Posted on by Stefaan Verhulst

Book by Ron Kohavi, Diane Tang, and Ya Xu: “Getting numbers is easy; getting numbers you can trust is hard. This practical guide by experimentation leaders at Google, LinkedIn, and Microsoft will teach you how to accelerate innovation using trustworthy online controlled experiments, or A/B tests. Based on practical experiences at companies that each run more than 20,000 controlled experiments a year, the authors share examples, pitfalls, and advice for students and industry professionals getting started with experiments, plus deeper dives into advanced topics for practitioners who want to improve the way they make data-driven decisions.

Learn how to use the scientific method to evaluate hypotheses using controlled experiments Define key metrics and ideally an Overall Evaluation Criterion Test for trustworthiness of the results and alert experimenters to violated assumptions. Build a scalable platform that lowers the marginal cost of experiments close to zero. Avoid pitfalls like carryover effects and Twyman’s law. Understand how statistical issues play out in practice….(More)”.

The Law and Economics of Online Republication

Curated on Posted on by Stefaan Verhulst

Paper by Ronen Perry: “Jerry publishes unlawful content about Newman on Facebook, Elaine shares Jerry’s post, the share automatically turns into a tweet because her Facebook and Twitter accounts are linked, and George immediately retweets it. Should Elaine and George be liable for these republications? The question is neither theoretical nor idiosyncratic. On occasion, it reaches the headlines, as when Jennifer Lawrence’s representatives announced she would sue every person involved in the dissemination, through various online platforms, of her illegally obtained nude pictures. Yet this is only the tip of the iceberg. Numerous potentially offensive items are reposted daily, their exposure expands in widening circles, and they sometimes “go viral.”

This Article is the first to provide a law and economics analysis of the question of liability for online republication. Its main thesis is that liability for republication generates a specter of multiple defendants which might dilute the originator’s liability and undermine its deterrent effect. The Article concludes that, subject to several exceptions and methodological caveats, only the originator should be liable. This seems to be the American rule, as enunciated in Batzel v. Smith and Barrett v. Rosenthal. It stands in stark contrast to the prevalent rules in other Western jurisdictions and has been challenged by scholars on various grounds since its very inception.

The Article unfolds in three Parts. Part I presents the legal framework. It first discusses the rules applicable to republication of self-created content, focusing on the emergence of the single publication rule and its natural extension to online republication. It then turns to republication of third-party content. American law makes a clear-cut distinction between offline republication which gives rise to a new cause of action against the republisher (subject to a few limited exceptions), and online republication which enjoys an almost absolute immunity under § 230 of the Communications Decency Act. Other Western jurisdictions employ more generous republisher liability regimes, which usually require endorsement, a knowing expansion of exposure or repetition.

Part II offers an economic justification for the American model. Law and economics literature has showed that attributing liability for constant indivisible harm to multiple injurers, where each could have single-handedly prevented that harm (“alternative care” settings), leads to dilution of liability. Online republication scenarios often involve multiple tortfeasors. However, they differ from previously analyzed phenomena because they are not alternative care situations, and because the harm—increased by the conduct of each tortfeasor—is not constant and indivisible. Part II argues that neither feature precludes the dilution argument. It explains that the impact of the multiplicity of injurers in the online republication context on liability and deterrence provides a general justification for the American rule. This rule’s relatively low administrative costs afford additional support.

Part III considers the possible limits of the theoretical argument. It maintains that exceptions to the exclusive originator liability rule should be recognized when the originator is unidentifiable or judgment-proof, and when either the republisher’s identity or the republication’s audience was unforeseeable. It also explains that the rule does not preclude liability for positive endorsement with a substantial addition, which constitutes a new original publication, or for the dissemination of illegally obtained content, which is an independent wrong. Lastly, Part III addresses possible challenges to the main argument’s underlying assumptions, namely that liability dilution is a real risk and that it is undesirable….(More)”.

A controlled trial for reproducibility

Curated on Posted on by Stefaan Verhulst

Marc P. Raphael, Paul E. Sheehan & Gary J. Vora at Nature: “In 2016, the US Defense Advanced Research Projects Agency (DARPA) told eight research groups that their proposals had made it through the review gauntlet and would soon get a few million dollars from its Biological Technologies Office (BTO). Along with congratulations, the teams received a reminder that their award came with an unusual requirement — an independent shadow team of scientists tasked with reproducing their results.

Thus began an intense, multi-year controlled trial in reproducibility. Each shadow team consists of three to five researchers, who visit the ‘performer’ team’s laboratory and often host visits themselves. Between 3% and 8% of the programme’s total funds go to this independent validation and verification (IV&V) work. But DARPA has the flexibility and resources for such herculean efforts to assess essential techniques. In one unusual instance, an IV&V laboratory needed a sophisticated US$200,000 microscopy and microfluidic set-up to make an accurate assessment.

These costs are high, but we think they are an essential investment to avoid wasting taxpayers’ money and to advance fundamental research towards beneficial applications. Here, we outline what we’ve learnt from implementing this programme, and how it could be applied more broadly….(More)”.

How Civic Technology Can Help Stop a Pandemic

Curated on Posted on by Stefaan Verhulst

Jaron Lanier and E. Glen Weyl at Foreign Affairs: “The spread of the novel coronavirus and the resulting COVID-19 pandemic have provided a powerful test of social and governance systems. Neither of the world’s two leading powers, China and the United States, has been particularly distinguished in responding. In China, an initial bout of political denial allowed the virus to spread for weeks, first domestically and then globally, before a set of forceful measures proved reasonably effective. (The Chinese government also should have been better prepared, given that viruses have jumped from animal hosts to humans within its territory on multiple occasions in the past.) The United States underwent its own bout of political denial before adopting social-distancing policies; even now, its lack of investment in public health leaves it ill-equipped for this sort of emergency.

The response of the bureaucratic and often technophobic European Union may prove even worse: Italy, although far from the epicenter of the outbreak, has four times the per capita rate of cases as China does, and even famously orderly Germany is already at half China’s rate. Nations in other parts of the world, such as information-manipulating Iran, provide worse examples yet.

Focusing on the countries that have done worst, however, may be less useful at this point than considering which country has so far done best: Taiwan. Despite being treated by the World Health Organization as part of China, and despite having done far broader testing than the United States (meaning the true rate of infection is far less hidden), Taiwan has only one-fifth the rate of known cases in the United States and less than one-tenth the rate in widely praised Singapore. Infections could yet spike again, especially with the global spread making visitors from around the world vectors of the virus. Yet the story of Taiwan’s initial success is worth sharing not just because of its lessons for containing the present pandemic but also because of its broader lessons about navigating pressing challenges around technology and democracy.

Taiwan’s success has rested on a fusion of technology, activism, and civic participation. A small but technologically cutting-edge democracy, living in the shadow of the superpower across the strait, Taiwan has in recent years developed one of the world’s most vibrant political cultures by making technology work to democracy’s advantage rather than detriment. This culture of civic technology has proved to be the country’s strongest immune response to the new coronavirus….(More)”.

The world after coronavirus

Curated on Posted on by Stefaan Verhulst

Yuval Noah Harari at the Financial Times: “Humankind is now facing a global crisis. Perhaps the biggest crisis of our generation. The decisions people and governments take in the next few weeks will probably shape the world for years to come. They will shape not just our healthcare systems but also our economy, politics and culture. We must act quickly and decisively. We should also take into account the long-term consequences of our actions.

When choosing between alternatives, we should ask ourselves not only how to overcome the immediate threat, but also what kind of world we will inhabit once the storm passes. Yes, the storm will pass, humankind will survive, most of us will still be alive — but we will inhabit a different world.  Many short-term emergency measures will become a fixture of life. That is the nature of emergencies. They fast-forward historical processes.

Decisions that in normal times could take years of deliberation are passed in a matter of hours. Immature and even dangerous technologies are pressed into service, because the risks of doing nothing are bigger. Entire countries serve as guinea-pigs in large-scale social experiments. What happens when everybody works from home and communicates only at a distance? What happens when entire schools and universities go online? In normal times, governments, businesses and educational boards would never agree to conduct such experiments. But these aren’t normal times. 

In this time of crisis, we face two particularly important choices. The first is between totalitarian surveillance and citizen empowerment. The second is between nationalist isolation and global solidarity. 

Under-the-skin surveillance

In order to stop the epidemic, entire populations need to comply with certain guidelines. There are two main ways of achieving this. One method is for the government to monitor people, and punish those who break the rules. Today, for the first time in human history, technology makes it possible to monitor everyone all the time. Fifty years ago, the KGB couldn’t follow 240m Soviet citizens 24 hours a day, nor could the KGB hope to effectively process all the information gathered. The KGB relied on human agents and analysts, and it just couldn’t place a human agent to follow every citizen. But now governments can rely on ubiquitous sensors and powerful algorithms instead of flesh-and-blood spooks. 

In their battle against the coronavirus epidemic several governments have already deployed the new surveillance tools. The most notable case is China. By closely monitoring people’s smartphones, making use of hundreds of millions of face-recognising cameras, and obliging people to check and report their body temperature and medical condition, the Chinese authorities can not only quickly identify suspected coronavirus carriers, but also track their movements and identify anyone they came into contact with. A range of mobile apps warn citizens about their proximity to infected patients…

If I could track my own medical condition 24 hours a day, I would learn not only whether I have become a health hazard to other people, but also which habits contribute to my health. And if I could access and analyse reliable statistics on the spread of coronavirus, I would be able to judge whether the government is telling me the truth and whether it is adopting the right policies to combat the epidemic. Whenever people talk about surveillance, remember that the same surveillance technology can usually be used not only by governments to monitor individuals — but also by individuals to monitor governments. 

The coronavirus epidemic is thus a major test of citizenship….(More)”.

Big Data and Democracy

Curated on Posted on by Stefaan Verhulst

Paper by Freek van Gils, Wieland Müller and Jens Prufer: “Recent technological developments have raised concerns about threats to democracy because of their potential to distort election outcomes: (a) data-driven voter research enabling political microtargeting, and (b) growing news consumption via social media and news aggregators that obfuscate the origin of news items, leading to voters’ unawareness about a news sender’s identity. We provide a theoretical framework in which we can analyze the effects that microtargeting by political interest groups and unawareness have on election outcomes in comparison to \conventional” news reporting. We show which voter groups suffer from which technological development, (a) or (b). While both microtargeting and unawareness have negative effects on voter welfare, we show that only unawareness can flip an election. Our model framework allows the theory-based discussion of policy proposals, such as to ban microtargeting or to require news platforms to signal the political orientation of a news item’s originator…(More)”.

The Coronavirus Tech Handbook

Curated on Posted on by Stefaan Verhulst

About: “The Coronavirus Tech Handbook provides a space for technologists, specialists, civic organisations and public & private institutions to collaborate on a rapid and sophisticated response to the coronavirus outbreak. It is an active and evolving resource with thousands of expert contributors.

In less than two weeks it has grown to cover areas including:

  • Detailed guidance for doctors and nurses,
  • Advice and tools for educators adjusting to remote teaching, 
  • Community of open-source ventilator designers
  • Comprehensive data and models for forecasting the spread of the virus.

Coronavirus Tech Handbook’s goal is to create a rapidly evolving open source technical knowledge base that will help all institutions across civil society and the public sector collaborate to fight the outbreak. 

Coronavirus Tech Handbook is not a place for the public to get advice, but a place for specialists to collaborate and make sure the best solutions are quickly shared and deployed….(More)”.

Like a moth to a flame, we’re drawn to metaphors to explain ourselves

Curated on Posted on by Stefaan Verhulst

Kenan Malik at The Guardian: “The selfish gene. The Big Bang. The greenhouse effect. Metaphors are at the heart of scientific thinking. They provide the means for both scientists and non-scientists to understand, think through and talk about abstract ideas in terms of more familiar objects or phenomena.

But if metaphors can illuminate, they can also constrain. In his new book, The Idea of the Brain, zoologist and historian Matthew Cobb tells the story of how scientists and philosophers have tried to understand the brain and how it works. In every age, Cobb shows, people have thought about the brain largely in terms of metaphors, drawn usually from the most exciting technology of the day, whether clocks or telephone exchanges or the contemporary obsession with computers. The brain, Cobb observes, “is more like a computer than like a clock”, but “even the simplest animal brain is not a computer like anything we have built, nor one we can yet envisage”.

Metaphors allow “insight and discovery” but are “inevitably partial” and “there will come a point when the understanding they allow will be outweighed by the limits they impose”. We may, Cobb suggests, be at that point in picturing the brain as a computer.

The paradox of neuroscience today is that we possess an unprecedented amount of data about the brain but barely a glimmer of a theory to explain how it works. Indeed, as the French neuroscientist Yves Frégnac has put it, making ample use of metaphor, it can feel as if “we are drowning in a flood of information” and that “all sense of global understanding [of brain function] is in acute danger of being washed away”.

It’s not just in science that metaphors are significant in shaping the ways in which we think. In 1980, the linguist George Lakoff and philosopher Mark Johnson set off the modern debate on this issue with their seminal work, Metaphors We Live By. Metaphors, they argued, are not linguistic flourishes but the fundamental building blocks of thought. We don’t simply talk or write with metaphors, we also think with them….(More)”

World Justice Project (WJP) Rule of Law Index®

Curated on Posted on by Stefaan Verhulst

Interactive Overview: “The World Justice Project (WJP) Rule of Law Index® is the world’s leading source for original, independent data on the rule of law. Now covering 128 countries and jurisdictions, the Index relies on national surveys of more than 130,000 households and 4,000 legal practitioners and experts to measure how the rule of law is experienced and perceived around the world.

Effective rule of law reduces corruption, combats poverty and disease, and protects people from injustices large and small. It is the foundation for communities of justice, opportunity, and peace—underpinning development, accountable government, and respect for fundamental rights.

Learn more about the rule of law and explore the full WJP Rule of Law Index 2020 report, including PDF report download, data insights, methodology, and more at the Index report resources page….(More)”

Testing Transparency

Curated on Posted on by Stefaan Verhulst

Paper by Brigham Daniels, Mark Buntaine & Tanner Bangerter: “In modern democracies, governmental transparency is thought to have great value. When it comes to addressing administrative corruption and mismanagement, many would agree with Justice Brandeis’s observation
that sunlight is the best disinfectant. Beyond this, many credit transparency with enabling meaningful citizen participation.

But even though transparency appears highly correlated with successful
governance in developed democracies, assumptions about administrative
transparency have remained empirically untested. Testing effects of transparency would prove particularly helpful in developing democracies
where transparency norms have not taken hold or only have done so slowly.

In these contexts, does administrative transparency really create the sorts of benefits attributed to it? Transparency might grease the gears of developed democracies, but what good is grease when many of the gears seem to be broken or missing entirely?

This Article presents empirical results from a first-of-its-kind field study that tested two major promises of administrative transparency in a developing democracy: that transparency increases public participation in government affairs and that it increases government accountability. To test these hypotheses, we used two randomized controlled trials.

Surprisingly, we found transparency had no significant effect in almost
any of our quantitative measurements, although our qualitative results
suggested that when transparency interventions exposed corruption, some
limited oversight could result. Our findings are particularly significant for
developing democracies and show, at least in this context, that Justice
Brandeis may have oversold the cleansing effects of transparency.

A few rays of transparency shining light on government action do not disinfect the system and cure government corruption and mismanagement. Once corruption and mismanagement are identified, it takes effective government institutions and action from civil society to successfully act as a disinfectant…(More)”.