Whatever happened to evidence-based policy making?

Curated on Posted on by Stefaan Verhulst

Speech by Professor Gary Banks: “One of the challenges in talking about EBPM (evidence-based policy making), which I had not fully appreciated last time, was that it means different things to different people, especially academics. As a result, disagreements, misunderstandings and controversies (or faux controversies) have abounded. And these may have contributed to the demise of the expression, if not the concept.

For example, some have interpreted the term EBPM so literally as to insist that the word “based” be replaced by “influenced”, arguing that policy decisions are rarely based on evidence alone. That of course is true, but few using the term (myself included) would have thought otherwise. And I am sure no-one in an audience such as this, especially in our nation’s capital, believes policy decisions could derive solely from evidence — or even rational analysis!

If you’ll pardon a quotation from my earlier address: “Values, interests, personalities, timing, circumstance and happenstance – in short, democracy – determine what actually happens” (EBPM: What is it? How do we get it?). Indeed it is precisely because of such multiple influences, that “evidence” has a potentially significant role to play.

So, adopting the position from Alice in Wonderland, I am inclined to stick with the term EBPM, which I choose to mean an approach to policy-making that makes systematic provision for evidence and analysis. Far from the deterministic straw man depicted in certain academic articles, it is an approach that seeks to achieve policy decisions that are better informed in a substantive sense, accepting that they will nevertheless ultimately be – and in a democracy need to be — political in nature.

A second and more significant area of debate concerns the meaning and value of “evidence” itself. There are a number of strands involved.

Evidentiary elitism?

One relates to methodology, and can be likened to the differences between the thresholds for a finding of guilt under civil and criminal law (“balance of probabilities” versus “beyond reasonable doubt”).

Some analysts have argued that, to be useful for policy, evidence must involve rigorous unbiased research techniques, the “gold standard” for which are “randomized control trials”. The “randomistas”, to use the term which headlines Andrew Leigh’s new book (Leigh, 2018), claim that only such a methodology is able to truly tell us “what works”

However adopting this exacting standard from the medical research world would leave policy makers with an excellent tool of limited application. Its forte is testing a specific policy or program relative to business as usual, akin to drug tests involving a placebo for a control group. And there are some inspiring examples of insights gained. But for many areas of public policy the technique is not practicable. Even where it is, it requires that a case has to some extent already been made. And while it can identify the extent to which a particular program “works”, it is less useful for understanding why, or whether something else might work even better.

That is not to say that any evidence will do. Setting the quality bar too low is the bigger problem in practice and the notion of a hierarchy of methodologies is helpful. However, no such analytical tools are self-sufficient for policy-making purposes and in my view are best thought of as components of a “cost benefit framework” – one that enables comparisons of different options, employing those estimation techniques that are most fit for purpose. Though challenging to populate fully with monetized data, CBA provides a coherent conceptual basis for assessing the net social impacts of different policy choices – which is what EBPM must aspire to as its contribution to (political) policy decisions….(More)”.

The democratic potential of civic applications

Curated on Posted on by Stefaan Verhulst

Paper by Jäske, Maija and Ertiö, Titiana: “Recently, digital democratic applications have increased in presence and scope. This study clarifies how civic applications – bottom-up technologies that use open data to solve governance and policy challenges – can contribute to democratic governance. While civic applications claim to deepen democracy, systematic frameworks for assessing the democratic potential of civic apps are missing, because apps are often evaluated against technical criteria. This study introduces a framework for evaluating the democratic potential of civic apps, distinguishing six criteria: inclusiveness, deliberation, influence, publicity, mobilization, and knowledge production. The framework is applied to a case study of the Finnish DataDemo competition in 2014 by analyzing the institutional design features of six civic applications. It is argued that in terms of democratic governance, the greatest potential of civic apps lies in enhancing publicity and mobilization, while they should not be expected to increase inclusiveness or direct influence in decisions. Thus, our study contributes to understanding how civic applications can improve democracy in times of open data abundance….(More)”.

Gradually, Then Suddenly

Curated on Posted on by Stefaan Verhulst

Blogpost by Tim O’Reilly: “There’s a passage in Ernest Hemingway’s novel The Sun Also Rises in which a character named Mike is asked how he went bankrupt. “Two ways,” he answers. “Gradually, then suddenly.”

Technological change happens in much the same way. Small changes accumulate, and suddenly the world is a different place. Throughout my career at O’Reilly Media, we’ve tracked and fostered a lot of “gradually, then suddenly” movements: the World Wide Web, open source software, big data, cloud computing, sensors and ubiquitous computing, and now the pervasive effects of AI and algorithmic systems on society and the economy.

What are some of the things that are in the middle of their “gradually, then suddenly” transition right now? The list is long; here are a few of the areas that are on my mind.

1) AI and algorithms are everywhere

The most important trend for readers of this newsletter to focus on is the development of new kinds of partnership between human and machine. We take for granted that algorithmic systems do much of the work at online sites like Google, Facebook, Amazon, and Twitter, but we haven’t fully grasped the implications. These systems are hybrids of human and machine. Uber, Lyft, and Amazon Robotics brought this pattern to the physical world, reframing the corporation as a vast, buzzing network of humans both guiding and guided by machines. In these systems, the algorithms decide who gets what and why; they’re changing the fundamentals of market coordination in ways that gradually, then suddenly, will become apparent.

2) The rest of the world is leapfrogging the US

The volume of mobile payments in China is $13 trillion versus the US’s $50 billion, while credit cards never took hold. Already Zipline’s on-demand drones are delivering 20% of all blood supplies in Rwanda and will be coming soon to other countries (including the US). In each case, the lack of existing infrastructure turned out to be an advantage in adopting a radically new model. Expect to see this pattern recur, as incumbents and old thinking hold back the adoption of new models..…

9) The crisis of faith in government

Ever since Jennifer Pahlka and I began working on the Gov 2.0 Summit back in 2008, we’ve been concerned that if we can’t get government up to speed on 21st century technology, a critical pillar of the good society will crumble. When we started that effort, we were focused primarily on government innovation; over time, through Jen’s work at Code for America and the United States Digital Service, that shifted to a focus on making sure that government services actually work for those who need them most. Michael Lewis’s latest book, The Fifth Risk, highlights just how bad things might get if we continue to neglect and undermine the machinery of government. It’s not just the political fracturing of our country that should concern us; it’s the fact that government plays a critical role in infrastructure, in innovation, and in the safety net. That role has gradually been eroded, and the cracks that are appearing in the foundation of our society are coming at the worst possible time….(More)”.

Political Selection and Bureaucratic Productivity

Curated on Posted on by Stefaan Verhulst

Paper by James P. Habyarimana et al: “Economic theory of public bureaucracies as complex organizations predicts that bureaucratic productivity can be shaped by the selection of different types of agents, beyond their incentives. This theory applies to the institutions of local government in the developing world, where nationally appointed bureaucrats and locally elected politicians together manage the implementation of public policies and the delivery of services. Yet, there is no evidence on whether (which) selection traits of these bureaucrats and politicians matter for the productivity of local bureaucracies.

This paper addresses the empirical gap by gathering rich data in an institutional context of district governments in Uganda, which is typical of the local state in poor countries. The paper measures traits such as the integrity, altruism, personality, and public service motivation of bureaucrats and politicians. It finds robust evidence that higher integrity among locally elected politicians is associated with substantively better delivery of public health services by district bureaucracies. Together with the theory, this evidence suggests that policy makers seeking to build local state capacity in poor countries should take political selection seriously….(More)”.

Defining subnational open government: does local context influence policy and practice?

Curated on Posted on by Stefaan Verhulst

M. Chatwin, G. Arku and E. Cleave in Policy Sciences: “What is open government? The contemporary conceptualization of open government remains rooted in transparency and accountability, but it is embedded within the political economy of policy, where forces of globalization through supranational organizations strongly influence the creation and dispersion of policy across the globe. Recognizing the direct impact of subnational governments on residents, in 2016 the Open Government Partnership (OGP) launched the Subnational Pioneer’s Pilot Project with 15 participating government authorities globally. Each subnational participant submitted an action plan for opening their government information and processes in 2017. The uniformity of the OGP action plan provides a unique opportunity to assess the conception of open government at the subnational level globally. This paper uses a document analysis to examine how open government is conceptualized at the subnational level, including the salience of various components, and how local context can influence the development of action plans that are responsive to the realities of each participating jurisdiction. This paper assesses whether being a part of the political economy of policy homogenizes the action plans of 15 subnational governments or allows for local context to influence the design of commitments still aligned within a general theme….(More)”.

Trusting Nudges: Toward A Bill of Rights for Nudging

Curated on Posted on by Stefaan Verhulst

Book by Cass R. Sunstein and Lucia A. Reisch: “Many “nudges” aim to make life simpler, safer, or easier for people to navigate, but what do members of the public really think about these policies? Drawing on surveys from numerous nations around the world, Sunstein and Reisch explore whether citizens approve of nudge policies. Their most important finding is simple and striking. In diverse countries, both democratic and nondemocratic, strong majorities approve of nudges designed to promote health, safety, and environmental protection—and their approval cuts across political divisions.

In recent years, many governments have implemented behaviorally informed policies, focusing on nudges—understood as interventions that preserve freedom of choice, but that also steer people in certain directions. In some circles, nudges have become controversial, with questions raised about whether they amount to forms of manipulation. This fascinating book carefully considers these criticisms and answers important questions. What do citizens actually think about behaviorally informed policies? Do citizens have identifiable principles in mind when they approve or disapprove of the policies? Do citizens of different nations agree with each other?

From the answers to these questions, the authors identify six principles of legitimacy—a “bill of rights” for nudging that build on strong public support for nudging policies around the world, while also recognizing what citizens disapprove of. Their bill of rights is designed to capture citizens’ central concerns, reflecting widespread commitments to freedom and welfare that transcend national boundaries….(More)”.

Agile research

Curated on Posted on by Stefaan Verhulst

Michael Twidale and Preben Hansen at First Monday: “Most of us struggle when starting a new research project, even if we have considerable prior experience. It is a new topic and we are unsure about what to do, how to do it and what it all means. We may not have reflected much on our research process. Furthermore the way that research is described in the literature can be rather disheartening. Those papers describe what seems to be a nice, clear, linear, logical, even inevitable progression through a series of stages. It seems like proper researchers carefully plan everything out in advance and then execute that plan. How very different from the mess, the bewilderment, the false starts, the dead ends, the reversions and changes that we make along the way. Are we just doing research wrong? If it feels like that to established researchers with decades of experience and a nice publication record, how much worse must it feel to a new researcher, such as a Ph.D. student? If they are lucky they may have a good mentoring experience, effectively serving an apprenticeship with a wise and nurturing adviser in a supportive group of fellow researchers. Even so, it can be all too easy to feel like an imposter who must be doing it all wrong because what you are doing is not at all like what you read about what others are doing.

In the light of these confusions, fears, doubts and mismatches with what you experience while doing research and what you think is the right and proper way as alluded to in all the papers you read, we want to explore ideas around a title, or at least a provocative metaphor of “agile research”. We want to ask the question: “how might we take the ideas, the methods and the underlying philosophy behind agile software development and explore how these might be applied in the context of doing research?” This paper is not about sharing a set of methods that we have developed but more about provoking a discussion about the issue: What might agile research be like? How might it work? When might it be useful? When might it be problematic? Is it worth trying? Are people doing it already?

We are not claiming that this idea is wholly new. Many people have been using small scale rapid iterative methods within the research process for a long time. Rather we think that it can be useful to consider all these and other possible methods in the light of the successful deployment of agile software development processes, and to contrast them with more conventional research processes that rely more on careful advance planning. That is not to say that the latter methods are bad, just that other methods that might be characterized as more agile can be useful in particular circumstances.

We believe that it is worth exploring this idea as a way of addressing the problems that arise in trying to do a new research project, especially where an exploratory approach is useful. This could be in a domain that is new to the researcher, or where the domain is new in some way, such as involving new use contexts, new ways of interacting, new technologies, novel technology combinations, or new appropriations of existing technologies. We suspect this may be especially useful in helping new researchers such as PhD students get a better understanding of the research process in a less daunting manner. This work builds on prior thinking about how agile may be applied in university teaching and administration (Twidale and Nichols, 2013)….(More)”.

Efficacious and Ethical Public Paternalism

Curated on Posted on by Stefaan Verhulst

Daniel M. Hausman in the Review of Behavioral Economics (Special Issue on Behavioral Economics and New Paternalism): “People often make bad judgments. A big brother or sister who was wise, well-informed, and properly-motivated could often make better decisions for almost everyone. But can governments, which are not staffed with ideal big brothers or sisters, improve upon the mediocre decisions individuals make? If so, when and how? The risks of extending the reach of government into guiding individual lives must also be addressed. This essay addresses three questions concerning when paternalistic policies can be efficacious, efficient, and safe: 1. In what circumstances can policy makers be confident that they know better than individuals how individuals can best promote their own well-being? 2. What are the methods governments can use to lead people to make decision that are better for themselves? 3. What are the moral pluses and minuses of these methods? Answering these questions defines a domain in which paternalistic policy is an attractive option….(More)”.

Participatory Design for Innovation in Access to Justice

Curated on Posted on by Stefaan Verhulst

Margaret Hagan at Daedalus: “Most access-to-justice technologies are designed by lawyers and reflect lawyers’ perspectives on what people need. Most of these technologies do not fulfill their promise because the people they are designed to serve do not use them. Participatory design, which was developed in Scandinavia as a process for creating better software, brings end users and other stakeholders into the design process to help decide what problems need to be solved and how. Work at the Stanford Legal Design Lab highlights new insights about what tools can provide the assistance that people actually need, and about where and how they are likely to access and use those tools. These participatory design models lead to more effective innovation and greater community engagement with courts and the legal system.

A decade into the push for innovation in access to justice, most efforts reflect the interests and concerns of courts and lawyers rather than the needs of the people the innovations are supposed to serve. New legal technologies and services, whether aiming to help people expunge their criminal records or to get divorced in more cooperative ways, have not been adopted by the general public. Instead, it is primarily lawyers who use them.1

One way to increase the likelihood that innovations will serve clients would be to involve clients in designing them. Participatory design emerged in Scandinavia in the 1970s as a way to think more effectively about decision-making in the workplace.  It evolved into a strategy for developing software in which potential users were invited to help define a vision of a product, and it has since been widely used for changing systems like elementary education, hospital services, and smart cities, which use data and technology to improve sustainability and foster economic development.3

Participatory design’s promise is that “system innovation” is more likely to be effective in producing tools that the target group will use and in spending existing resources efficiently to do so. Courts spend an enormous amount of money on information technology every year. But the technology often fails to meet courts’ goals: barely half of the people affected are satisfied with courts’ customer service….(More)”.

Firm Led by Google Veterans Uses A.I. to ‘Nudge’ Workers Toward Happiness

Curated on Posted on by Stefaan Verhulst

Daisuke Wakabayashi in the New York Times: “Technology companies like to promote artificial intelligence’s potential for solving some of the world’s toughest problems, like reducing automobile deaths and helping doctors diagnose diseases. A company started by three former Google employees is pitching A.I. as the answer to a more common problem: being happier at work.

The start-up, Humu, is based in Google’s hometown, and it builds on some of the so-called people-analytics programs pioneered by the internet giant, which has studied things like the traits that define great managers and how to foster better teamwork.

Humu wants to bring similar data-driven insights to other companies. It digs through employee surveys using artificial intelligence to identify one or two behavioral changes that are likely to make the biggest impact on elevating a work force’s happiness. Then it uses emails and text messages to “nudge” individual employees into small actions that advance the larger goal.

At a company where workers feel that the way decisions are made is opaque, Humu might nudge a manager before a meeting to ask the members of her team for input and to be prepared to change her mind. Humu might ask a different employee to come up with questions involving her team that she would like to have answered.

At the heart of Humu’s efforts is the company’s “nudge engine” (yes, it’s trademarked). It is based on the economist Richard Thaler’s Nobel Prize-winning research into how people often make decisions because of what is easier rather than what is in their best interest, and how a well-timed nudge can prompt them to make better choices.

Google has used this approach to coax employees into the corporate equivalent of eating their vegetables, prodding them to save more for retirement, waste less food at the cafeteria and opt for healthier snacks….

But will workers consider the nudges useful or manipulative?

Todd Haugh, an assistant professor of business law and ethics at Indiana University’s Kelley School of Business, said nudges could push workers into behaving in ways that benefited their employers’ interests over their own.

“The companies are the only ones who know what the purpose of the nudge is,” Professor Haugh said. “The individual who is designing the nudge is the one whose interests are going to be put in the forefront.”…(More)”.