Ten ways to optimise evidence-based policy


Paper by Peter Bragge: “Applying knowledge to problems has occupied the minds of great philosophers, scientists and other thinkers for centuries. In more modern times, the challenge of connecting knowledge to practice has been addressed through fields such as evidence-based medicine which have conceptualised optimal healthcare as integration of best available research evidence, clinical experience and patients’ values. Similar principles apply to evidence-based public policy, and literature in this field has been growing since the turn of the century.

The exponential rise in knowledge availability has greatly enhanced the ‘supply’ side of the evidence-into-practice equation – however substantial gaps between evidence and practice remain. Policymakers are therefore increasingly looking to academia to optimise evidence-informed policy. This article presents ten considerations for optimising evidence-based policy, drawn from experience in delivering applied behaviour change research to government….(More)”.

Retrofitting Social Science for the Practical & Moral


Kenneth Prewitt at Issues: “…We cannot reach this fresh thinking without first challenging two formulations that today’s social science considers settled. First, social science should not assume that the “usefulness of useless knowledge” works as our narrative. Yes, it works for natural sciences. But the logic doesn’t translate. Second, we should back off from exaggerated promises about “evidence-based policy,” perhaps terming it “evidence-influenced politics,” a framing that is more accurate descriptively (what happens) and prescriptively (what should happen). The prominence given to these two formulations gets in the way of an alternative positioning of social science as an agent of improvement. I discuss this alternative below, under the label of the Fourth Purpose….

…the “Fourth Purpose.” This joins the three purposes traditionally associated with American universities and colleges: Education, Research, and Public Service. The latter is best described as being “a good citizen,” engaged in volunteer work; it is an attractive feature of higher education, but not in any substantial manner present in the other two core purposes.

The Fourth Purpose is an altogether different vision. It institutionalizes what Ross characterized as a social science being in the “broadest sense practical and moral.” It succeeds only by being fully present in education and research, for instance, including experiential learning in the curriculum and expanding processes that convert research findings into social benefits. This involves more than scattered centers across the university working on particular social problems. As Bollinger puts it, the university itself becomes a hybrid actor, at once academic and practical. “A university,” he says, “is more than simply an infrastructure supporting schools, departments, and faculty in their academic pursuits. As research universities enter into the realm or realms of the outside world, the ‘university’ (i.e., the sum of its parts/constituents) is going to have capacities far beyond those of any segment, as well as effects (hopefully generally positive) radiating back into the institution.”

To oversimplify a bit, the Fourth Purpose has three steps. The first occurs in the lab, library, or field—resulting in fundamental findings. The second ventures into settings where nonacademic players and judgment come into play, actions are taken, and ethical choices confronted, that is, practices of the kind mentioned earlier: translation research, knowledge brokers, boundary organizations, coproduction. Academic and nonacademic players should both come away from these settings with enriched understanding and capabilities. For academics, the skills required for this step differ from, but complement, the more familiar skills of teacher and researcher. The new skills will have to be built into the fabric of the university if the Fourth Purpose is to succeed.

The third step cycles back to the campus. It involves scholarly understandings not previously available. It requires learning something new about the original research findings as a result of how they are interpreted, used, rejected, modified, or ignored in settings that, in fact, are controlling whether the research findings will be implemented as hoped. This itself is new knowledge. If paid attention to, and the cycle is repeated, endlessly, a new form of scholarship is added to our tool kit….(More)”.

OMB rethinks ‘protected’ or ‘open’ data binary with upcoming Evidence Act guidance


Jory Heckman at Federal News Network: “The Foundations for Evidence-Based Policymaking Act has ordered agencies to share their datasets internally and with other government partners — unless, of course, doing so would break the law.

Nearly a year after President Donald Trump signed the bill into law, agencies still have only a murky idea of what data they can share, and with whom. But soon, they’ll have more nuanced options of ranking the sensitivity of their datasets before sharing them out to others.

Chief Statistician Nancy Potok said the Office of Management and Budget will soon release proposed guidelines for agencies to provide “tiered” access to their data, based on the sensitivity of that information….

OMB, as part of its Evidence Act rollout, will also rethink how agencies ensure protected access to data for research. Potok said agency officials expect to pilot a single application governmentwide for people seeking access to sensitive data not available to the public.

The pilot resembles plans for a National Secure Data Service envisioned by the Commission on Evidence-Based Policymaking, an advisory group whose recommendations laid the groundwork for the Evidence Act.

“As a state-of-the-art resource for improving government’s capacity to use the data it already collects, the National Secure Data Service will be able to temporarily link existing data and provide secure access to those data for exclusively statistical purposes in connection with approved projects,” the commission wrote in its 2017 final report.

In an effort to strike a balance between access and privacy, Potok said OMB has also asked agencies to provide a list of the statutes that prohibit them from sharing data amongst themselves….(More)”.

To What Extent Does the EU General Data Protection Regulation (GDPR) Apply to Citizen Scientist-led Health Research with Mobile Devices?


Article by Edward Dove and Jiahong Chen: “In this article, we consider the possible application of the European General Data Protection Regulation (GDPR) to “citizen scientist”-led health research with mobile devices. We argue that the GDPR likely does cover this activity, depending on the specific context and the territorial scope. Remaining open questions that result from our analysis lead us to call for a lex specialis that would provide greater clarity and certainty regarding the processing of health data for research purposes, including by these non-traditional researchers…(More)”.

Becoming a data steward


Shalini Kurapati at the LSE Impact Blog: “In the context of higher education, data stewards are the first point of reference for all data related questions. In my role as a data steward at TU Delft, I was able to advise, support and train researchers on various aspects of data management throughout the life cycle of a research project, from initial planning to post-publication. This included storing, managing and sharing research outputs such as data, images, models and code.

Data stewards also advise researchers on the ethical, policy and legal considerations during data collection, processing and dissemination. In a way, they are general practitioners for research data management and can usually solve most problems faced by academics. In cases that require specialist intervention, they also serve as a key point for referral (eg: IT, patent, legal experts).

Data stewardship is often organised centrally through the university library. (Subject) Data librarians, research data consultants and research data officers, usually perform similar roles to data stewards. However, TU Delft operates a decentralised model, where data stewards are placed within faculties as disciplinary experts with research experience. This allows data stewards to provide discipline specific support to researchers, which is particularly beneficial, as the concept of what data is itself varies across disciplines….(More)”.

Timing Technology


Blog by Gwern Branwen: “Technological forecasts are often surprisingly prescient in terms of predicting that something was possible & desirable and what they predict eventually happens; but they are far less successful at predicting the timing, and almost always fail, with the success (and riches) going to another.

Why is their knowledge so useless? The right moment cannot be known exactly in advance, so attempts to forecast will typically be off by years or worse. For many claims, there is no way to invest in an idea except by going all in and launching a company, resulting in extreme variance in outcomes, even when the idea is good and the forecasts correct about the (eventual) outcome.

Progress can happen and can be foreseen long before, but the details and exact timing due to bottlenecks are too difficult to get right. Launching too early means failure, but being conservative & launching later is just as bad because regardless of forecasting, a good idea will draw overly-optimistic researchers or entrepreneurs to it like moths to a flame: all get immolated but the one with the dumb luck to kiss the flame at the perfect instant, who then wins everything, at which point everyone can see that the optimal time is past. All major success stories overshadow their long list of predecessors who did the same thing, but got unlucky. So, ideas can be divided into the overly-optimistic & likely doomed, or the fait accompli. On an individual level, ideas are worthless because so many others have them too—‘multiple invention’ is the rule, and not the exception.

This overall problem falls under the reinforcement learning paradigm, and successful approaches are analogous to Thompson sampling/posterior sampling: even an informed strategy can’t reliably beat random exploration which gradually shifts towards successful areas while continuing to take occasional long shots. Since people tend to systematically over-exploit, how is this implemented? Apparently by individuals acting suboptimally on the personal level, but optimally on societal level by serving as random exploration.

A major benefit of R&D, then, is in laying fallow until the ‘ripe time’ when they can be immediately exploited in previously-unpredictable ways; applied R&D or VC strategies should focus on maintaining diversity of investments, while continuing to flexibly revisit previous failures which forecasts indicate may have reached ‘ripe time’. This balances overall exploitation & exploration to progress as fast as possible, showing the usefulness of technological forecasting on a global level despite its uselessness to individuals….(More)”.

Supporting priority setting in science using research funding landscapes


Report by the Research on Research Institute: “In this working paper, we describe how to map research funding landscapes in order to support research funders in setting priorities. Based on data on scientific publications, a funding landscape highlights the research fields that are supported by different funders. The funding landscape described here has been created using data from the Dimensions database. It is presented using a freely available web-based tool that provides an interactive visualization of the landscape. We demonstrate the use of the tool through a case study in which we analyze funding of mental health research…(More)”.

Robotic Bureaucracy: Administrative Burden and Red Tape in University Research


Essay by Barry Bozeman and Jan Youtie: “…examines university research administration and the use of software systems that automate university research grants and contract administration, including the automatic sending of emails for reporting and compliance purposes. These systems are described as “robotic bureaucracy.” The rise of regulations and their contribution to administrative burden on university research have led university administrators to increasingly rely on robotic bureaucracy to handle compliance. This article draws on the administrative burden, behavioral public administration, and electronic communications and management literatures, which are increasingly focused on the psychological and cognitive bases of behavior. These literatures suggest that the assumptions behind robotic bureaucracy ignore the extent to which these systems shift the burden of compliance from administrators to researchers….(More)”.

Why Trust Science?


Book by Naomi Oreskes: “Do doctors really know what they are talking about when they tell us vaccines are safe? Should we take climate experts at their word when they warn us about the perils of global warming? Why should we trust science when our own politicians don’t? In this landmark book, Naomi Oreskes offers a bold and compelling defense of science, revealing why the social character of scientific knowledge is its greatest strength—and the greatest reason we can trust it.

Tracing the history and philosophy of science from the late nineteenth century to today, Oreskes explains that, contrary to popular belief, there is no single scientific method. Rather, the trustworthiness of scientific claims derives from the social process by which they are rigorously vetted. This process is not perfect—nothing ever is when humans are involved—but she draws vital lessons from cases where scientists got it wrong. Oreskes shows how consensus is a crucial indicator of when a scientific matter has been settled, and when the knowledge produced is likely to be trustworthy.

Based on the Tanner Lectures on Human Values at Princeton University, this timely and provocative book features critical responses by climate experts Ottmar Edenhofer and Martin Kowarsch, political scientist Jon Krosnick, philosopher of science Marc Lange, and science historian Susan Lindee, as well as a foreword by political theorist Stephen Macedo….(More)”.

Restricting data’s use: A spectrum of concerns in need of flexible approaches


Dharma Akmon and Susan Jekielek at IASSIST Quaterly: “As researchers consider making their data available to others, they are concerned with the responsible use of data. As a result, they often seek to place restrictions on secondary use. The Research Connections archive at ICPSR makes available the datasets of dozens of studies related to childcare and early education. Of the 103 studies archived to date, 20 have some restrictions on access. While ICPSR’s data access systems were designed primarily to accommodate public use data (i.e. data without disclosure concerns) and potentially disclosive data, our interactions with depositors reveal a more nuanced notion range of needs for restricting use. Some data present a relatively low risk of threatening participants’ confidentiality, yet the data producers still want to monitor who is accessing the data and how they plan to use them. Other studies contain data with such a high risk of disclosure that their use must be restricted to a virtual data enclave. Still other studies rest on agreements with participants that require continuing oversight of secondary use by data producers, funders, and participants. This paper describes data producers’ range of needs to restrict data access and discusses how systems can better accommodate these needs….(More)”.