How spooks are turning to superforecasting in the Cosmic Bazaar


The Economist: “Every morning for the past year, a group of British civil servants, diplomats, police officers and spies have woken up, logged onto a slick website and offered their best guess as to whether China will invade Taiwan by a particular date. Or whether Arctic sea ice will retrench by a certain amount. Or how far covid-19 infection rates will fall. These imponderables are part of Cosmic Bazaar, a forecasting tournament created by the British government to improve its intelligence analysis.

Since the website was launched in April 2020, more than 10,000 forecasts have been made by 1,300 forecasters, from 41 government departments and several allied countries. The site has around 200 regular forecasters, who must use only publicly available information to tackle the 30-40 questions that are live at any time. Cosmic Bazaar represents the gamification of intelligence. Users are ranked by a single, brutally simple measure: the accuracy of their predictions.

Forecasting tournaments like Cosmic Bazaar draw on a handful of basic ideas. One of them, as seen in this case, is the “wisdom of crowds”, a concept first illustrated by Francis Galton, a statistician, in 1907. Galton observed that in a contest to estimate the weight of an ox at a county fair, the median guess of nearly 800 people was accurate within 1% of the true figure.

Crowdsourcing, as this idea is now called, has been augmented by more recent research into whether and how people make good judgments. Experiments by Philip Tetlock of the University of Pennsylvania, and others, show that experts’ predictions are often no better than chance. Yet some people, dubbed “superforecasters”, often do make accurate predictions, largely because of the way they form judgments—such as having a commitment to revising predictions in light of new data, and being aware of typical human biases. Dr Tetlock’s ideas received publicity last year when Dominic Cummings, then an adviser to Boris Johnson, Britain’s prime minister, endorsed his book and hired a controversial superforecaster to work at Mr Johnson’s office in Downing Street….(More)”.

Lawmakers’ use of scientific evidence can be improved


Paper by D. Max Crowley et al: “This study is an experimental trial that demonstrates the potential for formal outreach strategies to change congressional use of research. Our results show that collaboration between policy and research communities can change policymakers’ value of science and result in legislation that appears to be more inclusive of research evidence. The findings of this study also demonstrated changes in researchers’ knowledge and motivation to engage with policymakers as well as their actual policy engagement behavior. Together, the observed changes in both policymakers and researchers randomized to receive an intervention for supporting legislative use of research evidence (i.e., the Research-to-Policy Collaboration model) provides support for the underlying theories around the social nature of research translation and evidence use….(More)”.

2030 Compass CoLab


About: “2030 Compass CoLab invites a group of experts, using an online platform, to contribute their perspectives on potential interactions between the goals in the UN’s 2030 Agenda for Sustainable Development.

By combining the insight of participants who posses broad and diverse knowledge, we hope to develop a richer understanding of how the Sustainable Development Goals (SDGs) may be complementary or conflicting.

Compass 2030 CoLab is part of a larger project, The Agenda 2030 Compass Methodology and toolbox for strategic decision making, funded by Vinnova, Sweden’s government agency for innovation.

Other elements of the larger project include:

  • Deliberations by a panel of experts who will convene in a series of live meetings to undertake in-depth analysis on interactions between the goals. 
  • Quanitative analysis of SDG indicators time series data, which will examine historical correlations between progress on the SDGs.
  • Development of a knowledge repository, residing in a new software tool under development as part of the project. This tool will be made available as a resource to guide the decisions of corporate executives, policy makers, and leaders of NGOs.

The overall project was inspired by the work of researchers at the Stockholm Environment Institute, described in Towards systemic and contextual priority setting for implementing the 2030 Agenda, a 2018 paper in Sustainability Science by Nina Weitz, Henrik Carlsen, Måns Nilsson, and Kristian Skånberg….(More)”.

The War on Professionalism


Jonathan Rauch in National Affairs: “…As a child, I asked my father, a lawyer, what the word “professional” meant. He replied, “it means you do something for a living.” He contrasted it with the term “amateur,” meaning someone who works for pleasure.

My father’s definition has merit. I recall it whenever I tell interns that, to me, professionalism means performing a job to the highest standards, even when I don’t feel like doing it at all. One might think of the doctor who shows up for emergency surgery on Christmas Eve, the journalist who takes care to verify every fact mentioned in a report, or the concert pianist who gives the audience the best he is capable of night after night, even on nights when he would much rather be doing anything else.

That concept of professionalism is a good starting point, but we can dig deeper by drawing on the work of the American Enterprise Institute’s Yuval Levin. In his book A Time to Build, Levin explores the role and meaning of institutions. Institutions, he says, are — or, when they function well, should be — forms, training and shaping people to work together toward a larger goal. The military is a classic example, as are churches and schools. These “structures of social life” provide the durable arrangements that frame our perceptions, mold our character, and delineate our social existence.

When institutions do not or cannot perform those shaping functions, they collapse into something more like platforms — stages upon which individuals perform in order to build audiences and self-advertise. He locates the collapse of trust in institutions — and the resulting public sense of anomie and disconnectedness — in the conversion of many institutions from places where people are formed to places where people perform. Thus a self-promoting real-estate magnate can become a self-promoting reality-TV host and then a self-promoting presidential candidate, hopping from one stage to the next, all while putting on pretty much the same show.

As institutions have drifted away from shaping us and toward displaying us, they have lost both efficacy and legitimacy. And we, in turn, have naturally lost confidence in them. Moreover, Levin argues, institutions have been taken for granted for so long, and yet are neglected so generally, that we have lost even the vocabulary for talking about what they are supposed to be doing. We don’t realize what we are missing, although we acutely feel the void.

Something very much like that has happened with professionalism. A combination of institutional absence, lazy thinking, and populist politics has collapsed the idea of professionalism down to the much flatter notion of elitism.

To some extent, it is natural to think of professionalism and elitism as two sides of the same coin. After all, many professionals are people with advanced degrees and high incomes who occupy elite positions in society. We imagine professionalism to be about excluding others from certain pursuits or occupations like law and medicine — a seemingly elitist practice. We think of it, too, as synonymous with professional schooling, something not everyone can aspire to.

Still, there is a world of difference between professionals and elites. Elites are influential by dint of who they are and whom they know. They are elite because they have social connections and powerful positions. Professionals, by contrast, are influential by dint of what they know and what they do. Their status is contingent on both their standing and their behavior.

That distinction gestures toward a fuller definition of professionalism, one that implies commitment to personal standards, social norms, and expert knowledge in furtherance of a mission or an institution. That is, professionalism defines a right way of doing things — a notion of best practices — that is grounded in dedication to a mission or an institution rather than personal advancement or partisan loyalty. As Levin says, professionalism “tends to yield a strong internal ethos among practitioners. In uncertain situations, a professional asks himself, ‘What should I do here, given my professional responsibilities?’ And his profession will generally have an answer to that question.”

As Levin notices, institutionalism and professionalism are cousins. Both institutions and professions organize individuals to accomplish missions, they seek to inculcate norms and guide behavior, they assemble and transmit knowledge and best practices across generations, they cultivate reputational capital over long spans of time, and they draw and enforce boundaries between insiders and outsiders….(More)”

Conducting City Diplomacy: A Survey of International Engagement in 47 Cities


Paper by Anna Kosovac, Kris Hartley, Michele Acuto, and Darcy Gunning: “The impact of global challenges such as climate change and the COVID-19 pandemic manifests most acutely in urban settings, rendering cities essential players on the global stage.

In the 2018 report Toward City Diplomacy, the Chicago Council on Global Affairs presented findings from a survey of 27 cities on the capacity of local governments around the world to network internationally—and the perceived barriers to that engagement. The report found that cities “need to invest in resources, expertise, and capacity to manage their relationships and responsibilities to conduct city diplomacy effectively.”

In our new survey of 47 cities, we find that advice to still ring true. City officials broadly recognize the importance of engaging internationally but lack the necessary formal diplomacy training and resources for conducting that engagement to maximum effect. Nevertheless, cities maintain a strong commitment to global agendas, and international frameworks are increasingly influential in municipal affairs. For example, more than half of survey respondents said they track their city’s performance against the metrics of the United Nations’ Sustainable Development Goals (SDGs).

Furthermore, we found that cities and their leaders are confident in their capacity to tackle global challenges. For instance, the majority of survey respondents said that city governments have greater potential for impact on climate change mitigation than their national government counterparts do, especially when acting collaboratively through city networks and multilateral urban programs.

The individual stories of five cities whose officials participated in the study offer lessons for a variety of challenges and approaches to city diplomacy. Based on the survey results, we discuss the three primary obstacles cities must overcome in order to strengthen the role of city diplomacy globally: inadequate funding and resources for international engagement, insufficient training in city diplomacy, and the failure of national and multilateral bodies to fully recognize and formalize city engagement in diplomacy.

We conclude with a framework for ensuring that city-diplomacy efforts are systematic and institutionalized rather than reliant on the personalities and connections of powerful city leaders. This capacity-building strategy can help cities leverage international coordination, information sharing, and intersectoral collaboration to address the complex and interconnected problems that will characterize the 21st century….(More)”.

Science and Scientists Held in High Esteem Across Global Publics


Pew Research: “As publics around the world look to scientists and the research and development process to bring new treatments and preventive strategies for the novel coronavirus, a new international survey finds scientists and their research are widely viewed in a positive light across global publics, and large majorities believe government investments in scientific research yield benefits for society.

Chart shows most value government investment in scientific research, being a world leader in science

Still, the wide-ranging survey, conducted before the COVID-19 outbreak reached pandemic proportions, reveals ambivalence about certain scientific developments – in areas such as artificial intelligence and genetically modified foods – often exists alongside high trust for scientists generally and positive views in other areas such as space exploration….

Scientists as a group are highly regarded, compared with other prominent groups and institutions in society. In all publics, majorities have at least some trust in scientists to do what is right. A median of 36% have “a lot” of trust in scientists, the same share who say this about the military, and much higher than the shares who say this about business leaders, the national government and the news media.

Still, an appreciation for practical experience, more so than expertise, in general, runs deep across publics. A median of 66% say it’s better to rely on people with practical experience to solve pressing problems, while a median of 28% say it’s better to rely on people who are considered experts about the problems, even if they don’t have much practical experience….(More)”.

Peer-Reviewed Scientific Journals Don’t Really Do Their Job


Article by Simine Vazire: “THE RUSH FOR scientific cures and treatments for Covid-19 has opened the floodgates of direct communication between scientists and the public. Instead of waiting for their work to go through the slow process of peer review at scientific journals, scientists are now often going straight to print themselves, posting write-ups of their work to public servers as soon as they’re complete. This disregard for the traditional gatekeepers has led to grave concerns among both scientists and commentators: Might not shoddy science—and dangerous scientific errors—make its way into the media, and spread before an author’s fellow experts can correct it? As two journalism professors suggested in an op-ed last month for The New York Times, it’s possible the recent spread of so-called preprints has only “sown confusion and discord with a general public not accustomed to the high level of uncertainty inherent in science.”

There’s another way to think about this development, however. Instead of showing (once again) that formal peer review is vital for good science, the last few months could just as well suggest the opposite. To me, at least—someone who’s served as an editor at seven different journals, and editor in chief at two—the recent spate of decisions to bypass traditional peer review gives the lie to a pair of myths that researchers have encouraged the public to believe for years: First, that peer-reviewed journals publish only trustworthy science; and second, that trustworthy science is published only in peer-reviewed journals.

Scientists allowed these myths to spread because it was convenient for us. Peer-reviewed journals came into existence largely to keep government regulators off our backs. Scientists believe that we are the best judges of the validity of each other’s work. That’s very likely true, but it’s a huge leap from that to “peer-reviewed journals publish only good science.” The most selective journals still allow flawed studies—even really terribly flawed ones—to be published all the time. Earlier this month, for instance, the journal Proceedings of the National Academy of Sciences put out a paper claiming that mandated face coverings are “the determinant in shaping the trends of the pandemic.” PNAS is a very prestigious journal, and their website claims that they are an “authoritative source” that works “to publish only the highest quality scientific research.” However, this paper was quickly and thoroughly criticized on social media; by last Thursday, 45 researchers had signed a letter formally calling for its retraction.

Now the jig is up. Scientists are writing papers that they want to share as quickly as possible, without waiting the months or sometimes years it takes to go through journal peer review. So they’re ditching the pretense that journals are a sure-fire quality control filter, and sharing their papers as self-published PDFs. This might be just the shakeup that peer review needs….(More)”.

The Data Assembly


Press Release: “The Governance Lab (The GovLab), an action research center at New York University Tandon School of Engineering, with the support of the Henry Luce Foundation, announced the creation of The Data Assembly. Beginning in New York City, the effort will explore how communities perceive the risks and benefits of data re-use for COVID-19. Understanding that policymakers often lack information about the concerns of different stakeholders, The Data Assembly’s deliberations will inform the creation of a responsible data re-use framework to guide the use of data and technology at the city and state level to fight COVID-19’s many consequences.

The Data Assembly will hold deliberations with civil rights organizations, key data holders and policymakers, and the public at large. Consultations with these stakeholders will take place through a series of remote engagements, including surveys and an online town hall meeting. This work will allow the project to consider the perspectives of people from different strata of society and how they might exercise some control over the flow of data.

After the completion of these data re-use deliberations, The Data Assembly will create a path forward for using data responsibly to solve public challenges. The first phases of the project will commence in New York City, seeking to engage with city residents and their leaders on data governance issues. 

“Data is increasingly the primary format for sharing information to understand crises and plan recovery efforts; empowering everyone to better understand how data is collected and how it should be used is paramount,” said Adrienne Schmoeker, Director of Civic Engagement & Strategy and Deputy Chief Analytics Officer at the NYC Mayor’s Office of Data Analytics. “We look forward to learning from the insights gathered by the GovLab through The Data Assembly work they are conducting in New York City.”…(More)”.

A Practical Guide for Establishing an Evidence Centre


Report by Alliance for Useful Evidence: “Since 2013, Nesta and the Alliance for Useful Evidence have supported the development of more than eight evidence centres. This report draws on insight from our own experience, published material and interviews with senior leaders from a range of evidence intermediaries.

The report identifies five common ingredients that contribute to successful evidence centres:

  1. Clear objectives: Good knowledge of the centre’s intended user group(s), clear outcomes to work towards and an evidence-informed theory of change.
  2. Robust organisational development: Commitment to create an independent and sustainable organisation with effective governance and the right mix of skills and experience, over a timescale that will be sufficient to make a difference.
  3. Engaged users: Understanding users’ evidence needs and working collaboratively with them to increase their capability, motivation and opportunity to use evidence in their decision-making.
  4. Rigorous curation and creation of evidence: A robust and transparent approach to selecting and generating high-quality evidence for the centre’s users.
  5. A focus on impact: Commitment to learn from the centre’s activities, including successes and failures, so that you can increase your effectiveness in achieving your objectives…(More)”.

UK parliamentary select committees: crowdsourcing for evidence-based policy or grandstanding?


Paper by the The LSE GV314 Group: “In the United Kingdom, the influence of parliamentary select committees on policy depends substantially on the ‘seriousness’ with which they approach the task of gathering and evaluating a wide range of evidence and producing reports and recommendations based on it. However, select committees are often charged with being concerned with ‘political theatre’ and ‘grandstanding’ rather than producing evidence-based policy recommendations. This study, based on a survey of 919 ‘discretionary’ witnesses, including those submitting written and oral evidence, examines the case for arguing that there is political bias and grandstanding in the way select committees go about selecting witnesses, interrogating them and using their evidence to put reports together. While the research finds some evidence of such ‘grandstanding’ it does not appear to be strong enough to suggest that the role of select committees is compromised as a crowdsourcer of evidence….(More)”.