The limits of expert judgment: Lessons from social science forecasting during the pandemic

Curated on Posted on by Stefaan Verhulst

Article by Cendri Hutcherson  Michael Varnum Imagine being a policymaker at the beginning of the COVID-19 pandemic. You have to decide which actions to recommend, how much risk to tolerate and what sacrifices to ask your citizens to bear.

Who would you turn to for an accurate prediction about how people would react? Many would recommend going to the experts — social scientists. But we are here to tell you this would be bad advice.

As psychological scientists with decades of combined experience studying decision-making, wisdom, expert judgment and societal change, we hoped social scientists’ predictions would be accurate and useful. But we also had our doubts.

Our discipline has been undergoing a crisis due to failed study replications and questionable research practices. If basic findings can’t be reproduced in controlled experiments, how confident can we be that our theories can explain complex real-world outcomes?

To find out how well social scientists could predict societal change, we ran the largest forecasting initiative in our field’s history using predictions about change in the first year of the COVID-19 pandemic as a test case….

Our findings, detailed in peer-reviewed papers in Nature Human Behaviour and in American Psychologist, paint a sobering picture. Despite the causal nature of most theories in the social sciences, and the fields’ emphasis on prediction in controlled settings, social scientists’ forecasts were generally not very good.

In both papers, we found that experts’ predictions were generally no more accurate than those made by samples of the general public. Further, their predictions were often worse than predictions generated by simple statistical models.

Our studies did still give us reasons to be optimistic. First, forecasts were more accurate when teams had specific expertise in the domain they were making predictions in. If someone was an expert in depression, for example, they were better at predicting societal trends in depression.

Second, when teams were made up of scientists from different fields working together, they tended to do better at forecasting. Finally, teams that used simpler models to generate their predictions and made use of past data generally outperformed those that didn’t.

These findings suggest that, despite the poor performance of the social scientists in our studies, there are steps scientists can take to improve their accuracy at this type of forecasting….(More)”.

Automating Public Services: Learning from Cancelled Systems

Curated on Posted on by Stefaan Verhulst

Report by Joanna Redden, Jessica Brand, Ina Sander and Harry Warne: “Pressure on public finances means that governments are trying to do more with less. Increasingly, policymakers are turning to technology to cut costs. But what if this technology doesn’t work as it should?

This report looks at the rise and fall of automated decision systems (ADS). If you’ve tried to get medical advice over the phone recently you’ve got some experience of an ADS – a computer system or algorithm designed to help or replace human decision making. These sorts of systems are being used by governments to consider when and how to act. The stakes are high. For example, they’re being used to try to detect crime and spot fraud, and to determine whether child protective services should act.

This study identifies 61 occasions across Australia, Canada, Europe, New Zealand and the United States when ADS projects were cancelled or paused. From this evidence, we’ve made recommendations designed to increase transparency and to protect communities and individuals…(More)”.

China Data Flows and Power in the Era of Chinese Big Tech

Curated on Posted on by Stefaan Verhulst

Paper by W. Gregory Voss and Emmanuel Pernot-Leplay: “Personal data have great economic interest today and their possession and control are the object of geopolitics, leading to their regulation by means that vary dependent on the strategic objectives of the jurisdiction considered. This study fills a gap in the literature in this area by analyzing holistically the regulation of personal data flows both into and from China, the world’s second largest economy. In doing so, it focuses on laws and regulations of three major power blocs: the United States, the European Union, and China, seen within the framework of geopolitics, and considering the rise of Chinese big tech.

First, this study analyzes ways that the United States—the champion of the free-flow of data that has helped feed the success of the Silicon Valley system—has in specific cases prevented data flows to China on grounds of individual data protection and national security. The danger of this approach and alternate protection through potential U.S. federal data privacy legislation are evoked. Second, the cross-border data flow restriction of the European Union’s General Data Protection Regulation (GDPR) is studied in the context of data exports to China, including where the data transit via the United States prior to their transfer to China. Next, after review of the conditions for a European Commission adequacy determination and an examination of recent data privacy legislation in China, the authors provide a preliminary negative assessment of the potential for such a determination for China, where government access is an important part of the picture. Difficult points are highlighted for investigation by data exporters to China, when relying on EU transfer mechanisms, following the Schrems II jurisprudence.

Finally, recent Chinese regulations establishing requirements for the export of data are studied. In this exercise, light is shed on compliance requirements for companies under Chinese law, provisions of Chinese data transfer regulations that are similar to the those of the GDPR, and aspects that show China’s own approach to restrictions on data transfers, such as an emphasis on national security protection. This study concludes with the observation that restrictions for data flows both into and out of China will continue and potentially be amplified, and economic actors will need to prepare themselves to navigate the relevant regulations examined in this study….(More)”.

The pandemic veneer: COVID-19 research as a mobilisation of collective intelligence by the global research community

Curated on Posted on by Stefaan Verhulst

Paper by Daniel W Hook and James R Wilsdon: “The global research community responded with speed and at scale to the emergence of COVID-19, with around 4.6% of all research outputs in 2020 related to the pandemic. That share almost doubled through 2021, to reach 8.6% of research outputs. This reflects a dramatic mobilisation of global collective intelligence in the face of a crisis. It also raises fundamental questions about the funding, organisation and operation of research. In this Perspective article, we present data that suggests that COVID-19 research reflects the characteristics of the underlying networks from which it emerged, and on which it built. The infrastructures on which COVID-19 research has relied – including highly skilled, flexible research capacity and collaborative networks – predated the pandemic, and are the product of sustained, long-term investment. As such, we argue that COVID-19 research should not be viewed as a distinct field, or one-off response to a specific crisis, but as a ‘pandemic veneer’ layered on top of longstanding interdisciplinary networks, capabilities and structures. These infrastructures of collective intelligence need to be better understood, valued and sustained as crucial elements of future pandemic or crisis response…(More)”.

Analyzing Big Data on a Shoestring Budget

Curated on Posted on by Stefaan Verhulst

Article by Toshiko Kaneda and Lori S. Ashford: “Big data has opened a new world for demographers and public health scientists to explore, to gain insights into social and health phenomena using the myriad digital traces we leave behind in our daily lives. But is analyzing big data practical and affordable? Researchers and organizations who have not made the leap might wonder: Do we need a lot more funding? Supercomputers? Armies of data scientists?

Three studies, presented recently in a PRB Demography Talk, show the feasibility of conducting research on a proverbial shoestring—using big data that are publicly, freely available to anyone with a personal computer and Wi-Fi connection.

Study 1: Can Google data help measure health care access more accurately?

The first study, presented by Luis Gabriel Cuervo of the Universitat Autònoma de Barcelona and the AMORE project, used Google mobility data to assess the effect of traffic congestion on people’s ability to access health services in Cali, Colombia, a city of 2.3 million. The study aimed to improve how health care accessibility is measured and communicated, to inform urban and health services planning.

Cuervo assembled a multidisciplinary research team, including mobility experts, to examine travel times from where people live to urgent and frequently used health services. The team used Google’s Distance Matrix API, which provides travel times and distance between origins and destinations, accounting for changing traffic conditions. The data are generated from Google Maps on people’s cell phones.

Combining this information with census and health services data, the study measured travel times repeatedly and revealed significant inequality by sociodemographic characteristics. On typical days, 60% of the city’s population lived more than 15 minutes by car from emergency care, with those in the poorest neighborhoods facing the longest travel times and a greater impact from traffic congestion.

Studies 2 and 3: Can Google data help predict changes in birth rates and examine excess deaths from COVID-19 related shutdowns?

In another study, Joshua Wilde from the Max Planck Institute for Demographic Research (MPIDR) and Portland State University asked, can Google search data predict whether COVID-related shutdowns will lead to a baby boom or bust?  In 2020, early in the pandemic, Wilde and team constructed a forecasting model based on volumes of Google searches with keywords related to conception, pregnancy, childbirth, and economic stability. Their thinking was that if searches increased sharply for keywords such as “pregnancy test” and “missed period,” one might expect higher birth rates seven to nine months later. On the other hand, prior research had associated unemployment with lower birth rates—so if unemployment-related searches climbed, one might expect a baby bust….(More)”.

Policy Guide on Social Impact Measurement for the Social and Solidarity Economy

Curated on Posted on by Stefaan Verhulst

OECD Report: “As social and solidarity economy (SSE) entities are increasingly requested to demonstrate their positive contribution to society, social impact measurement can help them understand the additional, net value generated by their activities, in the pursuit of their mission and beyond. Policy plays an important role to facilitate a conducive environment to unlock the uptake of social impact measurement among SSE actors. Drawing on a mapping exercise and good practice examples from over 33 countries, this international policy guide navigates how policy makers can support social impact measurement for the social and solidarity economy by: (i) improving the policy framework, (ii) delivering guidance, (iii) building evidence and (iv) supporting capacity. Building on the earlier publication Social Impact Measurement for the Social and Solidarity Economy released in 2021 the guide is published under the framework of the OECD Global Action “Promoting Social and Solidarity Economy Ecosystems”, funded by the European Union’s Foreign Partnership Instrument…(More)”.

An agenda for advancing trusted data collaboration in cities

Curated on Posted on by Stefaan Verhulst

Report by Hannah Chafetz, Sampriti Saxena, Adrienne Schmoeker, Stefaan G. Verhulst, & Andrew J. Zahuranec: “… Joined by experts across several domains including smart cities, the law, and data ecosystem, this effort was focused on developing solutions that could improve the design of Data Sharing Agreements…we assessed what is needed to implement each aspect of our Contractual Wheel of Data Collaboration–a tool developed as a part of the Contracts for Data Collaborations initiative that seeks to capture the elements involved in data collaborations and Data Sharing Agreements.

In what follows, we provide key suggestions from this Action Lab…

  1. The Elements of Principled Negotiations: Those seeking to develop a Data Sharing Agreement often struggle to work with collaborators or agree to common ends. There is a need for a common resource that Data Stewards can use to initiate a principled negotiation process. To address this need, we would identify the principles to inform negotiations and the elements that could help achieve those principles. For example, participants voiced a need for fairness, transparency, and reciprocity principles. These principles could be supported by having a shared language or outlining the minimum legal documents required for each party. The final product would be a checklist or visualization of principles and their associated elements.
  2. Data Responsibility Principles by Design: …
  3. Readiness Matrix: …
  4. A Decision Provenance Approach for Data Collaboration: ..
  5. The Contractual Wheel of Data Collaboration 2.0…
  6. A Repository of Legal Drafting Technologies:…(More)”.

Building Trust in AI: A Landscape Analysis of Government AI Programs

Curated on Posted on by Stefaan Verhulst

Paper by Susan Ariel Aaronson: “As countries around the world expand their use of artificial intelligence (AI), the Organisation for Economic Co-operation and Development (OECD) has developed the most comprehensive website on AI policy, the OECD.AI Policy Observatory. Although the website covers public policies on AI, the author of this paper found that many governments failed to evaluate or report on their AI initiatives. This lack of reporting is a missed opportunity for policy makers to learn from their programs (the author found that less than one percent of the programs listed on the OECD.AI website had been evaluated). In addition, the author found discrepancies between what governments said they were doing on the OECD.AI website and what they reported on their own websites. In some cases, there was no evidence of government actions; in other cases, links to government sites did not work. Evaluations of AI policies are important because they help governments demonstrate how they are building trust in both AI and AI governance and that policy makers are accountable to their fellow citizens…(More)”.

To harness telecom data for good, there are six challenges to overcome

Curated on Posted on by Stefaan Verhulst

Blog by Anat Lewin and Sveta Milusheva: “The global use of mobile phones generates a vast amount of data. What good can be done with these data? During the COVID-19 pandemic, we saw that aggregated data from mobile phones can tell us where groups of humans are going, how many of them are there, and how they are behaving as a cluster. When used effectively and responsibly, mobile phone data can be immensely helpful for development work and emergency response — particularly in resource-constrained countries.  For example, an African country that had, in recent years, experienced a cholera outbreak was ahead of the game. Since the legal and practical agreements were already in place to safely share aggregated mobile data, accessing newer information to support epidemiological modeling for COVID-19 was a straightforward exercise. The resulting datasets were used to produce insightful analyses that could better inform health, lockdown, and preventive policy measures in the country.

To better understand such challenges and opportunities, we led an effort to access and use anonymized, aggregated mobile phone data across 41 countries. During this process, we identified several recurring roadblocks and replicable successes, which we summarized in a paper along with our lessons learned. …(More)”.

The Right To Be Free From Automation

Curated on Posted on by Stefaan Verhulst

Essay by Ziyaad Bhorat: “Is it possible to free ourselves from automation? The idea sounds fanciful, if not outright absurd. Industrial and technological development have reached a planetary level, and automation, as the general substitution or augmentation of human work with artificial tools capable of completing tasks on their own, is the bedrock of all the technologies designed to save, assist and connect us. 

From industrial lathes to OpenAI’s ChatGPT, automation is one of the most groundbreaking achievements in the history of humanity. As a consequence of the human ingenuity and imagination involved in automating our tools, the sky is quite literally no longer a limit. 

But in thinking about our relationship to automation in contemporary life, my unease has grown. And I’m not alone — America’s Blueprint for an AI Bill of Rights and the European Union’s GDPR both express skepticism of automated tools and systems: The “use of technology, data and automated systems in ways that threaten the rights of the American public”; the “right not to be subject to a decision based solely on automated processing.” 

If we look a little deeper, we find this uneasy language in other places where people have been guarding three important abilities against automated technologies. Historically, we have found these abilities so important that we now include them in various contemporary rights frameworks: the right to work, the right to know and understand the source of the things we consume, and the right to make our own decisions. Whether we like it or not, therefore, communities and individuals are already asserting the importance of protecting people from the ubiquity of automated tools and systems.

Consider the case of one of South Africa’s largest retailers, Pick n Pay, which in 2016 tried to introduce self-checkout technology in its retail stores. In post-Apartheid South Africa, trade unions are immensely powerful and unemployment persistently high, so any retail firm that wants to introduce technology that might affect the demand for labor faces huge challenges. After the country’s largest union federation threatened to boycott the new Pick n Pay machines, the company scrapped its pilot. 

As the sociologist Christopher Andrews writes in “The Overworked Consumer,” self-checkout technology is by no means a universally good thing. Firms that introduce it need to deal with new forms of theft, maintenance and bottleneck, while customers end up doing more work themselves. These issues are in addition to the ill fortunes of displaced workers…(More)”.