Paper by Nate Breznau et al: “This study explores how researchers’ analytical choices affect the reliability of scientific findings. Most discussions of reliability problems in science focus on systematic biases. We broaden the lens to include conscious and unconscious decisions that researchers make during data analysis and that may lead to diverging results. We coordinated 161 researchers in 73 research teams and observed their research decisions as they used the same data to independently test the same prominent social science hypothesis: that greater immigration reduces support for social policies among the public. In this typical case of research based on secondary data, we find that research teams reported widely diverging numerical findings and substantive conclusions despite identical start conditions. Researchers’ expertise, prior beliefs, and expectations barely predicted the wide variation in research outcomes. More than 90% of the total variance in numerical results remained unexplained even after accounting for research decisions identified via qualitative coding of each team’s workflow. This reveals a universe of uncertainty that is hidden when considering a single study in isolation. The idiosyncratic nature of how researchers’ results and conclusions varied is a new explanation for why many scientific hypotheses remain contested. It calls for greater humility and clarity in reporting scientific findings..(More)”.
Building the analytic capacity to support critical technology strategy
Paper by Erica R.H. Fuchs: “Existing federal agencies relevant to the science and technology enterprise are appropriately focused on their missions, but the U.S. lacks the intellectual foundations, data infrastructure, and analytics to identify opportunities where the value of investment across missions (e.g., national security, economic prosperity, social well-being) is greater than the sum of its parts.
The U.S. government lacks systematic mechanisms to assess the nation’s strengths, weaknesses, and opportunities in technology and to assess the long chain of suppliers involved in producing products critical to national missions.
Two examples where modern data and analytics—leveraging star interdisciplinary talent from across the nation—and a cross-mission approach could transform outcomes include 1) the difficulties the federal government had in facilitating the production and distribution of personal protective equipment in spring 2020, and 2) the lack of clarity about the causes and solutions to the semiconductor shortage. Going forward, the scale-up of electric vehicles promises similar challenges…
The critical technology analytics (CTA) would identify 1) how emerging technologies and institutional innovations could potentially transform timely situational awareness of U.S. and global technology capabilities, 2) opportunities for innovation to transform U.S. domestic and international challenges, and 3) win-win opportunities across national missions. The program would be strategic and forward-looking, conducting work on a timeline of months and years rather than days and weeks, and would seek to generalize lessons from individual cases to inform the data and analytics capabilities that the government needs to build to support cross-mission critical technology policy…(More)”.
Policy Choice and the Wisdom of Crowds
Paper by Nicholas Otis: “Using data from seven large-scale randomized experiments, I test whether crowds of academic experts can forecast the relative effectiveness of policy interventions. Eight-hundred and sixty-three academic experts provided 9,295 forecasts of the causal effects from these experiments, which span a diverse set of interventions (e.g., information provision, psychotherapy, soft-skills training), outcomes (e.g., consumption, COVID-19 vaccination, employment), and locations (Jordan, Kenya, Sweden, the United States). For each policy comparisons (a pair of policies and an outcome), I calculate the percent of crowd forecasts that correctly rank policies by their experimentally estimated treatment effects. While only 65% of individual experts identify which of two competing policies will have a larger causal effect, the average forecast from bootstrapped crowds of 30 experts identifies the better policy 86% of the time, or 92% when restricting analysis to pairs of policies who effects differ at the p < 0.10 level. Only 10 experts are needed to produce an 18-percentage point (27%) improvement in policy choice…(More)”.
Architectures of Participation
Essay by Jay Lloyd and Annalee Saxenian: “Silicon Valley’s dynamism during the final three decades of the twentieth century highlighted the singular importance of social and professional networks to innovation. Since that time, contemporary and historical case studies have corroborated the link between networks and the pace of technological change. These studies have shown that networks of networks, or ecosystems, that are characterized by a mix of collaboration and competition, can accelerate learning and problem-solving.
However, these insights about networks, collaboration, and ecosystems remain surprisingly absent from public debates about science and technology policy. Since the end of World War II, innovation policy has targeted economic inputs such as funding for basic scientific research and a highly skilled workforce (via education, training, and/or immigration), as well as support for commercialization of technology, investments in information technology, and free trade. Work on national systems of innovation, by contrast, seeks to define the optimal ensembles of institutions and policies. Alternatively, policy attention is focused on achieving efficiencies and scale by gaining control over value chains, especially in critical industries such as semiconductors. Antitrust advocates have attributed stalled technological innovation to monopolistic concentration among large firms, arguing that divestiture or regulation is necessary to reinvigorate competition and speed gains for society. These approaches ignore the lessons of network research, potentially threatening the very ecosystems that could unlock competitive advantages. For example, attempts to strengthen value chains risk cutting producers off from global networks, leaving them vulnerable to shifting markets and technology and weakening the wider ecosystem. Breaking up large platform firms may likewise undermine less visible internal interdependencies that support innovation, while doing nothing to encourage external collaboration.
Networks of networks, or ecosystems, that are characterized by a mix of collaboration and competition, can accelerate learning and problem-solving.
How might the public sector promote and strengthen important network connections in a world of continuous flux? This essay reexamines innovation policy through the lens of the current era of cloud computing, arguing that the public sector has a regulatory role as well as a nurturing one to play in fostering innovation ecosystems…(More)”.
On the Power of Networks
Essay by Jay Lloyd: “A mosquito net made from lemons, a workout shirt that feeds sweat to cyanobacteria to generate electricity, a water filter using moss from the Andes—and a slime mold that produces eerie electronic music. For a few days in late June, I logged on to help judge the Biodesign Challenge, a seven-year-old competition where high school and college students showcase designs that use biotechnology to address real problems. Fifty-six teams from 18 countries presented their creations—some practical, others purely speculative.
The competition is, by design, cautiously optimistic about the potential for technology to solve problems such as plastic pollution or malaria or sexually transmitted diseases. This caution manifests in an emphasis on ethics as a first principle in design: many problems the students seek to solve are the results of previous “solutions” gone wrong. Underlying this is a conviction that technology can help build a world that not only works better but is also more just. The biodesign worldview starts with research to understand problems in context, then imagines a design for a biology-based solution, and often envisions how that technology could transform today’s power dynamics. Two projects this year speculated about using mRNA to reduce systemic racism and global inequality.
The Biodesign Challenge is a profoundly hopeful exercise in future-building, but the tensions inherent in this theory of change became clear at the awards ceremony, which coincided with the Supreme Court’s announcement of the reversal of Roe v. Wade, ending the right to abortion at the national level. The ceremony took place under a cloud, and these entrancing proposals for an imagined biofuture sharply juxtaposed with the results of the blunt exercise of political power.
Clearly, networks of people devoted to a cause can be formidable forces for change—and it’s possible that Biodesign Challenge itself could become such a network in the future. The group consists of more than 100 teachers and judges—artists, scientists, social scientists, and people from the biotech industry—and the challengers themselves, who Zoom in from Shanghai, Buenos Aires, Savannah, Cincinnati, Turkey, and elsewhere. As biotechnology matures around the world, it will be applied by networks of people who have determined which problems need to be addressed…(More)”.
By focusing on outputs, rather than people, we misunderstand the real impact of research
Blog by Paul Nightingale and Rebecca Vine: “Increases in funding for research come with a growing expectation that researchers will do more to improve social welfare, economic prosperity and more broadly foster innovation. It is widely accepted that innovation is a key driver of long-term economic growth and that public funding for research complements private investment. What is more contested is how research delivers impact. Whether it comes from the kinds of linear processes of knowledge transfer from researcher to user, sought for and often narrated in REF impact case studies. Or, if the indirect effects of research such as expertise, networks, instrumentation, methods and trained students, are as important as the discoveries….
One reason research is so important, is that as the economy has changed and demand for experts has increased. As we noted in a Treasury report over 20 years ago, often the most valuable output of research is ‘talent, not technology’. The ‘post-graduate premium’ that having a Masters qualification adds to starting salaries is evidence of this. But why is expertise so valuable? Experts don’t just know more than novices, they understand things differently, drawing on more abstract, ‘deeper’ representations. Research on chess-grandmasters, for example, shows that they understand chess piece configurations by seeing patterns. They can see a Sicilian defence, while novices just see a selection of chess pieces. Their expertise enables them to configure chess positions more effectively and solve problems more rapidly. They draw different conclusions than novices, typically starting closer to more robust solutions, finding solutions faster, and exploring fewer dead-ends….
Research is extremely important because innovation requires more diverse and deeper stocks of knowledge. Academics with field expertise and highly developed research skills can play a valuable and important role co-producing research and creating impact. These observations are drawn from our ESRC-funded research collaboration with the UK government – known as Project X. Within a year Project X became the mechanism to coordinate the Cabinet Office’s areas of research interest (ARIs) for government major project delivery. This required a sophisticated governance structure and the careful coordination of a mixed portfolio of practice-focused and theoretical research…(More)”.
Public Provides NASA with New Innovations through Prize Competitions, Crowdsourcing, Citizen Science Opportunities
NASA Report: “Whether problem-solving during the pandemic, establishing a long-term presence at the Moon, or advancing technology to adapt to life in space, NASA has leveraged open innovation tools to inspire solutions to some of our most timely challenges – while using the creativity of everyone from garage tinkerers to citizen scientists and students of all ages.
Open Innovation: Boosting NASA Higher, Faster, and Farther highlights some of those breakthroughs, which accelerate space technology development and discovery while giving the public a gateway to work with NASA. Open innovation initiatives include problem-focused challenges and prize competitions, data hackathons, citizen science, and crowdsourcing projects that invite the public to lend their skills, ideas, and time to support NASA research and development programs.
NASA engaged the public with 56 public prize competitions and challenges and 14 citizen science and crowdsourcing activities over fiscal years 2019 and 2020. NASA awarded $2.2 million in prize money, and members of the public submitted over 11,000 solutions during that period.
“NASA’s accomplishments have hardly been NASA’s alone. Tens of thousands more individuals from academic institutions, private companies, and other space agencies also contribute to these solutions. Open innovation expands the NASA community and broadens the agency’s capacity for innovation and discovery even further,” said Amy Kaminski, Prizes, Challenges, and Crowdsourcing program executive at NASA Headquarters in Washington. “We harness the perspectives, expertise, and enthusiasm of ‘the crowd’ to gain diverse solutions, speed up projects, and reduce costs.”
This edition of the publication highlights:
- How NASA used open innovation tools to accelerate the pace of problem-solving during the COVID-19 pandemic, enabling a sprint of creativity to create valuable solutions in support of this global crisis
- How NASA invited everyone to embrace the Moon as a technological testing ground through public prize competitions and challenges, sparking development that could help prolong human stays on the Moon and lay the foundation for human exploration to Mars and beyond
- How citizen scientists gather, sort, and upload data, resulting in fruitful partnerships between the public and NASA scientists
- How NASA’s student-focused challenges have changed lives and positively impacted underserved communities…(More)”.
The West already monopolized scientific publishing. Covid made it worse.
Samanth Subramanian at Quartz: “For nearly a decade, Jorge Contreras has been railing against the broken system of scientific publishing. Academic journals are dominated by the Western scientists, who not only fill their pages but also work for institutions that can afford the hefty subscription fees to these journals. “These issues have been brewing for decades,” said Contreras, a professor at the University of Utah’s College of Law who specializes in intellectual property in the sciences. “The covid crisis has certainly exacerbated things, though.”
The coronavirus pandemic triggered a torrent of academic papers. By August 2021, at least 210,000 new papers on covid-19 had been published, according to a Royal Society study. Of the 720,000-odd authors of these papers, nearly 270,000 were from the US, the UK, Italy or Spain.
These papers carry research forward, of course—but they also advance their authors’ careers, and earn them grants and patents. But many of these papers are often based on data gathered in the global south, by scientists who perhaps don’t have the resources to expand on their research and publish. Such scientists aren’t always credited in the papers their data give rise to; to make things worse, the papers appear in journals that are out of the financial reach of these scientists and their institutes.
These imbalances have, as Contreras said, been a part of the publishing landscape for years. (And it doesn’t occur just in the sciences; economists from the US or the UK, for instance, tend to study countries where English is the most common language.) But the pace and pressures of covid-19 have rendered these iniquities especially stark.
Scientists have paid to publish their covid-19 research—sometimes as much as $5,200 per article. Subscriber-only journals maintain their high fees, running into thousands of dollars a year; in 2020, the Dutch publishing house Elsevier, which puts out journals such as Cell and Gene, reported a profit of nearly $1 billion, at a margin higher than that of Apple or Amazon. And Western scientists are pressing to keep data out of GISAID, a genome database that compels users to acknowledge or collaborate with anyone who deposits the data…(More)”
Making Space for Everyone
Amy Paige Kaminski at Issues: “The story of how NASA came to see the public as instrumental in accomplishing its mission provides insights for R&D agencies trying to create societal value, relevance, and connection….Over the decades since, NASA’s approaches to connecting with citizens have evolved with the introduction of new information and communications technologies, social change, legal developments, scientific progress, and external trends in space activities and public engagement. The result has been an increasing and increasingly accessible set of opportunities that have enabled diverse segments of society to connect more closely with NASA’s work and, in turn, boost the agency’s techno-scientific and societal value….
Another significant change in public engagement practices has been providing more people with opportunities to do space-related R&D. Through the shuttle program, the agency enabled companies, universities, high schools, and an eclectic set of participants ranging from artists to garden seed companies to develop and fly payloads. The stated purpose was to advance knowledge of the effects of the space environment—a concept that was sometimes loosely defined.
Today NASA similarly encourages a broad set of players to use the International Space Station (ISS) for R&D. While some of the shuttle and ISS programs have charged fees to payload owners, NASA has instead offered grants, primarily to the university community, for competitively selected research projects in space science. The agency also invites various groups to propose experiments and technology development projects through government-wide programs such as the Small Business Innovative Research program, which aims to foster innovation in small businesses, as well as the Established Program to Stimulate Competitive Research (better known by its EPSCoR acronym), which seeks to enhance research infrastructure and competitiveness at the state level….(More)”.
Manufacturing Consensus
Essay by M. Anthony Mills: “…Yet, the achievement of consensus within science, however rare and special, rarely translates into consensus in social and political contexts. Take nuclear physics, a well-established field of natural science if ever there were one, in which there is a high degree of consensus. But agreement on the physics of nuclear fission is not sufficient for answering such complex social, political, and economic questions as whether nuclear energy is a safe and viable alternative energy source, whether and where to build nuclear power plants, or how to dispose of nuclear waste. Expertise in nuclear physics and literacy in its consensus views is obviously important for answering such questions, but inadequate. That’s because answering them also requires drawing on various other kinds of technical expertise — from statistics to risk assessment to engineering to environmental science — within which there may or may not be disciplinary consensus, not to mention grappling with practical challenges and deep value disagreements and conflicting interests.
It is in these contexts — where multiple kinds of scientific expertise are necessary but not sufficient for solving controversial political problems — that the dependence of non-experts on scientific expertise becomes fraught, as our debates over pandemic policies amply demonstrate. Here scientific experts may disagree about the meaning, implications, or limits of what they know. As a result, their authority to say what they know becomes precarious, and the public may challenge or even reject it. To make matters worse, we usually do not have the luxury of a scientific consensus in such controversial contexts anyway, because political decisions often have to be made long before a scientific consensus can be reached — or because the sciences involved are those in which a consensus is simply not available, and may never be.
To be sure, scientific experts can and do weigh in on controversial political decisions. For instance, scientific institutions, such as the National Academies of Sciences, will sometimes issue “consensus reports” or similar documents on topics of social and political significance, such as risk assessment, climate change, and pandemic policies. These usually draw on existing bodies of knowledge from widely varied disciplines and take considerable time and effort to produce. Such documents can be quite helpful and are frequently used to aid policy and regulatory decision-making, although they are not always available when needed for making a decision.
Yet the kind of consensus expressed in these documents is importantly distinct from the kind we have been discussing so far, even though they are both often labeled as such. The difference is between what philosopher of science Stephen P. Turner calls a “scientific consensus” and a “consensus of scientists.” A scientific consensus, as described earlier, is a relatively stable paradigm that structures and organizes scientific research. By contrast, a consensus of scientists is an organized, professional opinion, created in response to an explicit political or social need, often an official government request…(More)”.