The Problem with Science: The Reproducibility Crisis and What to do About It

Curated on Posted on by Stefaan Verhulst

Book by R. Barker Bausell: “Recent events have vividly underscored the societal importance of science, yet the majority of the public are unaware that a large proportion of published scientific results are simply wrong. The Problem with Science is an exploration of the manifestations and causes of this scientific crisis, accompanied by a description of the very promising corrective initiatives largely developed over the past decade to stem the spate of irreproducible results that have come to characterize many of our sciences.

More importantly, Dr. R. Barker Bausell has designed it to provide guidance to practicing and aspiring scientists regarding how (a) to change the way in which science has come to be both conducted and reported in order to avoid producing false positive, irreproducible results in their own work and (b) to change those institutional practices (primarily but not exclusively involving the traditional journal publishing process and the academic reward system) that have unwittingly contributed to the present crisis. There is a need for change in the scientific culture itself. A culture which prioritizes conducting research correctly in order to get things right rather than simply getting it published….(More)”.

OECD Science, Technology and Innovation Outlook

Curated on Posted on by Stefaan Verhulst

OECD: “…the COVID-19 crisis has triggered an unprecedented mobilisation of the science and innovation community. Public research agencies and organisations, private foundations and charities, and the health industry have set up an array of newly funded research initiatives worth billions of dollars in record time. Science is the only exit strategy from COVID-19.

Science and innovation have played essential roles in providing a better understanding of the virus and its transmission, and in developing hundreds of candidate therapeutics and vaccines over a very short period. Digital technologies have enabled large parts of the economy and society to continue to function, mitigating the impact of COVID-19. The pandemic has underscored more than in other recent crises the importance of science and innovation to being both prepared and reactive to upcoming crises….

The world is still in the midst of the COVID-19 crisis and many uncertainties remain….At the same time, many governments view the pandemic as a stark reminder of the need to transition to more sustainable, equitable and resilient societies. This is highlighted in many countries’ recovery packages, which include expenditures for R&D. Science and innovation will be essential to promote and deliver such transitions, but the pandemic has exposed limits in research and innovation systems that, if not addressed, will prevent this potential from being realised.

There is therefore a need to re-set STI policies to better equip governments with the instruments and capabilities to direct innovation efforts towards the goals of sustainability, inclusivity and resiliency.

1. Policy needs to be able to guide innovation efforts to where they are most needed. This has implications for how governments support research and innovation in firms, which account for about 70% of R&D expenditures in the OECD. The business R&D support policy mix has shifted in recent decades towards a greater reliance on tax compared to direct support instruments such as contracts, grants or awards. While effective for incentivising businesses to innovate, R&D tax incentives are indirect, untargeted and tend to generate incremental innovations. Well-designed direct measures for R&D are potentially better suited to supporting longer-term, high-risk research, and targeting innovations that either generate public goods (e.g. in health) or have a high potential for knowledge spillovers. Governments need to revisit their policy portfolios to ensure an appropriate balance between direct and indirect measures.

2. The multifaceted nature of addressing complex problems like COVID-19 and sustainability transitions underscores the need for transdisciplinary research to which current science system norms and institutions are ill-adapted. Disciplinary and hierarchical structures need to be adjusted to enable and promote transdisciplinary research that engages different disciplines and sectors to address complex challenges.

3. Governments should link support for emerging technologies, such as engineering biology and robotics, to broader missions like health resilience that encapsulate responsible innovation principles. The responsible innovation approach seeks to anticipate problems in the course of innovation and steer technology to best outcomes. It also emphasises the inclusion of stakeholders early in the innovation process.

4. Reforming PhD and post-doctoral training to support a diversity of career paths is essential for improving the ability of societies to react to crises and to deal with future challenges like climate change that require science-based responses. Reforms could also help relieve the precarity of early-career researchers, many of whom are employed on short-term contracts with no clear prospect of a permanent academic position. The crisis has also highlighted the need for academia to train and embrace a new cohort of digitally skilled research support professionals and scientists.

5. Global challenges require global solutions that draw on international STI co-operation. The development of COVID-19 vaccines has benefited from nascent global R&D preparedness measures, including agile technology platforms that can be activated as new pathogens emerge. The pandemic has created momentum to establish effective and sustainable global mechanisms to support the range and scope of R&D necessary to confront a wider range of global challenges. However, governments need to build trust and define common values to ensure a level playing field for scientific co-operation and an equitable distribution of its benefits.

6. Governments need to renew their policy frameworks and capabilities to fulfil a more ambitious STI policy agenda. Increasing policy emphasis on building resilience, which calls for policy agility, highlights the need for governments to acquire dynamic capabilities to adapt and learn in the face of rapidly changing environments. Engaging stakeholders and citizens in these efforts will expose policymakers to diverse knowledge and values, which should contribute to policy resilience. Governments should also continue to invest in evidence about their STI support policies with a view to improving them….(More)”.

How data analysis can enrich the liberal arts

Curated on Posted on by Stefaan Verhulst

The Economist: “…The arts can indeed seem as if they are under threat. Australia’s education ministry is doubling fees for history and philosophy while cutting those for stem subjects. Since 2017 America’s Republican Party has tried to close down the National Endowment for the Humanities (neh), a federal agency, only to be thwarted in Congress. In Britain, Dominic Cummings—who until November 2020 worked as the chief adviser to Boris Johnson, the prime minister—advocates for greater numeracy while decrying the prominence of bluffing “Oxbridge humanities graduates”. (Both men studied arts subjects at Oxford.)

However, little evidence yet exists that the burgeoning field of digital humanities is bankrupting the world of ink-stained books. Since the neh set up an office for the discipline in 2008, it has received just $60m of its $1.6bn kitty. Indeed, reuniting the humanities with sciences might protect their future. Dame Marina Warner, president of the Royal Society of Literature in London, points out that part of the problem is that “we’ve driven a great barrier” between the arts and stem subjects. This separation risks portraying the humanities as a trivial pursuit, rather than a necessary complement to scientific learning.

Until comparatively recently, no such division existed. Omar Khayyam wrote verse and cubic equations, Ada Lovelace believed science was poetical and Bertrand Russell won the Nobel prize for literature. In that tradition, Dame Marina proposes that all undergraduates take at least one course in both humanities and sciences, ideally with a language and computing. Introducing such a system in Britain would be “a cause for optimism”, she thinks. Most American universities already offer that breadth, which may explain why quantitative literary criticism thrived there. The sciences could benefit, too. Studies of junior doctors in America have found that those who engage with the arts score higher on tests of empathy.

Ms McGillivray says she has witnessed a “generational shift” since she was an undergraduate in the late 1990s. Mixing her love of mathematics and classics was not an option, so she spent seven years getting degrees in both. Now she sees lots of humanities students “who are really keen to learn about programming and statistics”. A recent paper she co-wrote suggested that British arts courses could offer basic coding lessons. One day, she reckons, “It’s going to happen…(More)”.

Is a racially-biased algorithm delaying health care for one million Black people?

Curated on Posted on by Stefaan Verhulst

Jyoti Madhusoodanan at Nature: “One million Black adults in the United States might be treated earlier for kidney disease if doctors were to remove a controversial ‘race-based correction factor’ from an algorithm they use to diagnose people and decide whether to administer medication, a comprehensive analysis finds.

Critics of the factor question its medical validity and say it potentially perpetuates racial bias — and that the latest study, published on 2 December in JAMA1, strengthens growing calls to discontinue its use.

“A population that is marginalized and much less likely to have necessary resources and support is the last group we want to put in a situation where they’re going to have delays in diagnosis and treatment,” says nephrologist Keith Norris at the University of California, Los Angeles, who argues for retiring the correction until there’s clear evidence that it’s necessary.

On the flip side, others say that the correction is based on scientific data that can’t be ignored, although they, too, agree that its basis on race is a problem….(More)”.

Wikipedia @ 20

Curated on Posted on by Stefaan Verhulst

Stories of an Incomplete Revolution edited by Joseph Reagle and Jackie Koerner (Open Access): “We have been looking things up in Wikipedia for twenty years. What began almost by accident—a wiki attached to a nascent online encyclopedia—has become the world’s most popular reference work. Regarded at first as the scholarly equivalent of a Big Mac, Wikipedia is now known for its reliable sourcing and as a bastion of (mostly) reasoned interaction. How has Wikipedia, built on a model of radical collaboration, remained true to its original mission of “free access to the sum of all human knowledge” when other tech phenomena have devolved into advertising platforms? In this book, scholars, activists, and volunteers reflect on Wikipedia’s first twenty years, revealing connections across disciplines and borders, languages and data, the professional and personal.

The contributors consider Wikipedia’s history, the richness of the connections that underpin it, and its founding vision. Their essays look at, among other things, the shift from bewilderment to respect in press coverage of Wikipedia; Wikipedia as “the most important laboratory for social scientific and computing research in history”; and the acknowledgment that “free access” includes not just access to the material but freedom to contribute—that the summation of all human knowledge is biased by who documents it….(More)”

The Modern World Has Finally Become Too Complex for Any of Us to Understand

Curated on Posted on by Stefaan Verhulst

Blog by Tim Maughan: “One of the dominant themes of the last few years is that nothing makes sense. Donald Trump is president, QAnon has mainstreamed fringe conspiracy theories, and hundreds of thousands are dead from a pandemic and climate change while many Americans do not believe that the pandemic or climate change are deadly. It’s incomprehensible.

I am here to tell you that the reason so much of the world seems incomprehensible is that it is incomprehensible. From social media to the global economy to supply chains, our lives rest precariously on systems that have become so complex, and we have yielded so much of it to technologies and autonomous actors that no one totally comprehends it all….

And those platforms of technology and software that glue all these huge networks together have become a complex system themselves. The internet might be the system that we interact with in the most direct and intimate ways, but most of us have little comprehension of what lies behind our finger-smudged touchscreens, truly understood by few. Made up of data centers, internet exchanges, huge corporations, tiny startups, investors, social media platforms, datasets, adtech companies, and billions of users and their connected devices, it’s a vast network dedicated to mining, creating, and moving data on scales we can’t comprehend. YouTube users upload more than 500 hours of video every minute — which works out as 82.2 yearsof video uploaded to YouTube every day. As of June 30, 2020, there are over 2.7 billion monthly active Facebook users, with 1.79 billion people on average logging on daily. Each day, 500 million tweets are sent— or 6,000 tweets every second, with a day’s worth of tweets filling a 10-million-page book. Every day, 65 billion messages are sent on WhatsApp. By 2025, it’s estimated that 463 million terabytes of data will be created each day — the equivalent of 212,765,957 DVDs.

So, what we’ve ended up with is a civilization built on the constant flow of physical goods, capital, and data, and the networks we’ve built to manage those flows in the most efficient ways have become so vast and complex that they’re now beyond the scale of any single (and, arguably, any group or team of) human understanding them. It’s tempting to think of these networks as huge organisms, with tentacles spanning the globe that touch everything and interlink with one another, but I’m not sure the metaphor is apt. An organism suggests some form of centralized intelligence, a nervous system with a brain at its center, processing data through feedback loops and making decisions. But the reality with these networks is much closer to the concept of distributed intelligence or distributed knowledge, where many different agents with limited information beyond their immediate environment interact in ways that lead to decision-making, often without them even knowing that’s what they’re doing….(More)”.

Public Value Science

Curated on Posted on by Stefaan Verhulst

Barry Bozeman in Issues in Science and Technology: “Why should the United States government support science? That question was apparently settled 75 years ago by Vannevar Bush in Science, the Endless Frontier: “Since health, well-being, and security are proper concerns of Government, scientific progress is, and must be, of vital interest to Government. Without scientific progress the national health would deteriorate; without scientific progress we could not hope for improvement in our standard of living or for an increased number of jobs for our citizens; and without scientific progress we could not have maintained our liberties against tyranny.”

Having dispensed with the question of why, all that remained was for policy-makers to decide, how much? Even at the dawn of modern science policy, costs and funding needs were at the center of deliberations. Though rarely discussed anymore, Endless Frontier did give specific attention to the question of how much. The proposed amounts seem, by today’s standards, modest: “It is estimated that an adequate program for Federal support of basic research in the colleges, universities, and research institutes and for financing important applied research in the public interest, will cost about 10 million dollars at the outset and may rise to about 50 million dollars annually when fully underway at the end of perhaps 5 years.”

In today’s dollars, $50 million translates to about $535 million, or less than 2% of what the federal government actually spent for basic research in 2018. One way to look at the legacy of Endless Frontier is that by answering the why question so convincingly, it logically followed that the how much question could always be answered simply by “more.”

In practice, however, the why question continues to seem so self-evident because it fails to consider a third question, who? As in, who benefits from this massive federal investment in research, and who does not? The question of who was also seemingly answered by Endless Frontier, which not only offered full employment as a major goal for expanded research but also embraced “the sound democratic principle that there should be no favored classes or special privilege.”

But I argue that this principle has now been soundly falsified. In an economic environment characterized by growth but also by extreme inequality, science and technology not only reinforce inequality but also, in some instances, help widen the gap. Science and technology can be a regressivefactor in the economy. Thus, it is time to rethink the economic equation justifying government support for science not just in terms of why and how much, but also in terms of who.

What logic supports my claim that under conditions of conspicuous inequality, science and technology research is often a regressive force? Simple: except in the case of the most basic of basic research (such as exploration of other galaxies), effects are never randomly distributed. Both the direct and indirect effects of science and technology tend to differentially affect citizens according to their socioeconomic power and purchasing power….(More)”.

Impact through Engagement: Co-production of administrative data research

Curated on Posted on by Stefaan Verhulst

Paper by Elizabeth Nelson and Frances Burns: “The Administrative Data Research Centre Northern Ireland (ADRC NI) is a research partnership between Queen’s University Belfast and Ulster University to facilitate access to linked administrative data for research purposes for public benefit and for evidence-based policy development. This requires a social licence extended by publics which is maintained by a robust approach to engagement and involvement.

Public engagement is central to the ADRC NI’s approach to research. Research impact is pursued and secured through robust engagement and co-production of research with publics and key stakeholders. This is done by focusing on data subjects (the cohort of people whose lives make up the datasets, placing value on experts by experience outside of academic knowledge, and working with public(s) as key data advocates, through project steering committees and targeted events with stakeholders. The work is led by a dedicated Public Engagement, Communications and Impact Manager.

While there are strengths and weaknesses to the ADRC NI approach, examples of successful partnerships and clear pathways to impact demonstrate its utility and ability to amplify the positive impact of administrative data research. Working with publics as data use becomes more ubiquitous in a post-COVID-19 world will become more critical. ADRC NI’s model is a potential way forward….(More)”.

See also Special Issue on Public Involvement and Engagement by the International Journal of Population Data Science.

Commission proposes measures to boost data sharing and support European data spaces

Curated on Posted on by Stefaan Verhulst

Press Release: “To better exploit the potential of ever-growing data in a trustworthy European framework, the Commission today proposes new rules on data governance. The Regulation will facilitate data sharing across the EU and between sectors to create wealth for society, increase control and trust of both citizens and companies regarding their data, and offer an alternative European model to data handling practice of major tech platforms.

The amount of data generated by public bodies, businesses and citizens is constantly growing. It is expected to multiply by five between 2018 and 2025. These new rules will allow this data to be harnessed and will pave the way for sectoral European data spaces to benefit society, citizens and companies. In the Commission’s data strategy of February this year, nine such data spaces have been proposed, ranging from industry to energy, and from health to the European Green Deal. They will, for example, contribute to the green transition by improving the management of energy consumption, make delivery of personalised medicine a reality, and facilitate access to public services.

The Regulation includes:

  • A number of measures to increase trust in data sharing, as the lack of trust is currently a major obstacle and results in high costs.
  • Create new EU rules on neutrality to allow novel data intermediaries to function as trustworthy organisers of data sharing.
  • Measures to facilitate the reuse of certain data held by the public sector. For example, the reuse of health data could advance research to find cures for rare or chronic diseases.
  • Means to give Europeans control on the use of the data they generate, by making it easier and safer for companies and individuals to voluntarily make their data available for the wider common good under clear conditions….(More)”.

Introducing Reach: find and track research being put into action

Curated on Posted on by Stefaan Verhulst

Blog by Dawn Duhaney: “At Wellcome Data Labs we’re releasing our first product, Reach. Our goal is to support funding organisations and researchers by making it easier to find and track scientific research being put into action by governments and global health organisations.

https://reach.wellcomedatalabs.org/
https://reach.wellcomedatalabs.org/

We focused on solving this problem in collaboration with our internal Insights and Analysis team for Wellcome and with partner organisations before deciding to release Reach more widely.

We found that evaluation teams wanted tools to help them measure the influence academic research was having on policy making institutions. We noticed that it is often challenging to track how scientific evidence makes its way into policy making. Institutions like the UK Government and the World Health Organisation have hundreds of thousands of policy documents available — it’s a heavily manual task to search through them to find evidence of our funded research.

At Wellcome we have some established methods for collecting evidence of policy influence from our funded research such as end of scheme reporting and via word of mouth. Through these methods we found great examples of how funded research was being put into policy and practice by government and global health organisations.

One example is from Kenya. The KEMRI Research Programme — a collaboration between the Kenyan Medical Research Institute, Wellcome and Oxford University launched a research programme to improve maternal health in 2005. Their research was cited in the World Health Organisation and with advocacy efforts from the KEMRI team influenced the development of new Kenyan national guidelines of paediatric care.

In Wellcome Data Labs we wanted to build a tool that would aid the discovery of evidence based policy making and be a step in the process of assessing research influence for evaluators, researchers and funding institutions….(More)”.