How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals

Curated on Posted on by Stefaan Verhulst

Paper by Eric Wu et al: “Medical artificial-intelligence (AI) algorithms are being increasingly proposed for the assessment and care of patients. Although the academic community has started to develop reporting guidelines for AI clinical trials, there are no established best practices for evaluating commercially available algorithms to ensure their reliability and safety. The path to safe and robust clinical AI requires that important regulatory questions be addressed. Are medical devices able to demonstrate performance that can be generalized to the entire intended population? Are commonly faced shortcomings of AI (overfitting to training data, vulnerability to data shifts, and bias against underrepresented patient subgroups) adequately quantified and addressed?

In the USA, the US Food and Drug Administration (FDA) is responsible for approving commercially marketed medical AI devices. The FDA releases publicly available information on approved devices in the form of a summary document that generally contains information about the device description, indications for use, and performance data of the device’s evaluation study. The FDA has recently called for improvement of test-data quality, improvement of trust and transparency with users, monitoring of algorithmic performance and bias on the intended population, and testing with clinicians in the loop. To understand the extent to which these concerns are addressed in practice, we have created an annotated database of FDA-approved medical AI devices and systematically analyzed how these devices were evaluated before approval. Additionally, we have conducted a case study of pneumothorax-triage devices and found that evaluating deep-learning models at a single site alone, which is often done, can mask weaknesses in the models and lead to worse performance across sites.

Fig. 1: Breakdown of 130 FDA-approved medical AI devices by body area.

figure1

Devices are categorized by risk level (square, high risk; circle, low risk). Blue indicates that a multi-site evaluation was reported; otherwise, symbols are gray. Red outline indicates a prospective study (key, right margin). Numbers in key indicate the number of devices with each characteristic….(More)”.

Data Access, Consumer Interests and Public Welfare

Curated on Posted on by Stefaan Verhulst

Book edited by Bundesministerium der Justiz und für Verbraucherschutz, and Max-Planck-Institut für Innovation und Wettbewerb: “Data are considered to be key for the functioning of the data economy as well as for pursuing multiple public interest concerns. Against this backdrop this book strives to device new data access rules for future legislation. To do so, the contributions first explain the justification for such rules from an economic and more general policy perspective. Then, building on the constitutional foundations and existing access regimes, they explore the potential of various fields of the law (competition and contract law, data protection and consumer law, sector-specific regulation) as a basis for the future legal framework. The book also addresses the need to coordinate data access rules with intellectual property rights and to integrate these rules as one of multiple measures in larger data governance systems. Finally, the book discusses the enforcement of the Government’s interest in using privately held data as well as potential data access rights of the users of connected devices….(More)”.

Our Tomorrows- A Community Sensemaking Approach

Curated on Posted on by Stefaan Verhulst

OPSI Case Study: “The Kansas vision for the early childhood system is:  All children will have their basic needs met and have equitable access to quality early childhood care and educational opportunities, so they are prepared to succeed in kindergarten and beyond. In 2019, the State of Kansas received a large federal grant (the Preschool Development Grant) to conduct a needs assessment and craft a strategic plan for the early childhood system where all children can thrive. The grant leadership team of state agencies utilized this opportunity to harness the power of Our Tomorrows’ innovative Community Sensemaking Approach to map families’ lived experiences and create policies and programming adaptive to families’ needs.

In this context, Our Tomorrows set out to achieve three goals:

1. Gather stories about thriving and surviving from families across Kansas utilizing a complexity-informed narrative research approach called SenseMaker.

2. Make sense of patterns that emerged from the stories through Community Sensemaking Workshops with stakeholders at various levels of the system.

3. Take action and ennoble bottom-up change through Community Action Labs.

From a complexity perspective, these goals translate to developing a ‘human sensor network,’ embedding citizen feedback loops and sensemaking processes into governance, and complexity-informed intervention via portfolios of safe-to-fail probes….(More)“

Combining Racial Groups in Data Analysis Can Mask Important Differences in Communities

Curated on Posted on by Stefaan Verhulst

Blog by Jonathan Schwabish and Alice Feng: “Surveys, datasets, and published research often lump together racial and ethnic groups, which can erase the experiences of certain communities. Combining groups with different experiences can mask how specific groups and communities are faring and, in turn, affect how government funds are distributed, how services are provided, and how groups are perceived.

Large surveys that collect data on race and ethnicity are used to disburse government funds and services in a number of ways. The US Department of Housing Urban Development, for instance, distributes millions of dollars annually to Native American tribes through the Indian Housing Block Grant. And statistics on race and ethnicity are used as evidence in employment discrimination lawsuits and to help determine whether banks are discriminating against people and communities of color.

Despite the potentially large effects these data can have, researchers don’t always disaggregate their analysis to more racial groups. Many point to small sample sizes as a limitation for including more race and ethnicity categories in their analysis, but efforts to gather more specific data and disaggregate available survey results are critical to creating better policy for everyone.

To illustrate how aggregating racial groups can mask important variation, we looked at the 2019 poverty rate across 139 detailed race categories in the Census Bureau’s annual American Community Survey (ACS). The ACS provides information that helps determine how more than $675 billion in government funds is distributed each year.

The official poverty rate in the United States stood at 10.5 percent in 2019, with significant variation across racial and ethnic groups. The primary question in the ACS concerning race includes 15 separate checkboxes, with space to print additional names or races for some options (a separate question refers to Hispanic or Latino origin).

Screenshot of the American Community Survey's race question

Although the survey offers ample latitude for interviewees to respond with their race, researchers have a tendency to aggregate racial categories. People who identify as Asian or Pacific Islander (API), for example, are often combined in economic analyses.

This aggregation can mask variation within racial or ethnic categories. As an example, one analysis that used the ACS showed 11 percent of children in the API group are in poverty, relative to 18 percent of the overall population. But that estimate could understate the poverty rate among children who identify as Pacific lslanders and could overstate the poverty rate among children who identify as Asian, which itself is a broad grouping that encompasses many different communities with various experiences. Similar aggregating can be found across economic literature, including on education, immigration (PDF), and wealth….(More)”.

Averting Catastrophe

Curated on Posted on by Stefaan Verhulst

Book by Cass Sunstein on “Decision Theory for COVID-19, Climate Change, and Potential Disasters of All Kinds…The world is increasingly confronted with new challenges related to climate change, globalization, disease, and technology. Governments are faced with having to decide how much risk is worth taking, how much destruction and death can be tolerated, and how much money should be invested in the hopes of avoiding catastrophe. Lacking full information, should decision-makers focus on avoiding the most catastrophic outcomes? When should extreme measures be taken to prevent as much destruction as possible?

Averting Catastrophe explores how governments ought to make decisions in times of imminent disaster. Cass R. Sunstein argues that using the “maximin rule,” which calls for choosing the approach that eliminates the worst of the worst-case scenarios, may be necessary when public officials lack important information, and when the worst-case scenario is too disastrous to contemplate. He underscores this argument by emphasizing the reality of “Knightian uncertainty,” found in circumstances in which it is not possible to assign probabilities to various outcomes. Sunstein brings foundational issues in decision theory in close contact with real problems in regulation, law, and daily life, and considers other potential future risks. At once an approachable introduction to decision-theory and a provocative argument for how governments ought to handle risk, Averting Catastrophe offers a definitive path forward in a world rife with uncertainty….(More)”.

The Promise of Access: Technology, Inequality, and the Political Economy of Hope

Curated on Posted on by Stefaan Verhulst

Book by Daniel Greene: “Why simple technological solutions to complex social issues continue to appeal to politicians and professionals who should (and often do) know better.

Why do we keep trying to solve poverty with technology? What makes us feel that we need to learn to code—or else? In The Promise of Access, Daniel Greene argues that the problem of poverty became a problem of technology in order to manage the contradictions of a changing economy. Greene shows how the digital divide emerged as a policy problem and why simple technological solutions to complex social issues continue to appeal to politicians and professionals who should (and often do) know better.

Greene shows why it is so hard to get rid of the idea—which he terms the access doctrine—that the problem of poverty can be solved with the right tools and the right skills. This way of thinking is so ingrained that is adopted by organizations that fight poverty—which often refashion themselves to resemble technology startups. Drawing on years of fieldwork, Greene explores how this plays out in the real world, examining organizational change in technology startups, public libraries, and a charter school in Washington, DC. He finds that as the libraries and school pursue technological solutions, they win praise and funding but also marginalize and alienate the populations they serve. Greene calls for new political alliances that can change the terms on which we understand technology and fight poverty….(More)”

Our Brain Typically Overlooks This Brilliant Problem-Solving Strategy

Curated on Posted on by Stefaan Verhulst

Diana Kwon in Scientific American: “For generations, the standard way to learn how to ride a bicycle was with training wheels or a tricycle. But in recent years, many parents have opted to train their kids with balance bikes, pedalless two-wheelers that enable children to develop the coordination needed for bicycling—a skill that is not as easily acquired with an extra set of wheels.

Given the benefits of balance bikes, why did it take so long for them to replace training wheels? There are plenty of other examples in which overlooked solutions that involve subtraction turn out to be better alternatives. In some European cities, for example, urban planners have gotten rid of traffic lights and road signs to make streets safer—an idea that runs counter to conventional traffic design.

Leidy Klotz, an engineer at the University of Virginia, noticed that minimalist designs, in which elements are removed from an existing model, were uncommon. So he reached out to Gabrielle Adams, a social psychologist at the university, to try to figure out why this was the case. The two researchers hypothesized that there might be a psychological explanation: when faced with a problem, people tend to select solutions that involve adding new elements rather than taking existing components away….

These findings, which were published today in Nature, suggest that “additive solutions have sort of a privileged status—they tend to come to mind quickly and easily,” says Benjamin Converse, a social psychologist at the University of Virginia and a co-author of the study. “Subtractive solutions are not necessarily harder to consider, but they take more effort to find.”…(More)”.

Democratic institutions and prosperity: The benefits of an open society

Curated on Posted on by Stefaan Verhulst

Paper by the European Parliamentary Research Service: “The ongoing structural transformation and the rapid spread of the technologies of the fourth industrial revolution are challenging current democratic institutions and their established forms of governance and regulation.At the same time, these changes offer vast opportunities to enhance, strengthen and expand the existing democratic framework to reflect a more complex and interdependent world. This process has already begun in many democratic societies but further progress is needed.
Examining these issues involves looking at the impact of ongoing complex and simultaneous changes on the theoretical framework underpinning beneficial democratic regulation. More specifically, combining economic, legal and political perspectives, it is necessary to explore how some adaptations to existing democratic institutions could further improve the functioning of democracies while also delivering additional economic benefits to citizens and society as whole. The introduction of a series of promising new tools could offer a potential way to support democratic decision-makers in regulating complexity and tackling ongoing and future challenges. The first of these tools is to use strategic foresight to anticipate and control future events; the second is collective intelligence, following the idea that citizens are collectively capable of providing better solutions to regulatory problems than are public administrations; the third and fourth are concerned with design-thinking and algorithmic regulation respectively. Design-based approaches are credited with opening up innovative options for policy-makers, while algorithms hold the promise of enabling decision-making to handle complex issues while remaining participatory….(More)”.

We Need to Talk About Data: Framing the Debate Around the Free Flow of Data and Data Sovereignty

Curated on Posted on by Stefaan Verhulst

Report by the Internet & Jurisdiction Policy Network: “The Report presents concerns and perspectives around the current use of the concepts “Free Flow of Data” and “Data Sovereignty” and offers key recommendations on how to move forward to foster a collaborative discussion on how to organize our common datasphere….

The Report presents three key findings for moving forward.

  • First, we need a debate that is global, multi-stakeholder, across sectors 
  • Second, the discussion needs to be reframed to ensure that this complex and novel issue is addressed in a much more nuanced manner
  • Third, the Report highlights the need to be innovative in the tools, frameworks and concepts we use to address the issue of data…(More)”

Decisions, Decisions, Decisions.

Curated on Posted on by Stefaan Verhulst

BSR Report on “Responsible Business Decision-Making Before, During, and After Public Health Emergencies: A Rights-Based Approach to Technology and Data Use…The COVID-19 public health emergency has surfaced important questions about the relationship between the right to privacy and other rights, such as the right to health, work, movement, expression, and assembly. Data and digital infrastructures can be used for many positive outcomes, such as facilitating “back to work” efforts, enhancing research into COVID-19 vaccines and treatments, and allowing the resumption of economic activity while also protecting public health.

However, these uses may also result in the infringement of privacy rights, new forms of discrimination, and harm to vulnerable groups. Some governments are using the emergency as an excuse to expand their power, leading to concerns that initiatives launched to address COVID-19 could become permanent forms of state surveillance.

As the providers of data, systems, and software, technology companies are often central in these public health emergency response efforts. For this reason, companies need to address the human rights risks associated with their involvement in disease response to avoid being connected to human rights violations.

This paper sets out the key elements of a human rights-based approach to the use of data and technology solutions during public health emergencies in today and tomorrow’s digital era, with a focus on the role of business and impacts to privacy.

These elements are primarily captured in a human rights-based decision-making framework for companies that can guide them through future public health emergencies. This framework can be found on page 5 of the report or can be downloaded separately.

COVID-19 is the first truly global pandemic of the modern age, but it won’t be the last. We hope this paper highlights lessons learned from COVID-19 that can be applied during the public health emergencies of the future….(More)”.