Stefaan Verhulst
Blog Post by Manuela Di Fusco: “Real-world data (RWD) and real-world evidence (RWE) are playing an increasing role in healthcare decision making.
The conduct of RWD studies involves many interconnected stages, ranging from the definition of research questions of high scientific interest, to the design of a study protocol and statistical plan, and the conduct of the analyses, quality reviews, publication and presentation to the scientific community. Every stage requires extensive knowledge, expertise and efforts from the multidisciplinary research team.
There are a number of well-accepted guidelines for good procedural practices in RWD . Despite their stress on the importance of data reliability, relevance and studies being fit for purpose, their recommendations generally focus on methods/analyses and transparent reporting of results. There often is little focus on feasibility concerns at the early stages of a study; ongoing RWD initiatives, too, focus on improving standards and practices for data collection and analyses.
RWD and RWE are playing an increasing role in healthcare decision making.”
The availability and use of new data sources, which have the ability to store health-related data, have been growing globally, and include mobile technologies, electronic patient-reported outcome tools and wearables [1].
As data sources exist in various formats, and are often created for non-research purposes, they have inherent associated limitations – such as missing data. Determining the best approach for collecting complete and quality data is of critical importance. At study conception, it is not always clear if it is reasonable to expect that the research question of interest could be fully answered and all analyses carried out. Numerous methodological and data collection challenges can emerge during study execution. However, some of these downstream study challenges could be proactively addressed through an early feasibility study, concurrent to protocol development. For example, during this exploratory study, datasets may be explored carefully to ensure data points deemed relevant for the study are routinely ascertained and captured sufficiently, despite potential missing data and/or other data source limitations.
Determining the best approach for collecting complete and quality data is of critical importance.”
This feasibility assessment serves primarily as a first step to gain knowledge of the data and ensure realistic assumptions are included in the protocol; relevant sensitivity analyses can test those assumptions, hence setting the basis for successful study development.
Below is a list of key feasibility questions which may guide the technical exploration and conceptualization of a retrospective RWD study. The list is based on experience supporting observational studies on a global scale and is not intended to be exhaustive and representative of all preparatory activities. This technical feasibility analysis should be carried out while considering other relevant aspects, including the novelty and strategic value of the study versus the existing evidence – in the form of randomized controlled trial data and other RWE –, the intended audience, data access/protection, reporting requirements and external validity aspects.
This feasibility assessment serves primarily as a first step to gain knowledge of the data and ensure realistic assumptions are included in the protocol…”
The list may support early discussions among study team members during the preparation and determination of a RWD study.
- Can the population be accurately identified in the data source?
Diagnosis and procedures can be identified through International Classification of Diseases codes; published code validation studies on the population of interest can be a useful guide.
- How generalizable is the population of the data source?
Generalizability issues should be recognized upfront. For example, the patient population for which data is available in the data source might be restricted to a specific geographic region, health insurance plan (e.g. Medicare or commercial), system (hospital/inpatient and ambulatory) or group (e.g. age, gender)…(More)”.
Elizabeth Stuart et al at John Hopkins School of Public Health: “Models can be enormously useful in the context of an epidemic if they synthesize evidence and scientific knowledge. The COVID-19 pandemic is a complex phenomenon and in such a dynamic setting it is nearly impossible to make informed decisions without the assistance models can provide. However, models don’t perfectly capture reality: They simplify reality to help answer specific questions.
Below are 10 tips for making sense of COVID-19 models for decision-making such as directing health care resources to certain areas or identifying how long social distancing policies may need to be in effect.
- Make sure the model fits the question you are trying to answer.
There are many different types of models and a wide variety of questions that models can be used to address. There are three that can be helpful for COVID-19:- Models that simplify how complex systems work, such as disease transmission. This is often done by putting people into compartments related to how a disease spreads, like “susceptible,” “infected,” and “recovered.” While these can be overly simplistic with few data inputs and don’t allow for the uncertainty that exists in a pandemic, they can be useful in the short term to understand basic structures. But these models generally cannot be implemented in ways that account for complex systems or when there is ongoing system or individual behavioral change.
- Forecasting models try to predict what will actually happen. They work by using existing data to project out conclusions over a relatively short time horizon. But these models are challenging to use for mid-term assessment—like a few months out—because of the evolving nature of pandemics.
- Strategic models show multiple scenarios to consider the potential implications of different interventions and contexts. These models try to capture some of the uncertainty about the underlying disease processes and behaviors. They might take a few values of such as the case fatality ratio or the effectiveness of social distancing measures, and play out different scenarios for disease spread over time. These kinds of models can be particularly useful for decision-making.
- Be mindful that forecast models are often built with the goal of change, which affects their shelf life.
The irony of many COVID-19 modeling purposes is that in some cases, especially for forecasting, a key purpose in building and disseminating the model is to invoke behavior change at individual or system levels—e.g., to reinforce the need for physical distancing.
This makes it difficult to assess the performance of forecasting models since the results of the model itself (and reactions to it) become part of the system. In these cases, a forecasting model may look like it was inaccurate, but it may have been accurate for an unmitigated scenario with no behavior change. In fact, a public health success may be when the forecasts do not come to be! - Look for models (and underlying collaborations) that include diverse aspects and expertise.
One of the challenges in modeling COVID-19 is the multitude of factors involved: infectious disease dynamics, social and behavioral factors such as how frequently individuals interact, economic factors such as employment and safety net policies, and more.
One benefit is that we do know that COVID-19 is an infectious disease and we have a good understanding about how related diseases spread. Likewise, health economists and public health experts have years of experience understanding complex social systems. Look for models, and their underlying collaborations, that take advantage of that breadth of existing knowledge….(More)”.
Essay by Stuart Whatley: “It is now a familiar story. A civilization that measures itself by its technological achievements is confronted with the limits of its power. A new threat, a sudden shock, has shown its tools to be wanting, yet it is now more dependent on them than ever before. While the few in a position to wrest back a semblance of control busy themselves preparing new models and methods, the nonessential masses hurl themselves at luminescent screens, like so many moths to the flame.
It is precisely at such moments of technological dependency that one might consider interrogating one’s relationship with technology more broadly. Yes, “this too shall pass,” because technology always holds the key to our salvation. The question is whether it also played a role in our original sin.
In 1909, following a watershed era of technological progress, but preceding the industrialized massacres of the Somme and Verdun, E.M. Forster imagined, in “The Machine Stops,” a future society in which the entirety of lived experience is administered by a kind of mechanical demiurge. The story is the perfect allegory for the moment, owing not least to its account of a society-wide sudden stop and its eerily prescient description of isolated lives experienced wholly through screens.
The denizens (for they are not citizens) of Forster’s world wile away their days in single-occupancy hexagonal underground rooms, where all of their basic needs are made available on demand. “The Machine…feeds us and clothes us and houses us,” they exclaim, “through it we speak to one another, through it we see one another, in it we have our being.” As such, one’s only duty is to abide by the “spirit of the age.” Whereas in the past that may have entailed sacrifices, always to ensure “that the Machine may progress, that the Machine may progress eternally,” most inhabitants now lead lives of leisure, “eating, or sleeping, or producing ideas.”
Yet despite all of their comforts and free time, they are a harried leisure class, because they have absorbed the values of the Machine itself. They are obsessed with efficiency, an impulse that they discharge by trying to render order (“ideas”) from the unmanageable glut of information that the machine spits out. One character, Vashti, is a fully initiated member of the cult of efficiency. She does not bother trying to acquire a bed to fit her smaller stature more comfortably, for she accepts that “to have an alternative size would have involved vast alterations in the Machine.” Nor does she have any interest in traveling, because she generates “no ideas in an air-ship.” To her mind, any habit that “was unproductive of ideas…had no connexion with the habits that really mattered.” Everyone simply accepts that although the machine’s video feeds do not convey the nuances of one’s facial expressions, they’re “good enough for all practical purposes.”
Chief among Vashti’s distractions is her son, Kuno, a Cassandra-like figure who dares to point out that, “The Machine develops—but not on our lines. The Machine proceeds—but not to our goal.” When the mechanical system eventually begins to break down (starting with the music-streaming service, then the beds), the people have no choice but to take further recourse in the Machine. Complaints are lodged with the Committee of the Mending Apparatus, but the Mending Apparatus itself turns out to be broken. Rather than protest further, the people pray and pine for the Machine’s quick recovery. By that “latter day,” Forster explains, they “had become so subservient that they readily adapted themselves to every caprice of the Machine.”…(More)”.
Blog post by Alex Engler: “Existing models have been valuable, but they were not designed to support these types of critical decisions. A new generation of models that estimate the risk of COVID-19 spread for precise geographies—at the county or even more localized level—would be much more informative for these questions. Rather than produce long-term predictions of deaths or hospital utilization, these models could estimate near-term relative risk to inform local policymaking. Going forward, governors and mayors need local, current, and actionable numbers.
Broadly speaking, better models would substantially aid in the “adaptive response” approach to re-opening the economy. In this strategy, policymakers cyclically loosen and re-tighten restrictions, attempting to work back towards a healthy economy without moving so fast as to allow infections to take off again. In an ideal process, restrictions would be eased at such a pace that balances a swift return to normalcy with reducing total COVID-19 infections. Of course, this is impossible in practice, and thus some continued adjustments—the flipping of various controls off and on again—will be necessary. More precise models can help improve this process, providing another lens into when it will be safe to relax restrictions, thus making it easier to do without a disruptive back-and-forth. A more-or-less continuous easing of restrictions is especially valuable, since it is unlikely that second or third rounds of interventions (such as social distancing) would achieve the same high rates of compliance as the first round.
The proliferation of Covid19 Data
These models can incorporate cases, test-positive rates, hospitalization information, deaths, excess deaths, and other known COVID-19 data. While all these data sources are incomplete, an expanding body of research on COVID-19 is making the data more interpretable. This research will become progressively more valuable with more data on the spread of COVID-19 in the U.S. rather than data from other countries or past pandemics.
Further, a broad range of non-COVID-19 data can also inform risk estimates: Population density, age distributions, poverty and uninsured rates, the number of essential frontline workers, and co-morbidity factors can also be included. Community mobility reports from Google and Unacast’s social distancing scorecard can identify how easing restrictions are changing behavior. Small area estimates also allow the models to account for the risk of spread from other nearby geographies. Geospatial statistics cannot account for infectious spread between two large neighboring states, but they would add value for adjacent zip codes. Lastly, many more data sources are in the works, like open patient data registries, the National Institutes of Health’s (NIH) study of asymptomatic persons, self-reported symptoms data from Facebook, and (potentially) new randomized surveys. In fact, there are so many diverse and relevant data streams, that models can add value simply be consolidating daily information into just a few top-line numbers that are comparable across the nation.
FiveThirtyEight has effectively explained that making these models is tremendously difficult due to incomplete data, especially since the U.S. is not testing enough or in statistically valuable ways. These challenges are real, but decision-makers are currently using this same highly flawed data to make inferences and policy choices. Despite the many known problems, elected officials and public health services have no choice. Frequently, they are evaluating the data without the time and expertise to make reasoned statistical interpretations based on epidemiological research, leaving significant opportunity for modeling to help….(More)”.
Press Release: “The National Academies of Sciences, Engineering, and Medicine and the National Science Foundation announced today the formation of a Societal Experts Action Network (SEAN) to connect social and behavioral science researchers with decision-makers who are leading the response to COVID-19. SEAN will respond to the most pressing social, behavioral, and economic questions that are being asked by federal, state, and local officials by working with appropriate experts to quickly provide actionable answers.
The new network’s activities will be overseen by an executive committee in coordination with the National Academies’ Standing Committee on Emerging Infectious Diseases and 21st Century Health Threats, established earlier this year to provide rapid expert input on urgent questions facing the federal government on the COVID-19 pandemic. Standing committee members Robert Groves, executive vice president and provost at Georgetown University, and Mary T. Bassett, director of the François-Xavier Bagnoud Center for Health and Human Rights at Harvard University, will co-chair the executive committee to manage SEAN’s solicitation of questions and expert responses, anticipate leaders’ research needs, and guide the dissemination of network findings.
SEAN will include individual researchers from a broad range of disciplines as well as leading national social and behavioral science institutions. Responses to decision-maker requests may range from individual phone calls and presentations to written committee documents such as Rapid Expert Consultations.
“This pandemic has broadly impacted all aspects of life — not just our health, but our work, families, education, supply chains, and even the global environment,” said Marcia McNutt, president of the National Academy of Sciences. “Therefore, to address the myriad questions that are being raised by mayors, governors, local representatives, and other leaders, we must recruit the full range of scientific expertise from across the social, natural, and biomedical sciences.”
“Our communities and our society at large are facing a range of complex issues on multiple fronts due to COVID-19,” said Arthur Lupia, head of the Directorate for Social, Behavioral, and Economic Sciences at the National Science Foundation. “These are human-centered issues affecting our daily lives — the education and well-being of our children, the strength of our economy, the health of our loved ones, neighbors, and so many more. Through SEAN, social and behavioral scientists will provide actionable, evidence-driven guidance to our leaders across the U.S. who are working to support our communities and speed their recovery.”…(More)”.
Article by Lily Scherlis: “The term “social distancing” trickled into the US news at the end of January, and by mid-March had become the governing creed of interpersonal relations for the time being. It surfaced in the midst of early doubts about the efficacy and ethics of the quarantine in China. The media began to recite it, wrapping it in scare quotes. The omnipresent quotation marks created the impression that reporters were holding the term at bay and contemplating it. By mid-March—after the flood of guidelines from the Center for Disease Control (CDC) and subsequent executive orders—social distancing had become sufficiently imperative for the term to be folded directly into sentences, shedding its quotation marks once and for all. But the initial presence of the quotes reflects the early mass fascination with the unfamiliar term. It materialized as if from nowhere: a scientific coinage, a spontaneous naming of a systematized set of behaviors miraculously devised by presumed experts.
“Social distancing” has actually lived several lives. It and its precursor, “social distance,” had long been used in a variety of colloquial and academic contexts, both as prescriptions and descriptions, before being taken up by epidemiologists in this century. In the nineteenth century, “social distance” was a polite euphemism used by the British to talk about class and by Americans to talk about race. It was then formally adopted in the 1920s by sociologists as a term to facilitate the quantitative codification that was then being introduced into the nascent study of race relations. In the second half of the twentieth century, psychiatry, anthropology, and zoology all adapted it for various purposes. And it was used in the 1990s in the United States to analyze what happened to the gay community when faced with straight fears of contagion. It was only in 2004 in a CDC publication on controlling the recent SARS outbreak that the term “social distance” was finally deployed for the first time by the medical community.
The history I trace here doesn’t presume that the doctors who appropriated it to control disease knew about its legacy, or that these links are relationships of causation. But there was something in the air in 2004 that encouraged the practices we now know as social distancing to be christened in this way—as if its past meanings had coalesced into a semantic atmosphere ripe for the emergence of this new use. Which is why if you think the term is weird, you’re right….(More)”.
Paper by Ishani Mukherjee and Sarah Giest: “Behavioural Insights Teams (BITs) have gained prominence in government as policy advisors and are increasingly linked to the way policy instruments are designed. Despite the rise of BITs as unique knowledge brokers mediating the use of behavioral insights for policymaking, they remain underexplored in the growing literature on policy advice and advisory systems. The article emphasizes that the visible impact that BITs have on the content of policy instruments, the level of political support they garner and their structural diversity in different political departments, all set them apart from typical policy brokers in policy advisory systems connecting the science-policy divide…(More)”.
Paper by Ciro Cattuto and Alessandro Spina: “Amid the outbreak of the SARS-CoV-2 pandemic, there has been a call to use innovative digital tools for the purpose of protecting public health. There are a number of proposals to embed digital solutions into the regulatory strategies adopted by public authorities to control the spread of the coronavirus more effectively. They range from algorithms to detect population movements by using telecommunication data to the use of artificial intelligence and high-performance computing power to detect patterns in the spread of the virus. However, the use of a mobile phone application for contact tracing is certainly the most popular.
These proposals, which have a very powerful persuasive force, and have apparently contributed to the success of public health response in a few Asian countries, also raise questions and criticisms in particular with regard to the risks that these novel digital surveillance systems pose for privacy and in the long term for our democracies.
With this short paper, we would like to describe the pattern that has led to the institutionalization of digital tools for public health purposes. By tracing their origins to “digital epidemiology”, an approach originated in the early 2010s, we will expose that, whilst there exists limited experimental knowledge on the use of digital tools for tracking disease, this is the first time in which they are being introduced by policy-makers into the set of non-clinical emergency strategies to a major public health crisis….(More)”
Book by Dariusz Jemielniak and Aleksandra Przegalinska: “Humans are hard-wired for collaboration, and new technologies of communication act as a super-amplifier of our natural collaborative mindset. This volume in the MIT Press Essential Knowledge series examines the emergence of a new kind of social collaboration enabled by networked technologies. This new collaborative society might be characterized as a series of services and startups that enable peer-to-peer exchanges and interactions though technology. Some believe that the economic aspects of the new collaboration have the potential to make society more equitable; others see collaborative communities based on sharing as a cover for social injustice and user exploitation.
The book covers the “sharing economy,” and the hijacking of the term by corporations; different models of peer production, and motivations to participate; collaborative media production and consumption, the definitions of “amateur” and “professional,” and the power of memes; hactivism and social movements, including Anonymous and anti-ACTA protest; collaborative knowledge creation, including citizen science; collaborative self-tracking; and internet-mediated social relations, as seen in the use of Instagram, Snapchat, and Tinder. Finally, the book considers the future of these collaborative tendencies and the disruptions caused by fake news, bots, and other challenges….(More)”.
Book by Philip N. Howard: “Artificially intelligent “bot” accounts attack politicians and public figures on social media. Conspiracy theorists publish junk news sites to promote their outlandish beliefs. Campaigners create fake dating profiles to attract young voters. We live in a world of technologies that misdirect our attention, poison our political conversations, and jeopardize our democracies. With massive amounts of social media and public polling data, and in-depth interviews with political consultants, bot writers, and journalists, Philip N. Howard offers ways to take these “lie machines” apart.
Lie Machines is full of riveting behind-the-scenes stories from the world’s biggest and most damagingly successful misinformation initiatives—including those used in Brexit and U.S. elections. Howard not only shows how these campaigns evolved from older propaganda operations but also exposes their new powers, gives us insight into their effectiveness, and explains how to shut them down…(More)”.