EBP+: Integrating science into policy evaluation using Evidential Pluralism

Curated on Posted on by Stefaan Verhulst

Article by Joe Jones, Alexandra Trofimov, Michael Wilde & Jon Williamson: “…While the need to integrate scientific evidence in policymaking is clear, there isn’t a universally accepted framework for doing so in practice. Orthodox evidence-based approaches take Randomised Controlled Trials (RCTs) as the gold standard of evidence. Others argue that social policy issues require theory-based methods to understand the complexities of policy interventions. These divisions may only further decrease trust in science at this critical time.

EBP+ offers a broader framework within which both orthodox and theory-based methods can sit. EBP+ also provides a systematic account of how to integrate and evaluate these different types of evidence. EBP+ can offer consistency and objectivity in policy evaluation, and could yield a unified approach that increases public trust in scientifically-informed policy…

EBP+ is motivated by Evidential Pluralism, a philosophical theory of causal enquiry that has been developed over the last 15 years. Evidential Pluralism encompasses two key claims. The first, object pluralism, says that establishing that A is a cause of B (e.g., that a policy intervention causes a specific outcome) requires establishing both that A and B are appropriately correlated and that there is some mechanism which links the two and which can account for the extent of the correlation. The second claim, study pluralism, maintains that assessing whether A is a cause of B requires assessing both association studies (studies that repeatedly measure A and B, together with potential confounders, to measure their association) and mechanistic studies (studies of features of the mechanisms linking A to B), where available…(More)”.

A diagrammatic representation of Evidential Pluralism
Evidential Pluralism (© Jon Williamson)

Narratives Online. Shared Stories in Social Media

Curated on Posted on by Stefaan Verhulst

Book by Ruth Page: “Stories are shared by millions of people online every day. They post and re-post interactions as they re-tell and respond to large-scale mediated events. These stories are important as they can bring people together, or polarise them in opposing groups. Narratives Online explores this new genre – the shared story – and uses carefully chosen case-studies to illustrate the complex processes of sharing as they are shaped by four international social media contexts: Wikipedia, Facebook, Twitter and YouTube. Building on discourse analytic research, Ruth Page develops a new framework – ‘Mediated Narrative Analysis’ – to address the large scale, multimodal nature of online narratives, helping researchers interpret the micro- and macro-level politics that are played out in computer-mediated communication…(More)”.

What Does Information Integrity Mean for Democracies?

Curated on Posted on by Stefaan Verhulst

Article by Kamya Yadav and Samantha Lai: “Democracies around the world are encountering unique challenges with the rise of new technologies. Experts continue to debate how social media has impacted democratic discourse, pointing to how algorithmic recommendations, influence operations, and cultural changes in norms of communication alter the way people consume information. Meanwhile, developments in artificial intelligence (AI) surface new concerns over how the technology might affect voters’ decision-making process. Already, we have seen its increased use in relation to political campaigning. 

In the run-up to Pakistan’s 2024 presidential elections, former Prime Minister Imran Khan used an artificially generated speech to campaign while imprisoned. Meanwhile, in the United States, a private company used an AI-generated imitation of President Biden’s voice to discourage people from voting. In response, the Federal Communications Commission outlawed the use of AI-generated robocalls.

Evolving technologies present new threats. Disinformation, misinformation, and propaganda are all different faces of the same problem: Our information environment—the ecosystem in which we disseminate, create, receive, and process information—is not secure and we lack coherent goals to direct policy actions. Formulating short-term, reactive policy to counter or mitigate the effects of disinformation or propaganda can only bring us so far. Beyond defending democracies from unending threats, we should also be looking at what it will take to strengthen them. This begs the question: How do we work toward building secure and resilient information ecosystems? How can policymakers and democratic governments identify policy areas that require further improvement and shape their actions accordingly?…(More)”.

How artificial intelligence can facilitate investigative journalism

Curated on Posted on by Stefaan Verhulst

Article by Luiz Fernando Toledo: “A few years ago, I worked on a project for a large Brazilian television channel whose objective was to analyze the profiles of more than 250 guardianship counselors in the city of São Paulo. These elected professionals have the mission of protecting the rights of children and adolescents in Brazil.

Critics had pointed out that some counselors did not have any expertise or prior experience working with young people and were only elected with the support of religious communities. The investigation sought to verify whether these elected counselors had professional training in working with children and adolescents or had any relationships with churches.

After requesting the counselors’ resumes through Brazil’s access to information law, a small team combed through each resume in depth—a laborious and time-consuming task. But today, this project might have required far less time and labor. Rapid developments in generative AI hold potential to significantly scale access and analysis of data needed for investigative journalism.

Many articles address the potential risks of generative AI for journalism and democracy, such as threats AI poses to the business model for journalism and its ability to facilitate the creation and spread of mis- and disinformation. No doubt there is cause for concern. But technology will continue to evolve, and it is up to journalists and researchers to understand how to use it in favor of the public interest.

I wanted to test how generative AI can help journalists, especially those that work with public documents and data. I tested several tools, including Ask Your PDF (ask questions to any documents in your computer), Chatbase (create your own chatbot), and Document Cloud (upload documents and ask GPT-like questions to hundreds of documents simultaneously).

These tools make use of the same mechanism that operates OpenAI’s famous ChatGPT—large language models that create human-like text. But they analyze the user’s own documents rather than information on the internet, ensuring more accurate answers by using specific, user-provided sources…(More)”.

Youth Media Literacy Program Fact Checking Manual

Curated on Posted on by Stefaan Verhulst

Internews: “As part of the USAID-funded Advancing Rights in Southern Africa Program (ARISA), Internews developed the Youth Media Literacy Program to enhance the digital literacy skills of young people. Drawing from university journalism students, and young leaders from civil society organizations in Botswana, Eswatini, Lesotho, and South Africa, the program equipped 124 young people to apply critical thinking to online communication and practice improved digital hygiene and digital security practices. The Youth Media Literacy Program Fact Checking Manual was developed to provide additional support and tools to combat misinformation and disinformation and improve online behavior and security…(More)”.

How to Run a Public Records Audit with a Team of Students

Curated on Posted on by Stefaan Verhulst

Article by By Lam Thuy Vo: “…The Markup (like many other organizations) uses public record requests as an important investigative tool, and we’ve published tips for fellow journalists on how to best craft their requests for specific investigations. But based on where government institutions are located, public record laws vary. Generally, government institutions are required to release documents to anyone who requests them, except when information falls under a specific exemption, like information that invades an individual’s privacy or if there are trade secrets. Federal institutions are governed by the Freedom of Information Act (FOIA), but state or local government agencies have their own state freedom of information laws, and they aren’t all identical. 

Public record audits take a step back. By sending the same freedom of information (FOI) request to agencies around the country, audits can help journalists, researchers and everyday people track which agency will release records and which may not, and if they’re complying with state laws. According to the national freedom of information coalition, “audits have led to legislative reforms and the establishment of ombudsman positions to represent the public’s interests.” 

The basics of auditing is simple: Send the same FOI request to different government agencies, document how you followed up, and document the outcome. Here’s how we coordinated this process with student reporters…(More)”.

Power and Governance in the Age of AI

Curated on Posted on by Stefaan Verhulst

Reflections by several experts: “The best way to think about ChatGPT is as the functional equivalent of expensive private education and tutoring. Yes, there is a free version, but there is also a paid subscription that gets you access to the latest breakthroughs and a more powerful version of the model. More money gets you more power and privileged access. As a result, in my courses at Middlebury College this spring, I was obliged to include the following statement in my syllabus:

“Policy on the use of ChatGPT: You may all use the free version however you like and are encouraged to do so. For purposes of equity, use of the subscription version is forbidden and will be considered a violation of the Honor Code. Your professor has both versions and knows the difference. To ensure you are learning as much as possible from the course readings, careful citation will be mandatory in both your informal and formal writing.”

The United States fails to live up to its founding values when it supports a luxury brand-driven approach to educating its future leaders that is accessible to the privileged and a few select lottery winners. One such “winning ticket” student in my class this spring argued that the quality-education-for-all issue was of such importance for the future of freedom that he would trade his individual good fortune at winning an education at Middlebury College for the elimination of ALL elite education in the United States so that quality education could be a right rather than a privilege.

A democracy cannot function if the entire game seems to be rigged and bought by elites. This is true for the United States and for democracies in the making or under challenge around the world. Consequently, in partnership with other liberal democracies, the U.S. government must do whatever it can to render both public and private governance more transparent and accountable. We should not expect authoritarian states to help us uphold liberal democratic values, nor should we expect corporations to do so voluntarily…(More)”.

The Future of Trust

Curated on Posted on by Stefaan Verhulst

Book by Ros Taylor: “In a society battered by economic, political, cultural and ecological collapse, where do we place our trust, now that it is more vital than ever for our survival? How has that trust – in our laws, our media, our governments – been lost, and how can it be won back? Examining the police, the rule of law, artificial intelligence, the 21st century city and social media, Ros Taylor imagines what life might be like in years to come if trust continues to erode.

Have conspiracy theories permanently damaged our society? Will technological advances, which require more and more of our human selves, ultimately be rejected by future generations? And in a world fast approaching irreversible levels of ecological damage, how can we trust the custodians of these institutions to do the right thing – even as humanity faces catastrophe?…(More)”.

Mark the good stuff: Content provenance and the fight against disinformation

Curated on Posted on by Stefaan Verhulst

BBC Blog: “BBC News’s Verify team is a dedicated group of 60 journalists who fact-check, verify video, counter disinformation, analyse data and – crucially – explain complex stories in the pursuit of truth. On Monday, March 4th, Verify published their first article using a new open media provenance technology called C2PA. The C2PA standard is a technology that records digitally signed information about the provenance of imagery, video and audio – information (or signals) that shows where a piece of media has come from and how it’s been edited. Like an audit trail or a history, these signals are called ‘content credentials’.

Content credentials can be used to help audiences distinguish between authentic, trustworthy media and content that has been faked. The digital signature attached to the provenance information ensures that when the media is “validated”, the person or computer reading the image can be sure that it came from the BBC (or any other source with its own x.509 certificate).

This is important for two reasons. First, it gives publishers like the BBC the ability to share transparently with our audiences what we do every day to deliver great journalism. It also allows us to mark content that is shared across third party platforms (like Facebook) so audiences can trust that when they see a piece of BBC content it does in fact come from the BBC.

For the past three years, BBC R&D has been an active partner in the development of the C2PA standard. It has been developed in collaboration with major media and technology partners, including Microsoft, the New York Times and Adobe. Membership in C2PA is growing to include organisations from all over the world, from established hardware manufacturers like Canon, to technology leaders like OpenAI, fellow media organisations like NHK, and even the Publicis Group covering the advertising industry. Google has now joined the C2PA steering committee and social media companies are leaning in too: Meta has recently announced they are actively assessing implementing C2PA across their platforms…(More)”.

Civic Trust: What’s In A Concept?

Curated on Posted on by Stefaan Verhulst

Article by Stefaan Verhulst, Andrew J. Zahuranec, Oscar Romero and Kim Ochilo: “We will only be able to improve civic trust once we know how to measure it…

A visualization of the ways to measure civic trust

Recently, there’s been a noticeable decline in trust toward institutions across different sectors of society. This is a serious issue, as evidenced by surveys including the Edelman Trust Barometer, Gallup, and Pew Research.

Diminishing trust presents substantial obstacles. It threatens to weaken the foundation of a pluralistic democracy, adversely affects public health, and hinders the collaboration needed to tackle worldwide challenges such as climate change. Trust forms the cornerstone of democratic social contracts and is crucial for maintaining the civic agreements essential for the prosperity and cohesion of communities, cities, and countries alike.

Yet to increase civic trust, we need to know what we mean by it and how to measure it, which turns out to be a challenging exercise. Toward that end, The GovLab at New York University and the New York Civic Engagement Commission joined forces to catalog and identify methodologies to quantify and understand the nuances of civic trust.

“Building trust across New York is essential if we want to deepen civic engagement,” said Sarah Sayeed, Chair and Executive Director of the Civic Engagement Commission. “Trust is the cornerstone of a healthy community and robust democracy.”

This blog delves into various strategies for developing metrics to measure civic trust, informed by our own desk research, which categorizes civic trust metrics into descriptive, diagnostic, and evaluative measures…(More)”.