Stefaan Verhulst

Article by Andrea Splendiani: “How can we use ontologies and semantics to improve the performance of LLM-based systems? A lot of articles talk about RAG (Retrieval-Augmented Generation), where ontologies and knowledge graphs help provide the right context to the AI engine.

But I would like to go one step further here, and explore how actual semantic annotation of data affects the performance of AI systems. Starting in this post with a basic question: does structured semantics, some would say FAIR data, actually help? And in what ways?

Going beyond opinion, here is a simple experiment to test what changes when different levels of semantics are added to data. This is, of course, far from a representative or unbiased experiment… Nevertheless, it’s real, it’s practical, and the results contain a few surprises.

The experiment

We use a small dataset (shown below) about people (fictional!) and their parents, and ask an LLM to compute a person’s ancestry. We start with data alone and then add more and more “semantic” information: some minor data cleansing, definitions, knowledge graphs, and ontologies. We then observe how the answers change.

We deliberately left some ambiguity in the dataset. For instance, we don’t know which of the two people highlighted in yellow is the father of “Niklas Zimmermann”.

The results

Here is a summary of the steps and results, with the actual queries and responses presented later.

1) We provided the data only. The response contained many errors, and the LLM was largely confused. What we learned: Data alone is not enough.

2) We provided the same data, with some minor cleansing. The results didn’t change. What we learned: LLMs can cope well with minor data quality issues.

3) We provided definitions for the data (in plain text). Substantial improvement! The LLM was able to reason by drawing on a broader understanding of the domain. However, some confusion remained around edge cases, and different hypotheses were still proposed. What we learned: LLMs can make significant use of semantic information provided through simple definitions.

4) We provided the same information encoded as an RDF Knowledge Graph. The results were essentially the same, but we now realised that we (and the AI) had been making many assumptions that were not actually supported by the data. And that we were, in fact, asking the wrong question! What we learned: We hadn’t fully understood the problem!

5) We provided the information as a curated RDF Knowledge Graph. The answers became more precise and trustworthy, with no alternative hypotheses proposed. What we learned: Not much additional value in terms of the actual results, but an improvement in trust and consistency.

6) We provided an ontology explaining the Knowledge Graph. Bingo! What we learned: Adding an ontology made a difference.

Here is a summary of the steps and results, with the actual queries and responses presented later…(More)”.

Paper by Mossavar-Rahmani Center For Business and Government: “Governments across the world are exploring how generative AI might improve their ability to effectively pass legislation and deliver public services. This includes the UK government, which has stated that utilising AI to reduce costs and improve public services is a national priority. Current efforts to deploy AI tools are focused on streamlining internal bureaucracy, such as writing ministerial briefings or analysing public consultations, and supporting the direct delivery of public services. In contrast, there has been limited exploration of the role generative AI could play in directly improving policy development and implementation, in support of a government’s high-level agenda or manifesto commitments.

This project seeks to add practical context to hypothetical considerations of how generative AI might impact policymaking. Published articles and commentary provide insights and recommendations, with varying degrees of specificity, on the opportunities, risks, and limitations of using AI to support national policymaking. However, there does not appear to be any instances of meaningful ‘direct experimentation’ with currently available generative AI tools, to explore the potential role of generative AI in more detail, identifying context-specific areas of strength, weakness, or risk.

This project is designed as an initial first step towards addressing this research gap. Using our direct experience of government policymaking, we begin to assess the performance and relative ‘value-add’ of using publicly available generative AI tools to support typical policymaking functions. This is conducted through ‘policy simulations’ in which we assessed how useful (or problematic) AI tools were in helping us to perform a range of policymaking tasks.

Our simulated ‘performance assessments’ were conducted in two phases. Firstly, we undertook basic assessments of the utility of publicly available generative AI across typical policymaking tasks for three different high-level policy contexts. Secondly, we conducted a more granular and in-depth assessment in an area where we have domain expertise (Regional Growth and Development), to explore the impact of output verification (by experts) and corpus design on the utility of generative AI in a national policymaking context.

From this research, we have identified an indicative set of policymaking functions for which generative AI appears to offer significant value (provided this deployment is specific, deliberate, and targeted). We also found that the use of generative AI does not replace the need for in-depth policy expertise but rather increases the utilisation of this expertise by a wider range of government stakeholders, and significant risks may be introduced in the absence of meaningful human engagement with AI outputs. Significant limitations of using generative AI to support policymaking also emerged from this preliminary research, alongside the well-documented broader risks associated with generative AI (such as unclear provenance, biased outputs, hallucinations, and broader security risks). The full set of our indicative findings are below, which we use to inform a proposed set of basic recommendations for governments seeking to better utilise generative AI to support national policymaking…(More)”.

Article by Rachel Santarsiero: “In a little-noticed webpage update, the Environmental Protection Agency has announced a plan to decommission its online web archive at the end of the year, a move that would wipe tens of thousands of publicly available environmental resources from the web. The discreet banner notice comes in stark contrast to EPA Administrator Lee Zeldin’s recent public takedowns of the agency’s power plant pollution standards and federal methane regulatory system.

“The EPA’s online archive makes information about a wide variety of environmental issues and past policies available to the public, dating back at least 20 years,” said Gretchen Gehrke, co-founder of the Environmental Data and Governance Initiative (EDGI). “This information is irreplaceable, and if the archive is decommissioned, that information will be lost forever.”

For decades, the web archive has been an unparalleled resource for researchers, legislators, policymakers, and the public writ large. It houses everything from fact sheets and environmental reports to agency policy changes and regulatory actions, making it a tool for “democratic oversight,” according to Gehrke, who for ten years has been tracking changes to vulnerable federal environmental data.

This isn’t the first time the environmental agency has tried to shutter the online web archive. In late January 2022, the Biden administration EPA announced plans to discontinue the online repository, citing burdensome financial upkeep; through a Freedom of Information Act request, Gehrke was able to determine that the agency spent roughly $2 million a year to maintain it. A coalition of environmentalists, historians, and librarians swiftly condemned the decision, with groups like the EDGI and American Society for Environmental History calling the announcement “alarming” and cause for a “tremendous gap” in the historical record…(More)”.

Report by CRS: “As automated tools for decisionmaking, including those using artificial intelligence (AI), have grown in popularity and sophistication, they raise questions about how their use intersects with federal antidiscrimination laws. Federal laws address various kinds of discrimination, including disparate treatment and disparate impact.

Disparate treatment generally occurs when a person makes a decision or classifies people on the basis of a protected trait, such as race, sex, religion, or another characteristic. Int’l Bhd. of Teamsters v. United States, 431 U.S. 324, 335 n.15 (1977).

Disparate impact occurs when a person makes a decision using neutral criteria, but the decision adversely and disproportionately affects members of a protected group, and either the decisionmaker cannot put forth a legally adequate justification for using the criteria, or an alternative exists that would serve the decisionmaker’s purposes with less of a disparate impact. Tex. Dep’t of Hous. & Cmty. Affs. v. Inclusive Cmtys. Project, Inc., 576 U.S. 519, 524–25 (2015).

How disparate treatment and disparate impact frameworks apply to decisionmaking via automated, algorithmic tools (including AI) is not always clear. Some situations involving algorithmic decisionmaking would clearly be disparate treatment under present law. For example, a person covered by antidiscrimination law who intentionally designs an algorithm to discriminate on the basis of a protected characteristic, or who selects an algorithm because of its discriminatory effect on a protected group, engages in disparate treatment. See Pers. Adm’r of Mass. v. Feeney, 442 U.S. 256, 279 (1979).

However, the law is unsettled on how to evaluate cases where no human consciously chooses to discriminate, and yet an automated system classifies people on the basis of protected characteristics. There is also substantial uncertainty over when an algorithmic system could be said to engage in disparate treatment via proxy—that is, classification based on a facially neutral trait that is closely correlated with a protected characteristic…(More)”.

Book edited by Trace Gale, Andrew Curry, Sue Robinson, and Sahar Hadidimoud: “…offers the first comprehensive synthesis of the Three Horizon approach—a participatory approach first introduced by Anthony Hodgson and colleagues, for understanding how change happens in complex systems, envisioning possible and desired futures, and navigating the paths between. The framework provides a versatile scaffolding for navigating complex transitions that translates readily across disciplines, sectors, and worldviews, enabling collaboration across difference—what scholars call a “boundary object”.  It reveals how current systems (Horizon 1) lose fit over time, how alternative possibilities (Horizon 3) emerge from the margins, and how the turbulent space between (Horizon 2) becomes the site of transformative change. 

This edited collection offers a comprehensive synthesis of Three Horizons theory and practice, weaving together contributions from the framework’s co-originators with global case studies, practical tips, and tools. It moves from conceptual foundations through applied practice to deeper questions of perception, hope, and transformative capacity—exploring collective sense-making, strategic reflection, and systemic learning. In a world navigating polycrisis—from climate change and technological disruption to social transformation—Three Horizons approach offers a rare integration of systemic thinking and creative imagination.  This book makes that integration accessible and actionable, an essential resource for academics and practitioners in possibility studies, future and anticipation studies, education, sustainability and environmental sciences, and social change…(More)”

Book edited by V. Joseph Hotz, Ruobin Gong, and Ian M. Schmutte: “… collection of new research findings and ideas that shed light on the current movement to advance data privacy protection methods and the impact these innovations have on the conduct of empirical analysis in economics, computer science and statistics using privacy-protected data. Its chapters provide short introductions and overviews of key issues related to privacy protection of data used in social science research, including outlines of new frameworks for characterizing and assessing privacy protection of data, discussion of challenges for disclosure limitation in the context of survey data and new designs to improve access by researchers and the general public to confidential data…(More)”.

Policy Primer from the Australian Academy of Health and Medical Sciences (AAHMS): Explores the meaning and purpose of horizon scanning, and some of the different shapes this can take; Maps the Australian health and medical science
horizon scanning landscape; Overviews key international approaches to and national systems for health and medical
science-related horizon scanning.
This primer is intended to inform discussion across government, research funders and the broader health and medical sciences (HMS) sector about the role and design of a nationally coordinated HMS horizon scanning capability.
It maps key past and present Australian approaches, as well as some existing international approaches, and identifies questions that could provide a starting point for exploring next steps that Australia could take towards realising a nationally coordinated horizon scanning model…(More)”.

Paper by Janosch Pfeffer et al: “Climate citizens’ assemblies convene randomly selected citizens who, after structured learning and deliberation, develop policy recommendations. Yet, whether they produce more ambitious recommendations and influence political decisions remains contested. Here we conduct a comparative, counterfactual assessment of 488 recommendations from five national and one state-level assembly in Western Europe, drawing on document analysis and expert interviews. The results show that citizens consistently proposed more ambitious climate measures than existing government policy. Recommendations sometimes influenced policy by adding momentum, enriching plans, breaking deadlocks or introducing novel ideas, but such influence was rare. Only the French assembly shows substantial policy impact, with one-third of its recommendations having influenced policy, due to strong power delegation, political leadership and resulting public pressure. In the other cases, policy impact was minimal or absent. These findings show that citizens’ assemblies can raise policy ambition, but translating recommendations into political decisions depends strongly on political conditions…(More)”.

Article by Zoe Baird: “The leaders of the major frontier AI developers are sounding the alarm at the speed at which AI is creating potential catastrophic risk, calling for a slowdown and government regulation. But Congress and the president are far from consensus on government’s role. Congress is unprepared and needs to devote serious attention to legislative options. As it does, it should quickly establish a bipartisan public-private AI Commission to define solutions for responding to public alarm and industry warnings.

There is precedent for using a Commission to build rapid consensus in response to a radical change in the threats at hand. The end of the Cold War was one such moment. The terrorist attack of September 11 was another. In each case, the existing machinery of the US government had been built for a world that no longer existed. The moment required a clear-eyed rethinking of government’s role itself, at a speed government rarely manages.

We are in such a moment now. The people building artificial intelligence — the engineers, researchers, and executives who understand the technology best — are the ones sounding the loudest alarms. That is not normal. Industries do not typically lobby for their own regulation or warn that their products could cause catastrophic harm. When this happens, the debate about whether government needs to act should be over. What remains are the hard questions: what should government’s role be, and how does it organize itself to protect the AI future while maintaining America’s lead in AI development?

Certain steps should be taken right away, like requiring embedded independent reviewers and government safety review, but for Congress to frame regulation it should establish an AI Commission and give it six months to make its initial report. This will provide a thoughtful space where points of view can be explored deeply and participants can learn from each other. Its membership should include congressional leaders of committees for technology, commerce, and national security; former government officials; public interest and private sector representatives, including but not limited to AI developers; and independent domain experts. There are four questions Congress should ask…(More)“.

Paper by Gary King and Steven Worthington: “We show that AI-generated article reviews are now better, by every measure we could construct, than reviews written by well-compensated human experts and judged to be of unusually high quality by journal editors and authors. We then discuss how this result may inform journal policy choices, with concerning downsides as well as potential paths to increase publication speed and quality; reduce adversarial interactions; improve the experience of editors, reviewers, authors, and readers; and advance science…(More)”.