Article by Andrea Splendiani: “How can we use ontologies and semantics to improve the performance of LLM-based systems? A lot of articles talk about RAG (Retrieval-Augmented Generation), where ontologies and knowledge graphs help provide the right context to the AI engine.
But I would like to go one step further here, and explore how actual semantic annotation of data affects the performance of AI systems. Starting in this post with a basic question: does structured semantics, some would say FAIR data, actually help? And in what ways?
Going beyond opinion, here is a simple experiment to test what changes when different levels of semantics are added to data. This is, of course, far from a representative or unbiased experiment… Nevertheless, it’s real, it’s practical, and the results contain a few surprises.
The experiment
We use a small dataset (shown below) about people (fictional!) and their parents, and ask an LLM to compute a person’s ancestry. We start with data alone and then add more and more “semantic” information: some minor data cleansing, definitions, knowledge graphs, and ontologies. We then observe how the answers change.
We deliberately left some ambiguity in the dataset. For instance, we don’t know which of the two people highlighted in yellow is the father of “Niklas Zimmermann”.
The results
Here is a summary of the steps and results, with the actual queries and responses presented later.
1) We provided the data only. The response contained many errors, and the LLM was largely confused. What we learned: Data alone is not enough.
2) We provided the same data, with some minor cleansing. The results didn’t change. What we learned: LLMs can cope well with minor data quality issues.
3) We provided definitions for the data (in plain text). Substantial improvement! The LLM was able to reason by drawing on a broader understanding of the domain. However, some confusion remained around edge cases, and different hypotheses were still proposed. What we learned: LLMs can make significant use of semantic information provided through simple definitions.
4) We provided the same information encoded as an RDF Knowledge Graph. The results were essentially the same, but we now realised that we (and the AI) had been making many assumptions that were not actually supported by the data. And that we were, in fact, asking the wrong question! What we learned: We hadn’t fully understood the problem!
5) We provided the information as a curated RDF Knowledge Graph. The answers became more precise and trustworthy, with no alternative hypotheses proposed. What we learned: Not much additional value in terms of the actual results, but an improvement in trust and consistency.
6) We provided an ontology explaining the Knowledge Graph. Bingo! What we learned: Adding an ontology made a difference.
Here is a summary of the steps and results, with the actual queries and responses presented later…(More)”.
