Article by Stefaan G. Verhulst and Roshni Singh: “Artificial intelligence systems are increasingly designed to remember us. Whether answering a question, drafting an email, or recommending a course of action, modern AI systems draw on accumulated knowledge about a user’s preferences, behaviors, goals, and past interactions to function effectively. This capacity for context — persistent memory about who we are and what we do — is not a secondary feature. It is foundational to how these systems generate value.
But context in AI is more than a technical convenience or feature. It is also a source of risk. The accumulation and reuse of personal information introduces privacy vulnerabilities, particularly when data from different domains is aggregated into a single, unified memory. This is, of course, not a new concern: as Helen Nissenbaum argued in Privacy in Context, privacy depends on maintaining appropriate information flows within specific social contexts, and risks emerge when those boundaries are collapsed. What AI changes is the scale, speed, and inferential power of such aggregation, turning what were once discrete data linkages into continuous, dynamic systems capable of generating new insights, predictions, and vulnerabilities far beyond the original contexts in which the data was produced.
And the persistence of context raises deeper questions about cognitive dependence: when AI systems continuously shape the informational environment in which users think, they do not merely respond to us but influence how we understand ourselves and make decisions. In doing so, they risk constraining what we have described as digital self-determination: the ability of individuals and communities to meaningfully shape the conditions under which their data is (re) used and how it, in turn, shapes them — shifting agency from the user to the system in often opaque and difficult-to-contest ways.
These risks are not limited to one category of AI. They apply across AI systems that store and reuse user data — from large language models and recommendation engines to agentic systems that act autonomously on a user’s behalf. What this article examines is not a particular technology, but a structural feature common to many: the use of context as memory, and the tradeoffs that follow.
Context is often treated as the accumulation of user data, but this framing is incomplete. Context is better understood as the relational structure that gives information meaning by situating it within social, temporal, and functional relationships. It is not simply what is stored, but how information is organized, linked, and interpreted within a given frame. Without these relationships, data may remain present but lose meaning or be misapplied across situations. As Jessica Talisman further elaborates, this spectrum runs from statistical proximity to formal logical commitment; AI systems that conflate these distinct levels of relational strength risk treating correlation as meaning.
In what follows, we draw on emerging writing on AI memory, context, and human-AI interaction to explore three interconnected dimensions of this problem. First, we examine why context matters so much for AI performance, and why it is better understood as a relational structure than as simple data storage. Second, we analyze the privacy risks that arise when contextual boundaries collapse. Third, we consider the cognitive risks of persistent memory: the possibility that AI systems come to shape not only what users do, but how they think. Across these dimensions, we also consider the implications for digital self-determination — that is, the extent to which individuals and communities retain meaningful agency over how they are represented, interpreted, and acted upon within context-aware AI systems. These concerns are especially acute for children and young users, for whom both data exposure and cognitive development are at stake…(More)”.
