Papers & posters

Heuristic judgments are used as substitutes, wherein a simple task is substituted for a more complicated one. Social engineering exploits these cognitive processes at the level of the situation, concatenating multiple heuristics into a response. The target attribute (e.g., caller and request legitimate) is inaccessible due to an absence of familiarity, specific knowledge, and visual cues, and the limited verification procedures available to a recipient who maintains no independent evidence of identity. In contrast, heuristic-associated social cues and associated social scripts are highly accessible, because scripts for routine transactions are retrieved from partial cues, with high frequency scripts dominating retrieval under time pressure. This paper defines the process of social script substitution: an illegitimate script is supplied that satisfies the accessible question, and compliance is a function of correct social script recognition rather than from failure to notice a cue. In an analogous manner to the choice-rule formulation of categorization, the probability of substitution can be defined as a ratio of cue evidence for the illegitimate script to cue evidence for both candidates, and that evidence is set by each cue's positional fit to the decision point, its similarity to the legitimate script, its power to discriminate the two scripts, and the resources available to evaluate it.

Security is a decision-making process, and the models a field uses to describe that process determine what it can see. Signal detection theory (SDT), the default account, treats a security decision as a single, timeless comparison of evidence to a criterion, summarized in two numbers. This paper argues that SDT is a useful teaching device and an incomplete description of defender and target cognition. It lacks an account of evidence accumulation, dynamic threshold change, information loss, and erroneous categorization. A leaky, competing accumulator model, in which evidence is sampled over time and decays without reinforcement, retains SDT’s core constructs (sensitivity, criterion) and adds history, decay, categorical status, and a confidence readout. I apply the model to two settings. In SOC alert triage, threshold narrows during backlogs and early signals decay across interruptions. In a social engineering victim, authority framing, urgency, and secondhand reassurance are aggregated with legitimate evidence. Four parameters (drift rate, leak rate, response threshold, starting point) anchor a set of documented biases that attackers exploit across sequential, multi-cue attacks.

Conversational AI could implant false memories in its users, at scale and by default, and through weaponization by bad actors. Psychology research shows false memories of events can be implanted using known techniques: personal information, guided imagination, praise, and repetition. Conversational AI can facilitate each one. Memory and personalization features hold more autobiographical detail than any human experimenter ever had, sycophantic models praise recall, and context compression overwrites the verbatim record that could expose a distortion. Memory poisoning attacks could exploit the same mechanisms deliberately. Once implanted, false memories are typically self-concealing, and they can irreparably change who people think they are and how they remember collective, societally important events. We argue that human memory is part of the AI attack surface, that AI-induced false memories need imminent empirical study, and that design principles need to be implemented to protect against what we term “false memory attacks”.

Some people are more inclined than others to treat generative AI as an agent with beliefs and goals rather than as a designed artefact (sometimes referred to as adopting an “intentional stance” towards it). We tested whether this disposition predicts behavioural reliance on AI advice, and whether it does so beyond general anthropomorphism. In a preregistered online experiment (N = 200), participants completed a judge-advisor forecasting task, giving an initial estimate, receiving advice from an AI, then giving a final estimate. Participants were randomly assigned to a mentalistic or mechanistic framing of the same advisor, and afterwards completed the Epistemic Stance Test for Generative AI (EST-GenAI), our psychometrically validated measure of dispositional intentional stance toward generative AI, and the Individual Differences in Anthropomorphism Questionnaire (IDAQ). While the mechanistic advisor received more weight overall, the dispositional intentional stance positively predicted reliance on AI advice, an association that held with each IDAQ subscale (technology, nature, animal) entered as a covariate and that was specific to the mentalistic framing: users low in intentional stance discounted the mentalistic advisor, while high-stance users weighted it as fully as the mechanistic one. Overall, those most inclined to treat AI as a thinking agent relied on it most, but only when it presented itself as one. This makes intentional stance taking a candidate cognitive security risk: a disposition that shapes where over- and under-reliance will land.

Software environments increasingly affect human cognition, yet they are not designed with cognitive shaping in mind. An intentional approach to crafting one’s own cognition requires an open science of what kind of cognition is fostered and hindered by different software environments. We introduce a framework for software development that supports both inference about cognition and user agency over the kind of cognition they want. Software becomes the interface through which users and developers collaboratively form cognition and formulate questions about cognition.

One hundred known errors were planted into 10 open-access psychology papers, and the modified papers were then run through frontier models and two commercial AI review tools. The best single system caught 71 of the 100 errors, while the worst caught 30. Pooling every system's output caught 93. Models are only partly correlated in the errors they find, which makes ensembling a big lever for finding issues in papers. Seven errors could not be caught by any system, and all seven were omissions: information deleted from a paper rather than mistakes inserted into it. Refine.ink contributes more unique catches than any other single system, though it is expensive on a per-error basis. False positives were not measured, and how this error distribution compares to the distribution of errors in real papers remains unknown. The papers, errors, model outputs, and full experiment log are public, so that others can build on this toward a comprehensive evaluation benchmark across disciplines.

We study the problem of mitigating information overload in large-scale digital ecosystems through dynamically learned taxation mechanisms. We model the interaction between a social planner and strategic message senders as a Stackelberg game, where the planner (leader) sets a tax policy on messaging and the senders (followers) adapt their behavior over time via reinforcement learning under limited interaction horizons, giving rise to non-equilibrium and boundedly rational behavior. The planner’s objective balances reducing aggregate message volume with promoting equity in access to user attention, formalized via a log-participation reward and the Gini coefficient. Leveraging a query-based follower oracle, we show that the leader’s policy optimization problem can be cast as a partially observable Markov decision process (POMDP), enabling the use of reinforcement learning to search over tax policies. In the one-step single-sender, one-step multi-sender, and multi-step single-sender environments, RL-trained tax policies reach or closely approximate the analytically computed optimum, including exact, stable convergence to the dynamic-programming optimum in the multi-step setting under our best hyperparameter configuration; we additionally observe that other hyperparameter choices in the multi-step setting reach a comparable peak but subsequently decay to a majority of it, a sensitivity we have not yet fully characterized. The multi-step, multi-sender environment is in progress. These results suggest that dynamic taxation, framed as a POMDP over learning-based follower responses, is a viable and promising approach to regulating information overload in attention-constrained environments.

Governments, industry, and academia have all remarked on the potential for generative artificial intelligence to greatly enhance the efficacy of adversarial influence operations by driving down the cost of convincing synthetic media. So far, this threat has not been realized. Significant research has explored facets of this conundrum, but there remains room for frameworks that explain why. This paper puts forward George Akerlof’s analysis of markets under quality uncertainty. Akerlof showed how information asymmetries can bring down a market; he also showed why they usually do not. Guarantees, licensing, and brand names establish verification externally, beyond the interaction between a customer and the purveyor of potentially dubious goods. This paper contends that similar mechanisms can be applied to synthetic media, and it identifies four pathways for securing collective sensemaking against GAI-enabled synthetic media: technological, institutional, legal, and social. It considers each against the influence operations that targeted the 2024 United States presidential election, an episode in which abundant synthetic media produced no observable collapse of the information environment. It also acknowledges substantial limitations: disinformation may not work as commonly assumed, audiences may respond to uncertainty by discounting everything rather than believing falsehoods, some audiences may prefer lemons, and verification mechanisms are themselves fragile.

Cognitive security is the “integrity of the conditions under which people and groups reason for themselves.” Research in this area has concentrated at two levels: the cognitive vulnerabilities of individuals, and the design of the systems and interfaces those individuals encounter. Both are necessary, but this position paper argues that a third level determines whether interventions at the first two can reliably function in organizations: the accountability regime that surrounds the human-AI interface. Critical infrastructure organizations have historically maintained two accountability regimes simultaneously: procedural, which emphasize shared procedural standards, documented workflows, and coordinated adherence to established protocols, and discretionary, which emphasize and protect professional judgment through designated roles that enable professionals to exercise tacit, contextual knowledge. Drawing on organizational research on accountability and comments I recently submitted to NIST on its Trustworthy AI in Critical Infrastructure Profile, I argue that AI is dismantling the organization’s ability to maintain both regimes. This collapse produces two opposing failure modes—mindless conformity and reckless deviation—that map directly onto this community’s concern with over- and under-reliance. I offer a diagnostic (the asymmetry test), three design principles, and a research agenda that treats the accountability regime as a moderator of every mitigation the Cognitive Vulnerability and Mitigation Catalog will contain.

Deepfake non-consensual intimate imagery is the dominant real-world use of generative image technology, yet it has received comparatively little attention from the technical AI community. Where deepfake NCII has been studied, it has largely been treated as an intellectual-property problem to be met with existing privacy and copyright law—a framing that neglects both its nature as a form of sexual violence and the expertise of the frontline that already responds to it. Reframing deepfake NCII as technology-facilitated sexual violence, I draw on a pilot study of semi-structured interviews with twelve U.S.-based sexual violence service providers to make two contributions. First, I characterize the frontline of deepfake NCII response as it currently operates, identifying four concerns: insufficient specialized training, scarce resources, the interlocking failure of under-legislation and under-enforcement, and a desire for stronger collaborative pipelines and prevention. Second, I synthesize these findings into a framework for frontline-centered AI governance, which treats the intervention and prevention workers who encounter harm first as constitutive participants in how AI systems are governed rather than as downstream recipients of governance decisions. I argue that this framework generalizes beyond deepfake NCII to other domains of AI safety.

Cognitive security concerns the integrity of attention, perception, memory, and judgment that individuals and institutions depend on to interpret the world and act effectively. AI-enabled manipulation, such as through generative models and recommender systems, has become a central focus of research, funding, and media coverage, often involving a deliberate attacker influencing a single, high-stakes decision. In contrast, this paper categorizes threats using two axes. The first axis, timing, distinguishes between acute capture, where a decision is hijacked within a single window, and chronic drift, where incremental nudges accumulate and gradually distort subsequent decisions. The second axis, source, differentiates adversarial harm, which involves an intentional attacker, from emergent harm, which arises from systemic incentives without deliberate intent. Each of the four resulting categories is based on a distinct threat model and requires tailored defenses, and two of them remain underexplored. Additionally, cognitive offloading exists outside this framework. Cognitive offloading, often accelerated by AI systems that automate reasoning or decision-making, does not redirect decisions toward incorrect outcomes but instead diminishes the capacity for independent decision-making. Most manipulation-focused defenses do not address it. We outline how each category can be measured, identify where the axes may obscure certain problems, and propose priorities for future research agendas.