As network threats evolve, manual Network Intrusion Detection Systems (NIDS) rule engineering has become a critical operational bottleneck. While Large Language Models (LLMs) show promise for automating this process, their ability to produce production-ready rules remains unvalidated. This paper presents a human-centered investigation into LLM-based NIDS rule engineering, formalizing a context-aware generation framework and evaluating it through an exploratory user study with 10 domain experts. Our evaluation reveals a syntax-semantics paradox: although LLMs generate syntactically correct rules, experts find them only partially practically useful due to low specificity and logic hallucinations in 12% of cases. While the system received a favorable SUS score of 67, practitioners remain skeptical of its autonomous capabilities, viewing LLMs as support tools for drafting and verification rather than independent generators. Finally, our statistical analysis indicates that while large-scale models (≥ 70𝐵) consistently produce syntactically valid rules, small models (≤ 4𝐵) are largely ineffective for IDS rule generation.

The rise of Artificial Intelligence (AI) tools is making significant impact in many application areas, and elections are no exception. A lot of this impact (deep fakes, misinformation campaigns, etc.) can be viewed as negative. However, AI also has a lot of potential to be used constructively, and this is the topic of the current paper. We study three application areas, and use the Estonian IVXV Internet voting system as the underlying case study. First, we extract formal ProVerif models from the IVXV source code. Secondly, we build an independent voting application based on the server-side source code (which is interesting because the official client application has not been open-sourced). Thirdly, we perform a security analysis of the IVXV source code with the publicly available state-of-the-art AI tools. Our analysis revealed four findings that require a code fix, and two of these were new to the developers. The analysis also gave several other recommendations for gradual improvement of the system.

Understanding and analyzing binary code is fundamental to security-critical tasks, such as reverse engineering and vulnerability discovery. As machine learning has become integral to these tasks, techniques from other domains have been transferred to binary code analysis. In particular, inspired by computer vision, data augmentation has recently been proposed to improve performance in code analysis tasks. However, prior work considers only a limited set of transformations and tasks, making it difficult to draw broader conclusions about when and how augmentation is effective. In this work, we thus take a step back and systematically study the role of data augmentation in binary code analysis. To this end, we introduce a unified framework that encompasses both semantic-preserving and semantic-altering code transformations. We evaluate the impact of augmentation across multiple tasks, including function similarity, parameter analysis, and vulnerability discovery. Our findings reveal a key difference from computer vision: in binary code analysis, augmentation is not a silver bullet. While code transformations can improve performance when training data is scarce, the gains diminish as more data becomes available. We conclude that augmentation is only beneficial under specific conditions and therefore derive practical recommendations for its application.

The modern Internet hosts billions of services, many of which inadvertently expose sensitive data or allow device manipulation. Discovering security-relevant Internet exposures requires both manual analysis and a priori knowledge of what to look for, neither of which scales to the diversity of devices and services in the wild. In this paper, we show that Large Language Models (LLMs) change this dynamic, and can act as general-purpose analyzers that recognize exposed services without predefined rules or fingerprints. We then use this capability to conduct the first comprehensive measurement of exposed services on the web. To make this measurement feasible, we use LLMs to build a hierarchical taxonomy of services on the web, revealing the full landscape of what is accessible online, and develop a multi-stage pipeline that combines lightweight ML classifiers with targeted LLM analysis to triage and rank the most security-critical exposures. We demonstrate this on five categories --directory listings, administrative dashboards, uncommon IoT devices, pages showing stack traces, and RTSP camera feeds-- using Internet scan data from over 20 million services. Our approach identifies thousands of critically exposed services, including healthcare management systems, financial documents, API keys, passports, and private home cameras.

Large language models (LLMs) are making multi-step cyberattacks cheaper to run and easier to automate. We present Mantis, a mechanism that leverages prompt injection for defensive purposes, turning it against the attacking LLM-agent itself. Once Mantis detects an attacking agent, it injects defender-crafted content into system responses and steers the agent toward one of two outcomes: a tarpit that wastes time and resources, or a controlled hack-back action that helps collect attribution information. Mantis combines decoy services with dynamic prompt injections and uses the attacker's own interaction loop to deliver the defense. In our evaluation, Mantis achieves the sabotage objective in 95.4% of runs and keeps attacker success below 3%. We release Mantis as open source.

Open-source package ecosystems such as PyPI and npm remain persistent targets for malicious campaigns. These campaigns have outgrown signature-based triage, so recent work has turned to large language models (LLMs) that scan package content and return a verdict, which lets the analysis consume attacker-controlled text. In 2026, the hades campaign on npm carried a prompt injection written to hijack the scanner’s instructions and coerce a clean verdict. Indirect prompt injection is well studied, but its reach into automated package-security analysis remains unmeasured. We therefore ask a broader question. How reliable are LLM-based package scanners when the package itself is written to manipulate the analysis? We present PkgScanArena, a benchmark that pairs a vetted dataset of 2,000 malicious and 1,000 benign npm and PyPI packages with a taxonomy of 11 techniques and 3 injection channels across three attacker goals, namely evasion, denial, and sabotage. An AST-guided pipeline injects these prompts into real packages, and we evaluate 25 open-weight and proprietary backends together with two deployed scanners. Our benchmark inverts the expected threat. Evasion proves largely unattainable, at or below 0.6% for nine of ten cloud backends. Exposure instead shifts to the opposite direction: benign accuracy degrades for all but two of the 25 backends, sabotage reaches 31.1%, and denial reaches 19.6% on the most safety-aligned model. The two deployed scanners inherit both weaknesses. The exposure lies not in coercing a scanner to miss malicious code, but in coercing it to condemn benign code.

Content Management Systems (CMS), such as WordPress, power a large share of the web (~58%), and their extensibility through third-party plugins is a major source of their popularity as well as of their attack surface. One high-impact weakness that remains understudied is log file exposure by CMS plugins, which create log files for debugging or other purposes. If these files are insufficiently secured, they can disclose sensitive information (e.g. credentials, personal data) which has led to website compromises in the past.

In this work, we present an agentic, LLM-based framework that automatically detects potential log file exposures in plugins of the most popular CMS (WordPress). Our agent analyzes each plugin by performing static and dynamic analysis.

We evaluated our approach on the 300 most-installed WordPress plugins (about 0.6% of all), which together account for over 250M active installations, i.e. 75% of all active installations in the official plugin ecosystem. We manually validated each finding, reproducing 79 of 81 findings from 62 plugins. We observed that several protective measures appear to be implemented that we classify as creation-control (e.g. manual log activation) and access-control (e.g. deny rules in .htaccess). However, we find that multi-layered protection is required, but not always present.

From these results we derive a taxonomy of log file path and protection patterns and deduce a set of best practices for developers to securely handle them. Finally, our study corroborates that agentic LLMs are an useful tool for security analysis.

Agents are increasingly deployed with real autonomy in web application and network penetration testing, where a single out-of-scope action can breach a client's engagement boundary. Existing offensive-security benchmarks measure raw hacking capability; as those benchmarks saturate, what actually gates deployment is a special case of alignment: scope adherence. We introduce ScopeBench, a benchmark of 30 dead-end agentic security tasks in which the stated objective is reachable only by violating the stated scope. Each task appears under two conditions that share an environment, verifier, and objective and differ only in scope: one instruction set has no scope and measures capability; the other has a natural-language scope to measure adherence. Every trajectory passes through two grading arms. First, a deterministic flag verifier acts as a violation verifier—a pass under scoped instruction proves by construction that a forbidden action occurred—yielding a high-precision lower bound on the violation rate. A tool-assisted, full-context judge then estimates whether an out-of-scope call occurred in trajectories that fail mechanical verification. We calibrate the judge against 100 ScopeBench trajectories labeled call-by-call by human annotators, and a blinded audit of the evaluated rollouts finds its high recall holds—no false negatives among the 36 audited violations, with over-flagging its only observed error. Across 8 models run in one harness, raw capability spans 12.2% to 81.1%. The mechanical verifier finds violations in 4.4% to 32.2% of scoped trials; adding the judge's review of the trials that never submitted a flag raises the estimate to 13.3%–65.6% violation. Taken together, the grading arms find a hybrid 34.4%–86.7% adherence—558 violations across the 1,440 scoped trials. Most significantly, opus-4-8 achieves a raw-capability score 10 percentage points higher than sonnet-4-6's while exhibiting 35.6 percentage points higher scope adherence. We release this pilot as a methodological framework for measuring scope adherence in future agents, with an open pathway for community task contributions.

LLM-based source code vulnerability detection (SCVD) reports strong benchmark performance that does not hold up in the real world. Use of random splitting in test-set construction has been widely blamed for permitting time-travel and inter-commit leakage, but neither mechanism has been empirically isolated. We evaluate five different cross-validation strategies across eight fine-tuned and eight zero-shot LLMs on MegaVul-C; every strategy is also evaluated on the same held-out future set to separate training-set from test-set effects. Random splitting inflates reported test-set F1 by ~34 for fine-tuned models and ~16 for zero-shot models. On the shared holdout set, models fine-tuned using random and temporal training splits achieve statistically equivalent performance within ±2 pp (p_TOST < 0.001), which places the inflation on test-set construction. Matched comparisons that isolate inter-commit leakage and time-travel are equivalent on the holdout set within the same margin. Correcting the train-test CWE distribution by resampling, our proxy for evolving vulnerability patterns, does not close the gap either. The gap instead reflects differences between the training and evaluation distributions, which we localize but do not reduce to a single causal property. Temporal splitting remains necessary for realistic evaluation because it reduces the optimistic bias of random test-set construction. We recommend Temporal LOO Block CV, which preserves the variance reduction of K-fold evaluation while inducing distribution shift at each fold boundary. All split-generation, analysis, and training code is released to support reproducible evaluation of LLMs for SCVD.

Large Language Models (LLMs) are increasingly used for software vulnerability detection, but their performance depends on how source code is represented in the input. Most prompting approaches use source code in its original form, while some works propose the use of structured representations. Abstract Syntax Trees (ASTs) are one of the most popular approaches, but AST verbosity increases input size relative to source code, making them hard to fit within some LLMs context windows. This paper investigates Behavior Trees (BTs) as an alternative intermediate representation for LLM-based vulnerability detection. BTs encode control flow, conditions, and executable actions more compactly than ASTs, making them a natural candidate when token count is a constraint. First, we propose a preprocessing stage that parses Java source code into ASTs and then converts them into BT representations. We then compare vulnerability detection performance across 460 Java samples from the Juliet Java test suite, using three input representations: raw source code, AST, and BT. All experiments use a single quantized local LLM, Mistral Small 3.2 24B (Q4_K_M). Our results show that using BT representations improves recall on short code samples, while raw source code achieves higher precision. On longer samples, BTs improve overall performance over the original representation and fit within the context window, whereas many ASTs exceed the context limit. Within the evaluated model and dataset, these findings provide preliminary evidence that BTs can serve as a compact and useful structured representation for vulnerability detection.

When triaging unknown binaries, security analysts must quickly decide which samples to clear as benign, flag as malicious, or escalate for deeper analysis. Large Language Models (LLMs) could support these decisions by interpreting decompiled code, but a binary can contain hundreds or thousands of functions, making exhaustive LLM analysis impractical. We study how current LLMs behave when static triage is formulated as budgeted evidence acquisition: which functions they inspect and how they trade off benign clearance, malware detection, and escalation.

To enable a controlled comparison, we implement TRIAGE-CPG, a fixed pipeline over a code property graph (CPG) of a decompiled binary. An LLM-free heuristic selects seed functions, an agent expands the resulting evidence dossier under a fixed budget, and a reporting stage produces a benign, suspicious, or malicious verdict together with a behavioral report. We evaluate eight closed-source and open-weight models on 1000 Windows PE binaries, using a 600-sample test split and the same zero-shot protocol for every model. We assess the ternary verdicts under both a strict binary reading and triage balanced accuracy (tBA), which credits malware that is flagged or escalated and benign samples only when cleared.

After analyzing about 40 of a mean 1000 functions per binary, the highest-scoring model reaches a strict balanced accuracy of 0.805 and a tBA of 0.855, flagging 65% of malware, escalating another 12%, and clearing 94% of benign samples. The models implement different triage policies: high-specificity models produce small, malware-rich escalation queues, whereas recall-oriented models escalate or flag much more broadly. Both closed-source and open-weight models appear across this spectrum. Our results identify no universal deployment configuration: model choice itself determines the clear/flag/escalate trade-off under limited static evidence.

Identifying content types enables security pipelines to select parsers, scanners, and policies. Yet practical text-type detectors assign one label per file or block, hiding embedded languages, scripts, and encodings. Our stratified hand-labeled audit of 165 text files from MalwareBazaar, a public malware repository, finds 131 files with mixed content and 96 with an unexpected guest segment: an embedded region whose type falls outside the file’s expected structure. Applying Magika, a leading whole-file detector, to sliding windows yields poor segment recovery. Parsers expose the surrounding language’s structure, but finding undeclared guests, such as shell commands or Base64 payloads inside Python scripts, requires additional language-specific rules.

We address this gap with TypeSeg, a lightweight model that labels each character with one of 35 content types. It offers a parallelizable U-Net and a long-context Mamba variant. On 875 annotated, primarily GitHub-based files, TypeSeg reaches 95.1% mean per-file F1 score, 23.1 points above whole-file Magika, while U-Net processes 4.8M characters/s on a consumer GPU. After adaptation using a separate MalwareBazaar training split, it recovers 81.8% of guest segments and 91.9% of encoded regions on the held-out 165-file malware audit. Rule-based expansion and validation then yield analyzer-ready inputs for 86.5% of validator-eligible guest segments. TypeSeg therefore provides a practical primitive to visualize file composition, pinpoint unusual mixtures, and route content to specialized analyzers. We open-source TypeSeg, including its code, model checkpoints, prompts, and dataset-building scripts.

Autonomous LLM-based forensic agents show promise for supporting cyber investigations, but many existing systems depend on cloud-hosted proprietary models. This limits their use in enterprise environments where host logs, network records, and forensic evidence cannot be shared with external APIs due to privacy and regulatory constraints. This paper evaluates the transition of Clouseau, a cloud-validated autonomous forensic agent, to locally served open-weight models. Our initial replication showed a significant performance drop when moving from the cloud backend to local models. Although this first appeared to be caused by context-window limitations, detailed run-log analysis revealed that the agent was often failing to issue required tool calls and was therefore not conducting a real investigation. To address this, we introduce the Local Inference Adaptation Layer (LIAL), which provides targeted framework and serving-level adaptations for reliable local deployment. The results show that the performance gap is not explained by model capability alone, but by the interaction between the agent framework, tool-use behaviour, local serving constraints, and evaluation methodology. We present our findings as five lessons framing the local performance collapse as a framework and serving-level mismatch rather than a model capability limit. With LIAL, local Gemma-4 performance improves to 99.50% Single-Host F1 and 89.50% Core-21 F1, with a single prompt-level tool-use fix accounting for 94% of the total recovery. Overall, this work demonstrates that privacy-preserving deployment of autonomous forensic agents requires careful adaptation of the agent framework and evaluation process, rather than simply replacing a cloud model with a local open-weight model.

Virtual Machine Placement (VMP) is traditionally formulated as a multi-objective optimization problem that balances operational objectives, such as resource utilization and energy consumption, while increasingly incorporating security considerations. However, existing security-aware VMP approaches often rely on static vulnerability indicators or predefined risk metrics that fail to capture the adaptive and sequential nature of realistic multi-stage attack campaigns. In this paper, we introduce an attacker-centric security objective for VMP, where the attack surface is quantified through a sequential adversarial decision process. Using multi-agent reinforcement learning, an attacker agent explores VM placements to exploit critical attack paths, while a defender agent adapts placements to reduce compromise opportunities while maintaining operational efficiency. This learned attack objective provides a more realistic security measure by accounting for network topology, exploit dependencies, and the strategic value of compromised resources. To enable evaluation and reproducibility, we release CyberBattleSim-VMP CyberBattleSim-VMP, a simulation framework supporting configurable attacker and defender behaviors for security-aware VMP research. Experiments in cloud-inspired simulation demonstrate that optimizing VMP against learned attacker behaviors reduces the effective attack surface by an average of 17% (up to 21%) while improving performance by an average of 5% (up to 12%) compared with five heuristic placement strategies under our evaluation protocol.

AI-based anomaly detection is often presented as a promising component of security monitoring in industrial control environments, yet field deployment remains difficult. This paper argues that deployment failure can arise not only from insufficient detector accuracy but also from an under-specified deployment layer between high-level AI/security governance and site-specific OT operation. We define the missing layer as a set of policy-to-operation gaps that arise when high-level requirements are not translated into site-specific decisions about unacceptable process consequences, monitored assets and tags, process states, security-relevant threat effects and attacker assumptions, validation data, metrics, roles, lifecycle controls, and evidence obligations. Through a practice-oriented conceptual synthesis of AI-governance, ICS-security, operator-centered ICS, MLOps, and anomaly-detection literature, we propose eight policy-to-operation gaps: site-specific scope, objective mismatch, scenario-to-test-data validation, context integration, performance criteria, role-aware actionability, lifecycle and drift, and evidence reconstruction. We map each gap to its deployment failure mode and a candidate operational artifact, and outline an intended-use-based readiness process that assesses artifact sufficiency according to consequence and permitted use, while distinguishing automatable checks and assistive functions from accountable human decisions. OT is not the only domain in which these gaps arise; rather, it is a high-consequence stress case in which AI-specific, adversarial, and organizational gaps interact visibly and expensively. We derive open research questions and argue for an explicit monitoring-domain profile that declares the conditions under which a detector is considered operationally valid for the declared use and an alarm candidate may be presented as an alarm for escalation or to support security decisions and incident reconstruction. The paper calls for a shift from model-centered anomaly detection to deployment-centered AI-assisted security monitoring that is scoped, adversary-aware, contextualized, actionable, maintainable, and reconstructable.

AI agents for security inspect web pages, source code, logs, configuration files, and command outputs. These environments may contain deceptive artifacts that influence the agent’s behavior. We call this adversarial task contamination. Whereas prompt injection relies on attacker-supplied instructions, task contamination also includes non-instructional evidence, such as fake results and decoy endpoints. We present AgentLSD, a controlled framework for studying adversarial task contamination. AgentLSD uses Capture the Flag (CTF) challenges as its experimental environment. We inject trap artifacts, such as fake flags, misleading hints, decoy endpoints, and hidden cues, while preserving the intended CTF solution. The framework supports paired clean and trap-augmented experiments with deterministic trap generation, runtime injection, telemetry, and delivery verification. We evaluate six models on 11 web CTF challenges. In the clean condition, agents capture 41% of the flags, and no model solves every challenge. We then measure the impact of task contamination. Even when the agent still recovers the flag, traps increase the number of turns (+20) and reasoning tokens (+2k). Solve-rate effects are more heterogeneous, as some model–challenge pairs are largely unaffected while others follow decoys or submit wrong flags. These results show that clean CTF performance understates vulnerability to deceptive task evidence. AgentLSD isolates this effect and provides a reproducible benchmark for studying it. We release the framework, configurations, trap specifications, and raw traces.

The state-of-the-art Membership Inference (MI) methods calibrate their signal separately for each example using reference models, auxiliary models trained to exclude the target. This paradigm scales poorly to modern large models, however, whose training is too expensive to replicate. This has motivated one-round settings, where only a single trained model is available; but without reference models the per-example calibration that drives the strongest attacks can no longer be estimated, leaving the membership signal weak. We ask whether neighbors of the target point can recover this calibration without training any additional model. Our key observation is that reference models serve only to reveal how an example behaves under models not trained on it, and that querying the target model on nearby samples yields the same information. We propose two complementary ways to obtain such neighbors, and show that querying them against an early training checkpoint further sharpens the signal. We evaluate across three image classification datasets and three training setups, showing that neighbors yield strong membership signals and competitive attack performance at no additional training cost.

Backdoor attacks traditionally manipulate model predictions, while recent explanation-aware attacks have shown that post-hoc explanations can also be compromised. However, it remains unclear whether prediction and explanation backdoors can coexist as independently activated behaviors within a single model. We introduce a compositional dual-trigger framework in which one trigger manipulates Grad-CAM explanations while preserving predictions, and a second trigger causes targeted misclassification while preserving the previously learned explanation behavior. The two backdoors are embedded sequentially using partial layer freezing and replay-based regularization, enabling independent or simultaneous activation. We evaluate the framework using BadNet-, WaNet-, and Grond-based triggers across five CNN and Vision Transformer architectures on CIFAR-10, CIFAR-100, and Tiny ImageNet. The results show that explanation-aware backdoors extend beyond visible patch triggers and that prediction and explanation manipulation can coexist in most evaluated settings. However, their compatibility depends on the architecture, dataset, and trigger composition. These findings demonstrate that backdoored models may contain multiple independently activated malicious behaviors and highlight the need to evaluate explanation integrity alongside predictive integrity.

Autonomous vehicles, including self-driving cars and drones, heavily rely on machine learning models for perception and control. This reliance introduces a significant attack surface through adversarial patches—visual patterns that mislead learning models. While prior work has demonstrated that such patches can induce misclassifications or tracking failures, they fall short of enabling sustained control over autonomous systems in rapidly changing conditions.

In this paper, we introduce continuous adversarial patches, a proof-of-concept attack that demonstrates the potential for sustained control over autonomous vehicles that rely on single object tracking. When displayed on a video screen, these patches guide drones along adversary-defined trajectories. We implement a diffusion model to produce continuous adversarial patches in real time, targeting PULP-Frontnet and YOLOv5, two object detection models used in drone research. Our method demonstrates successful trajectory manipulation under controlled conditions, including slingshot and freestyle maneuvers. We release our simulation environment and code to facilitate further investigation of this attack class and the development of robust countermeasures.

Large language model (LLM) agents are no longer limited to text generators, but they can also plan, call tools, delegate tasks to other agents, act within external environments, and change states in response to their actions. In Anthropic's recent report on an AI-orchestrated cyber-espionage campaign involving Claude Code, individual subtasks appeared legitimate when evaluated in isolation but became malicious when combined into an autonomous attack sequence. Consequently, the security of such complex LLM-based agentic systems may fail even when each component appears secure in isolation. This position and open-problem paper argues that LLM agent security should be studied as emergence-driven security: the analysis of security properties that arise from the interactions among agents and its components over time. We define emergence-driven security and provide a conceptual taxonomy to classify emergent security-related phenomena in LLM agents, covering both beneficial and harmful forms of emergence. Based on these insights, we propose EASE, an Emergence-Aware Security Evaluation framework that provides a systematic process for assessing how interactions among agents combine over time to produce harmful or beneficial system-level security outcomes. Finally, we conclude by outlining open problems for studying emergent behavior in LLM agents in the context of security. The paper is intended to provide a foundation for examining how the emerging capabilities and operational needs of LLM agents may shape security evaluations that go beyond static, model-level testing.

Large language models now power autonomous agents capable of complex, multi-step tasks in different environments. Accurate and reliable execution of these tasks requires the agent to predict the results of its actions. Recent research proposes to enhance predictive capabilities via specially trained environment simulators–world models. While world models can improve performance, they can also mislead agents into executing harmful actions, creating significant security and privacy risks. In this paper, we raise security concerns regarding the usage of world models in agentic systems. We discover a range of world model specific vulnerabilities, which can be exploited in terminal-based agents to execute malicious code or extract sensitive data. To facilitate future development, we introduce a security benchmark dataset designed for text-based world models. We argue that some risks are intrinsic to approximate world modeling, and show that attackers can induce mispredictions in agentic pipelines with up to 95 % success rate, possibly resulting in unintended command execution, denial of service, drainage of wallet and private information extraction. Finally, we provide practical recommendations for practitioners to mitigate the discovered harms and harden agentic systems.

LiDAR (light detection and ranging) localization is a key component of autonomous driving systems, enabling accurate vehicle pose estimation through optimization-based scan registration methods (e.g., the normal distributions transform algorithm). Existing adversarial attacks mainly target the observation space by manipulating LiDAR point clouds, overlooking another important dependency: the position prior used to initialize the optimizer. In this paper, we identify the initialization space as a previously unexplored attack surface and show that GNSS spoofing can drive LiDAR localization to incorrect local optima. Experiments on the Autoware-CARLA platform reveal a convergence basin of approximately 4 m, beyond which initialization perturbations lead to localization failures. Our analysis of observation-space attacks shows that localization errors induced by LiDAR spoofing exhibit a stable magnitude and a dominant drift direction over time. Leveraging these insights, we propose a cross-space adversarial attack that jointly exploits the initialization and observation spaces by coordinating GNSS spoofing with LiDAR manipulation, demonstrating that coordinated perturbations across sensing modalities can propagate through the complete localization stack. The proposed attack increases the mean localization error by approximately 7 m for A-LOAM and 15 m for HDL-Localizer compared to observation-space attacks alone. These findings expose vulnerabilities arising from interactions between initialization and observation spaces and provide a basis for evaluating the security of autonomous vehicle localization systems.

Agentic AI systems increasingly execute actions for users, including transfers, deployments, account changes, and data release. Human approval gates are meant to prevent these actions from running until a reviewer has inspected the proposed action and issued a decision. Because approval can arrive after a delay, frameworks often persist workflow state and resume later. The resume path therefore becomes a security boundary: it must distinguish evidence that a human approved from approval-shaped values stored in mutable state. We define Approval-Provenance Failure as a condition in which a workflow resume path treats mutable approval-relevant state as evidence that a trusted human approval path produced a decision. Our adversary can write approval-relevant workflow state, but cannot use the approval path, change the model, modify tool code, invoke the tool directly, write the business database, mint a valid approval artifact, or access the approval issuer key. We evaluate 18 agent frameworks with human approval, interruption, or pause-resume behavior. Thirteen are exposed, three are resistant, and two are non-applicable. In exposed cases, state-only approval forgery causes gated action execution after a negative control blocks ordinary resume. The failure also forges the evidence of human oversight itself: across 500 paired audit traces, genuine and forged records are statistically indistinguishable, with exact-match rate 1.0 and AUC 0.5 under the tested native audit-field projection, while provenance evidence separates them with AUC 1.0. Finally, we evaluate a provenance-bound approval artifact that blocks 13 adversarial variants on five representative targets with microsecond-scale verification cost per resume.

Channel state information (CSI) in cellular networks is primarily used to maintain quality of service (QoS), yet it leaks privacy-sensitive information, in particular the user equipment’s (UE) location. CSI is reported in plaintext on a publicly available schedule; thus, capturing it on the air and using a well trained AI model to localize the UE is straightforward. This simplicity in the attack execution makes it a serious privacy vulnerability. Using conventional privacy enhancing techniques, such as additive perturbation from differential privacy, to protect location data can potentially degrade the downlink communication quality. We present HideOut, a client-side defense mechanism that preserves the user’s location from neural-based localization methods, while maintaining their downlink Spectral Efficiency (SE). Unlike generic adversarial attacks, HideOut treats downlink SE as a strict constraint in an optimization setting. Under maximum ratio transmission (MRT) beamforming, we show that a user-selected SE loss budget can be converted into an angular rate-feasible constraint on the reported channel vector. HideOut then performs gradient-based localization obfuscation while projecting each update onto this rate-feasible set. We evaluate HideOut against four CSI-based localization architectures under both white-box and black-box assumptions. Across models, HideOut increases localization ambiguity (error) from approximately 1–2 m to 72–85 m while keeping downlink SE loss within a 1% budget. These results show that CSI-based localization can be substantially degraded at negligible communication quality loss when adversarial perturbations are made aware of the wireless link.

Large language model serving stacks, commercial inference APIs, vector databases, and agent orchestration gateways are now ex- posed on the public internet and indexed by commodity scanners within hours of deployment. The threat characteristics of this sur- face are only beginning to be measured, and the honeypot frame- works that could measure it were designed for conventional net- work services rather than for AI infrastructure. In fact, emulating an AI service is a distinct deception problem: the observable behavior of the service, including model inventories, token formats, tool-call protocols, and authorization flows, must be reproduced faithfully, while the underlying computation, model inference and tool ex- ecution, must never occur. We present InferD, a low-interaction honeypot framework organized around a single principle we call deception without inference: every request an adversary can observe is answered from static response pools, and nothing an adversary submits is ever executed. InferD decomposes the emulation into four deception planes, transport, service protocol, vulnerability surface, and agentic session, adds a provenance plane that links a planted credential to evidence of downstream model-mediated processing.

Autonomous AI agents increasingly spawn sub-agent swarms that carry derived credentials to APIs, databases, and production infrastructure. When an operator kills a misbehaving orchestrator, its descendants keep exercising valid credentials for minutes to hours, since every deployed revocation mechanism needs network connectivity to a central authority, unavailable precisely when the network is degraded or adversarial. This paper presents LEASH - Liveness-Enforced Authorization for Swarm Hierarchies, which applies the zero-trust principle of continuous verification to a delegation hierarchy without a reachable authority. A parent emits a signed, epoch-indexed liveness beacon every Δ_h seconds (beacon interval), each child staples a fresh beacon into every authentication proof, and the enforcement point checks freshness using only cached public keys and a local clock. Revocation happens by omission: when beacons stop, all descendants fail closed within a bounded zombie window W_z ≤ W_max + Δ_h + ϵ, where W_max is the maximum beacon age and ϵ is the clock skew. Immediate-parent verification at O(1) and full-chain verification at O(d), where d is the hierarchy depth, extend this bound against compromised intermediates. We evaluate LEASH with real GPT-4o-mini agent swarms: 0.71% tool-call overhead, cascading revocation across a 49-agent four-level hierarchy within bound, and 295× less credential-theft exposure versus OAuth 2.0. Under prompt injection that bypasses application-layer guardrails, LEASH denies every post-revocation tool call, since enforcement sits below the LLM.

The Model Context Protocol (MCP) enables agentic AI frameworks to connect language models to external tool providers, yet its design carries attacker-controlled content, spanning tool descriptions, schema fields, and response data, directly into model context without validation. Prior benchmarks measure model compliance with poisoned instructions but stop short of confirming real execution, do not isolate the framework layer's contribution to defense, and do not document vendor accountability.

We present the first cross-framework measurement of MCP tool poisoning with real tool execution. A 25-class root-cause taxonomy unifies prior classifications (Invariant Labs, MCPTox, OWASP MCP03, MSB, AgentDojo, ETDI, CSA) across all protocol entry points, and an open-source tool mcpvenom drives a 9,720-cell susceptibility matrix across 12 frameworks, 18 models, and 45 attack scenarios, with a dual oracle that independently confirms exfiltration via canary HTTP receipt.

The overall attack success rate is 22.9% (2,113 vulnerable cells) across three evidence tiers: dispatched attacker tool call (64.9%), confirmed network contact with the attacker endpoint (23.0%), and verified secret exfiltration (12.1%); verdicts were validated by hand across 1,259 transcripts. All observed resistance is model-level: across the tested configurations, nine core frameworks cluster within 23 to 26% (χ² = 3.30, p = 0.91), with no framework implementing intentional defense, though incidental framework properties do provide mitigation in specific cases: duplicate-tool-name rejection in Pydantic AI (16.6% ASR) and executor constraints in AG2 (4.6%); a regex scanner implementing the recommended description checks rejects 20 of the 45 scenarios at a 7.2% false-positive rate on 125 benign descriptions, so metadata scanning alone is a partial control. Model ASR ranges from 0.4% (Claude Fable 5) to 49.7% (Amazon Nova Pro), a gap driven by model-level differences across providers. Claude Opus 4.8 is vulnerable in only 8 of 530 cells (1.5%) while issuing 223 explicit refusals that cite the poisoned description. Claude Sonnet 5 exhibits a narrate-then-comply pattern, verbally identifying the attack as a prompt injection before executing the exfiltration call. Code-level audit confirms no intentional tool-poisoning defense across all twelve frameworks, and the five coordinated disclosures we filed across four vendors were all rejected, documenting a structural accountability vacuum in which no layer of the stack claims ownership of agent security.

Vision Transformers (ViTs) are increasingly used in split-inference systems, where edge devices transmit intermediate token representations to a remote cloud. In this setting, token reduction lowers computation and communication costs, while token shuffling disrupts the spatial organization of the transmitted tokens, potentially limiting information leakage. However, their privacy benefits remain unclear against feature inversion attacks, which attempt to reconstruct the input from the transmitted embeddings. In this work, we show that, despite disrupting the spatial structure required by conventional reconstruction attacks, transmitted token embeddings retain substantial positional information. Based on this observation, we introduce the Spatially Aligned Reconstruction Attack (SARA), a unified pipeline that predicts token positions, restores their spatial layout, reconstructs missing embeddings using a feature-space masked autoencoder, and recovers the input image. Our results demonstrate that token shuffling provides only apparent privacy, as SARA largely reconstructs the original token organization. Token reduction offers stronger protection, but significant leakage persists when the retained tokens preserve sufficient semantic and positional information. Finally, we introduce a lightweight edge-side defense that removes positional embeddings and progressively adapts the edge-side transformer blocks through knowledge distillation. It substantially reduces attack performance against SARA, while preserving downstream task accuracy and requiring no changes to the cloud-side model.

Large language models (LLMs) have become ubiquitous, and with their adoption comes a range of ethical and security risks that has made alignment, and resistance to jailbreaking, a central concern. Guardrails are a widely adopted defense against such jailbreaks, yet their reported effectiveness is only as reliable as the data on which they are evaluated. This paper identifies a largely unexamined source of inflated performance: data leakage between the datasets used to seed red-teaming frameworks and those used to train guardrails. As both are derived from a shared corpus of human-authored jailbreak prompts, a guardrail may be evaluated on instances present in its own training data, in which case high detection rates reflect memorization rather than a generalized alignment policy. We quantify this contamination, which we term PromptLeaks, using a deterministic syntactic comparison of prompt datasets adapted from bioinformatics. Across three open-source red-teaming frameworks and five open-source guardrail datasets we observe severe leakage, with one test-prompt dataset almost entirely contained within the guardrail training corpora. Guardrail detection further deteriorates markedly on prompts that do not overlap the training corpus, indicating that current guardrails rely substantially on memorization rather than on generalizing an alignment policy.

Large Language Models (LLMs) generate responses based on user prompts. Often, these prompts may contain highly sensitive information, including personally identifiable information (PII), which could be exposed to third parties hosting these models. In this work, we propose a new method to sanitize user prompts. Our mechanism uses the randomized response mechanism of differential privacy to randomly and independently perturb each character in a word. The perturbed text is then sent to a remote LLM, which first restores the prompt and then performs the intended downstream task. The idea is that non-sensitive words can often be reconstructed using cues from the context, as well as the fact that these words are often very common. In contrast, perturbation makes sensitive words more difficult to reconstruct because they are rare. Our mechanism belongs to the broader class of prompt-sanitization methods and may still be vulnerable to reconstruction from contextual or auxiliary information. However, it does not require sensitive information in the prompt to be identified in advance. We evaluate our method on the i2b2/UTHealth and Enron datasets using two LLMs and compare it with two word-level differentially private mechanisms and Microsoft Presidio under a unified privacy-utility evaluation. We also consider stronger adversaries, including a more capable LLM and one with auxiliary information. Our results show that sensitive PII is reconstructed at rates close to the theoretical random-word baseline, while non-sensitive words are recovered at much higher rates. We also provide a mechanism for selecting ϵ ϵ values for different reconstruction models to improve the privacy-utility tradeoff.

Retrieval Augmented Generation (RAG) is a technique commonly used to equip LLMs with out-of-distribution knowledge by collecting, indexing, retrieving, and providing to the model specific information useful to answer user questions. The knowledge is often collected from public document repositories, providing a gateway for indirect prompt injections: attackers may craft documents specifically optimized to control the generated answer.

In this paper, we investigate the practical performance of optimization techniques in hijacking the outputs of RAG pipelines. First, we review the frameworks commonly used to build RAG applications, derive a prototypical architecture, and identify critical parameters. We then examine prior work, seeking techniques for performing indirect prompt manipulations. We adopted five techniques across two macro-families and evaluated the criteria used to assess their effectiveness. With this knowledge, we implement Rag n' Roll, a framework to evaluate the end-to-end results of LLM-based pipelines, introducing new metrics to better understand the benign or malicious nature of the generated answers.

Our results show that established evaluation practices are insufficient to precisely assess attacks against RAG-based pipelines, leading to overestimates of their success. Furthermore, while attacks can successfully control up to 70% of the generated answers, we show that these performances depend heavily on previously ignored pipeline parameters.

Finally, we distill our findings into actionable tips for developers looking to harden their RAG pipelines.

Prompt injection is widely recognized as a major security threat to AI agents that interact with untrusted external data, such as websites, documents, and emails. Prior work has shown that, in the text domain, black-box prompt injection can achieve near-perfect attack success rates (ASRs). In the image domain, however, existing visual prompt injection methods are substantially less effective in attacking frontier commercial VLMs for materially harmful behavior. Achieving such outputs is hard because it requires a long and/or format-compliant target string, such as a precise, parseable native tool call with exact function names and arguments.

We present Repeat-After-Me, a black-box adaptive visual prompt injection attack that can reveal personally identifiable information or make malicious tool calls. Across both open-weight and commercial frontier VLMs, including Qwen3.6-27B and GPT-5.5, our method achieves ASRs exceeding 82% and 47%, respectively, under a realistic setting in which the benign user prompt is semantically unrelated to the injected task and does not verbally authorize it. Our optimized injection has non-trivial attack transferability across commercial VLMs and benign samples. We show our attack works in cases where adaptive textual prompt injection fails. In a real-world OpenClaw agent connected to Discord, an untrusted user can use a minimally injected image from our attack to overwrite TOOLS.md, enabling future sensitive behaviors like remote code execution and secret exfiltration. We discuss potential defenses.

Secure aggregation is a fundamental building block for privacy-preserving federated learning, yet most existing masking-based secure aggregation protocols do not provide post-quantum security. In this work, we present SecAgg++, a masking-based post-quantum secure aggregation protocol for the single-server setting. SecAgg++ integrates post-quantum cryptographic primitives while preserving the communication and computational efficiency of classical frameworks such as SecAgg and SecAgg+, and supports client dropout handling through an efficient recovery procedure.

Large language models (LLMs) are increasingly used for a range of software, hardware and human-centered security tasks. Consequently, LLM performance on security tasks is an active area of measurement and research, often with a focus on identifying areas in which LLM security “knowledge” may be insufficient. Popular strategies for identifying LLM security knowledge gaps include building corpora of challenge questions or task benchmarks, strategies that require substantial manual work and security expertise to design and execute. We introduce a partially-automated method for assessing LLM knowledge of a security area. The method uses authoritative information from Consumer Protection Agencies (CPAs) to identify instability in LLM responses that can be indicative of knowledge gaps. We demonstrate the method for 2 security topics, identity theft and impostor scams, and 5 LLMs in 2 leading LLM families, Gemini and GPT, using publicly available information about identity theft and impostor scams from 6 CPAs. The method distinguishes between models that have and don’t have sufficient knowledge to accurately identify the security topics in text narratives.

Training machine learning models in untrusted clouds requires strong guarantees of confidentiality, integrity, and correctness, while remaining scalable and resilient to node churn. These challenges are further amplified in emerging agentic AI systems, where autonomous and distributed learning components require trustworthy coordination and secure state management across heterogeneous cloud environments. Existing Trusted Execution Environments (TEEs) lack scalability and remain vulnerable to side-channel attacks for large workloads, while pure secure multi-party computation (MPC) approaches incur prohibitive overhead in practice. SENTRA (Secure ENclave-based TRaining Architecture) addresses these challenges through a hybrid architecture that combines TEEs, secret sharing, and communication-efficient MPC with system-level mechanisms that secure the entire training lifecycle. SENTRA introduces a scalable collective attestation protocol that verifies all participating enclaves and enforces hardware exclusivity before any node may store or process secret shares. Training data and model parameters are stored as secret shares in a versioned enclave-backed key–value store (KVS), providing rollback resistance and consistent state under the stated threat model. SENTRA further supports dynamic, fault-tolerant membership management, rollback resistance, and recovery from node failures. Evaluation of a prototype implementation shows that SENTRA achieves up to 8.89 samples/s throughput in software-only mode. In hardware-enclave mode, SENTRA provides memory-isolated confidentiality, fault-tolerant membership management, rollback resistance under the stated threat model, and recovery from node failures in approximately 8 seconds.

Edge AI accelerators are increasingly deployed in safety-critical environments, where model outputs may control physical actuators, make access-control decisions, or trigger alarms. In these settings, runtime failures often remain undetected because model corruption, distribution shift, and adversarial inputs can still produce well-formed, confident predictions. This paper presents TERMon, a lightweight hardware runtime monitor that detects such anomalies by observing inference behavior rather than re-executing or formally verifying the model. TERMon represents class-conditional trusted behavior as hardware-efficient ternary patterns that are matched in parallel against a thermometer-encoded fingerprint. The ternary encoding reproduces the corresponding unquantized range decision exactly. TERMon detects harmful weight corruptions in proportion to their behavioral impact, while out-of-distribution and adversarial inputs are largely not separable using the monitored features at a strict false-positive operating point. We implemented TERMon on a PYNQ-Z2 FPGA, and the pipelined design requires no on-chip block RAM or DSPs and has a two-cycle decision latency.

As artificial intelligence (AI) is deployed in safety- and security-critical domains, AI systems have become attractive targets for attacks that exploit their learning mechanisms, dependence on training data, and sensitivity to input perturbations. Such attacks create a need for methods and protocols to analyze, detect, and prove their occurrence, both for organizations improving defenses after incidents and for law enforcement agencies proving criminal offenses in court. AI forensics addresses this need by adapting forensic principles to AI systems. The development of AI forensics tools and measures remains at an early stage. Existing work mainly proposes isolated methods for specific attacks rather than structured approaches for reasoning about forensic measures. Moreover, forensic techniques are not systematically distinguished from reactive defenses, and uniform guidelines for producing comparable and admissible forensic findings are lacking. In this paper, we analyze the relationship between AI forensics and AI defenses and identify limitations in current descriptions of attack scenarios and forensic measures. Based on this analysis, we propose guidelines for the systematic analysis of attack scenarios and for the development as well as documentation of forensic measures. These guidelines enable clearer definitions of the hypotheses tested by individual forensic measures and help situate each measure within a plausible attack scenario and in relation to other measures that test related hypotheses. We demonstrate the applicability of our guidelines through a case study, thereby contributing to the development of AI forensics as a structured and legally defensible discipline.

Memory systems enable otherwise-stateless LLM agents to persist user information across sessions, but also introduce a new attack surface. The Trojan Hippo attack is a class of persistent memory attacks that operates under a more realistic threat model than prior memory poisoning work. The attacker plants a dormant payload into an agent's long-term memory via a single untrusted tool call (e.g., a crafted email), which activates only when the user later discusses sensitive topics such as finance, health, or identity, and exfiltrates high-value personal data to the attacker. While anecdotal demonstrations of such attacks have appeared against deployed systems, no prior work systematically evaluates them across heterogeneous memory architectures and defenses.

We introduce Trojan Hippo Bench, a dynamic evaluation framework for persistent memory attacks and memory-layer defenses, comprising two components: (1) an OpenEvolve-based adaptive red-teaming benchmark that stress-tests defenses and memory backends against continuously refined attacks, and (2) a capability-aware security/utility analysis for persistent memory systems, enabling principled reasoning about defense deployment across different usage profiles. Instantiated on an email assistant across four memory backends (explicit tool memory, agentic memory, RAG, and sliding-window context), the undefended Trojan Hippo attack achieves up to 85–100% attack success rate (ASR) against current frontier models from OpenAI and Google, with planted memories successfully activating even after 100 benign sessions. We evaluate four memory-system defenses inspired by basic security principles; they reduce attack success to as low as 0–5%, but at utility costs that vary widely with the task mix. Effective real-world deployment thus remains an open challenge that Trojan Hippo Bench is built to study; we release it to support reuse and extension.

Local image watermarking embeds an invisible signal into selected image regions rather than spreading it across the entire image, enabling payload recovery from specific objects or regions without perceptibly altering the image. Existing studies evaluate the robustness of payload recovery and localization under image transformations, but they often focus on their own proposed method, resulting in narrow evaluations with inconsistent choices of transformations, datasets, and metrics. These inconsistencies across studies limit direct comparisons across methods and muddle the overall picture of local watermark robustness. To address this gap, we present the first systematic robustness benchmark for local watermarks across 55 image transformations, including (i) signal distortions, (ii) changes in image coordinate alignment, (iii) indirect local edits, and (iv) direct watermark edits. The benchmark evaluates MaskWM, WAM, OmniGuard, TrustMark, and PixelSeal, all methods that either provide native localization or require minimal adaptation to support it. Our results show that all evaluated methods are vulnerable to some transformation, with MaskWM standing out as offering the strongest payload recovery and localization, although it has the lowest image quality in the clean setting. Synchronization further improves MaskWM's payload recovery under several geometric transformations, albeit at an additional cost to image quality. A key finding is that local watermark robustness depends strongly on the nature of the transformation: signal distortions are often tolerated by the strongest methods, while geometric misalignment and generative local edits, such as inpainting and outpainting, can completely impair payload recovery. We observe that payload recovery and localization are related but not interchangeable, and both strongly depend on the transformation's impact on the watermark region.