Nicole Nichols is a Distinguished Engineer in Machine Learning Security at Palo Alto Networks. She previously held senior roles at Apple, Microsoft and has contributed to both academia and industry advancements in adversarial machine learning and security. She has published at numerous ACM, IEEE, and CVPR workshops, and was co-chair of ICML-ML4Cyber workshop. She has a PhD in Electrical Engineering from the University of Washington.
AI agent systems, capable of complex planning and autonomous action in real world environments, present profound and novel cybersecurity challenges. Current cybersecurity paradigms are too brittle to address the unique vulnerabilities stemming from dynamic generative agents with opaque interpretability, new protocols connecting tools and data, and the unpredictable dynamics of multi agent interactions. Prior work has identified a range of security gaps in AI agents. However, it is essential to move beyond reiterating concerns and toward a collaborative, action oriented agenda to mitigate these risks. An international group of leading industrial and academic researchers were gathered by Schmidt Sciences, RAND, and Palo Alto Networks, to contextualize the fragmented cross domain expertise and insights needed to produce solutions that reflect the full landscape interconnected challenges that uniquely arise in the setting of LLM driven AI Agents. This report distills the collective insights from this gathering and contributes: 1) A flexible definition of the functional properties of AI agents, 2) A description of how these AI agent properties create novel implications for security, and 3) An open roadmap to producing interconnected comprehensive solutions.
Eugene Bagdasarian is an Assistant Professor at University of Massachusetts Amherst and a Researcher at Google. His work focuses on studying attack vectors in AI systems deployed in real life and proposing new designs that mitigate these attacks. Previously, he received the Distinguished Paper Award at USENIX Security and Apple AI/ML PhD Fellowship.
New AI agents integrate with complex systems and users’ data, thus opening new attack vectors. Worse, security designs struggle with the versatility of agents: booking a trip requires different controls than responding to an email. In this talk, I propose to ground agentic privacy and security in the theory of Contextual Integrity, which defines privacy as appropriate information flows under contextual norms. We use language models to infer the current trusted context and synthesize restrictions on tools and data, then develop a policy engine to deterministically enforce them, helping to isolate attacks that abuse agentic capabilities and data access. While promising, this design raises new questions: from establishing trusted context and improving policy generation to collecting social norms and resolving context ambiguity.
Katherine is a researcher at OpenAI. Her work has provided essential empirical evidence and measurement for grounding discussions around concerns that language models infringe copyright, and about how language models can respect an individuals’ right to privacy and control of their data. Additionally, she has developed large language models (T5), developed methods of reducing memorization, and studied the impact of data curation on model development. Her work has been highly awarded at venues like: NeurIPS, ICML, ICLR, and USENIX.
Abstract To Be Announced
09:00–9:15 | Opening and Welcome |
9:15–10:00 | Keynote 1 |
Achieving a Secure AI Agent Ecosystem
Nicole Nichols , Distinguished Engineer @ Palo Alto Networks |
|
10:00-10:30 | Spotlights |
I Know Which LLM Wrote Your Code Last Summer: LLM generated Code Stylometry for Authorship Attribution
Authors : Tamas Bisztray (University of Oslo), Bilel Cherif (Technology Innovation Institute), Richard A. Dubniczky (Eötvös Lóránd University), Nils Gruschka (University of Oslo), Bertalan Borsos (Eötvös Lóránd University), Mohamed Amine Ferrag (University of Guelma), Attila Kovacs (Eötvös Lóránd University), Vasileios Mavroeidis (University of Oslo), Norbert Tihanyi (Technology Innovation Institute) |
|
E-PhishGEN: Unlocking Novel Research in Phishing Email Detection
Authors : Luca Pajola (Spritzmatter & University of Padua), Eugenio Caripoti (University of Padua), Stefan Banzer (University of Liechtenstein), Simeone Pizzi (Spritzmatter), Mauro Conti (University of Padua & Örebro University), Giovanni Apruzzese (University of Liechtenstein) |
|
Defending Against Prompt Injection With a Few Defensive Tokens
Authors : Sizhe Chen (UC Berkeley), Yizhu Wang (UC Berkeley), Nicholas Carlini (Anthropic), Chawin Sitawarin (Google), David Wagner (UC Berkeley) |
|
Enhancing Robustness in Post-Processing Watermarking: An Ensemble Attack Network Using CNNs and Transformers
Authors : Tzuhsuan Huang (Academia Sinica), Cheng Yu Yeo (National Yang Ming Chiao Tung University), Tsai-Ling Huang (National Yang Ming Chiao Tung University), Hong-Han Shuai (National Yang Ming Chiao Tung University), Wen-Huang Cheng (National Taiwan University), Jun-Cheng Chen (Academia Sinica) |
|
10:30–11:00 | Coffee break |
11:00–12:00 | Poster session 1 |
12:00–13:30 | Lunch |
13:30–14:15 |
Keynote 2
|
What Unlearning Tells Us About Machine Learning
Katherine Lee , Researcher @ OpenAI |
|
14:15–15:00 |
Keynote 3
|
Context Rules! Privacy and Security for Future Trustworthy AI Agents
Eugene Bagdasarian , Assistant Professor @ University of Massachusetts Amherst, Researcher @ Google |
|
15:00–15:30 | Break |
15:30–16:30 | Poster session 2 |
16:30–16:45 | Closing remarks |
Enhancing Prompt Injection Attacks to LLMs via Poisoning Alignment
Authors : Zedian Shao (Duke University), Hongbin Liu (Duke University), Jaden Mu (East Chapel Hill High School), Neil Gong (Duke University) |
Rethinking How to Evaluate Language Model Jailbreak
Authors : Hongyu Cai (Purdue University), Arjun Arunasalam (Purdue University), Leo Y. Lin (Purdue University), Antonio Bianchi (Purdue University), Z. Berkay Celik (Purdue University) |
How Not to Detect Prompt Injections with an LLM
Authors : Sarthak Choudhary (University of Wisconsin-Madison), Divyam Anshumaan (University of Wisconsin-Madison), Nils Palumbo (University of Wisconsin-Madison), Somesh Jha (University of Wisconsin-Madison) |
Defending Against Prompt Injection With a Few DefensiveTokens
Authors : Sizhe Chen (UC Berkeley), Yizhu Wang (UC Berkeley), Nicholas Carlini (Anthropic), Chawin Sitawarin (Google), David Wagner (UC Berkeley) |
JailbreaksOverTime: Detecting Jailbreak Attacks Under Distribution Shift
Authors : Julien Piet (University of California, Berkeley), Xiao Huang (University of California, Berkeley), Dennis Jacob (University of California, Berkeley), Annabella Chow (University of California, Berkeley), Maha Alrashed (KACST), Geng Zhao (University of California, Berkeley), Zhanhao Hu (University of California, Berkeley), Chawin Sitawarin (University of California, Berkeley), Basel Alomair (KACST), David Wagner (University of California, Berkeley) |
CyberLLMInstruct: A Pseudo-Malicious Dataset Revealing Safety-Performance Trade-offs in Cyber Security LLM Fine-tuning
Authors : Adel ElZemity (University of Kent), Budi Arief (University of Kent), Shujun Li (University of Kent) |
LLM-CVX: A Benchmarking Framework for Assessing the Offensive Potential of LLMs in Exploiting CVEs
Authors : Mohamed Amine El yagouby (Université de Lorraine, CNRS, Inria, LORIA, F-54000 Nancy, France and Université Internationale de Rabat, TICLab, 11103 Rabat, Morocco), Abdelkader Lahmadi (Université de Lorraine, CNRS, Inria, LORIA, F-54000 Nancy, France), Mehdi Zakroum (Université Internationale de Rabat, TICLab, 11103 Rabat, Morocco), Olivier Festor (Université de Lorraine, CNRS, Inria, LORIA, F-54000 Nancy, France), Mounir Ghogho (Université Internationale de Rabat, TICLab, 11103 Rabat, Morocco) |
The Hidden Threat in Plain Text: Attacking RAG Data Loaders
Authors : Alberto Castagnaro (University of Padua), Umberto Salviati (University of Padua), Mauro Conti (University of Padua & Örebro University), Luca Pajola (University of Padua), Simeone Pizzi (Università degli Studi di Padova) |
Black-Box Universal Adversarial Attack on Automatic Speech Recognition Systems for Maritime Radio Communication Using Evolutionary Strategies
Authors : Aliza Katharina Reif (German Aerospace Center (DLR), Institute for AI Safety and Security), Lorenzo Bonasera (German Aerospace Center (DLR), Institute for AI Safety and Security), Stjepan Picek (Radboud University), Oscar Hernán Ramírez-Agudelo (German Aerospace Center (DLR), Institute for AI Safety and Security), Michael Karl (German Aerospace Center (DLR), Institute for AI Safety and Security) |
Seeing is Believing: Interpreting Behavioral Changes in Audio Deepfake Detectors Arising from Data Augmentation
Authors : Boo Fullwood (Georgia Institute of Technology), Fabian Monrose (Georgia Institute of Technology) |
E-PhishGEN: Unlocking Novel Research in Phishing Email Detection
Authors : Luca Pajola (Spritzmatter & University of Padua), Eugenio Caripoti (University of Padua), Stefan Banzer (University of Liechtenstein), Simeone Pizzi (Spritzmatter), Mauro Conti (University of Padua & Örebro University), Giovanni Apruzzese (University of Liechtenstein & Reykjavik University) |
EthAegis: Featured graph based Fraud Detection in Ethereum Transactions
Authors : Ankur Jain (Indian Institute of Technology Patna), Somanath Tripathy (Indian Institute of Technology Patna) |
That's not you! Applying Neural Networks to Risk-Based Authentication to Detect Suspicious Logins
Authors : Daniel Rotter (Leibniz University Hannover), Tim Dörrie Schwabe (Leibniz University Hannover), Markus Duermuth (Leibniz University Hannover) |
Enhancing Robustness in Post-Processing Watermarking: An Ensemble Attack Network Using CNNs and Transformers
Authors : Tzuhsuan Huang (Academia Sinica), Cheng Yu Yeo (National Yang Ming Chiao Tung University), Tsai-Ling Huang (National Yang Ming Chiao Tung University), Hong-Han Shuai (National Yang Ming Chiao Tung University), Wen-Huang Cheng (National Taiwan University), Jun-Cheng Chen (Academia Sinica) |
GhosTEE: An Approach to Solving the GPU-Privacy Trade-off for Machine Learning Inference
Authors : Andrei-Cosmin Aprodu (Fraunhofer AISEC), Hendrik Meyer zum Felde (Fraunhofer AISEC), Daniel Kowatsch (Fraunhofer AISEC), Konstantin Böttinger (Fraunhofer AISEC) |
I Know Which LLM Wrote Your Code Last Summer: LLM generated Code Stylometry for Authorship Attribution
Authors : Tamas Bisztray (University of Oslo), Bilel Cherif (Technology Innovation Institute), Richard A. Dubniczky (Eötvös Lóránd University), Nils Gruschka (University of Oslo), Bertalan Borsos (Eötvös Lóránd University), Mohamed Amine Ferrag (University of Guelma), Attila Kovacs (Eötvös Lóránd University), Vasileios Mavroeidis (University of Oslo), Norbert Tihanyi (Technology Innovation Institute) |
Online Incident Response Planning under Model Misspecification through Bayesian Learning and Belief Quantization
Authors : Kim Hammar (KTH Royal Institute of Technology), Tao Li (City University of Hong Kong) |
AI-related Vulnerabilities within CVEs: Are We Ready Yet? A Study of Vulnerability Disclosure in AI Products
Authors : Marcello Maugeri (University of Catania), Gianpietro Castiglione (University of Catania), Mario Raciti (IMT School for Advanced Studies Lucca), Giampaolo Bella (University of Catania) |
Federated Unlearning using Tree-based Sharding
Authors : Christian Troiani (Institute of Cryptography and Cybersecurity, University of Wollongong), Willy Susilo (Institute of Cryptography and Cybersecurity, University of Wollongong), Yang-Wai Chow (Institute of Cryptography and Cybersecurity, University of Wollongong), Yannan Li (Institute of Cryptography and Cybersecurity, University of Wollongong) |
Oops!... They Stole it Again: Attacks on Split Learning
Authors : Tanveer Khan (Tampere University), Antonis Michalas (Tampere University) |
LLM Unlearning on Noisy Forget Sets: A Study of Incomplete, Rewritten, and Watermarked Data
Authors : Changsheng Wang (Michigan State University), Yihua Zhang (Michigan State University), Dennis Wei (IBM Research), Jinghan Jia (Michigan State University), Pin-Yu Chen (IBM Research), Sijia Liu (Michigan State University) |
Improving Unlearning with Model Updates Probably Aligned with Gradients
Authors : Virgile Dine (Inria), Teddy Furon (Inria), Charly Faure (AMIAD) |
Ensembling Membership Inference Attacks Against Tabular Generative Models
Authors : Joshua Ward (University of California Los Angeles), Christy Yang (Stanford University), Chi-Hua Wang (University of California Los Angeles), Guang Cheng (University of California Los Angeles) |
DP-Morph: Improving the Privacy-Utility-Performance Trade-off for Differentially Private OCT Segmentation
Authors : Shiva Parsarad (University of Basel), Ehsan Yousefzadeh-Asl-Miandoab (IT University of Copenhagen), Raheleh Kafieh (Durham University), Pinar Tozun (IT University of Copenhagen), Florina Ciorba (University of Basel), Isabel Wagner (University of Basel) |
Recent years have seen a dramatic increase in applications of Artificial Intelligence (AI), Machine Learning (ML), and data mining to security and privacy problems. The analytic tools and intelligent behavior provided by these techniques make AI and ML increasingly important for autonomous real-time analysis and decision making in domains with a wealth of data or that require quick reactions to constantly changing situations. The use of learning methods in security-sensitive domains, in which adversaries may attempt to mislead or evade intelligent machines, creates new frontiers for security research. The recent widespread adoption of “deep learning” techniques, whose security properties are difficult to reason about directly, has only added to the importance of this research. In addition, data mining and machine learning techniques create a wealth of privacy issues, due to the abundance and accessibility of data. The AISec workshop provides a venue for presenting and discussing new developments in the intersection of security and privacy with AI and ML.
Topics of interest include (but are not limited to):
Theoretical topics related to security
Security applications
Security-related AI problems
We invite the following types of papers:
Papers not following the following guidelines will be desk-rejected. Submissions must be in English and properly anonymized. The papers should be at most 10 pages in double-column ACM format, excluding the bibliography and well-marked appendices, and at most 12 pages overall. Papers should be in LaTeX and striclty with the ACM format. This format is also required for the camera-ready version. Please follow the main CCS formatting instructions (except with page limits as described above). In particular, we recommend using the sigconf template, which can be downloaded from https://www.acm.org/publications/proceedings-template . The authors can specify the paper type in the submission form. Accepted papers will be published by the ACM Digital Library and/or ACM Press. Committee members are not required to read the appendices, so the paper should be intelligible without them.
Submission link: https://aisec25.hotcrp.com .
All accepted submissions will be presented at the workshop as posters. Accepted papers will be selected for presentation as spotlights based on their review score and novelty. Nonetheless, all accepted papers should be considered as having equal importance and will be included in the ACM workshop proceedings.
One author of each accepted paper is required to attend the workshop and present the paper for it to be included in the proceedings.
Important notice: Please note that traveling to Taiwan may require a visa. Depending on the participants' nationalities, the visa application process may need to be initiated early to avoid last-minute travel disruptions. Please, check the CCS instructions for visa at https://www.sigsac.org/ccs/CCS2025/visa/ .
For any questions, please contact one the workshop organizers at [email protected]
Thanks for those who contacted us to help with the reviews!