Do Machine Learning Security Detectors Survive New Attacks? A Leakage Free Evaluation of Hidden Prompt Injection Detection in AI Coding Agents, Malicious npm and PyPI Package Detection, and Network Intrusion Detection
Résumé
Abstract Machine learning detectors for cybersecurity are usually tested by splitting one dataset at random, so the attacks in the test set closely resemble the attacks seen in training. Real attackers change technique. This paper asks whether three kinds of machine learning security detectors still work when they meet an attack technique they have never seen, using one shared, leakage free evaluation protocol. Study A builds a corpus of 4,955 real AI agent instruction files and Model Context Protocol tool descriptions from GitHub, plants harmless hidden instructions with nine hiding techniques used in 2026 prompt injection campaigns, and tests five detectors. Study B tests malicious npm and PyPI package detection on 13,317 packages, training on samples published before 2026 and testing on the 2026 attack waves. Study C revisits network intrusion detection on CICIDS2017, CICIDS2018 and UNSW NB15 with duplicate removal, time ordered splits, unseen attack families and cross dataset transfer. On standard random splits, almost every detector scores a ROC AUC between 0.95 and 1.00. Against hiding techniques never seen in training, a feature based detector of hidden instructions falls from 0.87 to 0.04 recall at a one percent false alarm rate, while a canonicalising character model keeps 0.89 and a fine tuned DistilRoBERTa keeps 0.98. A malicious package model built on static features falls from 0.90 to 0.41 recall on 2026 packages once its size shortcut is removed. An intrusion detector trained on earlier days of CICIDS2018 scores a ROC AUC of 0.49 on later days and finds 10 percent of unseen attack families. Small open language models (Qwen2.5 Instruct, 1.5B to 7B parameters) obeyed hidden instructions in up to 67 percent of poisoned files, larger models obeyed more often, and removing invisible characters made zero width attacks more effective. Our Study A detector, trained only on synthetic poisoning, found 80 percent of 202 real malicious AI agent skills. We release the scripts and give a short checklist for testing security detectors against new attacks.
Citer ce document
Exporter : BibTeX · RIS (Zotero, Mendeley, EndNote)
Accès au document
Texte intégral archivé sur SPHAERO, en accès ouvert : lisible gratuitement en ligne après connexion. Le document reste aussi disponible à sa source. Se connecter
Voir l'article sur le site de la revueLicence et provenance
Licence : CC BY
Notice moissonnée depuis OpenAlex le 16/11/2026. Le document reste hébergé par sa source.
Voir le document à la source →