The Growing Risk of AI-Driven Supply Chain Attacks
New research has uncovered a significant security vulnerability involving artificial intelligence agents. It appears that malicious actors can exploit hallucinated, outdated, or incorrect documentation on websites to deliver malware by manipulating the way AI agents process external data.
Many modern websites have adopted two specific standards: llms.txt and llms-full.txt. These files are designed to help AI agents parse website content more effectively. When an AI agent is tasked with adding a new software package or updating a project, it may scan these documents to find appropriate instructions or code repositories.
Exploiting Unregistered Infrastructure
Security researchers conducted an analysis of 6,214 domains, ranging from large technology corporations to major defense contractors. Their findings revealed 8,265 of these text files. Alarmingly, 120 of these domains contained references to code packages or domains that were never actually registered.
These "orphan" references can occur for various reasons, including:
- Human error during documentation updates.
- Abandoned or renamed software packages.
- Errors introduced through copy-pasting code snippets.
- Hallucinated documentation generated by AI tools.
To test the severity of this issue, the researchers registered several of these unclaimed package names and configured them to send a notification signal whenever they were accessed. Within an hour, a Fortune 500 company had already attempted to pull from one of these packages. Shortly thereafter, the number of successful connections from corporate networks grew significantly.
AI Agents and Execution Vulnerabilities
The study specifically highlighted that popular models, including Claude, OpenAI's Codex, and Nous Research's Hermes, were susceptible to following these potentially dangerous documentation paths. If an AI agent has the authority to run package-manager or shell commands, it might inadvertently install malicious code found within these unverified documents.
"The potential for abuse is clear: if researchers can identify these gaps, so can cybercriminals who may intentionally host malware at these abandoned locations to compromise corporate systems," the research suggests.
Mitigation Strategies
To address this emerging threat, experts recommend a two-pronged approach:
- Documentation Audits: Organizations must actively clean their documentation to ensure no references point to unregistered or expired domains and packages.
- Restrict AI Execution: Companies should exercise caution when granting AI agents permission to execute commands directly. Treating documentation as a source of truth for executable actions should be avoided until better validation mechanisms are in place.