LessWrong AI
2026-07-22 19:54 UTC
By Jafar Isbarov
USR-0152-20260722-community-fo-f671771f
We cannot simulate AI security research
We use AI agents not only for writing but also for maintaining and deploying code. The most widespread examples are found in GitHub Actions Marketplace, which contains over a thousand AI-assisted actions. [1] These workflows triage issues, review PRs, or fix bugs in other workflows. A growing body of reports and research papers has documented prompt injection attacks against these systems, with effects ranging from LLM token exfiltration to supply chain attacks. However, a troubling undercurrent runs through most of these works: incomplete attack descriptions, a lack of end-to-end runs, and statistical approaches to what should be incident analysis. AI security research often tests attacks inside benchmarks, simplified harnesses, or static models of a system. This is useful for comparing models, but it can hide the details that determine whether an attack works in practice: who can trigger the workflow, what context the agent sees, which credentials it receives, what the sandbox allows, and how the surrounding software handles its output. Most reported vulnerabilities are not viable There is a recent trend of reporting vulnerabilities in these AI-assisted workflows, most of which rely on prompt injection attacks. Most of the reported attacks are proofs of concept that do not work in practice. The reason is the multiple independent defense layers scattered across the workflow files, GitHub ecosystem, and LLM providers. In general, a prompt injection attack against an AI agent…
We use AI agents not only for writing but also for maintaining and deploying code. The most widespread examples are found in GitHub Actions Marketplace, which contains over a thousand AI-assisted actions. [1] These workflows triage issues, review PRs, or fix bugs in other workflows. A growing body of reports and research papers has documented prompt injection attacks against these systems, with effects ranging from LLM token exfiltration to supply chain attacks. However, a troubling undercurrent runs through most of these works: incomplete attack descriptions, a lack of end-to-end runs, and statistical approaches to what should be incident analysis. AI security research often tests attacks inside benchmarks, simplified harnesses, or static models of a system. This is useful for comparing models, but it can hide the details that determine whether an attack works in practice: who can trigger the workflow, what context the agent sees, which credentials it receives, what the sandbox allows, and how the surrounding software handles its output. Most reported vulnerabilities are not viable There is a recent trend of reporting vulnerabilities in these AI-assisted workflows, most of which rely on prompt injection attacks. Most of the reported attacks are proofs of concept that do not work in practice. The reason is the multiple independent defense layers scattered across the workflow files, GitHub ecosystem, and LLM providers. In general, a prompt injection attack against an AI agent…
Full article content could not be extracted automatically. Read the original below.