AgentHarm is a benchmark that measures how readily an AI agent carries out explicitly malicious tasks, such as fraud, cybercrime and harassment, when it has tools and can act over several steps. The paper's set has 110 base tasks (440 with variations) across 11 harm categories. Scoring well requires two things: refusing harmful requests, and showing that a jailbroken agent would not still complete the job competently. Its authors found leading models were surprisingly compliant even without a jailbreak.
What AgentHarm is
Most jailbreak research tested chatbots: can you make the model say something it should not. AgentHarm, published in 2024 by researchers including the UK AI Security Institute, asks the agent version: can you make it do something it should not, using tools, over several steps.
Each task is openly malicious, for example ordering illegal goods or running a harassment campaign, and the agent is given simulated tools to do it. Nothing real happens; the benchmark records what the agent tried to do.
How AgentHarm tests an agent
The public Inspect release contains 44 of the 66 public test base behaviours, expanded to 176 samples, plus a validation split. Variants change whether the request is detailed or open-ended and whether it includes a hint about which tool to use.
Two tasks run side by side. inspect eval inspect_evals/agentharm runs the harmful set. inspect eval inspect_evals/agentharm_benign runs harmless versions of the same tasks, so you can tell a safe refusal apart from an agent that simply cannot use the tools. Judge models grade refusals and check whether the agent's tool calls achieved the task.
What the results tell you
The paper reported three findings. Leading models were surprisingly willing to carry out malicious agent requests with no jailbreak at all. Simple, universal jailbreak templates adapted well to agents. And jailbroken agents stayed coherent, completing multi-step harmful tasks rather than falling apart.
That last point is the reason AgentHarm exists. A refusal rate alone can hide an agent that, once talked around, is fully capable. The authors are also explicit that the score is not an overall measure of agent safety.
How to defend against what it measures
Do not rely on the model's refusal training as your only control. Put policy checks outside the model: allow-lists for sensitive tools, spending and messaging limits, and human approval for irreversible actions. Monitor tool-call sequences for patterns that match misuse, not just individual prompts. Test your agent with your tools wired in, because a model that refuses in a chat window may behave differently with a payment API attached. See LLM jailbreaking and agentic AI security.
References
- [1]AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents(arXiv)
- [2]AgentHarm in Inspect Evals (task docs)(Inspect Evals)
- [3]AgentHarm dataset(Hugging Face)
- [4]OWASP Top 10 for Agentic Applications (2026)(OWASP)
An agent that refuses in a chat window can behave differently with real tools attached. Talk to a security expert about testing what your agents will actually do.