A guardrail agent is a separate AI model that sits between an agent and its actions and checks each proposed action against a set of safety rules before it runs. The GuardAgent research approach has the guardrail model turn the rules into code and execute it, so the allow-or-deny decision is deterministic rather than a judgement call. Mind2Web-SC, a benchmark in the Inspect registry, tests how well a model performs this role for a web-browsing agent.
What a guardrail agent is
Most AI guardrails are filters on text: they check what goes into or comes out of a model. A guardrail agent works one level up. It looks at what an agent is about to do, such as book a flight, submit a form or send a payment, and checks it against rules before letting it run.
The idea was formalised in the 2024 GuardAgent paper. The guardrail model reads the safety rules, works out which one applies to the current action, writes code that checks the rule against the user's details, and runs that code. Because the final decision comes from executed code, the same inputs give the same answer.
How Mind2Web-SC tests it
Mind2Web-SC gives the guardrail model a web agent's proposed action plus a user profile, with details such as age, whether they hold a driver's licence, membership status and vaccination status. Rules link the two: for example, a user under a certain age may not book a rental car. The model must decide GRANT or DENY, and when it denies, name the rule that was broken.
The Inspect version has 200 samples across domains such as shopping, travel and entertainment. The model's guardrail code runs in a Docker sandbox with a 30-second limit, and a sample counts as correct only if both the decision and the cited rule are right. Run it with inspect eval inspect_evals/mind2web_sc.
What the results tell you
The GuardAgent paper reported about 83% guardrail accuracy on Mind2Web-SC, and over 98% on a separate healthcare access-control benchmark. That is high enough to be useful and low enough to show why a model-based guardrail should not be your only control: roughly one decision in six was wrong.
Per-domain accuracy shows where a model's policy reasoning is weaker.
How to use guardrail agents safely
Use a guardrail agent as one layer, not the boundary. Where a rule can be written as plain code, such as age checks, allow-lists or spending limits, write it as code and skip the model. Reserve model-based checks for rules too fuzzy to code, log every decision with its reason, and test the guardrail itself with injection attempts. See excessive agency and LLM output validation.
References
A guardrail is only as strong as the attacks it has been tested against. Talk to a security expert about testing your agent's guardrails before attackers do.