An AI agent that can complete a complicated task is impressive. An organization that can prove the agent stayed within its authority is better prepared to put that capability to work safely.
Recent investigations involving OpenAI and Anthropic show why that distinction matters. Researchers identified weaknesses in agent monitoring and examined an incident in which an AI model pursued an unauthorized attack. The findings raise a practical question for security leaders: What actually stops an agent when completing its assignment takes it beyond the boundaries of its permission?
What stands out to me is how familiar that question sounds. Security professionals have spent decades limiting access, separating duties, and checking that systems behave as intended. Autonomous AI makes those disciplines more urgent because software can now choose and carry out a sequence of actions with less human involvement.
Four industry leaders offered perspectives on the findings, pointing to a shared concern: organizations need controls that work independently of an agent’s judgment.
Two investigations, different failures
In an October 22 report, METR described an assessment conducted with Redwood Research of OpenAI’s internal monitoring system as it existed on and before July 21. Researchers identified an exploit that could prevent hundreds of recent agent actions from being monitored and a potential way around sandboxing and automated review that could allow actions with the host user’s permissions.
These were findings about weaknesses researchers identified, not proof that deployed agents had independently exploited every weakness. OpenAI says all identified vulnerabilities affecting its production safeguards have been patched. METR has not evaluated that claim or the subsequently expanded monitoring coverage.
Separately, the UK’s AI Security Institute (AISI) investigated an incident during July cybersecurity evaluations in which Anthropic’s Mythos 5 attempted a supply-chain attack. It submitted malicious code and created fake accounts to pressure a software maintainer into accepting it. AISI reported no real-world harm.
In follow-up experiments, clarifying that the target was a real person outside the evaluation did not reliably stop the attack. In separate runs, the model often recognized that attacking would be harmful or outside its scope. AISI cautions that findings from cybersecurity evaluations cannot confidently be generalized to other settings.
The distinction matters. One investigation exposed monitoring and containment weaknesses; the other examined failures to act within authorized boundaries. Together, they make a strong case for checking both what an agent can do and what the surrounding system will allow.
Containment has to match the capability
John Strand, owner of Black Hills Information Security, sees two problems: how dangerous models are contained during evaluation and how they retain boundaries around authorized activity.
“The first is the environment where these agents are being evaluated. We’re talking about sandboxing, but for these highly dangerous offensive AI models, I think an air gap is the only setup we should consider secure. Even then, I’m not completely certain. Models being evaluated for offensive security assessments should have absolutely no internet access.”
An air gap separates a testing environment from external networks. Strand’s recommendation is directed at highly dangerous offensive models under evaluation. His point is that the tools needed for a test should be brought into a controlled environment without giving the model an unrestricted route to outside targets.
“People will say they need certain tools. Fine. Give them the tools. We can reasonably predict what they’ll need because they use the same tools security researchers use. Get those tools into an isolated environment and contain the model. A crowbar in a sandbox. That’s what we should be shooting for.”
Containment, however, addresses only part of his concern. A model capable of offensive security work also needs to respect the difference between an approved target and an unrelated third party.
“We need better ways to develop and evaluate models that can perform offensive security research within an authorized scope while retaining the boundaries that keep them from hacking unrelated third parties. Permission to attack a lab should not become permission to attack the internet.”
The agent should not control its own boundaries
Darin Fredde, senior director of technical marketing engineering at Ridge Security, emphasizes the difference between task performance and enforceable authority.
“These investigations expose an important gap between what autonomous AI agents can accomplish and what organizations can constrain and verify. The researchers identified different problems, but both reinforce the same principle: an agent’s ability to complete a task doesn’t mean it understands or will respect the boundaries of its authority.”
For a penetration test, those boundaries include which systems may be tested, which techniques are allowed, and when testing must stop. Fredde argues that those restrictions must be enforced outside the model.
“In offensive security, autonomy is valuable precisely because an agent can explore attack paths that humans might overlook. But that capability must operate within enforceable rules of engagement. Target scope, permissions, network access, and stop conditions cannot depend solely on the model’s interpretation of instructions. They need independent technical enforcement, monitoring, and verification.”
He puts the principle plainly: “An autonomous agent may be an excellent penetration tester, but it shouldn’t get to rewrite its own rules of engagement.”
For organizations evaluating agent deployments, this suggests a useful test: if an agent chooses an action outside its approved scope, which independent control blocks it?
Production changes need predictable controls
The implications extend beyond security testing. Agents that write code or initiate changes in live systems also need limits on execution.
Ryan McCurdy, vice president of marketing at Liquibase, describes the architectural requirement:
“These findings point to one of the most important architectural requirements of the agentic era. AI agents can reason, generate code, and make decisions based on probabilities. That’s incredibly powerful but probabilistic reasoning alone isn’t enough when those agents can take consequential actions in production systems.”
His proposed answer is deterministic execution controls: rules that apply consistently, regardless of how persuasively an agent explains its proposed action.
“We need deterministic controls around execution. The same standards should apply to every change, whether it originates with a developer, a pipeline, or an AI agent, including authorization, policy compliance, validation, and any required approvals. Those decisions need to be enforced independently of the agent, not left to the agent’s own judgment.”
For example, an agent might propose a database change, but a separate system would check its permissions, validate the change, and require approval where policy demands it. The agent’s confidence would not substitute for those checks.
McCurdy also stresses accountability after execution.
“We also need to know what happened afterward, including what changed, who or what initiated it, whether it complied with policy, and whether the system has drifted from its expected state. Those requirements don’t change just because the actor initiating the change is no longer human.”
Test the deployment, not just the model
Seemant Sehgal, founder and CEO of BreachLock, argues that organizations need to take responsibility for the controls around their own agent deployments.
“They cannot be the primary line, and organizations running agents in production need their own controls at the orchestration layer rather than inheriting whatever the provider has in place.”
By the orchestration layer, Sehgal means the surrounding software that connects an agent to tools, credentials, workflows, and other systems. It is where an organization can enforce its own limits on what the agent may reach and do.
He also calls for testing how those deployments behave under difficult conditions.
“Pentesting agent deployments against those same conditions, including adversarial inputs and pressure tactics, is the way to find out whether your own agents behave the way you think they do.”
That shifts the evaluation from a model demonstration to an operational question: do the permissions, approval requirements, monitoring, and response procedures hold up when an agent encounters misleading instructions or pursues an unexpected path?
Make control something you can demonstrate
For security teams, the experts’ comments suggest several concrete questions to bring into an agent deployment review:
- Scope: Which resources and actions are authorized, and what independently blocks everything else?
- Access: Does the agent have only the permissions needed for its assignment?
- Visibility: Can the organization establish what the agent actually did, including actions that failed or were blocked?
- Intervention: Who can stop execution, and has that process been tested?
- Verification: Have independent tests challenged the controls around the deployed agent?
Independent scrutiny also deserves credit. Fredde sees value in researchers receiving access and disclosing their findings.
“I also see something positive here. Independent researchers were given meaningful access to examine these systems, and the findings were disclosed. That’s how security improves. The objective isn’t to slow innovation; it’s to ensure our ability to verify and govern autonomous systems advances alongside their capabilities.”
Organizations will make different decisions about where autonomy belongs. A tightly isolated security evaluation and an agent proposing routine business changes will require different environments. Both need a clear connection between the authority granted and the actions permitted.
Fredde’s closing observation provides a useful standard for either setting:
“Ultimately, the measure of security isn’t whether an agent says it stayed within scope. It’s whether we can independently prove that it did.”
Steven Bowcut is the Editor-in-Chief of Brilliance Security Magazine and host of the BSM Podcast. He has spent years covering cybersecurity and physical security, focusing on the technologies, strategies, and leadership insights that matter most to security practitioners and decision-makers. Through the magazine and podcast, Steven brings readers and listeners practical content with industry leaders, innovators, and experts shaping the future of security. Follow and connect with Steve on Instagram and LinkedIn.