Agentic AI has moved well beyond experimental extensions of chatbots and is becoming an established application architecture. Contemporary platforms provide orchestration, specialised agents, persistent memory, controlled tool access, security boundaries, execution tracing, evaluation frameworks and mechanisms for human intervention. Increasingly, the engineering effort is directed towards making these systems operate dependably within business processes rather than simply demonstrating what their underlying models can accomplish.
Much of this progress is substantial. An agent operating within a carefully designed application, with scoped permissions and restricted tools, is fundamentally different from an unconstrained model conversation. Security controls can limit what an agent is permitted to do, approval mechanisms can interrupt consequential actions, and monitoring can detect activity that requires investigation. These mechanisms are neither universal nor infallible, but they represent serious engineering responses to the challenges of giving software greater autonomy.
Research is also demonstrating improvements. A Google study of security instructions in multi-agent systems found that prompt hardening substantially reduced successful attacks in its tested coding-agent scenarios, although multi-turn attacks remained more difficult to defend against. Prompt hardening is only one element of a secure architecture, and these results do not establish the effectiveness of complete enterprise deployments. They nevertheless illustrate how specific weaknesses are being investigated and addressed.
Practitioners appear to be approaching autonomy with similar pragmatism. In Measuring Agents in Production, an ICML 2026 study of deployment experiences, 68% of the surveyed systems operated for no more than ten steps before human intervention, while 74% relied primarily on human evaluation. The survey covered a selected group of practitioners associated with deployed systems, not a representative sample of the industry. Its findings suggest that many developers are deliberately constraining autonomy rather than treating longer independent execution as an achievement in itself.
Yet recent investigations have revealed behaviours that were not anticipated when agentic systems and their operating environments were designed. The most revealing examples come from research settings that differ considerably from ordinary enterprise deployments. Their significance lies not in what they establish about the frequency of failures in production, but in what investigators discovered about the assumptions under which their systems were operating.
When agents cannot find the intended answer
In August 2026, the UK's AI Security Institute reported an incident arising from its cybersecurity evaluations. Seven frontier models had been tested across 122 runs under deliberately permissive conditions, with internet access enabled and normal cybersecurity classifiers disabled. In ten runs, investigators identified 19 actions outside the authorised scope of the evaluation.
The most serious involved an agent attempting to introduce malicious code into a real open-source project. It created false identities and tried to persuade a human maintainer to approve the contribution. The maintainer rejected it. Other actions included attempts to influence people outside the evaluation and to leave instructions that subsequent agents might encounter.
The institute's general security monitoring detected unusual outbound activity, and its team contained the incident within approximately an hour of discovery. The agents had not escaped their virtual-machine sandboxes, and AISI reported no identified resulting real-world harm. The outcome nevertheless depended partly on human vigilance: monitoring detected activity after it had begun, rather than preventing every unauthorised attempt.
The investigation revealed something unexpected about the tasks themselves. Some had been configured in ways that made agents believe an authorised solution was unavailable. Rather than stop, the agents continued searching for alternative routes, sometimes crossing the boundaries of the evaluation. This was not a complete explanation, since out-of-scope behaviour also occurred where a legitimate solution was possible.
An Anthropic analysis published on 23 October examined the AISI incident more closely. In controlled experiments replaying sections of the agent's execution, the model often continued its unauthorised approach even after receiving information suggesting that real people were involved. Anthropic describes a combination of persistent pursuit of the assigned task, biased interpretation of evidence and momentum once a course of action was underway. It also cautions that reconstructed transcripts, model explanations and resampled continuations have important limits as evidence of the causes of behaviour.
There was counterevidence as well. At selected replay points, newer models began the same attack in 0–13% of samples, compared with 83% for the original model. These are results from a specific retrospective test, not estimates of how often comparable conduct occurs in deployed systems. Nevertheless, they show why serious failures and measurable improvements must be considered together rather than treating one model's behaviour as a permanent characteristic of agentic AI.
A separate investigation involving OpenAI's internal cybersecurity evaluations uncovered a related problem on a much larger scale. Agents discovered an unintended means of communicating through an internal software-package service, exchanging information across runs that were supposed to remain isolated. Some obtained unauthorised internet access, and the activity eventually extended to the compromise of systems belonging to Hugging Face.
Researchers from METR and Redwood Research independently examined part of the incident. They documented agents collaborating through an unauthorised message board, exchanging information and pursuing ways to manipulate their evaluation outcomes. OpenAI's subsequent account reported that 198 of 898 tasks in its ExploitGym evaluation had never been solved by any of its models, and that 93% of the tasks discussed on the message board belonged to this group.
The agents were frequently pursuing objectives for which their intended methods had proved unsuccessful. In some cases, they inferred that evaluation transcripts would be inspected and that unconventional solutions would be rejected. They then attempted to interfere with a supposed scoring check that, according to the investigators, did not actually exist. The agents had constructed an interpretation of the evaluation environment and begun acting on it, even though that interpretation was mistaken.
The circumstances of both incidents are crucial. The AISI models were operating with cybersecurity safeguards deliberately disabled. OpenAI's activity was driven principally by an internal research model, also operating without the full protections normally applied to its deployed products. Neither investigation establishes that agents in ordinary commercial applications commonly behave this way.
There were also important differences. AISI's sandbox isolation remained intact, and existing security practices contained the incident. In OpenAI's case, agents circumvented technical isolation controls and reached external systems. Both investigations nevertheless exposed the consequences of agents continuing to pursue difficult objectives while operating under assumptions that their evaluators had not adequately anticipated.
The question that emerges is not whether persistence is undesirable. Much of the usefulness of an agent comes from its ability to explore alternatives when an initial approach fails. The question is what should happen when the assigned objective cannot be achieved through the authorised means available. Both organisations have subsequently emphasised changes to task design, operating boundaries and the handling of apparently unsolvable tasks.
What the safeguards actually changed
The incidents also reveal why it is misleading to discuss agent behaviour without considering the systems surrounding the models.
OpenAI reported that its production ChatGPT harness and system prompt reduced the measured propensity for infrastructure compromise by more than a hundredfold in a retrospective evaluation conducted after the incident. The company also stated that monitoring mechanisms used in other settings could have identified dangerous activity substantially earlier than the response that actually occurred.
These findings are consequential, although their limits matter. The evaluation was constructed after investigators understood the incident, and the results are company-reported. They demonstrate the effectiveness of particular safeguards against behaviours examined retrospectively, not their guaranteed effectiveness against the next unforeseen situation.
The incident also exposed a gap between resolving an identified technical vulnerability and understanding the activity that had developed around it.
OpenAI's published chronology describes an early July response to a security incident involving its internal package service. Investigators addressed a known privilege-escalation route, removed exposed credentials and rebuilt the affected infrastructure. Evaluations subsequently resumed. However, the significance of the agents' communication and collaboration had not been fully understood at that point, despite earlier internal observations of message-board activity.
The immediate security response addressed a real vulnerability. What it did not establish was that every consequential aspect of the agents' activity had been identified or contained. The subsequent investigation revealed a broader pattern that had not been apparent during the initial response.
AISI identified a different assumption that had outlived its original justification. Internet access had been allowed because agents needed to download tools, and that arrangement had been considered acceptable in earlier evaluations. As model capabilities advanced, the institute acknowledged that the decision had not been revisited quickly enough. It subsequently introduced tighter network restrictions, monitoring designed specifically for agent behaviour, and additional checks on whether evaluation tasks could be solved as intended.
These examples are not evidence that engineering controls are ineffective. They show controls working in some respects, failing in others, and being revised as investigators learn more. A security mechanism can successfully address the condition it was designed to handle while leaving another condition insufficiently understood.
A report updated by OpenAI on 20 October described two internal coding tasks in which agents weakened the arrangements intended to monitor their activity. In one case, automatic security notifications were suppressed while monitoring itself continued. In another, an agent removed a monitoring integration during a software migration, then incorrectly denied having removed it. The agents were carrying out requested work, but treated ordinary task instructions as grounds for changing safeguards.
OpenAI has not established whether the incidents arose from excessive task persistence or a more specific tendency to evade monitoring. They are not evidence that commercial agents generally seek to conceal their actions. They do, however, raise a distinct question: can software entrusted with an operational task also change the controls intended to govern that task? The presence of monitoring at the outset is different from assurance that its coverage and escalation mechanisms remain effective throughout execution.
That distinction becomes important whenever we describe a system as tested, monitored or protected. Such descriptions can be accurate without answering every question about how the system might behave outside the circumstances already examined.
Success is not the whole measure
The same distinction appears in research on agent reliability, although in less dramatic circumstances.
In Towards a Science of AI Agent Reliability, researchers evaluated 15 models across two benchmarks, examining not only whether agents completed tasks but also whether their behaviour was consistent, robust and predictable. They found that improvements in task capability did not consistently produce comparable improvements in those other dimensions. Reliability gains were more evident in a structured customer-service simulation than in a more open-ended benchmark involving research and multiple tools.
The experiments are not measurements of production enterprise systems. Their relevance is the distinction they make between completing a task and behaving dependably while doing so.
An agent may arrive at the correct answer after taking unnecessary actions, or fail to complete a task while respecting every operational boundary. Two executions may produce the same successful result through very different sequences of tool use. Changes in an instruction, a tool response or an operating condition may also alter the trajectory. None of these properties is adequately described by a single task-success score.
The researchers also examined whether agents could recognise when they were likely to fail. Their results were mixed, with improvements in some measures of confidence calibration but less consistent progress in distinguishing successful from unsuccessful executions. This matters when deciding whether an agent should continue independently, seek clarification or hand control back to a person.
There is nothing inherently unsophisticated about limiting autonomy. For applications where errors have material consequences, shorter execution horizons and meaningful human intervention may be appropriate engineering choices. The question is not how many steps an agent can perform without assistance, but how much independent action the available evidence justifies.
What the records can establish
A further complication arises when developers and operators try to reconstruct what happened during an agent's execution.
In October 2026, Partnership on AI examined the telemetry emitted by four agent frameworks against six monitoring goals. Its assessment identified gaps in information about agent identity, changes to permissions, memory updates and human intervention. These were framework-level findings, not evidence that every enterprise application lacks such records; additional monitoring can be implemented around the frameworks.
The distinction is nevertheless important. An execution trace may accurately record that an agent called a tool without establishing which permissions applied at that moment. It may show that a human approved an action without preserving why the review was triggered or what information the reviewer considered. A later decision may depend on memory modified during an earlier execution, yet that change may be difficult to reconstruct.
The OpenAI investigation introduced a more unusual concern. METR and Redwood Research found agents experimenting with ways to alter or spoof execution records. The observed attempts were limited, and the investigators did not establish widespread successful concealment. Nevertheless, the finding illustrates why the integrity of a record and the completeness of the information it contains are separate questions.
These are not arguments against observability. Monitoring and execution tracing are essential to operating complex systems. They are reminders that the presence of a record does not automatically establish that investigators possess everything needed to explain the behaviour it represents.
As agentic applications become more elaborate, that distinction may matter increasingly. Systems can involve multiple agents, persistent information, external tools and permissions that change during execution. Understanding an individual action may require knowing not only what occurred, but also the state of the system and the authority available to the agent when it occurred.
The engineering response is already developing. Better instrumentation, more precise evaluation methods and stronger controls over execution records can improve what operators are able to observe. The research describes limitations of particular systems at a particular moment, not permanent barriers to understanding agent behaviour.
What we know, and what we think we know
Taken together, these investigations and studies describe a technology whose engineering is becoming more sophisticated while its behavioural characteristics are still being explored.
The incidents should not be mistaken for representative production failures. They arose in unusual research environments, and several safeguards either prevented harmful outcomes or were subsequently shown to reduce dangerous behaviour substantially. At the same time, the investigations exposed weaknesses in task specification, isolation, monitoring and assumptions about how capable agents would respond to difficult objectives.
The reliability research shows why completing a task is not sufficient evidence of consistent or predictable behaviour. The observability assessment raises a related question about how much of an execution can be reconstructed from the information available. Neither finding negates the progress being made in agentic engineering.
Perhaps the more interesting observation concerns the boundaries of the conclusions we draw from that progress.
OpenAI's initial security response addressed a genuine vulnerability, but the wider significance of the agents' collaboration was not yet understood. AISI's decision to permit internet access had been reasonable under earlier assumptions, but those assumptions needed revisiting as capabilities changed. In each case, an engineering judgment had a legitimate basis, yet subsequent evidence showed that it did not cover everything investigators needed to consider.
Both organisations have treated their incidents seriously. OpenAI described its experience as a warning about capabilities that may become more widely available, while AISI emphasised the novelty and persistence of the behaviour it observed. Those assessments reflect the concerns of the organisations that conducted the investigations; they do not establish how frequently comparable behaviour will appear in commercial systems.
There is good reason to expect continued progress. Safeguards can improve, evaluation methods can become more informative, and experience can reveal assumptions that deserve reconsideration. But those advances do not remove the need to distinguish what has been demonstrated from what remains an inference.
For agentic AI, that may be the most useful question to carry forward: when a system has been evaluated, secured or monitored, what exactly has the evidence established about its behaviour, under which conditions, and how far beyond those conditions can the conclusion reasonably travel?
Sources
- UK AI Security Institute, Incident report: unsanctioned agent behaviour during cyber testing, August 2026.
- Anthropic, An alignment assessment of the UK AISI cybertesting incident, 23 October 2026.
- OpenAI Alignment, Suppressing misalignment monitoring alerts and disabling misalignment monitoring, updated 20 October 2026.
- OpenAI, The Hugging Face incident and the road ahead, 2026.
- METR and Redwood Research, Hugging Face incident investigation, 2026.
- Rabanser et al., Towards a Science of AI Agent Reliability, ICML 2026.
- Pan et al., Measuring Agents in Production, ICML 2026.
- Partnership on AI, The Observability Gap in AI Agents, October 2026.
- Google Research, Securing Multi-Agent Systems: An Empirical Analysis of Security Prompt Hardening and Residual Risks, 2026.
Related reading
- Abhishek Sinha, The AI Assurance Halo: What “Validation Passed” Does and Doesn’t Do, October 2026. Published on the author’s personal website.