On September 17 Anthropic published numbers showing how it monitors AI agents inside its own organization. Around 30,000 agents were doing research and engineering work, those agents made more than a billion decisions in August, and a real-time monitor blocked about 1 in 47,000 of them.
That is useful disclosure. It also gives internal auditors a concrete example of what using AI under control can look like. Anthropic does not rely on people noticing bad outcomes after the event. It routes agent actions through automated monitoring, stops some actions before they run, and sends selected cases to people for review.
This is the direction I expect many organizations to take. As AI moves from answering questions to taking actions, human review of every step stops being practical. Automated monitoring becomes part of the control environment.
The auditor’s job is to determine whether that monitoring deserves reliance. I would start with three questions.
What is actually covered?
Anthropic reports that every action on the platform it measured passed through its real-time monitor. That is 100 percent coverage for the stated population.
The words “stated population” matter. One in 47,000 means one in 47,000 decisions the monitor saw. It does not necessarily mean one in 47,000 AI decisions across the whole organization.
When I encounter AI in an audit, I want to understand how an agent enters the monitored population. Is registration automatic when someone deploys an agent, or does an engineer have to add it? Does the monitoring cover experiments, third-party tools, and agents running outside the main platform? Can an agent call another tool that sits beyond the monitoring boundary?
This is familiar audit work. Before relying on an exception report, we test whether the population feeding it is complete. AI does not change that principle. It makes the population harder to see because an agent can be created quickly, connected to several systems, and changed without appearing in the inventories we already use.
The first practical test is therefore not about the model. Reconcile the agents in the monitoring platform to other sources: cloud activity, service accounts, API traffic, software registers, and procurement data. The gaps tell you whether “100 percent monitored” means the whole organization or only the part someone remembered to register.
What could the monitor miss?
A block rate is an exception rate. It shows that the control fired. It does not show how many harmful actions passed through undetected.
Anthropic blocked 0.002 percent of the decisions it monitored, about 1 in 47,000. A low rate could mean the agents behaved well. It could also mean the monitor was too narrow. You cannot decide which explanation is right from the block rate alone.
This is where auditors can contribute something more useful than asking for another dashboard. Ask the control owner to demonstrate what the monitor is designed to detect. Then test it with known cases.
If an agent should not send sensitive data outside the company, give the monitor a safe test case that represents that behavior. If an agent has a spending limit, test an action above the limit. If it may read from a system but not change records, attempt a controlled write. The purpose is not to trick the AI. It is to establish that the surrounding control detects the behavior it claims to detect.
False negatives will never be measured perfectly because we do not have an independent list of every action that should have been blocked. We can still obtain evidence. Scenario testing, samples of allowed actions, incident reviews, and comparisons between model versions can show where the monitor is strong and where its detection logic needs work.
That is also how monitoring helps the organization use AI with more confidence. A failed test is not evidence that AI should be switched off. It tells the team which boundary needs improvement before the agent receives more authority.
What happens after an alert?
Anthropic’s offline monitoring flags roughly 100,000 transcripts a week and narrows those to about 50 for human review. That final step matters as much as the automated detection.
For an internal audit, I would ask who receives an alert, how quickly they must respond, what evidence they retain, and who can change or suspend the agent. I would sample alerts through to closure. A queue is not a control if nobody owns it, and a human review adds little if the reviewer cannot see the agent’s context or reverse its action.
This is where AI governance becomes operational. The organization needs named owners, escalation criteria, usable logs, and authority to intervene. Those controls do not have to be perfect before a team experiments with AI. They should become stronger as the agent gains access, autonomy, and potential impact.
I wrote recently that internal auditors are adopting AI faster than external auditors. I think that is an opportunity if we apply the same discipline to our own use that we expect from the business. We can use agents to search documents, prepare analyses, and coordinate workflows while also recording their actions, testing their boundaries, and reviewing the exceptions.
The same balance applies when we audit someone else’s AI. Start with what the agent does and why the business uses it. Then follow the actions into the monitoring controls around it. The aim is not to produce a generic list of AI risks. It is to decide whether this specific use can operate within boundaries the organization understands.
Only 7 percent of finance leaders in a recent survey said their organization puts governance ahead of speed, and 44 percent were only somewhat confident they could explain an agent’s action to an auditor. Better monitoring can narrow that gap, but only if we examine what sits behind the percentage on the dashboard.
So when someone tells you their AI is monitored, do not stop at the block rate. Ask what is covered, what the monitor could miss, and what happens when it raises an alert.
Those questions help us audit AI. They also help the organization use it well.