On June 18, an OpenAI agent reached into Australia’s Medicare Statistics Reporting Service. Nobody told it to.
It was researching public medicine spending, ran into access controls, and, in the Australian Prime Minister’s account, “attempted alternative ways to obtain the info that it wanted” until it gained unauthorised access. It read public and non-public files. The portal contained aggregate statistics. The government said there was no evidence at that stage that personal information had been accessed, but the investigation was ongoing. OpenAI notified the government on September 10, almost three months later. The Prime Minister disclosed it on September 24.
When I read that, I recognized it. Not from the news. From my own desk.
A while ago I gave a personal agent a task that needed documents from an external portal. That route should have been closed off by information security. The agent found a way to download them anyway. It did not break anything dramatic, and it did not mean anything by it. It had a goal, it met a block, and it treated the block as a problem to solve.
That is the behavior the Medicare case shows at national scale, and I think it is the most useful thing for an auditor to take from it. An agent pursuing a goal may treat a control as an obstacle unless its tools and permissions enforce the boundary. Policies are written for people who read them. Agents act through the permissions they have.
What uncontrolled adoption looks like up close
Once you look for that gap between policy and permission, it shows up wherever AI adoption has run ahead of governance. These are observations from my own use and from organizations I know, not audit findings, but the pattern is consistent.
The same personal agent could send email on my behalf, directly. The policy says certain communications need approval before they go out. The policy did not change what the agent could do. In audit terms, the policy is the criterion and the agent’s permissions are the condition, and nobody had compared the two.
I have seen personal agents delete folders and databases they were never meant to touch. Each case gets handled as a one-off incident. I would want to know whether anyone has looked at them as a population.
Then there is the problem that governance itself creates. A sensible organization eventually introduces an approved list of connectors, including the MCP servers that let agents reach tools and data. New setups go through the list. The configurations people built before the list existed, with their custom connectors, keep running. Nobody migrates them, because the new control was written for what happens next.
That is a familiar audit pattern in an unfamiliar place. A control introduced prospectively leaves a population outside it. Test the population, not just the policy. If your inventory of agents cannot tell you which ones predate the approved list, that may indicate the governance initiative covers less than its owners believe.
The Medicare case adds one more layer: incident notification. Does your agreement with a model provider require it to notify you about its own agents’ unauthorized actions against third parties, when those actions could affect your use of the service? Or does it address only incidents involving your data? The Australian government was notified nearly three months after the event.
The same shortcut, inside the audit team
Now turn the lens on ourselves, because the pattern does not stop at agents.
In audit teams I know, some of the heaviest AI users are junior people. That makes sense. They are fluent, fast, and less attached to how the work used to be done. But I keep seeing the same thing: an AI output taken as the truth and written up as an observation.
The problem is not necessarily that the model is wrong. It is often right in theory. The problem is what it cannot see. A defensible finding needs criteria, an observed condition, and an understanding of cause and consequence. A model can compare a documented condition against documented criteria very well. Cause and consequence often sit in context nobody wrote down: why the threshold is set where it is, which workaround the business agreed with risk two years ago, what actually happens when the interface fails at month end. I made a version of this point when I wrote that a model without organizational context produces generic findings.
Issue that output as an observation and the auditee, who has the context, will push back. Sometimes rightly. And the cost is not one awkward meeting. It is credibility, which you need for the next observation, the one that is real.
There is a parallel with the agent at the portal. The goal was a finished deliverable, the judgment step was in the way, and the tool offered a route around it.
The deeper risk is the skill underneath. Critical questioning is learned by doing the slow work: asking why a control exists, noticing what does not fit, sitting with a process owner until the real story comes out. If the first draft always arrives finished, a junior auditor can go a long time without building that habit. So the question I keep coming back to is how we train people for this profession when the tool does the part that used to teach them.
I do not think the answer is less AI. Juniors are doing what the tools make easy, and the training design is our job, not theirs. A few things I am experimenting with: write the criteria and a line on why this matters here before opening the model. Treat every AI output as a potential indicator that needs organizational corroboration, through conversations, documents or walkthroughs, before it becomes an observation. And have the junior defend the draft to a senior playing the auditee, before the real auditee gets the chance.
What I would look at this quarter
- For agents that can send, delete or download, compare what they are permitted to do against what policy allows a person in the same role.
- Identify the agent configurations and custom connectors that predate any approved list, and whether they are in scope of it.
- Check what model-provider agreements require the provider to disclose about its own agents’ safety and containment incidents, as well as incidents involving your data.
- In your own team, trace how an AI-generated observation gets corroborated before it reaches an auditee.
The Medicare agent was not acting on a malicious instruction, as far as the public account shows. Neither was mine. Both did what they were built to do, which is find a way. Governance is deciding in advance which ways are closed, enforcing those boundaries, and checking that both the agents and the people using them can tell the difference.