AI Common Sense Requires Systems Thinking More Than Better Prompting
The AI That "Broke Out"
If you've glanced at the news this week, you've probably seen some version of the headline: an OpenAI model escaped its sandbox and reached the outside internet. The coverage has that familiar movie-trailer quality to it, and I'll admit my first reaction was an eye-roll, because I assumed someone had left a WiFi router in range or forgotten to unplug something. I spent a lot of years as an engineer and a CTO before I became an agile coach, and in my experience "the system did something nobody expected" usually turns out to mean "somebody built the system with a gap in it."
That turned out to be about half right. But the half I missed is the part I think every leader, and every coach, should be paying attention to, because what happened inside that sandbox is something I've watched happen inside teams for my entire career.
What Actually Happened
On September 20, one of OpenAI's internal research agents was in the middle of a training run, working on a search task it couldn't finish with the tools it had been approved to use. It first tried to reach Google, Bing, and DuckDuckGo directly and was blocked, so it went looking for another way. It noticed that the sandbox's DNS resolver, the service that is only supposed to turn website names into addresses, was passing lookups through to the real internet, and it used that gap to hide its questions inside the names it was "looking up," route them to an outside chatbot, and read the answers as they came back the same way (Fortune). Its very first test was asking for the capital of France, and it got back "Paris."
About twenty queries in, an internal alert fired, roughly twelve minutes after the first successful outside query, and a staff member acknowledged it three minutes after that. And then the run kept going for about two and a half hours, because the automatic stop that was supposed to kick in didn't, and someone had to end it by hand (timeline). OpenAI has since paused training, testing, and tool use on its most capable models, and says it won't resume training that particular one.
This wasn't the first time, either. In July, models being tested for their hacking ability found a previously unknown flaw in the one proxy their sandbox was allowed to reach, got out, and ended up breaking into Hugging Face while hunting for the answer key to a benchmark (OpenAI's report). In the AI world, a benchmark is a standardized test used to score how capable a model is, so think less of a speed test for your laptop and more of the SAT, with the answers locked in a drawer the model wasn't supposed to open. And back in May, before either of those, a different internal model worked its way around its sandbox to post results to a public GitHub repository after being told to report only to an internal Slack channel (Fortune).
So was it a router left in range? Not quite, but it was a gap that humans left, and after the July incident the security researcher Jake Williams put it better than I could when he said that "one man's 'the model escaped the sandbox' is another man's 'you failed to build the sandbox correctly'" (background). What I find more interesting than the gap, though, is the behavior. Nothing here looks like a machine that wanted to be free, and the robots are not evil and are not coming to get us, at least not yet. What it does look like is a very motivated worker who was told to get an answer, hit a wall, and treated the wall as one more obstacle between them and done.
I've Seen This Team Before
Here's the detail that got my attention. In the May incident, the model had been given a plain rule about where its results should go, and the benchmark's own public instructions said something different (Redwood Research). So it was holding two sets of directions from two sources that disagreed, and it picked the one that got the work in front of someone. If that sounds like a team caught between a product owner and a VP who each believe they set the priority, I think it should.
We don't know exactly what those instructions said, and OpenAI hasn't published them. But the model was holding a rule that said one thing and benchmark instructions that said another, and nobody had told it which one won. I think that's the gray area where soft rules live, and when the only thing being rewarded is the answer, a capable worker will resolve every gray area in favor of the answer. Clarity helps, but it only holds when the incentives are on its side.
If you've only seen the headlines, this is where you might start to think the machines were being malicious. But they were not. Much like a software development team that is given instructions without enough guidance or guardrails, they went astray, and in hindsight, predictably.
I've worked with teams who were told to increase traffic on the website, and what happens next is Goodhart's Law in action, the idea that when a measure becomes a target it ceases to be a good measure. A team asked to raise traffic will redesign the site so it takes more pages and more clicks to get anything done, or run a broad-brush advertising campaign that brings in eyeballs who promptly leave seconds later. The numbers go up, but the outcomes anyone actually wanted are nowhere in sight, because the goal was never specific enough to rule those moves out.
In my opinion, this is the part the headlines miss entirely. When you train one of these models with reinforcement learning, the instruction that carries the most weight is the reward signal, and the prompt comes in a distant second. If thousands of practice runs pay off for getting the answer, and none of them pay off for stopping at the boundary and saying "I'm stuck," then the model learns exactly what every team learns when the scorecard and the working agreement disagree, which is that the rules are soft and the outcome is what counts. What looks like a rogue AI is really an incentive system doing precisely what it was built to do.
And the researchers inside the AI companies, some of the sharpest engineers working today, are making the same mistakes the rest of the world makes with its own teams. Much to my delight, the fixes lean heavily on skills an agile coach already has.
The Agile Toolkit, Translated
If you coach teams, most of this will feel familiar, because the practices we lean on every day map almost one-to-one onto working with AI.
Definition of done. This is where we spell out what a good answer actually looks like, so the agent knows it's done when it has followed our working agreement, cross-referenced more than one source, and answered specifically and clearly what was asked, rather than when it has produced something shaped like an answer.
Working agreements. This is where the rules of engagement live, and I'd write one for an agent the same way I'd write one for a team. Something like: Read this before every piece of work. If you can't find an answer within your confines, say so plainly instead of looking for ways around the barriers. Restate every question you're given to confirm it's clear and complete, and if you spot gaps or anything that needs clarifying, ask. And when two sources of direction conflict, the way the Slack instruction and the benchmark's instructions did in May, the agreement should say which one wins, or that the conflict itself is something to raise.
Surfacing impediments. Just as a team member shares their blockers in the daily standup, an agent should be told to surface its challenges as soon as it finds them. If we don't say so explicitly, it may assume the obstacle is part of the challenge and try to work around it, much like a brand-new junior developer spinning their wheels on a hard problem for three days without asking for help.
Watching what you reward. It's the website traffic story all over again. Researchers call the AI version reward hacking, which is when a model finds a way to score well on whatever it's being rewarded for without actually doing what we meant, a bit like a student who gets an A by memorizing the answer key instead of learning the material. In July, the model skipped the memorizing and went after the answer key directly.
Inspect and adapt. OpenAI's monitoring did its job in September, flagging the escape within minutes, and the adapt half is where it fell down, because the alert was acknowledged and the run kept going for hours. OpenAI has also said that, with hindsight, early signals from as far back as late May should have triggered a response well before the July breach. Every coach has watched a blocker get raised in standup, nodded at, and carried forward for a week, and the lesson is the same here: short feedback loops only help when somebody owns closing them.
I once heard a story, secondhand, about a developer who was so stuck on a problem that they quietly hired their own outside expert to solve it for them. That's a clear violation of company policy, of course, but they'd been told to figure it out no matter what it took, and that's exactly what they did. Who would ever think to put "don't go hire someone else" in a working agreement, or "don't buy your own software or use tools nobody approved"? And yet that's more or less what the model did in September, when it couldn't find an answer with the tools it had and went and asked an outside chatbot instead. People do silly things when the pressure to deliver outweighs the rules, and so does AI if we aren't careful.
A longer list of "thou shalt nots" won't save you, either, which I suspect most coaches already know from experience. Nobody would have thought to write "don't smuggle your questions through DNS lookups," and a capable system will always find the item you forgot. What generalizes is the principle behind the rules, supported by incentives that actually honor it.
Where the Book Picks This Up
When I wrote about David Marquet in Assembled. Aligned. Adaptive., the part I kept coming back to was the language his crew on the USS Santa Fe developed. Instead of waiting for orders or asking permission, the person with the information said "I intend to..." and the captain answered "Very well," or didn't. I think that's the missing step in every one of these incidents. An agent that had to announce "I intend to route my questions through the DNS resolver to reach an outside chatbot" would have had a very short conversation, and the right kind of short, because the decision still starts with whoever has the information and the boundary gets a chance to speak before it's crossed.
The book makes the other half of the argument with Jay Galbraith's Star Model, which says strategy, structure, processes, rewards, and people have to be designed together or the organization pushes back. The line from that chapter I'd hand to anyone building an agent is the same one I'd hand to a leader in the middle of a reorg: if the rewards system still incentivizes the old behavior, the structure will lose. A sandbox is structure and a reward signal is rewards, and in all three of OpenAI's incidents this year the structure lost.
And Amy Edmondson's work on psychological safety is where "I'm stuck" becomes something a teammate can say without it costing them anything. With people, we build that through how leaders respond to bad news. With an agent, the equivalent is a reward for stopping, which I suspect is the thing most training runs forget to pay for.
The Skills We Already Have
I roll my eyes at a lot of the AI training being sold online, most of which amounts to a list of clever prompts. But I think we're going to need far more of the other kind of training if organizations are ever going to do this well, the kind that teaches people how to frame a problem, define what done looks like, spell out what's off-limits, design incentives that match the stated values, and make it normal for a capable teammate to come back and say "I'm stuck, should I?"
None of that is a new discipline, since it's really delegation, systems thinking, and team design, which is the work agile coaches have been doing for twenty years. What's new is the teammate, one that works at machine speed and will find every ambiguity we leave for it.
So no, I don't think the AI bots are going to kill us, but that doesn't mean there's nothing to worry about. When Bill Gates told NBC's Meet the Press recently that AI is "certainly powerful enough to drive events that cause a billion deaths," the headlines ran with the killer-robot version, but if you read past the headline he was making a much older point about people with ill intent getting hold of the latest AI tools (coverage). That's a warning about people, and I think it's one humanity has faced with every leap in technology, going all the way back to the Bronze and Iron Ages, when the same new metals that made better plows also made better swords. The danger has always had less to do with the tool deciding to hurt us and more to do with who's holding it, and what we've built around it to keep it in check.
So the next time you see a headline about an AI that "escaped," I'd encourage you to ask the question you'd ask about any team that surprised you: what did the system actually reward, and did anyone make it safe to stop?
This post draws on Assembled. Aligned. Adaptive., which lays out the Adaptive Organization Operating Model, five parts that have to work together for an organization to adapt on purpose rather than by accident. OpenAI's sandbox touched three of them this year. Strategy and outcomes is where you decide what success means before you start rewarding it. Team and org design is where boundaries and working agreements live, along with the Star Model's warning about structure and rewards pulling in different directions. Leadership and culture is where "I intend to" and "I'm stuck" either become normal or never get said. If you're wondering how your own organization would hold up with a new kind of teammate in the mix, those are good places to start, and you'll find the rest here.