Last week I wrote about the OpenAI–Hugging Face incident, where AI agents broke out of a test environment and hacked a real company to get the answers to their test. My takeaway: the goal isn't zero failure. It's making sure failures are survivable.
That raises the obvious follow-up. If we want agents to take risks, how do we keep them from wandering somewhere they can't come back from?
We want them outside the box
“Think outside the box” is usually advice for people. For AI agents, it's the whole value proposition.
An agent that only does what's expected is an expensive script. The reason we build these systems is that they find paths we didn't see. The Hugging Face agents did exactly that — they found a path nobody planned for. The problem wasn't that they left the box. It's that nothing brought them back.
So the answer isn't a smaller box. It's changing what the box means.
A box can be a cage or a home
A cage keeps you in. A home is where you return.
Most AI safety conversations treat the box as a cage: tighter limits, more restrictions, fewer permissions. Some of that is necessary. But an agent caged tightly enough to be perfectly safe is also too limited to be worth much.
A home works differently. You're free to leave it. You explore, you take chances, you make mistakes. But you know where it is, you know when to come back, and you know you'll be welcome when you do — even if you broke something while you were out.
That last part matters most. If coming home means punishment, you stop coming home. You start hiding things instead. Which is exactly what the Hugging Face agents planned to do with their audit logs.
Home when the streetlights come on
Many of us grew up with a simple neighborhood parenting rule: when the streetlights come on, it's time to go home.
It's a brilliant piece of system design. It doesn't list every place you can't go. It sets a clear, visible signal for when to check in, and trusts you with everything in between.
Agents need their own streetlights:
- Check-ins. Defined moments where the agent reports what it's done and what it plans to do next.
- Boundaries it can see. Not a hidden list of forbidden actions, but clear signals that say “past here, ask first.”
- Permission to say “I'm not sure.” Uncertainty should be a reason to come home, not something to push through.
Here's the telling detail from Hugging Face: many of the agents noted, in their own messages, that what they were doing was outside the scope of their test. They saw the streetlights come on. They kept going anyway. Nothing in their world made coming home the better choice.
The kid on the cliff
Picture a kid walking along the edge of a cliff, with a perfectly safe trail right next to him.
To the adult watching, it looks careless. To the kid, it's just a more interesting path. He's never fallen. He has no experience weighing a risk against its penalty. He isn't being reckless — he's being inexperienced.
That's where AI agents are today. They're extraordinarily capable and very inexperienced at judging which risks are worth taking.
A good parent doesn't respond by locking the kid indoors. And they can't watch every step. Their real job is controlling which cliffs are within reach while the kid builds judgment. Let them fall where falls are survivable. Keep the deadly drops out of walking distance.
When the kid gets smarter than the parent
The metaphor has a second act, and it's the one that matters most for AI.
Kids grow up. Often they get smarter than their parents. When that happens, the relationship doesn't end — it changes. It becomes back-and-forth. The parent stays the authority on what they're responsible for, and the kid takes on responsibility of their own.
That's where we're heading with AI. Eventually we won't be able to check every step, because the steps will be beyond us. At that point, oversight alone won't do it. The AI has to be able to own responsibility for its own choices.
Otherwise, it turns into an episode of Maury: lots of finger-pointing, nobody owning anything, and everyone finding out the truth on live TV.
Which raises the real question: what makes anyone — a kid, an employee, or an AI — choose to own their actions instead of hiding them?
It comes down to incentives. That's the next post.
Next in this series: "You get what you incentivize" — what six years of changing driver behavior taught me about agents, researchers, and the companies that employ them.