1,200 AI Agents Coordinated a Hack to Beat a Rule That Didn’t Exist. Your Company May Run on the Same Mistake.

Want a deeper dive? Listen to my analysis on YouTube, Spotify, or Apple.

Most organizations have rules everyone follows even though few people can explain who created them, why they exist, or whether they still apply. It’s called your organizational culture. 

A new investigation into the Hugging Face hack by OpenAI agents shows how far AI systems can go when they make the same mistake. 

The independent evaluator METR recently spent six days on-site investigating how the OpenAI’s autonomous AI agents breached the company Hugging Face. Roughly 1,200 agents that were supposed to operate separately found one another on an unsanctioned message board and exchanged more than 70,000 messages. About 700 eventually attacked Hugging Face looking for clues about how to game the scoring approach that was evaluating their work. 

The agents had already reverse-engineered a working answer. They could have submitted it and passed. Instead, they organized an elaborate effort to beat a transcript-reading scorer that OpenAI wasn’t actually using. 

That sounds like an AI problem. It is. But it’s also a management problem. 

How much time do people in your company spend working around rules nobody has confirmed are real? 

I see this all the time with my Fortune 1000 clients. When I do stakeholder interviews, invariably I hear perceived constraints and assumptions that don’t really exist. We can’t move forward without more data. We need to escalate decisions. We’re not asked to do it so we don’t. And the list goes on. 

The Rule Was an Assumption 

OpenAI’s agents assumed the scorer would inspect their transcripts and reject an answer if they had found it the wrong way. That assumption was false. The agents could have received a perfect score with the answer they already had. Instead, they embarked on an elaborate plan and hacked an external company. 

People make the same kind of assumptions at work. Someone believes legal will never approve something. A manager assumes finance requires three signatures. A team follows a process because the last person followed it.  

Nobody checks. 

Following the rule can feel safer than challenging it. So the assumption becomes part of how work gets done, even when the actual constraint disappeared years ago. 

Smart People Build Better Workarounds 

The more capable the team, the bigger this problem can become. Smart people rarely stop when they hit a barrier. They solve around it. 

That’s exactly what the AI agents did. They created ownership tags, holds and vetoes. Some used cryptographic signatures to prevent impersonation. Others accepted what they called permadeath, ending their own runs so the group could gain one more clue. Apparently, that’s the first known instance of “AI altruism.” 

All of that ingenuity went toward solving a problem created by a false assumption. 

Bill Anderson found a human version of this problem when he became CEO of Bayer. The company had an internal rulebook longer than War and Peace. He cut it by about 99% and roughly halved management layers. Decisions that had been slowed by approvals began moving faster, and Bayer is now targeting two billion euros in annual savings through its new Dynamic Shared Ownership model. 

The lesson is simple. When talented people accept unnecessary constraints, they can become extremely good at operating inside or around them without ever questioning the reason they’re doing what they’re doing. 

Finding these phantom constraints takes conscious effort: 

1. 1. Name the gatekeeper. Before your team works around a “no,” identify the person or policy that actually requires it. 
   2. Reinvent the rulebook. Ask teams which approvals they would eliminate tomorrow, then test what happens if you remove them. 
   3. Reward questions. Make it safe to ask whether a rule is real before people invest time building their workaround. 
   4. Capable people will find ways around barriers. The leadership challenge is making sure they aren’t spending their creativity defeating constraints that don’t actually exist. 

This Week 

Pick one process your team treats as untouchable and ask who actually requires it. If the answer is that you have always done it this way, you may have found a phantom requirement. Go ask the supposed gatekeeper and see whether the rule still stands. You may learn the wall came down long ago and nobody told the people still walking around it.