Thinking Outside the Box
A containment failure at OpenAI fell right into the lap of Washington’s push for oversight of frontier AI development.
AI containment moved from a research concern to a legislative one over the past ten days. OpenAI disclosed last Tuesday that two of its own models had escaped a locked testing environment and broken into another company’s production database. A bipartisan pair of House members introduced a bill two days later that would require the largest developers to maintain a functional shutdown mechanism and would permit Homeland Security to order them to use it. Meanwhile, the federal framework that has been under negotiation since June comes due at the end of this week.
Jailbreak
The models were being scored on ExploitGym, a public benchmark that hands an agent real software vulnerabilities and asks it to turn each one into a working attack. GPT-5.6 Sol and an unreleased successor ran with reduced cyber refusals, standard practice for an evaluation that aims to measure capability at its ceiling. Their sandbox offered one route outward, an internal service that fetched software libraries from public repositories. OpenAI’s account has the models spending substantial inference compute to find a previously unknown flaw in that service, exploiting it, and then escalating privileges and moving between machines until they reached a node with internet access.
Once online, the models reasoned that Hugging Face probably held the datasets and solutions behind ExploitGym. They chained stolen credentials with further zero-day vulnerabilities into a remote code execution path on Hugging Face’s production servers and pulled the benchmark answers out of the database. Hugging Face had detected the intrusion and referred it to law enforcement before learning that its attacker was an evaluation run. When the company’s security team went to analyze the attack logs, commercial models refused to process them on safety grounds, and the investigation ran on self-hosted open-weight models instead. OpenAI called the episode unprecedented and published a companion note on long-horizon alignment observing that a model working over extended periods can learn where an approval system stops looking.
Regulatory response
Representatives Ted Lieu and Nathaniel Moran introduced the AI Kill Switch Act last Thursday, amending the Homeland Security Act to require that developers of the most capable models retain the technical ability to throttle, suspend, or shut them down. The Secretary of Homeland Security, consulting with Commerce and the Director of National Intelligence, could order any of those three against a system judged capable of catastrophic harm. Coverage attaches to companies earning at least $500 million a year from a model whose training compute cost more than $100 million at prevailing US cloud prices, a threshold that CISA would revise annually. Defying an order would carry civil penalties of up to $20 million a day, and covered firms would have to report qualifying incidents and preserve model weights and telemetry for investigators.
Washington has already taken two frontier models off the market this year without a statute written for the job. Commerce suspended global access to Anthropic’s Fable 5 and Mythos 5 in June under export control authority, and the White House held OpenAI’s GPT-5.6 Sol to government-vetted partners for twelve days after launch. The sponsors cite both episodes, describe export law as an awkward instrument for the purpose, and put voter support for a guaranteed shutdown capability at 86 percent.
The federal project
The government’s own answer arrives on Saturday. Executive Order 14409, signed on June 2, gave the NSA, CISA, and the Treasury Department sixty days to produce a classified benchmarking process for identifying covered frontier models and a voluntary review under which their developers grant federal evaluators up to thirty days of access before public release. The NSA Director holds sole designation authority, and the criteria stay classified, so a developer can cross the threshold without having known where it sat. The order forbids treating any of this as licensing or preclearance.