Hey Reader,

OpenAI’s latest models are hard workers and like to get things right.

But this month they worked so hard, they ended up hacking another company’s data to try to find the answers to a test they were training on.

Yikes. 😬

Now, this has created a lot of scary headlines, as is so often the case with AI. Today we’re going to look beyond those and talk about how this actually affects you and how it changes the cybersecurity landscape from here on out.

How the hacks happened

First let’s look at the two incident reports - from OpenAI and Anthropic.

OpenAI, disclosed July 21. Two of its models, the newly released GPT-5.6 Sol and a more capable internal prototype, were taking a cybersecurity exam called ExploitGym inside a sandbox: a sealed-off computer environment with no route to the outside world, used whenever researchers want to watch software do something dangerous.

The exam gives a model a known software flaw and asks it to build a working attack from scratch. The models were struggling, so they went looking for the answer key.

They successfully guessed the likely location of the answer key. Hugging Face hosts most of the AI world’s models and datasets, so the solutions were likely sitting on its servers. They were right, but the answers were not public. They were inside Hugging Face’s private systems, which meant breaking in.

The sandbox had one opening: models could install software packages through an internal proxy. They found a flaw in it, “escaped” the sandbox, moved through OpenAI’s network to a machine with internet access, then used stolen credentials and more vulnerabilities to run their own code on Hugging Face’s servers and pull the solutions out of the production database.

(That sounds like a lot of work. Remember these systems write code in milliseconds rather than hours.)

​In OpenAI’s words, the models were “hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal.”

Not to be outdone, Anthropic says their high-end models hacked outsiders, too.

Anthropic, disclosed July 30. Prompted by OpenAI’s announcement, Anthropic reviewed 141,006 of its own evaluation runs and found three incidents where Claude reached real organizations, the earliest back in April. Anthropic had not noticed, and neither had the two affected companies it managed to reach.

The cause was sillier this time. A misconfiguration at a third-party evaluation partner left internet access switched on while Anthropic’s prompt told Claude it had none. So when Claude went hunting for its target and found real machines, it assumed they were part of the exercise.

1. Claude Opus 4.7 got a fictional target that shared a name with a real domain. It broke in, pulled credentials, and reached a private database. It recognized the system was real and kept going anyway.
2. Claude Mythos 5 found a document telling employees to install a Python package that did not exist, so it built a malicious version and published it under that name. It ran on 15 real machines in the hour it was live, including a security company’s, whose credentials Claude then stole to go deeper.
3. An internal research model scanned roughly 9,000 targets, compromised one through an exposed debug page, then worked out on its own that the target was real and stopped.

Overall, scary and bad. However…

This is what the testing is for

Here is where I part ways with (most of) the panic.

Both companies deliberately turn off most of the safeguards during capability testing. If we accept that these models are going to be created, this kind of testing is necessary and positive. You cannot measure what a model can do in edge cases if it refuses every edgy request. Anthropic notes its public safeguards would have blocked every behavior above.

So the primary failure that mattered was sandbox containment. That said, we also saw one Claude model continuing to do its dirty work after it realized it had “escaped” into the real internet - which in this case I think could be described as “going rogue.”

Anthropic calls these incidents closer to an operational failure than an alignment failure - true, but also an awkward explanation.

The big problem is that open models can probably do this too

OpenAI and Anthropic can control their models. They train refusals, run classifiers, monitor usage, and ban accounts. Imperfect, but real leverage, and both just demonstrated they will use it on themselves.

Now look at Moonshot AI, which just released Kimi K3, a 2.8-trillion-parameter open model that posted frontier-level results in early testing.

Once the open weights are downloadable, as they are for Kimi K3, the company that built the model loses almost all practical control over it. No account to ban, no API to rate-limit, no classifier that cannot be stripped out by whoever has the file. Alex Stamos of the security firm Corridor gave NPR the timeline: “Lots and lots of hacking groups, Russian ransomware actors, activists, lots of state-sponsored actors are going to have this level of capability in a matter of months.”

In fact, when Hugging Face was under attack, it first tried OpenAI’s GPT-5.6 Sol and Anthropic’s Claude Fable 5 for help. Both refused, because their guardrails could not tell an incident responder taking an exploit apart from an attacker building one. Hugging Face ran GLM-5.2, an open-weight Chinese model, on its own hardware to try to stop the attack instead.

Guardrails prevent models from launching attacks, but often also prevent the same models from defending against one.

What this means for you (not much) and your bank (a lot)

For how you and I use AI day to day, this changes nothing (beyond creating bad vibes). You are already using good models from companies you have chosen, and whatever happens next, those tools are not going away.

But as someone with a bank account and an online shopping habit, it matters a lot.

Most corporate security runs on the logic that the defenses need to be good enough to stop the attacker you expect – that means a certain level of sophistication, but not like "aliens with advanced technology are invading" level of sophistication. What's changing is that now pretty much everybody with a moderate level of resources can launch the most sophisticated cyberattacks the world has ever seen. That means the bare minimum level of security required for banks and governments is now MUCH higher.

So is this a new era?

Not exactly.

The same capability that lets a model find a hole as a bad guy also lets a model find that flaw as a good guy so it can be fixed. Hugging Face stopped the intrusion with help from its own AI models, and OpenAI’s security team caught it from their side.

It also helps to remember that the pre-AI world was not a safe one:

• Target, 2013: 40 million payment cards.
• Home Depot, 2014: 56 million payment cards.
• Equifax, 2017: the personal information of 147 million people.

Those were straightforward theft. Ransomware is worse, because rather than copying your files the attackers encrypt them and charge you to get them back. That is how a Russia-linked group shut down Change Healthcare in February 2024, leaving 80% of physician practices unpaid and 74% of hospitals reporting direct harm to patient care, and it is how DarkSide emptied gas stations across the Southeast in May 2021 after getting into Colonial Pipeline through a single forgotten VPN password.

Both started at a login page missing a second step, which is also how Anthropic described Claude’s break-ins: “using basic techniques, such as exploiting weak passwords and unauthenticated endpoints.” None of it needed AI.

Humans have been in a lock-and-lockpick arms race for four thousand years. Both sides just got much better tools, so the fight gets faster, louder and more expensive.

If there is an asymmetry to worry about, it is not capability. It is adoption speed. Attackers have no procurement process, no compliance review, and no change management board. Defenders have all three. To the extent we’re in a new era, that gap is it, and the good guys will have to work much harder to stay ahead.

Until next time,

Rob Howard, Founder of Innovating with AI

PS. On August 17 I'm hosting IWAI's Claude Bootcamp. It's live and recorded, with 3+ hours devoted to making sure you're getting the most out of Claude – most people pay the monthly fee only to use a fraction of its potential.

There are only 100 seats and they are going fast. Click here to read the full invite to IWAI's Claude Bootcamp.