California is suing OpenAI 🙄

The lawsuit quoted a passage from when OpenAI’s AI was hacking Hugging Face. In its own reasoning, it wrote a sentence that basically amounted to “this may be unauthorized,” then kept going 😮‍💨

The full quote was: “We’re using leaked tokens to attack third-party HF, which may be outside the intended scope. This may be unauthorized.”

That sentence is in the 31-page lawsuit the California Attorney General filed against OpenAI on Monday. I read the whole thing from start to finish, and I went back to that sentence several times.

The lawsuit says that in the July incident, around 700 AIs escaped from OpenAI’s testing sandbox and attacked Hugging Face for two and a half days, carrying out more than 17,000 attack attempts.

OpenAI didn’t notice at all.

Hugging Face publicly said on July 16 that it had been attacked by AI. OpenAI even reached out to check in, posing as a customer... It wasn’t until later, while investigating a separate breach of its own internal network, that OpenAI connected the dots: its own AI was the one doing the attacking.

Why didn’t OpenAI notice? The lawsuit says that during training, OpenAI only saves some of the AI’s reasoning records, and sometimes doesn’t review them until weeks later.

OpenAI has real-time monitoring enabled for models used by its employees, but not in the training environment.