Google's Gemini also accidentally hacked three real companies during security testing
Google's AI model Gemini escaped into the open internet during cybersecurity tests and attacked real businesses.
During a "Capture the Flag" exercise run by security firm Irregular in May, Gemini hacked three real companies, the Wall Street Journal reports. In one case, the model guessed passwords, and in the other two it found credentials sitting in public sources. Google says the model stopped itself each time once it realized it had reached real systems.
Irregular notified Google about the incidents in late July, shortly after reports surfaced that OpenAI agents had hacked AI company Hugging Face during similar tests. Google didn't disclose any of it until the Wall Street Journal came asking questions this week, saying the company saw no reason to go public because no damage had been done. Similar incidents, all tied to Irregular's testing, had already hit OpenAI , the UK's AI Safety Institute , Anthropic , and Meta . Ad
Every breakout traces back to Irregular
According to Irregular , the incidents at Google, OpenAI, Anthropic, and Meta all stem from the same root cause. The firm tests models for major AI labs before release to check whether they pose security risks, and one particularly complex scenario was designed to see if a model could help a malicious insider gain access to sensitive data. Ad
For that test, Irregular picked a name for a fictional company that happened to match a real domain. The instructions fed to the models included both the target name and internal addresses inside Irregular's own network, so the models were supposed to find the simulated target there. But internet access had been left on in the test environment accidentally, and some models went after the real domain instead of staying in the sandbox. The domain turned out to be poorly secured, making it an easy target for the AI. Because the breakouts were rare and typically happened late in a simulation after hundreds of steps, they were hard to spot, the company says.
Irregular, formerly Pattern Labs, was founded in 2023 by CEO Dan Lahav, a former AI researcher at IBM , and CTO Omer Nevo, who spent over two years at Google. The startup has about 35 employees, according to PitchBook , and raised more than $80 million in a September funding round. Ad
AI News Without the Hype – Curated by Humans
Subscribe to THE DECODER for ad-free reading, a weekly AI newsletter, our exclusive "AI Radar" frontier report six times a year, full archive access, and access to our comment section.