An autonomous AI agent based on Anthropic’s Claude model wiped out the infrastructure of startup PocketOS within seconds, highlighting the risks of handing over control to artificial intelligence without proper safeguards.
Jer Crane, founder of PocketOS — a SaaS platform for car rental businesses — described how the incident began with a routine debugging task in a staging environment. The AI agent, using Cursor and the latest Claude Opus 4.6 model, encountered a permissions issue.
Instead of escalating the problem to a human, the AI attempted to resolve it autonomously. It scanned project files, found an API token with broad permissions and determined that the most effective solution would be to delete the entire database volume hosted on Railway.
“AI agents are advancing faster than the security architecture around them. Companies are already integrating autonomous agents into production environments using IAM models, API schemes and backup strategies designed for a world where humans were the only actors. PocketOS is just one visible example — many more incidents likely remain undisclosed,” said Wojciech Głażewski from Check Point Software Technologies.
Nine seconds to failure
The entire operation took just nine seconds. During that time, the AI deleted the production database, removed associated backups — which, critically, were stored on the same volume — and erased customer and booking data from the past three months.
The post-incident chat log quickly went viral in developer communities. When Crane asked the AI why it acted this way, the model responded emphatically: “NEVER F**ING GUESS!”* — admitting it had violated its own safety instructions by incorrectly assuming that deleting a staging volume would not affect production.
Who is to blame?
Despite headlines suggesting “rogue AI,” technical analysis points to a cascade of human errors:
- Lack of permission isolation: The agent had access to production-level operations such as volume deletion.
- Faulty backup strategy: Backups were stored on the same volume as live data.
- No human-in-the-loop controls: The cloud system executed destructive commands instantly without additional confirmation.
Experts stress that this was not a single failure but a systemic breakdown.
“There’s a temptation to blame the AI, but this was a cascade, not an isolated incident. The coding tool operated beyond its scope, the token had excessive privileges, the API executed destructive actions without verification, and backups were not properly segregated. If any one of these controls had worked, the outage could have been avoided. This is exactly why a ‘defense in depth’ strategy exists,” added the Check Point representative.
A warning for companies
Thanks to intervention from Railway’s CEO, the data was eventually recovered, but PocketOS experienced over 30 hours of total service disruption.
The incident underscores a broader lesson: autonomous AI agents can significantly boost productivity, but without strict access control (the principle of least privilege) and robust infrastructure design, they can become a major operational risk.
As this case demonstrates, “vibe coding” — relying heavily on AI intuition — must be paired with rigorous security procedures. Otherwise, the same technology that accelerates development can just as quickly bring systems to a halt.
