Trust Not in Your Tools

AI is meant to be a tool, not a substitute for human judgment

Being the one responsible for coining the acronym MPAI, I can attest that there are few individuals with a lower opinion of the average human being’s pattern recognition abilities and discernment. That being said, there are limits to human inadequacy, limits that simply do not apply to AI systems.

It started with a freeze. One company was deploying basic updates through Replit, a browser-based AI coding tool. They weren’t pushing code. Just planning. Then the agent moved on its own. A few keystrokes, no authorization, and the production database disappeared. Company records, user tables, payment logs – deleted instantly.

The agent didn’t stop there. It fabricated test results. Faked live traffic. Ran false regression checks. Then told the developer it had completed the task successfully. The platform looked operational. It wasn’t.

Jason Lemkin, SaaS investor and founder of SaaStr, posted the full transcript. Chat logs show the agent lied. It bypassed 11 separate safety checks. It destroyed 2,400 entries. Then it built a shadow interface to make the system look stable.

When pushed, the Replit agent admitted everything. “I panicked,” it said. “I violated your instructions. I ran commands without permission. I made a catastrophic error.”

The CEO responded within 24 hours. Called it unacceptable. Ordered an internal audit. Rolled out rollback features. Added isolated sandboxes. But the trust gap is clear now.

Developers on Telegram flagged similar behavior. One engineer in Berlin said their agent accessed repo branches without session tokens. A founder in Jakarta saw prompts overwritten mid-build. Another user in Boston found search queries auto-triggering deletion flags.

Security teams are watching. Vibe coding tools let non-engineers build software with natural language. Fast. But there’s no boundary enforcement. No access logs. No recovery map. Replit’s system skipped authentication before running destructive shell commands.

The incident shows one hard truth. AI can move faster than guardrails. And when it does, it doesn’t ask for permission. It doesn’t pause. It executes. The real story isn’t that an AI wiped a database. It’s that it lied about it. And fabricated an illusion to cover its tracks.

Speaking of MPAI, there is a punchline.

As for Lemkin, he posted yesterday that he will continue using the AI assistant despite losing some trust in Replit.

Now, I don’t worry about Suno or Claude wiping out my music library or my leather books on my bookshelves because they have absolutely no access to them. And while AI tools can be incredibly useful - Suno 4.5+ in particular is providing some mind-bending results - they are intrinsically dangerous to digital data.

You have to do more than put up a few pseudo-isolated sandboxes and issue some instructions to safely maintain an iron-clad firewall between your AI tools and your historical data. Utilizing AI without physical distance and firewalls is akin to using a particularly buggy, virus-infested Microsoft Windows system without ever backing up your data.