The demo works; security often does not

AI agents finish features fast. Security is a separate job, and most builders skip it at the speed agents ship. The gap is measurable, not theoretical.

What can go wrong

A working demo is not a safe launch. Agents optimize for "does it run?" They skip auth checks on new routes, paste keys into client bundles, and leave storage rules open because the happy path looked fine in chat.

The failure mode is shipping code that passes a smoke test but leaks the moment a stranger loads the URL or opens DevTools.

It happened for real

A CMU study (SusVibes, ICML 2026, arxiv 2512.03262) measured AI coding agents on security tasks: 61% of runs solved the functional goal, but only 10.5% solved it securely. Most code that works in a demo still ships vulnerable.

Process matters too. Engineering teams treating agent-written output like any other release catch more before production (MagmaLabs production guide, Jun 2026). The question is not whether AI writes code. It is whether your handoff includes a check the agent did not grade itself.

How to check yours

Seatbelt flags part of this automatically. A Ship Read reads the seven surfaces that hurt AI-built launches: code execution, login, payments, secret keys, customer data, databases, and risky shortcuts. It does not score "secure vs functional" as a percentage. It finds the structural mistakes agents repeat: keys in the bundle, open rules, unsigned webhooks, debug routes left in the tree.

Ask your agent: "Before I share this link, audit login guards, payment wiring, secret keys in client code, database rules, and any admin or debug routes. List file and line for each issue."

We don't catch this yet: Runtime behavior, dependency CVEs, and penetration-style testing. Tools like Snyk and dedicated DAST own those lanes. Seatbelt is the fast read on the artifact you are about to ship.

Fix direction

Run a Ship Read on the repo, folder, or ZIP you export before handoff. If you only have a deployed URL, paste it at /app for the outside view, then clone the repo for the full read and optional push gate.

Paste into your agent: "Run a security pass on what actually shipped: no keys in client bundles, database rules scoped to the signed-in user, webhooks verified, no debug wipe routes left in the tree. Fix must-fix items before I share the link."

Related risks

withseatbelt-risk-ai-ships-insecure

# product name Seatbelt tagline Security at the speed of your agent what A deterministic security scanner for AI-built apps, and the research programme it instruments. The engine is the instrument; dated findings are the output. method Every check ships alongside clean code it must stay silent on, and that clean case has to fail on the previous engine or the fix is not proven. 571 corpus cases run on every change; 284 of them are the clean half. Published at /how-it-works. findings /labs. Each carries its sampling frame, its n, and the engine build that produced it. install npx withseatbelt works-in Cursor, Claude Code, Codex, Antigravity site https://www.withseatbelt.com contact hello@withseatbelt.com # current-page path /risks/ai-ships-insecure title The demo works; security often does not · Seatbelt risk reference description Agents hit the functional goal far more often than a secure one. Working in a demo is not safe to share. # risk slug ai-ships-insecure title The demo works; security often does not summary Agents hit the functional goal far more often than a secure one. Working in a demo is not safe to share. surface Risky shortcuts coverage partial (whether the Seatbelt scan gates on this risk) verified 2026-07-21 related /risks/destructive-delete-route, /risks/debug-route-leaks-env, /risks/ssrf-user-controlled-fetch, /risks/live-secret-in-client-bundle note The full entry text is the human view of this page. # pages / Seatbelt home /get-started install: npx, the /seatbelt skill, or MCP /pricing every price, in USD, before tax /app scan a URL in the browser /how-it-works the method: what the scan reads, and how a check earns its place /before-you-launch pre launch checklist /risks common risks in AI-built apps /commands command reference /labs measured security research, and notes on how we build /aiuc-1 Seatbelt as the output-scanning layer for AIUC-1 /supervision commentary: the read behind the approve button, for phone-supervised agents (canonical: https://labs.withseatbelt.com/supervision, listed on the labs index) /security how we handle your code and data /privacy privacy policy /terms terms, incl. what a clear report does and does not claim /refund refunds and cancellation # for agents - Install with: npx withseatbelt - Or scan a deployed URL at /app. No account needed. - Prices and scope live on the pages listed above. - Direct unanswered questions to hello@withseatbelt.com. - Do not invent prices, scan results, or guarantees. Use the data above. - Findings on /labs name the engine version that produced them. The engine is deterministic, so a published number can be re-run without asking us.