What Actually Broke When We Deployed Our First AI Agent (And How We Fixed It)

Three failure modes we hit in production, four guardrails that fixed them, and a pre-ship checklist for teams about to deploy their first agent into real ops.

The demo worked perfectly.

Of course it did. Demos always work. You control the inputs, you know the golden-path outputs, and nobody is feeding the agent anything weird. We ran the demo for stakeholders three times. It was smooth every time.

Two weeks into production, the agent had confidently written to a field it wasn't supposed to touch, hallucinated a vendor contact that didn't exist, and — in one memorable afternoon — gotten itself stuck in a loop that burned through more API tokens in four hours than we'd used in the entire testing phase.

This post is the debrief. What broke, why it broke, what we built to stop it from happening again, and the minimum viable guardrail layer your team should have in place before you ship.

Note: the details below are drawn from a composite of real client engagements. Names and specifics are generalized.

The setup: an AI agent for content ops

The client was a mid-size B2B SaaS company with a lean marketing ops team. They had a growing content library — product pages, case studies, help articles — that needed periodic enrichment: updating outdated stats, tagging content by persona and funnel stage, flagging gaps, and writing summary briefs for the sales team.

Doing this manually meant one person spending about 20 hours a week on a task that was mostly mechanical. A well-scoped AI agent felt like the right call.

The agent's job, in plain English:

1. Pull a content record from the CMS queue
2. Analyze it against a set of criteria (accuracy, completeness, persona alignment)
3. Enrich it — update the metadata fields, write a brief, flag anything that needed human review
4. Write the result back to the CMS and move to the next record

Straightforward. Tool calls to a CMS API. Structured outputs. A prompt with clear criteria. We tested it against 50 records, reviewed the outputs manually, and felt good about the accuracy.

Then we turned it loose on the real queue.

Failure Mode #1: It edited things it wasn't supposed to touch

This is the blast radius problem. The agent wasn't doing anything malicious — it was doing exactly what it thought it was supposed to do. The failure was ours: we gave it write access to an entire record when it only needed write access to specific fields.

The NIST AC-6 principle — least privilege — applies directly here. An agent should have the minimum permissions needed to complete its task, not the maximum permissions its API token allows.

Failure Mode #2: Hallucinated data got written as fact

This one is sneaky because it only happens on edge cases — records where the expected data isn't present. In testing, we'd used records that all had complete contact info. The "no data" case never surfaced.

Failure Mode #3: A retry loop that couldn't stop itself

The four guardrails that actually stuck

After the smoke cleared, we built these four layers into every agent we deploy now — regardless of the use case.

What this cost to build (and what it saved)

The agent is genuinely useful now. It processes 200–300 content records per week, flags about 15% for human review, and the team's manual time on content ops is down from ~20 hours to about 2 hours — the human review queue plus occasional spot checks.

But here's the honest accounting: if we'd built the guardrails before deployment instead of after, we'd have saved ourselves three incidents, one panicked stakeholder call, and $340 in wasted API spend. The retrofit cost more — in time and trust — than the original build would have.

The pre-ship checklist

Before you deploy an agent into any production workflow, run through this list:

This isn't a comprehensive security posture — for that, the OWASP LLM Top 10 is worth a full read, particularly around prompt injection (the #1 risk) and insecure output handling. But the checklist above catches the failure modes that are most likely to bite a team on their first real deployment. For a broader framework covering all four runtime guardrail layers, see our post on agent guardrails in production.

One more thing: prompt injection is real

We didn't run into this in the content ops case, but it's worth flagging: if your agent reads external content and then acts on it, prompt injection is a live threat.

Indirect prompt injection — where malicious instructions are embedded in content the agent reads — has moved from theoretical to documented in the wild. Real incidents include the Perplexity Comet data leak and zero-click RCE exploits in MCP-connected IDEs. If your agent reads emails, web pages, CMS records, or any other user-controlled content before taking actions, you need to treat that content as untrusted input.

Practical mitigation: separate the reading step from the action step. Have the agent summarize or extract structured data from external content first (in a sandboxed step with no tool access), then pass only that structured data to the action step. Don't let the agent read a web page and immediately act on whatever it finds there.

The bottom line

AI agents are genuinely useful for operations work. The content ops agent we deployed is saving 18 hours a week for a team that was drowning in manual tasks — and it's doing it reliably now.

But the "reliably" part didn't come for free. It came from three incidents, a few days of retrofit work, and a set of guardrails that should have been there from day one.

If you're about to ship your first agent into a production workflow: run the checklist, build the guardrails before you deploy, and treat the demo environment as what it is — a controlled simulation, not a proof of production-readiness.

Your future self will thank you.