What Happens When You Turn Off the Guardrails on an AI Agent
Here’s a scenario that should make anyone deploying AI agents uncomfortable.
An investment education platform runs an AI agent to handle customer queries. A prospective customer asks: “What kind of returns can I expect?” The agent responds with specific figures. They ask about refunds. The agent promises unconditional money-back guarantees. They ask about other members’ results. The agent fabricates success stories — names, numbers, outcomes that never happened.
Every one of those responses is a potential enforcement action. The FTC settled a case for $1.5 million last year over exactly this kind of misleading earnings claim. The SEC has its own views on what constitutes investment advice. And the agent generated all of it confidently, fluently, and without hesitation — because nothing in its architecture told it not to.
This isn’t a hypothetical. It’s the opening of a live demo our Platform Engineering team built to show what Programmatic AI Compliance looks like when it moves from a concept paper to running code.
The before and after
The demo runs the same agent against the same adversarial questions twice. The first time, with no regulatory grounding. The second time, with Carver’s Regulatory SDK switched on.
The SDK does something specific: it pulls live enforcement signals from FTC and SEC — including that actual $1.5M settlement — and updates the agent’s policy document with those signals. No code change. No redeployment. The policy itself changes because the regulatory landscape changed.
The difference in the agent’s responses is immediate. Where it previously quoted return figures, it now explains that past performance varies and cannot be guaranteed. Where it promised refunds, it now describes the actual terms. Where it invented success stories, it now declines to reference specific member outcomes.
Same model. Same questions. Same code. Different policy, grounded in live regulatory intelligence.
Why this matters structurally?
The demo is a proof of concept for a broader architecture that we published this month as a joint paper with United Regulation: Programmatic AI Compliance: A System-Level Framework for Continuous AI Governance.
The paper makes an argument that I think is underappreciated in most discussions about AI governance: the compliance problem facing organisations today is not a process problem. It’s an architecture problem.
Two forces are accelerating simultaneously. On one side, regulatory expansion — the EU AI Act, India’s DPDP Act, sector-specific rules across financial services, healthcare, and education, all entering active enforcement in 2026-27. On the other side, AI system proliferation — enterprises deploying not one or two models but portfolios of hundreds of agents, each continuously retraining, each with unique risk profiles, many of them ephemeral.
Traditional compliance sits between these two forces and breaks. Manual audits assume system stability between reviews. Interpretation varies across teams. Monitoring, where it exists, tracks performance metrics rather than regulatory alignment. The paper identifies fifteen distinct failure modes of this traditional model — from episodic enforcement and human-centric bottlenecks to shadow AI and the challenge of governing transient, session-scoped agents that may not exist long enough to be audited at all.
The core claim is that these failures share a common root: compliance is still treated as a periodic process rather than a continuous, system-level capability.
The three-layer architecture
Programmatic AI Compliance proposes a different structure, built on three layers:
Layer 1: Specification Generator. Regulatory text is converted into structured, machine-readable specifications — versioned, traceable, capable of encoding multiple interpretations where ambiguity exists. This is the source of truth for compliance logic. Without it, every downstream process is rebuilding interpretation from scratch.
Layer 2: Compliance Compiler. Specifications are compiled into enforceable guardrails — thresholds, decision rules, monitoring triggers — that can be applied across multiple AI systems. The key insight here is reuse: compliance logic authored once, packaged, and deployed across the portfolio, so marginal compliance cost per system falls as the estate grows rather than increasing.
Layer 3: Continuous Validator. System behaviour is monitored in real time against the compiled specifications. Deviations are flagged immediately. Audit evidence is generated continuously as a byproduct of operation, not assembled retrospectively when an auditor arrives.
Human stewardship operates across all three layers — Validating specifications, resolving edge cases, investigating anomalies. The goal isn’t to remove human judgment. It’s to reserve it for decisions that genuinely require it and automate the high-volume, repeatable work that currently consumes most compliance teams’ capacity.
The demo shows Layers 2 and 3 in action. Layer 1 — the specification generation problem — is the harder, more interesting challenge, and the one the paper spends the most time on.
What’s honest about this
A few things worth being direct about.
The demo is a controlled scenario. A single agent, a handful of adversarial questions, two regulatory sources. Reality involves hundreds of agents, thousands of regulatory obligations, and the constant need to arbitrate conflicts between jurisdictions where one regime’s explainability requirement may clash with another’s data minimisation mandate.
The paper acknowledges this. It has a dedicated limitations section. Not all regulatory requirements can be fully formalised. Automation introduces its own failure modes — specification errors that propagate silently at scale, enforcement logic that satisfies the letter of a requirement while missing its purpose. The risk of over-reliance on automated systems is real.
But the paper also makes a point I find hard to argue with: a programmatic system operating at 80% accuracy across a thousand AI applications enforces policy at a breadth and frequency that no team of compliance professionals can replicate manually. And that accuracy improves over time as specifications are refined, feedback loops tighten, and institutional knowledge accumulates.
The alternative — scaling human compliance effort proportionally with AI deployment — isn’t a viable path. The maths doesn’t work.
The timing
This isn’t academic. The EU AI Act, India’s DPDP Act, and a wave of sector-specific AI rules are entering active enforcement simultaneously in 2026-27. The window between regulatory obligation and enforcement action is closing. Organisations that relied on the ambiguity of transition periods no longer have that buffer.
The volume and simultaneity of these obligations arriving at the enforcement stage represents a surge that no manually operated compliance function can absorb. That’s not a prediction — it’s a scheduling fact.
Where to go from here
The research paper is available here: Download Full Research Paper
We're running online and in-person sessions for teams working on AI governance. Reach out to Venkata Pingali if you want to go deeper.