Why Human Oversight of Autonomous Legal AI Must Be Built In, Not Bolted On

The standard answer to autonomous AI is to keep a human in the loop. Applied to systems that act at machine speed, that is closer to reassurance than control. A lawyer supervising an agent that drafts filings in seconds cannot review each step. Oversight collapses into a signature.

The Evidence: Signatures Go Unchecked

That signatures go unchecked is now well documented. Damien Charlotin's database records more than 1,600 AI-hallucinated citations in court filings. A Stanford study found even purpose-built legal tools hallucinate between 17% and 33% of the time. The problem is structural, not a matter of model accuracy. Recent work on AI's epistemic risks traces it to cognitive offloading and self-reinforcing feedback loops.

‍

Agency by Design: Relocating Control

Agency by design responds by relocating control. The decisive choices are made before the system runs, not during it. Real-time oversight fails because human attention cannot match machine tempo. Constraints set at design time are fixed once, deliberately, and enforced automatically. The EU AI Act's Article 14 gestures at this, but frames oversight as a runtime feature bolted onto a high-risk system. Even that obligation is now deferred to December 2027. The harder question is whether the workflow is built so that the human's decision is load-bearing where it counts.

‍

Three design commitments follow.

‍

1. Bounded Authority

An agent's autonomy should be graduated by consequence. Reversible acts such as searching, drafting, and summarising can run unattended. Irreversible ones, such as filing to a court, binding a client, or waiving a right, must route to a human. Where that line falls is a legal-governance decision, not a technical default, and it belongs to someone accountable for the outcome.

2. Meaningful Friction

A confirmation step works only if it forces genuine verification rather than a reflexive click. Good design makes checking the primary source the path of least resistance.

3. Traceability

Every consequential step leaves a record of what the agent proposed, what a human approved or overrode, and why.

‍

Reconnecting Responsibility and Control

Together these do what the loop cannot. They reconnect responsibility and control. A signature is meant to certify that a person has vouched for the work. A workflow that leaves the person unable to verify has quietly severed that connection. Agency by design restores it, but not for free. Every escalation and every point of friction trades against the efficiency that made the agent attractive, and that trade-off cannot be engineered away. It has to be chosen. The open question is who sets the thresholds, whether the vendor, the firm, the regulator, or the lawyer who signs.

‍

The question is not whether a human can override the machine. It is whether the workflow was ever built to let them.

Sources:

Regulation (EU) 2024/1689 (EU AI Act), Article 14
Digital Omnibus on AI (2026 amendments to Regulation (EU) 2024/1689)
Magesh, V., Surani, F., Dahl, M., Suzgun, M., Manning, C. D., & Ho, D. E. (2025). Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools. Journal of Empirical Legal Studies. https://onlinelibrary.wiley.com/doi/full/10.1111/jels.12413
Charlotin, D. AI Hallucination Cases Database. https://www.damiencharlotin.com/hallucinations/
Hogan Lovells, EU legislators agree to delay for high-risk AI rules. https://www.hoganlovells.com/en/publications/eu-legislators-agree-to-delay-for-highrisk-ai-rules
White & Case, EU agrees Digital Omnibus deal to simplify AI rules. https://www.whitecase.com/insight-alert/eu-agrees-digital-omnibus-deal-simplify-ai-rules
Covington (Inside Privacy), EU AI Act Update: Timeline Relief, Targeted Simplification, and New Prohibitions. https://www.insideprivacy.com/artificial-intelligence/eu-ai-act-update-timeline-relief-targeted-simplification-and-new-prohibitions/
User-provided analysis on AI epistemic risks (persuasion/manipulation, cognitive offloading, feedback loops) — original authorship not independently identified; attribute before publication.

‍

United States
447 Broadway,
2nd Floor Suite #563,
New York 10013

Listen to Carver Conversations on Moltbook

AI Regulatory OS
Regulatory Intelligence
Regulatory Sources
Regulatory Platforms
Horizon Scanning
Regulatory Monitoring
Intelligence vs Compliance

Pricing
Podcasts
Knowledge Base
Resources
Glossary
Use cases

Home
Core Solution
Regulatory OS Solution
RegWatch
Technology
About Us

hello@carveragents.ai
github.com/carveragents