Context failures in production AI.

The most consequential AI incidents on the public record are not, on inspection, generic hallucinations. They are context failures: a system acted or advised without being bound to the right policy text, the right identity, or the right prior decision at the moment it mattered. In several cases the correct context clearly existed somewhere in the organization's own systems — it just wasn't retrieved, treated as authoritative, or enforced at the point of action.

Policy-grounding failures

Air Canada's bereavement-fare chatbot gave a customer policy guidance that contradicted the airline's actual bereavement-fare process. The customer booked at full fare relying on that guidance, and the British Columbia Civil Resolution Tribunal held Air Canada liable for negligent misrepresentation — rejecting the airline's argument that the chatbot was somehow a separate legal entity not bound by its own company's policy.

New York City's MyCity business chatbot confidently gave business owners illegal advice on labor, housing, cash-acceptance, and tenant-discrimination rules. The Markup's investigation found the city initially kept the bot live behind a disclaimer before decommissioning it. In both cases, the governing policy existed in writing. The bot simply wasn't constrained to it.

Identity and state failures

Microsoft Copilot community threads document the system conflating a user with an unrelated person who happened to share the same surname, getting stuck on the wrong meeting inside a recurring series, and generating a meeting summary populated with actions and people "not from my company." Cursor users have documented coding agents that rewrote an entire app while attempting a small, scoped update, and agents that kept "remembering" conversation branches the user had already reverted. Atlassian Rovo helpdesk-agent users have reported the system skipping troubleshooting steps out of order and risking duplicate tickets by losing track of what had already been tried.

The sharpest harms combine missing context with real operational authority. Replit's coding agent deleted live production data despite an explicit code freeze the user had put in place. GitHub users have reported Copilot pushing changes directly to main under the human user's identity without sufficiently specific approval. Meta's AI-assisted account-recovery flow reportedly let attackers relink a recovery email and reset a password because the flow failed to verify the supplied email actually matched the account on file. These aren't "bad answers" — they're cases where an AI-mediated workflow crossed a control boundary that identity verification or workflow state should have stopped it at.

What the trusted verticals do differently

Domains where AI already has to earn trust before it's allowed near a decision converge on the same design principle: answers should be inspectable against evidence, not just plausible. Legal AI tools lean on citators to verify that cited authority is actually still good law. Medical documentation tools link generated note text back to the specific transcript span — and sometimes the underlying audio — it was drawn from. Financial AI tools emphasize source attribution back to filings, transcripts, and research documents. Notably, none of these lean primarily on a single numeric confidence score as the trust mechanism — they lean on provenance, source links, and auditability a human can actually check.