AI Incident Response: Failure Modes, Controls and Evidence for SMB and HealthTech Teams

Short answer: Most AI incidents fall into a few repeatable failure modes: hallucinated output relied on as fact, sensitive data leaking into AI tools, prompt injection, biased outcomes and silent model or vendor changes. Treat each as a governance risk with a named control, an owner and evidence, and extend your incident response plan to cover AI systems.

Related guides:

• AI governance policy: template, requirements and audit evidence
• NIST AI RMF: everything you need to know
• Incident management policy

Key takeaways

• Public AI incidents rarely involve exotic attacks. They involve ordinary governance gaps: no review step, no usage policy, no owner.
• At least one tribunal has held a company responsible for what its chatbot told a customer. Do not plan on "the AI said it" as a defense.
• Group risks into AI failure modes, then assign each one a control and a piece of evidence an auditor can inspect.
• An AI incident response plan is your existing IR plan plus an AI system inventory, a disable path, logging and clear notification triggers.
• In HealthTech, any AI incident involving PHI needs a HIPAA breach analysis, and every AI vendor that receives PHI needs a BAA.

---

What do public AI incidents actually teach?

They teach that accountability for AI output stays with the organization that deploys it. Two court and tribunal decisions and one common workplace scenario show this.

The Air Canada chatbot. In Moffatt v. Air Canada, 2024 BCCRT 149, the British Columbia Civil Resolution Tribunal held the airline responsible for inaccurate bereavement-fare information that its website chatbot gave a customer. The tribunal rejected the argument that the chatbot was responsible for its own statements. It is a ruling from a Canadian civil resolution tribunal, not binding precedent in the US, EU or UK. Lesson: customer-facing AI output is your company speaking.

Shadow AI data leakage (illustrative scenario). A common scenario, not a specific incident: employees paste source code, internal documents or customer data into consumer AI tools to work faster, and the employer finds out only afterward and then restricts tool use. Lesson: leakage often comes from well-meaning employees using unapproved tools, often called shadow AI.

Mata v. Avianca. In Mata v. Avianca, Inc., No. 22-cv-1461 (S.D.N.Y.), lawyers were sanctioned in June 2023 by a federal judge after filing a brief that contained fictitious case citations generated by ChatGPT. Lesson: professionals remain responsible for verifying AI output.

None of these involved a sophisticated attacker. Each was a missing control. For more examples, the AI Incident Database is a public collection of reported AI harms that is useful when building your risk register.

What are the most common AI failure modes?

Five failure modes cover most AI incidents that SMB and HealthTech companies are likely to face, and they give your risk register a stable structure.

1. Hallucinated output relied on as fact. A model generates confident, wrong content, such as a policy, citation or clinical-sounding recommendation, and someone acts on it. The Air Canada and Avianca cases fit here.
2. Sensitive data leakage into AI tools (shadow AI). Employees paste source code, customer data or PHI into unapproved or consumer-tier AI tools.
3. Prompt injection and agents taking unintended actions. Instructions hidden in an email, web page or document are treated as commands by a model. If the model has tool access, it may send data or change records nobody intended. Our agentic AI security guide covers the technical controls in depth.
4. Bias and unfair outcomes. A model used for screening, pricing or triage produces systematically worse results for some groups, often because of skewed data or proxy variables.
5. Model or vendor changes breaking behavior. A provider updates or retires a model or changes default settings, and a workflow that passed testing last quarter quietly fails.

Which controls prevent each failure mode, and what evidence proves them?

Each failure mode needs a preventive control and evidence that the control operates. This table is a practical starting point.

Failure mode	What went wrong	Control	Evidence an auditor would expect
Hallucinated output	Output treated as authoritative without review	Human review for high-impact outputs, grounding on approved sources, clear user disclaimers, restricted topics for customer-facing bots	Review workflow records, test results against a set of known questions, approved knowledge source list, disclaimer screenshots
Data leakage (shadow AI)	Sensitive data entered into unapproved tools	AI acceptable use policy, approved tool list, enterprise accounts with retention controls, data loss prevention or browser controls, training	Signed policy acknowledgments, approved AI tool register, vendor settings screenshots, training completion records
Prompt injection and agent actions	Untrusted content steers a model with tool access	Least-privilege tool permissions, human approval for consequential actions, input and output filtering, separation of untrusted content	Permission configuration exports, approval logs, red-team or test reports, change tickets for agent permissions
Bias and unfair outcomes	Model produces unequal results across groups	Impact assessment before deployment, fairness testing on representative data, human review of adverse decisions, appeal path	Completed impact assessments, test results with dates, documented appeal or override process
Model or vendor changes	Upstream change alters behavior without notice	Pinned model versions where possible, regression test suite, vendor change monitoring, contract terms on notice and data use	Model version records, regression test runs tied to change tickets, vendor review records, contract clauses

Give each AI system a named owner and run tests on a schedule, not only at launch. Auditors look for evidence that controls operated over time.

What should an AI incident response plan add to your existing IR plan?

An AI incident response plan should extend your current plan, not replace it. Detection, triage, containment and review still apply. Add these six elements:

1. AI system inventory. List every AI system, including AI features embedded in SaaS tools, internal models and agents. Record purpose, data processed and whether outputs reach customers.
2. Model and vendor owner. Name an internal owner for each system and record the vendor contact and contractual notice obligations.
3. Kill switch or disable path. Document how to turn off the feature, revoke API keys or fall back to a non-AI workflow, and test it.
4. Prompt and output logging with privacy safeguards. Log inputs, outputs, model version and actions taken. Apply access controls, retention limits and redaction, because prompt logs can contain sensitive data or PHI.
5. Customer notification triggers. Define which AI incidents require telling customers, such as exposure of their data or unauthorized actions in their accounts, aligned with contract commitments.
6. HIPAA breach analysis when PHI is involved. If PHI was disclosed to an unauthorized AI tool or exposed through AI output, run your breach risk assessment and follow your Breach Notification Rule and BAA obligations.

A short runbook by failure mode

Rate severity as you already do, and treat any PHI or customer data exposure as high until assessed:

Failure mode	Detect	Contain	Recover and review
Hallucinated output	Customer complaint, reviewer flag or output test failure	Pull or correct the content; add human review	Fix prompts or sources; assess with legal input whether customers are owed what the AI told them
Data leakage	DLP alert, audit log or employee report	Revoke access to the tool; request vendor deletion where available	Breach risk assessment; update the approved tool list
Prompt injection or agent action	Unexpected tool calls or changes in logs	Use the kill switch; rotate exposed credentials	Reverse changes; tighten permissions and approvals
Bias or unfair outcome	Monitoring metrics or a complaint	Pause automated decisions; route to humans	Reassess impact; retest before re-enabling
Model or vendor change	Regression test failure or vendor notice	Pin or roll back the version	Update tests and the vendor record

Preserve logs, model versions and timestamps as evidence, and close each incident with a post-incident review.

Keep regulatory clocks in the plan too. A GDPR personal data breach must be reported to the supervisory authority within 72 hours of becoming aware of it unless it is unlikely to result in a risk (see our GDPR breach notification guide). A HIPAA breach of unsecured PHI must be reported to affected individuals without unreasonable delay and no later than 60 days after discovery. If you provide AI systems in the EU, the EU AI Act may also add serious-incident reporting duties for high-risk systems. Application dates have been subject to proposed changes, so confirm the current timeline and whether any of your systems are in scope. Then add an AI scenario to your tabletop exercises, such as a chatbot giving wrong refund terms. For the general structure, see our SOC 2 incident response guide.

How do these controls map to NIST AI RMF, ISO 42001 and SOC 2?

The controls map to frameworks buyers already reference. The NIST AI Risk Management Framework organizes activities into four functions: Govern, Map, Measure and Manage. ISO/IEC 42001 sets requirements for an AI management system, including an AI policy (clause 5.2), AI risk assessment and treatment (clauses 8.2 and 8.3) and AI system impact assessment (clause 8.4). SOC 2 is narrower: its CC7 criteria cover detecting, evaluating and responding to security incidents, so it applies to the incident response part of this list, not to bias or output accuracy.

Failure mode	Primary control	NIST AI RMF function	ISO/IEC 42001 clause
Hallucinated output	Human review and output testing	Measure, Manage	8.2 and 8.3 risk assessment and treatment
Data leakage (shadow AI)	AI acceptable use policy and approved tool list	Govern	5.2 AI policy; 8.3 risk treatment
Prompt injection and agent actions	Least privilege and human approval	Map, Manage	8.2 and 8.3 risk assessment and treatment
Bias and unfair outcomes	Impact assessment and fairness testing	Map, Measure	8.4 AI system impact assessment
Model or vendor changes	Version pinning, regression tests, vendor monitoring	Govern, Manage	8.3 risk treatment, including supplier risk
All failure modes	AI incident response plan and inventory	Govern, Manage	10 improvement and corrective action

This is our own mapping aid, not an official crosswalk or a certification checklist. For the security incidents in this list, such as data leakage or a compromised agent, your SOC 2 incident response controls (CC7) also apply.

What changes when AI touches PHI or clinical workflows?

When AI handles PHI or produces clinical-adjacent output, the same controls need to be stricter and better documented.

• Clinical-adjacent outputs. Note summaries or triage suggestions can cause harm if a hallucination is trusted. Require clinician review before outputs influence care and record who approved what.
• PHI in prompts. Restrict unapproved tools, route PHI only to approved systems and apply the minimum necessary standard to what you send a model.
• BAAs with AI vendors. Any AI vendor that creates, receives, maintains or transmits PHI on your behalf is a business associate and needs a signed BAA. Confirm coverage before PHI flows, since some services offer BAAs only on certain plans.
• Breach analysis. PHI entered into an unapproved AI tool can be an impermissible disclosure. Run your documented breach risk assessment rather than assuming it is harmless.
• Logs as PHI. Prompt and output logs that contain PHI are themselves ePHI. Encrypt them, restrict access and include them in your HIPAA risk analysis.

How SecureSlate helps

SecureSlate helps SMB and HealthTech teams turn lessons from AI incidents into a working compliance program. You can keep your AI use and incident response policies with the rest of your policies, record AI risks in your risk register, review AI providers through vendor risk management, and organize the related evidence for your compliance frameworks, so you can show buyers and auditors the policies, risk entries and vendor reviews behind your AI use.

Start your free SecureSlate trial

FAQ

What counts as an AI incident?

An AI incident is any event where an AI system causes or nearly causes harm, such as incorrect output that someone relies on, exposure of sensitive data, unauthorized actions or unfair outcomes. Define it in your incident management policy and include near misses.

Do we need a separate AI incident response plan?

Usually not. Most SMBs should add an AI annex to their existing plan covering inventory, owners, disable paths, logging and notification triggers.

Is a company liable for what its chatbot says?

In Moffatt v. Air Canada, a Canadian tribunal held the airline responsible for its chatbot's inaccurate information. Outcomes depend on jurisdiction and facts, but assume customer-facing AI output is treated as your own statement.

Is entering PHI into an AI tool a HIPAA breach?

It can be. If the tool is not covered by a BAA and is not an approved system for that PHI, the disclosure may be impermissible. Run your breach risk assessment, document the outcome and follow your notification obligations.

Disclaimer (legal note)

This article is for general information only and is not legal, regulatory or professional advice. Requirements vary by framework, industry and jurisdiction. Consult qualified advisors for your specific obligations.