Introduction
According to a 2025 JAMA Network Open survey of 2,174 nonfederal U.S. hospitals, 52.2% already use machine-learning-based predictive AI integrated with their EHRs — yet 32.6% of those hospitals report no local evaluation practices for the AI they've deployed. That gap between adoption and governance is where regulatory exposure lives.
Healthcare AI doesn't operate in a compliance vacuum. It inherits every existing regulatory obligation and triggers new ones. Three distinct frameworks apply in non-overlapping ways:
- HIPAA governs how AI handles patient data
- FDA determines whether AI qualifies as a medical device
- OSHA addresses how AI affects the safety of the people delivering care
Each framework has a distinct jurisdictional scope, its own enforcement mechanism, and technical requirements that don't overlap. An organization that conflates them — or treats compliance as a single unified checklist — typically ends up with gaps in all three.
This guide breaks down what each framework requires, where organizations are creating compliance gaps right now, and how to build governance that actually scales.
TL;DR
- HIPAA applies to any AI system that accesses, processes, or transmits ePHI — no exceptions
- FDA classification turns on whether clinicians can meaningfully review an AI tool's reasoning or primarily rely on its output
- OSHA's General Duty Clause extends to AI-driven cognitive and physical hazards for healthcare workers, including alert fatigue
- The four most common compliance gaps are overbroad PHI access, missing BAAs, insufficient audit trails, and CDS misclassification
- Governance built into the architecture outperforms periodic compliance reviews
HIPAA Compliance Requirements for Healthcare AI
HIPAA's Privacy Rule, Security Rule, and Minimum Necessary standard under 45 CFR §164.502(b) apply without modification to any AI system that accesses ePHI. HHS's general applicability guidance makes clear that HIPAA obligations extend to all covered entities and business associates handling PHI — AI systems are not exempt, and enforcement reflects that.
A recent example: in March 2025, HHS OCR settled with Health Fitness Corporation for $227,816 after a server misconfiguration allowed ePHI to become discoverable by web crawlers. The root cause was a software configuration failure — the same category of technical gap that AI deployments create when access controls aren't enforced at the system level.
Business Associate Agreements as the Deployment Gate
Any AI vendor that creates, receives, maintains, or transmits PHI on a covered entity's behalf must execute a HIPAA-compliant Business Associate Agreement before any PHI flows to their systems. This is a legal prerequisite, not a paperwork formality.
Three rules govern BAA enforcement in practice:
- BAA verification must function as a hard deployment gate, not a post-launch administrative task
- Vendors unwilling to sign BAAs cannot legally operate in PHI-touching workflows, regardless of how capable their technology is
- The covered entity bears full liability for any PHI breach involving a vendor without an executed BAA
Minimum Necessary Access and Technical Enforcement
HIPAA's Minimum Necessary standard requires AI agents to be technically restricted to only the PHI their specific function requires. Policy statements don't satisfy this requirement; the restriction must be enforced at the system level.
That system-level enforcement has direct implications for how AI agents are scoped. Broad EHR access violates Minimum Necessary even when overall system permissions appear correct — an AI agent handling appointment scheduling has no legitimate need to access clinical notes or medication histories.
Attribute-Based Access Control (ABAC) is the technical mechanism that satisfies this requirement for AI agents, restricting access dynamically based on the agent's function, the data's sensitivity, and the context of the request.
Cybic builds access controls — including RBAC and encrypted data protection — directly into the AI architecture. When an auditor asks how minimum necessary access is enforced, the answer is in the system design, not a policy document.
Audit Controls and Encryption Requirements
45 CFR §164.312(b) requires hardware, software, and procedural mechanisms that record and examine activity in systems containing or using ePHI. For AI agents, that means operation-level logs capturing:
- Which agent accessed which PHI
- What action was performed
- Who authorized the workflow
- When the access occurred
A log showing an AI tool was "used" is not compliant — it's a standalone HIPAA violation.
Additional technical requirements:
- ePHI must be encrypted in transit and at rest
- AI agents must be authenticated with an identity linked to a human authorizer
- Breach notification obligations under 45 CFR §§164.400–414 apply whether a breach was caused by a human or an AI system
FDA Regulatory Framework for Healthcare AI
Under Section 201(h) of the FD&C Act, software intended for use in the diagnosis, cure, mitigation, treatment, or prevention of disease qualifies as Software as a Medical Device (SaMD) and is subject to FDA regulation. Administrative AI tools — scheduling, billing, prior authorization routing — generally fall outside FDA jurisdiction. But that boundary shifts the moment a tool's intended use crosses into clinical territory, and misclassification creates real enforcement exposure.
The FDA currently lists 499 AI-enabled medical devices authorized for marketing in the U.S. That number reflects the scale of regulated AI already operating in healthcare — and the baseline expectation that clinical AI tools go through appropriate review.
The CDS Line That Determines Regulatory Burden
The FDA's 2022 Clinical Decision Support Software guidance establishes a four-criteria test for non-device CDS. The most consequential criterion: whether clinicians can independently review the basis for a recommendation and reach their own conclusions.
If yes → non-device CDS, lower regulatory burden. If no → device territory, requiring premarket authorization.
The practical problem is that many organizations label AI tools as "decision support" when those tools are functionally routing patients, recommending treatments, or generating outputs that clinicians ratify rather than genuinely review. That mislabeling doesn't change what the tool actually does. It surfaces during post-deployment audits, when fixing classification errors costs significantly more than getting it right upfront.
Ask these three questions to identify potential misclassification:
- Does the AI provide directives rather than recommendations?
- Can clinicians meaningfully review the reasoning behind the output?
- Do clinicians primarily rely on the AI's output rather than independently verifying it?
If the answers are yes, no, and yes — that tool warrants FDA classification review.
Premarket Pathways and Predetermined Change Control Plans
Three pathways exist for AI-enabled devices, based on risk classification:
| Pathway | Risk Class | Use Case |
|---|---|---|
| 510(k) | Class I/II | Most AI-enabled devices; substantial equivalence to predicate |
| De Novo | Novel low-to-moderate risk | New device types without a predicate |
| PMA | Class III | High-risk devices requiring clinical data |
510(k) is the dominant pathway for cleared AI devices.
Cleared devices don't stay static. AI models drift post-deployment, and updates can trigger new FDA submissions — unless a Predetermined Change Control Plan (PCCP) is in place. The FDA's August 2025 PCCP guidance specifies that a valid plan must include:
- A Description of Modifications: what changes are planned
- A Modification Protocol: how changes will be validated, with acceptance criteria
- An Impact Assessment: risks, benefits, and mitigations
PCCPs allow manufacturers to implement model updates within a pre-approved framework without restarting the submission process.
Postmarket Obligations
Cleared status doesn't end the compliance work. Manufacturers must monitor real-world performance, document corrections, and report adverse events under the MDR program (21 CFR Part 803).
FDA's September 2023 cybersecurity guidance adds further requirements for AI-enabled medical devices:
- A software bill of materials (SBOM)
- Secure-by-design architecture
- Ongoing vulnerability management programs
OSHA and AI in Healthcare: The Often-Overlooked Framework
OSHA's General Duty Clause (OSH Act Section 5(a)(1)) requires employers to provide a workplace free from recognized hazards likely to cause serious harm. AI systems that affect the physical or cognitive work environment of healthcare workers fall within that scope.
One JAMA Internal Medicine study found physicians overrode 91.2% of drug-allergy alerts and 89.4% of high-severity drug-interaction alerts. A JAMIA study found override rates for medication-related CDS alerts averaging around 90%.
When AI systems generate a constant stream of alerts that clinicians override by default, that's a documented cognitive hazard — and it falls squarely within OSHA's jurisdiction.
That hazard framing leads to an important scope boundary: OSHA covers workers, not patients. A clinical AI tool that harms a patient triggers FDA and HIPAA analysis. The same tool's effect on the clinicians operating it — fatigue, distraction, unsafe workload — is an OSHA question. These obligations run in parallel.
Practical OSHA compliance steps for AI deployment:
- Conduct hazard assessments that explicitly include AI system impacts on workers
- Document risk controls for identified AI-related hazards
- Train clinical staff on AI tool limitations and safe interaction procedures
- Implement machine guarding for robotic systems per 29 CFR 1910.212
- Follow lockout/tagout procedures per 29 CFR 1910.147 during AI system servicing
- Maintain incident records for any workplace harm involving AI systems
The Most Common AI Compliance Gaps Healthcare Organizations Are Creating Right Now
Gap 1: AI Agents with Overbroad PHI Access
The most pervasive HIPAA violation is AI systems with broad access to EHRs or clinical data warehouses, without operation-level controls restricting each agent to the minimum PHI its specific function requires.
Policy statements don't fix this. If an AI agent technically can access a full patient record when its function only requires appointment data, the minimum necessary standard is violated, regardless of what the policy document says. Technical enforcement via ABAC is mandatory.
Gap 2: Missing BAAs and Undocumented Vendor Relationships
Many AI tools embedded in EHR platforms, scheduling systems, and administrative workflows process PHI without executed BAAs. This happens because AI vendor discovery (identifying every tool that touches PHI) rarely happens before deployment.
Without a BAA, the covered entity bears full liability for any PHI breach involving that vendor. Vendor discovery must precede governance work. Key steps before any AI deployment include:
- Inventorying every tool that processes or accesses PHI
- Confirming executed BAAs with each vendor
- Documenting data flows for ongoing audit readiness
Gap 3: Absent or Insufficient Audit Trails
Most organizations have some logging for AI tools. Very few have the operation-level specificity HIPAA requires.
| Log Type | What It Captures | Compliant? |
|---|---|---|
| Session log | "AI tool was used at 2:14 PM" | ❌ No |
| Operation-level log | Agent ID + PHI fields accessed + action performed + human authorizer + timestamp | ✅ Yes |
The gap between these two isn't a minor documentation issue: it's a standalone HIPAA violation. Compliant audit logs need to be tamper-evident and generated automatically as a byproduct of operations, not reconstructed after the fact.
Cybic builds auditability and traceability directly into AI workflow architecture, so compliant logging is a structural output of operations rather than a compliance layer retrofitted afterward.
Gap 4: CDS Misclassification Hiding FDA Exposure
Healthcare organizations consistently mislabel AI tools as non-device CDS to avoid FDA review. The misclassification is rarely intentional; it usually reflects an incomplete understanding of the FDA's four-criteria test.
Remediation after deployment is expensive. Identifying misclassified tools before deployment is a classification review. The difference in cost and operational disruption is substantial. Run the FDA's four-criteria test against every AI tool before deployment, not after a compliance incident forces the issue.
Building a Healthcare AI Governance Framework That Scales
Start with an AI Inventory
Organizations underestimate the AI already operating in their facilities. Before any governance work begins, document every AI tool in use. For each tool, capture:
- Name, function, and deployment context
- Vendor name and contract status
- PHI access scope (what data can the tool access?)
- FDA classification status
- BAA execution status
- Current audit trail capability
Tools with PHI access but no executed BAA, tools with overbroad access, and tools potentially misclassified as non-device CDS should be flagged for immediate remediation.
Shift from Periodic Review to Continuous Enforcement
Annual risk assessments can't keep pace with AI systems that update continuously. Static governance creates a compliance posture that's accurate once a year and stale the other 364 days.
Governance for AI must include:
- Real-time policy enforcement across models and agents
- Automated compliance monitoring that generates evidence continuously
- Audit trails produced as an operational byproduct — not reconstructed for audits
Architectural decisions drive this distinction. When governance, RBAC, encrypted data protection, and auditability are built into AI systems at the engineering level — as Cybic does for healthcare clients — compliance becomes an operational property of the system, not a layer added afterward. Checklists appended post-deployment reflect a past configuration. Governance embedded in the architecture reflects what the system does right now.
Establish Cross-Functional Governance and Future-Proof Vendor Contracts
Governance structure needs executive ownership, cross-functional representation (clinical, legal, IT, compliance), and clear decision-making authority.
AI vendor contracts should require:
- Executed BAA before any PHI access
- Local validation evidence for your patient population
- Bias assessment documentation
- Audit rights
- Indemnification for AI system failures
The Joint Commission's Responsible Use of AI in Healthcare (RUAIH) guidance provides a concrete alignment target. Organizations that build governance structures around RUAIH now are positioned for accreditation reviews as these standards move from advisory to required.
Frequently Asked Questions
What are the 5 main HIPAA rules?
The five rules are the Privacy Rule (governing PHI use and disclosure), the Security Rule (technical and administrative safeguards for ePHI), the Breach Notification Rule (reporting requirements for PHI breaches), the Omnibus Rule (extending obligations to business associates), and the Enforcement Rule (penalties for violations). All five apply equally to AI systems handling ePHI.
What is the FDA guidance on AI?
The FDA's primary AI guidance includes the 2022 Clinical Decision Support Software guidance (which distinguishes non-device CDS from SaMD) and the AI/ML-Based SaMD Action Plan. The FDA also issued final cybersecurity guidance in September 2023 and maintains a public list of AI-enabled medical devices.
Can AI tools be HIPAA compliant?
Yes, when the right safeguards are in place: access controls, minimum necessary enforcement, operation-level audit trails, encryption, and executed BAAs with AI vendors. HIPAA compliance is an architecture decision, not just a policy one.
What is the difference between SaMD and CDS under FDA regulations?
SaMD is software intended to diagnose, treat, or prevent disease, and requires FDA premarket review. CDS that allows clinicians to independently review the basis for recommendations and reach their own conclusions is typically not classified as a medical device. The determining question is whether the clinician can meaningfully verify the AI's reasoning before acting on it.
Does OSHA apply to AI systems in healthcare?
Yes, under the General Duty Clause. AI systems that affect the physical or cognitive safety of healthcare workers (including robotic tools, automated dispensing systems, and AI that causes alert fatigue) fall within OSHA's scope. OSHA protects workers, not patients, and operates separately from HIPAA and FDA obligations.
What are the biggest AI compliance risks healthcare organizations face today?
Four gaps appear most often in practice:
- AI agents with overbroad PHI access and no minimum necessary controls
- Missing BAAs with AI vendors
- Absent operation-level audit trails that fail HIPAA's audit controls standard
- AI tools misclassified as non-device CDS when they function as autonomous decision-makers requiring FDA review
