AI Is Already Processing Claims — But It Can't Legally Decide Them

AI is already doing the heavy lifting in claims processing — and it's been doing so at scale. A NAIC survey of 93 insurers found that 84% use AI or machine learning in claims-related processes, and adoption spans every major line: 92% of health insurers, 88% of auto insurers, 70% of home insurers, and 58% of life insurers report current or planned AI usage.

But there's a legal line AI cannot cross: it cannot be the sole decision-maker on a claim denial. The CMS Final Rule requires human review before any adverse determination, and state-level safeguards impose the same duty.

By late 2025, 23 states plus Washington, D.C. had adopted the NAIC AI Model Bulletin, which pushes insurers toward governance, documentation, and audit procedures. The pattern is consistent: AI can recommend, flag, and accelerate — but a human must make the final call.

Adding to the complexity, regulation is a patchwork. ERISA preempts state insurance laws for most self-funded employer plans, meaning many workers' claims fall outside state AI protections entirely — and what "full and fair" review means for AI tools has yet to be interpreted through guidance or updated regulation. Meanwhile, the federal AI Framework pushes toward preemption of state AI laws while states simultaneously expand AI-specific insurance statutes.

That creates a compliance reality for every insurer, TPA, and claims operation:

• Position AI as decision support, not decision-maker — never let it be the sole authority on a denial
• Document per-decision human review, not sample audits — CMS and plaintiff lawyers will demand the record
• Track override rates — when AI decisions are overridden less than 2% of the time, regulators question whether human review is real
• Test models for bias — nearly one-third of health insurers don't regularly test AI for bias or discrimination

The stakes are not theoretical. Class actions against UnitedHealthcare, Cigna, Humana, and State Farm are progressing through federal courts, and the Colossus litigation of the 2000s established that opaque claims-automation systems draw regulatory and legal scrutiny. In Hensley v. Computer Sciences Corporation, plaintiffs alleged Colossus was calibrated to reduce claim payouts by up to 15%, and the settlement required transparency modifications and independent adjuster judgment. Every wrongful AI denial is a discrete plaintiff with a discrete cause of action.

The real challenge for claims teams is speed without legal exposure. AI delivers the speed; the exposure comes from skipping the human layer. That's where done-for-you AI agents fit. Agents by AIQ designs, builds, and runs AI agents that handle the repetitive busywork around claims workflows — answering calls, following up with claimants, automating document collection — while keeping human oversight intact. The model is a tool; human judgment is the legal floor.

The Litigation and Regulatory Landscape You're Operating In

Insurance companies are increasingly leveraging AI to streamline claims processing, but the legal landscape is fraught with challenges. Regulators and courts are scrutinizing AI's role in claims adjudication more closely than ever. The enforcement reality today involves a complex web of class actions, regulatory mandates, and legal precedents that define what AI can and cannot do in insurance claims processing.

Class actions against major insurers are setting precedents. Cases like Estate of Lokken v. UnitedHealthcare and Kisting-Leung v. Cigna Corp. highlight the acute litigation risk. According to recent legal analyses, every adverse AI decision is a potential cause of action. Insurers face significant legal exposure if their AI systems deny claims without proper human review. For instance, Kelly v. State Farm Fire & Casualty Co. alleges that AI algorithms disproportionately impacted Black and non-white policyholders, emphasizing the need for fairness and transparency.

Regulatory requirements also demand a robust human-in-the-loop framework. The CMS Final Rule mandates that AI tools must be applied only after considering individual patient circumstances, with human review required before adverse determinations. This rule, effective January 1, 2024, ensures that AI does not make sole decisions in claims processing. Additionally, the NAIC's AI Model Bulletin, adopted by 23 states plus Washington, D.C., requires insurers to establish governance, documentation, and audit procedures. However, enforcement relies on existing unfair trade practice and unfair claim settlement practice statutes, creating a patchwork of compliance requirements.

The regulatory landscape is further complicated by federal preemption and ERISA gaps. The Trump administration's AI Framework promotes limited federal restrictions and industry-led standards, while related actions seek to preempt state AI regulations. However, states are expanding AI-specific insurance statutes, creating compliance uncertainty. ERISA preemption creates a protection gap for most workers with employer-sponsored insurance, who are in self-funded plans exempt from state laws. The legal landscape is dynamic, with future congressional action likely to shape federal frameworks and their preemption of state consumer protections.

Navigating these challenges requires a thorough understanding of the regulatory and legal environment. Legal experts emphasize the importance of compliance and transparency. For instance, the Colossus litigation of the 2000s established that opaque claims-automation systems draw regulatory scrutiny. Modern AI faces similar transparency, bias, and oversight tests. According to historical precedent, insurers must ensure that AI tools are transparent and that human oversight is indispensable.

For small and mid-size businesses, including those in the insurance sector, compliance with these regulations is crucial. Market trends show that AI adoption in claims processing is widespread, with 84% of surveyed insurers using AI or machine learning. However, the legal landscape is evolving, and insurers must stay ahead of regulatory changes. Partnering with a service like Agents by AIQ can help businesses navigate these complexities, ensuring that their AI tools are compliant and effective. The team behind the AI Business Sites platform can design, build, connect, and operate done-for-you AI agents tailored to specific business needs, handling calls, follow-ups, and workflow automation. This shifts the burden of compliance and risk management from the business to the experts, allowing owner-operators to focus on their core operations.

Human-in-the-Loop: The Compliance Standard That Keeps Coming Up

If an AI denies a claim and no one meaningfully reviews it, regulators and courts increasingly treat that as a legal defect — not a technicality. Across federal rules, state statutes, and pending class actions, one requirement keeps surfacing: a human must genuinely review adverse determinations before they become final.

The convergence is striking. CMS's Final Rule requires that algorithmic tools be applied only after considering individual patient circumstances, with human review required before any adverse determination. State safeguards echo the same principle — AI cannot be the sole decision-maker on a denial, according to the Medicare Rights Center. And by late 2025, 23 states plus Washington, D.C. had adopted the NAIC AI Model Bulletin, which requires governance, documentation, and audit procedures.

The hard part is proving the review was real. Regulators have a concrete benchmark: compliance guidance warns that if AI decisions are overridden less than 2% of the time, regulators question whether human review is genuine. Law professor Jennifer Oliva has observed that where human-in-the-loop rules exist, humans appear to be "just sort of approving what the AI is deciding" (PBS NewsHour).

That skepticism is why documentation standards are tightening. The floor for compliance now looks like:

• A per-decision audit record for every adverse determination — not a sample
• Logged model version, input features, score, reviewing human, and timestamp
• Tracked override rates, with a 2% floor treated as a red flag

Sample-based audits won't survive ERISA discovery. Plaintiffs in cases like Estate of Lokken v. UnitedHealthcare and Kisting-Leung v. Cigna Corp. — which a federal court allowed to proceed in March 2025 (Fenwick) — can demand records for individual denials, and a gap in the log becomes its own evidence. As one compliance analysis puts it, a per-decision human review record is the floor, not the ceiling.

The historical precedent reinforces the point. Post-Colossus settlements mandated that adjusters retain independent judgment and document deviations from system recommendations, notes Houston Harbaugh attorney Christopher M. Jacobs, who frames the duty plainly: automation "merely reframes" the obligation of fair claim handling. For teams building claims workflows — including those Agents by AIQ supports with automated intake and follow-up — the lesson is architectural: keep the human decision point inside the workflow, with the evidence trail built in, rather than retrofitted after the first subpoena arrives.

Bias Testing, Transparency, and Vendor Accountability

Most insurers have a compliance plan for their AI models. Far fewer can prove it — and that gap is where regulators and plaintiffs' lawyers are now focusing their attention.

The headline statistic is sobering: nearly one-third of health insurers do not regularly test their AI models for bias or discrimination, according to industry tracking of AI insurance regulation. Bias testing isn't an optional best practice anymore. A widely cited study found that algorithms using healthcare costs as a proxy for need systematically underestimated the health needs of Black patients — exactly the kind of flaw that bias testing is designed to catch before it shapes a denial.

The training data itself deserves equal scrutiny. Claims AI trained only on historical claims-paid history will reproduce historical denial patterns, baked-in inequities and all. The KFF analysis of AI in claims review makes clear that auditors and regulators expect insurers to demonstrate diligence over what their models learned and from whom.

Transparency is the second pillar. Regulators are actively debating whether insurers must disclose AI use to consumers when it significantly affects claims decisions, and the principle is simple: if AI influences people's finances and well-being, consumers deserve to know how those decisions are made. History reinforces the point — the Colossus litigation of the 2000s showed that opaque claims-automation systems draw regulatory and legal scrutiny, with the Hensley settlement requiring changes to transparency and marketing practices.

Vendor accountability is the third, and it's where forward planning pays off. A NAIC model law governing third-party AI vendors is anticipated in 2026, and insurers that wait for it will be scrambling. Practical contractual controls to implement now include:

• Audit rights allowing the insurer to inspect vendor model documentation, training data provenance, and performance reports
• Clear allocation of responsibility for bias testing, monitoring, and remediation between insurer and vendor
• Notification obligations when a vendor retrains, updates, or materially changes a model used in claims decisions
• Documentation support so the insurer can produce per-decision human review records if ERISA discovery demands them

As Dan Silverboard of Holland & Knight puts it, there is "this increased compliance burden on health insurers to demonstrate that they've conducted their diligence on these programs." Demonstration is the operative word — regulators want evidence, not assurances.

For small agencies and brokerages adopting AI tools, the same logic applies at a smaller scale. When we build agents at Agents by AIQ, the client owns everything and the tools integrate with systems the business already runs — which also means the audit trail stays with the business, not locked inside a vendor's black box. That ownership matters when a regulator, an auditor, or a plaintiff's attorney asks you to show your work.

How to Deploy AI Claims Support the Right Way

Getting AI into claims workflows isn't the hard part anymore — doing it without becoming the next class-action defendant is. The legal landscape is fragmented and fast-moving, but the safe path is surprisingly clear: AI supports decisions, humans make them, and everything gets documented.

Start with decision-support, not decision-making. Federal CMS rules require human review before any adverse determination, and state safeguards echo the same principle — AI cannot be the sole decision-maker on a denial, according to Medicare Rights Center analysis. Courts are watching too: pending class actions against major insurers allege denials issued without proper individual review, per Fenwick's regulatory tracking. Treat AI as a tool that drafts, flags, and summarizes — never as the final arbiter.

Build audit logging and human review workflows from day one. Compliance guidance is blunt: "Document human-in-the-loop for every adverse decision — sample-based audits won't satisfy CMS or plaintiff lawyers," as one claims adjudication analysis puts it. Log the model version, inputs, output, reviewing human, and timestamp for every decision. Then track your override rate — if humans disagree with the AI less than 2% of the time, regulators will question whether the review is real.

Plan for the multi-state patchwork. By late 2025, 23 states plus Washington, D.C. had adopted the NAIC AI Model Bulletin, each with variations, and a third-party vendor model law is anticipated in 2026. Colorado's AI Act treats claims as "consequential decisions" requiring impact assessments and human appeal routes, with comparable measures in Arizona, Texas, Washington, and elsewhere. Design for the strictest standard, not the average one.

Your implementation checklist should cover:

• Per-decision audit logs with named human reviewer and timestamp
• Documented override processes — and an override rate above the 2% credibility threshold
• Regular bias testing (nearly one-third of health insurers skip this, per Fenwick's reporting)
• Consumer disclosure of AI use where it materially affects outcomes
• Vendor contracts that address the coming NAIC third-party rules

Here's the practical reality for smaller firms: the highest-value, lowest-risk AI work in claims is administrative, not adjudicative. As Houston Harbaugh's analysis of the Colossus litigation notes, "automation does not absolve insurers of the duty of fair claim handling; it merely reframes it." Scheduling, intake, document collection, status calls, and follow-ups carry none of the adverse-determination risk.

That's exactly where done-for-you agents fit. Agents by AIQ builds and operates AI receptionists, follow-up agents, and workflow automation that handle the busywork around claims — intake calls, document requests, status updates — while every consequential decision stays with a licensed human. If missed calls and manual admin work are slowing your claims operation, book a call to scope an agent for your business.