At a Glance: Today's issue dives into the FDA's new unspoken rule for AI submissions: explainability is now non-negotiable. We also break down the engineering challenges behind SKIA's newly cleared AR surgical guidance system. This week is all about proving not just that your device works, but how.

DIGITAL HEALTH

The Black Box Breaker: Why 99% Accuracy Isn't Enough for the FDA

Your AI model has 99% sensitivity, solid validation, and a great data set. And yet, the FDA just sent you a deficiency letter. It's not because your numbers are wrong. It's because you can't explain them in a way a clinician can trust and act on. This is the new reality for AI/ML device submissions.

What the Public Information Reports

The dialogue in the industry reveals a clear shift in the FDA's focus. Reviewers are moving beyond asking 'How precise is the AI?' to a much tougher question: 'Can you describe what this AI does and justify a clinician’s ability to rely on its results?' High performance is now just the table stakes; it's no longer the entire game.

This isn't based on a single new guidance document titled 'Explainability.' Instead, as the analysis points out, it's the combined weight of several key documents. The Good Machine Learning Practice (GMLP) principles, the Predetermined Change Control Plan (PCCP) guidance, and IMDRF N41 on clinical evaluation together create an effective explainability standard. The message is clear: transparency is a foundational requirement, not a nice to have.

What Could Cause This Type of Failure

Submissions are failing because development teams often treat explainability as a documentation task to be handled at the end of a project, rather than a core design principle from the start. A team might present a beautiful model architecture and impressive metrics but offer only a single sentence about using SHAP values for explainability. That's no longer enough.

The core of the issue is a failure to bridge the gap between data science and clinical practice. A model's output, without context, is just a number. The FDA needs to see a documented chain of logic connecting the input data to the final result, an analysis of how the model performs on different patient subgroups, and a clear understanding of its limitations.

This gap often leads to a lengthy and expensive remediation process, sometimes lasting 6 to 12 months, as teams scramble to generate the evidence the FDA is asking for. The problem isn't the model's performance; it's the lack of a documented, transparent, and clinically relevant narrative explaining how that performance is achieved and where it might falter.

Regulatory & Standards Context

The shift to the Quality Management System Regulation (QMSR), which aligns 21 CFR Part 820 with ISO 13485, adds another layer here. This has real, practical implications. Your AI's traceability and explainability records can't just live in a GitHub repo or a technical whitepaper anymore. They must be integrated into your formal QMS.

Specifically, things like data origin documentation, algorithm version control, a clear description of your training and validation datasets (including demographics), and subgroup performance analyses are now quality records. They need to be managed with the same rigor as any other design history file component. If an auditor asks you to trace an output back to the data that generated it, you need a QMS-compliant paper trail.

Design Playbook - Learning from the Event

Check: Can you write an Algorithm Description Document in plain English? This document should explain your model's function, training, and validation process to a clinician, not just a data scientist. It must detail the demographic makeup of your training data, how you controlled for quality, and how known biases were addressed. Simply referencing a published paper won't cut it.

Audit: Is your model traceability part of your official QMS? You need to prove that your data lineage, algorithm version history, and validation records are under formal design controls. This isn't just a technical file. Ensure these records integrate with your existing QMS to meet the new QMSR expectations and survive an audit.

Check: Have you defined and validated your human oversight model? What is the exact workflow when a clinician disagrees with the AI's output? How is the AI's performance monitored in the field? GMLP principles require that clinical experts are involved during the AI's development, not just as reviewers at the end. You need to document this interaction model.

Audit: Does your FMEA address risks from misunderstood or misinterpreted AI outputs? The risk isn't just that the AI is wrong; it's that a correct output is used incorrectly by a clinician. Your risk analysis must include failure modes related to comprehensibility. This means running human factors studies on how users interpret the UI, confidence scores, and displayed limitations.

DIGITAL HEALTH

AR in the OR: Engineering SKIA’s Newly Cleared Surgical Guidance System

Let's talk about putting augmented reality onto a moving, breathing patient. SKIA just got 510(k) clearance for their SKIA HEAD, a tablet based AR surgical guidance system. And while projecting a 3D model is cool, the real engineering nightmare is keeping that model perfectly aligned with the patient's anatomy in the real world.

What the Public Information Reports

According to the announcement, SKIA HEAD is a software platform that runs on a tablet. It uses medical grade Structure Sensors, which are popular structured light 3D scanners, to map the patient and the operating room environment. The system takes preoperative imaging, like a CT or MRI scan, and turns it into a 3D model that gets overlaid directly onto the patient's body in real time.

The goal is to give surgeons 'x-ray vision,' helping them see internal anatomy without looking away at a separate monitor. The partnership with Structure suggests a focus on making this technology more accessible, moving away from large, fixed navigation systems and toward a more portable solution suitable for smaller hospitals or outpatient centers.

What Could Cause This Type of Failure

The single biggest challenge for any surgical AR system is registration and tracking. Registration is the initial process of aligning the virtual 3D model to the physical patient. If this is off by even a few millimeters, the entire system is not just useless, it's dangerous. This step often requires identifying specific anatomical landmarks on both the patient and the 3D model to create a perfect match.

Once registered, the system must continuously track the patient, the camera (the tablet), and any surgical instruments to maintain that alignment. A failure here could be caused by sensor occlusion, where a surgeon's hand or an instrument blocks the sensor's view. Another common problem is drift, where small tracking errors accumulate over time, causing the virtual overlay to slowly slide away from the real anatomy. Without a robust system to detect and correct for these issues, the surgeon could be misled.

Regulatory & Standards Context

For a device like this, software is the core of the system, so IEC 62304, 'Medical device software – Software life cycle processes,' is non negotiable. Your entire software development, verification, and validation process needs to be documented under this standard. But the bigger concern is risk management under ISO 14971.

The primary risk is displaying inaccurate information that misleads a surgeon. Your risk analysis must meticulously break down every possible cause of misalignment: initial registration error, tracking drift, latency between movement and display update, and software bugs. The FDA's guidance on Computer Assisted Surgery (CAS) systems would also be highly relevant here, as it sets expectations for validating the accuracy and reliability of such navigation tools.

Design Playbook - Learning from the Event

Check: What is your workflow for confirming initial patient registration? This is a critical use-case that blends human factors and system design. You need a simple, foolproof way for the surgeon to verify that the virtual overlay is perfectly aligned before making any clinical decisions. This could involve touching several known anatomical points and confirming they match on screen.

Audit: How does your system handle tracking loss or sensor occlusion? What happens when the sensor's view is temporarily blocked? A silent failure is the worst possible outcome. Your system must immediately and clearly indicate the loss of tracking, perhaps by making the overlay disappear or turning it bright red. This failure mode and its mitigation must be a key line item in your FMEA.

Check: Is your end to end system latency well under 100 milliseconds? Any noticeable lag between patient movement and the AR overlay update will destroy the illusion of alignment and can make the system feel unusable or unsafe. You need to budget this latency across the entire pipeline: sensor capture, data processing, 3D model rendering, and the tablet's display refresh.

Audit: What is your validation strategy for proving submillimeter accuracy? You can't just say the system is accurate; you have to prove it with data. This requires a formal validation plan using phantom models with known dimensions and a high precision ground truth measurement system, like an optical CMM. You need to demonstrate, with statistical confidence, that your system's accuracy meets its design specifications under a range of simulated use conditions.