Cybersecurity

Shadow AI in Medical Device Companies: Find It Before Your Next Audit

Your quality system is a map of how your company is supposed to work. Every controlled process, every design output, every record has a place in it. Shadow AI is the part of how your company actually works now that the map can't see.

An engineer pastes a failing test log into free ChatGPT to get unstuck. A regulatory writer asks an AI assistant to tighten the language in a 510(k) response. A quality manager drafts a CAPA with a chatbot. None of it is malicious. All of it is invisible to the system you'll be inspected against. And the first principle to sit with is the uncomfortable one: you can't audit what you don't know about.

This essay is about the AI your team uses to build the device and do your work — the tools that touch your design files, your quality records, and your development environment. It is not about AI inside the device, the SaMD and algorithm questions the FDA governs through its own guidance. That is a different problem for a different essay. The board-level case for why any of this matters — the tension between moving fast, staying defensible, and protecting your IP — is made in The AI Governance Gap. This is the operator's follow-on: how to find the shadow AI already in your company, and what to do the day you find something you wish you hadn't.

Why it's worse in a device company

In most companies, shadow AI is an IP and security problem. In a medical device company, it is also a records problem, and that changes the stakes.

Consider where the exposure actually lands:

• An engineer pastes a design output into a consumer tool. That output belongs in your design history file — now the design-and-development records QMSR keeps under ISO 13485. Now a controlled design record has been processed by a tool you never validated, hosted by a vendor you never qualified, under terms that may let it train on the files and text your employees provide.
• AI gets silently switched on inside your eQMS, PLM, or CAD. The vendor pushes an update; a feature appears; your proprietary device geometry starts flowing somewhere new. You didn't add a supplier — one added itself.
• AI drafts a CAPA, a complaint investigation, or a submission response. These are quality records and regulatory representations. If the tool invents a number or softens a fact, the integrity problem is now inside a record an investigator can pull.
• A team member uses AI on a personal phone because the approved-tools list didn't include what they needed. That usage is not just ungoverned — it is structurally uninspectable.

The through-line is that a device company already operates under a regime that assumes its environment is controlled. IEC 81001-5-1, the secure-development standard the FDA points to, requires that the environment you build in — workstations, servers, your source control — be secured before you write a line of code. As Chris Gates put it in Introduction to Medical Device Cybersecurity, that is the one place the infrastructure-security world and new-product development actually overlap: is the machine you're developing on secure? An AI tool with quiet access to that environment is a new, unmanaged actor inside a boundary you were supposed to have drawn.

And the regime itself changed this year. The Quality Management System Regulation took effect on February 2, 2026, folding ISO 13485:2016 into 21 CFR Part 820 and bringing an updated inspection process with it. The bar for what counts as a controlled process and a trustworthy record did not get lower. Data integrity under Part 11 did not get more forgiving. Whatever your team is doing with AI, you are now doing it under a quality system regulation that expects you to know.

The two kinds of shadow AI

There are two versions of this problem, and they need different responses.

The first is the AI your team reaches for. People adopt tools because the sanctioned path is too slow or doesn't exist. There is an apocryphal story about Stanford building its campus with no sidewalks for the first year — they watched where people wore paths in the grass, and put the sidewalks there. The lesson for governance is the same. Your engineers are already showing you where the demand is. The mistake is pre-building the sidewalks: publishing a rigid approved-tools list, watching people go off-roading through the quad anyway, and calling the resulting mess a compliance problem instead of a design signal.

The second is the AI you didn't choose at all. Jira Cloud added AI without asking anyone. Many major CAD programs are adding it. Adobe, your office suite, your accounting software, quite possibly your eQMS — the vendor ships an update, you install it, and a feature is now doing something with your data inside a license you already pay for. Some of it is harmless and useful: plain-English search that translates to a query, duplicate-ticket detection. Some of it trains the vendor's model on what you feed it.

That second kind is the one nobody has a policy for, and in a device company it has a sharp edge. Picture the version of a story told in a general-business setting: someone designs a manifold for a rocket, labels it "manifold," the CAD tool trains on the design-plus-label, and a competitor later types "I need a manifold" and gets a starting point that looks suspiciously familiar. Now swap the part. A catheter tip geometry. A pump manifold. An implant housing, labeled by function, sitting in a CAD tool that quietly added a training-enabled AI feature. The same risk pattern applies, and the thing being exfiltrated is the thing you were going to patent.

There is no clean setting to switch this off across every tool you own, and you can't realistically read every EULA every time it changes. What you can do is make it a rule that someone reviews how your data is used — and whether you can opt out of training — every time a new vendor or a new terms-of-service enters the building. The honest test for each new AI feature is the one an attendee at one of these sessions offered: is this actually making the product better, or is it just the vendor keeping up with the Joneses?

Discover, don't ban

The instinct, once a CEO sees the exposure, is to ban. Restrict AI to an approved list and forbid the rest. It feels like control. It produces the opposite.

When you ban the tools people rely on, you don't stop the usage — you move it onto personal phones and unmanaged accounts, where you have no terms, no auditing, and no visibility at all. You have taken a problem you could have seen and made it one you can't. For a company that will one day stand in front of an inspector and account for its processes, that is exactly the wrong trade.

The exception is the active fire. If discovery turns up a specific high-risk exposure happening right now — PHI or an unreleased invention flowing into a consumer tool today — block that one immediately and inventory the rest in parallel. Containment of a known exposure doesn't wait for the full census.

The better sequence is to get ahead of it:

1. Find out who's using AI, for what, and with which data. Not to punish — to understand. This is the step that turns an uninspectable gap into an inventory.
2. Pay for the business or enterprise tier of the tools people actually want. Claude Team or Enterprise, ChatGPT Enterprise, and their peers do not train on your data by default and come with a degree of built-in auditing. If your workflows touch protected health information — identifiable clinical data, anything you handle as or on behalf of a covered entity — that is where you scope a Business Associate Agreement and confirm the safeguards, rather than hoping a consumer login was good enough. A paid tier is not itself HIPAA authorization — confirm the specific product and configuration is BAA-covered before any PHI flows. A BAA is neither automatically required nor automatically sufficient; whether you need one depends on whether PHI is actually in the workflow and whose behalf you're acting on. Get that determination right on purpose.
3. Inventory your existing software for silent AI features. Have someone check whether Jira, your CRM, your design tools, your eQMS, and your CAD have quietly added AI — and whether those features protect your data or feed it upstream.
4. Write it down as a policy your team can point to — one that references your quality system, so the approved-tools decision lives where your other controlled decisions live.

Once you know what's in use and you're paying for the safer, contractually governed versions, you can consolidate later as you learn what earns its keep. The order matters: discovery first, because everything downstream depends on it.

A note on how to write that policy, because the usual approach backfires. Rigid rules invite the reflexive no — the team member who fights anything that looks like a mandate. Principles travel further than rules. A short set that a device team can actually live by:

• We protect our IP. We don't paste sensitive material into free tools.
• We use the tools the company pays for. If you need a different one, ask, and we'll buy a license.
• We assume AI can be wrong. Outputs that land in a record get checked.
• We don't put data into tools we haven't reviewed.

Principles ship the why, and the why is what survives contact with a real workflow at 6 p.m. on a deadline. The handbook can come later.

The credential firewall

Some of the most useful AI is the kind that acts on your systems — opening issues, searching repositories, moving data. That is also the kind that tempts you to hand an AI agent your credentials. Don't.

An engineer at one of these sessions wanted an AI tool that would open a ticket whenever it hit a recurring bug. It needed an access token, and he balked: I don't know if I want that accessible with more and more AI tools running on my laptop. His instinct was correct, and the fix is a pattern worth making standard.

Build a narrow tool that sits between the AI and the API. The tool holds the credentials. The AI holds the tool. You inspect the tool. That's your firewall. The AI can call a defined operation — "open an issue," "look up a record" — but it never sees the token, and the only doors it has are the ones you built. That sharply limits the blast radius. It doesn't eliminate it: a door built too wide is still a way out — a "look up a record" that returns any record is its own exfiltration path — so scope each operation to the minimum the job needs. This is the same discipline as a scoped API token: grant the minimum access the job needs and nothing more. Auditors already understand the principle of least privilege. This is least privilege applied to AI agents instead of employees.

When you find something: incident response

Discovery will eventually turn up something you don't like. Someone pasted a design specification into a consumer chatbot. A clinical dataset went into a tool with permissive terms. This is the moment the whole exercise pays off — if you have a response, and not a panic. Handle it as an incident, with a sequence:

Contain. Stop the ongoing exposure first. Revoke access to the tool, rotate any credentials that were in play, and preserve what happened — who, what, which tool, what data, when. You are going to want that record.

Classify the record. The severity depends entirely on what was exposed. Was it a design output bound for the design history file? A quality record — a CAPA, a complaint file? A regulatory submission? Protected health information? Or genuinely nothing sensitive — a formatting question, a public reference? The response scales to the answer, and getting the classification right is most of the work.

Assess the exposure as a scenario, not a verdict. Here is where cooler heads matter, because the categorical version of this advice is wrong. Disclosing information to an AI vendor does not automatically vaporize your trade-secret rights or your patent position. Trade-secret protection turns on whether you took reasonable measures to keep the information secret and whether it actually lost its secrecy; a disclosure under enterprise terms with no training and a confidentiality posture is a very different fact pattern from a paste into a free consumer tool. Patent rights turn on public disclosure and your filing timeline — and while the U.S. grants a one-year grace period, most other countries do not, so a public disclosure can forfeit foreign rights the moment it happens. A PHI disclosure is evaluated under HIPAA's breach-assessment framework — its definitions, exceptions, and four-factor risk analysis. An impermissible disclosure of unsecured PHI is presumed a reportable breach unless an exception applies or you can document a low probability of compromise — the burden sits with you. The job in the moment is to run the scenario: what was disclosed, to whom, under what terms, and what does that mean for this record. Loop in qualified regulatory and legal counsel for anything that touches a submission, a patent, or PHI. Don't self-diagnose a categorical loss, and don't wave it away either.

Document and feed it back. Record the incident and its disposition the way you'd record any other quality event, and evaluate — under your CAPA procedure — whether it rises to a CAPA. Not every formatting-question paste does; a design spec in a consumer chatbot might. Where it does, containing the exposure is your correction, and closing the gap that let it happen — the policy, the approved-tools list — is the corrective action that keeps it from recurring. The paste happened because a real need met a missing sidewalk. Close that gap, or you will investigate the same incident again next quarter. The companies that get this right are the ones whose AI usage outgrew their governance and noticed before it cost them.

The reason to build this muscle before you need it is the title of this essay. An FDA investigator or a notified-body auditor examines your processes, and the top reasons devices draw 483 observations and warning letters are rarely exotic — they are lapses in documentation, traceability, and control. "We didn't know our team was doing that" is not a finding you want to discover in the room. Far better to have found it yourself, contained it, classified it, and written it down. An incident you handled is evidence of a functioning quality system. An incident you never saw is the opposite.

Where to start this quarter

You do not need a committee to begin. You need an inventory.

• Run the discovery. Ask your team, honestly and without threat, what AI they use and what they put into it. You cannot govern — or defend — what you can't see.
• Pay for the safer, governed versions of the two or three tools that matter most, with terms that keep your data out of training and a BAA where PHI is genuinely in play.
• Inventory your existing licenses for silent AI features, starting with the systems that hold your design and quality data.
• Write four principles, not forty rules, and tie them to your quality system.
• Put a credential firewall in front of any AI that touches your repositories or APIs.
• Draft your incident response before you need it, so the day you find a design spec in a chatbot you run a play instead of a fire drill.

Shadow AI is not a sign your team is careless. It is a sign they are trying to do their jobs faster than your governance has caught up — which is the most normal thing in the world right now. The companies writing this playbook are writing it as they go. The ones who will pass their next audit are writing it with their eyes open.

Not sure where your team actually stands? The AI Workflow Assessment surfaces where AI is already in use across your organization and flags the workflows that touch sensitive and regulated data — the discovery step, done for you.