Why Shadow AI is a Forensic Nightmare

Working with clients, Dan Jensen, CISSP wanted to know where machine learning (ML) models had been deployed across their environments. That meant looking for simple answers: who built them, what data trained them and how secure they were. It wasn’t easy, thanks to the creep of shadow IT.

Disclaimer: The views and opinions expressed in this article belong solely to the author and do not necessarily reflect those of ISC2.

It was supposed to be a straightforward afternoon spent putting together a spreadsheet so I could run penetration tests against those systems later. But what my informal survey uncovered instead was that employees had already been feeding production data into AI tools. Not maliciously; just to get their work done.

In one finance department, a user had fed sensitive data into an AI chat interface but wasn’t certain which one. The data “probably” contained Personally Identifiable Information (PII) – but, because it was copied and pasted, we had no way of knowing how much exposure had actually occurred. In a nearby operations group, someone casually mentioned spinning up a local LLM with no specifics on its use case (I asked them to please hit pause). It was this latter conversation that really got my attention.

None of these events had been subject to security review. More importantly, nobody tracked how much data was involved, whether it was secure, or, crucially, where the logs were. The evidence simply didn’t exist and, to the people involved, it wasn’t even an IT security issue. They were just using tools.

That last part is what turned this from a housekeeping problem into a four-alarm fire. Unlike a traditional database, these models leave almost no trail. There were no query logs, no record of who asked what and no way to reconstruct what an endpoint had returned after the fact. A model could be answering an outsider’s probes with real customer data right now, and the owner would have no alarm and no log to prove what was taken. The exposure is silent and, forensically, it is unrecoverable. Detection and response become meaningless words if an attack can’t even be identified.

Understanding the Risk: Membership Inference

To understand why this is catastrophic, let’s remember how these models fail. The attack that matters most here is called membership inference. It doesn’t steal the model or its source code; it asks one simple, effective question: Was this specific person’s record used to train this model?

That sounds academic, until we remember what corporate training sets actually consist of – such as people flagged for collections, accounts in hardship, or customers matching a sensitive profile. Confirming someone is in that dataset discloses the sensitive fact itself, without a single raw record ever leaving your network.

The reason this attack works comes down to memorization. When a model has more capacity than its data can fill (too many parameters and/or too few examples), it stops learning patterns and starts storing rows. It becomes like a covers band with 40 recording-perfect note-for-note songs locked down cold: they’re flawless and supremely confident on those 40 tracks. But, if you call out “Play ‘Free Bird’”, they have no idea what to do; they never learned music, they just memorized 40 recordings. An underfed model does the exact same thing, and that misplaced confidence on data it has already seen is the exact fingerprint an attacker exploits. 

What My Testing Showed

I created several machines and models on the cloud, downloaded and trained them on synthetic PII and conducted several test attacks against the surface using the tools that a well-equipped script-kiddy could deploy. Two findings reshaped how I think about AI risk.

First: the danger scales inversely with data volume – this is the counterintuitive part. At a healthy training size above several thousand records, an undefended model leaks, but modestly. As the dataset shrinks toward the few-hundred-record range typical of a niche business-unit project, the model can no longer find broad patterns. Instead, it falls back on memorizing rows outright, meaning the attacker’s advantage climbs sharply.

In my testing, attack success rates rose substantially as the data shrank. The smallest, least-governed models – those least likely to have had any security oversight – are the most exposed. That’s the exact opposite of where most security programs focus.

Second: the standard defense is a brake, but not a cure. The recommended mitigation for preventing membership attacks is structural regularization. This works by putting constraints that stop a model from contorting itself to fit individual rows. The good news is that I found this to work and at scale it costs almost nothing in model accuracy, which is why it belongs in every build standard.

There are two important caveats, though: it never drives the leakage to zero and, on small datasets, holding the defense in place starts trading away genuine accuracy. On small data sets, there is no winning position.

My practical takeaway was uncomfortable: for a small model on a sensitive dataset, we may not be able to have both strong privacy and full utility, and we certainly can’t assume we got privacy for free.

My Advice for Practitioners

I came out of this with five actions I now treat as non-negotiable. None of them require heavy LLM knowledge or AI math; they require you to take the blind spot seriously.

• Find the shadow AI first: I cannot protect, or even reason about, a model I do not know exists. The highest-risk systems are the ones built quietly on local machines or unsanctioned cloud accounts, outside review, logging and inventory. I have to treat an undisclosed model exactly as I would an undisclosed database full of raw customer records, because functionally that is what it is. Finding them is job one.
• Build a locked dataset inventory: For every model that touches sensitive data, I maintain a controlled register: what dataset trained it, which unit owns it, how many records it holds and how sensitive those records are under my regulatory obligations. Training-set size turned out to be the single strongest predictor of risk in my testing, so I flag any model trained on a few hundred high-value records as high-risk by default.
• Recall what I can’t vouch for: If a model is reachable from outside, was trained on a small or selective cohort and had no privacy defenses applied, it matches my worst-case profile precisely. With shadow AI, I usually can’t prove a given endpoint isn’t that case. Faced with uncharacterized exposure and a severe worst case, I pull it and retrain it properly before it goes back out. That is not an overreaction; it is my only move to closes a gap I can’t otherwise measure.
• Plant canary records: I seed a handful of known [synthetic] records into training data, then periodically check whether the model treats them with the tell-tale over-confidence of memorization. If it does, I have forensic evidence of leakage I’d otherwise never see, and queries probing those records will surface in logs as adversarial activity. The specific techniques will evolve, so I treat it as a principle: build testable markers into my data so memorization leaves a trace.
• Log inputs, outputs and confidence: Most teams log the prompt and the answer. Fewer log the confidence score attached to that answer and confidence is the tell. These attacks reveal themselves as a model being unusually certain on particular records. Crucially, this logging only matters if it is running before an incident. If it’s not on today, a future audit cannot tell me whether yesterday’s traffic was an attack, because the evidence was never written down. I don’t get to reconstruct it later.

The Bottom Line

Our collective intuition – that the biggest, best-funded systems carry the most risk – fails completely when applied to AI. Privacy risk concentrates where data is scarcest and oversight is thinnest: in those small, homegrown, barely documented models that never went through a review. The models we can see are manageable. It’s the ones we can’t see that should keep us up at night.

Dan Jensen, CISSP, has over 30 years of experience in IT infrastructure and zero trust data defense. He has held management, technical and advisory roles, with responsibility for evaluating and protecting critical infrastructure and data. His cybersecurity work currently spans AI behavioral security, incident prevention and response, and automated threat mitigation.

Related Insights

• Foundations of Digital Forensics Course
• We Have AI Everywhere but Control Almost Nowhere
• Taming AI Sprawl and Shadow AI Through Strong Governance