Organizations have seen AI work; they have production systems running where agents are processing clinical literature, surfacing competitive intelligence, supporting HTA submissions, and in some cases, even flagging safety signals. And yet something is happening inside these production systems that the dashboards are not showing. It is not an error or a failure. It is something quieter and much harder to catch.
The outputs look right. Recall is strong, and the systems are faster than ever. But beneath those headline metrics, a subtle pattern is emerging: what they find—and what they miss—is narrowing in the wrong direction. In pharma, that matters because the rarest events are often the most consequential.
Two Converging Problems That Nobody is Connecting
1. The Model Collapse
It is mathematically documented. When AI models are trained on AI-generated data, they progressively lose the ability to represent the edges of distributions, including rare and unusual data points. Each generation of training amplifies the center and erodes the tails. The model gets better at the common case and quietly worse at everything else.
2. The Provenance Collapse
Unlike model collapse, which has a body of research behind it, this is a pattern emerging in enterprise AI deployments rather than a formally documented phenomenon. It is the point at which AI-generated content re-enters training pipelines, knowledge bases, and evaluation frameworks at such scale that nobody can trace where a claim actually came from anymore. The original human observation might be the clinician who noticed something odd or the analyst who spotted a pattern nobody else saw. These observations get cited, summarized, re-synthesized, and eventually replaced by a model output that points back to another model output. The chain of custody breaks without an error message.
These two problems point toward a broader risk: a slow epistemological narrowing. AI systems converging on the most common, most validated, most central version of knowledge and systematically losing touch with the edges where most of the consequential things might actually happen.
The Validation Loop Nobody is Auditing
What makes this particularly difficult to detect is how it hides inside practices that are individually defensible.
AI-on-AI validation is now standard practice with LLM-as-judge.
- Synthetic training data generated by AI and used to fine-tune the next model.
- RAG systems retrieving AI-generated summaries as if they were source documents.
- Prompt libraries built by prompting AI to generate better prompts.
- Red-teaming done by AI adversaries trained on the same distribution as the model they are testing.
Each of these is defensible in isolation.
However, together they create a closed loop where the reference point for “is this good?” is itself a model output. The insights captured by domain experts with decades of experience in looking at the edge are being progressively removed from the validation chain for convenience and speed. That institutional knowledge of the exception is not captured in the pipeline; it is not that it was deprioritized, but that the pipeline was never designed to hold it.
And the metrics look reassuring: accuracy is up, hallucinations are down, and benchmark scores keep improving. But that is partly because the benchmarks often reflect the same distributions the models were trained on. An LLM judge may agree with a model not because the answer is correct, but because both share similar training patterns—and similar blind spots. The evaluation can be internally consistent without being right.
The Paradox: Absence Does Not Trigger an Alert
When an AI system misses an outlier, it does not return an error or flag low confidence. It returns a perfectly coherent answer that reflects the common case, while the uncommon case simply does not appear. There is no alert, red flag, or audit trail entry that says, “We did not find the thing you needed because it was in the tail of the distribution we were trained to compress.”
The system appears to be working perfectly right up until the moment the missing thing matters enormously.
A pattern worth naming comes up repeatedly in conversations with medical affairs and regulatory teams: the question they bring to AI is usually well-formed, the search string is correct, the inclusion criteria are defined, and the system executes precisely what was asked. What gets lost is the adjacent signal, the cross-indication pattern, or the regional safety report that an experienced reviewer would have noticed while looking for something else.
In Pharma, The Edges Are Crucial
1. In pharmacovigilance: Signal detection is fundamentally a rare event problem. The adverse reactions that matter most do not appear in the common case. They surface in a subpopulation too small to register in the training distribution but large enough to matter when the drug is at scale. They appear in a small-sample study published in a regional journal that never reached the major indexing databases. They emerge from a spontaneous report filed in one market that has not yet crossed a global detection threshold.
AI systems optimised on large, well-indexed, English-language safety datasets will be systematically better at finding signals that were already findable and systematically worse at the ones that require looking at the edges.
2. In systematic literature reviews: The backbone of HTA submissions to NICE, G-BA, HAS, etc., an SLR agent executes faster than any human team. Search, screen, select, synthesize—the PRISMA flow diagram looks complete and evidence tables are populated. But what the agent is optimized to find is common studies: well-indexed studies, studies with clean abstracts, studies with consistent MeSH tagging, and studies with enough citations to be statistically visible. Older studies in regional journals, conference abstracts that never became full publications, and grey literature that NICE actively weights in its appraisals get filtered out not because the agent assessed them as irrelevant, but because they were never in the distribution it learned to look for.
The evidence base has quietly narrowed. The recall metric said everything was fine. Recall of what, exactly? That is the question nobody asked.
The Strategic Dimension
What is beginning to surface across AI deployments in pharma is a pattern of convergence – similar retrieval sources, similar foundation models, similar evaluation approaches. That is partly efficiency. It is also, quietly, a competitive risk. The signal that creates strategic advantage is almost always weak, early, or contradicts the prevailing view. Those are precisely the signals that narrowing systems are worst at finding.
The organization that maintains genuine domain expertise will have a structural advantage that compounds as AI systems converge. The expert who notices things that do not fit is about to become the most valuable person in the room—and in AI development.
Industry has broadly adopted the language of human-in-the-loop, which has the right instinct with the wrong architecture. Human-in-the-loop places the domain expert at the end of the process for reviewing, approving, and catching what the model missed. Human-in-the-lead places domain expertise at the center of the design, shaping what the agent looks for, defining the edges of the distribution it must not compress, and maintaining the provenance chain that makes outputs defensible to a regulator, a payer, or an HTA body.
The distinction matters because the two approaches produce fundamentally different systems under pressure. A human-in-the-loop system catches obvious errors. A human-in-the-lead system is structurally less likely to produce the quiet narrowing in the first place because the domain expert is directing what gets looked for, not just reviewing what came back.
|
Human-in-the-loop
|
Human-in-the-lead
|
|---|---|
|
Domain expert reviews output
|
Domain expert directs the workflow
|
|
Catches errors after the fact
|
Prevents distributional narrowing by design
|
|
Expertise as a checkpoint
|
Expertise as the architecture
|
|
AI decides, human approves
|
Human frames, AI executes
|
Three Things Worth Doing Now
Keep domain experts in the design process, not just the review process. The question is not "does this output look correct?" It is "are we looking in the right places?" Those are different questions and only one of them requires a domain expert at the start.
- Build provenance infrastructure before scaling. Every AI output entering a knowledge base, a regulatory submission, or a training pipeline should carry traceable metadata.
- Treat data diversity as a quality control mechanism. The breadth of inputs feeding AI systems is the primary defense against model collapse. Epistemically monocultural AI systems fail the same way at the edges, until the edges are all that matters.
The issue is not whether AI belongs in pharma workflows. It is how those workflows are designed.
Systems built to optimize for the common case will get progressively better at finding the common case. That is useful, but it also creates a predictable blind spot: rare signals become easier to miss precisely because the system is becoming more efficient.
In pharma, that is a serious design problem.
Domain experts therefore cannot sit at the end of the process, reviewing outputs. They need to shape the system upstream: what it searches for, what it treats as anomalous, and where it is required to look beyond the dominant pattern.
The risk is not that AI gets things wrong at random. It is that it gets very good at being right in the same places, while becoming systematically less attentive to the places that matter most.
