A Tale of Two Pilots

Why is AI success in Pharma so hard to predict?

If one AI pilot produced a spectacular scientific breakthrough, and the other produced results that were merely fine, which one would you bet on for a long-term partnership? Keep reading. The answer might surprise you.

Two top-20 pharmaceutical companies. Both evaluate the same type of AI platform: one that analyzes clinical trial data to identify which patients respond best to treatment, optimize study design, and support development decisions. Same technology. Same evaluation criteria. Same six-month timeline.

Very different outcomes. But not in the direction anyone expected.

PART 1

The Obvious Winner

Pilot A: The Home Run

The first pilot was in a Phase III oncology program. High unmet need. The kind of program that gets written about.

The team shared data from a completed trial, and very rapidly, the AI platform identified a patient subgroup where the treatment effect nearly doubled. The findings were statistically significant and reproducible across validation analyses. The team was visibly excited. One senior scientist leaned over to a colleague mid-presentation and said, quietly but not quietly enough: 

"This is a huge effect."

It looked like exactly the result every AI company dreams of delivering. A clear, compelling, data-driven breakthrough, the kind of finding that makes you think a major partnership announcement is only weeks away.

Pilot B: The Disappointment

The second pilot was a Phase II in an immunology program. An established franchise with a crowded competitive landscape,m the company had three related assets in development and was actively deciding how to sequence them. Solid program. Not glamorous. The team wanted to know whether there were responder populations that conventional analyses had missed.

The AI models were applied to the dataset. The results came back. And they were... fine. No dramatic subgroup. No doubling of effect size. No headline-worthy breakthrough. The AI largely expanded what the team already suspected, with a few signals that were interesting but not surprising.

At the final readout, the reaction was not what anyone expected. The head of the program - a clinical development leader who had clearly done her homework - came prepared. And she had questions. A lot of them.

How was the model validated, and on what dataset? If we changed the patient population in the next study, would the model need to be retrained - and how long would that take? What happens when the endpoint changes between Phase II and Phase III - does the methodology hold? If a regulatory reviewer asked us to justify a stratification decision based on this output, what would we hand them? How transparent is the underlying methodology to someone outside your team?

It felt, honestly, like a cross-examination. The room was quieter than it had been all meeting. Nobody was congratulating anyone.

If you had attended both presentations, you would have drawn an obvious conclusion: Pilot A is about to become a major partnership. Pilot B just got torn apart.

Almost everyone in the room walked out thinking the same thing.

PART 2

The Surprise

 The outcome was exactly the opposite.

Pilot A never expanded. No additional projects. No budget. No deployment. The pilot quietly disappeared - not cancelled with fanfare, just never mentioned again. The spectacular finding sat in a presentation deck, admired by the people who had seen it and forgotten by everyone else.

Pilot B became a strategic collaboration. Multiple studies. Additional indications. Executive sponsorship. Long-term adoption. The team that gave a polite "we'll think about it" had, it turned out, been thinking very seriously. Those pointed questions at the readout weren't a sign of discomfort - they were the final steps of a decision process that had already started.

AI pilots in pharma are rarely won or lost on the algorithm. They're won or lost on five other dimensions

PART 3

The Five Determinants 

01  The Champion

In Pilot A, the champion was a brilliant statistician with deep technical expertise - but his influence stopped at the edge of his function. When budget discussions began, there was no executive backing the project, no political momentum. The result was scientifically compelling and organizationally stranded.

In Pilot B, the champion was a clinical development leader who could translate AI findings into language every stakeholder understood - from the bench scientist to the CFO. Those tough questions at the readout weren't skepticism. They were a procurement checklist. She needed to know if the methodology would survive a regulatory conversation and whether it could scale across her pipeline. Those are not the questions of someone who is unimpressed. Those are the questions of someone who is serious.

AI doesn't get adopted by organizations. People adopt AI. Champions matter more than results.

02  The Program

Pilot A was attached to a program that was nearly finished. Most major investment decisions had already been made. The development plan was largely locked. Even a valuable AI insight had nowhere useful to land.

Pilot B was attached to a flagship asset - one of the company's genuine strategic priorities, with multiple upcoming trials across indications and senior leadership paying attention. Every useful insight had leverage because there were real decisions still on the table.

Insight only create value when decisions are still open. depends entirely on where it.

03  Results Must Be Verifiable

Here's something that surprises people outside pharma: a spectacular result can actually be harder to act on than a modest one. Pilot A's dramatic subgroup finding triggered organizational scrutiny rather than excitement. Was this subgroup defined prospectively or identified post-hoc? Could the finding be replicated in an independent dataset? Could a biostatistician at the FDA reproduce the result?

None of these questions implied the result was wrong. But the path from "we believe this is real" to "we can act on this" in a regulated, high-stakes environment is long. 

Pilot B's more modest findings came packaged for action: clinical rationale, biological plausibility, pre-specified validation, and clear methodology any internal reviewer or regulator could follow.

Being right is not enough. You have to be right in a way people can verify and justify.

04  Trust and Credibility

Pilot A's AI team entered a large, mature pharma company with deeply entrenched vendor relationships, without a sponsor who could vouch for them at the right level. Every interaction was transactional. The vendor was treated like a contractor, because that's all the engagement structure allowed it to be.

Pilot B's team approached the engagement differently - not as a client to impress, but as a scientific collaborator to understand. They asked questions most vendors don't ask: What decisions are actually at stake? What would a useful answer look like versus an interesting one? They surfaced limitations in their own methodology before anyone else asked. When a result was ambiguous, they said so.

That behavior built something no single result could: credibility as a scientific partner rather than a technology provider.

Adoption follows trust. And trust is built in the small moments, not the big ones.

05  Actionability

Pilot A was initiated by the statistician himself as a proof-of-concept on a recently completed trial - born as an intellectual exercise rather than a business decision. When the exciting result came back, there was nothing left to decide. Acting would require re-opening regulatory discussions with new retrospective data, or funding an entirely new validation study. Neither had been budgeted. Neither had an executive owner.

Pilot B was the opposite. Every analysis was tied to a decision that hadn't been made yet. Adjusted inclusion criteria, revised stratification strategy, updated sample size assumptions, changes to the Phase III design currently on the drawing board. The AI didn't just produce findings - it changed what people did the following week.

Pharma doesn't buy insights. Pharma buys better decisions.

CLOSING

The Questions That Actually Matter

When people evaluate AI pilots, the first question they tend to ask is: Did the model work? Did it find something real? That's important. But it's not the only question.

• Does the champion have the influence to turn a finding into a decision?
• Is the program at a stage where insights could actually change something?
• Are the results verifiable in a way that could survive internal and regulatory scrutiny?
• Can the vendor earn enough trust to be treated as a partner, not a contractor?
• Will someone be able to do something differently tomorrow because of what the AI found?

The AI pilot that changes decisions is the one that wins.

That's the tale of two pilots.

‍