AI needs a reality check

Anthropic recently acquired stealth startup Coefficient Bio for 400 million. But there’s only one true test of any healthcare AI: Did it work in humans? Did it create a medicine that saved someone’s life? And bluntly, most companies have not achieved that. Let’s look at the number of treatments brought to market. Isomorphic? None. Lila? The same. Marketing claims in AI rarely survive contact with reality. That’s because making real progress in healthcare is hard. To test a new treatment, you need to take it through a Phase 3 clinical trial. That’s typically 10 years and 2 billion. To test a diagnostic, you need to demonstrate clinical benefit, pass a rigorous third-party test, and build a full quality management system—before your product is even permitted into the clinic. To uncover and prove new human biology? That could take decades of scientific experimentation. CLOSE THE GAP So what do we need to do? The industry needs to close the gap between where AI models are trained and where medicine actually happens. That hard graft is what the best AI companies in the field are doing. Companies like Insilico Medicine and Recursion are advancing AI-discovered assets through clinical trials. At Owkin, we’ve taken OKN4395, our oncology drug, into the Phase 1a clinical INVOKE trial. Beyond that, we’ve trained our AI on real patient data for years and brought MSIntuit CRC through Europe’s CE mark into pathology practice. This is hard work, but bringing your AI to patients has a big upside: It forces your AI to be better. From our experience, we’ve had to tackle unexpected, knotty problems. When we were first bringing diagnostic AI to the clinic, we realized that the models wouldn’t generalize well across population changes or scanner setups. We had to develop simple but robust methods to adapt our models to the vagaries of individual locations and technologies. IMPROVE THE FEEDBACK LOOP IN REAL TIME We think that this “reality check”—testing our models’ results with real patients—is so important, that we’ve built it into the structure of our INVOKE trial. In a traditional trial, the design looks only at the essential indicators of trial success and the interim results would decide whether the trial progresses. That’s it. But unlike a traditional trial, we’re using ongoing data from our patient participants to improve our AI. Where our AI’s predictions about patients’ responses have missed the mark, we have retrained it on the real data to improve its performance. It’s a positive feedback loop: The more information we get from real-life trials, the better our AI gets, the better it works for patients, the more models we can test. This is where the field is headed. There are different flavors. Some companies insert extra steps—like testing their AIs’ results on in vitro model systems (outside the body, like in Petri dishes)—but eventually no drug-discovery, trial-design, diagnostic, or clinical AI can be successful without showing that the AI’s results work in humans. But it doesn’t all have to come from clinical trials. MODEL TRAINING DATA CAN BE VARIED You can bring initial model predictions closer to reality by training those AI models on rich patient data. The more detailed the data descriptions, the broader the range of modalities, the more likely the signals the models pick up are real. When you need to test new AI-generated hypotheses and you can’t do it with existing patient data, you can get as close to the patient as possible in vitro. For example, patient-derived organoids preserve human biological complexity that lab-grown cell lines and animal models lack, while also bringing a wealth of clinical information about the patient of origin. And you can test how models’ predictions of patients’ responses fare in the wild—outside rigorously controlled testing settings—with real human patients. Quelel horreur! That’s the beauty of having a full stack ecosystem. When you make models that are used routinely in the clinic, like our diagnostic models, you get a real sense of their strengths, limitations, and where the real addressable clinical pain-points are. At Owkin, we do all of these things. It’s not easy. It stretches us. And it forces us to confront the real barriers to bringing treatments to patients. This is the point in the article where I should be making my own visionary, outlandish claims—something to really put my marketing team into panic mode. Something about how the future is going to change forever, about how close we are to some epoch-defining shiftyou know the kind of thing. But let me actually finish with something more grounded. It’s easy to get excited about the promise of AI. Believe me, I do. But it’s even more satisfying to watch all those dreams and expectations collide with reality, evaporate—and see what survives. Because that is what’s real. Thomas Clozel, MD, is cofounder and CEO of Owkin.