The scientific method for AI

What’s clear to me is that folks are throwing the kitchen sink at AI, asking it to solve everything with little understanding of what it can and cannot do. Most of the attention seems to be on applications, while the guardrails and evals side seems to be falling behind. Adoption in biotech is heterogeneous, and so is people’s familiarity with AI. That seems to compound the problem.

It’s moments like these where work by Tacit Labs and OpenAI comes in handy. Before we hand everyone the power to design drugs, we need systems that work the way they should, with safeguards appropriate to how they’re used.

The FDA’s warning to Purolea Cosmetics Lab is a strange example of what can go wrong. Asked about missing process validation, the company said its AI agent had never told it this was required. The letter also recorded that drug production had ceased. Overreliance poses a real threat—more on that in another essay. For now, I’d like to touch on how we build these systems reliably.

The AI most folks use is general-purpose and can seem “all-knowing.” But a subject-matter expert can spot flaws that someone outside the field might miss. In life sciences, LifeSciBench puts some numbers to this: GPT-5.5 passed 25.7% of tasks and GPT-Rosalind passed 36.1% in the June 2026 evaluation. A pass required 70% of a task’s rubric points.

One way to address this is to specialize AI for the domain. I see three ways to do that:

• Training: train a model on data and objectives chosen for the domain.
• Post-training: adapt an existing model through fine-tuning and other forms of feedback.
• The application layer: add relevant context through retrieval (RAG), tools, and skills.

Maybe we need all three. I haven’t run enough experiments to know.

That said, I believe AI for life sciences needs to stand up to scrutiny from experts in each domain. A genomics expert should be able to tell me whether a system weighs the evidence for a variant’s pathogenicity and considers alternative explanations. That alone doesn’t establish a mechanistic understanding of biology, but it feels like the right direction.

The way I think we accomplish this is how we accomplish most things in science: a hypothesis, positive and negative controls, and an experiment that could support or challenge the hypothesis.

For target triage, my hypothesis might be that curated retrieval improves the ranking of experimentally supported targets. Targets with known positive and negative results in the same experimental setting give us reference controls. We can hold the cases and evaluation criteria constant, change one component, and see whether the improvement survives independent review.

This is also why ablation studies matter: remove a component and see what changes. With the scale at which we’re combining tools and agents, those experiments get harder. Components interact, so I’d want repeated runs and comparisons of different combinations, accounting for the compute budget.

If expert feedback helps us improve the system, we’d then need fresh cases to see whether the improvement holds.