Fine-tuning scientific AI

I’ve been diving into GeneBench-Pro, and it’s sent me back to the three ways to specialize AI in my last note. Fine-tuning still looks promising to me, but I’m curious how much can be done by giving a model better context.

In SciAgent, giving the untuned models tools actually hurt performance. After tool-use training, those tools helped. Aviary gives me another reason to try fine-tuning: training a Llama-3.1-8B-Instruct agent on successful tool-use trajectories, followed by expert iteration, took LitQA2 test accuracy from 30% to 72% in the PaperQA environment. The agent already had the tools. Learning to use them made a substantial difference.

Biomni improved performance over its ReAct variant using the same Claude Sonnet 3.7 backbone, without task-specific fine-tuning. So there’s room to improve how the agent works around the model, too. And BioT5+ found that removing the added PubMed Central and bioRxiv training data slightly hurt its molecule-description and description-guided design results.

What caught my attention in LABBench2 is that models do worse when they have to find the relevant figures and tables in papers themselves. The authors suggest in the discussion that better reasoning alone won’t solve this. Evidence curation clearly matters here, but I’d like to run a study to test that suggestion.

It’s funny how much of this comes back to context. More reasoning helps on GeneBench-Pro, too. I’m curious whether better context could save reasoning tokens, improve the answers even when the model has plenty of room to reason, or both. I don’t have an answer to that yet.

I’m also very curious to compare individual models fine-tuned for these tasks with closed frontier models, given the same evidence and tools. These papers use different tasks and setups, so they don’t settle that comparison. I’ll be running tests and adding the results here.

If you’re interested in collaborating, send me a text or email. If you have evals you’d like me to test against, let me know.