Evaluating Scientific AI on Ten Public GeneBench-Pro Tasks

Each profile ran the same ten tasks once. Native product modes are shown separately from API effort settings. The table retains endpoint misses, formatting failures, resource limits and operational failures as different outcomes.

The shared-field comparison covers 67 settings. Ten fields gained a pass and eleven lost one. Ungraded fields do not enter this intersection. It answers a different question from the whole-task comparison across all 90 settings.

The Exa arm’s recorded estimate was $188.0754: $186.6760 in solver usage, $0.1960 in Exa fees and a $1.2034 conservative failed-request allowance. The allowance is not verified usage from the failed response. Local Nemotron hardware cost is unpriced; these figures are usage estimates rather than invoices.

Muse Medium received an excerpt explaining first-order uncertainty propagation. Its next saved Python computation propagated measurement variance through transformations and used the result in regression weights. There is a concrete correspondence between delivered methods text and subsequent code.

That setting had already passed without web access. The code did not explicitly attribute the choice to the source. In the other three searched passes, the core method was present before search. These traces help locate possible changes in behavior; they do not establish that an excerpt caused a pass.

The audit inspected all seven gain traces, all four searched passes and three searched misses beyond the call inventory. The other nine searched misses were not semantically audited. This examines saved tool calls, code and outputs, rather than private internal reasoning.

Search responses already contained highlights, so zero fetches does not mean zero source exposure. A highlight also does not establish that a full paper was read or applied.

GeneBench-Pro supplies this study’s tasks. Its simulated data and known targets let us inspect concrete analytical decisions and check numerical endpoints. The full benchmark has 129 questions; this study uses ten released cases.

LABBench2, by Laurent and colleagues, evaluates practical biology research capabilities across information, data and literature tasks. Its access and tool comparisons are relevant to separating information availability from successful use.

LifeSciBench, by Liu and colleagues, uses expert-authored research tasks and detailed rubrics to evaluate scientific correctness and usefulness. That broader perspective matters when endpoints alone do not explain the quality of an analysis.

This study builds on those efforts. It does not include new LABBench2 or LifeSciBench evaluations, or pool their scores with GeneBench-Pro.

The baseline covers 26 model/route/effort profiles across five families, with one observation per profile on each of the ten tasks. Effort labels and native modes are provider-specific configurations, rather than equal compute budgets. The public package is pinned at revision eb75a3c0996b3cedcc9af685bad02fd166848fa2.

A whole-task pass requires every required endpoint to pass the saved grader and the final answer to meet its output contract. The grader checks deterministic endpoints and schema, not the semantic quality of a reasoning string. Some task conventions remain ambiguous; the saved labels are endpoint outcomes, rather than a comprehensive expert assessment of each analysis.

The Exa comparison reuses 90 baseline settings and adds 90 new observations. Task prompts, model/effort settings, data, graders and per-case solver limits were preserved. Serving-context allowances, guard versions, timing and concurrency differed. A single observation per setting and these historical differences limit attribution and do not estimate run-to-run reliability.

Search used a shared interface backed by a Luna helper and, in a separate arm, Exa. Exa supplied bounded highlights; fetch remained a guarded direct publisher reader. Only methods and official documentation were permitted. Benchmark papers, solutions, graders and prior responses were excluded from retrieval.

The Exa owners are 34 initial settings, two continuations and 54 later settings. The last 56 used a repaired guard with a common 42-query methods menu; the final 54 also used up to five Muse workers. All Nemotron settings belong to the earlier stratum. Limits retained 20 web calls within 120 aggregate tools, 120 model responses, 40,000 output tokens per request, 256,000 per run and 60 awake minutes.

Download the complete versioned study data (JSON). It preserves the baseline, the earlier Luna-backed comparisons and Exa as separate conditions. This article presents the baseline reasoning sweep and the Exa follow-up; the archived arms have not been replaced or removed. The data exclude answer values, raw transcripts, credentials and private reasoning.

The main source has SHA-256 df499f6f288848cd03635b28577e882098c3c82281ba9ac576d4d793e07b503f; the follow-up source has SHA-256 c3f3dad2b4b876db1ee4fca9242baae7ab016e8fd35ea494efad8246462c451b. All optional runs ended, collection was sealed and the monitor was retired. No new observations were created for this article.

• Li, Shringarpure, Wong and Ho. GeneBench-Pro: Evaluating Multistage Statistical Reasoning in Genomics, Quantitative Biology, and Translational Biomedicine. Public task package.
• Laurent and colleagues. LABBench2: An Improved Benchmark for AI Systems Performing Biology Research.
• Liu and colleagues. LifeSciBench: Evaluating Language Models on Realistic, Expert-Level Tasks in the Life Sciences.
• Fine-tuning scientific AI — the question that led to these experiments.