A research agent finds several useful candidates. Another learns to rank unseen candidates accurately. Those achievements can diverge, even in the same experimental campaign.
Autonomous Science Bench, introduced October 13, evaluates budgeted campaigns grounded in published experiments. Success requires both a discovery gate for measured designs and a learning gate for ranking held-out designs. Its pilot reveals different weaknesses across agent configurations. Read the research paper.
Finding a good candidate is not the same as learning a rule
An autonomous science agent has a limited experimental budget. It must decide what to test and use the measurements to guide later work. Two achievements can diverge: finding promising candidates and learning a model that predicts unseen candidates well. An agent can exploit one productive region without understanding the wider design space, or build a useful predictor while spending too cautiously to discover enough strong designs.
Autonomous Science Bench evaluates both requirements. Its initial release comprises six design-build-test-learn campaigns grounded in published experiments across biology, chemistry and materials science. Agents choose batches, receive measurements through a lab service and eventually deliver a predictive model.
Two gates prevent one success from hiding another failure
The discovery gate evaluates the designs the agent actually measured. The learning gate evaluates its model's ranking of held-out designs. Passing both asks for productive experimentation and transferable inference from the results. Neither gate alone can establish the complete achievement the benchmark targets.
This is a useful departure from fixed-dataset science evaluations. When an agent chooses its own observations, the quality of its experimental policy shapes what it can learn. A final predictor trained on a supplied dataset bypasses that decision problem. The campaign structure therefore links experiment selection and model building while keeping their outcomes separately visible.
The pilot measures configurations, not models in isolation
Across eight agent-model configurations and one hundred and forty-four trials, the best configuration succeeds in sixty-seven percent of trials. The hardest campaign is solved in only three of twenty-four attempts. Gate-level analysis shows different weaknesses: some configurations rank unseen designs well but discover too few promising measured candidates; others discover candidates without producing a good held-out ranking.
The full paper stresses that scaffolds differ in tools and prompts. Results therefore compare agent-model configurations, not pure model capability under an identical wrapper. Reasoning effort also varies with the stated configuration. Those details belong beside the headline score because changing the harness can alter both experiment management and the final analysis.
The agent chooses observations as well as answers
The lab service creates a sequential information problem: results from earlier batches become evidence for choosing later designs. An agent can spend its budget exploiting a region that already looks promising or exploring alternatives that improve its final predictive model. The two gates expose the different costs of those choices rather than assuming one policy serves both objectives equally well.
This also changes how a failure should be investigated. Low discovery with good held-out ranking suggests a different bottleneck from high discovery with weak ranking. The first may call for better experimental selection; the second may call for better inference from measurements. These are interpretations supported by the benchmark structure, not proof that one particular internal mechanism caused each observed failure.
A replay is not a newly executed experiment
Version 0.1 uses replay and twin backends whose available design spaces depend on recorded measurements or surrogate coverage. Published data may also have appeared in training or become accessible if network restrictions fail. The authors identify unpublished live experiments as a stronger future boundary. The current benchmark should not be described as proof that an agent autonomously ran a wet lab and discovered previously unmeasured results.
Its contribution is nevertheless concrete: it makes discovery yield, predictive transfer and budget use inspectable together. For teams building scientific agents, the two-gate view helps locate whether improvement should target the choice of experiments or the learning from their outcomes. A convincing autonomous campaign needs an account of both what was found and what knowledge survived beyond the tested candidates.
References
Questions, news tips or corrections? Contact [email protected].
