Scientific search is often evaluated by whether it retrieves a relevant paper. A researcher curating experimental data needs a stricter answer. The requested assay, population, outcome and data availability must belong to the same qualifying experiment. Finding all the right words somewhere in a document does not establish that conjunction.
SciCurateBench tests this distinction using raw scientific literature rather than cleaned snippets. The task is closer to the work that makes literature review slow: tracing conditions across long files, rejecting near-matches and retaining evidence for why a study does or does not qualify. Its value lies in exposing a gap that broad relevance scores can hide.
The benchmark asks for a conjunction
SciCurateBench, posted October 9, uses expert-annotated papers to test conjunctive experimental criteria in raw scientific documents. Its LangPoli system turns natural-language policies into executable retrieval actions and improves over the evaluated base model. Read the research paper.
The corpus contains 200 expert-annotated papers from major life-science journals. Systems must verify conjunctive biological experimental criteria across unprocessed literature. This requires maintaining relationships between facts, not merely detecting their separate presence.
A candidate paper may discuss one assay in the introduction, report another in its methods and describe a desired outcome only in prior work. A loose summariser can combine those pieces into a persuasive but unsupported answer. The benchmark’s structure makes that kind of evidence assembly a failure, even when the document is broadly on topic.
Raw files preserve the mess that snippets remove
Scientific curation often depends on tables, methods, supplementary material and qualifications spread across a long document. A benchmark built from short, polished passages can remove the very retrieval and association work that makes the task difficult. SciCurateBench instead starts from raw, unprocessed literature.
That choice changes the error analysis. A system may know what a criterion means yet fail to locate the relevant evidence or connect it to the right experimental group. Conversely, a keyword hit may be easy to find but insufficient to decide eligibility. The proposed policy-and-actor arrangement attempts to turn higher-level guidance into concrete retrieval operations. Its success should therefore be assessed with evidence traces as well as classification F1. For a researcher using the output, the decisive value is being able to inspect the association between each condition and its source, particularly when the system rejects a paper that initially looked promising.
Policies become executable retrieval actions
The accompanying LangPoli system has a policy agent write natural-language action guidelines that actor agents convert into executable tools. The authors report paper-classification F1 of 83.16% versus 58.22% for the base language model, with additional structured retrieval and re-ranking evaluations.
Those results support the integrated approach on the tested tasks. They do not make automatically generated tools infallible. A tool can search the wrong section, parse a table incorrectly or apply a criterion to the wrong study arm. The policy-to-code interface therefore needs execution checks and source-level verification, not just a convincing description of the intended method.
A useful rejection can save more time than another hit
For an expert, an answer that identifies the one disqualifying condition can be more useful than a ranked list of broadly relevant papers. The system should retain both the passage supporting each criterion and the link showing that those passages refer to the same experiment.
The larger implication is that practical scientific assistance should be measured at the level of evidence configurations. Search relevance is an entry point. Curation requires preserving relationships and acknowledging unresolved conditions. That shift makes evaluations harder, but it also brings them closer to the decisions researchers need the assistant to help make.
Questions, news tips or corrections? Contact [email protected].
