EvaluationsGiskard’s StereoTales tests what models generate, not only what they recognize
The September 28 research update describes multilingual stereotype discovery and a gap between judging harmful associations and avoiding them.
Evaluations
The tests and evidence that separate a stronger model from a better-looking score.
Benchmarks · Reliability · Cost & performance
12 articles
Evaluations
EvaluationsPatronus’ SpeedrunBench asks whether an agent can improve a successful route
The September benchmark separates completion from efficiency and makes the search budget part of the result.
EvaluationsAISI and EvalEval put the conditions behind benchmark scores into shared records
Configuration and inference budgets help explain why apparently comparable evaluations diverge.
EvaluationsProvenanceGuard tests whether an agent cites the right source for a true claim
The study separates pooled factual support from source attribution, while showing the cost of conservative blocking.
EvaluationsAnthropic gives METR a closer view of its safety decisions
Six embedded evaluators receive employee-like access, while predefined exclusions and redaction authority remain important limits on what independent oversight can reveal.
EvaluationsTogether's ParallelKernelBench tests whether AI code can coordinate GPUs
The June benchmark separates kernels that run correctly from kernels that also beat a reference implementation, exposing the communication work that single-device tests miss.
Hover to animateEvaluationsNVIDIA V2D turns human video into robot practice, one stage at a time
The released workflow reaches GR00T N1.7 training. A small real-robot test shows why simulated practice and real adaptation need to be examined separately.
Hover to animateEvaluationsOpenAI and Ironclad: One missed rule can derail an AI workflow
Their contracting-workflow test shows why a better average score is not the same as a workflow a company can switch on. Mandatory rules need their own acceptance test.
Hover to animateEvaluationsSurge AI: Correct spreadsheet maths can still produce the wrong answer
GDP.xlsx tests whether models find the evidence across a workbook before calculating. A missing sheet can matter more than another improvement in arithmetic.
Hover to animateEvaluationsMercor: Winning an accounting test does not prove lower workplace costs
Its bounded comparison favors AI on structured tasks. The next question is whether that advantage survives ordinary workplace context, review and exceptions.
Hover to animateEvaluationsHarvey: One invented fact can invalidate an otherwise complete AI answer
Artificial Analysis and Harvey now gate headline credit on grounded answers. That separates two problems model teams must solve: missing requirements and unsupported claims.
Hover to animateEvaluationsFactory: Old test cases can help check AI-written repairs
Legacy-Bench exposes a gap between an agent saying it is done and a repair passing independent checks. Preserved outputs and edge cases can matter as much as a stronger coding model.
No articles in this topic yet
Explore the other topics in this desk.
Explore the research library for paper-based analysis.