LabAuto Bench

How well do AI agents turn a manual wet-lab SOP into an Opentrons Flex protocol an automation engineer would accept? The simulator only tells you a protocol runs. We also score whether it makes lab sense.

Leaderboard · Track A (SOP → executable protocol)

Guided = the SOP spells out automation rules (tip changes, mixing cadence). Bench = the same task worded like a scientist's bench SOP, without those hints.

Per-task scores

What this first run shows

How scoring works

Layer	Weight	What
L1 Executable	gate	Runs end to end in opentrons_simulate. Fail = 0.
L2 Process sense	40	Rule checks the simulator does not do. Each distinct problem type costs one third of the layer.
L4 Outcome	40	Final well volumes and, where it matters, what went into each well.
L3 Efficiency	20	Transfers vs. the reference protocol (mixing is not penalized).

Process checks:

Known limits (scorer )

• Aspiration height, blow-out, touch-tip and multi-channel pipetting are not modelled yet.
• Timing rules are checked on one representative well; temperature is checked at the first setpoint.
• Liquid dispensed into wells the task does not list is not penalized.
• The sop-to-protocol skill and this benchmark were written by the same team. Treat its score as a self-reported baseline, not an independent result.

Data policy

Tasks are authored in-house (CC BY 4.0) and carry a canary string. SOP text and expected volumes are not published here so the task set stays useful as a test set. Agents run locally; we never receive your agent.