# MAGI benchmark run 20261015T171223Z-suite-nist_anova-scripted-7361e5 Status: `success`. ## Overview A run of the plan `suite-nist_anova` at commit `2925aeab226272b243436386a1c4e01d97c1377e` (tree clean: yes) with the agent `scripted` (`reference`): 22 task(s) of `nist_anova` 0.1.1, 22 result line(s) (one per task, epoch and attempt). The per-suite numbers are in `summary.json`, rendered as `summary.md`; each is defined by the code of `verify_bundle.py`, which recomputes them from `results.jsonl`. Anyone can run verify_bundle.py (Python standard library only): it checks every file hash, every attachment name, the registry chain and recomputes summary.json from results.jsonl. Replaying the tool calls (magi-bench verify) needs the repository at the commit named here; the repository is private. ## Data availability and provenance statements No third-party data file is redistributed. Each dataset is fetched from its publisher at a pinned URL and verified by the sha256 of its bytes, or regenerated from a seeded generator (`manifest.json` → `data`): | dataset | kind | URL | sha256 of the file | licence | |---|---|---|---|---| | d | file | https://raw.githubusercontent.com/scipy/scipy/92d2a8592782ee19a1161d0bf3fc2241ba78bb63/scipy/stats/tests/data/nist_anova/SmLs03.dat | `eb23c78904a4d0a238a144503a4f7db7eb849be21937ef1e20f1dbff2d264355` | Public domain (US Government work, NIST) | | d | file | https://raw.githubusercontent.com/scipy/scipy/92d2a8592782ee19a1161d0bf3fc2241ba78bb63/scipy/stats/tests/data/nist_anova/SmLs02.dat | `9c23f2c51935740a123d6882ee41b874b697b3618bdfe2915d1b76370817a080` | Public domain (US Government work, NIST) | | d | file | https://raw.githubusercontent.com/scipy/scipy/92d2a8592782ee19a1161d0bf3fc2241ba78bb63/scipy/stats/tests/data/nist_anova/SmLs07.dat | `42d492d7577ebe456fa9a204789ea8370c844f508aa3242a8e57c1976b133890` | Public domain (US Government work, NIST) | | d | file | https://raw.githubusercontent.com/scipy/scipy/92d2a8592782ee19a1161d0bf3fc2241ba78bb63/scipy/stats/tests/data/nist_anova/SmLs09.dat | `aa610b390c8d947ec258e32df0f43467fcb1820f47c5ce25a993ae1d9c715ae7` | Public domain (US Government work, NIST) | | d | file | https://raw.githubusercontent.com/scipy/scipy/92d2a8592782ee19a1161d0bf3fc2241ba78bb63/scipy/stats/tests/data/nist_anova/SmLs08.dat | `aabc5043423a82786b79fca02b04f8f31f9bcfef25683f15a67edcf359c84d18` | Public domain (US Government work, NIST) | | d | file | https://raw.githubusercontent.com/scipy/scipy/92d2a8592782ee19a1161d0bf3fc2241ba78bb63/scipy/stats/tests/data/nist_anova/SmLs05.dat | `8894da0165059fb0cf2d377d6d087f10c2842b50cf1b47e737cc64ec14253d11` | Public domain (US Government work, NIST) | | d | file | https://raw.githubusercontent.com/scipy/scipy/92d2a8592782ee19a1161d0bf3fc2241ba78bb63/scipy/stats/tests/data/nist_anova/AtmWtAg.dat | `41d7748bb1f870d8400017c53993eea65862ffd482aae1693be84d93245c303f` | Public domain (US Government work, NIST) | | d | file | https://raw.githubusercontent.com/scipy/scipy/92d2a8592782ee19a1161d0bf3fc2241ba78bb63/scipy/stats/tests/data/nist_anova/SiRstv.dat | `c7dc09da0d6a9f37f80caff0f45fd688e883303120e2d776cfab94efb1b4ef13` | Public domain (US Government work, NIST) | | d | file | https://raw.githubusercontent.com/scipy/scipy/92d2a8592782ee19a1161d0bf3fc2241ba78bb63/scipy/stats/tests/data/nist_anova/SmLs01.dat | `29d9c979d46d845ed72c0fb0ed79083c8313833bbcf72040326640d87e2e40e4` | Public domain (US Government work, NIST) | | d | file | https://raw.githubusercontent.com/scipy/scipy/92d2a8592782ee19a1161d0bf3fc2241ba78bb63/scipy/stats/tests/data/nist_anova/SmLs06.dat | `ae23a4613f8b46098984fa422b7db1ef16270f7f20c2787f128d57f0e2b778df` | Public domain (US Government work, NIST) | | d | file | https://raw.githubusercontent.com/scipy/scipy/92d2a8592782ee19a1161d0bf3fc2241ba78bb63/scipy/stats/tests/data/nist_anova/SmLs04.dat | `7c8f87a60245f5c7467418a7725855cce71608273964966e5da954c22ad5fa57` | Public domain (US Government work, NIST) | Published values quoted for verification are in `sources/`, each with its locator and an excerpt of at most 15 words; a benchmark's own gold is quoted from its pinned release with its licence. ## Computational requirements `env/platform.json` lists the Python version, the operating system and architecture, the BLAS, `uv --version`, the sha256 of `uv.lock` and every installed package as name==version. Tool replay (`magi-bench verify`) needs the repository at the commit above and `uv sync --frozen --all-packages`. ## Description of the files - `manifest.json`: what was run, by which code, agent and model, with which environment, at what cost. - `results.jsonl`: one task result per task, epoch and attempt, sorted. - `specs/`: each task exactly as it ran (canonical JSON; its sha256 is the result's `task_sha256`). - `traces/`: the raw records of every call (the source of truth); `trajectories/`: the event view derived from them. - `attachments/`: prompts, tool declarations, vendor contents and raw responses, each named by the sha256 of its canonical bytes. - `sources/`, `native/`, `tools/registry.json`, `env/platform.json`. - `SHA256SUMS`: the sha256 of every other file; its own sha256 is the run digest. Canonical JSON: keys sorted, no whitespace, ASCII, floats as Python's `repr`, non-finite numbers as the strings `"NaN"`, `"Infinity"` and `"-Infinity"`. ## Instructions to replicators 1. Integrity, without the repository: `python3 verify_bundle.py` (standard library only). 2. With the repository at the commit: `uv sync --frozen --all-packages`, then `uv run magi-bench verify ` replays the tool calls and rescores every task. ## List of tasks - `nist_anova.atmwtag.agent` (T2, agent, split public) - `nist_anova.atmwtag.tool` (T1, tool, split public) - `nist_anova.sirstv.agent` (T2, agent, split public) - `nist_anova.sirstv.tool` (T1, tool, split public) - `nist_anova.smls01.agent` (T2, agent, split public) - `nist_anova.smls01.tool` (T1, tool, split public) - `nist_anova.smls02.agent` (T2, agent, split public) - `nist_anova.smls02.tool` (T1, tool, split public) - `nist_anova.smls03.agent` (T2, agent, split public) - `nist_anova.smls03.tool` (T1, tool, split public) - `nist_anova.smls04.agent` (T2, agent, split public) - `nist_anova.smls04.tool` (T1, tool, split public) - `nist_anova.smls05.agent` (T2, agent, split public) - `nist_anova.smls05.tool` (T1, tool, split public) - `nist_anova.smls06.agent` (T2, agent, split public) - `nist_anova.smls06.tool` (T1, tool, split public) - `nist_anova.smls07.agent` (T2, agent, split public) - `nist_anova.smls07.tool` (T1, tool, split public) - `nist_anova.smls08.agent` (T2, agent, split public) - `nist_anova.smls08.tool` (T1, tool, split public) - `nist_anova.smls09.agent` (T2, agent, split public) - `nist_anova.smls09.tool` (T1, tool, split public)