# MAGI benchmark run 20261015T171204Z-suite-nist_univariate-scripted-ca602b Status: `success`. ## Overview A run of the plan `suite-nist_univariate` at commit `2925aeab226272b243436386a1c4e01d97c1377e` (tree clean: yes) with the agent `scripted` (`reference`): 18 task(s) of `nist_univariate` 0.1.0, 18 result line(s) (one per task, epoch and attempt). The per-suite numbers are in `summary.json`, rendered as `summary.md`; each is defined by the code of `verify_bundle.py`, which recomputes them from `results.jsonl`. Anyone can run verify_bundle.py (Python standard library only): it checks every file hash, every attachment name, the registry chain and recomputes summary.json from results.jsonl. Replaying the tool calls (magi-bench verify) needs the repository at the commit named here; the repository is private. ## Data availability and provenance statements No third-party data file is redistributed. Each dataset is fetched from its publisher at a pinned URL and verified by the sha256 of its bytes, or regenerated from a seeded generator (`manifest.json` → `data`): | dataset | kind | URL | sha256 of the file | licence | |---|---|---|---|---| | d | file | https://raw.githubusercontent.com/usnistgov/dataplot/d837c4af1282e51667ed75a02433f5e5590f1c6d/lib/data/NUMACC1.DAT | `ce7c7a0fc6d7cedd0d5e1f9911afb9337544be6836730beb9968ca0a4571d375` | Public domain (US Government work, NIST) | | d | file | https://raw.githubusercontent.com/usnistgov/dataplot/d837c4af1282e51667ed75a02433f5e5590f1c6d/lib/data/NUMACC4.DAT | `ca310dc767f5130f980f8280bbe69e26ba414a4d85a89fc11b6c767b828e5fe4` | Public domain (US Government work, NIST) | | d | file | https://web.archive.org/web/20000118203016id_/http://www.itl.nist.gov/div898/strd/univ/data/Lottery.dat | `e79813a132cf9901a85970615a83ca0c6b851c24832c78160734c71da6ff6c3a` | Public domain (US Government work, NIST) | | d | file | https://raw.githubusercontent.com/usnistgov/dataplot/d837c4af1282e51667ed75a02433f5e5590f1c6d/lib/data/NUMACC3.DAT | `66df1b96596735936f37fd889e7c1c6ce998eb2b523bfc46c98ec1737e992aec` | Public domain (US Government work, NIST) | | d | file | https://raw.githubusercontent.com/usnistgov/dataplot/d837c4af1282e51667ed75a02433f5e5590f1c6d/lib/data/PIDIGITS.DAT | `e9421a2f4c0a31c505099332b4d22e507e808953c3c57bddae9226f25587d7d2` | Public domain (US Government work, NIST) | | d | file | https://raw.githubusercontent.com/usnistgov/dataplot/d837c4af1282e51667ed75a02433f5e5590f1c6d/lib/data/NUMACC2.DAT | `5be29522dfbe299541f0ec98ec1c22a51a5587bf06561ba21a92c209fb7ef0f9` | Public domain (US Government work, NIST) | | d | file | https://raw.githubusercontent.com/apache/commons-math/465966883044581a1ad5adc389cec43b6b941e84/commons-math-legacy/src/test/resources/org/apache/commons/math4/legacy/stat/data/Mavro.txt | `dece6cbf88cc99d465df6796d5a0b75ec6ab16ce4f0b0347d27693dff6f0a62d` | Public domain (US Government work, NIST) | | d | file | https://web.archive.org/web/20010211025406id_/http://www.itl.nist.gov/div898/strd/univ/data/Lew.dat | `53cdc7d066ed1dd726528b4bdd8c51034d37fff6709195de8a0013e5131340e4` | Public domain (US Government work, NIST) | | d | file | https://raw.githubusercontent.com/apache/commons-math/465966883044581a1ad5adc389cec43b6b941e84/commons-math-legacy/src/test/resources/org/apache/commons/math4/legacy/stat/data/Michelso.txt | `99b853f9ad5144a28b13ec94824c52ca62ff38e22070e08e0451a746418a82af` | Public domain (US Government work, NIST) | Published values quoted for verification are in `sources/`, each with its locator and an excerpt of at most 15 words; a benchmark's own gold is quoted from its pinned release with its licence. ## Computational requirements `env/platform.json` lists the Python version, the operating system and architecture, the BLAS, `uv --version`, the sha256 of `uv.lock` and every installed package as name==version. Tool replay (`magi-bench verify`) needs the repository at the commit above and `uv sync --frozen --all-packages`. ## Description of the files - `manifest.json`: what was run, by which code, agent and model, with which environment, at what cost. - `results.jsonl`: one task result per task, epoch and attempt, sorted. - `specs/`: each task exactly as it ran (canonical JSON; its sha256 is the result's `task_sha256`). - `traces/`: the raw records of every call (the source of truth); `trajectories/`: the event view derived from them. - `attachments/`: prompts, tool declarations, vendor contents and raw responses, each named by the sha256 of its canonical bytes. - `sources/`, `native/`, `tools/registry.json`, `env/platform.json`. - `SHA256SUMS`: the sha256 of every other file; its own sha256 is the run digest. Canonical JSON: keys sorted, no whitespace, ASCII, floats as Python's `repr`, non-finite numbers as the strings `"NaN"`, `"Infinity"` and `"-Infinity"`. ## Instructions to replicators 1. Integrity, without the repository: `python3 verify_bundle.py` (standard library only). 2. With the repository at the commit: `uv sync --frozen --all-packages`, then `uv run magi-bench verify ` replays the tool calls and rescores every task. ## List of tasks - `nist_univariate.lew.agent` (T2, agent, split public) - `nist_univariate.lew.tool` (T1, tool, split public) - `nist_univariate.lottery.agent` (T2, agent, split public) - `nist_univariate.lottery.tool` (T1, tool, split public) - `nist_univariate.mavro.agent` (T2, agent, split public) - `nist_univariate.mavro.tool` (T1, tool, split public) - `nist_univariate.michelso.agent` (T2, agent, split public) - `nist_univariate.michelso.tool` (T1, tool, split public) - `nist_univariate.numacc1.agent` (T2, agent, split public) - `nist_univariate.numacc1.tool` (T1, tool, split public) - `nist_univariate.numacc2.agent` (T2, agent, split public) - `nist_univariate.numacc2.tool` (T1, tool, split public) - `nist_univariate.numacc3.agent` (T2, agent, split public) - `nist_univariate.numacc3.tool` (T1, tool, split public) - `nist_univariate.numacc4.agent` (T2, agent, split public) - `nist_univariate.numacc4.tool` (T1, tool, split public) - `nist_univariate.pidigits.agent` (T2, agent, split public) - `nist_univariate.pidigits.tool` (T1, tool, split public)