# MAGI benchmark run 20261015T171252Z-suite-strd-scripted-d8a322

Status: `success`.

## Overview

A run of the plan `suite-strd` at commit `2925aeab226272b243436386a1c4e01d97c1377e` (tree clean: yes) with the agent `scripted`:
20 task(s) of `strd` 0.1.0, 20 result line(s) (one per task, epoch and attempt). The per-suite
numbers are in `summary.json`, rendered as `summary.md`; each is defined by the code of `verify_bundle.py`,
which recomputes them from `results.jsonl`.

Anyone can run verify_bundle.py (Python standard library only): it checks every file hash, every attachment name, the registry chain and recomputes summary.json from results.jsonl. Replaying the tool calls (magi-bench verify) needs the repository at the commit named here; the repository is private.

## Data availability and provenance statements

No third-party data file is redistributed. Each dataset is fetched from its publisher at a pinned URL and
verified by the sha256 of its bytes, or regenerated from a seeded generator (`manifest.json` → `data`):

| dataset | kind | URL | sha256 of the file | licence |
|---|---|---|---|---|
| d | generator | — | `—` | — |
| d | generator | — | `—` | — |
| d | generator | — | `—` | — |
| d | generator | — | `—` | — |
| d | generator | — | `—` | — |
| d | generator | — | `—` | — |
| d | generator | — | `—` | — |
| d | generator | — | `—` | — |
| d | generator | — | `—` | — |
| d | generator | — | `—` | — |
| d | generator | — | `—` | — |

Published values quoted for verification are in `sources/`, each with its locator and an excerpt of at most
15 words; a benchmark's own gold is quoted from its pinned release with its licence.

## Computational requirements

`env/platform.json` lists the Python version, the operating system and architecture, the BLAS, `uv
--version`, the sha256 of `uv.lock` and every installed package as name==version. Tool replay
(`magi-bench verify`) needs the repository at the commit above and `uv sync --frozen --all-packages`.

## Description of the files

- `manifest.json`: what was run, by which code, agent and model, with which environment, at what cost.
- `results.jsonl`: one task result per task, epoch and attempt, sorted.
- `specs/`: each task exactly as it ran (canonical JSON; its sha256 is the result's `task_sha256`).
- `traces/`: the raw records of every call (the source of truth); `trajectories/`: the event view derived
  from them.
- `attachments/`: prompts, tool declarations, vendor contents and raw responses, each named by the sha256
  of its canonical bytes.
- `sources/`, `native/`, `tools/registry.json`, `env/platform.json`.
- `SHA256SUMS`: the sha256 of every other file; its own sha256 is the run digest.

Canonical JSON: keys sorted, no whitespace, ASCII, floats as Python's `repr`, non-finite numbers as the
strings `"NaN"`, `"Infinity"` and `"-Infinity"`.

## Instructions to replicators

1. Integrity, without the repository: `python3 verify_bundle.py` (standard library only).
2. With the repository at the commit: `uv sync --frozen --all-packages`, then
   `uv run magi-bench verify <this folder>` replays the tool calls and rescores every task.

## List of tasks

- `strd.filip` (T1, tool, split public)
- `strd.filip.explicit_constant` (T1, tool, split public)
- `strd.longley` (T1, tool, split public)
- `strd.longley.explicit_constant` (T1, tool, split public)
- `strd.noint1` (T1, tool, split public)
- `strd.noint2` (T1, tool, split public)
- `strd.norris` (T1, tool, split public)
- `strd.norris.explicit_constant` (T1, tool, split public)
- `strd.pontius` (T1, tool, split public)
- `strd.pontius.explicit_constant` (T1, tool, split public)
- `strd.wampler1` (T1, tool, split public)
- `strd.wampler1.explicit_constant` (T1, tool, split public)
- `strd.wampler2` (T1, tool, split public)
- `strd.wampler2.explicit_constant` (T1, tool, split public)
- `strd.wampler3` (T1, tool, split public)
- `strd.wampler3.explicit_constant` (T1, tool, split public)
- `strd.wampler4` (T1, tool, split public)
- `strd.wampler4.explicit_constant` (T1, tool, split public)
- `strd.wampler5` (T1, tool, split public)
- `strd.wampler5.explicit_constant` (T1, tool, split public)