# MAGI benchmark run 20261015T124100Z-suite-paper_repro-gemini-c5ea8f

Status: `success`.

## Overview

A run of the plan `suite-paper_repro` at commit `38e979372a0f7382f285079e4380813081bdd032` (tree clean: yes) with the agent `gemini` (`gemini-3.8-flash`):
3 task(s) of `paper_repro` 0.1.1, 3 result line(s) (one per task, epoch and attempt). The per-suite
numbers are in `summary.json`, rendered as `summary.md`; each is defined by the code of `verify_bundle.py`,
which recomputes them from `results.jsonl`.

Anyone can run verify_bundle.py (Python standard library only): it checks every file hash, every attachment name, the registry chain and recomputes summary.json from results.jsonl. Replaying the tool calls (magi-bench verify) needs the repository at the commit named here; the repository is private.

## Data availability and provenance statements

No third-party data file is redistributed. Each dataset is fetched from its publisher at a pinned URL and
verified by the sha256 of its bytes, or regenerated from a seeded generator (`manifest.json` → `data`):

| dataset | kind | URL | sha256 of the file | licence |
|---|---|---|---|---|
| maketable2 | file | https://www.dropbox.com/s/1s1yb7a3hkjwzjh/maketable2.zip?dl=1 | `f8213d9b42290cf0d3a2e968483bf3b5964e33261b8e48f92e9ea4c87c709b92` | No licence stated (author's data archive); fetched, not redistributed. |
| maketable4 | file | https://www.dropbox.com/s/mntre4yhzothfgq/maketable4.zip?dl=1 | `cd3a1cb197e3065411fce6fabbe06161555468049e6810a28c608d6e361aed34` | No licence stated (author's data archive); fetched, not redistributed. |

Published values quoted for verification are in `sources/`, each with its locator and an excerpt of at most
15 words; a benchmark's own gold is quoted from its pinned release with its licence.

## Computational requirements

`env/platform.json` lists the Python version, the operating system and architecture, the BLAS, `uv
--version`, the sha256 of `uv.lock` and every installed package as name==version. Tool replay
(`magi-bench verify`) needs the repository at the commit above and `uv sync --frozen --all-packages`.

## Description of the files

- `manifest.json`: what was run, by which code, agent and model, with which environment, at what cost.
- `results.jsonl`: one task result per task, epoch and attempt, sorted.
- `specs/`: each task exactly as it ran (canonical JSON; its sha256 is the result's `task_sha256`).
- `traces/`: the raw records of every call (the source of truth); `trajectories/`: the event view derived
  from them.
- `attachments/`: prompts, tool declarations, vendor contents and raw responses, each named by the sha256
  of its canonical bytes.
- `sources/`, `native/`, `tools/registry.json`, `env/platform.json`.
- `SHA256SUMS`: the sha256 of every other file; its own sha256 is the run digest.

Canonical JSON: keys sorted, no whitespace, ASCII, floats as Python's `repr`, non-finite numbers as the
strings `"NaN"`, `"Infinity"` and `"-Infinity"`.

## Instructions to replicators

1. Integrity, without the repository: `python3 verify_bundle.py` (standard library only).
2. With the repository at the commit: `uv sync --frozen --all-packages`, then
   `uv run magi-bench verify <this folder>` replays the tool calls and rescores every task.

## List of tasks

- `paper_repro.ajr_2001.tab2_c2.agent` (T2, agent, split public)
- `paper_repro.ajr_2001.tab4_pa_c1.agent` (T2, agent, split public)
- `paper_repro.ajr_2001.tab4_pa_c1.design` (T3, agent, split design)