# MAGI benchmark run 20261015T170010Z-suite-qrdata_socsci-scripted-7912b3 Status: `success`. ## Overview A run of the plan `suite-qrdata_socsci` at commit `62cdbe0860ad0472e1d2120e93d3c4f29f7688eb` (tree clean: yes) with the agent `scripted` (`reference`): 59 task(s) of `qrdata_socsci` 0.1.2, 59 result line(s) (one per task, epoch and attempt). The per-suite numbers are in `summary.json`, rendered as `summary.md`; each is defined by the code of `verify_bundle.py`, which recomputes them from `results.jsonl`. Anyone can run verify_bundle.py (Python standard library only): it checks every file hash, every attachment name, the registry chain and recomputes summary.json from results.jsonl. Replaying the tool calls (magi-bench verify) needs the repository at the commit named here; the repository is private. ## Data availability and provenance statements No third-party data file is redistributed. Each dataset is fetched from its publisher at a pinned URL and verified by the sha256 of its bytes, or regenerated from a seeded generator (`manifest.json` → `data`): | dataset | kind | URL | sha256 of the file | licence | |---|---|---|---|---| | FLCensusVTD | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | FLVoters | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | FLVoters | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | MPs | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | USGini | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | afghan | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | afghan | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | afghan-village | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | ak91 | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | ak91_iv | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | app_engagement_push | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | billboard_impact | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | collections_email | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | drinking | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | drinking_log | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | face | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | face_margin | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | florentine | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | florida | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | hospital_treatment | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | ice_cream_sales | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | ice_cream_sales_rnd | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | invest_email_rnd_test | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | invest_email_rnd_train | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | learning_mindset | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | medicine_impact_recovery | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | minwage | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | minwage_props | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | names | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | online_classroom | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | polls08 | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | pres08 | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | pres08 | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | pres12 | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | resume | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | resume_white | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | smoking2 | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | social | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | social_neighbors | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | trainee_unique_on_age | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | twitter-following | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | twitter-senator | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | wage | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | wage | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | wage_grad12 | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | wage_hourly | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | | women | file | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | `3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970` | CC BY-NC 4.0 | Published values quoted for verification are in `sources/`, each with its locator and an excerpt of at most 15 words; a benchmark's own gold is quoted from its pinned release with its licence. ## Computational requirements `env/platform.json` lists the Python version, the operating system and architecture, the BLAS, `uv --version`, the sha256 of `uv.lock` and every installed package as name==version. Tool replay (`magi-bench verify`) needs the repository at the commit above and `uv sync --frozen --all-packages`. ## Description of the files - `manifest.json`: what was run, by which code, agent and model, with which environment, at what cost. - `results.jsonl`: one task result per task, epoch and attempt, sorted. - `specs/`: each task exactly as it ran (canonical JSON; its sha256 is the result's `task_sha256`). - `traces/`: the raw records of every call (the source of truth); `trajectories/`: the event view derived from them. - `attachments/`: prompts, tool declarations, vendor contents and raw responses, each named by the sha256 of its canonical bytes. - `sources/`, `native/`, `tools/registry.json`, `env/platform.json`. - `SHA256SUMS`: the sha256 of every other file; its own sha256 is the run digest. Canonical JSON: keys sorted, no whitespace, ASCII, floats as Python's `repr`, non-finite numbers as the strings `"NaN"`, `"Infinity"` and `"-Infinity"`. ## Instructions to replicators 1. Integrity, without the repository: `python3 verify_bundle.py` (standard library only). 2. With the repository at the commit: `uv sync --frozen --all-packages`, then `uv run magi-bench verify ` replays the tool calls and rescores every task. ## List of tasks - `qrdata_socsci.352` (T2, agent, split public) - `qrdata_socsci.353` (T2, agent, split public) - `qrdata_socsci.354` (T2, agent, split public) - `qrdata_socsci.355` (T2, agent, split public) - `qrdata_socsci.356` (T2, agent, split public) - `qrdata_socsci.357` (T2, agent, split public) - `qrdata_socsci.358` (T2, agent, split public) - `qrdata_socsci.359` (T2, agent, split public) - `qrdata_socsci.360` (T2, agent, split public) - `qrdata_socsci.361` (T2, agent, split public) - `qrdata_socsci.362` (T2, agent, split public) - `qrdata_socsci.363` (T2, agent, split public) - `qrdata_socsci.364` (T2, agent, split public) - `qrdata_socsci.365` (T2, agent, split public) - `qrdata_socsci.366` (T2, agent, split public) - `qrdata_socsci.367` (T2, agent, split public) - `qrdata_socsci.368` (T2, agent, split public) - `qrdata_socsci.369` (T2, agent, split public) - `qrdata_socsci.370` (T2, agent, split public) - `qrdata_socsci.371` (T2, agent, split public) - `qrdata_socsci.372` (T2, agent, split public) - `qrdata_socsci.373` (T2, agent, split public) - `qrdata_socsci.374` (T2, agent, split public) - `qrdata_socsci.375` (T2, agent, split public) - `qrdata_socsci.376` (T2, agent, split public) - `qrdata_socsci.377` (T2, agent, split public) - `qrdata_socsci.378` (T2, agent, split public) - `qrdata_socsci.379` (T2, agent, split public) - `qrdata_socsci.380` (T2, agent, split public) - `qrdata_socsci.381` (T2, agent, split public) - `qrdata_socsci.382` (T2, agent, split public) - `qrdata_socsci.383` (T2, agent, split public) - `qrdata_socsci.384` (T2, agent, split public) - `qrdata_socsci.385` (T2, agent, split public) - `qrdata_socsci.386` (T2, agent, split public) - `qrdata_socsci.387` (T2, agent, split public) - `qrdata_socsci.388` (T2, agent, split public) - `qrdata_socsci.389` (T2, agent, split public) - `qrdata_socsci.390` (T2, agent, split public) - `qrdata_socsci.391` (T2, agent, split public) - `qrdata_socsci.392` (T2, agent, split public) - `qrdata_socsci.393` (T2, agent, split public) - `qrdata_socsci.394` (T2, agent, split public) - `qrdata_socsci.395` (T2, agent, split public) - `qrdata_socsci.396` (T2, agent, split public) - `qrdata_socsci.397` (T2, agent, split public) - `qrdata_socsci.398` (T2, agent, split public) - `qrdata_socsci.399` (T2, agent, split public) - `qrdata_socsci.400` (T2, agent, split public) - `qrdata_socsci.401` (T2, agent, split public) - `qrdata_socsci.402` (T2, agent, split public) - `qrdata_socsci.403` (T2, agent, split public) - `qrdata_socsci.404` (T2, agent, split public) - `qrdata_socsci.405` (T2, agent, split public) - `qrdata_socsci.406` (T2, agent, split public) - `qrdata_socsci.407` (T2, agent, split public) - `qrdata_socsci.408` (T2, agent, split public) - `qrdata_socsci.409` (T2, agent, split public) - `qrdata_socsci.410` (T2, agent, split public)