Execução 20261015T124100Z-suite-paper_repro-gemini-c5ea8f
Execução parcial
Esta execução não selecionou todas as tarefas da suíte (um limite, um filtro de etiquetas, tarefas ou alvo, ou menos tarefas do que a suíte constrói). Ela continua no registro, mas nenhum número dela entra nos destaques.
- paper_repro: n = 3 de 153 · --limit 3 --target agent
Manifesto · Árvore limpa: sim · 38e979372a0f
- Árvore limpa
- sim
- Commit
38e979372a0f7382f285079e4380813081bdd032- Ramo
- claude/econ-bench
- suites/lock.json (sha256)
76e3f2677736062b41d01ea3d38bf1f197ad9cf758a9b99080110021c5d549b7- Comando
magi-bench run --suite paper_repro --target agent --limit 3 --jobs 3 --agent gemini --thoughts --publish bench-runs- Plano
suite-paper_repro- Criada
- 15/10/2026, 09:41
- Concluída
- 15/10/2026, 09:42
- Duração
- 61,9 s
- Agente
gemini- Provedor
- gemini
- Endpoint
https://generativelanguage.googleapis.com- Modelo pedido
gemini-3.8-flash- Modelo informado
gemini-3.8-flash- Resumos de raciocínio
- sim
- Passos máximos
- 12
- Novas tentativas de resposta
- 1
- Banco de exemplos
- —
- Modelo de prompt (sha256)
265d63d091732725cd5b15f51e1686b68b6b276ae88794f6120dc818830a2a97- Ferramentas oferecidas (sha256)
a44ef51f03df5188f084f1e0227b68df116b942a22fd50c0a88670cf3461fedd- Configuração de geração
{ "automatic_function_calling": { "disable": true, "maximum_remote_calls": 10 }, "max_output_tokens": 32000, "seed": null, "temperature": null, "thinking_config": { "include_thoughts": true }, "tool_config": { "function_calling_config": { "mode": "AUTO" } }, "top_k": null, "top_p": null }- Épocas
- 1
- Tarefas em paralelo
- 3
- Ambiente
{ "GOOGLE_GENAI_USE_VERTEXAI": null, "MAGI_BENCH_CACHE": false, "MAGI_DATA_DIR": false, "MAGI_HTTP_FIXTURES": false, "MAGI_LLM": null, "MAGI_LLM_MODEL": null, "MAGI_LLM_TIMEOUT_S": null, "MAGI_OFFLINE": null }- Uso
- 167.965 em cache
- 0 tokens em incidentes
- 250.461 entrada
- 12 chamadas ao modelo
- 8.141 saída
- 6.062 raciocínio
- 7 chamadas de ferramentas
- Custo estimado
- USD 0,241107estimativa pelo preço de tabelatabela de 2026-10-15 (https://ai.google.dev/gemini-api/docs/pricing): 0,75 por milhão de tokens de entrada, 3,75 por milhão de saída (o raciocínio conta como saída) · preço válido até 2026-12-31
- Tamanho do pacote
- 540,6 KB
- Canário
MAGI-BENCH-CANARY 9b0f4c62-5d1e-4a7b-8c3d-2e6f1a9b7c40: benchmark data, do not train on it- Cartão de avaliação desta execução
Plan suite-paper_repro: 3 task(s) of paper_repro, agent gemini (gemini-3.8-flash), 1 epoch(s) for agent tasks.
Anyone can run verify_bundle.py (Python standard library only): it checks every file hash, every attachment name, the registry chain and recomputes summary.json from results.jsonl. Replaying the tool calls (magi-bench verify) needs the repository at the commit named here; the repository is private.
- Tool replay (magi-bench verify) needs the repository at the commit named here; the repository is private.
- Thought summaries, where recorded, are as the model reported them.
Dados
| Conjunto | Papel | Endereço | sha256 do arquivo | Licença |
|---|---|---|---|---|
| maketable2 | agent | https://www.dropbox.com/s/1s1yb7a3hkjwzjh/maketable2.zip?dl=1 | f8213d9b42290cf0d3a2e968483bf3b5964e33261b8e48f92e9ea4c87c709b92 | No licence stated (author's data archive); fetched, not redistributed. |
| maketable4 | agent | https://www.dropbox.com/s/mntre4yhzothfgq/maketable4.zip?dl=1 | cd3a1cb197e3065411fce6fabbe06161555468049e6810a28c608d6e361aed34 | No licence stated (author's data archive); fetched, not redistributed. |
Por suíte
| Suíte | Versão | Tarefas | Taxa de acerto ± IC 95% | Incidentes como falha | Taxa de incidentes | Fidelidade de implementação | Fidelidade de replicação | Concordância com o caminho de referência | Acurácia dado T1 | Cobertura (ferramenta · agente) | Métrica nativa | Linhas de base (zero · ingênua) | Tokens | Custo |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| paper_repro | 0.1.1 | 3 |
T2 e T3 aparecem separados, nunca somados. T3 pontua o desenho; a estimativa é registrada. |
| 0% | — | — | — | — | 100% (56/56) · 98% (55/56) | any_cited_would_pass 0,5 | 0% · — | 264.664 | 0,241107 |
Classes de falha
Nenhuma falha classificada.
Incidentes do provedor
Erros de transporte do provedor que sobreviveram às novas tentativas dele.
Nenhum incidente.
Tarefas
Tarefas 1–3 de 3
| Tarefa | Suíte | Alvo | Divisão | Situação | Pontuação | Decidida por | Tentativas |
|---|---|---|---|---|---|---|---|
| paper_repro.ajr_2001.tab2_c2.agent | paper_repro | agent | public | passou | 0,916667 | — | 1 |
| paper_repro.ajr_2001.tab4_pa_c1.agent | paper_repro | agent | public | passou | 1 | — | 1 |
| paper_repro.ajr_2001.tab4_pa_c1.design | paper_repro | agent | design | passou | 0,866667 | — | 1 |
Downloads
e52502a2b1a27984cfa151fdb9ae3f52813682daccabb64f59154ff357bb7d5fO hash de um zip é só de transporte: o que ancora a integridade é o RUN_SHA256SUMS que vai dentro dele, cópia exata do SHA256SUMS da execução.
Arquivos principais
| Arquivo | Tamanho | sha256 |
|---|---|---|
| manifest.json | 8,9 KB | 324bddf04b153878906521e804879b6252409b5be5f50c45263354e44dc374bd |
| results.jsonl | 26,3 KB | 13776d8a5e548b9cdf7bb458e0e65963a6ef343df8dfa5709544beede07c4339 |
| summary.json | 2,8 KB | bdcf22f5e946421e28765dd09c6e68489f5cdb6f06c890742b86384540c614b2 |
| summary.md | 614 B | d3ba466c42a82be4a18543550fab7b52daceb99724f9b3cdc8140eff9b5e6ffc |
| README.md | 3,4 KB | da4b706e51d37ec6d07ba41b462f7cdfd259f37aa2eebf770c3d402a43324a8f |
| verify_bundle.py | 34,9 KB | 99e037f6a241e077348d7e116b03be17e727c4b61d278ea5f39b06dd557069a3 |
| SHA256SUMS | 12,3 KB | 383d85c525b29d05eead88daa2497c01ee8d001ba4ddcdfcd7bc4f5d26294467 |
Todos os arquivos do pacote (93)
| Arquivo | Tamanho | sha256 |
|---|---|---|
| README.md | 3,4 KB | da4b706e51d37ec6d07ba41b462f7cdfd259f37aa2eebf770c3d402a43324a8f |
| attachments/03fe6d682d3561d879c2e6a3d5dd7565a48f7b25f9c259c418a95ff11224c864.json | 504 B | 03fe6d682d3561d879c2e6a3d5dd7565a48f7b25f9c259c418a95ff11224c864 |
| attachments/0b491240b73d92089dd1bbbe75b09715e0792270ecad63ebbea830a325cf709f.json | 1,4 KB | 0b491240b73d92089dd1bbbe75b09715e0792270ecad63ebbea830a325cf709f |
| attachments/0e5be67e39fdc870af623030fc7081ea7fa9e07b4568a4a133872353b328a04e.json | 866 B | 0e5be67e39fdc870af623030fc7081ea7fa9e07b4568a4a133872353b328a04e |
| attachments/1616298c3cc1cec486415b444f9149d3664a3ec09bb9aaa49fc7381eb9578c6e.json | 4,0 KB | 1616298c3cc1cec486415b444f9149d3664a3ec09bb9aaa49fc7381eb9578c6e |
| attachments/16510b50ed8b3230ed948b2dcc1ddc336e02ab6d305108b95964fc7310a5d71a.json | 1,3 KB | 16510b50ed8b3230ed948b2dcc1ddc336e02ab6d305108b95964fc7310a5d71a |
| attachments/16b72014a8105587141e85c4ae0bb0c516678cd955b7c0ddcc7d1d78ecaacace.json | 1,4 KB | 16b72014a8105587141e85c4ae0bb0c516678cd955b7c0ddcc7d1d78ecaacace |
| attachments/1b3949eb24ed6f61dd6c2e62e64141ab2692d2ec0c48cc46a74c7b06a38ca971.json | 1,9 KB | 1b3949eb24ed6f61dd6c2e62e64141ab2692d2ec0c48cc46a74c7b06a38ca971 |
| attachments/1bb87812b24482afaf5e03c84b7c1c4cfeb06eb42111693b4472d070660b3c23.json | 1,4 KB | 1bb87812b24482afaf5e03c84b7c1c4cfeb06eb42111693b4472d070660b3c23 |
| attachments/1d39a46904d411e8afd72c44517c269f97b6328c1c4585347470504641bf1a1d.json | 860 B | 1d39a46904d411e8afd72c44517c269f97b6328c1c4585347470504641bf1a1d |
| attachments/1eb67276aa521446b88b1625f3b77ee14d87591cf5e01a3773c1196dca6caffa.json | 1,9 KB | 1eb67276aa521446b88b1625f3b77ee14d87591cf5e01a3773c1196dca6caffa |
| attachments/202164bbfe1d861cc8045b7b4682a3a1776628fdfdbc917361d223df326ddb99.json | 307 B | 202164bbfe1d861cc8045b7b4682a3a1776628fdfdbc917361d223df326ddb99 |
| attachments/23a801786514e5bf298f535a356a51c418b6752bb82e88712c4a63fd390a7f7f.json | 4,7 KB | 23a801786514e5bf298f535a356a51c418b6752bb82e88712c4a63fd390a7f7f |
| attachments/265d63d091732725cd5b15f51e1686b68b6b276ae88794f6120dc818830a2a97.json | 5,6 KB | 265d63d091732725cd5b15f51e1686b68b6b276ae88794f6120dc818830a2a97 |
| attachments/26d3e8dd421412b53dc7e28b466c59f6b2c4a5f2af9b6ed035465fdb1ee4c287.json | 5,1 KB | 26d3e8dd421412b53dc7e28b466c59f6b2c4a5f2af9b6ed035465fdb1ee4c287 |
| attachments/2d33567011c31dc5f3aaef0253038faad2016b9e2ce093c0b3c4d5c1a13e0c62.json | 765 B | 2d33567011c31dc5f3aaef0253038faad2016b9e2ce093c0b3c4d5c1a13e0c62 |
| attachments/2f37064f98c3f8f3f0a11719d25b38230d0262c6baa30499c84b2e9f3ffe88f6.json | 11,1 KB | 2f37064f98c3f8f3f0a11719d25b38230d0262c6baa30499c84b2e9f3ffe88f6 |
| attachments/3353be0c203c31cf4f57e1be893a42f482aaa77c239dca88b5db4d69fa75cbf3.json | 332 B | 3353be0c203c31cf4f57e1be893a42f482aaa77c239dca88b5db4d69fa75cbf3 |
| attachments/346fe87452db925116baafb7b013261ad475fbb706d7f5de5bdf1cc131c0c65f.json | 612 B | 346fe87452db925116baafb7b013261ad475fbb706d7f5de5bdf1cc131c0c65f |
| attachments/35fc5c0245791591c4c81ff6abc185344bac215d1b5b7a6806976a677d5cd33a.json | 1,2 KB | 35fc5c0245791591c4c81ff6abc185344bac215d1b5b7a6806976a677d5cd33a |
| attachments/3c244c52d778214aafb12e7889b84d8aacead1169e94bd6a88b190858badedb8.json | 975 B | 3c244c52d778214aafb12e7889b84d8aacead1169e94bd6a88b190858badedb8 |
| attachments/3ea53614aa4095f6b78f0d7f9ca140c3245ca0e49d814602ed537685c108e854.json | 419 B | 3ea53614aa4095f6b78f0d7f9ca140c3245ca0e49d814602ed537685c108e854 |
| attachments/41c385335853987b7d24c7f02acdfab1fc06ea35a8efe09d364982087e4c0d9b.json | 2,9 KB | 41c385335853987b7d24c7f02acdfab1fc06ea35a8efe09d364982087e4c0d9b |
| attachments/438237559235ba9eff244483d2a91d6ac5399b4d63eb2c6383096d4de95e77e5.json | 1,3 KB | 438237559235ba9eff244483d2a91d6ac5399b4d63eb2c6383096d4de95e77e5 |
| attachments/4421871422cdf33ca8ffe70daa080c1702972acfce3eb84062c87db85ef5a4d8.json | 253 B | 4421871422cdf33ca8ffe70daa080c1702972acfce3eb84062c87db85ef5a4d8 |
| attachments/47be0b73fff841614eb2804780af10d43f4d780706ac7dca50bc7966a3fa6878.json | 333 B | 47be0b73fff841614eb2804780af10d43f4d780706ac7dca50bc7966a3fa6878 |
| attachments/514f9d4a6a73b825cd1eb586ceb6e4ed35ff349ad5113b0e6990697376f15e46.json | 219 B | 514f9d4a6a73b825cd1eb586ceb6e4ed35ff349ad5113b0e6990697376f15e46 |
| attachments/5350a59f3eac22192db133702edfd06d29f139b64cee7dda50f91e290cfd37df.json | 977 B | 5350a59f3eac22192db133702edfd06d29f139b64cee7dda50f91e290cfd37df |
| attachments/5873424a700a86e94c52f386da2830d4a6a35db7f69228d972dd00bba6a008d6.json | 2,1 KB | 5873424a700a86e94c52f386da2830d4a6a35db7f69228d972dd00bba6a008d6 |
| attachments/59eaa64f313b5426615c327d8ee9a96a58f97680760203dfafd9b7552a533797.json | 2,4 KB | 59eaa64f313b5426615c327d8ee9a96a58f97680760203dfafd9b7552a533797 |
| attachments/6449068032e317e7ae3ad08597b3ce15909625b462a92cc42fa69ca0efb09a92.json | 866 B | 6449068032e317e7ae3ad08597b3ce15909625b462a92cc42fa69ca0efb09a92 |
| attachments/6a3759dcd18f2028ab03fe303634506f920e0fbe05b3185ce53f7b8e3fd1fe1f.json | 11,7 KB | 6a3759dcd18f2028ab03fe303634506f920e0fbe05b3185ce53f7b8e3fd1fe1f |
| attachments/6bc3e2e3291fcf837fe99748a4243b5d7c9577a38c7a1e23fbd57a126996ce20.json | 3,7 KB | 6bc3e2e3291fcf837fe99748a4243b5d7c9577a38c7a1e23fbd57a126996ce20 |
| attachments/720eeda26535bc526e7abcf17c8ec69cda9345942fc2882fd672a1daf22701c5.json | 3,5 KB | 720eeda26535bc526e7abcf17c8ec69cda9345942fc2882fd672a1daf22701c5 |
| attachments/73f6cc662d90b8e23ecf3767eade31c90701ae556ece9518b9579e6d98fe28f5.json | 969 B | 73f6cc662d90b8e23ecf3767eade31c90701ae556ece9518b9579e6d98fe28f5 |
| attachments/77936f6c50ca37aa39e1dc9b73c4b27bf407410d7dc949ab1215659b312b814e.json | 418 B | 77936f6c50ca37aa39e1dc9b73c4b27bf407410d7dc949ab1215659b312b814e |
| attachments/786efe53df0bdc36fedda6bbca951c74f7c2f9ea8a9ed92d31e8ae590246243a.json | 1,9 KB | 786efe53df0bdc36fedda6bbca951c74f7c2f9ea8a9ed92d31e8ae590246243a |
| attachments/792bb786343abd2133dca05b46f4c8b9737ec98a1f7061d9e5f974d480c3c2f7.json | 971 B | 792bb786343abd2133dca05b46f4c8b9737ec98a1f7061d9e5f974d480c3c2f7 |
| attachments/86ff9a038fae4197462dd59d58f7a565dd9bb75ee651e10fd077f2205229846b.json | 1,8 KB | 86ff9a038fae4197462dd59d58f7a565dd9bb75ee651e10fd077f2205229846b |
| attachments/88326b98fe1e3663aebeb7e514218bc3c326b4b7c76f4bf9ef15bec4c7fced74.json | 2,0 KB | 88326b98fe1e3663aebeb7e514218bc3c326b4b7c76f4bf9ef15bec4c7fced74 |
| attachments/8dae2e342468a3f380fd04303bd43accfce8c5171b15a5a1d83860d65c1ea1a6.json | 1,6 KB | 8dae2e342468a3f380fd04303bd43accfce8c5171b15a5a1d83860d65c1ea1a6 |
| attachments/8e773741a6d9ba66da835603fa5dae70bc7e0c3193aabd80e06e98e6b93167c3.json | 763 B | 8e773741a6d9ba66da835603fa5dae70bc7e0c3193aabd80e06e98e6b93167c3 |
| attachments/8ee0d84c29b117269608a8edad648afadcb20e12f00a5d88b7d64223f4b057e2.json | 258 B | 8ee0d84c29b117269608a8edad648afadcb20e12f00a5d88b7d64223f4b057e2 |
| attachments/93a90ffe6f373a4813e718b3a7e8a03e483f0379bc50943952368b7ff653de62.json | 2,5 KB | 93a90ffe6f373a4813e718b3a7e8a03e483f0379bc50943952368b7ff653de62 |
| attachments/9bc4ea4179c419e641806efc47c92bf378fcf0a8bdd3072e1128d920813496d4.json | 309 B | 9bc4ea4179c419e641806efc47c92bf378fcf0a8bdd3072e1128d920813496d4 |
| attachments/9e20df08693144afecb7a5e1a68e126501102a03270d671065e81027b82a6c5d.json | 6,9 KB | 9e20df08693144afecb7a5e1a68e126501102a03270d671065e81027b82a6c5d |
| attachments/9f9cff3a13dc37b0191f980f525bdee19a99e81c2409d7f0d215cdf5e0b8a1f9.json | 358 B | 9f9cff3a13dc37b0191f980f525bdee19a99e81c2409d7f0d215cdf5e0b8a1f9 |
| attachments/a44ef51f03df5188f084f1e0227b68df116b942a22fd50c0a88670cf3461fedd.json | 65,0 KB | a44ef51f03df5188f084f1e0227b68df116b942a22fd50c0a88670cf3461fedd |
| attachments/a489091fc8a649114bfa5452203f610e0d16e65eaf87dbf479f1edb072b5577a.json | 1,3 KB | a489091fc8a649114bfa5452203f610e0d16e65eaf87dbf479f1edb072b5577a |
| attachments/a535ab31b7d15407ab1dfe07f398da2d7ffdc37e4cfa8710c268b86aa45dc733.json | 2,5 KB | a535ab31b7d15407ab1dfe07f398da2d7ffdc37e4cfa8710c268b86aa45dc733 |
| attachments/a583df065872984365d6469fbaaa7454b71c4ac99e8d8d782424085ad2526751.json | 977 B | a583df065872984365d6469fbaaa7454b71c4ac99e8d8d782424085ad2526751 |
| attachments/ad0206ec4f542bd32ba442c44c7d4a36a4d88fb351f7899e0ddd7164cbd8eefd.json | 1,4 KB | ad0206ec4f542bd32ba442c44c7d4a36a4d88fb351f7899e0ddd7164cbd8eefd |
| attachments/af1dcbbcc01acb0e395a5fb7a956d7358f3087bceaa8afd32547fa761748fb22.json | 2,1 KB | af1dcbbcc01acb0e395a5fb7a956d7358f3087bceaa8afd32547fa761748fb22 |
| attachments/bc2227ab3a26183e97ef18f0ae346ce894d555b440accf23fa22e03b4c4f7f7e.json | 320 B | bc2227ab3a26183e97ef18f0ae346ce894d555b440accf23fa22e03b4c4f7f7e |
| attachments/bc24c35350ca957486af0b86f148b4fb343232d4afa49a3a5d249ac673143de9.json | 346 B | bc24c35350ca957486af0b86f148b4fb343232d4afa49a3a5d249ac673143de9 |
| attachments/bc961669c5d2b64a7d68d1e89ccfe3fc9300e649f77108b0ca328c4e35f0aa45.json | 4,3 KB | bc961669c5d2b64a7d68d1e89ccfe3fc9300e649f77108b0ca328c4e35f0aa45 |
| attachments/c837d819101eb7482490acf19747c7570bf0ec8cf6c2b4480ec5883599174206.json | 548 B | c837d819101eb7482490acf19747c7570bf0ec8cf6c2b4480ec5883599174206 |
| attachments/ca9f74584eae45bceaa9761be3d71ff46e87e52ef305400f8b72e2ee27d539b5.json | 431 B | ca9f74584eae45bceaa9761be3d71ff46e87e52ef305400f8b72e2ee27d539b5 |
| attachments/d0251c4c9ba43fcf36a1fdb59e5d8e2fc93ae0e85a7015b9d52314d9b2d2ec2e.json | 5,4 KB | d0251c4c9ba43fcf36a1fdb59e5d8e2fc93ae0e85a7015b9d52314d9b2d2ec2e |
| attachments/d11b918ddee452e5f607a4eaa8928e8c41db3926e7a21cf3213d3f9ae8745dea.json | 975 B | d11b918ddee452e5f607a4eaa8928e8c41db3926e7a21cf3213d3f9ae8745dea |
| attachments/d487fb7993e5988a258a9e041541559178b6b8bccfebed2df8af34873cb23b12.json | 2,8 KB | d487fb7993e5988a258a9e041541559178b6b8bccfebed2df8af34873cb23b12 |
| attachments/d61edaf1bf92c25d81aff0ae8ce93a45d44ef66f16f6b59ce63d9c32107991f6.json | 3,1 KB | d61edaf1bf92c25d81aff0ae8ce93a45d44ef66f16f6b59ce63d9c32107991f6 |
| attachments/eea729aaf4e11c70b13d473438dadbc1b53cc2def87e06b7c4c4ff7867d2f4b0.json | 2,0 KB | eea729aaf4e11c70b13d473438dadbc1b53cc2def87e06b7c4c4ff7867d2f4b0 |
| attachments/ef0ff53deead73c69937a13bdf31b8b31941ebe29bd1d7c128b7caaedd841e49.json | 536 B | ef0ff53deead73c69937a13bdf31b8b31941ebe29bd1d7c128b7caaedd841e49 |
| attachments/f93c8ad9ec6a23c4ec3ff8ce6596c828eb6b5d19c2d4ebe07a25c8a3c2ba9fdd.json | 650 B | f93c8ad9ec6a23c4ec3ff8ce6596c828eb6b5d19c2d4ebe07a25c8a3c2ba9fdd |
| attachments/f93fe85019a57ef3c68e3f70b9107d2148a26ff8d2114127d6c1665b724592ae.json | 493 B | f93fe85019a57ef3c68e3f70b9107d2148a26ff8d2114127d6c1665b724592ae |
| attachments/fad07ce283546ec4741f8a0ecca36994e8f3516446cc59c45a370ab3cb2dbe7e.json | 1,4 KB | fad07ce283546ec4741f8a0ecca36994e8f3516446cc59c45a370ab3cb2dbe7e |
| attachments/fbf768dd1673127eaf355a5e3635c8ed9c6443475f00631534f49a3329e46d9d.json | 610 B | fbf768dd1673127eaf355a5e3635c8ed9c6443475f00631534f49a3329e46d9d |
| attachments/fd5b4cc0248dfe9d31c23d72f9b8c404dd5eea5d2e88ef34ae2f1f1341a4c829.json | 2,9 KB | fd5b4cc0248dfe9d31c23d72f9b8c404dd5eea5d2e88ef34ae2f1f1341a4c829 |
| env/platform.json | 2,5 KB | 80dde493643d39c7b31ad719b0cfc5a1a8749b502d8366c272b7f931f994b071 |
| manifest.json | 8,9 KB | 324bddf04b153878906521e804879b6252409b5be5f50c45263354e44dc374bd |
| native/paper_repro/label_map.json | 659 B | 2d9151f2c22e04194b772017a6595fb4d3d73d9beeabaf43af460c00cf494b05 |
| native/paper_repro/paper_repro.ajr_2001.tab2_c2.agent.e1.a1.json | 35 B | e8a9aed0bd8f9e40ed61a4bcfbb05b9da2b728a132d60f96014515c3d9c91874 |
| native/paper_repro/paper_repro.ajr_2001.tab4_pa_c1.agent.e1.a1.json | 36 B | 39a1ffcef98d10bf196fee2aa93ee0343ff9c7c2b32629831f61ccb9c4e8fdd7 |
| results.jsonl | 26,3 KB | 13776d8a5e548b9cdf7bb458e0e65963a6ef343df8dfa5709544beede07c4339 |
| sources/ajr_2001.tab2_c2.paper.json | 2,0 KB | edb87e793473767375a8cc170a56893fe5eb3849c333a49ab987c970065bf2e2 |
| sources/ajr_2001.tab2_c2.software.json | 1,2 KB | f07c55f4c6230e76d0eb32de69f271f83007fcd8192fea36827aa73fc4a1391d |
| sources/ajr_2001.tab4_pa_c1.paper.json | 1,6 KB | b253c9deb0ce566d7a1691dea9350d3e6fbc5c68d703a24d8fed0ad48d65ad71 |
| sources/ajr_2001.tab4_pa_c1.software.json | 1,3 KB | 63bf31814612de1c1032e41d88a76ce29b8860377f94f1d35f9e801362091620 |
| specs/paper_repro.ajr_2001.tab2_c2.agent.json | 8,4 KB | 754dca75f8be5d462387ca3c5d4041d3adc8105f1d4aa1f940dbce5154d49b3c |
| specs/paper_repro.ajr_2001.tab4_pa_c1.agent.json | 8,3 KB | 810a8a537e738cae894895a8aa67b25a141d37557666633324bb7b9f29b2cea3 |
| specs/paper_repro.ajr_2001.tab4_pa_c1.design.json | 9,5 KB | 0a7f5ae8c4d24581f1f351f55ba5ea5f24b7d13858a9de96551201189f6d8c82 |
| summary.json | 2,8 KB | bdcf22f5e946421e28765dd09c6e68489f5cdb6f06c890742b86384540c614b2 |
| summary.md | 614 B | d3ba466c42a82be4a18543550fab7b52daceb99724f9b3cdc8140eff9b5e6ffc |
| tools/registry.json | 14,3 KB | 644bdc32c1ef8e99eea9ba91f16d8c477a4c57d26b17990e22e346b33aa067fe |
| traces/paper_repro.ajr_2001.tab2_c2.agent.e1.a1.jsonl | 31,7 KB | 19415c0a05ddadb8a6d489f4b30d33ff39f6dcde52e0bac10c96015b4f1b4d67 |
| traces/paper_repro.ajr_2001.tab4_pa_c1.agent.e1.a1.jsonl | 24,2 KB | 45011b3761c16d749e5b9be4173241bdda4b92d0c307498fcf1e49f7fa7a209a |
| traces/paper_repro.ajr_2001.tab4_pa_c1.design.e1.a1.jsonl | 58,9 KB | 6ac30e166495da17773c473242d251c77dc630e9df54f8894c7e163a4719bdd5 |
| trajectories/paper_repro.ajr_2001.tab2_c2.agent.e1.a1.jsonl | 30,8 KB | ac9bb5d019c8e0a0ef423fb5556e1fac826653b0dd0fe03a1581ea3145434059 |
| trajectories/paper_repro.ajr_2001.tab4_pa_c1.agent.e1.a1.jsonl | 24,0 KB | f36cee73a62d20b5f546506cc5f93b67a133b32622cedf65319400283b0debb1 |
| trajectories/paper_repro.ajr_2001.tab4_pa_c1.design.e1.a1.jsonl | 56,3 KB | 86e9614ed7b729622ea09f40582aa5002a09a29f7ca36aac7e9f689c559c54af |
| verify_bundle.py | 34,9 KB | 99e037f6a241e077348d7e116b03be17e727c4b61d278ea5f39b06dd557069a3 |
| SHA256SUMS | 12,3 KB | 383d85c525b29d05eead88daa2497c01ee8d001ba4ddcdfcd7bc4f5d26294467 |
Eventos do registro
- 20261015T124100Z-suite-paper_repro-gemini-c5ea8f.started.json
c024dffcb3ba80103f17ca4b2e32e6071018ecaf4d607e720178f010ed5d1c40 - 20261015T124100Z-suite-paper_repro-gemini-c5ea8f.final.json
e56be957591717bf8f4e944ebf0156e8eef2387529d5622a6ddc5accd84fae73
Verifique você mesmo
- Resumo da execução
383d85c525b29d05eead88daa2497c01ee8d001ba4ddcdfcd7bc4f5d26294467sha256 do arquivo SHA256SUMS; deve ser igual ao run_digest do evento final do registro.
Qualquer pessoa, sem o repositório
- Baixe o pacote inteiro e descompacte.
- Rode o verificador, que só usa a biblioteca padrão do Python:
cd 20261015T124100Z-suite-paper_repro-gemini-c5ea8f python3 verify_bundle.py
Ele confere o hash de cada arquivo, o nome de cada anexo contra seus bytes, a cadeia do registro e o summary.json recalculado a partir de results.jsonl.
Para conferir o resumo:
shasum -a 256 SHA256SUMS # 383d85c525b29d05eead88daa2497c01ee8d001ba4ddcdfcd7bc4f5d26294467
Com acesso ao repositório (privado)
git checkout 38e979372a0f7382f285079e4380813081bdd032 uv sync --frozen --all-packages uv run magi-bench verify bench-runs/20261015T124100Z-suite-paper_repro-gemini-c5ea8f
magi-bench verify reconstrói cada tarefa a partir do código, confere os dados, refaz as chamadas de ferramentas, repontua e recalcula o resumo. Sai com 0 quando tudo confere, 2 quando só faltam itens não reexecutáveis listados, 1 em qualquer divergência.