Pular para o conteúdo
Voltar aos benchmarks

Execução 20261015T124100Z-suite-paper_repro-gemini-c5ea8f

concluída15/10/2026, 09:41geminigemini-3.8-flashCadeia: íntegraseleção parcial

Execução parcial

Esta execução não selecionou todas as tarefas da suíte (um limite, um filtro de etiquetas, tarefas ou alvo, ou menos tarefas do que a suíte constrói). Ela continua no registro, mas nenhum número dela entra nos destaques.

  • paper_repro: n = 3 de 153 · --limit 3 --target agent
Manifesto · Árvore limpa: sim · 38e979372a0f
Árvore limpa
sim
Commit
38e979372a0f7382f285079e4380813081bdd032
Ramo
claude/econ-bench
suites/lock.json (sha256)
76e3f2677736062b41d01ea3d38bf1f197ad9cf758a9b99080110021c5d549b7
Comando
magi-bench run --suite paper_repro --target agent --limit 3 --jobs 3 --agent gemini --thoughts --publish bench-runs
Plano
suite-paper_repro
Criada
15/10/2026, 09:41
Concluída
15/10/2026, 09:42
Duração
61,9 s
Agente
gemini
Provedor
gemini
Endpoint
https://generativelanguage.googleapis.com
Modelo pedido
gemini-3.8-flash
Modelo informado
gemini-3.8-flash
Resumos de raciocínio
sim
Passos máximos
12
Novas tentativas de resposta
1
Banco de exemplos
—
Modelo de prompt (sha256)
265d63d091732725cd5b15f51e1686b68b6b276ae88794f6120dc818830a2a97
Ferramentas oferecidas (sha256)
a44ef51f03df5188f084f1e0227b68df116b942a22fd50c0a88670cf3461fedd
Configuração de geração
{
  "automatic_function_calling": {
    "disable": true,
    "maximum_remote_calls": 10
  },
  "max_output_tokens": 32000,
  "seed": null,
  "temperature": null,
  "thinking_config": {
    "include_thoughts": true
  },
  "tool_config": {
    "function_calling_config": {
      "mode": "AUTO"
    }
  },
  "top_k": null,
  "top_p": null
}
Épocas
1
Tarefas em paralelo
3
Ambiente
{
  "GOOGLE_GENAI_USE_VERTEXAI": null,
  "MAGI_BENCH_CACHE": false,
  "MAGI_DATA_DIR": false,
  "MAGI_HTTP_FIXTURES": false,
  "MAGI_LLM": null,
  "MAGI_LLM_MODEL": null,
  "MAGI_LLM_TIMEOUT_S": null,
  "MAGI_OFFLINE": null
}
Uso
  • 167.965 em cache
  • 0 tokens em incidentes
  • 250.461 entrada
  • 12 chamadas ao modelo
  • 8.141 saída
  • 6.062 raciocínio
  • 7 chamadas de ferramentas
Custo estimado
USD 0,241107estimativa pelo preço de tabela
tabela de 2026-10-15 (https://ai.google.dev/gemini-api/docs/pricing): 0,75 por milhão de tokens de entrada, 3,75 por milhão de saída (o raciocínio conta como saída) · preço válido até 2026-12-31
Tamanho do pacote
540,6 KB
Canário
MAGI-BENCH-CANARY 9b0f4c62-5d1e-4a7b-8c3d-2e6f1a9b7c40: benchmark data, do not train on it
Cartão de avaliação desta execução

Plan suite-paper_repro: 3 task(s) of paper_repro, agent gemini (gemini-3.8-flash), 1 epoch(s) for agent tasks.

Anyone can run verify_bundle.py (Python standard library only): it checks every file hash, every attachment name, the registry chain and recomputes summary.json from results.jsonl. Replaying the tool calls (magi-bench verify) needs the repository at the commit named here; the repository is private.

  • Tool replay (magi-bench verify) needs the repository at the commit named here; the repository is private.
  • Thought summaries, where recorded, are as the model reported them.

Dados

Dados
ConjuntoPapelEndereçosha256 do arquivoLicença
maketable2agenthttps://www.dropbox.com/s/1s1yb7a3hkjwzjh/maketable2.zip?dl=1f8213d9b42290cf0d3a2e968483bf3b5964e33261b8e48f92e9ea4c87c709b92No licence stated (author's data archive); fetched, not redistributed.
maketable4agenthttps://www.dropbox.com/s/mntre4yhzothfgq/maketable4.zip?dl=1cd3a1cb197e3065411fce6fabbe06161555468049e6810a28c608d6e361aed34No licence stated (author's data archive); fetched, not redistributed.

Por suíte

Números de cada suíte desta execução
SuíteVersãoTarefasTaxa de acerto ± IC 95%Incidentes como falhaTaxa de incidentesFidelidade de implementaçãoFidelidade de replicaçãoConcordância com o caminho de referênciaAcurácia dado T1Cobertura (ferramenta · agente)Métrica nativaLinhas de base (zero · ingênua)TokensCusto
paper_repro0.1.13
  • agente (T2) 100% n=2 (100%–100%)
  • desenho (T3) 100% n=1 (21%–100%)
T2 e T3 aparecem separados, nunca somados. T3 pontua o desenho; a estimativa é registrada.
  • agente (T2) 100%
  • desenho (T3) 100%
0%————100% (56/56) · 98% (55/56)
any_cited_would_pass 0,5
0% · —264.6640,241107

Classes de falha

Nenhuma falha classificada.

Incidentes do provedor

Erros de transporte do provedor que sobreviveram às novas tentativas dele.

Nenhum incidente.

Tarefas

Tarefas 1–3 de 3

Tarefas desta execução
TarefaSuíteAlvoDivisãoSituaçãoPontuaçãoDecidida porTentativas
paper_repro.ajr_2001.tab2_c2.agentpaper_reproagentpublicpassou0,916667—1
paper_repro.ajr_2001.tab4_pa_c1.agentpaper_reproagentpublicpassou1—1
paper_repro.ajr_2001.tab4_pa_c1.designpaper_reproagentdesignpassou0,866667—1

Downloads

Pacote inteiro (.zip)
230,0 KB · sha256 e52502a2b1a27984cfa151fdb9ae3f52813682daccabb64f59154ff357bb7d5f

O hash de um zip é só de transporte: o que ancora a integridade é o RUN_SHA256SUMS que vai dentro dele, cópia exata do SHA256SUMS da execução.

Arquivos principais

Arquivos principais
ArquivoTamanhosha256
manifest.json8,9 KB324bddf04b153878906521e804879b6252409b5be5f50c45263354e44dc374bd
results.jsonl26,3 KB13776d8a5e548b9cdf7bb458e0e65963a6ef343df8dfa5709544beede07c4339
summary.json2,8 KBbdcf22f5e946421e28765dd09c6e68489f5cdb6f06c890742b86384540c614b2
summary.md614 Bd3ba466c42a82be4a18543550fab7b52daceb99724f9b3cdc8140eff9b5e6ffc
README.md3,4 KBda4b706e51d37ec6d07ba41b462f7cdfd259f37aa2eebf770c3d402a43324a8f
verify_bundle.py34,9 KB99e037f6a241e077348d7e116b03be17e727c4b61d278ea5f39b06dd557069a3
SHA256SUMS12,3 KB383d85c525b29d05eead88daa2497c01ee8d001ba4ddcdfcd7bc4f5d26294467
Todos os arquivos do pacote (93)
Todos os arquivos do pacote (93)
ArquivoTamanhosha256
README.md3,4 KBda4b706e51d37ec6d07ba41b462f7cdfd259f37aa2eebf770c3d402a43324a8f
attachments/03fe6d682d3561d879c2e6a3d5dd7565a48f7b25f9c259c418a95ff11224c864.json504 B03fe6d682d3561d879c2e6a3d5dd7565a48f7b25f9c259c418a95ff11224c864
attachments/0b491240b73d92089dd1bbbe75b09715e0792270ecad63ebbea830a325cf709f.json1,4 KB0b491240b73d92089dd1bbbe75b09715e0792270ecad63ebbea830a325cf709f
attachments/0e5be67e39fdc870af623030fc7081ea7fa9e07b4568a4a133872353b328a04e.json866 B0e5be67e39fdc870af623030fc7081ea7fa9e07b4568a4a133872353b328a04e
attachments/1616298c3cc1cec486415b444f9149d3664a3ec09bb9aaa49fc7381eb9578c6e.json4,0 KB1616298c3cc1cec486415b444f9149d3664a3ec09bb9aaa49fc7381eb9578c6e
attachments/16510b50ed8b3230ed948b2dcc1ddc336e02ab6d305108b95964fc7310a5d71a.json1,3 KB16510b50ed8b3230ed948b2dcc1ddc336e02ab6d305108b95964fc7310a5d71a
attachments/16b72014a8105587141e85c4ae0bb0c516678cd955b7c0ddcc7d1d78ecaacace.json1,4 KB16b72014a8105587141e85c4ae0bb0c516678cd955b7c0ddcc7d1d78ecaacace
attachments/1b3949eb24ed6f61dd6c2e62e64141ab2692d2ec0c48cc46a74c7b06a38ca971.json1,9 KB1b3949eb24ed6f61dd6c2e62e64141ab2692d2ec0c48cc46a74c7b06a38ca971
attachments/1bb87812b24482afaf5e03c84b7c1c4cfeb06eb42111693b4472d070660b3c23.json1,4 KB1bb87812b24482afaf5e03c84b7c1c4cfeb06eb42111693b4472d070660b3c23
attachments/1d39a46904d411e8afd72c44517c269f97b6328c1c4585347470504641bf1a1d.json860 B1d39a46904d411e8afd72c44517c269f97b6328c1c4585347470504641bf1a1d
attachments/1eb67276aa521446b88b1625f3b77ee14d87591cf5e01a3773c1196dca6caffa.json1,9 KB1eb67276aa521446b88b1625f3b77ee14d87591cf5e01a3773c1196dca6caffa
attachments/202164bbfe1d861cc8045b7b4682a3a1776628fdfdbc917361d223df326ddb99.json307 B202164bbfe1d861cc8045b7b4682a3a1776628fdfdbc917361d223df326ddb99
attachments/23a801786514e5bf298f535a356a51c418b6752bb82e88712c4a63fd390a7f7f.json4,7 KB23a801786514e5bf298f535a356a51c418b6752bb82e88712c4a63fd390a7f7f
attachments/265d63d091732725cd5b15f51e1686b68b6b276ae88794f6120dc818830a2a97.json5,6 KB265d63d091732725cd5b15f51e1686b68b6b276ae88794f6120dc818830a2a97
attachments/26d3e8dd421412b53dc7e28b466c59f6b2c4a5f2af9b6ed035465fdb1ee4c287.json5,1 KB26d3e8dd421412b53dc7e28b466c59f6b2c4a5f2af9b6ed035465fdb1ee4c287
attachments/2d33567011c31dc5f3aaef0253038faad2016b9e2ce093c0b3c4d5c1a13e0c62.json765 B2d33567011c31dc5f3aaef0253038faad2016b9e2ce093c0b3c4d5c1a13e0c62
attachments/2f37064f98c3f8f3f0a11719d25b38230d0262c6baa30499c84b2e9f3ffe88f6.json11,1 KB2f37064f98c3f8f3f0a11719d25b38230d0262c6baa30499c84b2e9f3ffe88f6
attachments/3353be0c203c31cf4f57e1be893a42f482aaa77c239dca88b5db4d69fa75cbf3.json332 B3353be0c203c31cf4f57e1be893a42f482aaa77c239dca88b5db4d69fa75cbf3
attachments/346fe87452db925116baafb7b013261ad475fbb706d7f5de5bdf1cc131c0c65f.json612 B346fe87452db925116baafb7b013261ad475fbb706d7f5de5bdf1cc131c0c65f
attachments/35fc5c0245791591c4c81ff6abc185344bac215d1b5b7a6806976a677d5cd33a.json1,2 KB35fc5c0245791591c4c81ff6abc185344bac215d1b5b7a6806976a677d5cd33a
attachments/3c244c52d778214aafb12e7889b84d8aacead1169e94bd6a88b190858badedb8.json975 B3c244c52d778214aafb12e7889b84d8aacead1169e94bd6a88b190858badedb8
attachments/3ea53614aa4095f6b78f0d7f9ca140c3245ca0e49d814602ed537685c108e854.json419 B3ea53614aa4095f6b78f0d7f9ca140c3245ca0e49d814602ed537685c108e854
attachments/41c385335853987b7d24c7f02acdfab1fc06ea35a8efe09d364982087e4c0d9b.json2,9 KB41c385335853987b7d24c7f02acdfab1fc06ea35a8efe09d364982087e4c0d9b
attachments/438237559235ba9eff244483d2a91d6ac5399b4d63eb2c6383096d4de95e77e5.json1,3 KB438237559235ba9eff244483d2a91d6ac5399b4d63eb2c6383096d4de95e77e5
attachments/4421871422cdf33ca8ffe70daa080c1702972acfce3eb84062c87db85ef5a4d8.json253 B4421871422cdf33ca8ffe70daa080c1702972acfce3eb84062c87db85ef5a4d8
attachments/47be0b73fff841614eb2804780af10d43f4d780706ac7dca50bc7966a3fa6878.json333 B47be0b73fff841614eb2804780af10d43f4d780706ac7dca50bc7966a3fa6878
attachments/514f9d4a6a73b825cd1eb586ceb6e4ed35ff349ad5113b0e6990697376f15e46.json219 B514f9d4a6a73b825cd1eb586ceb6e4ed35ff349ad5113b0e6990697376f15e46
attachments/5350a59f3eac22192db133702edfd06d29f139b64cee7dda50f91e290cfd37df.json977 B5350a59f3eac22192db133702edfd06d29f139b64cee7dda50f91e290cfd37df
attachments/5873424a700a86e94c52f386da2830d4a6a35db7f69228d972dd00bba6a008d6.json2,1 KB5873424a700a86e94c52f386da2830d4a6a35db7f69228d972dd00bba6a008d6
attachments/59eaa64f313b5426615c327d8ee9a96a58f97680760203dfafd9b7552a533797.json2,4 KB59eaa64f313b5426615c327d8ee9a96a58f97680760203dfafd9b7552a533797
attachments/6449068032e317e7ae3ad08597b3ce15909625b462a92cc42fa69ca0efb09a92.json866 B6449068032e317e7ae3ad08597b3ce15909625b462a92cc42fa69ca0efb09a92
attachments/6a3759dcd18f2028ab03fe303634506f920e0fbe05b3185ce53f7b8e3fd1fe1f.json11,7 KB6a3759dcd18f2028ab03fe303634506f920e0fbe05b3185ce53f7b8e3fd1fe1f
attachments/6bc3e2e3291fcf837fe99748a4243b5d7c9577a38c7a1e23fbd57a126996ce20.json3,7 KB6bc3e2e3291fcf837fe99748a4243b5d7c9577a38c7a1e23fbd57a126996ce20
attachments/720eeda26535bc526e7abcf17c8ec69cda9345942fc2882fd672a1daf22701c5.json3,5 KB720eeda26535bc526e7abcf17c8ec69cda9345942fc2882fd672a1daf22701c5
attachments/73f6cc662d90b8e23ecf3767eade31c90701ae556ece9518b9579e6d98fe28f5.json969 B73f6cc662d90b8e23ecf3767eade31c90701ae556ece9518b9579e6d98fe28f5
attachments/77936f6c50ca37aa39e1dc9b73c4b27bf407410d7dc949ab1215659b312b814e.json418 B77936f6c50ca37aa39e1dc9b73c4b27bf407410d7dc949ab1215659b312b814e
attachments/786efe53df0bdc36fedda6bbca951c74f7c2f9ea8a9ed92d31e8ae590246243a.json1,9 KB786efe53df0bdc36fedda6bbca951c74f7c2f9ea8a9ed92d31e8ae590246243a
attachments/792bb786343abd2133dca05b46f4c8b9737ec98a1f7061d9e5f974d480c3c2f7.json971 B792bb786343abd2133dca05b46f4c8b9737ec98a1f7061d9e5f974d480c3c2f7
attachments/86ff9a038fae4197462dd59d58f7a565dd9bb75ee651e10fd077f2205229846b.json1,8 KB86ff9a038fae4197462dd59d58f7a565dd9bb75ee651e10fd077f2205229846b
attachments/88326b98fe1e3663aebeb7e514218bc3c326b4b7c76f4bf9ef15bec4c7fced74.json2,0 KB88326b98fe1e3663aebeb7e514218bc3c326b4b7c76f4bf9ef15bec4c7fced74
attachments/8dae2e342468a3f380fd04303bd43accfce8c5171b15a5a1d83860d65c1ea1a6.json1,6 KB8dae2e342468a3f380fd04303bd43accfce8c5171b15a5a1d83860d65c1ea1a6
attachments/8e773741a6d9ba66da835603fa5dae70bc7e0c3193aabd80e06e98e6b93167c3.json763 B8e773741a6d9ba66da835603fa5dae70bc7e0c3193aabd80e06e98e6b93167c3
attachments/8ee0d84c29b117269608a8edad648afadcb20e12f00a5d88b7d64223f4b057e2.json258 B8ee0d84c29b117269608a8edad648afadcb20e12f00a5d88b7d64223f4b057e2
attachments/93a90ffe6f373a4813e718b3a7e8a03e483f0379bc50943952368b7ff653de62.json2,5 KB93a90ffe6f373a4813e718b3a7e8a03e483f0379bc50943952368b7ff653de62
attachments/9bc4ea4179c419e641806efc47c92bf378fcf0a8bdd3072e1128d920813496d4.json309 B9bc4ea4179c419e641806efc47c92bf378fcf0a8bdd3072e1128d920813496d4
attachments/9e20df08693144afecb7a5e1a68e126501102a03270d671065e81027b82a6c5d.json6,9 KB9e20df08693144afecb7a5e1a68e126501102a03270d671065e81027b82a6c5d
attachments/9f9cff3a13dc37b0191f980f525bdee19a99e81c2409d7f0d215cdf5e0b8a1f9.json358 B9f9cff3a13dc37b0191f980f525bdee19a99e81c2409d7f0d215cdf5e0b8a1f9
attachments/a44ef51f03df5188f084f1e0227b68df116b942a22fd50c0a88670cf3461fedd.json65,0 KBa44ef51f03df5188f084f1e0227b68df116b942a22fd50c0a88670cf3461fedd
attachments/a489091fc8a649114bfa5452203f610e0d16e65eaf87dbf479f1edb072b5577a.json1,3 KBa489091fc8a649114bfa5452203f610e0d16e65eaf87dbf479f1edb072b5577a
attachments/a535ab31b7d15407ab1dfe07f398da2d7ffdc37e4cfa8710c268b86aa45dc733.json2,5 KBa535ab31b7d15407ab1dfe07f398da2d7ffdc37e4cfa8710c268b86aa45dc733
attachments/a583df065872984365d6469fbaaa7454b71c4ac99e8d8d782424085ad2526751.json977 Ba583df065872984365d6469fbaaa7454b71c4ac99e8d8d782424085ad2526751
attachments/ad0206ec4f542bd32ba442c44c7d4a36a4d88fb351f7899e0ddd7164cbd8eefd.json1,4 KBad0206ec4f542bd32ba442c44c7d4a36a4d88fb351f7899e0ddd7164cbd8eefd
attachments/af1dcbbcc01acb0e395a5fb7a956d7358f3087bceaa8afd32547fa761748fb22.json2,1 KBaf1dcbbcc01acb0e395a5fb7a956d7358f3087bceaa8afd32547fa761748fb22
attachments/bc2227ab3a26183e97ef18f0ae346ce894d555b440accf23fa22e03b4c4f7f7e.json320 Bbc2227ab3a26183e97ef18f0ae346ce894d555b440accf23fa22e03b4c4f7f7e
attachments/bc24c35350ca957486af0b86f148b4fb343232d4afa49a3a5d249ac673143de9.json346 Bbc24c35350ca957486af0b86f148b4fb343232d4afa49a3a5d249ac673143de9
attachments/bc961669c5d2b64a7d68d1e89ccfe3fc9300e649f77108b0ca328c4e35f0aa45.json4,3 KBbc961669c5d2b64a7d68d1e89ccfe3fc9300e649f77108b0ca328c4e35f0aa45
attachments/c837d819101eb7482490acf19747c7570bf0ec8cf6c2b4480ec5883599174206.json548 Bc837d819101eb7482490acf19747c7570bf0ec8cf6c2b4480ec5883599174206
attachments/ca9f74584eae45bceaa9761be3d71ff46e87e52ef305400f8b72e2ee27d539b5.json431 Bca9f74584eae45bceaa9761be3d71ff46e87e52ef305400f8b72e2ee27d539b5
attachments/d0251c4c9ba43fcf36a1fdb59e5d8e2fc93ae0e85a7015b9d52314d9b2d2ec2e.json5,4 KBd0251c4c9ba43fcf36a1fdb59e5d8e2fc93ae0e85a7015b9d52314d9b2d2ec2e
attachments/d11b918ddee452e5f607a4eaa8928e8c41db3926e7a21cf3213d3f9ae8745dea.json975 Bd11b918ddee452e5f607a4eaa8928e8c41db3926e7a21cf3213d3f9ae8745dea
attachments/d487fb7993e5988a258a9e041541559178b6b8bccfebed2df8af34873cb23b12.json2,8 KBd487fb7993e5988a258a9e041541559178b6b8bccfebed2df8af34873cb23b12
attachments/d61edaf1bf92c25d81aff0ae8ce93a45d44ef66f16f6b59ce63d9c32107991f6.json3,1 KBd61edaf1bf92c25d81aff0ae8ce93a45d44ef66f16f6b59ce63d9c32107991f6
attachments/eea729aaf4e11c70b13d473438dadbc1b53cc2def87e06b7c4c4ff7867d2f4b0.json2,0 KBeea729aaf4e11c70b13d473438dadbc1b53cc2def87e06b7c4c4ff7867d2f4b0
attachments/ef0ff53deead73c69937a13bdf31b8b31941ebe29bd1d7c128b7caaedd841e49.json536 Bef0ff53deead73c69937a13bdf31b8b31941ebe29bd1d7c128b7caaedd841e49
attachments/f93c8ad9ec6a23c4ec3ff8ce6596c828eb6b5d19c2d4ebe07a25c8a3c2ba9fdd.json650 Bf93c8ad9ec6a23c4ec3ff8ce6596c828eb6b5d19c2d4ebe07a25c8a3c2ba9fdd
attachments/f93fe85019a57ef3c68e3f70b9107d2148a26ff8d2114127d6c1665b724592ae.json493 Bf93fe85019a57ef3c68e3f70b9107d2148a26ff8d2114127d6c1665b724592ae
attachments/fad07ce283546ec4741f8a0ecca36994e8f3516446cc59c45a370ab3cb2dbe7e.json1,4 KBfad07ce283546ec4741f8a0ecca36994e8f3516446cc59c45a370ab3cb2dbe7e
attachments/fbf768dd1673127eaf355a5e3635c8ed9c6443475f00631534f49a3329e46d9d.json610 Bfbf768dd1673127eaf355a5e3635c8ed9c6443475f00631534f49a3329e46d9d
attachments/fd5b4cc0248dfe9d31c23d72f9b8c404dd5eea5d2e88ef34ae2f1f1341a4c829.json2,9 KBfd5b4cc0248dfe9d31c23d72f9b8c404dd5eea5d2e88ef34ae2f1f1341a4c829
env/platform.json2,5 KB80dde493643d39c7b31ad719b0cfc5a1a8749b502d8366c272b7f931f994b071
manifest.json8,9 KB324bddf04b153878906521e804879b6252409b5be5f50c45263354e44dc374bd
native/paper_repro/label_map.json659 B2d9151f2c22e04194b772017a6595fb4d3d73d9beeabaf43af460c00cf494b05
native/paper_repro/paper_repro.ajr_2001.tab2_c2.agent.e1.a1.json35 Be8a9aed0bd8f9e40ed61a4bcfbb05b9da2b728a132d60f96014515c3d9c91874
native/paper_repro/paper_repro.ajr_2001.tab4_pa_c1.agent.e1.a1.json36 B39a1ffcef98d10bf196fee2aa93ee0343ff9c7c2b32629831f61ccb9c4e8fdd7
results.jsonl26,3 KB13776d8a5e548b9cdf7bb458e0e65963a6ef343df8dfa5709544beede07c4339
sources/ajr_2001.tab2_c2.paper.json2,0 KBedb87e793473767375a8cc170a56893fe5eb3849c333a49ab987c970065bf2e2
sources/ajr_2001.tab2_c2.software.json1,2 KBf07c55f4c6230e76d0eb32de69f271f83007fcd8192fea36827aa73fc4a1391d
sources/ajr_2001.tab4_pa_c1.paper.json1,6 KBb253c9deb0ce566d7a1691dea9350d3e6fbc5c68d703a24d8fed0ad48d65ad71
sources/ajr_2001.tab4_pa_c1.software.json1,3 KB63bf31814612de1c1032e41d88a76ce29b8860377f94f1d35f9e801362091620
specs/paper_repro.ajr_2001.tab2_c2.agent.json8,4 KB754dca75f8be5d462387ca3c5d4041d3adc8105f1d4aa1f940dbce5154d49b3c
specs/paper_repro.ajr_2001.tab4_pa_c1.agent.json8,3 KB810a8a537e738cae894895a8aa67b25a141d37557666633324bb7b9f29b2cea3
specs/paper_repro.ajr_2001.tab4_pa_c1.design.json9,5 KB0a7f5ae8c4d24581f1f351f55ba5ea5f24b7d13858a9de96551201189f6d8c82
summary.json2,8 KBbdcf22f5e946421e28765dd09c6e68489f5cdb6f06c890742b86384540c614b2
summary.md614 Bd3ba466c42a82be4a18543550fab7b52daceb99724f9b3cdc8140eff9b5e6ffc
tools/registry.json14,3 KB644bdc32c1ef8e99eea9ba91f16d8c477a4c57d26b17990e22e346b33aa067fe
traces/paper_repro.ajr_2001.tab2_c2.agent.e1.a1.jsonl31,7 KB19415c0a05ddadb8a6d489f4b30d33ff39f6dcde52e0bac10c96015b4f1b4d67
traces/paper_repro.ajr_2001.tab4_pa_c1.agent.e1.a1.jsonl24,2 KB45011b3761c16d749e5b9be4173241bdda4b92d0c307498fcf1e49f7fa7a209a
traces/paper_repro.ajr_2001.tab4_pa_c1.design.e1.a1.jsonl58,9 KB6ac30e166495da17773c473242d251c77dc630e9df54f8894c7e163a4719bdd5
trajectories/paper_repro.ajr_2001.tab2_c2.agent.e1.a1.jsonl30,8 KBac9bb5d019c8e0a0ef423fb5556e1fac826653b0dd0fe03a1581ea3145434059
trajectories/paper_repro.ajr_2001.tab4_pa_c1.agent.e1.a1.jsonl24,0 KBf36cee73a62d20b5f546506cc5f93b67a133b32622cedf65319400283b0debb1
trajectories/paper_repro.ajr_2001.tab4_pa_c1.design.e1.a1.jsonl56,3 KB86e9614ed7b729622ea09f40582aa5002a09a29f7ca36aac7e9f689c559c54af
verify_bundle.py34,9 KB99e037f6a241e077348d7e116b03be17e727c4b61d278ea5f39b06dd557069a3
SHA256SUMS12,3 KB383d85c525b29d05eead88daa2497c01ee8d001ba4ddcdfcd7bc4f5d26294467

Eventos do registro

Verifique você mesmo

Resumo da execução
383d85c525b29d05eead88daa2497c01ee8d001ba4ddcdfcd7bc4f5d26294467
sha256 do arquivo SHA256SUMS; deve ser igual ao run_digest do evento final do registro.

Qualquer pessoa, sem o repositório

  1. Baixe o pacote inteiro e descompacte.
  2. Rode o verificador, que só usa a biblioteca padrão do Python:
    cd 20261015T124100Z-suite-paper_repro-gemini-c5ea8f
    python3 verify_bundle.py

Ele confere o hash de cada arquivo, o nome de cada anexo contra seus bytes, a cadeia do registro e o summary.json recalculado a partir de results.jsonl.

Para conferir o resumo:

shasum -a 256 SHA256SUMS   # 383d85c525b29d05eead88daa2497c01ee8d001ba4ddcdfcd7bc4f5d26294467

Com acesso ao repositório (privado)

git checkout 38e979372a0f7382f285079e4380813081bdd032
uv sync --frozen --all-packages
uv run magi-bench verify bench-runs/20261015T124100Z-suite-paper_repro-gemini-c5ea8f

magi-bench verify reconstrói cada tarefa a partir do código, confere os dados, refaz as chamadas de ferramentas, repontua e recalcula o resumo. Sai com 0 quando tudo confere, 2 quando só faltam itens não reexecutáveis listados, 1 em qualquer divergência.