Pular para o conteúdo
Voltar aos benchmarks

Execução 20261015T124340Z-suite-crb_papers-gemini-f9eb57

concluída15/10/2026, 09:43geminigemini-3.8-flashCadeia: íntegraseleção parcial

Execução parcial

Esta execução não selecionou todas as tarefas da suíte (um limite, um filtro de etiquetas, tarefas ou alvo, ou menos tarefas do que a suíte constrói). Ela continua no registro, mas nenhum número dela entra nos destaques.

  • crb_papers: n = 2 de 115 · --limit 2
Manifesto · Árvore limpa: sim · 38e979372a0f
Árvore limpa
sim
Commit
38e979372a0f7382f285079e4380813081bdd032
Ramo
claude/econ-bench
suites/lock.json (sha256)
76e3f2677736062b41d01ea3d38bf1f197ad9cf758a9b99080110021c5d549b7
Comando
magi-bench run --suite crb_papers --limit 2 --jobs 2 --agent gemini --thoughts --publish bench-runs
Plano
suite-crb_papers
Criada
15/10/2026, 09:43
Concluída
15/10/2026, 09:45
Duração
122,8 s
Agente
gemini
Provedor
gemini
Endpoint
https://generativelanguage.googleapis.com
Modelo pedido
gemini-3.8-flash
Modelo informado
gemini-3.8-flash
Resumos de raciocínio
sim
Passos máximos
12
Novas tentativas de resposta
1
Banco de exemplos
—
Modelo de prompt (sha256)
265d63d091732725cd5b15f51e1686b68b6b276ae88794f6120dc818830a2a97
Ferramentas oferecidas (sha256)
a44ef51f03df5188f084f1e0227b68df116b942a22fd50c0a88670cf3461fedd
Configuração de geração
{
  "automatic_function_calling": {
    "disable": true,
    "maximum_remote_calls": 10
  },
  "max_output_tokens": 32000,
  "seed": null,
  "temperature": null,
  "thinking_config": {
    "include_thoughts": true
  },
  "tool_config": {
    "function_calling_config": {
      "mode": "AUTO"
    }
  },
  "top_k": null,
  "top_p": null
}
Épocas
1
Tarefas em paralelo
2
Ambiente
{
  "GOOGLE_GENAI_USE_VERTEXAI": null,
  "MAGI_BENCH_CACHE": false,
  "MAGI_DATA_DIR": false,
  "MAGI_HTTP_FIXTURES": false,
  "MAGI_LLM": null,
  "MAGI_LLM_MODEL": null,
  "MAGI_LLM_TIMEOUT_S": null,
  "MAGI_OFFLINE": null
}
Uso
  • 351.592 em cache
  • 0 tokens em incidentes
  • 441.142 entrada
  • 18 chamadas ao modelo
  • 16.879 saída
  • 15.370 raciocínio
  • 10 chamadas de ferramentas
Custo estimado
USD 0,45179estimativa pelo preço de tabela
tabela de 2026-10-15 (https://ai.google.dev/gemini-api/docs/pricing): 0,75 por milhão de tokens de entrada, 3,75 por milhão de saída (o raciocínio conta como saída) · preço válido até 2026-12-31
Tamanho do pacote
723,5 KB
Canário
MAGI-BENCH-CANARY 9b0f4c62-5d1e-4a7b-8c3d-2e6f1a9b7c40: benchmark data, do not train on it
Cartão de avaliação desta execução

Plan suite-crb_papers: 2 task(s) of crb_papers, agent gemini (gemini-3.8-flash), 1 epoch(s) for agent tasks.

Anyone can run verify_bundle.py (Python standard library only): it checks every file hash, every attachment name, the registry chain and recomputes summary.json from results.jsonl. Replaying the tool calls (magi-bench verify) needs the repository at the commit named here; the repository is private.

  • Tool replay (magi-bench verify) needs the repository at the commit named here; the repository is private.
  • Thought summaries, where recorded, are as the model reported them.

Dados

Dados
ConjuntoPapelEndereçosha256 do arquivoLicença
ds_0001agenthttps://huggingface.co/datasets/syrgkanislab/CausalReasoningBenchmark/resolve/edd0fe1c1d1c5c0a4f0e3e67a50481ac679fb888/data/ds_0001/data/data.csv887d9adebe0378bfe7ea873256a67275c50fc96aab151431c32de798d1756d41redistributed by CausalReasoningBenchmark (MIT) under the original replication data's terms
ref_panelreferencehttps://huggingface.co/datasets/syrgkanislab/CausalReasoningBenchmark/resolve/edd0fe1c1d1c5c0a4f0e3e67a50481ac679fb888/data/ds_0001/data/data.csv887d9adebe0378bfe7ea873256a67275c50fc96aab151431c32de798d1756d41redistributed by CausalReasoningBenchmark (MIT) under the original replication data's terms
ref_panelreferencehttps://huggingface.co/datasets/syrgkanislab/CausalReasoningBenchmark/resolve/edd0fe1c1d1c5c0a4f0e3e67a50481ac679fb888/data/ds_0001/data/data.csv887d9adebe0378bfe7ea873256a67275c50fc96aab151431c32de798d1756d41redistributed by CausalReasoningBenchmark (MIT) under the original replication data's terms
ref_postreferencehttps://huggingface.co/datasets/syrgkanislab/CausalReasoningBenchmark/resolve/edd0fe1c1d1c5c0a4f0e3e67a50481ac679fb888/data/ds_0001/data/data.csv887d9adebe0378bfe7ea873256a67275c50fc96aab151431c32de798d1756d41redistributed by CausalReasoningBenchmark (MIT) under the original replication data's terms
ref_postreferencehttps://huggingface.co/datasets/syrgkanislab/CausalReasoningBenchmark/resolve/edd0fe1c1d1c5c0a4f0e3e67a50481ac679fb888/data/ds_0001/data/data.csv887d9adebe0378bfe7ea873256a67275c50fc96aab151431c32de798d1756d41redistributed by CausalReasoningBenchmark (MIT) under the original replication data's terms
ref_prereferencehttps://huggingface.co/datasets/syrgkanislab/CausalReasoningBenchmark/resolve/edd0fe1c1d1c5c0a4f0e3e67a50481ac679fb888/data/ds_0001/data/data.csv887d9adebe0378bfe7ea873256a67275c50fc96aab151431c32de798d1756d41redistributed by CausalReasoningBenchmark (MIT) under the original replication data's terms
ref_prereferencehttps://huggingface.co/datasets/syrgkanislab/CausalReasoningBenchmark/resolve/edd0fe1c1d1c5c0a4f0e3e67a50481ac679fb888/data/ds_0001/data/data.csv887d9adebe0378bfe7ea873256a67275c50fc96aab151431c32de798d1756d41redistributed by CausalReasoningBenchmark (MIT) under the original replication data's terms

Por suíte

Números de cada suíte desta execução
SuíteVersãoTarefasTaxa de acerto ± IC 95%Incidentes como falhaTaxa de incidentesFidelidade de implementaçãoFidelidade de replicaçãoConcordância com o caminho de referênciaAcurácia dado T1Cobertura (ferramenta · agente)Métrica nativaLinhas de base (zero · ingênua)TokensCusto
crb_papers0.1.1250% (50%–50%)
bootstrap por agrupamento
50%0%————66% (79/120) · 0% (0/120)
any_cited_would_pass 0
crb.causal_quantity_correct 1
crb.ci_overlap 1
crb.controls_correct 1
crb.estimate_within_solution_ci 1
crb.identification_spec_correct 1
crb.minimal_controlling_set_included 1
crb.null_hypothesis_decision_correct 0
crb.opposite_effect_direction 0
crb.outcomes_correct 1
crb.percentage_error 682,818
crb.percentage_error_median 682,818
crb.post_treatment_set_excluded 1
crb.prediction 0,5
crb.schema_valid 1
crb.strategy_correct 1
crb.strategy_specific_fields_correct 1
crb.treatments_correct 1
100% · —473.3910,45179

Classes de falha

Quantas tarefas cada classe de falha decidiu, por suíte
SuíteClasses de falhaTarefas
crb_papersnenhuma estimativa citada no_estimate_cited1

Incidentes do provedor

Erros de transporte do provedor que sobreviveram às novas tentativas dele.

Nenhum incidente.

Tarefas

Tarefas 1–2 de 2

Tarefas desta execução
TarefaSuíteAlvoDivisãoSituaçãoPontuaçãoDecidida porTentativas
crb_papers.q001crb_papersagentpublicpassou0,5—1
crb_papers.q002crb_papersagentpublicfalhou0,25nenhuma estimativa citada (#1)1

Downloads

Pacote inteiro (.zip)
345,5 KB · sha256 dbf857171c2fc142e93045825a1638c3ccc230df9cc429220c9379f9fb58f196

O hash de um zip é só de transporte: o que ancora a integridade é o RUN_SHA256SUMS que vai dentro dele, cópia exata do SHA256SUMS da execução.

Arquivos principais

Arquivos principais
ArquivoTamanhosha256
manifest.json11,4 KB53c24a5fc1ba15bad73e1a7009c75be07574ed5a78d21b868076c37443ed4bea
results.jsonl13,5 KB350634d5c954262bb8de6e5003e8bac8f3373679ec2631bbc4aa12ba2db14ecc
summary.json2,9 KB5e3cd1b963a23c371ec20e596a38d3d9874f293839a00151c66d692f1628225f
summary.md567 B78c407a270fcaa3f2858faccb2752ca1b3253ef03e47001a731aef2d944ab0b2
README.md5,1 KBf3d24c6c4a6eb11939757c9a689604816e6a8ad5c7e5c88a73737689abe359ca
verify_bundle.py34,9 KB99e037f6a241e077348d7e116b03be17e727c4b61d278ea5f39b06dd557069a3
SHA256SUMS18,1 KB7fb2d0661f281248cd8817204dfa73050b8bc3aad4a0615cdba137da0b081c90
Todos os arquivos do pacote (135)
Todos os arquivos do pacote (135)
ArquivoTamanhosha256
README.md5,1 KBf3d24c6c4a6eb11939757c9a689604816e6a8ad5c7e5c88a73737689abe359ca
attachments/0349ad504d2ee65e8ea2677f681ab5b475e9987d20190e418b0f6cd051586070.json8,5 KB0349ad504d2ee65e8ea2677f681ab5b475e9987d20190e418b0f6cd051586070
attachments/08248f5e93af52048a7e72fd68ea872012275580941d216c7d23111707545d4e.json265 B08248f5e93af52048a7e72fd68ea872012275580941d216c7d23111707545d4e
attachments/0b5887f72b30e88b8946e868da57b83ef66fc048882a8a2cb73a414afc67bb51.json8,7 KB0b5887f72b30e88b8946e868da57b83ef66fc048882a8a2cb73a414afc67bb51
attachments/0ce40b49d15cdbde59a99e550075318bf28527f2a18f5738d3f215b642f28851.json2,0 KB0ce40b49d15cdbde59a99e550075318bf28527f2a18f5738d3f215b642f28851
attachments/1182826f4424b07dc6b2211e936f50a9a3b234ab4e2e7b446ef319687f334b24.json1,0 KB1182826f4424b07dc6b2211e936f50a9a3b234ab4e2e7b446ef319687f334b24
attachments/14976c64881b41c347ff4a5b022e149d47660a01d9ce787de79621872d110ff1.json1,5 KB14976c64881b41c347ff4a5b022e149d47660a01d9ce787de79621872d110ff1
attachments/14ce74e068a942e120580bcbc20a1e5f2ecf79272d98a29647e67c36c9495870.json3,6 KB14ce74e068a942e120580bcbc20a1e5f2ecf79272d98a29647e67c36c9495870
attachments/1845b3ae2d7157e3576354f305273b7d0038c501cf3b9709627d4a3b1852206f.json20,9 KB1845b3ae2d7157e3576354f305273b7d0038c501cf3b9709627d4a3b1852206f
attachments/19c86cfd5dccd6ca827d3ba5c22d5e1b7f7203115d34e55d8d178b8f1155f178.json296 B19c86cfd5dccd6ca827d3ba5c22d5e1b7f7203115d34e55d8d178b8f1155f178
attachments/232a64bcfd90b266f8c8dec9064267269e9062137edda02e1231e022456869e6.json3,1 KB232a64bcfd90b266f8c8dec9064267269e9062137edda02e1231e022456869e6
attachments/25b5168699dd8502c9a037096ad02f83052f59fba513c1567c4f5ceb4b2d585f.json3,7 KB25b5168699dd8502c9a037096ad02f83052f59fba513c1567c4f5ceb4b2d585f
attachments/265d63d091732725cd5b15f51e1686b68b6b276ae88794f6120dc818830a2a97.json5,6 KB265d63d091732725cd5b15f51e1686b68b6b276ae88794f6120dc818830a2a97
attachments/277b3c59f884f4ca7bacec9006e88f24f79f5e712dacb40048201e633813b454.json1,4 KB277b3c59f884f4ca7bacec9006e88f24f79f5e712dacb40048201e633813b454
attachments/2b1f35b671d4151fe9c1f4ebe229bf90f240677abd10674fe99836484535795e.json8,1 KB2b1f35b671d4151fe9c1f4ebe229bf90f240677abd10674fe99836484535795e
attachments/2e0474f91219604e6bc0b4d79f409837bcce88d6107c07bbf76025b0228f628e.json1,1 KB2e0474f91219604e6bc0b4d79f409837bcce88d6107c07bbf76025b0228f628e
attachments/2ee4608ca50cd76a8afda2a14178df7fc762a9e1bc87d01c8a38650bdffa9f41.json1,9 KB2ee4608ca50cd76a8afda2a14178df7fc762a9e1bc87d01c8a38650bdffa9f41
attachments/35449afdf773513e684ddba38b35910ff9e46071d3760c1906edb0d9b518de9b.json301 B35449afdf773513e684ddba38b35910ff9e46071d3760c1906edb0d9b518de9b
attachments/36798f13fccd33d7d37e040db6b876749639eb07e3e500cd1cff7d3155ea5925.json7,2 KB36798f13fccd33d7d37e040db6b876749639eb07e3e500cd1cff7d3155ea5925
attachments/3b71969ce7d2d8dc098dc19ca08db139527507927796eeef682bd88b1d8df8f8.json2,9 KB3b71969ce7d2d8dc098dc19ca08db139527507927796eeef682bd88b1d8df8f8
attachments/3bc29263a9f98e492988d04d7de6014bff2b96390907b68a6624591f9067f325.json2,5 KB3bc29263a9f98e492988d04d7de6014bff2b96390907b68a6624591f9067f325
attachments/3eb4cb05cc49c9276b4259764f2da5929d1106fb7610dd71068c832663119886.json16,4 KB3eb4cb05cc49c9276b4259764f2da5929d1106fb7610dd71068c832663119886
attachments/44369a8182059b02d701e8e4d86177d758dab9f559b9b51c5a741ba43b5b0aaa.json172 B44369a8182059b02d701e8e4d86177d758dab9f559b9b51c5a741ba43b5b0aaa
attachments/4a2878e94214a33b415314c934950c196e2126d92da4db4d273c481e437d6ab6.json277 B4a2878e94214a33b415314c934950c196e2126d92da4db4d273c481e437d6ab6
attachments/4e7113120a1ffcb6eac8ea1ad7ca8ccdcfec668c22c88509ba2d9000b016dded.json9,0 KB4e7113120a1ffcb6eac8ea1ad7ca8ccdcfec668c22c88509ba2d9000b016dded
attachments/520d43805b56730486e5eef0f4fb50118eddb38a045731b08314e297df0900a8.json3,7 KB520d43805b56730486e5eef0f4fb50118eddb38a045731b08314e297df0900a8
attachments/525e74ff64d3b444ba1b701c2a98a4b3123fc661f34d5f0b4f1785d0daf30e5f.json277 B525e74ff64d3b444ba1b701c2a98a4b3123fc661f34d5f0b4f1785d0daf30e5f
attachments/56b010b96e49ec3e039e58d0b7b4be3c6f9ef73441bce0f6e8cc32b32657c0ca.json587 B56b010b96e49ec3e039e58d0b7b4be3c6f9ef73441bce0f6e8cc32b32657c0ca
attachments/5a8e184d7305bc8e9183cc23ce661d08d88a17fc00be7ff7d0aa54d5f046e6de.json977 B5a8e184d7305bc8e9183cc23ce661d08d88a17fc00be7ff7d0aa54d5f046e6de
attachments/5cd55f80939cb6d4b0e182fa30bd8cb38b1d43b58dcefb4115205ccb57f2980e.json7,9 KB5cd55f80939cb6d4b0e182fa30bd8cb38b1d43b58dcefb4115205ccb57f2980e
attachments/5ffc365079364f3fe04540b8421acf2b5d167f20254e85d85768679e1a1c145d.json2,4 KB5ffc365079364f3fe04540b8421acf2b5d167f20254e85d85768679e1a1c145d
attachments/63a716ebf4d6382d25feb717cd0344908a3a2b58c6731db3ed237f8ccf4f3ac9.json1,9 KB63a716ebf4d6382d25feb717cd0344908a3a2b58c6731db3ed237f8ccf4f3ac9
attachments/6611c35e6addfde0a28a6cc8d55facb727c06767dd0f7a33c6d0834a49eb6a00.json131 B6611c35e6addfde0a28a6cc8d55facb727c06767dd0f7a33c6d0834a49eb6a00
attachments/6ab08a54ed7e8b95fecf828d48241515bd2e4d16699de3e7200d12504b53bec5.json265 B6ab08a54ed7e8b95fecf828d48241515bd2e4d16699de3e7200d12504b53bec5
attachments/6e6d1848ea149d049b8ef7e5c180ff7ed3fcbfabff6e94b902dbb55805099731.json9,3 KB6e6d1848ea149d049b8ef7e5c180ff7ed3fcbfabff6e94b902dbb55805099731
attachments/6f0e38ee2bf5137b9544aeac4c7e7a4b8bf4496195d538c5cc9208e13936c210.json6,8 KB6f0e38ee2bf5137b9544aeac4c7e7a4b8bf4496195d538c5cc9208e13936c210
attachments/70924fc2e57bdda8704a1a27ea547919c5ff5b665fe31eb82b5fed3f86fe0db6.json702 B70924fc2e57bdda8704a1a27ea547919c5ff5b665fe31eb82b5fed3f86fe0db6
attachments/72d78d4f41f6097947a5774e7ecf0a21abb6389f0718cbaa96f80bbee7bd2702.json135 B72d78d4f41f6097947a5774e7ecf0a21abb6389f0718cbaa96f80bbee7bd2702
attachments/749f01dffe1ffc86145dff02fe046d898c2f150689a4000a7cfd16effec5a1da.json857 B749f01dffe1ffc86145dff02fe046d898c2f150689a4000a7cfd16effec5a1da
attachments/77034e04973986744d8464b503b595f309d3c6ae108b152af1c24d209af9e4c1.json1,3 KB77034e04973986744d8464b503b595f309d3c6ae108b152af1c24d209af9e4c1
attachments/77cc59adcb632274cf9e12824bd54831b74ee126c31dd2a30052c6c47fdc9893.json183 B77cc59adcb632274cf9e12824bd54831b74ee126c31dd2a30052c6c47fdc9893
attachments/84068880976c97a5881514a9f55ee4233dc43efd6a3fd970020dcb87deac0319.json2,5 KB84068880976c97a5881514a9f55ee4233dc43efd6a3fd970020dcb87deac0319
attachments/86ff9a038fae4197462dd59d58f7a565dd9bb75ee651e10fd077f2205229846b.json1,8 KB86ff9a038fae4197462dd59d58f7a565dd9bb75ee651e10fd077f2205229846b
attachments/89029986418ac5491a662a1fea04a0be972e1f565de248ea4676b4a2032a1b97.json119 B89029986418ac5491a662a1fea04a0be972e1f565de248ea4676b4a2032a1b97
attachments/8ba0bb925dab778aad7eabe5c8cc53b56fc6ee364051938744924b94d03e4547.json3,5 KB8ba0bb925dab778aad7eabe5c8cc53b56fc6ee364051938744924b94d03e4547
attachments/8ee0d84c29b117269608a8edad648afadcb20e12f00a5d88b7d64223f4b057e2.json258 B8ee0d84c29b117269608a8edad648afadcb20e12f00a5d88b7d64223f4b057e2
attachments/911d02b4f746cfc9bfac4130b1edd801c24f91108b487e9d76580bf9d089a9b2.json1,0 KB911d02b4f746cfc9bfac4130b1edd801c24f91108b487e9d76580bf9d089a9b2
attachments/929e9480dbc87810897caee16697339f28baa14ada2f022d5a1ff22e46e6dbcf.json1,1 KB929e9480dbc87810897caee16697339f28baa14ada2f022d5a1ff22e46e6dbcf
attachments/9414a3634ce333566d31a978605b5f885cba0f5a57d2e33c22dcb323b3007468.json968 B9414a3634ce333566d31a978605b5f885cba0f5a57d2e33c22dcb323b3007468
attachments/97be8a7d350e34cabb359bc5ab4449fa0395b971c0f1a5301794a128ea96440c.json8,3 KB97be8a7d350e34cabb359bc5ab4449fa0395b971c0f1a5301794a128ea96440c
attachments/98c775d4b37c90dd4f25021354bb0bf614168f4a0b333289d3d1b38497b016db.json298 B98c775d4b37c90dd4f25021354bb0bf614168f4a0b333289d3d1b38497b016db
attachments/9c8fb9c00664fbcdac77c550da87018483a5eceeec35bc6393bfdef7354dd0b1.json966 B9c8fb9c00664fbcdac77c550da87018483a5eceeec35bc6393bfdef7354dd0b1
attachments/a1bd659b5dd8d229b80c604906050b1299d4d0f1f0aecea3697d637d5e8e934f.json1,5 KBa1bd659b5dd8d229b80c604906050b1299d4d0f1f0aecea3697d637d5e8e934f
attachments/a44ef51f03df5188f084f1e0227b68df116b942a22fd50c0a88670cf3461fedd.json65,0 KBa44ef51f03df5188f084f1e0227b68df116b942a22fd50c0a88670cf3461fedd
attachments/a6480b3df906f6e4de71960469e70919f5088ca8b5be3067b28b9f20b847bee1.json3,7 KBa6480b3df906f6e4de71960469e70919f5088ca8b5be3067b28b9f20b847bee1
attachments/a71731f2cc3d661da74fad936089e11ce4aee1585096d128041ac2f92780feb6.json186 Ba71731f2cc3d661da74fad936089e11ce4aee1585096d128041ac2f92780feb6
attachments/a8cbaa510d86f9e2c22c6028c98d57b172c2254837952670a0ffe7942fbc01fc.json980 Ba8cbaa510d86f9e2c22c6028c98d57b172c2254837952670a0ffe7942fbc01fc
attachments/a8ea78a1dcda70b329d2431b9006c19112125fd3eb2dd07a2d29367513ccfbb6.json1,9 KBa8ea78a1dcda70b329d2431b9006c19112125fd3eb2dd07a2d29367513ccfbb6
attachments/ad43389872fe2834a4a1c30877bab1152ddac46fe09805222817ade182422346.json2,5 KBad43389872fe2834a4a1c30877bab1152ddac46fe09805222817ade182422346
attachments/ad6499278f1a36fac7d24708087d29e4c077899be8329f6dd4737e95fbf73f20.json1,9 KBad6499278f1a36fac7d24708087d29e4c077899be8329f6dd4737e95fbf73f20
attachments/adafbd7101f872d2f91ae4ced0e92014ba948342dbe89bc8fdbd3594cc59d04f.json188 Badafbd7101f872d2f91ae4ced0e92014ba948342dbe89bc8fdbd3594cc59d04f
attachments/add4ac65f249cd3f954f09e116df1242208ea845b464d0acd9ceb0a73a9074b8.json3,3 KBadd4ac65f249cd3f954f09e116df1242208ea845b464d0acd9ceb0a73a9074b8
attachments/afc78744fff0e0295c10176d61e07af68331c07d1d08407737b1b50c725d179c.json2,9 KBafc78744fff0e0295c10176d61e07af68331c07d1d08407737b1b50c725d179c
attachments/afca660be6bc7a62855347f87c6663549fc06967b0b1d7b391f96e0b32f96a5b.json176 Bafca660be6bc7a62855347f87c6663549fc06967b0b1d7b391f96e0b32f96a5b
attachments/b380824b62444d04dcaf3fe473c9e4f54e7fed829b81c777cfef02440b59001f.json3,7 KBb380824b62444d04dcaf3fe473c9e4f54e7fed829b81c777cfef02440b59001f
attachments/b4e81bdbac54a2780735dadb835972fda33b5a7ec4acb19a91c4790b0e2d5cec.json149 Bb4e81bdbac54a2780735dadb835972fda33b5a7ec4acb19a91c4790b0e2d5cec
attachments/b4faefae8975f199367e57472c2bd31f1c69dd4be187e07522434757dff87545.json868 Bb4faefae8975f199367e57472c2bd31f1c69dd4be187e07522434757dff87545
attachments/b71cd81e9a23141f8874c14ba702364a52433080af729f19d825667ec7d07a26.json1,9 KBb71cd81e9a23141f8874c14ba702364a52433080af729f19d825667ec7d07a26
attachments/ba930f1fbe4ef660084376693b78cb375031ad0cb12d8d844aa92c7cc0f1e450.json871 Bba930f1fbe4ef660084376693b78cb375031ad0cb12d8d844aa92c7cc0f1e450
attachments/bd5c0a2aba3749128b99bcd5924b923d4662391742b014e197374019698d4154.json302 Bbd5c0a2aba3749128b99bcd5924b923d4662391742b014e197374019698d4154
attachments/be1536c245bbdf4c85666cf6d8403981e9e5f03b21fcc9f8dec9759a62682df8.json9,7 KBbe1536c245bbdf4c85666cf6d8403981e9e5f03b21fcc9f8dec9759a62682df8
attachments/bf7389f9937205bbaa5bb9e3847e84f64607acf98a241fa22e60743cea07395a.json1,1 KBbf7389f9937205bbaa5bb9e3847e84f64607acf98a241fa22e60743cea07395a
attachments/c0063d02ddc4fa92175984f9a67c6e1763150165b3da516ae03ae25b1e4a0d36.json700 Bc0063d02ddc4fa92175984f9a67c6e1763150165b3da516ae03ae25b1e4a0d36
attachments/c08892b07a2d4b8129e2cceffcbcda3fe62158745146e58eadc4280a52866f9a.json3,6 KBc08892b07a2d4b8129e2cceffcbcda3fe62158745146e58eadc4280a52866f9a
attachments/c20998afbe68aba59d21fbdc7d325dc610ec3a7d7c67d6bf60fbad08be4b9230.json979 Bc20998afbe68aba59d21fbdc7d325dc610ec3a7d7c67d6bf60fbad08be4b9230
attachments/c3b256597ff701b8b287ad8223ebfb1bffa7d5f487cbcaaf466b2da843532e8e.json7,7 KBc3b256597ff701b8b287ad8223ebfb1bffa7d5f487cbcaaf466b2da843532e8e
attachments/c7fa26cc43b9e4ef95a75e2e10da5478280a550a595783dd52780b0f26971aa4.json1,8 KBc7fa26cc43b9e4ef95a75e2e10da5478280a550a595783dd52780b0f26971aa4
attachments/cb65b4b1f955bb2522467a4141cd5477904a8584f4049d244e1c756de0b988f7.json300 Bcb65b4b1f955bb2522467a4141cd5477904a8584f4049d244e1c756de0b988f7
attachments/cc1a570fa37bc5ba04776a6ad987003161e48ab51ae89ddea86455246a58ddee.json122 Bcc1a570fa37bc5ba04776a6ad987003161e48ab51ae89ddea86455246a58ddee
attachments/cc232dd8916f301bd239f0434eecd4e2dfcff9bb7c4d49102dfbace48a1f6f89.json1,5 KBcc232dd8916f301bd239f0434eecd4e2dfcff9bb7c4d49102dfbace48a1f6f89
attachments/ce60574e1b157d16493d1ffc21129b6be18c524b0f6dda54a503aa0a451a9081.json2,3 KBce60574e1b157d16493d1ffc21129b6be18c524b0f6dda54a503aa0a451a9081
attachments/cfacd5a5b9f0372926886019e59de558e208c25f0a8cc7d0b717d3e90ce34f19.json135 Bcfacd5a5b9f0372926886019e59de558e208c25f0a8cc7d0b717d3e90ce34f19
attachments/d0b183f5dc5b7397b81ed35d46eb0b17214aa9a8bd1a9098776b2fee62615071.json2,5 KBd0b183f5dc5b7397b81ed35d46eb0b17214aa9a8bd1a9098776b2fee62615071
attachments/d18c200471caced38394cfaecbd84f55885ba97999ee8cb768238690b4682102.json1,1 KBd18c200471caced38394cfaecbd84f55885ba97999ee8cb768238690b4682102
attachments/d23b44b8ee7cf00948ddce1ae524f430766686ffe8afd2e16d512ac174151de8.json3,7 KBd23b44b8ee7cf00948ddce1ae524f430766686ffe8afd2e16d512ac174151de8
attachments/d3103a8b05f9804fb3934ab5b96b79a0260bd36940c52ea6c2966fdb6e133863.json176 Bd3103a8b05f9804fb3934ab5b96b79a0260bd36940c52ea6c2966fdb6e133863
attachments/d43e0477a222747398fc31ccfdefac72847621c0635419d6a43365117bcefb25.json3,3 KBd43e0477a222747398fc31ccfdefac72847621c0635419d6a43365117bcefb25
attachments/d7514702cbcf2bace89b1b547796dec6591b45606d594995a794dd099f410aca.json122 Bd7514702cbcf2bace89b1b547796dec6591b45606d594995a794dd099f410aca
attachments/d95ca13566f44e045dff2a3b1098f6ab5b1ae4246c1e28443d298acb40366116.json187 Bd95ca13566f44e045dff2a3b1098f6ab5b1ae4246c1e28443d298acb40366116
attachments/dc4393c68090f61ddc751ab61e056554bcb6e381e3c70be4c1fee5c2b047eaf5.json1,1 KBdc4393c68090f61ddc751ab61e056554bcb6e381e3c70be4c1fee5c2b047eaf5
attachments/dd4261aea148e82247df6ad45053d1608d0e798f04aa4e4be07d6acef3f7bf1a.json587 Bdd4261aea148e82247df6ad45053d1608d0e798f04aa4e4be07d6acef3f7bf1a
attachments/de14a037452fe35482389634a3348e7781987c5958e5b1b9426e5ad4a600d207.json980 Bde14a037452fe35482389634a3348e7781987c5958e5b1b9426e5ad4a600d207
attachments/dfc9420f2ad64b06c21ac76c4676d2253ed0d0ad181fe80f5a638b497cb2aa4a.json1,1 KBdfc9420f2ad64b06c21ac76c4676d2253ed0d0ad181fe80f5a638b497cb2aa4a
attachments/e22f0bf72a8c6902ac85384d9dc1f5139fda047611406566d571c3d5cf52c845.json171 Be22f0bf72a8c6902ac85384d9dc1f5139fda047611406566d571c3d5cf52c845
attachments/e3708a1dbe26082e776a1dfbcc93ebc6ae42702adcaa60e9544d8d1a2c1a929a.json702 Be3708a1dbe26082e776a1dfbcc93ebc6ae42702adcaa60e9544d8d1a2c1a929a
attachments/e3d7e428b164c07cb6de9400e0bf34ffa953dbaa310feba0526e1c29677d0338.json299 Be3d7e428b164c07cb6de9400e0bf34ffa953dbaa310feba0526e1c29677d0338
attachments/e6755f0b00c1e2526daeaf821bf9818e932f3fd1fbe461cbdfae586d3dd74a3b.json142 Be6755f0b00c1e2526daeaf821bf9818e932f3fd1fbe461cbdfae586d3dd74a3b
attachments/e8a1d27116e6d1f1daa1e3a4f6e81b4cf1f484f02fab1b6a7858ddb54770b857.json131 Be8a1d27116e6d1f1daa1e3a4f6e81b4cf1f484f02fab1b6a7858ddb54770b857
attachments/ea4e9a916d850a49641c44ec82b5e53ed684a80d0dc6071fef1e642535b4032d.json700 Bea4e9a916d850a49641c44ec82b5e53ed684a80d0dc6071fef1e642535b4032d
attachments/eb4fbeafdccf5e2304d9c8f77027b61225a4c83aecda499ee45eac1954ae954c.json8,2 KBeb4fbeafdccf5e2304d9c8f77027b61225a4c83aecda499ee45eac1954ae954c
attachments/f1855c7f6ec671211acf1efefc5509707204f381d49badef1d3d40d88a0af430.json1,0 KBf1855c7f6ec671211acf1efefc5509707204f381d49badef1d3d40d88a0af430
attachments/f388d8c35125bceb67f4fef51b5a6a535b1c7b1a3952e869bc2523673d399630.json119 Bf388d8c35125bceb67f4fef51b5a6a535b1c7b1a3952e869bc2523673d399630
attachments/f5d6ba71b9e31570bedb983cccc74ad9925b34baf9ddcbb03c4125d3742d5241.json2,5 KBf5d6ba71b9e31570bedb983cccc74ad9925b34baf9ddcbb03c4125d3742d5241
attachments/ff591b081c8521138cf967efbcc7757b187c5a20f4729ef7deb49d2cbe06d926.json188 Bff591b081c8521138cf967efbcc7757b187c5a20f4729ef7deb49d2cbe06d926
env/platform.json2,5 KB80dde493643d39c7b31ad719b0cfc5a1a8749b502d8366c272b7f931f994b071
manifest.json11,4 KB53c24a5fc1ba15bad73e1a7009c75be07574ed5a78d21b868076c37443ed4bea
native/crb_papers/crb_papers.q001.e1.a1.json4,2 KB378d3799adfe05d96660a99a24ae74b7832fd9f896a1a7834704162aa1824c28
native/crb_papers/crb_papers.q002.e1.a1.json2,7 KB9bd6231a489f452d24b079c0de0b34edd9bbf9cdf727dd1bac959f835d038e72
native/crb_papers/label_map.json339 Bcba7ec05bb03a8f3f17b8cc4600b1fe92085e4eab21bcb3076ceed640fcf293d
results.jsonl13,5 KB350634d5c954262bb8de6e5003e8bac8f3373679ec2631bbc4aa12ba2db14ecc
sources/crb_papers.leaderboard.gpt53.causal_quantity_correct.json1,5 KB1cc8aaf136aa30c6120cd64def9bcb145b0227c7b28fdb4f7cd46176b7c0002c
sources/crb_papers.leaderboard.gpt53.controls_correct.json1,4 KBf6db2a4a1ab637b402249767da86f5a3235aa0ca7a59ebb57253418a44523385
sources/crb_papers.leaderboard.gpt53.did_identification_spec_correct.json1,7 KB8a4c965e850c7626c5a9c79b440729182a6db5d17fbffdb11877489f48d757cf
sources/crb_papers.leaderboard.gpt53.estimate_within_solution_ci.json1,4 KBdba4e6c5a3b9ab1a8f7485312d3a1d07166af503c20dda44a541342cef7007ec
sources/crb_papers.leaderboard.gpt53.identification_spec_correct.json1,5 KB121097c312c8a5f7546f69a26a1a5f52974b364f0d02309aaf5964dcc108cf6c
sources/crb_papers.leaderboard.gpt53.iv_identification_spec_correct.json1,7 KB0ab29b5028cad3edc19b790dbbbd0eccd0333f536a438677f412b1b87fd12295
sources/crb_papers.leaderboard.gpt53.median_percentage_error.json1,4 KB511ae8f774cb765bec3eb74e90b8a50f84bd7825a722439854729ce83c3931fd
sources/crb_papers.leaderboard.gpt53.null_hypothesis_decision_correct.json1,5 KB73b65323f48e784ec5efcbb9105f7b0408189f9db24bd723ab344a3787314c69
sources/crb_papers.leaderboard.gpt53.rd_identification_spec_correct.json1,7 KBd49a31c9c05393d8708eb12af35e5e7382bd4322077ebbbe4834a9d6033f94dc
sources/crb_papers.leaderboard.gpt53.strategy_correct.json1,4 KB7d9ae46d500d3f3618adecb2b724bf5c2552d19fdd85c543ec2183dfbfe1cc0d
sources/crb_papers.q001.gold.json1,1 KBf94b1cb4c5213c4c094560cf14adf1a1f7861fc70d814c2762854bcb3548bb1a
sources/crb_papers.q001.script.json1,1 KB6fc299a4e335e96d097faef7c25caf91f92d74495f0576a47e22a235e19f4483
sources/crb_papers.q002.gold.json1,1 KBf7e13dc9bffe76d7431f954732a6be55903161383abd942086865d850196e631
sources/crb_papers.q002.script.json1,1 KB0cf899fd404d607e1be86f5dcdd79b1df3a5cf14e5d0067a1c8d2267ffd56613
specs/crb_papers.q001.json11,7 KB44a5e5480015355a7aea2e2e1af1002757cbfc73f90de81ba14ea745091865ce
specs/crb_papers.q002.json11,7 KBa3c6b8bf77b51436dac39f782e54dbcfc9a80198527c85334dda1eb36e07d3f0
summary.json2,9 KB5e3cd1b963a23c371ec20e596a38d3d9874f293839a00151c66d692f1628225f
summary.md567 B78c407a270fcaa3f2858faccb2752ca1b3253ef03e47001a731aef2d944ab0b2
tools/registry.json14,3 KB644bdc32c1ef8e99eea9ba91f16d8c477a4c57d26b17990e22e346b33aa067fe
traces/crb_papers.q001.e1.a1.jsonl74,5 KBdb68ec312d688a0ac08fb2e683efee64bda23d5caaf343c1b82b2b5e3e8f97f8
traces/crb_papers.q002.e1.a1.jsonl71,6 KB20e4e2ae569d020959ed56ce5afec243b2ebba5274f1f505799a97b50cf560ca
trajectories/crb_papers.q001.e1.a1.jsonl69,3 KB5bd6f79a2201531aa2f2a37d06813da12bebf2cfaaf659c57c259670a88efc96
trajectories/crb_papers.q002.e1.a1.jsonl65,5 KBd1b5a372ece61f8bc1fff3469b1560016c9d003166acdd641f9f097d890eb01f
verify_bundle.py34,9 KB99e037f6a241e077348d7e116b03be17e727c4b61d278ea5f39b06dd557069a3
SHA256SUMS18,1 KB7fb2d0661f281248cd8817204dfa73050b8bc3aad4a0615cdba137da0b081c90

Eventos do registro

Verifique você mesmo

Resumo da execução
7fb2d0661f281248cd8817204dfa73050b8bc3aad4a0615cdba137da0b081c90
sha256 do arquivo SHA256SUMS; deve ser igual ao run_digest do evento final do registro.

Qualquer pessoa, sem o repositório

  1. Baixe o pacote inteiro e descompacte.
  2. Rode o verificador, que só usa a biblioteca padrão do Python:
    cd 20261015T124340Z-suite-crb_papers-gemini-f9eb57
    python3 verify_bundle.py

Ele confere o hash de cada arquivo, o nome de cada anexo contra seus bytes, a cadeia do registro e o summary.json recalculado a partir de results.jsonl.

Para conferir o resumo:

shasum -a 256 SHA256SUMS   # 7fb2d0661f281248cd8817204dfa73050b8bc3aad4a0615cdba137da0b081c90

Com acesso ao repositório (privado)

git checkout 38e979372a0f7382f285079e4380813081bdd032
uv sync --frozen --all-packages
uv run magi-bench verify bench-runs/20261015T124340Z-suite-crb_papers-gemini-f9eb57

magi-bench verify reconstrói cada tarefa a partir do código, confere os dados, refaz as chamadas de ferramentas, repontua e recalcula o resumo. Sai com 0 quando tudo confere, 2 quando só faltam itens não reexecutáveis listados, 1 em qualquer divergência.