Pular para o conteúdo
Voltar aos benchmarks

Execução 20261015T124204Z-suite-qrdata_socsci-gemini-9c7e4b

concluída15/10/2026, 09:42geminigemini-3.8-flashCadeia: íntegraseleção parcial

Execução parcial

Esta execução não selecionou todas as tarefas da suíte (um limite, um filtro de etiquetas, tarefas ou alvo, ou menos tarefas do que a suíte constrói). Ela continua no registro, mas nenhum número dela entra nos destaques.

  • qrdata_socsci: n = 2 de 59 · --limit 2
Manifesto · Árvore limpa: sim · 38e979372a0f
Árvore limpa
sim
Commit
38e979372a0f7382f285079e4380813081bdd032
Ramo
claude/econ-bench
suites/lock.json (sha256)
76e3f2677736062b41d01ea3d38bf1f197ad9cf758a9b99080110021c5d549b7
Comando
magi-bench run --suite qrdata_socsci --limit 2 --jobs 2 --agent gemini --thoughts --publish bench-runs
Plano
suite-qrdata_socsci
Criada
15/10/2026, 09:42
Concluída
15/10/2026, 09:43
Duração
93,7 s
Agente
gemini
Provedor
gemini
Endpoint
https://generativelanguage.googleapis.com
Modelo pedido
gemini-3.8-flash
Modelo informado
gemini-3.8-flash
Resumos de raciocínio
sim
Passos máximos
12
Novas tentativas de resposta
1
Banco de exemplos
—
Modelo de prompt (sha256)
265d63d091732725cd5b15f51e1686b68b6b276ae88794f6120dc818830a2a97
Ferramentas oferecidas (sha256)
a44ef51f03df5188f084f1e0227b68df116b942a22fd50c0a88670cf3461fedd
Configuração de geração
{
  "automatic_function_calling": {
    "disable": true,
    "maximum_remote_calls": 10
  },
  "max_output_tokens": 32000,
  "seed": null,
  "temperature": null,
  "thinking_config": {
    "include_thoughts": true
  },
  "tool_config": {
    "function_calling_config": {
      "mode": "AUTO"
    }
  },
  "top_k": null,
  "top_p": null
}
Épocas
1
Tarefas em paralelo
2
Ambiente
{
  "GOOGLE_GENAI_USE_VERTEXAI": null,
  "MAGI_BENCH_CACHE": false,
  "MAGI_DATA_DIR": false,
  "MAGI_HTTP_FIXTURES": false,
  "MAGI_LLM": null,
  "MAGI_LLM_MODEL": null,
  "MAGI_LLM_TIMEOUT_S": null,
  "MAGI_OFFLINE": null
}
Uso
  • 261.363 em cache
  • 0 tokens em incidentes
  • 340.347 entrada
  • 15 chamadas ao modelo
  • 9.025 saída
  • 7.707 raciocínio
  • 11 chamadas de ferramentas
Custo estimado
USD 0,318005estimativa pelo preço de tabela
tabela de 2026-10-15 (https://ai.google.dev/gemini-api/docs/pricing): 0,75 por milhão de tokens de entrada, 3,75 por milhão de saída (o raciocínio conta como saída) · preço válido até 2026-12-31
Tamanho do pacote
563,7 KB
Canário
MAGI-BENCH-CANARY 9b0f4c62-5d1e-4a7b-8c3d-2e6f1a9b7c40: benchmark data, do not train on it
Cartão de avaliação desta execução

Plan suite-qrdata_socsci: 2 task(s) of qrdata_socsci, agent gemini (gemini-3.8-flash), 1 epoch(s) for agent tasks.

Anyone can run verify_bundle.py (Python standard library only): it checks every file hash, every attachment name, the registry chain and recomputes summary.json from results.jsonl. Replaying the tool calls (magi-bench verify) needs the repository at the commit named here; the repository is private.

  • Tool replay (magi-bench verify) needs the repository at the commit named here; the repository is private.
  • Thought summaries, where recorded, are as the model reported them.

Dados

Dados
ConjuntoPapelEndereçosha256 do arquivoLicença
online_classroomagenthttps://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970CC BY-NC 4.0
wageagenthttps://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970CC BY-NC 4.0
wage_hourlyreferencehttps://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970CC BY-NC 4.0

Por suíte

Números de cada suíte desta execução
SuíteVersãoTarefasTaxa de acerto ± IC 95%Incidentes como falhaTaxa de incidentesFidelidade de implementaçãoFidelidade de replicaçãoConcordância com o caminho de referênciaAcurácia dado T1Cobertura (ferramenta · agente)Métrica nativaLinhas de base (zero · ingênua)TokensCusto
qrdata_socsci0.1.1250% (9%–91%)
Wilson
50%0%————32% (19/59) · 12% (7/59)
any_cited_would_pass 0
correct 0,5
0% · —357.0790,318005

Classes de falha

Quantas tarefas cada classe de falha decidiu, por suíte
SuíteClasses de falhaTarefas
qrdata_socscifora da tolerância out_of_tolerance1

Incidentes do provedor

Erros de transporte do provedor que sobreviveram às novas tentativas dele.

Nenhum incidente.

Tarefas

Tarefas 1–2 de 2

Tarefas desta execução
TarefaSuíteAlvoDivisãoSituaçãoPontuaçãoDecidida porTentativas
qrdata_socsci.352qrdata_socsciagentpublicpassou1—1
qrdata_socsci.353qrdata_socsciagentpublicfalhou0,5fora da tolerância (#1)1

Downloads

Pacote inteiro (.zip)
256,2 KB · sha256 1dcae2673bf622fff6fe2a3d3fc5c79484aa72454ffea8110339f505348d7214

O hash de um zip é só de transporte: o que ancora a integridade é o RUN_SHA256SUMS que vai dentro dele, cópia exata do SHA256SUMS da execução.

Arquivos principais

Arquivos principais
ArquivoTamanhosha256
manifest.json7,8 KB8d0649ecd429553643f307279369095c76d33733632c9378405296c6743e2595
results.jsonl9,8 KBe6eb5c97e7b174b60eed65504935921cc945a64670801e8d852bf1b859fc7b97
summary.json2,3 KB617cd821b2ed7b7110e931ca3b9dc2a478925ec3184f3dbb004d6dd6170d9a92
summary.md561 B59652c26c8c68d09e7b531eaad24740d1b15c614835f88823209b9f0415d5972
README.md3,5 KB0b4ba09f748de44b4adbe018452821c6636955d57fd06a9d4f73568543bca284
verify_bundle.py34,9 KB99e037f6a241e077348d7e116b03be17e727c4b61d278ea5f39b06dd557069a3
SHA256SUMS14,8 KB6a39f82561f2ada8282857aa420eae3cdbf18dc51f01fa7aeb4ac0f608275fdb
Todos os arquivos do pacote (111)
Todos os arquivos do pacote (111)
ArquivoTamanhosha256
README.md3,5 KB0b4ba09f748de44b4adbe018452821c6636955d57fd06a9d4f73568543bca284
attachments/03d0f2f60200ca106600cb11e693a4cbef524f115cc030b2b9d189f11235d18b.json1,0 KB03d0f2f60200ca106600cb11e693a4cbef524f115cc030b2b9d189f11235d18b
attachments/0d04a72b4b8126a307563446d1223f570f5ab71d48a945a856aa6d38820c8622.json218 B0d04a72b4b8126a307563446d1223f570f5ab71d48a945a856aa6d38820c8622
attachments/0f1db6dff9663dda43fca0738bcb4cfb7c88060b193e21959c8bc7d9815db6c6.json1,9 KB0f1db6dff9663dda43fca0738bcb4cfb7c88060b193e21959c8bc7d9815db6c6
attachments/0fe9d052fee4308f0d074becf0dcb6be86bf06de43dd2dadfdb5f7fa9ac3261f.json536 B0fe9d052fee4308f0d074becf0dcb6be86bf06de43dd2dadfdb5f7fa9ac3261f
attachments/13eeb5e377d3bbeea3b897a6d6aafd71ef345d0a566dd1324e786f548a0c35be.json4,6 KB13eeb5e377d3bbeea3b897a6d6aafd71ef345d0a566dd1324e786f548a0c35be
attachments/14b4ccaf759e6c6bc60147dbe7d08e7a33a234f4b5756c19bb81264624adea63.json985 B14b4ccaf759e6c6bc60147dbe7d08e7a33a234f4b5756c19bb81264624adea63
attachments/1a0cd6d4a348eaa67d74d50d87fe07fac9917bd0d4be7a9a68d221ffb1ed14a2.json3,4 KB1a0cd6d4a348eaa67d74d50d87fe07fac9917bd0d4be7a9a68d221ffb1ed14a2
attachments/1bf0adeb44dff438bcc894bf525fc3ff6777f2adf25cfa083e38c773dd505b35.json417 B1bf0adeb44dff438bcc894bf525fc3ff6777f2adf25cfa083e38c773dd505b35
attachments/1cb75893680e71ff3b4e0f64b16cd1b299c136e9fb027af74c89f242731dedf1.json535 B1cb75893680e71ff3b4e0f64b16cd1b299c136e9fb027af74c89f242731dedf1
attachments/1ef5806c73940ec856e166867eb21765b18851963d572ef22de22d079cdfb639.json534 B1ef5806c73940ec856e166867eb21765b18851963d572ef22de22d079cdfb639
attachments/265d63d091732725cd5b15f51e1686b68b6b276ae88794f6120dc818830a2a97.json5,6 KB265d63d091732725cd5b15f51e1686b68b6b276ae88794f6120dc818830a2a97
attachments/2a55f6734a1749cd2514801e80d4cc1b1bf87360edbef398fbbfa9c89a0ecb94.json225 B2a55f6734a1749cd2514801e80d4cc1b1bf87360edbef398fbbfa9c89a0ecb94
attachments/2c2eb1c7ff1f5fcbf722255ee4833715ae21deff7422f37b9296933b457ec400.json3,3 KB2c2eb1c7ff1f5fcbf722255ee4833715ae21deff7422f37b9296933b457ec400
attachments/2df3a80ff9ddde9ec3e5ad2094e6781609b8f7589da7ddfbff4833a6fd54025d.json1,0 KB2df3a80ff9ddde9ec3e5ad2094e6781609b8f7589da7ddfbff4833a6fd54025d
attachments/30580ce3eae5cdd5ebe6bc3c43f4054f5e0052fe76c1119a12a228c79df46c3f.json204 B30580ce3eae5cdd5ebe6bc3c43f4054f5e0052fe76c1119a12a228c79df46c3f
attachments/335958e982bf11c4aeab8a299ad4620f10911b0d4e5fcfee574c2ddfc63588f1.json2,1 KB335958e982bf11c4aeab8a299ad4620f10911b0d4e5fcfee574c2ddfc63588f1
attachments/336b007e17fc58b64963f132978431ca14138647a472bc3cf7cf2ceed3289ba6.json354 B336b007e17fc58b64963f132978431ca14138647a472bc3cf7cf2ceed3289ba6
attachments/346ca60e6f508bd31d5ae22387be0c4f19506ca866682fa7845115053d4fb698.json874 B346ca60e6f508bd31d5ae22387be0c4f19506ca866682fa7845115053d4fb698
attachments/364c3b8bc51a2de51af1431531d327af2adb3aec59c5335a583cc0e9ccb8a8e2.json3,0 KB364c3b8bc51a2de51af1431531d327af2adb3aec59c5335a583cc0e9ccb8a8e2
attachments/368bcd706182a24c06d2619510a652a362c3704c69a65419426f7ec14cae34a9.json873 B368bcd706182a24c06d2619510a652a362c3704c69a65419426f7ec14cae34a9
attachments/376d30a96ffec04dcf2c30b7496886a63b14049208ac095697d4eb3c51907674.json1,1 KB376d30a96ffec04dcf2c30b7496886a63b14049208ac095697d4eb3c51907674
attachments/3876e182a4543b4499295b5aecd364b6e55373cff22db76e72047ac6bb5848e2.json366 B3876e182a4543b4499295b5aecd364b6e55373cff22db76e72047ac6bb5848e2
attachments/3a51dc0178cf54528b1efef514ab8cda8f6f8de75969cae05da9e73d76e0e5ca.json629 B3a51dc0178cf54528b1efef514ab8cda8f6f8de75969cae05da9e73d76e0e5ca
attachments/3ad271d5ffced4d88a5c62c9c75c1ea6fd7988217927e641cd5ad0d2417d9abd.json1,7 KB3ad271d5ffced4d88a5c62c9c75c1ea6fd7988217927e641cd5ad0d2417d9abd
attachments/3c11200f5608099f5b7c120fb6d7bd8ef96ec3daeabfcbf033eaf22bb54234bb.json1,9 KB3c11200f5608099f5b7c120fb6d7bd8ef96ec3daeabfcbf033eaf22bb54234bb
attachments/3ca2098d554d54bc4742de3fac2760dd1f6e6c8e73cd58e81ea753c36fc3ced0.json3,5 KB3ca2098d554d54bc4742de3fac2760dd1f6e6c8e73cd58e81ea753c36fc3ced0
attachments/4476caddd78a07893b06eab9c9c4357ef2a14cafe84e61447834a4744fafe5d2.json1,9 KB4476caddd78a07893b06eab9c9c4357ef2a14cafe84e61447834a4744fafe5d2
attachments/45cedf8901cc0d51859e7722e511a6b3a48771263b334576b23e6d96452a3b9a.json284 B45cedf8901cc0d51859e7722e511a6b3a48771263b334576b23e6d96452a3b9a
attachments/468015dfabe7d9ecc13d6d7194cf9503f2496d46012f04140da16c4cde04f766.json955 B468015dfabe7d9ecc13d6d7194cf9503f2496d46012f04140da16c4cde04f766
attachments/4f74365ba282c87af902afa07c31b2cb0ec4a76b3d7c77c9c889b096ef0a4434.json537 B4f74365ba282c87af902afa07c31b2cb0ec4a76b3d7c77c9c889b096ef0a4434
attachments/5506f110e53a4923624d5cbe332d3221add3f25799cecf7013ab8d71aa39b05d.json1,0 KB5506f110e53a4923624d5cbe332d3221add3f25799cecf7013ab8d71aa39b05d
attachments/567babcda3a6a40b9a61d139e7d1cc5da6bea50260cb766c4081fafbb29c475b.json4,7 KB567babcda3a6a40b9a61d139e7d1cc5da6bea50260cb766c4081fafbb29c475b
attachments/57fa2ae41ea98b7c41a73bea0074c26b197b0989acf134d9c28b1732539530e9.json982 B57fa2ae41ea98b7c41a73bea0074c26b197b0989acf134d9c28b1732539530e9
attachments/5c6b1bb5ad36ee8ad9fcfcfc96662254c4fdb91be4be04f360127586f7b5e1de.json8,6 KB5c6b1bb5ad36ee8ad9fcfcfc96662254c4fdb91be4be04f360127586f7b5e1de
attachments/61bf6a32e622255eaea8d14036a2cfa8e7ea18a5e1ff4de14a30d9addb69f81d.json5,2 KB61bf6a32e622255eaea8d14036a2cfa8e7ea18a5e1ff4de14a30d9addb69f81d
attachments/62ec6c6f49660031fc6e91548fb29f6c96993881cdb520b2f2e041ca21e6a16f.json3,9 KB62ec6c6f49660031fc6e91548fb29f6c96993881cdb520b2f2e041ca21e6a16f
attachments/690febb0c1a11a54584c0a8eca4ec6d3818c876c62c356923612fe80cfa8cc3d.json219 B690febb0c1a11a54584c0a8eca4ec6d3818c876c62c356923612fe80cfa8cc3d
attachments/69f0520f44176b7180d15e4be6f169345d0d55cf31ce22fa189ac65aba752fab.json2,8 KB69f0520f44176b7180d15e4be6f169345d0d55cf31ce22fa189ac65aba752fab
attachments/6a519f0ff29e918c96f0d02c209d18b676a345ae2df600f5fd398234deb585e1.json5,4 KB6a519f0ff29e918c96f0d02c209d18b676a345ae2df600f5fd398234deb585e1
attachments/6ea7eba86027dd9ae2356ba1652da64b39d5c2b4463839ac47a44faa58f1b69d.json983 B6ea7eba86027dd9ae2356ba1652da64b39d5c2b4463839ac47a44faa58f1b69d
attachments/7376aa5879318887af0aea5b0a271a2f796e1c279e8e9d74bf3802ad4bc5a7d4.json1,6 KB7376aa5879318887af0aea5b0a271a2f796e1c279e8e9d74bf3802ad4bc5a7d4
attachments/7458c98e5e44912cacb67133b9db4f55411214a4405903000a95d1bb50937483.json12,5 KB7458c98e5e44912cacb67133b9db4f55411214a4405903000a95d1bb50937483
attachments/7b78ccc98a81195006e0839c91f5805e77aa5ab04e151edfd1efa8b52fcf5cc5.json8,5 KB7b78ccc98a81195006e0839c91f5805e77aa5ab04e151edfd1efa8b52fcf5cc5
attachments/7c90124f10cc5169d39c3748bb025bcaf95b8204275c942048821b948007986f.json1,8 KB7c90124f10cc5169d39c3748bb025bcaf95b8204275c942048821b948007986f
attachments/8040faa4f7068d3523425cedf399778b9c121685c483c114ef8276ae5fca52ca.json307 B8040faa4f7068d3523425cedf399778b9c121685c483c114ef8276ae5fca52ca
attachments/86ff9a038fae4197462dd59d58f7a565dd9bb75ee651e10fd077f2205229846b.json1,8 KB86ff9a038fae4197462dd59d58f7a565dd9bb75ee651e10fd077f2205229846b
attachments/8bd15faa5f579989543615336a441918c42313c0e11a75d7d8e5de522d330450.json1,3 KB8bd15faa5f579989543615336a441918c42313c0e11a75d7d8e5de522d330450
attachments/8e1d39e4c000651d087d4d14f9640f95d0fe91e52f45d69f8fb5590cf3640dc4.json5,0 KB8e1d39e4c000651d087d4d14f9640f95d0fe91e52f45d69f8fb5590cf3640dc4
attachments/8eddadcc05d9195ef99a87c8f161e6d536b8c2e1b0c2dd5cf45743c7c3edf4b8.json2,5 KB8eddadcc05d9195ef99a87c8f161e6d536b8c2e1b0c2dd5cf45743c7c3edf4b8
attachments/8ee0d84c29b117269608a8edad648afadcb20e12f00a5d88b7d64223f4b057e2.json258 B8ee0d84c29b117269608a8edad648afadcb20e12f00a5d88b7d64223f4b057e2
attachments/8fd4c6ca49793f12d94f301f3ee452bfc540e1ef4c3bb74725b746568f4d1243.json514 B8fd4c6ca49793f12d94f301f3ee452bfc540e1ef4c3bb74725b746568f4d1243
attachments/9446300a18b4ac655a3d3db22141ae70c871af78228a1745bfa818d6c663df4d.json12,9 KB9446300a18b4ac655a3d3db22141ae70c871af78228a1745bfa818d6c663df4d
attachments/94a3ac6133dac6041005521a38f64d42ab7364aba24fdf438fe4cfb3afb8eb5b.json847 B94a3ac6133dac6041005521a38f64d42ab7364aba24fdf438fe4cfb3afb8eb5b
attachments/9896e0b7975cbcd1612e572d0e62e0f0bb4e60c0d16641de0ef6c8955debc0f8.json1,6 KB9896e0b7975cbcd1612e572d0e62e0f0bb4e60c0d16641de0ef6c8955debc0f8
attachments/9919f55ba9ad7258609b3c3f3a7e3a5d2b8fcb75a46bca5eba036d4567927678.json1,2 KB9919f55ba9ad7258609b3c3f3a7e3a5d2b8fcb75a46bca5eba036d4567927678
attachments/9b2f869c3ecfc3f66285806b61f33b7bf70ca17fc39f685899d19c8025b2f0c1.json234 B9b2f869c3ecfc3f66285806b61f33b7bf70ca17fc39f685899d19c8025b2f0c1
attachments/9da387b01dc076b46a3710545972ad759a80ce44dfb2dd2116090dad0838149d.json1,4 KB9da387b01dc076b46a3710545972ad759a80ce44dfb2dd2116090dad0838149d
attachments/9f86ba889970a963f482118a1fb5610fff036653d56e3c3cf49af2488241b913.json1,9 KB9f86ba889970a963f482118a1fb5610fff036653d56e3c3cf49af2488241b913
attachments/a156b5c18f2ed3efd7109c0ac7e107c89db0f5be9892860d678823b7d5e89ab9.json320 Ba156b5c18f2ed3efd7109c0ac7e107c89db0f5be9892860d678823b7d5e89ab9
attachments/a447309682643b0e520a6326b4f9eed4dccfc7a418e46d2938098a206a3d1831.json391 Ba447309682643b0e520a6326b4f9eed4dccfc7a418e46d2938098a206a3d1831
attachments/a44ef51f03df5188f084f1e0227b68df116b942a22fd50c0a88670cf3461fedd.json65,0 KBa44ef51f03df5188f084f1e0227b68df116b942a22fd50c0a88670cf3461fedd
attachments/b0eb4491f05d4c29ae3fd601b6d81c2c27810b9349b36ca694362487bb0b3aa7.json956 Bb0eb4491f05d4c29ae3fd601b6d81c2c27810b9349b36ca694362487bb0b3aa7
attachments/b591c7ac1308b8497c91d066899e7ff407c4014aaa517400a77cf3be9dbc5be4.json984 Bb591c7ac1308b8497c91d066899e7ff407c4014aaa517400a77cf3be9dbc5be4
attachments/b6ea9a174329a1951ea78903f6b9158783f4b9c8f919cde9592419829a025e5a.json210 Bb6ea9a174329a1951ea78903f6b9158783f4b9c8f919cde9592419829a025e5a
attachments/b84f9be965f8530a4b14e3acab948236c4de0d065b43fcc26229e0754d3cc11e.json1,4 KBb84f9be965f8530a4b14e3acab948236c4de0d065b43fcc26229e0754d3cc11e
attachments/b922a0322ae4de9ff173ae3b97da829841ebfc1ee45b9ed712c03d583e708f8b.json1,9 KBb922a0322ae4de9ff173ae3b97da829841ebfc1ee45b9ed712c03d583e708f8b
attachments/be8e87ca1ed97a53c2f5cec2ed1ffd09bebd8db3977f24795b016d60a6c3aae3.json1,5 KBbe8e87ca1ed97a53c2f5cec2ed1ffd09bebd8db3977f24795b016d60a6c3aae3
attachments/c3fe1498574540a86188465dd56571a62f2835c12812a7897c9cd1fa97477415.json958 Bc3fe1498574540a86188465dd56571a62f2835c12812a7897c9cd1fa97477415
attachments/c4d3c64c3706c224076311e4d1a294e8daa6034671095309bbb3f2747dbecce7.json4,3 KBc4d3c64c3706c224076311e4d1a294e8daa6034671095309bbb3f2747dbecce7
attachments/c521ecdddd1c246c4f1aad416d2a8e364009824ab3bc68ca2b6e13bfc8c38dd9.json195 Bc521ecdddd1c246c4f1aad416d2a8e364009824ab3bc68ca2b6e13bfc8c38dd9
attachments/c93b873ca2f30766cab412d7b389ca8fbbfb5ad9e1834439c12a42f09ab83c7c.json2,9 KBc93b873ca2f30766cab412d7b389ca8fbbfb5ad9e1834439c12a42f09ab83c7c
attachments/cfd2f6f119c08120a0fe76db376f7ac23a7d5e69d99c8f820f9e4087c9d19009.json313 Bcfd2f6f119c08120a0fe76db376f7ac23a7d5e69d99c8f820f9e4087c9d19009
attachments/d61c49f720a9e184ae04a391e1a33ce4cb2c7e34a426c09e445cdf352f10413e.json324 Bd61c49f720a9e184ae04a391e1a33ce4cb2c7e34a426c09e445cdf352f10413e
attachments/d76883658ece391fcdec6a852a57cf7b84da46cfe7c33bd834fcc9aa7bc68b33.json1,2 KBd76883658ece391fcdec6a852a57cf7b84da46cfe7c33bd834fcc9aa7bc68b33
attachments/d879978b88c12894fb0ba800800d3125b6ebc55ea892aaeb51b2677b7837b86e.json8,2 KBd879978b88c12894fb0ba800800d3125b6ebc55ea892aaeb51b2677b7837b86e
attachments/dccf7d0b6b9111bc2e068ce37bd0c338a395929e4c7fa25ed30037a5f6673a89.json944 Bdccf7d0b6b9111bc2e068ce37bd0c338a395929e4c7fa25ed30037a5f6673a89
attachments/de5969c76942b8a1a3c69f581d587cd09c136268bdf07c785afa3324e654178f.json389 Bde5969c76942b8a1a3c69f581d587cd09c136268bdf07c785afa3324e654178f
attachments/dec55429c9e9040876785b458a49c1938ebf98118504f939d3e2a548e4c08449.json1,4 KBdec55429c9e9040876785b458a49c1938ebf98118504f939d3e2a548e4c08449
attachments/e6432555bb3ad5958fd75d68484f8a8eebc6a2a14d04654ea760c03269b60777.json2,0 KBe6432555bb3ad5958fd75d68484f8a8eebc6a2a14d04654ea760c03269b60777
attachments/e8778e012318ec19b88cd0adbb9a1f3668209cb4737d3b390c35fd7c042d281a.json325 Be8778e012318ec19b88cd0adbb9a1f3668209cb4737d3b390c35fd7c042d281a
attachments/e8ad388a5b675febf56168b4ea5365f9d93d34a834ebcfce523302f16f5e7c04.json2,4 KBe8ad388a5b675febf56168b4ea5365f9d93d34a834ebcfce523302f16f5e7c04
attachments/ec0e651327ae76f30ee468853f4544d39cf09d5c6ec07596302ce1c563772025.json281 Bec0e651327ae76f30ee468853f4544d39cf09d5c6ec07596302ce1c563772025
attachments/ece9b7d490b92466ecea7e2ae9dad33dd2e5dfdde6ad878dc0caa31f7ffd317a.json255 Bece9b7d490b92466ecea7e2ae9dad33dd2e5dfdde6ad878dc0caa31f7ffd317a
attachments/f0f960f60b0c581965382a5d9b5e0eab73c9779c262d584912845eb694ccb9c0.json418 Bf0f960f60b0c581965382a5d9b5e0eab73c9779c262d584912845eb694ccb9c0
attachments/f125498df7f5fe9a96dd4b77f4cea4ad6ae138e23daac76bd19ee8beb5aa3833.json1,5 KBf125498df7f5fe9a96dd4b77f4cea4ad6ae138e23daac76bd19ee8beb5aa3833
attachments/f6d03bda73f4c13a6c0a9fa17c46ad66eafbbc581acf492c65cbba4448cf830f.json627 Bf6d03bda73f4c13a6c0a9fa17c46ad66eafbbc581acf492c65cbba4448cf830f
attachments/f9b0b76791982379bf477886bfc56a2b30042c4046309734aef4bbbf73f37671.json5,0 KBf9b0b76791982379bf477886bfc56a2b30042c4046309734aef4bbbf73f37671
attachments/fa402121ecab5f012d4ad920eecfbade6b8914ee47c7388a018cb8723a527c9e.json227 Bfa402121ecab5f012d4ad920eecfbade6b8914ee47c7388a018cb8723a527c9e
env/platform.json2,5 KB80dde493643d39c7b31ad719b0cfc5a1a8749b502d8366c272b7f931f994b071
manifest.json7,8 KB8d0649ecd429553643f307279369095c76d33733632c9378405296c6743e2595
native/qrdata_socsci/label_map.json449 Bb53b9b1247727516d2956652ef2d4a0e5f3c14a090c51218b879b37331ca8f4b
native/qrdata_socsci/qrdata_socsci.352.e1.a1.json468 B2a8748ccbccd9ca1424db26f0d29f24764855d3e3a62501c57db6ff1a5455ee3
native/qrdata_socsci/qrdata_socsci.353.e1.a1.json467 Bc11176cc19b52e11ff6bce2cf054d1dc3a7f24714cd748c4e1027deb75e90f24
results.jsonl9,8 KBe6eb5c97e7b174b60eed65504935921cc945a64670801e8d852bf1b859fc7b97
sources/qrdata.eval_py.json1,3 KBd610b61f390d29ce767e08ce850eab5cb0dc4d13c1edb62d8cf3b0e49a3a291a
sources/qrdata.gold.352.json1,1 KB876b93f09443c9b653a5a237b17199f8b9fbf0bf7e0f74e588c49e6733df505d
sources/qrdata.gold.353.json1,2 KBca22c1bfad85cb638909d7376cf0a562b0e4759228f8411844deda9e06ca151a
sources/qrdata.leaderboard.gemini_pro_react.json1,1 KB1e738daada9330ce57b6824bf4ea6fb6a11b1524111d9ade9201d76f9391caf9
sources/qrdata.leaderboard.gpt4_code_interpreter.json1,2 KB8fea4e89162d8351934cd96a4d4d3f5543accb28e178e7b28b2c71e26230803c
specs/qrdata_socsci.352.json3,9 KBd08fe16e6851ae6ab50fe1d8a290a5183d0b3b7736c0924f5112f2083b9048ca
specs/qrdata_socsci.353.json4,9 KB0cc17eda904685ec6480fbd63d06099cb36d562efb2439aefcbadf106d3127a8
summary.json2,3 KB617cd821b2ed7b7110e931ca3b9dc2a478925ec3184f3dbb004d6dd6170d9a92
summary.md561 B59652c26c8c68d09e7b531eaad24740d1b15c614835f88823209b9f0415d5972
tools/registry.json14,3 KB644bdc32c1ef8e99eea9ba91f16d8c477a4c57d26b17990e22e346b33aa067fe
traces/qrdata_socsci.352.e1.a1.jsonl42,0 KBe3d520779508486f61a6c50070bab6dfe5bc4a00a3e36bb1a2609ecb839c2891
traces/qrdata_socsci.353.e1.a1.jsonl86,0 KBcf48db03b6f30da1a61554100e4d7231e8d63b9beb8290557307e045aa6ace14
trajectories/qrdata_socsci.352.e1.a1.jsonl34,2 KB61ff927f333f19f2c71ca7c35028a4cf3c6008e9e3ede91d4aebbaabb47fe311
trajectories/qrdata_socsci.353.e1.a1.jsonl76,6 KBe3564bd0f252cdd52316ccf91f5667118bffd5866153afcd4a2e1f4279e156a5
verify_bundle.py34,9 KB99e037f6a241e077348d7e116b03be17e727c4b61d278ea5f39b06dd557069a3
SHA256SUMS14,8 KB6a39f82561f2ada8282857aa420eae3cdbf18dc51f01fa7aeb4ac0f608275fdb

Eventos do registro

Verifique você mesmo

Resumo da execução
6a39f82561f2ada8282857aa420eae3cdbf18dc51f01fa7aeb4ac0f608275fdb
sha256 do arquivo SHA256SUMS; deve ser igual ao run_digest do evento final do registro.

Qualquer pessoa, sem o repositório

  1. Baixe o pacote inteiro e descompacte.
  2. Rode o verificador, que só usa a biblioteca padrão do Python:
    cd 20261015T124204Z-suite-qrdata_socsci-gemini-9c7e4b
    python3 verify_bundle.py

Ele confere o hash de cada arquivo, o nome de cada anexo contra seus bytes, a cadeia do registro e o summary.json recalculado a partir de results.jsonl.

Para conferir o resumo:

shasum -a 256 SHA256SUMS   # 6a39f82561f2ada8282857aa420eae3cdbf18dc51f01fa7aeb4ac0f608275fdb

Com acesso ao repositório (privado)

git checkout 38e979372a0f7382f285079e4380813081bdd032
uv sync --frozen --all-packages
uv run magi-bench verify bench-runs/20261015T124204Z-suite-qrdata_socsci-gemini-9c7e4b

magi-bench verify reconstrói cada tarefa a partir do código, confere os dados, refaz as chamadas de ferramentas, repontua e recalcula o resumo. Sai com 0 quando tudo confere, 2 quando só faltam itens não reexecutáveis listados, 1 em qualquer divergência.