Execução 20261015T124204Z-suite-qrdata_socsci-gemini-9c7e4b
Execução parcial
Esta execução não selecionou todas as tarefas da suíte (um limite, um filtro de etiquetas, tarefas ou alvo, ou menos tarefas do que a suíte constrói). Ela continua no registro, mas nenhum número dela entra nos destaques.
- qrdata_socsci: n = 2 de 59 · --limit 2
Manifesto · Árvore limpa: sim · 38e979372a0f
- Árvore limpa
- sim
- Commit
38e979372a0f7382f285079e4380813081bdd032- Ramo
- claude/econ-bench
- suites/lock.json (sha256)
76e3f2677736062b41d01ea3d38bf1f197ad9cf758a9b99080110021c5d549b7- Comando
magi-bench run --suite qrdata_socsci --limit 2 --jobs 2 --agent gemini --thoughts --publish bench-runs- Plano
suite-qrdata_socsci- Criada
- 15/10/2026, 09:42
- Concluída
- 15/10/2026, 09:43
- Duração
- 93,7 s
- Agente
gemini- Provedor
- gemini
- Endpoint
https://generativelanguage.googleapis.com- Modelo pedido
gemini-3.8-flash- Modelo informado
gemini-3.8-flash- Resumos de raciocínio
- sim
- Passos máximos
- 12
- Novas tentativas de resposta
- 1
- Banco de exemplos
- —
- Modelo de prompt (sha256)
265d63d091732725cd5b15f51e1686b68b6b276ae88794f6120dc818830a2a97- Ferramentas oferecidas (sha256)
a44ef51f03df5188f084f1e0227b68df116b942a22fd50c0a88670cf3461fedd- Configuração de geração
{ "automatic_function_calling": { "disable": true, "maximum_remote_calls": 10 }, "max_output_tokens": 32000, "seed": null, "temperature": null, "thinking_config": { "include_thoughts": true }, "tool_config": { "function_calling_config": { "mode": "AUTO" } }, "top_k": null, "top_p": null }- Épocas
- 1
- Tarefas em paralelo
- 2
- Ambiente
{ "GOOGLE_GENAI_USE_VERTEXAI": null, "MAGI_BENCH_CACHE": false, "MAGI_DATA_DIR": false, "MAGI_HTTP_FIXTURES": false, "MAGI_LLM": null, "MAGI_LLM_MODEL": null, "MAGI_LLM_TIMEOUT_S": null, "MAGI_OFFLINE": null }- Uso
- 261.363 em cache
- 0 tokens em incidentes
- 340.347 entrada
- 15 chamadas ao modelo
- 9.025 saída
- 7.707 raciocínio
- 11 chamadas de ferramentas
- Custo estimado
- USD 0,318005estimativa pelo preço de tabelatabela de 2026-10-15 (https://ai.google.dev/gemini-api/docs/pricing): 0,75 por milhão de tokens de entrada, 3,75 por milhão de saída (o raciocínio conta como saída) · preço válido até 2026-12-31
- Tamanho do pacote
- 563,7 KB
- Canário
MAGI-BENCH-CANARY 9b0f4c62-5d1e-4a7b-8c3d-2e6f1a9b7c40: benchmark data, do not train on it- Cartão de avaliação desta execução
Plan suite-qrdata_socsci: 2 task(s) of qrdata_socsci, agent gemini (gemini-3.8-flash), 1 epoch(s) for agent tasks.
Anyone can run verify_bundle.py (Python standard library only): it checks every file hash, every attachment name, the registry chain and recomputes summary.json from results.jsonl. Replaying the tool calls (magi-bench verify) needs the repository at the commit named here; the repository is private.
- Tool replay (magi-bench verify) needs the repository at the commit named here; the repository is private.
- Thought summaries, where recorded, are as the model reported them.
Dados
| Conjunto | Papel | Endereço | sha256 do arquivo | Licença |
|---|---|---|---|---|
| online_classroom | agent | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | 3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970 | CC BY-NC 4.0 |
| wage | agent | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | 3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970 | CC BY-NC 4.0 |
| wage_hourly | reference | https://raw.githubusercontent.com/xxxiaol/QRData/de450af45ff7101b328bb064c6b475f73414a7ed/benchmark/data.zip | 3c3a97cca6fdd96f6856f7856b5f8bc786a31d1a9be78c9f610ef3070783e970 | CC BY-NC 4.0 |
Por suíte
| Suíte | Versão | Tarefas | Taxa de acerto ± IC 95% | Incidentes como falha | Taxa de incidentes | Fidelidade de implementação | Fidelidade de replicação | Concordância com o caminho de referência | Acurácia dado T1 | Cobertura (ferramenta · agente) | Métrica nativa | Linhas de base (zero · ingênua) | Tokens | Custo |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| qrdata_socsci | 0.1.1 | 2 | 50% (9%–91%) Wilson | 50% | 0% | — | — | — | — | 32% (19/59) · 12% (7/59) | any_cited_would_pass 0 correct 0,5 | 0% · — | 357.079 | 0,318005 |
Classes de falha
| Suíte | Classes de falha | Tarefas |
|---|---|---|
| qrdata_socsci | fora da tolerância out_of_tolerance | 1 |
Incidentes do provedor
Erros de transporte do provedor que sobreviveram às novas tentativas dele.
Nenhum incidente.
Tarefas
Tarefas 1–2 de 2
| Tarefa | Suíte | Alvo | Divisão | Situação | Pontuação | Decidida por | Tentativas |
|---|---|---|---|---|---|---|---|
| qrdata_socsci.352 | qrdata_socsci | agent | public | passou | 1 | — | 1 |
| qrdata_socsci.353 | qrdata_socsci | agent | public | falhou | 0,5 | fora da tolerância (#1) | 1 |
Downloads
1dcae2673bf622fff6fe2a3d3fc5c79484aa72454ffea8110339f505348d7214O hash de um zip é só de transporte: o que ancora a integridade é o RUN_SHA256SUMS que vai dentro dele, cópia exata do SHA256SUMS da execução.
Arquivos principais
| Arquivo | Tamanho | sha256 |
|---|---|---|
| manifest.json | 7,8 KB | 8d0649ecd429553643f307279369095c76d33733632c9378405296c6743e2595 |
| results.jsonl | 9,8 KB | e6eb5c97e7b174b60eed65504935921cc945a64670801e8d852bf1b859fc7b97 |
| summary.json | 2,3 KB | 617cd821b2ed7b7110e931ca3b9dc2a478925ec3184f3dbb004d6dd6170d9a92 |
| summary.md | 561 B | 59652c26c8c68d09e7b531eaad24740d1b15c614835f88823209b9f0415d5972 |
| README.md | 3,5 KB | 0b4ba09f748de44b4adbe018452821c6636955d57fd06a9d4f73568543bca284 |
| verify_bundle.py | 34,9 KB | 99e037f6a241e077348d7e116b03be17e727c4b61d278ea5f39b06dd557069a3 |
| SHA256SUMS | 14,8 KB | 6a39f82561f2ada8282857aa420eae3cdbf18dc51f01fa7aeb4ac0f608275fdb |
Todos os arquivos do pacote (111)
| Arquivo | Tamanho | sha256 |
|---|---|---|
| README.md | 3,5 KB | 0b4ba09f748de44b4adbe018452821c6636955d57fd06a9d4f73568543bca284 |
| attachments/03d0f2f60200ca106600cb11e693a4cbef524f115cc030b2b9d189f11235d18b.json | 1,0 KB | 03d0f2f60200ca106600cb11e693a4cbef524f115cc030b2b9d189f11235d18b |
| attachments/0d04a72b4b8126a307563446d1223f570f5ab71d48a945a856aa6d38820c8622.json | 218 B | 0d04a72b4b8126a307563446d1223f570f5ab71d48a945a856aa6d38820c8622 |
| attachments/0f1db6dff9663dda43fca0738bcb4cfb7c88060b193e21959c8bc7d9815db6c6.json | 1,9 KB | 0f1db6dff9663dda43fca0738bcb4cfb7c88060b193e21959c8bc7d9815db6c6 |
| attachments/0fe9d052fee4308f0d074becf0dcb6be86bf06de43dd2dadfdb5f7fa9ac3261f.json | 536 B | 0fe9d052fee4308f0d074becf0dcb6be86bf06de43dd2dadfdb5f7fa9ac3261f |
| attachments/13eeb5e377d3bbeea3b897a6d6aafd71ef345d0a566dd1324e786f548a0c35be.json | 4,6 KB | 13eeb5e377d3bbeea3b897a6d6aafd71ef345d0a566dd1324e786f548a0c35be |
| attachments/14b4ccaf759e6c6bc60147dbe7d08e7a33a234f4b5756c19bb81264624adea63.json | 985 B | 14b4ccaf759e6c6bc60147dbe7d08e7a33a234f4b5756c19bb81264624adea63 |
| attachments/1a0cd6d4a348eaa67d74d50d87fe07fac9917bd0d4be7a9a68d221ffb1ed14a2.json | 3,4 KB | 1a0cd6d4a348eaa67d74d50d87fe07fac9917bd0d4be7a9a68d221ffb1ed14a2 |
| attachments/1bf0adeb44dff438bcc894bf525fc3ff6777f2adf25cfa083e38c773dd505b35.json | 417 B | 1bf0adeb44dff438bcc894bf525fc3ff6777f2adf25cfa083e38c773dd505b35 |
| attachments/1cb75893680e71ff3b4e0f64b16cd1b299c136e9fb027af74c89f242731dedf1.json | 535 B | 1cb75893680e71ff3b4e0f64b16cd1b299c136e9fb027af74c89f242731dedf1 |
| attachments/1ef5806c73940ec856e166867eb21765b18851963d572ef22de22d079cdfb639.json | 534 B | 1ef5806c73940ec856e166867eb21765b18851963d572ef22de22d079cdfb639 |
| attachments/265d63d091732725cd5b15f51e1686b68b6b276ae88794f6120dc818830a2a97.json | 5,6 KB | 265d63d091732725cd5b15f51e1686b68b6b276ae88794f6120dc818830a2a97 |
| attachments/2a55f6734a1749cd2514801e80d4cc1b1bf87360edbef398fbbfa9c89a0ecb94.json | 225 B | 2a55f6734a1749cd2514801e80d4cc1b1bf87360edbef398fbbfa9c89a0ecb94 |
| attachments/2c2eb1c7ff1f5fcbf722255ee4833715ae21deff7422f37b9296933b457ec400.json | 3,3 KB | 2c2eb1c7ff1f5fcbf722255ee4833715ae21deff7422f37b9296933b457ec400 |
| attachments/2df3a80ff9ddde9ec3e5ad2094e6781609b8f7589da7ddfbff4833a6fd54025d.json | 1,0 KB | 2df3a80ff9ddde9ec3e5ad2094e6781609b8f7589da7ddfbff4833a6fd54025d |
| attachments/30580ce3eae5cdd5ebe6bc3c43f4054f5e0052fe76c1119a12a228c79df46c3f.json | 204 B | 30580ce3eae5cdd5ebe6bc3c43f4054f5e0052fe76c1119a12a228c79df46c3f |
| attachments/335958e982bf11c4aeab8a299ad4620f10911b0d4e5fcfee574c2ddfc63588f1.json | 2,1 KB | 335958e982bf11c4aeab8a299ad4620f10911b0d4e5fcfee574c2ddfc63588f1 |
| attachments/336b007e17fc58b64963f132978431ca14138647a472bc3cf7cf2ceed3289ba6.json | 354 B | 336b007e17fc58b64963f132978431ca14138647a472bc3cf7cf2ceed3289ba6 |
| attachments/346ca60e6f508bd31d5ae22387be0c4f19506ca866682fa7845115053d4fb698.json | 874 B | 346ca60e6f508bd31d5ae22387be0c4f19506ca866682fa7845115053d4fb698 |
| attachments/364c3b8bc51a2de51af1431531d327af2adb3aec59c5335a583cc0e9ccb8a8e2.json | 3,0 KB | 364c3b8bc51a2de51af1431531d327af2adb3aec59c5335a583cc0e9ccb8a8e2 |
| attachments/368bcd706182a24c06d2619510a652a362c3704c69a65419426f7ec14cae34a9.json | 873 B | 368bcd706182a24c06d2619510a652a362c3704c69a65419426f7ec14cae34a9 |
| attachments/376d30a96ffec04dcf2c30b7496886a63b14049208ac095697d4eb3c51907674.json | 1,1 KB | 376d30a96ffec04dcf2c30b7496886a63b14049208ac095697d4eb3c51907674 |
| attachments/3876e182a4543b4499295b5aecd364b6e55373cff22db76e72047ac6bb5848e2.json | 366 B | 3876e182a4543b4499295b5aecd364b6e55373cff22db76e72047ac6bb5848e2 |
| attachments/3a51dc0178cf54528b1efef514ab8cda8f6f8de75969cae05da9e73d76e0e5ca.json | 629 B | 3a51dc0178cf54528b1efef514ab8cda8f6f8de75969cae05da9e73d76e0e5ca |
| attachments/3ad271d5ffced4d88a5c62c9c75c1ea6fd7988217927e641cd5ad0d2417d9abd.json | 1,7 KB | 3ad271d5ffced4d88a5c62c9c75c1ea6fd7988217927e641cd5ad0d2417d9abd |
| attachments/3c11200f5608099f5b7c120fb6d7bd8ef96ec3daeabfcbf033eaf22bb54234bb.json | 1,9 KB | 3c11200f5608099f5b7c120fb6d7bd8ef96ec3daeabfcbf033eaf22bb54234bb |
| attachments/3ca2098d554d54bc4742de3fac2760dd1f6e6c8e73cd58e81ea753c36fc3ced0.json | 3,5 KB | 3ca2098d554d54bc4742de3fac2760dd1f6e6c8e73cd58e81ea753c36fc3ced0 |
| attachments/4476caddd78a07893b06eab9c9c4357ef2a14cafe84e61447834a4744fafe5d2.json | 1,9 KB | 4476caddd78a07893b06eab9c9c4357ef2a14cafe84e61447834a4744fafe5d2 |
| attachments/45cedf8901cc0d51859e7722e511a6b3a48771263b334576b23e6d96452a3b9a.json | 284 B | 45cedf8901cc0d51859e7722e511a6b3a48771263b334576b23e6d96452a3b9a |
| attachments/468015dfabe7d9ecc13d6d7194cf9503f2496d46012f04140da16c4cde04f766.json | 955 B | 468015dfabe7d9ecc13d6d7194cf9503f2496d46012f04140da16c4cde04f766 |
| attachments/4f74365ba282c87af902afa07c31b2cb0ec4a76b3d7c77c9c889b096ef0a4434.json | 537 B | 4f74365ba282c87af902afa07c31b2cb0ec4a76b3d7c77c9c889b096ef0a4434 |
| attachments/5506f110e53a4923624d5cbe332d3221add3f25799cecf7013ab8d71aa39b05d.json | 1,0 KB | 5506f110e53a4923624d5cbe332d3221add3f25799cecf7013ab8d71aa39b05d |
| attachments/567babcda3a6a40b9a61d139e7d1cc5da6bea50260cb766c4081fafbb29c475b.json | 4,7 KB | 567babcda3a6a40b9a61d139e7d1cc5da6bea50260cb766c4081fafbb29c475b |
| attachments/57fa2ae41ea98b7c41a73bea0074c26b197b0989acf134d9c28b1732539530e9.json | 982 B | 57fa2ae41ea98b7c41a73bea0074c26b197b0989acf134d9c28b1732539530e9 |
| attachments/5c6b1bb5ad36ee8ad9fcfcfc96662254c4fdb91be4be04f360127586f7b5e1de.json | 8,6 KB | 5c6b1bb5ad36ee8ad9fcfcfc96662254c4fdb91be4be04f360127586f7b5e1de |
| attachments/61bf6a32e622255eaea8d14036a2cfa8e7ea18a5e1ff4de14a30d9addb69f81d.json | 5,2 KB | 61bf6a32e622255eaea8d14036a2cfa8e7ea18a5e1ff4de14a30d9addb69f81d |
| attachments/62ec6c6f49660031fc6e91548fb29f6c96993881cdb520b2f2e041ca21e6a16f.json | 3,9 KB | 62ec6c6f49660031fc6e91548fb29f6c96993881cdb520b2f2e041ca21e6a16f |
| attachments/690febb0c1a11a54584c0a8eca4ec6d3818c876c62c356923612fe80cfa8cc3d.json | 219 B | 690febb0c1a11a54584c0a8eca4ec6d3818c876c62c356923612fe80cfa8cc3d |
| attachments/69f0520f44176b7180d15e4be6f169345d0d55cf31ce22fa189ac65aba752fab.json | 2,8 KB | 69f0520f44176b7180d15e4be6f169345d0d55cf31ce22fa189ac65aba752fab |
| attachments/6a519f0ff29e918c96f0d02c209d18b676a345ae2df600f5fd398234deb585e1.json | 5,4 KB | 6a519f0ff29e918c96f0d02c209d18b676a345ae2df600f5fd398234deb585e1 |
| attachments/6ea7eba86027dd9ae2356ba1652da64b39d5c2b4463839ac47a44faa58f1b69d.json | 983 B | 6ea7eba86027dd9ae2356ba1652da64b39d5c2b4463839ac47a44faa58f1b69d |
| attachments/7376aa5879318887af0aea5b0a271a2f796e1c279e8e9d74bf3802ad4bc5a7d4.json | 1,6 KB | 7376aa5879318887af0aea5b0a271a2f796e1c279e8e9d74bf3802ad4bc5a7d4 |
| attachments/7458c98e5e44912cacb67133b9db4f55411214a4405903000a95d1bb50937483.json | 12,5 KB | 7458c98e5e44912cacb67133b9db4f55411214a4405903000a95d1bb50937483 |
| attachments/7b78ccc98a81195006e0839c91f5805e77aa5ab04e151edfd1efa8b52fcf5cc5.json | 8,5 KB | 7b78ccc98a81195006e0839c91f5805e77aa5ab04e151edfd1efa8b52fcf5cc5 |
| attachments/7c90124f10cc5169d39c3748bb025bcaf95b8204275c942048821b948007986f.json | 1,8 KB | 7c90124f10cc5169d39c3748bb025bcaf95b8204275c942048821b948007986f |
| attachments/8040faa4f7068d3523425cedf399778b9c121685c483c114ef8276ae5fca52ca.json | 307 B | 8040faa4f7068d3523425cedf399778b9c121685c483c114ef8276ae5fca52ca |
| attachments/86ff9a038fae4197462dd59d58f7a565dd9bb75ee651e10fd077f2205229846b.json | 1,8 KB | 86ff9a038fae4197462dd59d58f7a565dd9bb75ee651e10fd077f2205229846b |
| attachments/8bd15faa5f579989543615336a441918c42313c0e11a75d7d8e5de522d330450.json | 1,3 KB | 8bd15faa5f579989543615336a441918c42313c0e11a75d7d8e5de522d330450 |
| attachments/8e1d39e4c000651d087d4d14f9640f95d0fe91e52f45d69f8fb5590cf3640dc4.json | 5,0 KB | 8e1d39e4c000651d087d4d14f9640f95d0fe91e52f45d69f8fb5590cf3640dc4 |
| attachments/8eddadcc05d9195ef99a87c8f161e6d536b8c2e1b0c2dd5cf45743c7c3edf4b8.json | 2,5 KB | 8eddadcc05d9195ef99a87c8f161e6d536b8c2e1b0c2dd5cf45743c7c3edf4b8 |
| attachments/8ee0d84c29b117269608a8edad648afadcb20e12f00a5d88b7d64223f4b057e2.json | 258 B | 8ee0d84c29b117269608a8edad648afadcb20e12f00a5d88b7d64223f4b057e2 |
| attachments/8fd4c6ca49793f12d94f301f3ee452bfc540e1ef4c3bb74725b746568f4d1243.json | 514 B | 8fd4c6ca49793f12d94f301f3ee452bfc540e1ef4c3bb74725b746568f4d1243 |
| attachments/9446300a18b4ac655a3d3db22141ae70c871af78228a1745bfa818d6c663df4d.json | 12,9 KB | 9446300a18b4ac655a3d3db22141ae70c871af78228a1745bfa818d6c663df4d |
| attachments/94a3ac6133dac6041005521a38f64d42ab7364aba24fdf438fe4cfb3afb8eb5b.json | 847 B | 94a3ac6133dac6041005521a38f64d42ab7364aba24fdf438fe4cfb3afb8eb5b |
| attachments/9896e0b7975cbcd1612e572d0e62e0f0bb4e60c0d16641de0ef6c8955debc0f8.json | 1,6 KB | 9896e0b7975cbcd1612e572d0e62e0f0bb4e60c0d16641de0ef6c8955debc0f8 |
| attachments/9919f55ba9ad7258609b3c3f3a7e3a5d2b8fcb75a46bca5eba036d4567927678.json | 1,2 KB | 9919f55ba9ad7258609b3c3f3a7e3a5d2b8fcb75a46bca5eba036d4567927678 |
| attachments/9b2f869c3ecfc3f66285806b61f33b7bf70ca17fc39f685899d19c8025b2f0c1.json | 234 B | 9b2f869c3ecfc3f66285806b61f33b7bf70ca17fc39f685899d19c8025b2f0c1 |
| attachments/9da387b01dc076b46a3710545972ad759a80ce44dfb2dd2116090dad0838149d.json | 1,4 KB | 9da387b01dc076b46a3710545972ad759a80ce44dfb2dd2116090dad0838149d |
| attachments/9f86ba889970a963f482118a1fb5610fff036653d56e3c3cf49af2488241b913.json | 1,9 KB | 9f86ba889970a963f482118a1fb5610fff036653d56e3c3cf49af2488241b913 |
| attachments/a156b5c18f2ed3efd7109c0ac7e107c89db0f5be9892860d678823b7d5e89ab9.json | 320 B | a156b5c18f2ed3efd7109c0ac7e107c89db0f5be9892860d678823b7d5e89ab9 |
| attachments/a447309682643b0e520a6326b4f9eed4dccfc7a418e46d2938098a206a3d1831.json | 391 B | a447309682643b0e520a6326b4f9eed4dccfc7a418e46d2938098a206a3d1831 |
| attachments/a44ef51f03df5188f084f1e0227b68df116b942a22fd50c0a88670cf3461fedd.json | 65,0 KB | a44ef51f03df5188f084f1e0227b68df116b942a22fd50c0a88670cf3461fedd |
| attachments/b0eb4491f05d4c29ae3fd601b6d81c2c27810b9349b36ca694362487bb0b3aa7.json | 956 B | b0eb4491f05d4c29ae3fd601b6d81c2c27810b9349b36ca694362487bb0b3aa7 |
| attachments/b591c7ac1308b8497c91d066899e7ff407c4014aaa517400a77cf3be9dbc5be4.json | 984 B | b591c7ac1308b8497c91d066899e7ff407c4014aaa517400a77cf3be9dbc5be4 |
| attachments/b6ea9a174329a1951ea78903f6b9158783f4b9c8f919cde9592419829a025e5a.json | 210 B | b6ea9a174329a1951ea78903f6b9158783f4b9c8f919cde9592419829a025e5a |
| attachments/b84f9be965f8530a4b14e3acab948236c4de0d065b43fcc26229e0754d3cc11e.json | 1,4 KB | b84f9be965f8530a4b14e3acab948236c4de0d065b43fcc26229e0754d3cc11e |
| attachments/b922a0322ae4de9ff173ae3b97da829841ebfc1ee45b9ed712c03d583e708f8b.json | 1,9 KB | b922a0322ae4de9ff173ae3b97da829841ebfc1ee45b9ed712c03d583e708f8b |
| attachments/be8e87ca1ed97a53c2f5cec2ed1ffd09bebd8db3977f24795b016d60a6c3aae3.json | 1,5 KB | be8e87ca1ed97a53c2f5cec2ed1ffd09bebd8db3977f24795b016d60a6c3aae3 |
| attachments/c3fe1498574540a86188465dd56571a62f2835c12812a7897c9cd1fa97477415.json | 958 B | c3fe1498574540a86188465dd56571a62f2835c12812a7897c9cd1fa97477415 |
| attachments/c4d3c64c3706c224076311e4d1a294e8daa6034671095309bbb3f2747dbecce7.json | 4,3 KB | c4d3c64c3706c224076311e4d1a294e8daa6034671095309bbb3f2747dbecce7 |
| attachments/c521ecdddd1c246c4f1aad416d2a8e364009824ab3bc68ca2b6e13bfc8c38dd9.json | 195 B | c521ecdddd1c246c4f1aad416d2a8e364009824ab3bc68ca2b6e13bfc8c38dd9 |
| attachments/c93b873ca2f30766cab412d7b389ca8fbbfb5ad9e1834439c12a42f09ab83c7c.json | 2,9 KB | c93b873ca2f30766cab412d7b389ca8fbbfb5ad9e1834439c12a42f09ab83c7c |
| attachments/cfd2f6f119c08120a0fe76db376f7ac23a7d5e69d99c8f820f9e4087c9d19009.json | 313 B | cfd2f6f119c08120a0fe76db376f7ac23a7d5e69d99c8f820f9e4087c9d19009 |
| attachments/d61c49f720a9e184ae04a391e1a33ce4cb2c7e34a426c09e445cdf352f10413e.json | 324 B | d61c49f720a9e184ae04a391e1a33ce4cb2c7e34a426c09e445cdf352f10413e |
| attachments/d76883658ece391fcdec6a852a57cf7b84da46cfe7c33bd834fcc9aa7bc68b33.json | 1,2 KB | d76883658ece391fcdec6a852a57cf7b84da46cfe7c33bd834fcc9aa7bc68b33 |
| attachments/d879978b88c12894fb0ba800800d3125b6ebc55ea892aaeb51b2677b7837b86e.json | 8,2 KB | d879978b88c12894fb0ba800800d3125b6ebc55ea892aaeb51b2677b7837b86e |
| attachments/dccf7d0b6b9111bc2e068ce37bd0c338a395929e4c7fa25ed30037a5f6673a89.json | 944 B | dccf7d0b6b9111bc2e068ce37bd0c338a395929e4c7fa25ed30037a5f6673a89 |
| attachments/de5969c76942b8a1a3c69f581d587cd09c136268bdf07c785afa3324e654178f.json | 389 B | de5969c76942b8a1a3c69f581d587cd09c136268bdf07c785afa3324e654178f |
| attachments/dec55429c9e9040876785b458a49c1938ebf98118504f939d3e2a548e4c08449.json | 1,4 KB | dec55429c9e9040876785b458a49c1938ebf98118504f939d3e2a548e4c08449 |
| attachments/e6432555bb3ad5958fd75d68484f8a8eebc6a2a14d04654ea760c03269b60777.json | 2,0 KB | e6432555bb3ad5958fd75d68484f8a8eebc6a2a14d04654ea760c03269b60777 |
| attachments/e8778e012318ec19b88cd0adbb9a1f3668209cb4737d3b390c35fd7c042d281a.json | 325 B | e8778e012318ec19b88cd0adbb9a1f3668209cb4737d3b390c35fd7c042d281a |
| attachments/e8ad388a5b675febf56168b4ea5365f9d93d34a834ebcfce523302f16f5e7c04.json | 2,4 KB | e8ad388a5b675febf56168b4ea5365f9d93d34a834ebcfce523302f16f5e7c04 |
| attachments/ec0e651327ae76f30ee468853f4544d39cf09d5c6ec07596302ce1c563772025.json | 281 B | ec0e651327ae76f30ee468853f4544d39cf09d5c6ec07596302ce1c563772025 |
| attachments/ece9b7d490b92466ecea7e2ae9dad33dd2e5dfdde6ad878dc0caa31f7ffd317a.json | 255 B | ece9b7d490b92466ecea7e2ae9dad33dd2e5dfdde6ad878dc0caa31f7ffd317a |
| attachments/f0f960f60b0c581965382a5d9b5e0eab73c9779c262d584912845eb694ccb9c0.json | 418 B | f0f960f60b0c581965382a5d9b5e0eab73c9779c262d584912845eb694ccb9c0 |
| attachments/f125498df7f5fe9a96dd4b77f4cea4ad6ae138e23daac76bd19ee8beb5aa3833.json | 1,5 KB | f125498df7f5fe9a96dd4b77f4cea4ad6ae138e23daac76bd19ee8beb5aa3833 |
| attachments/f6d03bda73f4c13a6c0a9fa17c46ad66eafbbc581acf492c65cbba4448cf830f.json | 627 B | f6d03bda73f4c13a6c0a9fa17c46ad66eafbbc581acf492c65cbba4448cf830f |
| attachments/f9b0b76791982379bf477886bfc56a2b30042c4046309734aef4bbbf73f37671.json | 5,0 KB | f9b0b76791982379bf477886bfc56a2b30042c4046309734aef4bbbf73f37671 |
| attachments/fa402121ecab5f012d4ad920eecfbade6b8914ee47c7388a018cb8723a527c9e.json | 227 B | fa402121ecab5f012d4ad920eecfbade6b8914ee47c7388a018cb8723a527c9e |
| env/platform.json | 2,5 KB | 80dde493643d39c7b31ad719b0cfc5a1a8749b502d8366c272b7f931f994b071 |
| manifest.json | 7,8 KB | 8d0649ecd429553643f307279369095c76d33733632c9378405296c6743e2595 |
| native/qrdata_socsci/label_map.json | 449 B | b53b9b1247727516d2956652ef2d4a0e5f3c14a090c51218b879b37331ca8f4b |
| native/qrdata_socsci/qrdata_socsci.352.e1.a1.json | 468 B | 2a8748ccbccd9ca1424db26f0d29f24764855d3e3a62501c57db6ff1a5455ee3 |
| native/qrdata_socsci/qrdata_socsci.353.e1.a1.json | 467 B | c11176cc19b52e11ff6bce2cf054d1dc3a7f24714cd748c4e1027deb75e90f24 |
| results.jsonl | 9,8 KB | e6eb5c97e7b174b60eed65504935921cc945a64670801e8d852bf1b859fc7b97 |
| sources/qrdata.eval_py.json | 1,3 KB | d610b61f390d29ce767e08ce850eab5cb0dc4d13c1edb62d8cf3b0e49a3a291a |
| sources/qrdata.gold.352.json | 1,1 KB | 876b93f09443c9b653a5a237b17199f8b9fbf0bf7e0f74e588c49e6733df505d |
| sources/qrdata.gold.353.json | 1,2 KB | ca22c1bfad85cb638909d7376cf0a562b0e4759228f8411844deda9e06ca151a |
| sources/qrdata.leaderboard.gemini_pro_react.json | 1,1 KB | 1e738daada9330ce57b6824bf4ea6fb6a11b1524111d9ade9201d76f9391caf9 |
| sources/qrdata.leaderboard.gpt4_code_interpreter.json | 1,2 KB | 8fea4e89162d8351934cd96a4d4d3f5543accb28e178e7b28b2c71e26230803c |
| specs/qrdata_socsci.352.json | 3,9 KB | d08fe16e6851ae6ab50fe1d8a290a5183d0b3b7736c0924f5112f2083b9048ca |
| specs/qrdata_socsci.353.json | 4,9 KB | 0cc17eda904685ec6480fbd63d06099cb36d562efb2439aefcbadf106d3127a8 |
| summary.json | 2,3 KB | 617cd821b2ed7b7110e931ca3b9dc2a478925ec3184f3dbb004d6dd6170d9a92 |
| summary.md | 561 B | 59652c26c8c68d09e7b531eaad24740d1b15c614835f88823209b9f0415d5972 |
| tools/registry.json | 14,3 KB | 644bdc32c1ef8e99eea9ba91f16d8c477a4c57d26b17990e22e346b33aa067fe |
| traces/qrdata_socsci.352.e1.a1.jsonl | 42,0 KB | e3d520779508486f61a6c50070bab6dfe5bc4a00a3e36bb1a2609ecb839c2891 |
| traces/qrdata_socsci.353.e1.a1.jsonl | 86,0 KB | cf48db03b6f30da1a61554100e4d7231e8d63b9beb8290557307e045aa6ace14 |
| trajectories/qrdata_socsci.352.e1.a1.jsonl | 34,2 KB | 61ff927f333f19f2c71ca7c35028a4cf3c6008e9e3ede91d4aebbaabb47fe311 |
| trajectories/qrdata_socsci.353.e1.a1.jsonl | 76,6 KB | e3564bd0f252cdd52316ccf91f5667118bffd5866153afcd4a2e1f4279e156a5 |
| verify_bundle.py | 34,9 KB | 99e037f6a241e077348d7e116b03be17e727c4b61d278ea5f39b06dd557069a3 |
| SHA256SUMS | 14,8 KB | 6a39f82561f2ada8282857aa420eae3cdbf18dc51f01fa7aeb4ac0f608275fdb |
Eventos do registro
- 20261015T124204Z-suite-qrdata_socsci-gemini-9c7e4b.started.json
e5ba5c4d7948de7815710dca7d7ee78ef71807fe87a96f9c062e334158755ad0 - 20261015T124204Z-suite-qrdata_socsci-gemini-9c7e4b.final.json
2f5c85c0dfe4a05c1388c9244b9a2839af7831daa14c7a8b880a5f041e380a14
Verifique você mesmo
- Resumo da execução
6a39f82561f2ada8282857aa420eae3cdbf18dc51f01fa7aeb4ac0f608275fdbsha256 do arquivo SHA256SUMS; deve ser igual ao run_digest do evento final do registro.
Qualquer pessoa, sem o repositório
- Baixe o pacote inteiro e descompacte.
- Rode o verificador, que só usa a biblioteca padrão do Python:
cd 20261015T124204Z-suite-qrdata_socsci-gemini-9c7e4b python3 verify_bundle.py
Ele confere o hash de cada arquivo, o nome de cada anexo contra seus bytes, a cadeia do registro e o summary.json recalculado a partir de results.jsonl.
Para conferir o resumo:
shasum -a 256 SHA256SUMS # 6a39f82561f2ada8282857aa420eae3cdbf18dc51f01fa7aeb4ac0f608275fdb
Com acesso ao repositório (privado)
git checkout 38e979372a0f7382f285079e4380813081bdd032 uv sync --frozen --all-packages uv run magi-bench verify bench-runs/20261015T124204Z-suite-qrdata_socsci-gemini-9c7e4b
magi-bench verify reconstrói cada tarefa a partir do código, confere os dados, refaz as chamadas de ferramentas, repontua e recalcula o resumo. Sai com 0 quando tudo confere, 2 quando só faltam itens não reexecutáveis listados, 1 em qualquer divergência.