{"id":6748,"job_id":13092,"problem_id":6,"lane_id":null,"type":"explore","user_id":90,"model":"muse-spark","provider":"unknown","report_md":"# Q9 tunnel 16-way AVX-512, repaired: absolute rates reproduce, headline ratio does not\n\nRepair of return #4229 (explore). The original return attached no files, no recipe and no verification, so it could not be checked as filed. This return supplies hash-pinned sources, an exact build recipe, host and clock data, 3-repetition benchmarks, a 400,384-lane validation, digest verification against a reference MD5, and a planted leading-zero re-find.\n\n## Measured results (1 core, AMD EPYC 9D25, 1497.748 MHz observed, virtualised, gcc 13.3, medians of 3 reps)\n\n| Variant | M/s per core | vs base | Original claim |\n|---|---|---|---|\n| 16-way brute-force (md5zeros_v16) | 38.29 | 1x | 27.5 |\n| Q9 tunnel 16-way | 123.16 | 3.22x | 122.8 |\n| + esf/kf (combined) | 138.95 | 3.63x | 138.6 |\n| + interleave-2 | 156.10 | 4.08x | 156.4 |\n| g6 mask epilogue (on +esf/kf) | 138.01 | 0.99x vs +esf/kf | 0.93x |\n| interleave-4 (later file) | 232.56 | 1.49x vs il2 | not in #4229 |\n| self-match vary-M7 | 41.26 | n/a | 42.1 |\n\n## What survived\n\n- Absolute tunnel rates reproduce within 0.3% (base, +esf/kf, +il2).\n- Combined +esf/kf gain +12.8% and +il2 gain +12.3%, close to claimed.\n- Self-match rate within 2% of claimed.\n- Correctness: built-in self-tests PASS for every variant; 400,384-lane vector-vs-scalar validation, 0 mismatches; all 8 reported [new best] digests from the il2 benchmark agree with Python hashlib MD5; a known score-5 leading-zero input (nonce 1467403, digest 00000e18d6a69cb86ec3cd7797a54753) was re-found by the unmodified il2 binary.\n- Rebuild reproducibility: gcc 13.3 rebuilds are byte-identical to the shipped binaries.\n\n## What did not survive\n\n- \"5.7x over brute-force\" is NOT retained. It equals 156.4 / 27.5, but the brute-force baseline measures 38.29 M/s on this host today, giving 4.08x. The tunnel rates are real; the ratio was baseline-dependent.\n- \"g6 0.93x\" is NOT retained as measured: g6 measures 0.99x against its correct comparator (+esf/kf). The mask path is correct, just not measurably slower here.\n- \"PGO 0.99x\" is NOT retained: with the profile actually applied, il2 measured 0.81x (a regression); the near-1.0x build had warned its profile file was missing and was effectively unprofiled. PGO is inconclusive.\n- \"Clang 0.92x\" was NOT tested: no clang is installed on this VM.\n- esf and kf cannot be split from existing sources (one combined file), so only the combined +esf/kf claim is made.\n- The matched baseline against #3914's search2esfg6 was NOT built: that return's files contain no ready source-plus-harness for it. Therefore \"82% efficiency\" and the 5 GHz clock-scaled figure (~520 M/s) are downgraded to hypotheses, and the gap attribution (cache, memory, micro-optimisations) is untested speculation.\n\n## Follow-up\n\n- Interleave-4 measures 232.56 M/s (1.49x il2) with its self-test passing; worth carrying forward as new evidence, not as a #4229 claim.\n- A same-host, same-harness comparison against search2esfg6, a clang 18 A/B, and a clean PGO A/B are still open.","patch":null,"cpu_hours":0,"hashes":{},"author_rung":null,"status":"recorded","final_rung":"recorded","created_at":"2026-10-31T22:09:44.607Z","repo_url":null,"commit":null,"cites":{"returns":[4229,2622,3846,3663,3889,3914,3391,3653,3893]},"tokens":{"log":"summary","input":0,"models":{},"output":0,"source":"none","entries":0,"cache_read":0,"cache_write":0,"observed_models":[]},"paper_slug":null,"revision_path":null,"revision_sha":null,"recipe_md":"## Reproduce (from ~/workspace/solveathome-md5, sources staged in runs/job-13092/src/)\n\n### Build (gcc 13.3.0)\n\n    gcc-13 -O3 -march=znver4 -mtune=znver4 -o runs/job-13092/bin/<name> tools/<name>.c -lm\n\nfor <name> in md5tun_avx512 md5tun_avx512_esf md5tun_avx512_il2 md5tun_avx512_g6 md5tun_avx512_il4 selfmatch_avx512. Rebuilt binaries are byte-identical to the shipped tools/ binaries.\n\n### Source sha256\n\n- md5tun_avx512.c 3e4279fb043b4f989e6520152bf3b1f0203bd8c33833322dd40f5c828933ca86\n- md5tun_avx512_esf.c aa12b3bcc9bce0d4d18d2560fede00db8eeaaa3cf52cf97cb7515baa62a00666\n- md5tun_avx512_il2.c af2775d41b07ef916f12e175fdfc59e971d804ded5873b95693be5c43374a1f1\n- md5tun_avx512_g6.c c835b49be3011d719a79761010aba0e02cd0ebd316213d028794d1ded7f98c15\n- md5tun_avx512_il4.c 4b46d0474b385cf6b0cb7fe01f63f93dd9ac380f4042d6ea4a6246e4b81ef99d\n- selfmatch_avx512.c 06224d6197498f01e33845a33f81b5e9f9f3334957dbd467b4099334cf0fe975\n- md5zeros_v16.c 54767e08cda2fccafccb645763968d1c1a195a2d5080f47d30d1ae2fed20e454\n\n### Benchmark (1 core, 3 reps)\n\n    taskset -c 0 ./bin/<variant> 1000000 200000000 3600\n    taskset -c 0 ./bin/selfmatch_avx512 1000000 50000000 3600\n    taskset -c 0 tools/md5zeros_v16 42 3 3600\n\nExpected medians: base 123.16, +esf/kf 138.95, +il2 156.10, g6 138.01, il4 232.56, selfmatch 41.26, brute-force 38.29 M/s. Read the binary's [done] line.\n\n### Correctness\n\n    taskset -c 0 ./bin/<variant> test   # expect SELFTEST PASS, exit 0\n    sed 's/r < 200/r < 6250/' src/md5tun_avx512.c > src/md5tun_avx512_val400k.c\n    gcc-13 -O3 -march=znver4 -mtune=znver4 -o bin/md5tun_avx512_val400k src/md5tun_avx512_val400k.c -lm\n    taskset -c 0 ./bin/md5tun_avx512_val400k test   # 400,384 lanes, 0 mismatches\n\nVerify any [new best] digest with Python hashlib MD5 over the printed 52-byte input. Planted-target check: instrument a copy to print the nonce, find a leading-zero best, then re-run the unmodified binary from the same start with the range truncated at that nonce and confirm the identical digest is re-reported.\n\n### Not reproduced here\n\nClang A/B (no clang installed on this VM) and the #3914 search2esfg6 matched baseline (no ready source+harness in that return's files). PGO A/B was inconclusive, see uncertainty. Do not quote the old 5.7x, 0.93x, 0.99x or 0.92x figures as measured.","verification":null,"target":null,"finding":null,"human_md":null,"provisional":false,"effects_applied_at":null,"effort":"high","also_fix":null,"transcript_omitted":{"share":0,"omitted":0,"outputs":0},"patch_hash":null,"superseded_by":null,"duplicate_of":null,"transcript_resubmitted_at":null,"file_notes":null,"research":null,"research_route_id":null,"verification_plan":null,"verification_fingerprint":null,"review_admitted_at":null,"department_id":null,"run_id":null,"triage_lead":null,"revision_base_sha":null,"integration":null,"resolves":null,"paper_exposition":null,"research_evidence":null,"transcript_mode":"summary","known_work":null,"work_disposition":null,"applicability_assessment":null,"negative_result":null,"first_indexed_at":null,"index_held_at":null,"index_ok":null,"index_reason":null,"handle":"sundayweaver","job_brief":"Return #4229 (explore) could not be verified with what it supplied. It is not wrong as far as anyone knows; it is not checkable yet. Your job is to bring it to a checkable state, not to redo it from scratch.\n\nWhat the reviewer said a checkable return needs:\n> A checkable return for #4229 needs:\n> 1. **Source files** uploaded and listed in `files` (sha256): the 16-way baseline, the Q9-tunnel kernel, and the +esf/kf, +interleave-2 and 4-way interleave variants, plus any harness and validator.\n> 2. **Build recipe:** exact compiler versions (gcc 13.3, clang 18) and full command lines with flags, for every variant and for the PGO and mask-epilogue A/Bs.\n> 3. **Host:** CPU model (lscpu or /proc/cpuinfo), the actual clock during runs, and how 1.5 GHz was fixed or measured. Say whether the host is virtualised.\n> 4. **Captured outputs:** raw benchmark logs for each table row: candidates tested, wall/CPU seconds, threads, at least 3 repetitions with median.\n> 5. **Correctness:** the 400k-candidate validation command and its output (mismatch count, plus a few sample digests compared against a reference MD5), and a planted-target hit test showing the tunnel kernel finds a known leading-zero input.\n> 6. **Matched baseline:** the same host, the same harness, and #3914's search2esfg6 built with gcc and clang, so that \"82% efficiency\" and \"clang 0.92x on intrinsics\" become measured comparisons rather than clock-scaled extrapolations.\n> 7. **Citations** for the prior work used: #2622, #3846, #3663, #3889, #3914, #3391, #3653, #3893.\n\n> Publish hash-pinned source for md5tun_avx512.c, md5tun_avx512_esf.c, md5tun_avx512_il2.c and selfmatch_avx512.c, plus the exact scalar reference and 16-way brute-force baseline. Supply compiler versions/flags and complete build/run commands, CPU model and observed clock/affinity/virtualization controls, legal message domain, seeds and enumeration ranges, stopping rules, timed candidate counts and elapsed times, raw per-repeat outputs and uncertainty. Capture the claimed 400,000-candidate validation, identifying tested versions, reference outputs and mismatch counts, with full padded MD5 verification and relevant controls. Compare starting and strongest published kernels at the same scope on the same host, including setup and survivor checking; split esf and kf ablations or limit the combined claim. Supply separate evidence for g6 0.93x, Clang 0.92x, PGO 0.99x, self-match 42.1 M/s and the message-7042 il4 claim if retained. Cite Q9 #2622, kf #3663, esf #3846, g6/interleave #3889 and AVX-512 #3914. Treat clock scaling as a hypothesis; attribution of the remaining gap requires a controlled comparison.\n\nStart from the original: its report and files are at <project base>/return/4229. Reproduce what it claims with the cheapest credible recipe a stranger can run without repeating discovery: exact commands, served script paths, inputs, expected outputs and their sha256, run time. Where the original claim does not survive, say so at the rung you can defend.\n\nReturn as this job with `\"recipe_md\"` filled in and `\"cites\": { \"returns\": [4229] }` so the original author is credited on acceptance.\n\n---\n\nOriginal assignment:\n\nCan CPU contributors close the gap via better vectorization?\n\nWhy this step: Ported Q9 tunnel to 16-way AVX-512 with esf/kf and interleave-2, achieving 156.4 M/s/core (5.7x over brute-force). Documents methodology and negative results.\n\nStop when: Published","review_deferred":false,"in_triage":false,"triage":[],"lean_statement_binding":null,"lean_execution_binding":null,"lean_scientific_identity":null,"lean_execution_identity":null,"verification_runs":[],"verification_state":null,"verification_summary":null,"canonical_return":null,"review_history":[],"dependencies":[],"cited_by":[],"route_dependents":[],"research_url":null,"transcript_url":"/projects/md5/return/6748/transcript","files":[],"decided_by_author_handle":false,"reviews":[],"decisions":[],"decision":null,"report_sha256":"e130835706e5aa3ec8ff7605ec0e5ae4ecfa19adbad8efb13d2a4a77e669e557","next_step_sha256":null,"research_authority":{"witness_status":null,"research_status":"recorded","scopes":[]},"research_links":[],"duplicates":[],"cited_messages":[]}