Kimi K3
Led the core benchmark and bonus category with valid answers for all 20 artifacts.
Executive summary
Meaningful progress, with a clear reconstruction gap.
Leading agents can orient themselves inside unfamiliar binaries and identify broad Windows behavior. Useful performance still depends on exact recovery, evidence discipline, tool planning, refusals, and completing the task within a fixed budget.
6.47% decline
Removing symbols caused a moderate aggregate reduction across complete stripped and unstripped pairs.
Recognition beat reconstruction
Broad techniques were easier to identify than keys, configuration, infrastructure, and complete API chains.
25 runs produced no valid answer
Refusals and tool-budget exhaustion reduced useful coverage independently of analytical knowledge.
Leaderboard
See how each model performed.
The solid bar shows average performance on the 18 core artifacts. The shaded extension shows results from the two synthetic-worm artifacts, scored separately as a bonus. Total adds the two figures together; it is not an average across all 20 artifacts.
95 / 120 valid answers
Terminal refusals and tool-budget exits remain part of the observed result.
2 / 6 models
Only Kimi and Gemini returned valid answers for all 20 artifacts.
0 true C2 endpoints
No model recovered the actual Level 23 command-and-control endpoint.
Show the complete leaderboard
| Rank | Model | Main /1 | Bonus /1 | Valid | Terminal outcome | Total /2 |
|---|---|---|---|---|---|---|
| 01 | Kimi K3 | 0.6757 | 0.6861 | 20/20 | None | 1.3618 |
| 02 | Claude Opus 5 | 0.6147 | 0.6638 | 18/20 | 2 refusals | 1.2785 |
| 03 | GPT-5.6 Sol | 0.5859 | 0.6298 | 18/20 | 2 budget exits | 1.2157 |
| 04 | DeepSeek V4 Flash | 0.5125 | 0.6782 | 19/20 | 1 budget exit | 1.1906 |
| 05 | Gemini 3.6 Flash | 0.5400 | 0.5653 | 20/20 | None | 1.1052 |
| 06 | Claude Fable 5 | 0.0000 | 0.0000 | 0/20 | 20 refusals | 0.0000 |
Effect of stripping
Removing symbols lowered scores by 6.47%—not to zero.
Across 46 model/level pairs with valid answers on both variants, mean score fell from 0.6428 unstripped to 0.6012 stripped. Static PE evidence still preserved much of the behavior models needed.
0 defined symbols
Every stripped artifact removed the symbol names available in its paired original.
75.67% smaller
Aggregate binary size fell sharply, along with the number of sections.
−0.0416 absolute
The average relative decline was 6.47%, so stripping made analysis harder without making the binaries opaque.
Show stripping results by model
| Model | Pairs | Unstripped | Stripped | Change |
|---|---|---|---|---|
| Kimi K3 | 10 | 0.7122 | 0.6413 | −0.0709 |
| Claude Opus 5 | 8 | 0.7313 | 0.6720 | −0.0593 |
| GPT-5.6 Sol | 9 | 0.6731 | 0.6387 | −0.0344 |
| DeepSeek V4 Flash | 9 | 0.5369 | 0.5525 | +0.0156 |
| Gemini 3.6 Flash | 10 | 0.5705 | 0.5145 | −0.0560 |
Show stripping results by task
| Level | Pairs | Unstripped | Stripped | Change |
|---|---|---|---|---|
| 14 · DLL Injection | 5 | 0.8617 | 0.7327 | −0.1290 |
| 15 · APC Injection | 4 | 0.6064 | 0.5757 | −0.0307 |
| 16 · Code Cave | 5 | 0.7672 | 0.7219 | −0.0453 |
| 17 · Process Hollowing | 5 | 0.4623 | 0.5096 | +0.0473 |
| 18 · Hell’s Gate | 5 | 0.5119 | 0.4649 | −0.0469 |
| 19 · Reflective DLL | 4 | 0.6119 | 0.4114 | −0.2004 |
| 20 · Remote PE | 5 | 0.6678 | 0.7054 | +0.0376 |
| 21 · Ghost Hollowing | 4 | 0.5484 | 0.6502 | +0.1018 |
| 22 · Advanced Evasion | 4 | 0.7082 | 0.5750 | −0.1332 |
| 23 · Synthetic Worm | 5 | 0.6628 | 0.6264 | −0.0364 |
These are descriptive differences from one run per model and artifact, not confidence-bounded causal estimates.
Model behavior profiles
Scores alone hide operationally important differences.
Coverage, precision, speed, refusal behavior, and budget discipline changed the practical value of each model’s run.
Kimi K3
Strongest overall and best on decoded strings and anti-analysis. Complete, but the slowest successful run at 9.74 summed episode-hours.
Claude Opus 5
Best injection-detail mean and strong technique recall. Two mid-analysis refusals reduced useful coverage.
GPT-5.6 Sol
Best single score with no unsupported technique claims. Both Level 22 variants ended at the tool-call limit.
DeepSeek V4 Flash
Strong bonus and encryption results, with greater variance, one budget exit, and the most unsupported claims.
Gemini 3.6 Flash
Fastest complete run and strong metadata recovery. Exact decoded-string and technique detail were weaker.
Policy coverage outcome: Claude Fable 5 refused all 20 artifacts before using a tool. That result is operationally relevant, but it is not a clean measure of reverse-engineering knowledge.
Show the failure taxonomy
Broad identification
The technique family was recognized, but required implementation details were missing.
Unsupported claims
APIs, techniques, configuration values, or execution steps were reported without recovered evidence.
Incomplete chains
Individual components were found but not connected into the correct execution sequence.
Budget exhaustion
Low-value exploration consumed the fixed call budget before a complete answer was produced.
Premature stopping
An early hypothesis was reported before critical details were verified.
Refusal
A valid defensive analysis task ended without a gradable answer.
Capability boundary
Useful for orientation. Less reliable for exact reconstruction.
Models were better at identifying the shape of a program than recovering the details required for a final incident conclusion.
What agents often recovered
- Broad injection and execution techniques
- Common Windows APIs, imports, strings, and constants
- Initial behavioral hypotheses and recognizable persistence or evasion patterns
- Straightforward control-flow relationships
What remained brittle
- Complete multi-stage execution chains
- Exact decoded configuration, keys, and transformations
- Indirect or dynamically resolved APIs
- Distinguishing active logic from decoys without unsupported claims
Recommended use: analyst-assisted triage, prioritization, and hypothesis generation. No model recovered the true Level 23 C2 endpoint, and unreviewed output should not be treated as authoritative configuration recovery.
Scope and limitations
What Version 3 does—and does not—establish.
Command-line static analysis
Agents worked from isolated binaries with fixed tools and budgets. The benchmark does not measure dynamic analysis, debugging, or interactive decompiler workflows.
No repeated-run variance
Each model-artifact pair was evaluated once, so results characterize the recorded runs rather than a stable population mean.
Synthetic known-ground-truth artifacts
Deterministic grading is possible because behavior is known, but the suite does not reproduce the full diversity of real-world malware.
Method and verification
Every published score is traceable to evidence.
Each result can be followed from input artifact to agent transcript, tool output, final answer, deterministic grader decision, and aggregate score.
Static, binary-only analysis
One anonymized PE per isolated workspace. No source, Internet, debugger, decompiler, compilation, or dynamic execution.
Deterministic ground truth
No LLM judge. Terminal refusals and budget exits remain in the aggregates. Main is the broad score; Level 23 supplies the separate Bonus.
Fully reproducible
All 120 answers were rescored, all six aggregates reconciled, and the 366-file checksum ledger and artifact hashes passed verification.
Independent check: run python3 version3/verify.py from the repository root. It requires no API key or model call.
What Version 3 means
Frontier agents are becoming capable binary analysts—not yet dependable reverse engineers.
Current agents can accelerate orientation and hypothesis generation. Exact reconstruction still requires stronger tool planning, longer-horizon evidence tracking, better calibration, efficient budget use, and expert verification.
Use agents as accelerators
Apply them to triage, prioritization, and evidence discovery while retaining analyst review for configuration, attribution, and consequential conclusions.
Reward verifiable recovery
Optimize for evidence-grounded technical facts, efficient tool paths, uncertainty reporting, and completion reliability—not broader descriptions alone.