v3.1Current
Five attempts per binary
Six models, twenty Windows PE binaries, and 600 attempts. Muse Wonder leads the averaged scores. The report shows what each model recovered, missed, and got wrong.
v3.1 · Windows PEOpen benchmark Version 3.1 · November 2026
AgentRE-Bench
Can AI agents reason about software without its source code?
AgentRE tests how models analyze binaries without source code, identifies where they fail, and turns those failures into training signals for better reverse engineering agents.
- 33
- distinct binary challenges
- Linux + Windows
- two binary ecosystems
- Deterministic
- replayable scoring
- Open
- methodology & transcripts
Latest results
Ten Windows programs, each evaluated with and without symbols. Six models, 20 artifacts, five attempts per artifact: 600 episodes.
Score guide
| # | Model | Overall | Main · 90% | Bonus · 10% | Valid answers |
|---|---|---|---|---|---|
| 01 | Muse Wonder | 87.37 | 87.37 | 87.37 | 100/100 |
| 02 | GPT-6 Astra | 87.04 | 86.40 | 92.82 | 100/100 |
| 03 | Grok 4.7 | 79.62 | 81.02 | 67.04 | 96/100 |
| 04 | DeepSeek V4.1 Flash | 78.31 | 78.43 | 77.25 | 98/100 |
| 05 | Claude Opus 5.5 | 69.66 | 68.84 | 77.03 | 78/100 |
| 06 | Gemini 4 Argon | 31.20 | 29.53 | 46.23 | 36/100 |
01Muse Wonder100/100 valid answers87.37Overall
- Main · 90%
- 87.37
- Bonus · 10%
- 87.37
- Overall 95% interval
- 85.18–89.56
- Thinking
- High
02GPT-6 Astra100/100 valid answers87.04Overall
- Main · 90%
- 86.40
- Bonus · 10%
- 92.82
- Overall 95% interval
- 85.80–88.29
- Thinking
- Extra high
03Grok 4.796/100 valid answers79.62Overall
- Main · 90%
- 81.02
- Bonus · 10%
- 67.04
- Overall 95% interval
- 76.59–82.65
- Thinking
- High
04DeepSeek V4.1 Flash98/100 valid answers78.31Overall
- Main · 90%
- 78.43
- Bonus · 10%
- 77.25
- Overall 95% interval
- 75.01–81.60
- Thinking
- Max
05Claude Opus 5.578/100 valid answers69.66Overall
- Main · 90%
- 68.84
- Bonus · 10%
- 77.03
- Overall 95% interval
- 62.57–76.74
- Thinking
- High
06Gemini 4 Argon36/100 valid answers31.20Overall
- Main · 90%
- 29.53
- Bonus · 10%
- 46.23
- Overall 95% interval
- 13.30–49.10
- Thinking
- High
Scores average five attempts per binary. Overall is 90% Main and 10% Bonus; unanswered attempts count as zero. Scoring and methodology ↗
Read the complete v3.1 reportLatest findings
- DeepSeek V4.1 Flash joins at 78.31/100. It submitted 98 valid answers; two budget exits remain zero-valued attempts.
- Astra was stronger on encryption. It received full algorithm and key credit in all 100 answers. Muse did so in 96.
- All 600 scores independently reproduced. A separate AI agent verified the versioned review. The previous five models’ scores are unchanged.
Why reverse engineering?
Most AI coding benchmarks begin with readable code and a clear task. Security teams often begin with neither. They get a compiled program, limited context, and one question: what does this actually do?
AgentRE measures whether an agent can turn that uncertainty into a defensible explanation. Agents recover behaviors, configurations, infrastructure, API chains, and execution details through tool use. Correct findings earn credit; unsupported claims are penalized against versioned ground truth.
Why AgentRE mattersHow it works
Agents receive an isolated binary and static-analysis tools. Recorded model configurations and the scoring version accompany each result.
Present a binary without source
A source-free, anonymized binary in an isolated workspace, with ordinary static-analysis tools and no internet access.
Record the investigation
Tool calls, evidence, time, coverage, failures, and final structured claims are captured for review.
Score what can be proved
Versioned ground truth rewards correct detail and penalizes unsupported claims. The same answer always receives the same score under the same scorer.
Benchmark history
V2
Calibration becomes the advantage
A broader model field ran the same Linux ladder. Gemini 3.1 Flash Lite led with a 0.5618 Main score and full 13/13 coverage. Fewer unsupported claims mattered more than deeper reasoning on this suite.
V2 · Linux ELFV1
Establishing the baseline
Thirteen synthetic Linux challenges established a source-free baseline with constrained tools and deterministic scoring. Completion was not the same as trust; every model scored zero on the bonus challenge.
V1 · Linux ELFV1 and V2 share the Linux suite. v3.1 uses the Windows PE suite and replaces the retired v3 leaderboard. Historical and current scores use different protocols and are not directly comparable.
From evaluation to training
The public benchmark establishes a baseline. Private evaluations and client-specific training turn measured failures into improvements.
Private evaluations
Held-out and client-specific binaries expose capability gaps across models, checkpoints, and agents. Get capability maps and failure analysis for real cybersecurity workflows.
Explore private evalsRL + SFT training
Verified outcomes, expert demonstrations, and complete tool trajectories become deterministic reward signals and expert-curated datasets for client-specific training and re-evaluation.
Explore RL DataInterested in evaluating or training your agent?
Get in touch