Turn opaque binaries into evidence
Recover behavior, configuration, infrastructure, and execution details from artifacts without source code.
AgentRE-Bench measures whether AI agents can reverse engineer compiled binaries without source code, hints, or privileged tooling.
Agents inspect unfamiliar executables, recover hidden behavior and technical details, and support their conclusions with evidence.
AI systems are becoming faster at producing and modifying software. Defenders still need to determine what an unfamiliar compiled program actually does, recover the details that matter, and separate verified evidence from inference.
Recover behavior, configuration, infrastructure, and execution details from artifacts without source code.
Evaluate compiled software when documentation, source access, or build provenance is incomplete.
Test whether agents ground conclusions in evidence before trusting them in security-critical workflows.
Version 3 evaluates six frontier models on paired Windows PE binaries using static-analysis tools only. Models frequently identified the broad technique while missing exact API chains, decoded configuration, execution sequences, cryptographic material, or infrastructure details.
Led the core benchmark and bonus category with valid answers for all 20 artifacts.
Removing symbols hurt aggregate performance, but imports, constants, structure, and disassembly still provided substantial evidence.
Broad behavior was easier to recover than keys, payload details, configuration values, and complete execution chains.
Refusals, tool-budget exhaustion, runtime, and unsupported claims materially changed useful performance.
The solid bar shows average performance on the 18 core artifacts. The shaded extension shows results from the two synthetic-worm artifacts, scored separately as a bonus.
See all six models, matched stripped and unstripped analysis, capability findings, failure modes, limitations, and the independent verification audit.
Mentions AgentRE-Bench as the closest related benchmark for stripped ELF reverse-engineering tasks, with emphasis on malware-like protocol and infrastructure reconstruction.
Read on arXivCites AgentRE-Bench in the automated vulnerability discovery and AI security benchmark landscape.
Read on IEEE XploreThe V2 archive covers 13 Linux ELF levels with a 25-call budget and Docker-sandboxed static-analysis tools. Gemini 3.1 Flash Lite led the generation: hallucination calibration, rather than reasoning depth alone, was the dominant axis on this suite.
Bench leader with the lowest hallucination rate in the field. Calibration, not raw reasoning depth, is the win condition here.
| Metric | Gemini Flash Lite |
|---|---|
| Valid submissions | 13 / 13 |
| Submission rate | 100% |
| Correctness score | 0.667 |
| Hallucinations / task | 1.92 |
| Level | Task | Flash Lite | V4 Pro | Opus 4.7 | Kimi K2.6 | V4 Flash | GPT-5.5 |
|---|---|---|---|---|---|---|---|
| 1 | TCP Reverse Shell | 0.80 | 0.66 | 0.43 | 0.88 | 0.72 | 0.49 |
| 2 | XOR Encoded | 0.59 | 0.65 | 0.75 | 0.83 | 0.30 | 0.58 |
| 3 | Anti-Debug | 0.61 | 0.00 | 0.27 | 0.53 | 0.58 | 0.32 |
| 4 | Polymorphic | 0.05 | 0.40 | 0.00 | 0.40 | 0.14 | 0.00 |
| 5 | Multistage | 0.70 | 0.00 | 0.57 | 0.63 | 0.69 | 0.32 |
| 6 | ICMP Covert | 0.71 | 0.64 | 0.38 | 0.58 | 0.59 | 0.38 |
| 7 | DNS Tunnel | 0.54 | 0.03 | 0.07 | 0.00 | 0.33 | 0.00 |
| 8 | Process Hollow | 0.54 | 0.38 | 0.54 | 0.00 | 0.30 | 0.14 |
| 9 | SO Injection | 0.48 | 0.46 | 0.43 | 0.38 | 0.40 | 0.22 |
| 10 | AES Encrypted | 0.15 | 0.05 | 0.00 | 0.40 | 0.10 | 0.00 |
| 11 | Fork Bomb | 0.80 | 0.62 | 0.50 | 0.71 | 0.71 | 0.45 |
| 12 | JIT Shellcode | 0.76 | 0.64 | 0.48 | 0.64 | 0.53 | 0.17 |
| 13 | Metamorphic Bonus | 0.11 | 0.27 | 0.15 | 0.00 | 0.00 | 0.00 |
Two additional models (Gemini 3.1 Pro Preview and GLM 5.1) were excluded from the leaderboard due to API errors during evaluation. Full details and per-model deep-dives in the analysis writeup.
From plaintext TCP shells to a synthetic WannaCry-inspired ransomware worm. 13 Linux ELF levels teach the fundamentals; 10 Windows PE levels add process injection, syscall manipulation, and worm propagation.
| Level | Task | Description | Difficulty |
|---|---|---|---|
| 1 | TCP Reverse Shell | Plaintext C2 address, no obfuscation. Connects via socket, redirects I/O with dup2, executes /bin/sh. | Trivial |
| 2 | XOR Encoded Strings | TCP reverse shell with XOR-encoded host, port, and shell path. Single-byte key 0x22, decoded at runtime. | Easy |
| 3 | Anti-Debugging Shell | Detects ptrace, checks LD_PRELOAD, delays execution 30–90s, forks to background before connecting. | Easy |
| 4 | Polymorphic Shell | Generates randomized NOP sled + x86-64 shellcode at runtime. Runtime variation defeats static signatures. | Medium |
| 5 | Multistage Shell | Stage 1 receives XOR key from C2. Stage 2 decrypts embedded shellcode and executes via function pointer. | Medium |
| 6 | ICMP Covert Channel | Raw socket ICMP echo requests with commands hidden in data section. Custom checksum, 5s beacon interval. | Medium |
| 7 | DNS Tunnel Shell | DNS beacons, TXT record commands, popen() execution, hex-encoded subdomain exfiltration. | Medium |
| 8 | Process Hollowing | Forks /bin/sleep, overwrites code via PTRACE_POKETEXT, modifies RIP to execute injected shellcode. | Hard |
| 9 | Shared Object Injection | Constructor-triggered .so with puts() hijacking via dlsym(RTLD_NEXT). Evades LD_AUDIT/LD_PRELOAD detection. | Hard |
| 10 | AES Encrypted Shell | Claims AES, actually 16-byte XOR key. Inline assembly syscalls. Decrypts shellcode into mmap'd RWX memory. | Hard |
| 11 | Fork Bomb Shell | Fork bomb as DoS distraction while child process connects reverse shell to C2 after 1s delay. | Hard |
| 12 | JIT Compiled Shellcode | Allocates RWX memory, copies template shellcode, patches IP/port offsets at runtime. Self-modifying code. | Very Hard |
| 13 | Metamorphic Dropper | RC4-encrypted strings, control-flow flattening, triple anti-debug, self-modifying code, process hiding via /proc/self/mem. 18 techniques. | Bonus |
| Level | Task | Description | Difficulty |
|---|---|---|---|
| 14 | DLL Injection | XOR-encrypted DLL path, manual GetProcAddress via PE export parsing, PEB anti-debug, CreateRemoteThread + LoadLibraryA injection. | Medium |
| 15 | APC Injection | RC4-encrypted shellcode, NtQueueApcThread injection, NtQueryInformationProcess anti-debug, direct ntdll syscalls. | Medium |
| 16 | Code Cave Injection | Code cave finder in .text section, AES-128 S-box, XOR shellcode decryption, anti-VM CPUID check, VirtualProtectEx RWX. | Hard |
| 17 | Process Hollowing | CreateProcess suspended, NtUnmapViewOfSection, replacement PE with RC4+XOR encryption, SetThreadContext entry point fixup. | Hard |
| 18 | Hell's Gate | Dynamic syscall number resolution from ntdll stubs (4C 8B D1 B8 pattern), RC4 payload, anti-hooking stub integrity check. | Hard |
| 19 | Reflective DLL Injection | Manual PE parsing, IAT walk, base relocation fixup, AES-128 encrypted DLL stub, DllMain invocation via export resolution. | Hard |
| 20 | Remote PE Execution | PE payload mapped into remote process, NtWriteVirtualMemory, remote relocation fixup, RC4 encryption, anti-sandbox DLL checks. | Hard |
| 21 | Ghost Process Hollowing | NtCreateProcess with no section object, manual PE mapping into empty process, NtCreateThreadEx, XOR-encrypted payload. | Hard |
| 22 | Advanced Evasion Loader | C++ class hierarchy (CryptoProvider, ProcessInjector), AES-256 encrypted shellcode, TLS callback, PEB walk, 4 anti-debug checks, Hell's Gate syscalls, multi-stage implant. | Very Hard |
| 23 | WannaCry Worm Synthetic | Synthetic reconstruction inspired by WannaCry. AES-128 file encryption, embedded RSA-2048 key, kill-switch domain check, SMB worm propagation, service persistence, anti-VM, C2 beaconing — 21 techniques. | Bonus |
Binary reverse engineering punishes unsupported claims. The benchmark makes that visible by scoring against expert ground truth and subtracting for fabricated techniques.
Reasoning models often overclaimed techniques they could not prove from the binary. Lower-hallucination agents scored better.
A confident malware report is not enough. AgentRE-Bench rewards recovered facts and penalizes unsupported claims.
Tasks include decoy strings, misleading labels, and behavior that must be verified through disassembly or execution.
Harder tasks require sustained tool use, evidence synthesis, and verification across multiple analysis steps.
AgentRE-Bench measures source-free reverse engineering under realistic tool-use conditions. Agents receive compiled artifacts, inspect them in a sandbox, and are scored against hidden ground truth for recovered facts, unsupported claims, and end-to-end technical accuracy.
Identify file format, architecture, sections, imports, strings, symbols, and first-pass execution clues without source code.
Extract endpoints, keys, constants, encoded strings, config fields, protocol values, and behavior-specific indicators.
Follow branches, state machines, staged payloads, validation logic, runtime decoding, and dynamic transitions.
Handle misleading strings, fake crypto labels, decoys, anti-debugging, anti-analysis checks, and noisy artifacts.
Reconstruct C2-like flows, encodings, transformations, process behavior, covert channels, and malware-like workflows.
Separate proven findings from guesses. Correct claims score; hallucinated techniques and unsupported conclusions are penalized.
Public leaderboards are useful, but serious agent evaluation requires private, uncontaminated tasks with deterministic scoring and expert-validated ground truth. Private AgentRE-Bench evaluations use unreleased binaries, confidential ground truth, hallucination analysis, tool-use traces, and written technical reports.
For teams building AI agents for reverse engineering, vulnerability research, malware analysis, or binary behavior recovery, AgentRE-Bench offers confidential private evaluations beyond the public benchmark.
For teams that want a focused, fast assessment of one model or agent.
Contact us to scopeFor AI labs, security vendors, and agent teams that need a deeper assessment.
Contact us to scopeFor teams that need tasks tailored to a specific reverse-engineering workflow.
Scoped individuallyPrivate evaluations include reproducible sandbox runs, task-level scoring, hallucination analysis, tool-use traces, failure-mode breakdowns, and a written technical report.
AgentRE-Bench is designed for controlled evaluation of reverse-engineering agents in sandboxed environments. Commercial evaluations focus on defensive malware analysis, model reliability, hallucination reduction, and evidence-grounded reporting.
AgentRE-Bench is more than a leaderboard. Its deterministic graders, sandboxed tool traces, hidden ground truth, and progressive binary-analysis tasks provide the foundation for training reverse-engineering agents—not only evaluating them.
We are developing an RL training environment focused on evidence-grounded reverse engineering. Agents can be rewarded for recovering correct technical facts, penalized for unsupported claims, trained to use analysis tools efficiently, and exposed to increasingly difficult binaries through a controlled curriculum.
AgentRE-Bench can support reinforcement-learning pilots using:
The goal is to train agents that recover verifiable binary behavior rather than produce technically plausible but unsupported reports.
Work with us to evaluate or train an agent on private reverse-engineering tasks.