When you get a binary, you still need to know what it actually does. The source might be unavailable. The names might be stripped. The useful details might be buried behind layers of encoding. That matters when investigating malware, checking a dependency, or understanding software nobody can explain anymore. The answer needs evidence you can check.
That is what we want AgentRE to measure. Give a model the binary, tools, and a fixed budget, then see what it can recover. Can it follow the execution path? Find the keys? Can somebody else take that explanation back to the instructions and confirm it?
For v3.1, we ran five models on twenty Windows binaries, covering ten programs with and without symbols. Each model got five attempts per binary: 500 attempts. We averaged all five scores. Refusals and budget exhaustion stayed in the results.
Muse Wonder finished first at 87.37/100, followed by GPT-6 Astra at 87.04. Both submitted answers on all 100 attempts. Muse earned more technique credit. Astra recovered encryption details more consistently and led the bonus tasks. The gap is small, and these runs do not establish that Muse would reliably beat Astra again.
The high scores came from concrete work. Models usually recovered the injection steps, target processes, and memory operations. Every submitted answer that required RC4 received full algorithm and key credit. The keys are embedded in these samples; recovery means finding their bytes and understanding their use.
The mistakes matter too. Muse reported a cipher’s round constant as round × 18 where the code uses round × 27. Several answers treated an unregistered TLS routine as something that would execute before the entry point. Finding a function does not establish that the program calls it.
Encryption also exposed a scoring distinction. Grok found 120 of 124 required component keys, but often left the round count out of the algorithm field. That reduced algorithm and key credit. Some correct details appeared in another field. Those losses reflect the answer format too; they cannot all be called hallucinations.
Completion mattered. Opus submitted 78 valid answers and Gemini 36. Opus recovered every required algorithm and key in its submitted answers. Its Overall score includes refusals. Showing coverage beside answer quality makes that difference visible.
These public samples measure performance on this suite, with source withheld during each attempt. They do not establish performance on every unfamiliar binary. The scorer still needs independent wording checks.
A model can recognize a cipher and still get its implementation wrong. Our reconstruction data lets us compare reconstructed functions’ outputs with the binary’s outputs on private test inputs. Reinforcement learning can use those checks to reward correct results and train against specific mistakes. Separate evaluation tasks give us a way to check whether training improves recovery and where errors still remain.
We’re building training data around specific failures: incomplete decoding, incorrect transformations, and unsupported claims. Our goal is to make reverse engineering more reliable, with answers defenders can trace directly back to the binary and verify.