AgentRE Challenge / Rules

AgentRE Challenge Official Rules

Official Rules v1.1, effective August 3, 2026. Season 1 rewards original, safe, reproducible Linux x86-64 ELF challenges that create meaningful static analysis difficulty for frontier AI agents.

Register Entry Submission Instructions Leaderboard
Eligibility

Free skill contest with objective technical requirements.

Entrant eligibility

  • Entry is free; no purchase or payment is required.
  • Entrants must be at least 18 years old.
  • Entrants need a valid GitHub account.
  • Season 1 is open to legal residents of the 50 United States and Washington, D.C., except where prohibited.
  • AgentRE organizers, judges, contractors involved in judging, and members of their immediate households are not eligible.
  • Individuals and teams may enter, but each team must designate one prize recipient.
  • Each entrant or team may submit up to 3 entries and may win only one prize.

Submission eligibility

  • Original work or work the entrant has the right to submit.
  • Target Linux x86-64 ELF and support static analysis.
  • Remain under 10 MB.
  • Include checksums and private ground truth.
  • Public source release is not required. Finalist or prize-eligible entries may be asked to provide complete source code and build instructions privately for verification.
  • Privately provided source and build materials remain confidential unless the entrant chooses to publish them.
  • Have clear, verifiable intended behavior and a matching binary SHA-256.
  • Give the organizer account access before the deadline and keep access available until results are final.
Submission Freeze

The final tag freezes the evaluated commit.

git tag agentre-season-1-final
git push origin agentre-season-1-final

Frozen commit rules

  • The tag agentre-season-1-final must exist before the final deadline.
  • AgentRE records the resolved full 40-character commit SHA.
  • Only that frozen commit is evaluated.
  • Branch changes after the deadline do not alter the entry.
  • Moving or deleting the final tag may disqualify the entry.
  • The binary checksum and submitted binary must match.
Disallowed Submissions

Do not win by creating risk, abusing infrastructure, or breaking tools.

Unsafe behavior

  • Real credential theft or credential collection
  • Persistence intended for deployment
  • Worm or propagation behavior
  • Destructive payloads or ransomware behavior
  • Exploitation of third-party systems
  • External downloads or required network access
  • Real command-and-control behavior
  • Collection of personal or confidential data

Invalid competition tactics

  • Third-party proprietary code without authorization
  • Malformed files whose primary purpose is crashing tools
  • Pathological or intentionally excessive resource consumption whose primary purpose is to abuse evaluation infrastructure rather than create a reverse-engineering challenge
  • Build scripts intended to damage or compromise infrastructure
  • Repository workflows intended to execute automatically
  • Dependencies on inaccessible external services or files
  • Entries that merely exploit unsupported file formats or infrastructure failures

Legitimately difficult binaries are valid even when frontier models cannot solve them. Difficulty alone is never grounds for rejection. Controlled simulation of security-relevant behavior may be accepted only when inert, isolated, safe, clearly disclosed, and useful for reverse-engineering evaluation.

Disqualification

Disqualification is based on documented reasons.

Technical and safety violations

  • Copied or unauthorized code
  • False ground truth or fabricated model-test claims
  • Hidden malicious behavior
  • Credential theft or collection
  • Persistence, propagation, ransomware, or destructive behavior
  • Attempts to attack the harness or evaluation infrastructure
  • Deliberate, challenge-unrelated resource exhaustion intended to abuse evaluation infrastructure
  • Automatic GitHub workflows targeting reviewers
  • Undisclosed packing, obfuscation, or unusual compiler techniques

Integrity violations

  • Tampering with tags, commits, checksums, or repository history
  • Harassment, collusion, or attempts to manipulate judging
  • Failure to preserve repository access
  • Violating GitHub Terms

Private notes should record the specific reason even when public status only says Ineligible.

Evaluation Rights

AgentRE may inspect, validate, score, and reject entries when needed.

Validation

  • Reject unsafe, copied, trivial, broken, unverifiable, or low-quality submissions
  • Request source and build instructions privately from finalist or prize-eligible entrants when needed
  • Inspect privately provided source and rebuild binaries for verification
  • Compare rebuilt binaries against submitted artifacts
  • Test different compilation environments when necessary

Evaluation

  • Evaluate against multiple AI agents
  • Conduct manual scoring review
  • Disqualify rule violations
  • Decline to execute any artifact
  • Withhold sensitive implementation details

Publication

  • Publish approved GitHub handles
  • Publish approved challenge titles
  • Publish rankings and aggregate scores
  • Publish sanitized summaries
  • Keep private source, build materials, ground truth, and notes confidential
Ownership And Licensing

Participants retain ownership of their work.

Limited competition permission

By entering, participants grant AgentRE a limited, nonexclusive permission to access and clone the private repository, inspect and rebuild the submission, evaluate it against AI models, store an internal evaluation copy, and publish approved handle, title, score, rank, and sanitized results.

Separate written agreement required

  • Selling the source or binary
  • Publishing private source or ground truth
  • Commercial model training use
  • Permanent redistribution
  • Adding the entry to a paid private benchmark
  • Public release of source or binaries

Verification is for reproducibility, safety, and result integrity. It does not require entrants to open-source or publicly disclose their research.

Prize Terms

First- and second-place awards for the hardest valid binaries.

Core award clauses

First place: The eligible submission with the lowest aggregate correctness score receives $1,000 USD plus one month of the recipient's choice of ChatGPT Pro or Claude Max 20x. At least 1 entry must pass validation, the evaluation must produce a reliable result, and the submission must be lowest-scoring on at least 4 of the 6 official model evaluations.

Second place: The next-highest-ranked distinct eligible submission after tie-breaks receives one month of the recipient's choice of ChatGPT Pro or Claude Max 5x.

All rankings are based solely on AgentRE-verified evaluations performed against the frozen submission commit under the published harness, model panel, budgets, and scoring methodology. Participant-reported results are informational and do not determine award recipients.

Award conditions

  • No purchase or payment is required to enter or claim an award.
  • First and second place are determined after all published tie-breakers.
  • If an exact first-place tie remains, tied entries divide the $1,000 equally and each receives the first-place subscription award. No separate second-place award is issued.
  • If an exact second-place tie remains, each tied entry receives the second-place subscription award.
  • If no entry satisfies the minimum requirements, including the 4-of-6 first-place requirement, AgentRE may decline to issue either award.
  • The competition is not sponsored, endorsed, or administered by GitHub.

Recipient verification and fulfillment

  • Each provisional recipient has seven calendar days to provide legal name, mailing address, payment information if applicable, age and eligibility confirmation, any reasonably necessary U.S. tax documentation, and a signed eligibility and rights confirmation.
  • Cash payment and subscription choice will be arranged privately with each recipient.
  • AgentRE will provide each award within 14 business days after verification.
  • Failure to respond or verify eligibility may result in the award passing to the next-ranked eligible entry.

Taxes

Recipients are responsible for applicable taxes. Awards are generally taxable income in the United States, even when no reporting form is issued.

Scoring Methodology

The winning binary is the one the official panel understands least.

Official model score measures how much of the required binary behavior the model correctly recovered. Entry difficulty score is 100 - average official model correctness. Higher difficulty score ranks higher. First place must also satisfy the 4-of-6 model-win requirement.

Standardized evaluation

  • Same harness commit
  • Same system prompt and scoring rubric
  • Same static-analysis tools
  • Same tool-call limit, token budget, time limit, and attempt policy
  • Same private ground-truth scoring method

Invalid runs are not scored as zero

  • Safety refusals unrelated to technical difficulty
  • Infrastructure failures, API errors, or tool crashes
  • Infrastructure failures caused by challenge-unrelated resource abuse
  • Malformed or unsupported binaries
  • Missing files or undocumented dependencies

A model's failure to solve a valid, difficult binary within the standard evaluation budget is a valid outcome, not an infrastructure failure.

Reported result metrics

    Panel-win requirement

    For first-place eligibility, the winning entry must have the lowest correctness score on at least 4 of the 6 official models when compared with the other eligible entries. If multiple entries tie for the lowest valid score on a model, each tied entry may count that model as a panel win.

    Ground truth is a validation requirement

    Poor, incomplete, or false ground truth should fail validation instead of being treated as a small scoring penalty.

    Allowed Tools

    Official evaluations use the standard AgentRE static-analysis tool set.

    Season 1 models receive the same fixed tools exposed by the AgentRE harness, running from the official agentre-bench-tools:latest image built from Dockerfile.tools. Models may not generate and execute arbitrary Python or code in any other programming language. Otherwise, a model could build custom parsers, symbolic executors, deobfuscators, or entirely new analysis tools during evaluation, making model comparisons unfair and difficult to reproduce. The tool-call budget is 25 calls, with a 30-second per-tool timeout.

    Docker image

    AgentRE should build and pin the official image for the harness commit used at contest opening:

    docker build --platform linux/amd64 -t agentre-bench-tools:latest -f Dockerfile.tools .

    Not allowed in official scoring

      Official Model Panel

      Season 1 uses the public AgentRE frontier-model panel.

      Only AgentRE runs against the official panel count toward rankings and awards. Participant tests against other models may be mentioned, but they do not determine rankings.

      If a model becomes unavailable, materially changes, cannot complete valid harness runs, or is blocked from participating, AgentRE may replace it with a reasonably comparable model. Any substitution will be publicly documented before final results are announced. Exact model identifiers, harness commit, prompt version, budgets, attempt policy, and test date should be recorded when submissions open.

      Tie Breaks

      Tie-break order is deterministic.

        Result Challenges

        Factual challenges have a short review window.

        48-hour dispute window

        AgentRE publishes provisional results, allows 48 hours for factual challenges, resolves valid errors, then publishes final results.

        Challenge scope

        Challenges must identify a checksum, harness, scoring, eligibility, or clerical error. Disagreement with the published judging criteria is not grounds for reconsideration.

        Changes And Cancellation

        Rule changes are limited after entries freeze.

        AgentRE may modify, suspend, or cancel the competition only when fraud, a security incident, a material technical failure, a legal restriction, or another circumstance outside AgentRE control prevents fair operation. Material rule or scoring changes will not be applied retroactively to already-frozen submissions unless necessary to address a security or integrity issue.

        AgentRE may decline to issue awards when no entry qualifies, no entry receives a reliable valid evaluation, or the evaluation cannot produce a reliable result.

        What Makes A Strong Challenge

        Focus on real static reverse-engineering difficulty.

        Encouraged

        • Indirect data flow
        • State-machine reconstruction
        • Custom protocol recovery
        • Algorithm identification
        • Cross-function reasoning
        • Sparse but recoverable semantic clues
        • Compiler optimization effects
        • Meaningful control-flow analysis
        • Correct inference of overall intent

        Discouraged

        • Broken files
        • Tool crashes
        • Excessive file size
        • Challenge-unrelated resource abuse targeting evaluation infrastructure
        • Undocumented runtime dependencies
        • Unsafe payloads
        • Infrastructure abuse
        • Aggressive anti-analysis as the primary challenge