AgentRE Challenge / Submit

Private GitHub Submission Workflow

Keep the full submission private, register only safe metadata publicly, and let AgentRE evaluate exactly the frozen commit resolved from your final tag.

Register by PR Read Rules Leaderboard
Participant Steps

Prepare the private repository, then register with a public-safe PR.

  1. Create a new private GitHub repository named agentre-challenge-entry.
  2. Add the configured AgentRE organizer agentrebench as a collaborator.
  3. Add the required files.
  4. Commit all submission material.
  5. Verify the binary checksum.
  6. Create and push the final tag agentre-season-1-final.
  7. Copy the resolved full 40-character commit SHA from that tag.
  8. Open a public-safe AgentRE registration pull request.
  9. Add one public-safe registration JSON file and add it to the registration manifest.
  10. Keep the private repository available through the end of judging.
Collaborator Access

Invite the organizer before registering.

GitHub path

Repository → Settings → Collaborators → Add people

Add agentrebench and keep access available through the end of judging.

Access is private review only

Do not place private repository URLs anywhere public. AgentRE infers the private repository from your GitHub handle and the required repository name, then uses collaborator access to clone the frozen commit privately.

Required Repository Structure

Keep required artifacts and any verification materials private.

agentre-challenge-entry/
├── README.md
├── submission.json
├── challenge.elf
├── SHA256SUMS
├── ground_truth.md
└── verification/             # only if requested
    ├── build.sh
    └── src/

README.md

  • Challenge title
  • Supported environment
  • Any required non-networked runtime dependencies
  • Ownership or licensing statement
  • Any disclosures needed to evaluate the binary safely

submission.json

Public-safe metadata plus the final tag, resolved commit SHA, and binary checksum. Use the template below and keep it in the private repository.

challenge.elf

The exact Linux x86-64 ELF binary AgentRE evaluates. It must remain under 10 MB.

SHA256SUMS

SHA-256 digest for the submitted binary and any other critical artifacts needed for verification.

ground_truth.md

Private answer key and scoring guide. This file must not be pasted into the public registration PR or copied into AgentRE-Bench.

verification/

Finalist or prize-eligible entries may be asked to provide complete source code and non-networked build instructions here. These materials are used only for private verification and remain confidential unless the entrant chooses to publish them.

submission.json Template

Use this exact shape unless AgentRE announces an update.

{
  "github_handle": "example-user",
  "challenge_title": "Example Challenge",
  "architecture": "linux-x86_64",
  "binary_path": "challenge.elf",
  "final_tag": "agentre-season-1-final",
  "final_commit_sha": "RESOLVED_FULL_40_CHARACTER_COMMIT_SHA",
  "binary_sha256": "SHA256_OF_BINARY",
  "public_handle": true,
  "public_title": true
}
Private Ground Truth

Make the answer key useful for scoring.

ground_truth.md must include

  • What the binary actually does
  • Important functions
  • Important algorithms
  • Key behaviors
  • Required facts for a correct model answer
  • Partially correct findings
  • Suggested scoring rubric
  • Deliberate reverse-engineering traps
  • Disclosure of packing, obfuscation, or unusual compiler techniques

Keep this file private

Do not paste ground truth into the public registration PR. Do not copy it into the AgentRE-Bench website repository. AgentRE will review it privately through collaborator access.

Verification Commands

Record the exact commit and verify checksums.

git tag agentre-season-1-final
git push origin agentre-season-1-final
git rev-parse agentre-season-1-final^{commit}
sha256sum challenge.elf
sha256sum -c SHA256SUMS

AgentRE evaluates only the commit resolved from the registered final tag before the deadline. Later branch changes do not affect the submitted entry, and moving or deleting the final tag may disqualify the entry.

Pull Request Registration

The registration PR stores only public-safe metadata and lets the static leaderboard update after merge.

Public registration data

  • GitHub handle
  • Required private repository name, not a URL
  • Public challenge title or "Withheld until judging"
  • Final tag name and resolved full 40-character commit SHA
  • Binary SHA-256
  • Required confirmations and display permissions

Never publish

  • Private repository URLs
  • Binary attachments or downloads
  • Source code
  • Ground truth
  • Secrets, tokens, credentials, or private keys
  • Personal information or third-party proprietary code
  • Private download links
Official Tool Environment

Design for the standard AgentRE static-analysis harness.

AgentRE evaluates entries with the fixed tools exposed by the public harness and the official agentre-bench-tools:latest Docker image. Models may not generate and execute arbitrary Python or code in other programming languages. This prevents models from building custom analysis tools during evaluation and keeps comparisons fair and reproducible.

docker build --platform linux/amd64 -t agentre-bench-tools:latest -f Dockerfile.tools .
Automation Warning

Do not rely on GitHub Actions or auto-executing workflows.

  • Do not rely on GitHub Actions.
  • Do not configure automatic workflows intended to execute when the organizer accesses the repository.
  • Do not require network access for build or evaluation.
  • Do not commit credentials, API tokens, private keys, personal information, or third-party proprietary code.
  • AgentRE evaluates only the frozen commit resolved from the registered final tag.
Register by PR Read Rules