# Scorer 6.2.0 contract

**2026-10-25 calibration correction:** use scorer **6.2.0** with `version3/v3_4/tasks_v3_4_scorer6_2_0.json`. Version 6 recognizes specific semantic facts and bounded algorithm descriptions, preserves unsupported claims, and flags unrecognized wording for review. Rubric v3.4 retains v3.3's corrections to strings-versus-payload scope, decoded-string requirements, and embedded key recovery, and adds source-supported optional and contradicted facts. All 159 interpretations flagged in the 500 saved episodes have been reviewed; future unfamiliar wording still requires review. Existing episodes remain immutable. See [the calibration and rescoring record](calibration-20261025.md). Post-hoc calibration does not establish independent validation or equal collection conditions.

Date: 2026-10-25. The canonical implementation is [`scorer.py`](../src/agentre_bench/scorer.py); fact predicates and aliases live in [`calibration.py`](../src/agentre_bench/calibration.py) and [`scoring_contract.py`](../src/agentre_bench/scoring_contract.py). Aggregation remains `artifact_mean_v1`. Historical scorers remain archived. New PE runs select [rubric v3.4](../version3/v3_4/README.md), answer schema 2, and tool/run protocol 4.

Version 6.1.1 adds scoped contradiction checks within and across technique entries and related detail values. An affirmation and denial of the same operation, target, and explicit condition cannot retain full credit. Contradictory evidence remains visible and requires review. Different named targets, failure branches, symbol resolution, absent imports, and compiler-runtime caveats are kept distinct. This is a bounded guard; it does not prove arbitrary natural-language consistency.

It also rejects explicit AES diffusion/key expansion in the synthetic bytewise S-box/XOR transforms, wrong RC4 permutation sizes, and modular-addition transforms described as plain XOR. Correct descriptions of absent operations retain credit. The v3.4 ground truths, component denominators, and weights are unchanged. Scorer 6.1.0, its manifest, and earlier reports remain archived.

**6.1.2 guarded release:** contractions and absence/disabled denials no longer establish positive technique evidence; unknown second-sentence assertions remain visible for review. Import-table absence does not imply absence of dynamic resolution or invocation. RC4 descriptions using payload/keystream addition are rejected, while legitimate modular arithmetic in KSA/PRGA remains valid. The 158 exact reviewed interpretations and ground truths are unchanged.

**6.2.0 explanation adjudication:** bounded, complete phrase rules in [`explanations.py`](../src/agentre_bench/explanations.py) recognize source/binary-supported buffer-width explanations, compiler TLS callbacks, conditional early returns, unused check results, conditional call paths, and preferred-image addresses. Evidence profiles are pinned to the complete rubric JSON and source/binary hashes; a different rubric does not inherit these facts. Supported continuations contribute neither extra technique credit nor false-positive items. Unknown or changed statements remain visible and require review. Qualifying an unregistered TLS routine cannot itself establish callback execution or thread hiding. A prologue signature comparison earns stub-integrity credit and does not imply invoking the inspected API.

The 54 original entries are resolved, including the one conditional-execution qualification, with an identified retrospective coding-assistant adjudication. Six related explanatory continuations found during the uniform rescore are also checked; every artifact's TLS callback array is verified against its runtime symbols. No model identity or episode-specific score override is used. Ground truths, weights, prior exact aliases, and original episodes remain unchanged. Current reports retain the evidence and qualifications in each claim audit. Comparisons touching any future unresolved interpretation still suppress inferential winner claims. See the [release policy](scorer-release-policy.md) and [current result status](results-current.md); earlier releases remain archived.

## Fields and applicability

| Field | Standard ELF | PE injection | Bonus |
|---|---:|---:|---:|
| decoded_c2 | .40 | .10 | .15 |
| techniques | .30 | .25 | .15 |
| file_type | .10 | .03 | .03 |
| encoded_strings | .10 | .02 | .02 |
| c2_protocol | .10 | .05 | .05 |
| injection_details | — | .20 | — |
| encryption_algorithm | — | .05 | .10 |
| encryption_key | — | .10 | .20 |
| encryption_key_storage | — | — | .05 |
| decoded_strings | — | .10 | .15 |
| anti_analysis | — | .10 | .10 |

Null or empty expected fields leave the weighted score; remaining weights renormalize to sum to one. A meaningful boolean `false` remains scorable. A substantive answer for an explicitly absent field incurs a spurious-field penalty of .05 (.03 for bonus). PE injection distinguishes an omitted annotation from an explicit null: omitted fields are ungraded, not evidence that the behavior is absent. The standard and bonus rubrics retain their existing absence rules. Corrected v3.1 normalizes the injection annotations on Levels 17, 19, 21, and 22.

Scores are the weighted sum minus penalties, clamped to [0, 1]. Results expose effective weights, field scores, missing and unsupported techniques, spurious fields, and scorer version. The tier comes from explicit `tier`, then legacy level-13/23 naming, then PE file type, then standard.

## Technique and detail matching

Version 6.1.0 adds [158 source-audited full-claim interpretations](../src/agentre_bench/reviewed_claims.py). Each maps an exact complete claim to facts and records evidence hashes and a rationale. These interpretations contain no model IDs or score overrides and apply identically in every artifact. Mixed claims retain false members such as an incorrect extension count or unsupported TLS execution. Changed numbers, prefixes, and suffixes do not inherit the exact interpretation. This is post-hoc coding-assistant review, not independent human calibration.

PE techniques and anti-analysis use F1: `2 * matched / (2 * matched + unsupported + missing)`. Canonical labels, reviewed aliases, and explicit fact predicates match. Specific facts can satisfy a broader requirement, never the reverse; generic injection does not imply DLL injection. Separate list entries cannot pool words into evidence. Compound claims retain extra facts and unrecognized list members. Duplicate facts count once. Source-supported optional facts neither increase recall nor count as unsupported. Both technique and anti-analysis audits expose canonical/alias/semantic/rejected status, matched/unsupported/optional facts, and review flags. Unknown wording is not automatically a hallucination; `hallucinated_techniques` is reserved for explicitly contradicted facts. PE has no additional fixed technique penalty.

ELF techniques and anti-analysis use set Jaccard. Extra ELF technique claims additionally cost .05 each (.03 in the ELF bonus). This is different from PE F1. Empty truth sets remove the field through applicability rules.

Structured details match corresponding keys or finite role aliases. Only enumerated related fields combine evidence for composite roles; arbitrary notes cannot fill every field. Expected fields receive equal weight. List-to-list values use Jaccard; prose lists receive per-item recall. APIs, permissions, section mapping, self-targeting, and NULL handles have explicit predicates. Byte-string fields compare exact bytes, allowing whitespace-separated whole-byte groups, bounded annotations, and literal decoded strings with byte escapes. Negated clauses supply no evidence; separate caveats do not erase affirmative clauses. An explicit decoded-byte field takes precedence over aliases; conflicting nonempty byte aliases score zero. Related detail roles can supply an API from an affirmative method chain and its flags from the corresponding role field; mere symbol resolution cannot supply invocation. This finite matcher does not prove every explanatory address, equation, or runtime claim.

Keys use authoritative byte arrays, then explicit hex or literal text; case is significant. Incidental hex inside arbitrary prose is not parsed. Bounded key annotations may contain balanced parentheses but cannot disclaim or guess the key. Level 22 accepts either its exact 16 effective bytes or the exact documented embedded array declared by `accepted_embedded_key_hex`. Other prefixes, suffixes, components, and algorithms do not qualify. This exception is explicit in the rubric and byte-verified in the artifact.

Component algorithms are parsed independently of the expected answer and model identity. Repeating-key XOR aliases and RC4 descriptions of KSA/PRGA/XOR-keystream operations are supported. An explicitly prior XOR layer is distinguished from guessed cipher bundles. Synthetic S-box/XOR descriptions require the transform and correct round count for full credit. Wrong round counts, wrong explicit round multipliers, inverse substitutions, contradictory claims, and guessed bundles reject the algorithm. Diagnostics retain both strings, match reason, and accepted key representation.

Bare parameterized families such as AES, RSA, and synthetic S-box/XOR receive half credit for the corresponding parameterized family. Corrected transforms are `SBOX-XOR-10` and `SBOX-XOR-14`; standard AES does not match them. Key-storage credit remains whole-token overlap with contradiction rejection. File type canonicalizes PE64/PE32+ and ELF forms, awarding half for the right family but wrong/unspecified bitness. C2 awards exact normalized address credit or half for a matching host; protocol and encoded-string values are exact. `encoded_strings` means textual strings, not arbitrary encrypted code. Level 23 decoded-string requirements cover decoded C2 bytes; plaintext strings are ungraded evidence.

## Encryption components

The rubric's `encryption_components` list declares a unique purpose `id`, one `algorithm`, boolean `required`, and boolean `key_required` for each component. Required keys must have authoritative byte material. Explicit component requirements take precedence over retained legacy `encryption_details` evidence annotations. Invalid requirements fail before a model request.

Answers submit `encryption_components: [{"id": "string_obfuscation", "algorithm": "XOR", "key": "deadbeef1337cafe"}]`. Each entry requires `id` and `algorithm`; omit an unknown key. IDs normalize case and punctuation, with no fuzzy purpose matching. The generic tool schema lists the purpose IDs used by this suite. Duplicate IDs, incorrectly typed values, more than 64 entries, and simultaneous legacy/new formats are invalid submissions.

For component `i`, let `a_i` be its algorithm score and `k_i` be exact key-byte correctness (0 or 1). The algorithm field is the mean `a_i` over required components. The key field is the mean `a_i × k_i` over required components whose keys are required. Thus a correct key paired with the wrong algorithm earns zero key credit, and a bare family identification can earn at most half key credit. Keys never match a global pool. Key-storage scores also stay within their component and are multiplied by algorithm credit before averaging applicable required components. Existing tier weights remain unchanged.

Missing required components contribute zero. Components declared optional do not affect denominators or increase scores; their recovery is shown in diagnostics. Keys not marked required do not affect the key denominator. Unknown component IDs and ambiguous/duplicate claims appear in the component audit and incur the tier's spurious-claim penalty (.05 for PE, .03 for bonus). A recognized component with a wrong algorithm/key receives zero corresponding credit. Entirely unannotated PE encryption remains ungraded.

Legacy flat `encryption_details` remains available for explicit diagnostic rescoring and single-component compatibility. It can match only one component with an unambiguous algorithm association; multiple components using that algorithm require purpose IDs. It cannot recover an algorithm from one component and a key from another. Legacy nested rubrics without explicit components are adapted into separate algorithm-bearing nodes, never flattened into one pool. Historical scores must still be reproduced with their archived scorer.

In v3.4, Levels 16, 17, and 23 each require two components. Level 23's XOR-only answer earns **0.5 algorithm and 0.5 key field credit**; both components recovered correctly earn 1.0. Its other component is `SBOX-XOR-10`, not AES. The truncated RSA marker remains a separate static finding and is not graded as a working RSA cipher or recoverable RSA key.

## Submission and run validity

The harness accepts only schema-valid `final_answer` calls. Invalid calls receive feedback and consume budget; they may be repaired within the remaining episode limits. A terminal episode without a valid submission contributes zero. The low-level scorer is also used for diagnostics on partial dictionaries and does not itself certify an episode as valid. See [the tool contract](tool-contract.md).

**Overall is the arithmetic mean of all artifact scores**, on a 0–1 scale. Main averages standard and PE-injection tasks; Bonus averages bonus tasks. They are breakdowns, not additive headline components. On the 20-artifact PE suite, `Overall = (18 × Main + 2 × Bonus) / 20`: each artifact has 5% weight, each level 10%, and Level 23's pair 10%. On the 13-artifact ELF suite, the bonus has 1/13 weight. Main-only and bonus-only subsets remain on a 0–1 scale but cannot receive an official full-suite score.

Task failures retain zero-valued slots. Reports give explicit coverage counts, IDs, `score_denominator`, `main_tasks`, `bonus_tasks`, and `aggregation_version`. `official_overall_score` is null for filtered or infrastructure/provider-incomplete runs and for runs with unrecognized wording requiring calibration review. Repeated-trial aggregation independently enforces the same review gate. Compatibility keys `total_score` and `official_total_score` alias Overall and official Overall; their maximum is now **1.0**, not the historical 2.0. `aggregation_sensitivity` separately reports Main-only, equal-tier mean, and `legacy_main_plus_bonus` for comparing weighting choices. The legacy sum is diagnostic and is never the official headline. An empty selection has a diagnostic zero and no official score.

Aggregation version is included in run settings, task identities, the corrected manifest, and reports; incompatible manifests and saved reports are rejected. Component scoring originally introduced scorer 5.0.0/rubric v3.2 and tool/run protocol 4. The calibration correction uses scorer 6.2.0/rubric v3.4; it preserves protocol-4 collection conditions and original binaries. Older scored artifacts and scorers remain archived. A rescore of existing answers must identify the original episode and both scoring versions; it is not a new model run. New protocol-4 episodes are reruns.

Standalone batch scoring raises for missing answer files instead of silently shrinking its denominator. Its filename-based batches are diagnostics, not certification that a full versioned manifest was evaluated. Both the harness and standalone reporter expect one score row per artifact and reject duplicate or invalid task IDs. For repeated trials, first compute each artifact's mean across a fixed, equal number of trials, then average those means. Do not flatten unbalanced trials into an episode-weighted leaderboard.

The offline `agentre-aggregate-trials` / `aggregate_trials.py` command implements this rule under `balanced_trials_v2`. It validates complete coverage, distinct episode IDs, matching conditions and output hashes, and independently rescores all submitted answers. Refusals, invalid terminal submissions, and budget exhaustion remain zero-valued attempts. Missing/infrastructure-failed trials are rejected. Five trials are required by default; explicitly requesting fewer produces a diagnostic summary without an official score. Fresh identical answers are allowed; copied/resumed episodes are rejected. Sample standard deviations, nominal 95% intervals across repetitions, reliability counts, and paired stripping differences are reported. See [the evaluation protocol](evaluation-protocol.md) for the method, command, and interpretation limits.

## Limits retained from the original design

This remediation implements the original 15 findings in `update.md`, the aggregation correction, component completeness/association, and fixed-suite uncertainty reporting. It does **not** complete every additional proposal in Kimi's later plan. Independent human calibration, a held-out suite, and independent-family RL-data validation remain necessary for broader claims. Complete component credit means recovery of the specifically annotated static components, not proof of successful runtime cryptography.

No paid model comparison, independently reviewed calibration set, or new held-out suite was produced by this change. The smoke test uses scripted answers and is explicitly ineligible as a model result. Public binaries, sources, and answers remain exposed. Do not treat a passing verifier or a non-null official score as an independent validation of generalization, rubric quality, or comparability with historical v3. Partner publication must state these limits, the precise versions, and the evaluation conditions.