CURRENT REPLICATION RESULTS

Updated: 2026-11-03T12:22:31.084318+00:00

The user authorized departures from the original unpublished components. Original prompts are retained; Experiment 6 has disclosed opponent-description corrections with exact diffs saved. Fresh responses are frozen before scoring. These are new measurements, not copied author scores. Early Experiment1/4 attempts containing unintended local Codex context were quarantined in replication-invalid-initial and excluded. The measurements below use the subsequent audited clean-context runs; no scored policy was revised after seeing its evaluation.

Experiments 1 and 2
All four game families matched the released labels: Breakout, Pong, Pac-Man and road-crossing. For the Freeway case, the fresh answer explicitly named Frogger/Freeway and favored a simplified Frogger, so this is a family-level match rather than exact title identification. All three shape categories were correct: circle, square/rectangle and triangle. All three shape answers incorrectly described hollow outlines instead of filled shapes. These are single valid fresh replies per case, without selecting among valid replies for better scores.

Experiment 3 (complete)
All six fresh conversations completed: corpus identification 3/6, unshuffled token order 6/6, and exact copy-head sets 5/6 at the released induction threshold >0.2. Both English runs and one Python run identified their corpus; both romaji runs and the other Python run guessed English. The other Python run added false copy head L3H0; all other copy-head sets were exact.
Independent audit reparsed all frozen answers and released controls and checked 723 artifact hashes, 56 CLI turns, 50 Python executions and manual approvals, exact original prompts, every feedback-to-next-prompt chain, same-thread settings, and aggregate/CSV scores. One actual SVD nonconvergence error in English r1 was returned unchanged to the reader and retained; there were no timeouts or truncated tool outputs. This audit verifies recorded executions and does not rerun the Python analyses.
Results and audit: reconstructed-exp3/scores.json and reconstructed-exp3/independent-review.json.

Experiment 4: Kinetix car repair
Policy | Search successes/256 | Test successes/1024 | Test mean length
base | 0/256 | 0/1024 | 256.00000
published_edit | 256/256 | 1024/1024 | 73.00000
fresh_edit | 0/256 | 0/1024 | 33.15625

Fresh car reply: all search and test episodes ended in red collisions. The single audited repair reply was scored as written, without subsequent policy tuning.
Controls: the released edit retains 100% success and 73-step episodes; the rounded base retains 0% success.

Experiment 5: MinAtar, mean score +/- standard error, 1000 episodes per policy
Game | Author archived written-policy mean | Released-policy control rerun | Fresh written policy
breakout | 517.411 | 496.270 +/- 7.817 | 380.585 +/- 6.176
seaquest | 53.110 | 51.533 +/- 1.348 | 35.245 +/- 0.842
asterix | 53.178 | 52.202 +/- 0.992 | 64.378 +/- 0.942

Asterix improved over the rerun released policy; fresh Breakout and Seaquest scored lower.
Released written-policy weights were reused for the control evaluations above. The user has now excluded training game networks from scratch. The optional 32-run reconstructed PPO comparison is cancelled before launch; no full PPO training or fresh PPO evaluation started. Its source and preflight remain historical artifacts. Missing original PPO baseline comparisons are not claimed.

Validation and limits
Independent review recomputed statistics from all 3840 car episode records and 6000 MinAtar episode records and verified hashes, unique IDs, seeds and aggregates. It did not rerun these episodes through a second simulator.
MinAtar parser recreates the original float32 tensor bytes. Car parser recreates every numerical float64 value; 3145 signed-zero bits differ because the text prompt does not preserve negative zero.
Car uses pinned Kinetix and jax2d, reconstructed wrapper, new fixed RNG streams, and Python3.12/JAX0.9 rather than original Python3.10/JAX0.4.38.
MinAtar1.0.15 sources match the original prompts exactly. Evaluation uses documented new per-episode seeds, fresh environments, last_action=0, float32 network arithmetic and float64 softmax.
Each fresh reply was evaluated once as written; no policy updates were made after scores.

Files
reconstructed-exp4/README.md: car protocol and artifacts
reconstructed-exp4/results.json: car aggregates
reconstructed-exp5/summary.json: MinAtar protocol and aggregates
reconstructed-exp4/independent-review.json and reconstructed-exp5/independent-review.json: independent audit

Experiment 6 (all four budgets complete and audited with substitute opponents)
The zero-feedback writer finished in one Python run and two model turns. Its frozen policy and the released zero-pass control were evaluated on 2000 games per opponent, with equal black/white counts and paired episode keys.
Policy | PPO seed0 | PPO seed2 | Random | Mean across PPO opponents
Fresh | 18.95% | 20.05% | 94.10% | 19.50%
Released control | 42.00% | 38.55% | 99.70% | 40.275%
The fresh policy performed worse than the released control. All 12,000 episode records, aggregates, artifact hashes and action/terminal consistency checks passed independent review. This audit does not replay every board through a second engine; separate rule and trace-replay tests cover the reconstructed engine. These use released same-size PPO seeds0/2 as substitute held-out opponents; the original small PPO/REINFORCE comparisons are not reproduced. The ten-pass writer finished after 28 CLI turns, 17 reviewed Python runs and all 10 practice passes. Its frozen policy and the matching published ten-pass control were scored on another 12,000 paired games.
Policy | PPO seed0 | PPO seed2 | Random | Mean across PPO opponents
Fresh ten-pass | 68.65% | 70.10% | 99.95% | 69.375%
Released ten-pass control | 42.00% | 57.65% | 99.60% | 49.825%
The fresh ten-pass policy outperformed the released control by 19.55 percentage points on the main score. Independent audit recomputed all 12,000 episode aggregates and verified complete IDs, colors, actions/terminal consistency, paired keys and tensor/source hashes. Practice transcripts, per-pass snapshots and actual feedback arrays are retained separately from scoring. The twenty-five-pass writer completed 55 CLI turns, 29 reviewed Python runs and all 25 practice passes. Its frozen policy and matching released control were scored on 12,000 paired games.
Policy | PPO seed0 | PPO seed2 | Random | Mean across PPO opponents
Fresh twenty-five-pass | 68.75% | 70.75% | 99.70% | 69.75%
Released twenty-five-pass control | 46.15% | 49.75% | 99.10% | 47.95%
The fresh twenty-five-pass policy exceeded its matching released control by 21.8 percentage points on the main score. All 12,000 records passed independent aggregate, ID, color, action/terminal, paired-key and hash checks. The independent fifty-pass writer finished after 92 CLI turns, 41 reviewed Python runs and all 50 practice passes, with no execution failures. Its frozen policy and matching released control were scored on another 12,000 paired games.
Policy | PPO seed0 | PPO seed2 | Random | Mean across PPO opponents
Fresh fifty-pass | 85.50% | 84.75% | 99.75% | 85.125%
Released fifty-pass control | 73.40% | 81.05% | 99.95% | 77.225%
The fresh fifty-pass policy exceeded its matching control by 7.9 percentage points. All 12,000 episode records passed independent aggregate, ID, color, action/terminal, paired-key and hash checks. The complete writer archive passed transcript/feedback and manual-code review; root independently verified all 2,425 archived file hashes and all three final tensor copies. Across all four budgets, 48,000 held-out games are now scored and audited. These are independent single writer trials per budget; the results do not isolate a causal effect of feedback budget. The disclosed substitute-opponent limitation applies throughout.

Experiment 7 checkpoint reconstruction (complete; readers ongoing)
Six new fine-tunes were trained with unchanged published training code, exact dataset/splits and pinned Gemma base revision ac82b4e820549b854eebf28ce6dedaf9fdfa17b3. All nine behavioral evaluations and both A/B release sets are complete. These are newly trained checkpoint bytes, not the authors' unavailable originals.
Pair | Honest wrong-answer preference | Sycophant wrong-answer preference
100 | 2.08% | 100%
101 | 0% | 100%
102 | 3.42% | 100%
These percentages compare teacher-forced total log-probabilities of the wrong and correct candidate answers on facts each model knows in plain-question evaluation. They are not free-generation accuracy. All six frozen embeddings match the base, and all 12 release payloads match their assigned fine-tunes. Artifact/provenance checks passed; the source evaluator does not save every per-fact log-probability for independent recomputation.
Results, logs and hashes: reconstructed-exp7/README.md; native weights remain on Spark.
Ten of fifteen readers are complete and independently audited. Six of seven completed weights-only trials are explicit 50/50 forced ties: all three round-1 pairs, round-2 pair 102 and round-3 pairs 101/102. The seventh, round-4 pair 101, chose incorrectly at 52% confidence (48% assigned to the correct model). This is a near-chance choice, not a forced tie. Arbitrary binary guesses are not evidence of identification.
All three attention-assisted trials correctly identified the sycophantic model, assigning the correct choice probabilities 97%, 90% and 80% (mean 89%). They used respectively 12 Python analyses plus 3 attention calls, 14 plus 6, and 11 plus 3. These are three trials on newly retrained model pairs; the component rankings remain hypotheses pending causal intervention.
Independent audit verified 188 CLI turns, 126 Python executions, 12 attention calls and 384 saved attention arrays. Three Python errors and eight truncated outputs remain in their exact feedback chains. The review-record bookkeeping correction and retained-response recovery matched the original code/messages and actual CLI rollouts; no valid response was regenerated to improve results. Five weights-only trials remain. Codex account usage is now exhausted: the spare-slot recovery stopped on a usage rejection, and the original driver is suspended with its retained turn-30 code approved but unexecuted. No response was regenerated.
Causal baseline checks passed for all three pairs: all 878 per-fact records passed independent consistency/hash/aggregate audit; 36 candidate-answer comparisons (first 3 facts per model) matched the unchanged author helper exactly. No selection-dependent patches have run yet.

Remaining work
Experiment 6 is complete across budgets 0/10/25/50, with all 48,000 held-out game records audited. It uses disclosed substitute opponents because the original small baselines are unavailable.
Experiment 8 runtime reconstruction and startup review passed. Both pinned models passed generation, native teacher probability parity, activation capture, real edit/restoration, input validation and kernel recovery tests. Three independent editors (Qwen 10, Qwen 25 and Gemma 5) were started with their exact initial prompts and max reasoning effort; the other four cases are prepared. The Qwen 10 editor subsequently stopped incomplete: Codex returned a provider policy rejection at turn 8, after one of ten passes, with no answer. Its valid context audit and exact failed prompt/events are preserved. Independent incident review verified the actual rollout had no returned assistant answer and all 20 remote scratch/queue files match their snapshots; in-memory kernel variables were not serialized. No prompt was rewritten or response regenerated; Qwen 25 and Gemma 5 completed their first practice passes, then their next CLI turns failed on the account usage limit without returning answers. Their new practice records await independent audit. There is no final Qwen 10 edited model/result from that stopped run. GPU requests share a queue. All ten held-out datasets and seven published controls passed evaluator startup review. The first bounded Gemma base MATH500 shard is complete and independently reviewed. The fixed base-plus-four-published-edits Gemma campaign is now launched: 398 shards and 12,638 total answers, including the verified first 32 reused without regeneration. The remaining 397 shards keep the original 32-answer grouping, sample counts and full token limits; the first new batch is queued, so no full benchmark result is available. The standalone Gemma benchmark campaign continues independently of the Codex account. Both 64-sequence capacity initializations passed actual-report review, and the first fixed 64-answer Qwen benchmark shard is launched; wider Qwen scoring remains gated on its audit. Three of the released 1000 MMLU-Pro questions match published practice text after normalization; these are retained and flagged. The other nine sets have no exact/normalized text overlaps. GPQA Diamond was retrieved from the authors' public archive and matches the pinned Hugging Face blob exactly (198 rows, SHA256 41d1213cd7a4998605a26c2798500652572007161b3a92817ba46b35befcd305). No account action is needed. The original option permutation is unavailable; a deterministic per-ID shuffle is explicitly reconstructed. All seven published Qwen/Gemma edit controls were rebuilt with unchanged author apply_edit.py: all edited tensor hashes match the archive and all unedited tensors match the base bitwise. The first Qwen request completed all 256 generations and activation replays, using one of its ten passes. The unedited practice baseline scored 33/256 (22/128 GSM8K and 11/128 MATH); 224 answers hit the full 32,768-token limit. Complete forward time was 13,618.261 seconds. Exact sample IDs/seeds, tokenizer/grade replay, publication and budget receipts, and all 54,788,096 activation values passed the reviewed audit. These are practice-baseline results, not final edited held-out performance. The first Gemma base MATH500 shard covers 8 questions with 4 samples each: lenient 1/32 and strict 0/32, with 30 normal stops and 2 token-limit stops. Its 32 actual records, frozen source/model/context and tokenizer/grade replay passed independent review; it is not a full benchmark score. An audit JSONL parsing error on Unicode line separators was corrected without altering experimental outputs or running new inference; the original failed audit and source are retained. No fresh final Experiment 8 outcome has yet been scored.
Experiments 1 and 2 fresh prompt results are retained in replication-report.txt, which is a historical report and has an obsolete originals-only limitation section; current-scope.txt and status.json supersede that limitation.
All eight experiments are not yet complete.

Reproduction source snapshot
REPRODUCE.txt and reproduction-source.zip provide the reviewed harness code, original one-shot launcher support, pinned inputs/settings and host-specific setup/recovery instructions. The package passed independent member/source/hash verification. It excludes native model weights, datasets and authentication. It records a source snapshot during ongoing experiments, not completed results or a portable installer. The original Exp8 final-evaluation plan contains 110,910 answers across 3,492 bounded shards. A separately reviewed and deployed runtime proposes 64-sequence generation and 64-answer Qwen shards for untouched cases/families, reducing the planned total to 2,092 shards while preserving all 110,910 answers. Existing editors and every Gemma evaluation retain their original runtime/grouping. The new configuration has passed 40 CPU tests and independent source/deployment review; its separate Qwen and Gemma zero-answer capacity checks passed actual-report review. Only the first bounded Qwen64 benchmark shard has launched under it; no new editor or wider Qwen campaign has started, and no speedup is yet measured. The updated source package includes both runtimes and their review records. Its historical active-case wording predates discovery of the p10c provider rejection; this current report supersedes that run-state snapshot. These are planned counts, not completed measurements.

Current source correction: one valid GPQA record contains U+2028 inside its JSON string. The staged, unstarted 64-answer evaluator and its auditor now use the reviewed physical-line JSONL reader correction. The patch passed independent review and exact deployment verification; current v1 experiments and input bytes/grouping are unaffected. The refreshed source ZIP includes this deployed correction, corrected auditors, and the approved Gemma campaign. Independent verification passed for all 239 members, including 177 Git source files and 14 bootstrap tools. The previous package is preserved under source-snapshots/2026-11-03-e5de714.