AI & Technical question

A frontier lab says its model scores well on public benchmarks but still fails on long-horizon, repo-scale engineering tasks. How would you design a contamination-resistant evaluation or RL environment that surfaces those failures, while preserving reproducibility, trustworthy reward signals, and automated code-correctness verification?

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Ability to design a rigorous, hard to game evaluation and training environment for long horizon coding tasks.

Design tasks sourced from real, recent repository history, using issues and their actual merged fixes as ground truth, rather than easily memorized public benchmark problems.
Prevent contamination by continuously rotating in newer repository snapshots and issues past the model's training cutoff, and by structurally transforming problems so they cannot be pattern matched from memorized training data.
Ensure reproducibility by fully containerizing the repository environment and dependencies for each task, so results are stable across runs and not dependent on external, changing state.
Build the reward signal around automated, objective verification, running the actual test suite or a constructed set of checks against the repository, rather than a model graded or subjective score that can be gamed.
Extend coverage to long horizon, multi step tasks by requiring the agent to make a sequence of changes across multiple files and pass integration tests, not just a single isolated function fix.
Validate the environment itself periodically by checking whether reward hacking is occurring, for example an agent passing tests without a genuinely correct fix, and patching the verification logic accordingly.

Sources tasks from real, recent, rotating repository data specifically to resist contamination, rather than relying on a static public dataset.
Grounds the reward signal in automated test execution, an objective, hard to game verification method, instead of a subjective model grade.
Addresses reproducibility explicitly through environment containerization, a common real failure point in long horizon coding evals.
Includes a process to detect and patch reward hacking, showing awareness that agents can exploit a flawed verification system.

Relying on a static, well known dataset that is likely already contaminated in frontier model training data.
Using a subjective or model graded reward signal that is easier for an agent to game than genuine correctness.

How would you detect if an agent was gaming the automated verification rather than solving the task correctly?
How often would you need to refresh the task set to stay ahead of contamination?