AI & Technical question
Enterprise customers report that North agents lose track of objectives on long-running tasks as context accumulates. How would you choose among progressive tool disclosure, context summarization/compaction, persistent filesystem offloading, and trajectory instrumentation, and what metrics would tell you those changes actually improved long-horizon performance?
- Cohere
- AI & Technical
- Hard
Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.
Start a mock interview on this question · Mock interview from a job description
What this question tests
Whether you can turn a vague agent reliability complaint into a diagnosis and a sequenced technical plan, and whether you know how to measure long-horizon agent quality rather than just task completion.
How to approach it
- Pin down what losing track means here: drifting off the goal, repeating tool calls, or forgetting earlier constraints as turns pile up.
- Add trajectory instrumentation first, since you need visibility into real failure sequences before choosing among the other three fixes.
- Use that data to match technique to failure mode: progressive disclosure for early context bloat, compaction for long sessions, filesystem offloading for state that must survive beyond one context window.
- Sequence compaction next as the cheapest reliability win, then disclosure and offloading for the hardest enterprise workflows.
- Roll out on the worst-performing account segment behind a flag and compare against the current baseline on the same tasks.
What a strong answer includes
- Ties each technique to a specific failure mode instead of listing all four as generically good.
- Defines a concrete metric beyond task completion: objective recall, scored by comparing the agent's final action to the original instruction, since a task can finish while drifting from the goal.
- Names guardrails: tool-call efficiency versus a human baseline, and latency or cost added by each technique.
- Flags that compaction risks losing information the agent needs later, while filesystem offloading adds retrieval and security risk.
Common mistakes
- Picks one technique without diagnosing which failure mode is actually happening.
- Never defines how success is measured, leaving the answer as architecture opinion only.
Likely follow-up questions
- How would you validate an LLM judge that scores objective drift.
- What would you do if compaction and filesystem offloading trade off against each other on cost.
More ai & technical questions
- You have two quarters to make North agents production-ready for long, multi-step enterprise workflows. What would you ship first in the MVP of the execution layer, tool orchestration, parallel execution, sub-agent delegation, sandboxed code execution, or failure recovery, and how would you justify the tradeoffs between capability, reliability, and security-first enterprise requirements?Cohere · AI & Technical · Hard
- North engineering wants to move quickly on new harness capabilities, while Modeling needs proof that those design choices help rather than constrain model behavior. What operating process would you set up so harness proposals are validated with Modeling before implementation, evals are shared across both teams, and regressions can be diagnosed as model gaps versus scaffolding gaps?Cohere · AI & Technical · Hard
- Design an evaluation framework for North agents that measures enterprise task completion, long-horizon reliability, and failure recovery across tools and sub-agents, while remaining compatible with both the product harness and model training infrastructure. What would you include, how would you score it, and how would you avoid overfitting the evals to the current harness?Cohere · AI & Technical · Hard
- North can adopt parts of an external agent/orchestration framework or build them in-house. What decision criteria would you use, and how would compliance, auditability, multi-tenancy, restricted or air-gapped deployments, and vendor lock-in affect your recommendation?Cohere · AI & Technical · Hard
- Design North’s third-party integrations experience end to end: connector framework, APIs, SDKs, plugin model, docs, and review lifecycle. How would you optimize for fast time-to-first-integration for partners and customers while preserving enterprise-grade security, identity control, and governance?Cohere · AI & Technical · Hard
- North runs inside a customer’s own infrastructure and positions itself as security-first enterprise AI. How should that deployment model change your integration product decisions, for example connector execution model, credential handling, least-privilege permissions, auditability, tool access, and which partners or categories you support first?Cohere · AI & Technical · Hard
More questions from Cohere
Learn the skill behind it
Chapters of the AI PM course that teach what this question tests.
- Chapter 1: Foundations: the model and the decisions it forces on you
- Chapter 8: Evals: define good and make the number defensible
- Chapter 6: Agents and agentic architecture