AI & Technical question

Design a pipeline for file processing to ensure completeness and reliability.

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Data pipeline design focused on the specific asked properties, completeness and reliability, not just throughput.

Clarify what kind of files are processed, documents, images, or logs, and what processing means, validation, transformation, or indexing.
Propose a durable ingestion queue so incoming files are never lost even if downstream processing is temporarily down.
Design for completeness: track every file through explicit states, received, processing, completed, failed, in a status store so nothing silently disappears.
Design for reliability: retries with exponential backoff for transient failures, and a dead-letter queue for files that repeatedly fail, surfaced for manual review.
Add idempotency: ensure re-processing a file after a retry doesn't produce duplicate or corrupted output.
Define success as the percent of files reaching a terminal completed state within SLA, and mean time to detect and resolve stuck files.

Directly addresses both named properties, completeness via explicit state tracking, reliability via retries and a dead-letter queue.
Names idempotency explicitly, a subtle but critical correctness requirement for any retryable pipeline.
Proposes a measurable SLA-based success metric tied to the two properties actually asked about.

Describing a generic pipeline, ingest, transform, store, without addressing how completeness and reliability are actually guaranteed.
Ignoring idempotency, leading to duplicate or corrupted processing on retries.

Chapters of the AI PM course that teach what this question tests.