Durable Long-Running Agentic Query Execution

Redesign an in-memory ask service so agentic queries that may run for hours execute durably, resume after interruptions, finish after the browser or network disconnects, and stream progress to the client. Treat the LLM and search APIs as black boxes. Requirements Begin with an existing ask endpoint backed by an ask service that calls an LLM API and a search API; treat both downstream APIs as black boxes. Support agentic queries that can take hours to finish. Keep a query running to completion when the browser tab closes. Keep execution independent of a temporary loss of the client's internet connection. Make execution durable and resumable instead of holding all query state only in memory. Stream progress while the query is running and allow the client to recover that progress after reconnecting. Notes The central design decisions are where durable execution state lives, how a query resumes without losing or duplicating work, and how the progress stream reconnects to an execution that outlives the original request. Clarify the required delivery semantics for work and progress events, the checkpoint boundary, retention after completion, cancellation behavior, and whether multiple clients may observe the same query. A strong reference design separates the lifetime of the HTTP request from the lifetime of the query: The API authenticates the caller, creates a stable query ID and durable job record in QUEUED state, and returns immediately. A transactional outbox publishes the query ID to the work queue so a crash cannot leave a committed job permanently unqueued. The job record is the source of truth for ownership, status, checkpoint, retry count, cancellation, and terminal result. A durable workflow engine or leased queue assigns work to stateless executors. An executor periodically renews its lease and persists a checkpoint after each recoverable step. A replacement executor may claim an expired lease and resume from the last committed checkpoint. Each external operation carries an idempotency key when the dependency supports one. Otherwise, persist call intent and result around the operation and design for at-least-once execution; exactly-once effects cannot be guaranteed against a black-box dependency that offers no idempotency or reconciliation mechanism. Commit each checkpoint and its progress event together, or use a transactional outbox when the event log is separate. Events carry a stable, monotonically increasing sequence number. An SSE or WebSocket gateway tails the log without owning the query; a reconnecting client supplies its last sequence number, replays missed events, and then resumes the live stream. Completion and cancellation use conditional state transitions so only one executor can publish a terminal result. Retain the result and progress log for a defined period, then expire them according to product and privacy requirements. For a failure walkthrough, suppose an executor crashes after checkpoint 12. Its lease expires, another executor claims the job, reloads checkpoint 12, and retries only work not durably recorded as complete. Re-delivered progress events retain their original sequence numbers, so clients can deduplicate them by query ID and sequence number. If the crash happened during a non-idempotent black-box call, the system must reconcile the saved call intent or expose the ambiguity rather than silently claiming exactly-once behavior. The LLM and search APIs are fixed dependencies, so the design should focus on orchestration, durability, recovery, and client progress rather than their internal implementation. Preparation Draw the request, durable-execution, and progress-delivery paths separately, then explain how their lifetimes differ. Walk through browser closure, network loss, executor interruption, retry, and client reconnection while tracking persisted state at each transition. Define completion, retry, cancellation, and progress-delivery semantics before selecting storage and messaging components.