Evaluate Subjective, Nondeterministic Agent Outputs

Design an evaluation approach for an agent whose outputs are subjective and nondeterministic. The interviewer challenges LLM-as-judge and user-feedback proposals with evaluator overconfidence, missing feedback, and developer bias, then asks how online user signals can close the gap. Requirements Explain how to evaluate an agent whose output is both subjective and nondeterministic. Address three explicit challenges to the evaluation plan: An LLM used as a judge may be overconfident. Users may not provide explicit feedback. Developer-authored evaluation criteria may encode developer bias. Discuss whether online user signals can substitute for missing direct feedback, and identify what those signals would establish about output quality. Notes The discussion is intentionally adversarial: naming user feedback or LLM-as-judge is only the starting point, not a complete answer. The interviewer pushes on the trustworthiness and availability of each evaluator, so keep separate the quality signal, its source, and its failure modes. The candidate proposed online user signals after the explicit-feedback and LLM-judge objections, but the round moved on without confirming a preferred answer. Preparation Rehearse a structured evaluation plan for subjective, nondeterministic outputs, then defend it against all three stated objections. Prepare concrete examples of online behavior signals and explain what each can and cannot prove about agent quality. Practise separating evaluator confidence from evaluator correctness when discussing an LLM judge.