Evaluating LLM Features: Knowing They Work and Keeping Them Working
Building a golden dataset from real usage, deterministic checks first, LLM-as-judge with rubrics and its biases, pairwise comparison, running evals in CI on prompt or model changes, tracking cost and latency alongside quality, online signals, tracing, and how to avoid overfitting your eval set.
Systems & architecture8 min