Building an eval suite for a system that shipped without one
Most AI systems we audit were shipped on vibes: someone tried a few prompts, it looked fine, it went live. Here is how we retrofit real evaluation onto a running system without pausing the roadmap.