Measuring the Unmeasurable: Evaluating Generative AI in Practice

Getting a generative AI prototype to impress a demo audience is easy. Getting it to behave reliably in production is another story. In production, you don't control what users ask or what the model answers. That's not a bug, that's the whole point of using LLMs. But it is also what makes evaluation genuinely hard, and what separates a convincing proof of concept from a trustworthy production system.

This session tackles the core paradox of building with large language models: how do you harness their creativity while maintaining control over their outputs? We will use a RAG-based chatbot as our running example, a real-world case where you have to guarantee output quality without ever controlling what users actually ask, and where linguistic fluency and factual accuracy constantly pull in opposite directions, to show what a complete, reproducible evaluation pipeline actually looks like in practice.

Along the way, we will cover why traditional metrics fall short, how to use LLMs as judges and where that approach quietly fails, when human evaluation is non-negotiable, and how to set up continuous monitoring to catch drift before your users do.

You will leave with practical frameworks and a clear methodology to make evaluation a constant driver of development, from the first prototype to the last production release.