Testing Challenges in the Age of AI

AI apps break QA: non‑determinism, latency, rate limits and schema violations make assertion tests flaky, turning CIs red and eroding confidence. Security/privacy add complexity: data must be anonymized, prompts sanitized, leaks prevented—areas beyond standard scans. Real users submit jailbreaks, informal queries and RAG patterns vary between staging and production. Agent workflows worsen this: multi‑step tool orchestration with retries, approvals and inter‑agent coordination exceed unit or integration scopes.
This session presents a layered testing strategy: fast, mock‑based tests with (mokksy.dev); behavioral tests validating tool calls and security; and live‑model sanity checks with LLMs as evaluation and assertion mechanisms (promptfoo). Examples using JetBrains Koog agentic framework demonstrate these patterns i practice.
Attendees will gain a clear map of AI testing pitfalls and actionable techniques to balance development speed with quality in unpredictable environments.