Beyond assertEquals: Probabilistic Security and Bias Testing for Java LLM Applications

The LLM scanners like Augustus and Garak are established tools for security evaluation of LLM-based applications, as they are scanning by running through a fixed sample of adversarial attack data, and evaluating a response. However, LLMs are non-deterministic by design, meaning that each input can generate a different response. It is not only non-determinism, but also the vast linguistic attack surface and the wide variety of threats that demonstrate that traditional Unit testing methods are insufficient. This talk proposes a Java tool for security and safety testing for LLM-based applications using probabilistic testing methods, and security contracts. In contrast to the traditional scanners, Tiberius fits naturally into a standard Java test suite, and integrates with the probabilistic testing tool PUnit to support multi-trial scanning and statistical assertions. Different adversarial probes can be integrated and tested directly in Java code, e.g., application specific adversarial attacks, like prompt injections and jailbreaks. A scan (multi-) run can execute 200+ attack probes against a deployed model and serializes the results into versioned json test-fixtures, which can be used in downstream regression tests. The tool furthermore includes a fingerprinting capability, and detects systemic bias, as even system prompts and guardrails can be biased. The audience will leave knowing Java-native LLM security and safety testing techniques.