When benchmarks go bad - what we learned from measuring performance wrong

This talk explores common performance measurement mistakes, cognitive biases, and misleading intuitions. Franz and Holly explain why performance testing matters, how it can go wrong, and offer practical guidance for improving application performance, including test system setup, load generators, and the USE method.

talk.summaryAiDisclaimer

Francesco NigroIBM

Holly CumminsIBM

talks.description

The world of performance analysis is littered with flawed claims, cognitive biases, dangerous intuitions, and beguiling fallacies. Both Holly and Franz have been guilty of all of the above. Repeatedly. Especially Holly. But this is a no-judgement zone. Some measurement anti-patterns are subtle, and some are downright counter-intuitive. In this talk, Franz and Holly will explain why measuring performance is important, and talk through some of the ways it can go wrong. That would be depressing if that was all there was, so they’ll also introduce a toolbox of questions and principles that you can use to improve the performance of your own applications.
These include:

How to set up a test system
Recommended load generators
The USE method

performance

biases

measurement

load testing

talks.speakers

Francesco Nigro

IBM

Italy

I've been working for many years in the computer field. In the last 15 years I've cultivated a strong passion in Java development and the under the hood details of OpenJDK, C and (x86) Assembly.
Big fan of DDD (Domain Driven Design) world, I've developed several Event-Sourcing high performance solutions in the medical and IoT field.
I'm an active member of various online communities on performance (https://groups.google.com/forum/#!forum/mechanical-sympathy), Senior Principal (Software) Performance Engineer and Performance Lead for Red Hat/IBM on Quarkus, Red Hat Top Inventor (2019).
I've collaborated to different projects related high-performance computing both as committer and contributors eg Quarkus, Vert-x, Netty committer, JCTools author, PMC of ActiveMQ Apache Artemis (Messaging Broker), HdrHistogram, JGroups-raft, ...

Holly Cummins

IBM

United Kingdom

Holly Cummins is a Senior Technical Staff Member on the IBM Quarkus team and a Java Champion. Over her career, Holly has been a full-stack javascript developer, a build architect, a client-facing consultant, a JVM performance engineer, and an innovation leader. Holly has led projects to understand climate risks, count fish, help a blind athlete run ultra-marathons in the desert solo, and invent stories (although not at all the same time). She gets worked up about sustainability, technical empathy, extreme programming, the importance of proper testing, and automating all the things. You can find her at https://hollycummins.com, or follow her on socials at @holly_cummins.

Holly Cummins is a Senior Principal Software Engineer on the Red Hat Quarkus team and a Java Champion. Over her career, Holly has been a full-stack javascript developer, a build architect, a client-facing consultant, a JVM performance engineer, and an innovation leader. Holly has led projects to understand climate risks, count fish, help a blind athlete run ultra-marathons in the desert solo, and invent stories (although not at all the same time). She gets worked up about sustainability, technical empathy, extreme programming, the importance of proper testing, and automating all the things. You can find her at http://hollycummins.com, or follow her on socials at @holly_cummins.