Devops, Agile, Method. & TestsTools-in-Action25min
Catching Every Bug: Integrating Testing, Observability, and Determinism. And then what?
This talk introduces autonomous testing, a new approach combining property-based, end-to-end, and observability techniques to test production-like systems deterministically. Using deterministic hypervisors and snapshots, it enables reproducible, system-wide testing that can catch elusive bugs and transform development workflows by restoring confidence and encouraging innovation.
talk.summaryAiDisclaimer
Clément HurlinIndependent Freelancer
We've all encountered some nasty bugs. The worst kind are bugs that only occur in production and rarely. That is why we want to test our applications better, matching the production setups as closely as possible and injecting input that resembles real data. However, if we do that, we often loose determinism, making bugs impossible to reproduce.
In this talk, I will present to you a new technique called autonomous testing. Autonomous testing combines the best of different fields. From property-based testing, it takes automatic generation of user inputs. From end-to-end testing, it takes system-wide testing. It also borrows from observability, by aggregating data from a large number of different runs. The magic of autonomous testing is that it achieves all this, while also preserving determinism.
To achieve this, autonomous testing leverages deterministic hypervisors and OSes. Coupled with the ability to take snapshots of your systems; a deterministic hypervisor enables you to reproduce end-to-end tests (involving multiple systems), to pause, and to rewind time (thanks to snapshots).
I will demonstrate how all these features are combined together to obtain a testing framework that can virtually catch any bug if given enough computing power. Finally I will conclude with a discussion how the ability to catch every bug changes development workflows: once you are in this high confidence state, you can start taking risk again!
In this talk, I will present to you a new technique called autonomous testing. Autonomous testing combines the best of different fields. From property-based testing, it takes automatic generation of user inputs. From end-to-end testing, it takes system-wide testing. It also borrows from observability, by aggregating data from a large number of different runs. The magic of autonomous testing is that it achieves all this, while also preserving determinism.
To achieve this, autonomous testing leverages deterministic hypervisors and OSes. Coupled with the ability to take snapshots of your systems; a deterministic hypervisor enables you to reproduce end-to-end tests (involving multiple systems), to pause, and to rewind time (thanks to snapshots).
I will demonstrate how all these features are combined together to obtain a testing framework that can virtually catch any bug if given enough computing power. Finally I will conclude with a discussion how the ability to catch every bug changes development workflows: once you are in this high confidence state, you can start taking risk again!
Clément Hurlin
Clément is a French software engineer and technical leader based in Nancy, France. He began his career with a Ph.D. thesis on the verification of object-oriented programs using linear logic. Following his academic work, Clément joined Prove & Run, where he co-authored a compiler and prover for a C-like functional programming language. This experience marked the beginning of his journey as a software engineering leader, inspiring teams and establishing effective processes.
After Prove & Run, Clément joined Tweag where he contributed as a software engineer and technical lead on fintech and healthcare accounts, using a variety of techs, including functional programming, Python data pipelines, and static analyzers. Clément served as the Director of the Build Systems Department at Tweag until March. In this role, he oversaw the growth of software engineers, supported tech leads on client projects, and contributed to company strategy, hiring, and sales activities. In parallel, he worked part-time as a Staff Software Engineer for selected clients.
Lastly Clément joined as the Engineering lead for a startup doing static analysis of builds, in order to automatically parallelize them.
After Prove & Run, Clément joined Tweag where he contributed as a software engineer and technical lead on fintech and healthcare accounts, using a variety of techs, including functional programming, Python data pipelines, and static analyzers. Clément served as the Director of the Build Systems Department at Tweag until March. In this role, he oversaw the growth of software engineers, supported tech leads on client projects, and contributed to company strategy, hiring, and sales activities. In parallel, he worked part-time as a Staff Software Engineer for selected clients.
Lastly Clément joined as the Engineering lead for a startup doing static analysis of builds, in order to automatically parallelize them.