Apache Spark Declarative Pipelines in Action: Lessons from Processing Live Aircraft Data at Global Scale

Apache Spark 4.1, released in December 2025, introduces Spark Declarative Pipelines (SDP), a paradigm shift that transforms how data engineers design and maintain complex data pipelines. This hands-on session provides a critical review of SDP's design decisions and comprehensive introduction, demonstrating how declarative pipelines replace traditional imperative Spark code.
We'll build a live data pipeline processing real avionics data streamed from tens of thousands of aircraft—from tiny Cessnas to massive Airbus A380s—flying overhead right now. This demo uses an open-sourced PySpark data source I developed with OpenSky founders from Oxford and ETH Zurich, enabling direct access to live ADS-B flight telemetry.

We'll create pipelines using SDP's streaming tables for ingestion and materialized views for transformations—all in declarative configuration. No complex "glue code"—just define what your pipeline should accomplish. The demo runs on a forever-free Lakehouse account implementing all OSS lakehouse foundations: table formats (Iceberg and Delta), governance with Unity Catalog, and Apache Spark with SDP.

As icing on the cake, I'll show AI integration for NLP queries against live IoT data. Attendees can replicate everything at no cost.
You'll leave with practical SDP knowledge and best practices for pipeline modernization. WARNING: After this breakout session, you may catch the virus of tracking live aircraft data—don't worry, the data pipeline won't be your bottleneck!