Engineering multimodal AI video pipelines at scale: from zero to hero

Today, a growing number of applications rely on video as a primary data source. Analyzing video at scale requires more than running individual models; it demands well designed multimodal pipelines that combine vision, audio, and text while remaining accurate, cost-efficient, and compliant.

In this session, we build a high-throughput pipeline for video streams. Participants will see how raw feeds are transformed into aligned, queryable components by orchestrating ASR and visual embeddings, and by producing higher-level outputs such as speaker identity, facial recognition, and summaries under noisy conditions.

We focus on key engineering challenges: audio-video synchronization and clock drift, stream fragmentation and context handling, hallucination mitigation, and scaling the system while controlling latency, cost and resilience.

Key takeaways include designing scalable multimodal pipelines, solving alignment and signal-quality issues at scale, and applying practical patterns for building compliant, auditable, and resilient video analysis systems.

Familiarity with data pipelines and basic ML concepts is helpful but not required.