Agentic Engineering & ToolingConference50min
Run Frontier Open Models on consumer hardware
A deep dive into fast, production-grade on-device inference on Macs using the MLX ecosystem. Covers unified memory, continuous batching, KV-cache compression, speculative decoding, and model porting pitfalls, with real benchmark results and engineering tradeoffs for balancing accuracy, latency, and memory on consumer hardware.
talk.summaryAiDisclaimer
Prince CanumaNeywa Labs
talkDetail.whenAndWhere
Wednesday, October 7, 14:00-14:50
TBA 2
talks.roomOccupancytalks.noOccupancyInfo
Getting a model to run on a Mac is easy. Getting it to run fast — with production-grade throughput, stable memory behavior, and accuracy that survives optimization — is where the real engineering lives. This session goes deep into the inference stack behind the MLX ecosystem (mlx-vlm, mlx-audio, nativ — 5M+ downloads).
We'll work through the full pipeline with measured numbers at every stage:
Unified memory in practice — why the architecture that makes 512GB of "VRAM" possible also changes how you batch, allocate, and evict; and where memory bandwidth, not compute, becomes the wall across M-series generations.
Continuous batching without a scheduler's luxury — handling concurrent requests when the same RAM serves the OS, the app, and the model.
KV-cache compression and quantization — which techniques hold accuracy on real long-context workloads versus which only look good in benchmarks, with results from MATH500 and LongBench evaluations.
Speculative decoding across chip generations — when draft models pay off and when the overhead eats the win.
Porting war stories — bugs from porting 1000+ models, and the testing methodology that now catches them.
You'll leave able to reason about the accuracy/latency/memory triangle on consumer hardware — and knowing which server-side habits to unlearn.
For engineers building or deploying on-device inference. Familiarity with transformer inference assumed.
We'll work through the full pipeline with measured numbers at every stage:
Unified memory in practice — why the architecture that makes 512GB of "VRAM" possible also changes how you batch, allocate, and evict; and where memory bandwidth, not compute, becomes the wall across M-series generations.
Continuous batching without a scheduler's luxury — handling concurrent requests when the same RAM serves the OS, the app, and the model.
KV-cache compression and quantization — which techniques hold accuracy on real long-context workloads versus which only look good in benchmarks, with results from MATH500 and LongBench evaluations.
Speculative decoding across chip generations — when draft models pay off and when the overhead eats the win.
Porting war stories — bugs from porting 1000+ models, and the testing methodology that now catches them.
You'll leave able to reason about the accuracy/latency/memory triangle on consumer hardware — and knowing which server-side habits to unlearn.
For engineers building or deploying on-device inference. Familiarity with transformer inference assumed.
Prince Canuma
Prince Canuma is Founder and CEO of Neywa Labs, building the inference stack for Apple Silicon. He is the creator and primary maintainer of the MLX open-source ecosystem — mlx-vlm, mlx-audio, and mlx-embeddings — with over 4M combined downloads, foundational to on-device AI on Apple hardware. He previously led developer experience at Neptune AI and worked at Arcee on post-training and inference optimization. His work focuses on model optimization and making AI accessible on local hardware.