Java Language & PlatformConference50min
Building LLM Inference Engines in Java with TornadoVM and the NVIDIA Ecosystem
The session shows how Java can run GPU-accelerated local LLMs natively with TornadoVM, using NVIDIA libraries without JNI. It covers GPULlama3.java, quantized inference, Quarkus and LangChain4J integration, and how TornadoVM plus Project Babylon/Panama position Java as a strong AI and HPC platform.
talk.summaryAiDisclaimer
Thanos StratikopoulosTornadoVM
Michalis PapadimitriouNeo4j
Orion PapadakisTornadoVM
As Java enters the AI era, developers are looking for ways to run local large language models (LLMs) without leaving the JVM. This session presents lessons learned from building GPU-accelerated LLM inference engines natively in Java using TornadoVM, an open-source framework that transparently offloads Java programs to GPUs and other accelerators.
Beyond generating optimized CUDA code directly from Java, TornadoVM now provides seamless access to the NVIDIA software ecosystem, enabling Java applications to leverage libraries such as cuBLAS, cuDNN, cuFFT, and CUDA runtime APIs without requiring JNI or manually written native code. This allows developers to combine high-level Java programming with the performance of NVIDIA's AI and HPC libraries.
Using GPULlama3.java, a lightweight GPU-accelerated inference engine for Llama models, the session explores techniques for supporting quantized data types (FP16, Q8, and Q4), integrating with Quarkus and LangChain4J, and optimizing LLM inference directly in Java. Along the way, attendees will see how modern Java can target GPUs while interoperating with the broader NVIDIA ecosystem.
The session concludes by discussing how TornadoVM complements emerging JVM technologies such as Project Babylon and Project Panama, and how these efforts are helping establish Java as a first-class platform for AI, accelerated computing, and high-performance machine learning.
Beyond generating optimized CUDA code directly from Java, TornadoVM now provides seamless access to the NVIDIA software ecosystem, enabling Java applications to leverage libraries such as cuBLAS, cuDNN, cuFFT, and CUDA runtime APIs without requiring JNI or manually written native code. This allows developers to combine high-level Java programming with the performance of NVIDIA's AI and HPC libraries.
Using GPULlama3.java, a lightweight GPU-accelerated inference engine for Llama models, the session explores techniques for supporting quantized data types (FP16, Q8, and Q4), integrating with Quarkus and LangChain4J, and optimizing LLM inference directly in Java. Along the way, attendees will see how modern Java can target GPUs while interoperating with the broader NVIDIA ecosystem.
The session concludes by discussing how TornadoVM complements emerging JVM technologies such as Project Babylon and Project Panama, and how these efforts are helping establish Java as a first-class platform for AI, accelerated computing, and high-performance machine learning.
Thanos Stratikopoulos
Dr. Athanasios Stratikopoulos (male) is a Research Fellow at the University of Manchester with specialization on heterogeneous architectures and reconfigurable accelerators. He has authored more than 20 research articles in the field of hardware acceleration, system software and programming languages. Currently his work involves heterogeneous architectures ranging from low-power devices to high-end cloud deployments. He is one of the lead developers of TornadoVM and has been part of the team for the last eight years. In addition to his core contributions to the system's technical development, Dr. Stratikopoulos leads the project's communication and dissemination efforts, helping to articulate its goals and advancements to both academic and industrial audiences through talks, documentation, and outreach activities.
Michalis Papadimitriou
Michalis Papadimitriou is a Senior Software Engineer at Neo4j and a Research Fellow at the University of Manchester. His core expertise includes open-source software development, hardware abstractions for high-level programming languages, compiler optimizations for GPU computing, and enabling large language model (LLM) inference on GPUs for the Java Virtual Machine (JVM).
He is focused on advancing high-performance AI and graph computing systems, with particular interest in GPU acceleration for machine learning workloads on the JVM through the TornadoVM framework, and continues to actively maintain the open-source GPULlama3.java project.
Before joining Neo4j, Michalis was a Research Fellow at the University of Manchester, where he contributed to the TornadoVM project as a Staff Software Engineer. Prior to that, he worked across a range of software stacks at Huawei Technologies and contributed to the open-source machine learning compiler Apache TVM while at OctoAI (formerly OctoML), which was later acquired by NVIDIA.
He is focused on advancing high-performance AI and graph computing systems, with particular interest in GPU acceleration for machine learning workloads on the JVM through the TornadoVM framework, and continues to actively maintain the open-source GPULlama3.java project.
Before joining Neo4j, Michalis was a Research Fellow at the University of Manchester, where he contributed to the TornadoVM project as a Staff Software Engineer. Prior to that, he worked across a range of software stacks at Huawei Technologies and contributed to the open-source machine learning compiler Apache TVM while at OctoAI (formerly OctoML), which was later acquired by NVIDIA.