Byte Size15min
chATLAS: How we built an AI assistant for the ATLAS experiment
This talk shares practical lessons from building a production-level AI assistant using Retrieval Augmented Generation (RAG) for scientific collaborations. It covers each stage of the RAG pipeline—data scraping, embeddings, storage, retrieval improvements, agentic RAG, and benchmarking—highlighting solutions to real-world challenges and enabling others to build similar applications.
Joe EganUniversity College London / ATLAS collaboration
This talk will take the audience along the journey of building a production-level AI assistant for a large scientific collaboration.
The intended audience is scientific collaborations and companies who are interested in building their own RAG applications, and could learn from our experience how to overcome challenges related to putting RAG into production.
It will touch on each stage of the Retrieval Augmented Generation (RAG) pipeline, explaining how it works, some challenges and our experience addressing them.
Talk outline:
The intended audience is scientific collaborations and companies who are interested in building their own RAG applications, and could learn from our experience how to overcome challenges related to putting RAG into production.
It will touch on each stage of the Retrieval Augmented Generation (RAG) pipeline, explaining how it works, some challenges and our experience addressing them.
Talk outline:
- Motivation: The problem of efficient information retrieval in ATLAS
- How retrieval augmented generation (RAG) can solve this.
- Walk through our RAG pipeline, describing solutions to challenges at each stage:
- Scraping documentation: Scheduled scraping pipelines in CI/CD, multimodal LLMs
- Embeddings: Chunking strategies, fine-tuning with synthetic Q&A pairs, mining hard negatives
- Embedding storage: Building a dedicated postgres server for hosting embeddings
- Retrieval improvements: Sparse and dense retrieval, Reciprocal RAG Fusion.
- Agentic RAG: LLM as a judge, second retrieval.
- Benchmarking: Creating a custom benchmark
- MCP: Allowing others to build tools on top of ours
Joe Egan
Hi, my name is Joe. I'm a PhD researcher in High Energy Physics at University College London, working within the CDT in Data Intensive Science and the ATLAS collaboration at CERN. I build tools to increase the efficiency of large experimental collaborations, so that we can learn more about physics!
One such tool is Contur, which repurposes published measurements from ATLAS and CMS to constrain physics beyond the Standard Model. This data re-use allows theorists to directly compare the signatures of their model against data from hundreds of analyses, all without the need for a bespoke search from the LHC experiments. This reduces model to insight time from years to hours.
Another challenge in large scientific collaborations is the amount of internal documentation, often disjointed, spread across multiple forms and difficult to search. To address this problem of efficient information retrieval, we propose chATLAS, an AI assistant for the ATLAS collaboration that works using retrieval augmented generation (RAG). I am the core developer of chATLAS, tasked with scaling the tool from proof-of-concept to production. In 6 months of production, chATLAS has answered 5,000 queries from ATLAS physicists, and was recently endorsed by a group convener as the go-to tool for information retrieval in the collaboration of over 3,000 physicists.
Outside particle physics, I have applied natural language processing and large language models to social science, with a quantitative study of immigration rhetoric in UK politics with The Guardian.
One such tool is Contur, which repurposes published measurements from ATLAS and CMS to constrain physics beyond the Standard Model. This data re-use allows theorists to directly compare the signatures of their model against data from hundreds of analyses, all without the need for a bespoke search from the LHC experiments. This reduces model to insight time from years to hours.
Another challenge in large scientific collaborations is the amount of internal documentation, often disjointed, spread across multiple forms and difficult to search. To address this problem of efficient information retrieval, we propose chATLAS, an AI assistant for the ATLAS collaboration that works using retrieval augmented generation (RAG). I am the core developer of chATLAS, tasked with scaling the tool from proof-of-concept to production. In 6 months of production, chATLAS has answered 5,000 queries from ATLAS physicists, and was recently endorsed by a group convener as the go-to tool for information retrieval in the collaboration of over 3,000 physicists.
Outside particle physics, I have applied natural language processing and large language models to social science, with a quantitative study of immigration rhetoric in UK politics with The Guardian.
comments.speakerNotEnabledComments