 |
Deep PapersAuthor: Arize AI
Deep Papers is a podcast series featuring deep dives on todays most important AI papers and research. Hosted by Arize AI founders and engineers, each episode profiles the people and techniques behind cutting-edge breakthroughs in machine learning. Language: en-us Genres: Mathematics, Science, Technology Contact email: Get it Feed URL: Get it iTunes ID: Get it |
Listen Now...
Sleep-time Compute: Beyond Inference Scaling at Test-time
Friday, 2 May, 2025
What if your LLM could think ahead—preparing answers before questions are even asked?In this week's paper read, we dive into a groundbreaking new paper from researchers at Letta, introducing sleep-time compute: a novel technique that lets models do their heavy lifting offline, well before the user query arrives. By predicting likely questions and precomputing key reasoning steps, sleep-time compute dramatically reduces test-time latency and cost—without sacrificing performance.We explore new benchmarks—Stateful GSM-Symbolic, Stateful AIME, and the multi-query extension of GSM—that show up to 5x lower compute at inference, 2.5x lower cost per query, and up to 18% higher accuracy when scaled.You’ll also see how this method applies to realistic agent use cases and what makes it most effective.If you care about LLM efficiency, scalability, or cutting-edge research.Explore more AI research, or sign up to hear the next session live: arize.com/ai-research-papersLearn more about AI observability and evaluation, join the Arize AI Slack community or get the latest on LinkedIn and X.
