Models tell you what to discard
Systems and Performance
Machine Learning
Optimization
Read transcript · Original paper
AI-generated voices discussing a paper selected by Arjun Srivastava.
About this paper
This paper introduces FastGen, a novel method that uses lightweight model profiling and adaptive key-value caching to significantly reduce memory footprint without noticeable quality loss.