Language Models Need Sleep

A new study proposes a 'sleep' mechanism for large language models where they periodically consolidate recent context into persistent fast weights during offline passes to improve performance on long-horizon tasks like math reasoning and graph retrieval.