Recursive Language Models Beat Context Rot

A deep dive into recursive language models (RLMs) that avoid the context bottleneck by keeping massive context in an external symbolic workspace. The root LLM acts as an active researcher and manager, writing and running code in a REPL to interrogate the context, delegating subtasks to sub-LLMs, and using tools like searches and regex to prune data. We explore how this context-centric decomposition enables long-horizon reasoning, review dramatic gains on the OolongPairs benchmark (moving from near-zero to as high as 58% F1), and discuss scaling to 10 million tokens, practical costs, and the potential to train models to get better at delegation through reinforcement learning.

Note:  This podcast was AI-generated, and sometimes AI can make mistakes.  Please double-check any critical information.

Sponsored by Embersilk LLC