Every time I'm not reading, I'm thinking about reading.
Title | Category | Medium | Tags |
|---|---|---|---|
This is more formally defined below.
autocurricula (def): a machine learning paradigm where AI agents (usually in multi-agent environments) automatically generate increasingly complex tasks and challenges for themselves to solve, rather than relying on human-designed training scenarios.
As of November 5, 2025, we hadn't employed self-play for LLMs. More specifically, LLMs aren't able to generate problems in which another LLM is learning to solve and challenging the knowledge of an LLM in that way.
Noam Brown at OpenAI notes that
“self-play works for two player zero sum games…like Go, Poker and Starcraft but is so much harder to use in real world domains”.
RL applications to help scale AI labs:
When training examples are ordered from easy to hard, convergence is faster and quality of local minima is higher (Yoshua Bengio).
This mimics how humans learn as well, starting from the basics and then learning harder and harder problems sequentially.
basins of attraction (def): set of initial conditions (starting points) in a dynamical system that evolve over time towards a specific long-term behavior (known as the attractor)
The following points demonstrate the bottlenecks of current AI systems.
The marginal information gain from showing tasks based on what humans find easy or difficult is less than the potential information gain from showing tasks based on what the model finds easy or difficult. Put another way, we need to zero in on what the actual model is ready to learn next based on its own definition of easy or difficult, a region called its learnable frontier.
The bottleneck is no longer maintaining the learnable frontier given a fixed set of tasks, but instead finding new tasks to feed to the learnable frontier.
This is a separate step before the training process - this is a fixed process. Possibly employing a continuous learning method can make the agents more versatile.
We want self-play for language models. Current methods that are framing the frontier:
However, this doesn't achieve state-of-the-art performance at larger scales.
This method automatically samples and adapts the task distribution, and even environment distribution, to the learner during training while introducing the second antagonist model. The multi-agent reinforcement learning setup looks like:
This neat trick ensures that generated environments are always solvable by at least one student, the antagonist. If the protagonist (the person we really care about) ever catches up to its counterpart, the teacher will make the game more challenging again, but not so difficult that its ally can’t solve it.
While more stable, this method still has a major issue of sparse rewards since the teacher's rewards depend on the agents' actions on the task provided. This further causes a long horizon.
With evolutionary algorithms, we can generate new, diverse tasks. This is a method inspired by biological evolution.
evolutionary algorithms (def): optimization techniques where population of candidate solutions evolves over time to find optimal or near-optimal solutions to a problem.
Here, LLMs can take an environment and mutate it (as seen in AlphaEvolve). EAs can be used with learned environment generators.