Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning

3 months ago by cm0002 to c/Aii

Reinforcement learning with verifiable rewards without human-annotated data, often referred to as zero RL, has emerged as a powerful paradigm for eliciting chain-of-thought reasoning. However, due to computational constraints, existing studies are largely restricted to small models, leaving the training dynamics and emergent capabilities at a large scale unexplored. To meaningfully explore this frontier, we aim to elicit high-quality reasoning behaviors from the model. However, we find that naive scaling often suffers from poor readability, token redundancy, and a lack of adaptive reasoning depth. To address these challenges, we present a stable and efficient training pipeline, incorporating algorithmic and system optimizations such as clipped importance sampling, training-inference ratio correction, and mixed-precision control. Our experiments offer three key findings that validate the "bitter lesson" of scaling: (1) scaling to 1T parameters significantly enhances sample efficiency and performance ceilings; (2) the training process progresses sequentially through an initial discovery phase followed by a sharpening phase; and (3) the model spontaneously develops advanced cognitive behaviors, including anthropomorphism, structured formatting, self-verification, parallel reasoning, and context anxiety, rendering hand-crafted heuristics redundant. Evaluated on seven mathematical benchmarks, Ring-2.5-1T-Zero achieves competitive performance. Additionally, to assess CoT quality beyond final-answer correctness, we propose a structured evaluation framework across three dimensions: comprehensibility, reproducibility, and efficiency, where our model demonstrates clear advantages in producing structured and concise reasoning traces. By sharing our observed emergent phenomena, we hope to provide the community with deeper insights into scaling behaviors, particularly at the 1-trillion scale.

The researchers trained a trillion-parameter model using zero reinforcement learning to see how reasoning capabilities emerge at a massive scale without relying on human-annotated data. They found that pushing the parameter count to a trillion drastically improves both sample efficiency and the overall performance ceiling when compared to a smaller 104-billion parameter baseline strongly validating the concept that raw scale and computation eventually outpace hand-crafted human heuristics.

They also discovered that the training process reliably unfolds in two distinct sequential stages. The model starts with a discovery phase where it actively expands its reasoning boundaries by unlocking dormant pathways, and then it moves into a sharpening phase where it refines its policy within those established limits. Notably, the model spontaneously developed advanced cognitive strategies entirely on its own.

It began using structured formatting, parallel reasoning, self-verification, context anxiety, and even anthropomorphic expressions of frustration during complex tasks without any explicit human prompting. To keep the training stable at such a massive scale, the team relied on simple optimization techniques like clipped importance sampling and mixed-precision control. They also created a new evaluation framework to judge the actual quality of the reasoning steps based on comprehensibility, reproducibility, and token efficiency instead of just looking at the final answer.

Aii
Aii

@programming.dev

login for more options
391
477
278

AI related news and articles.

Rules:

  • No Videos.
  • No self promotion: Don't post links to your articles.

go to feed...