Topic Guide
What Is Scaling laws?
Scaling laws is a subject covered in depth across 3 podcast episodes in our database. Below you'll find key concepts, expert insights, and the top episodes to listen to — all distilled from hours of conversation by leading experts.
Key Concepts in Scaling laws
Scaling laws for pre-training
These are empirical observations that predict how model performance improves with increased compute, data, and model size during the initial training phase. Gemini 3 notably confirmed these laws remain intact, despite researchers not fully understanding the underlying 'how' or 'why' they work.
Scaling laws for post-training
Two new scaling laws driving recent AI progress: Reinforcement Learning with Verified Rewards (RLVR) and test-time compute. RLVR involves training AI models using outcomes that can be objectively verified (e.g., did a sale convert, did the model balance the books), while test-time compute refers to allowing models to 'think' or process for longer during inference. These laws enabled significant progress even when pre-training hardware was stalled.
Low-cost producer of tokens
In the AI industry, this refers to the entity that can generate AI output (tokens) at the lowest computational cost. Gavin Baker highlights that Google's TPUs have given them this advantage, allowing them to exert economic pressure on competitors. This metric is uniquely important in AI, unlike most traditional tech industries where low-cost production hasn't been the primary driver of market value.
Reasoning (in ai)
Refers to an AI model's ability to 'think' or chain together logical steps to solve problems or generate more sophisticated outputs. The advent of reasoning models (like the first from OpenAI) significantly accelerated AI intelligence levels and enabled a 'flywheel' where user interactions and verifiable outcomes can be fed back to continuously improve the model.
Data centers in space
A visionary concept for AI infrastructure, proposing orbiting satellites equipped with chips for compute. From first principles, they offer superior conditions: constant, intense solar power (eliminating batteries) and free cooling to near absolute zero in the vacuum of space (reducing complex cooling systems), potentially leading to faster, lower-cost inference and training via laser communication.
Saas ai margin mistake
The error made by incumbent Software-as-a-Service (SaaS) companies who are reluctant to adopt AI agents because it would reduce their traditional 70-90% gross margins to 35-40%. This is compared to brick-and-mortar retailers ignoring e-commerce. Gavin argues this will leave them vulnerable to AI-native startups willing to operate at these lower margins, eventually leading to their displacement.
What Experts Say About Scaling laws
- 1.NVIDIA has transitioned to "extreme co-design" across the entire computing stack, from individual components like GPUs and CPUs to full data center infrastructure, to solve complex distributed problems that no longer fit a single computer.
- 2.The necessity for extreme co-design arises from the goal to achieve speedups "a million times faster" than simply increasing the number of computers, requiring intricate sharding of algorithms, data, and models.
- 3.Jensen Huang leads with a unique organizational structure where a large staff of 60+ technical experts collaborates in group settings, allowing for a comprehensive, cross-disciplinary approach to problem-solving inherent to co-design.
- 4.A crucial strategic decision for NVIDIA was embedding CUDA into GeForce GPUs, despite incurring significant costs that severely impacted profitability and temporarily reduced the company's market cap, to establish a vital developer "install base" which ultimately powered the deep learning revolution.
- 5.Huang believes that the "install base defines an architecture" more than its elegance, illustrating this with the success of x86 versus more elegantly designed RISC architectures.
- 6.Jensen Huang's leadership strategy for bold bets involves intensely reasoning about a future outcome, then systematically shaping the belief systems of his team, board, and external partners over an extended period to ensure widespread buy-in for strategic shifts.