Transformers Can't Think In Physics: Anima Anandkumar on Accelerated Understanding

Why it matters

In this episode, Caltech professor Anima Anandkumar and researcher Benedikt Jenik dive into the gap between large language models and physical world modeling. They argue that current transformers excel at symbolic reasoning but lack an intrinsic grasp of causality, time, and space continuity. The conversation focuses on combining neural operators, generative simulation, and scientific knowledge to build AI systems that truly accelerate discovery—an essential watch for anyone in AI for Science.

Key takeaways

• Transformers excel at linguistic and symbolic patterns but struggle to internalize physical laws, demanding fundamental architectural innovation.
• Embedding physical constraints into neural networks—e.g., neural operators—is key to improving extrapolation and generalization.
• Accelerating scientific discovery requires AI that generates hypotheses and high-fidelity simulations, not just processes existing text.
• Cross-disciplinary collaboration is essential to building physically aware AI systems.