Genesis AI - Simulating The World
This is the era of generative AI models — they can generate text, images, audio, video, and 3D objects... And if we combine all of these together, we might create an entire world!
CMU, together with more than 20 other research laboratories, has open-sourced a generative physics engine called Genesis, meaning "beginning" or "origin." As the name suggests, this could truly be the starting point of a new world.
Eventually, this collaborative team developed Genesis, a generative physics engine capable of creating 4D dynamic worlds, built upon a physics simulation platform for general robotics and physical AI applications.
While Genesis's technical paper is yet to be published, according to the official documentation, Genesis's main features include:
- Effortless installation with an extremely simple and user-friendly API.
- Unprecedented parallel simulation speed: Genesis AI is the world's fastest physics engine, running 10 to 80 times faster than existing GPU-accelerated robotics simulators (like Isaac Gym/Sim/Lab, Mujoco MJX, etc.) - yes, this sounds like science fiction - while maintaining simulation accuracy and fidelity.
- A unified framework supporting various SOTA physics solvers, capable of modeling different materials and physical phenomena.
- Performance-optimized photorealistic ray-tracing rendering.
- Differentiability: Genesis AI was designed with full compatibility for differentiable simulation in mind. Currently, its MPM solver and Tool Solver are differentiable, with differentiability for other solvers coming soon (starting with rigid body simulation).
- Physically accurate and differentiable tactile sensors.
- Native support for generative simulation, allowing generation of various modal data through language prompts: interactive scenes, task proposals, rewards, assets, character actions, policies, trajectories, camera movements, and (physically accurate) videos.
Additionally, Genesis AI supports various hardware platforms and operating systems.
Genesis AI Comprehensive Physics Simulation Platform
Genesis AI is a comprehensive physics simulation platform designed for general robotics, embodied AI, and physical AI applications. It possesses multiple attributes:
- A ground-up rebuilt general physics engine capable of simulating a wide range of materials and physical phenomena
- A lightweight, ultra-fast, Python-based, and user-friendly robotics simulation platform
- A powerful and fast photorealistic rendering system
- A generative data engine that transforms natural language descriptions from user prompts into various forms of data
Genesis AI's core technologies include:
- Fully interactive 3D scenes
- Open-world articulated object generation
- Voice audio, facial animations, and emotions
Currently, the underlying physics engine and simulation platform are being open-sourced. Access to the generative framework will be gradually rolled out in the near future.
Genesis AI Shows Outstanding Performance and Impressive Results
As a highly optimized physics engine, Genesis AI can leverage GPU-accelerated parallel computing to deliver unprecedented simulation speeds across various scenarios. When simulating manipulation scenarios, Genesis AI runs at 43 million frames per second, which is 430,000 times faster than real-time speed.
Genesis: Speed Comparison with Common CPU- and GPU-Based Robotics Simulators
Zhou Xian stated that the GPU-parallelized IK (Inverse Kinematics) solver of Genesis AI can solve the IK for 10,000 Franka robot arms within 2 milliseconds.
Next, let’s look at a concrete example demonstration.
Generating 4D Dynamic and Physical Worlds
The physics engine of Genesis AI is powered by a VLM-based generative agent, which uses the APIs provided by the simulation infrastructure as tools to create 4D dynamic worlds. These worlds then serve as foundational data sources for extracting various pattern data.
By integrating generative camera and object motion modules, Genesis AI can produce physically accurate and view-consistent videos and other forms of data.
Moreover, Genesis AI supports simulating a wide range of materials, including rigid bodies, articulated bodies, cloth, fluids, smoke, deformable bodies, thin-shell materials, elastic/plastic materials, robotic muscles, and more. Simulating a layer of chocolate sauce, naturally, is no challenge at all.
The texture of shredded foam also looks incredibly realistic.
The textures of planets and spaceships are also highly detailed, resembling scenes straight out of a big-budget sci-fi movie.
The physics of a bullet shattering a water balloon appears as if it were captured by a high-end high-speed camera.
A pot of alphabet gummies looks delightfully bouncy and elastic.
The simulation of inflatable dolls is spot-on, humorously capturing real-world behavior with great accuracy.
Character Motion Generation
With such a high-quality physics engine, it’s great news for the game development industry. Many complex movements and effects can now be quickly generated using simple prompts:
Prompt: A mini version of Goku holding a stick runs on a desktop for 3 seconds, then jumps into the air, swinging his right arm downward as he lands. The camera starts with a close-up of his face, then smoothly follows the character while gradually zooming out. When Goku reaches the highest point of the jump, the action pauses for a few seconds. The camera rotates 360 degrees around the character, then slowly rises before continuing the action.
The time cost of designing animations has been significantly reduced.
Robot Strategy Generation
Genesis AI can leverage generative robotic agents and the physics engine to automatically generate robotic strategies and demonstration data for various skills across different scenarios. This means researchers can quickly obtain robot motion plans that adhere to physical laws within a simulation environment, and reliably transfer these plans to physical robots.
Below are some examples of different types of robots performing various tasks.
Prompt: A moving Franka robotic arm uses a bowl and a microwave to make popcorn.
Prompt: Yushui Go2 quadruped robot running in the rain (Sim).
For example, the process from prompt to motion strategy in a simulation environment and then transferring it to a physical robot can be incredibly smooth:
Prompt: Yushui H1-2 humanoid robot walking forward (Sim2Real).
Performing a handstand requires precise balance control and full-body coordination. Such a high-level action can now be achieved through Genesis AI for Sim2Real:
Prompt: A quadruped robot performs a handstand using its front two legs (Sim2Real).
A handstand is not enough. With Genesis' assistance, the robotic dog can learn "gymnastics skills" even faster, flawlessly executing two straight-body backflips:
Prompt: A quadruped robot performs two consecutive backflips (Sim2Real).
Actions that involve interacting with real-world objects, like pulling a chair, are no problem either:
Motion manipulation of a large underactuated robot (Sim2Real).
3D and Fully Interactive Scene Generation
The generative framework of Genesis AI supports the creation of 3D and fully interactive scenes, which can be used for training robotic skills.
A household indoor scene featuring a living room (including a dining area), a bathroom, a study, and a bedroom.
The interior of a restaurant.
Open-World Articulated Object Generation
Genesis AI can also generate articulated objects and their interaction processes, such as opening and closing car doors, flipping a laptop open and shut, or folding a metal blade.
Soft Robots
Genesis AI is the first platform to provide comprehensive support for soft muscles and soft robots, as well as their interactions with rigid robots. It also includes a URDF-like configuration system tailored for soft robots.
Genesis AI can also simulate hybrid robots with soft skin and rigid skeletons.
Speech Audio, Facial Expressions, and Emotion Generation
Audio and facial expressions are also modalities that Genesis AI aims to integrate. Below are two examples:
A character's emotion shifts from neutral to angry, then changes to happy.
Genesis AI generalizes the transition of emotions across different facial expressions.
Conclusion
Finally, Zhou Xian showcased a Tetris game built with Genesis, where the blocks are made of jelly material and move according to realistic physical laws.
We may have come across similar videos before, but those were carefully crafted by visual effects artists. Now, Genesis AI can export them with a single click and further transform them into real, achievable technological breakthroughs.
Professor Gan Chuang shared his experience participating in this project on X: "Since 2018, I decided to shift my research focus from vision to embodied AI because I was fascinated by creating general intelligent agents that could interact with the physical world and other intelligent entities with human-like flexibility — we call this field embodied AGI."
He also wrote: "Honestly, sometimes I feel this simulator might be too advanced to release, but we believe that making it fully open-source and building a strong community around our mission is crucial! Join the Genesis AI community! We want to convince the robotics research community that 'Generative Physics Simulator is all You Need!'"
It truly makes us look forward to the practical applications of Genesis!
References
Open source address
https://github.com/Genesis-Embodied-AI/Genesis
Project page
https://genesis-embodied-ai.github.io/
Documentation address