Skip to content
Neural Daily

ModelsNews

NVIDIA Cosmos3-H-Dreams simulates surgery in real time

NVIDIA has released Cosmos3-H-Dreams, a generative simulator trained on real gallbladder surgery video and robot kinematics that renders the surgical scene live as instruments move. It runs on a single RTX PRO 6000 GPU, with open checkpoints and a training recipe.

By: Mikola Máté Patrik · · 3 min read

NVIDIA illustration: a surgeon at a robotic console sends actions to a surgical robot, which streams a video feed back
Image: NVIDIA · Press kit · Official blog header image · Source

What happened

On October 20 NVIDIA introduced Cosmos3-H-Dreams (Hugging Face blog). It is a generative simulator: given a starting frame and instrument commands, it generates in real time what happens inside the abdomen. NVIDIA says it is the first real-time generative surgical simulator trained on clinical cholecystectomy recordings to support live kinematic control.

Its predecessor, Cosmos-H-Dreams, learned tabletop suturing on the dVRK research robot. The new version learns from video and robot data recorded during real procedures on the CMR Surgical Versius platform.

What is new

  • Base: the 16-billion-parameter Cosmos3-H-Embodied-Nano, pretrained on the Open-H-Embodiment dataset.
  • Controls: 28 action dimensions covering pose and gripper for both arms, energy controls, and thumbstick axes and buttons. It tracks two instruments, a left Cadiere grasper and a right monopolar hook.
  • Training: 832 x 480 video with actions aligned at 10 Hz. The teacher first learns 24 future frames, then 72, which is 2.4 and 7.2 seconds.
  • Speed: Self-Forcing distillation turns the teacher into a causal student that generates block by block as new commands arrive. Two checkpoints ship, one with four denoising steps and one with two, trading visual quality against speed.
  • Runtime: with the FlashDreams streaming inference library it runs interactively on a single NVIDIA RTX PRO 6000 GPU.

Why it matters

Classic numerical simulation models geometry and tissue deformation explicitly, which is slow and expensive. The generative approach learns those dynamics from video and robot data instead. Inside the body every small movement changes the scene: tissue stretches, reflections shift, structures get occluded and smoke reduces visibility. Generating that consistently in real time is a bottleneck for surgical robotics and for training autonomous surgical policies.

NVIDIA is explicit that this is a research simulator, not a validated system for patient care, and that visual realism is not proof of biomechanical accuracy. Next steps it lists include covering the full procedure, combining generative and numerical simulation, and evaluation with surgeons.

Try it

The model is on Hugging Face, with code and examples on GitHub. It can be adapted to other datasets and robots with synchronized video and compatible action representations, and NVIDIA published a step-by-step recipe for teacher fine-tuning and distillation. For medtech teams working with surgical robot data, it is a ready starting point for their own simulator.

    Produced with the help of AI, reviewed by an editor. How we make our articles

    Reviewed and edited by the Neural Daily desk. About

    Related articles