Dominic Jainy is a leading IT professional and expert in physical AI, specializing in the intersection of robotics, machine learning, and healthcare infrastructure. With a deep focus on how digital twins and vision-language-action models can revolutionize medical environments, he has become a key voice in the development of autonomous systems designed to alleviate the mounting pressures on global healthcare. In this conversation, we explore the technical foundations and future implications of hospital automation, specifically focusing on the Project Rheo blueprint and the shift toward continuous, simulation-based training.
The discussion covers the strategic importance of automating surgical subtasks to mitigate clinician shortfalls and the use of high-fidelity simulations to bridge the data gap in chaotic hospital settings. Dominic details the differences in training workflows for varied robotic tasks, the role of synthetic data in overcoming environmental shifts, and the curriculum-based approaches necessary for complex multi-stage procedures.
Healthcare systems face a massive clinician shortfall and high costs for every minute of operating room time. How does surgical subtask automation address these specific bottlenecks, and which repetitive tasks should be prioritized to allow surgeons to focus on critical decisions?
The global healthcare crisis is no longer a distant threat; we are looking at a projected shortfall of 10 million clinicians by 2030. In the operating room, where every minute can cost tens of dollars, inefficiency is a luxury we can’t afford. Surgical subtask automation targets the “friction” of a procedure—repetitive, high-volume actions like suturing or surgical tray pick-and-place—that consume a surgeon’s cognitive bandwidth without requiring their high-level diagnostic expertise. By delegating these tasks to autonomous agents, we aren’t just saving time; we are increasing procedural throughput and democratizing access to care for the billions who currently face diagnostic and surgical gaps. The priority must be on these predictable, repetitive sequences, allowing the human expert to remain the pilot while the robot handles the “autopilot” elements of the workflow.
Hospitals are often chaotic environments with unique layouts and unpredictable human interactions. Since capturing real-world data for every edge case is unsafe and expensive, how do digital twins help robots master navigation and workflow variations before they ever enter a physical ward?
Real-world hospitals are heterogeneous and high-stakes, making it operationally infeasible to capture exhaustive data on every possible edge case, such as emergency interruptions or rare equipment failures. Digital twins serve as the foundational “data substrate,” allowing us to build a digital hospital where robots can experience thousands of navigation patterns and human interactions safely. Using tools like the Isaac Lab-Arena track, we can swap scenes, objects, and embodiments with minimal friction to see how a robot reacts to a crowded hallway or a sudden change in policy. This simulation-led approach reduces clinical risk significantly because the robot has already “lived” through these chaotic permutations before it ever encounters a live patient or a busy nurse. It transforms the hospital into a continuous training environment that exists entirely in bits before it ever moves into atoms.
When a developer chooses between arena-scale tasks like moving case carts and high-precision bimanual tasks like assembling a trocar, how do the training workflows differ? What are the specific benefits of using vision-language-action models for these different levels of manipulation?
The workflow splits based on the complexity and scale of the physical interaction. For arena-scale tasks like pushing a case cart or picking up a tray, we utilize the Isaac Lab-Arena model for rapid composition, focusing on locomotion-manipulation where the robot moves through a scene. Conversely, for high-precision bimanual tasks like assembling a trocar, we use a task-centric Isaac Lab track that defines the scene configuration explicitly, including wrist cameras and rigid object configurations for the trocar components. The NVIDIA Isaac GR00T vision-language-action (VLA) models are transformative here because they allow the robot to process multimodal inputs—seeing the tray, understanding the command, and executing the motor control—all within a single policy. This creates a more intuitive “physical AI” that can generalize across different tasks, whether it’s the gross motor skill of cart pushing or the fine motor skill of multi-part tool assembly.
Once a few expert demonstrations are recorded using motion controllers, how can synthetic data generation and domain transfer tools improve a robot’s success rate? How do you specifically account for shifts in lighting, clutter, or room geometry across different hospital facilities?
Recording just one or two expert demonstrations via motion controllers like Meta Quest is only the starting point; the real magic happens in the “multiplication” phase. Through synthetic data generation pipelines, we can take those few successful “seeds” and diversify them into a massive dataset that covers variations in object placement and lighting. For example, our benchmarks for surgical tray pick-and-place show that a base model might have a 0.00 success rate when moved to a new, unfamiliar scene. However, by using Cosmos-augmented models for generative transfer, we can see success rates in those same shifted scenes jump to 0.30 or higher. This “domain shift” is the key enabler for hospital deployment, as it prepares the robot for the specific lighting, clutter, and geometry quirks of a facility it has never physically visited.
Complex procedures often require moving from supervised fine-tuning to online reinforcement learning. What does a curriculum-based approach look like for multi-stage tasks, and what metrics indicate that a robot is ready to move from a “lift and align” phase to “insertion”?
A curriculum-based approach breaks down a daunting task—like the four-stage “Assemble Trocar” procedure—into manageable milestones: lift, align, insert, and place. We typically start with Supervised Fine-Tuning (SFT) to get the robot to a baseline level, but as the tasks get harder, we switch to Online Reinforcement Learning using Proximal Policy Optimization (PPO). The metrics are quite telling: for the “insert” stage, which is notoriously difficult, a base SFT model might only achieve a 32% success rate, whereas RL post-training can push that success rate up to 85%. We monitor the success hold steps and episode lengths to determine readiness; for instance, if a robot can consistently hold a tray for 150 steps without failure, it is likely ready to transition from a simple “lift” to the more complex “alignment” and “insertion” phases.
Before deploying an autonomous system, how should developers use WebRTC streams and vision-language model agents to validate policies? What are the essential steps for running an end-to-end integration smoke test to ensure the digital agent and physical robot are communicating correctly?
The final validation before a robot touches the hospital floor is the end-to-end integration smoke test. We use a triggered policy runner that streams camera observations at 30 FPS over WebRTC while exposing a trigger endpoint for an external orchestrator. This allows a VLM-based digital agent to observe the live feed and suggest or authorize actions, effectively acting as a monitoring and assistance layer. To run this test, you connect the UI livestream to the WebRTC server—typically on ports 8080 and 8081—and verify that the digital agent’s commands result in the correct physical response from the robot. This ensures that the entire communication stack, from the vision-language model down to the motor controllers, is synchronized and capable of closed-loop operation.
What is your forecast for hospital automation?
I believe we are moving toward a future where hospitals are no longer just static buildings, but are instead “living” AI environments. My forecast is that within the next decade, the “digital twin” of a hospital will be as standard as its blueprint, serving as a permanent sandbox for continuous learning and policy updates. We will see a shift where robots are not just specialized tools for one surgery, but versatile physical agents capable of navigating complex wards to deliver supplies and perform surgical subtasks autonomously. As we bridge the data gap through simulation, automation will become the primary way we scale clinician capacity, ultimately making high-quality, high-throughput healthcare a global standard rather than a local privilege.