Nvidia trains humanoid robots using the Apple Vision Pro headset

Nvidia is using the Apple Vision Pro augmented reality headset to develop a humanoid robot, writes 9to5Mac. Developers train Project GR00T’s humanoid robot AI model using real and synthetic data, allowing them to save time and reduce costs.

Image source: 9to5Mac

Training basic AI models for humanoid robots requires enormous amounts of data. One of the ways to capture (digitize) human movements is telecontrol – the operation of a robot in the mode of copying human movements, which is becoming an increasingly expensive and time-consuming process, Nvidia noted.

Last month, Nvidia presented at the SIGGRAPH computer graphics conference a reference telecontrol workflow using Nvidia AI and Nvidia Omniverse that allows AI researchers and developers to generate massive amounts of synthetic motion and perception data from a minimal amount of remotely captured human motion.

And it is to capture human movements that the Apple Vision Pro augmented reality headset is used. After recording a small number of telecontrol demonstrations using Apple Vision Pro, developers then model the data in NVIDIA Isaac Sim, a reference application for designing, simulating, testing and training AI robots, and use the MimicGen NIM microservice to generate synthetic sets data from records. That is, they expand the database with a real demonstration of movements with synthetic data sets.

Using the resulting database, the Project GR00T humanoid robot model is being trained. Developers then use the Robocasa NIM microservice in Isaac Lab, a robot learning framework, to generate experiences to retrain the robot.