NVIDIA & Orbbec partner to advance robotics vision technology

In a notable development at Computex, NVIDIA has shared its advanced vision for robotics and the burgeoning domain of “physical AI.” This vision was further supplemented by Orbbec’s announcement that its Gemini 330 series Stereo Vision 3D cameras will now be integrated with NVIDIA's Isaac Perceptor. Isaac Perceptor is a reference workflow designed for autonomous mobile robots (AMRs) built on the GPU-accelerated Isaac ROS framework.

The integration of Orbbec’s cameras aims to significantly enhance the vision capabilities of AMRs. These cameras are set to provide improved depth and RGB data output, even under challenging conditions. This enhancement will facilitate more accurate environmental sensing and navigation for robots and contribute valuable data for simulation environments.

Orbbec's Gemini 330 series cameras incorporate built-in depth processing, delivering high-precision Depth+RGB vision. This precision is crucial for the NVIDIA Isaac Perceptor AI-based perception workflow utilised by AMRs in indoor and outdoor environments. The integration promises improved depth quality and longer-range sensing capabilities under varied lighting conditions, allowing Isaac Perceptor to output 3D reconstruction and obstacle maps of any unstructured environment efficiently.

The Gemini 335/335L/336/336L camera models notably operate in passive and active laser-illuminated modes. This multi-mode functionality ensures high-quality depth and RGB data output, even in the most challenging lighting conditions. The depth algorithms are processed directly in the camera via Orbbec’s latest depth engine ASIC, effectively alleviating the processing burden on the NVIDIA Jetson Orin module-based compute. These cameras also feature internal Inertial Measurement Units (IMU) and temperature sensors, boasting a working range of 0.2-10 meters. Additional attributes include global shutter image sensors, wide field-of-view lenses, high frame rates, low latency, and precise multi-camera synchronisation.

Furthermore, Orbbec announced the Gemini 336/336L variants, designed to enhance indoor performance by adding Near-Infrared (NIR) bandpass filters. These filters are instrumental in reducing potential depth map “holes” caused by glare from reflective surfaces and minimising “ghost” images from repetitive environmental patterns.

Amit Banerjee, Head of Partnerships at Orbbec, remarked, “The integration of Orbbec’s cameras with NVIDIA Isaac Perceptor will provide robot developers the precise Depth+RGB vision capabilities necessary to enhance the performance and reliability of AMRs across various sectors.”

He added, “This collaboration extends beyond physical applications, with Orbbec camera simulation models also available through the NVIDIA Isaac Sim platform and NVIDIA Metropolis framework, ensuring customers can fully leverage NVIDIA-accelerated technologies from physical run-time models to physically-based simulation environments.”

In addition to their application in AMRs, the Gemini 330 series cameras are highly suited for robot arm applications requiring AI vision for tasks such as bin-picking, palletisation, scanning, and sorting, especially in environments where reducing glare and resulting holes from glossy surfaces is critical.