Zero-Shot Object Recognition & 3D Pose Estimation

A robot perception pipeline that detects arbitrary objects from text prompts and estimates 3D position, size, and orientation from RGB-D.

Overview

  • Period: Aug 2026 – present (K3I)
  • Robot perception pipeline that detects arbitrary objects from a text prompt alone in a conveyor environment and estimates 3D position, size, and orientation from RGB-D
  • Handles new target objects without per-object training data or CAD models

Live demo

Key work

  • SAM3 text-prompt segmentation — objects specified by text (“thermos”, “book”), no retraining
  • Hybrid tracking — SAM3 every 5th frame, optical flow in between → 9–13 FPS
  • RGB-D 3D pose — depth to point cloud, Open3D OBB for position/size/orientation; static objects within ±1 cm, rotation stabilized with zero axis flips
  • Occlusion handling — Kalman-filter trust check withholds pose during occlusion, same ID on reappearance
  • ROS2 pipeline — results published as ROS2 topics, RViz visualization, validated against an Isaac Sim conveyor

ROS2 pipeline

ROS2 node and topic layout

Published poses

/perception/detections view — pose withheld during occlusion

Isaac Sim integration

Isaac Sim conveyor integration

SAM3 · ROS2 Jazzy · Open3D · RealSense D455 · RGB-D · Kalman filter · RViz · Isaac Sim · Docker

Code: github.com/ingon1026/pose-anything (MIT) · Docker: hub.docker.com/r/ingon1026/pose-anything