Open-vocabulary metric perception

Open vocabulary that measures — not just names

Foundation models can find objects from text, but they are poor rulers. Classical RGB-D fitting can measure, but only after someone decides which points belong to which object — and naïve single-view fits often get orientation wrong. This system treats those as one problem: open-vocabulary masks for identity, geometry for metric pose and size.

Gap: Open-vocab models name and mask; they do not reliably return metric pose and extent for manipulation
Approach: SAM3 text prompts for instances; depth + nonlinear fitting for tracked boxes and cylinders
Status: Live ROS 2 stack on RealSense, ~5 Hz, offline-replayable geometry

Text prompts → Instance masks → Metric primitive fit → Motion-aware tracking

Videos

Demo

Open-vocabulary RGB-D detection and metric shape tracking

What existing approaches lack

Closed-set detectors and CAD-based pose estimators deliver metric output only for categories prepared in advance. Open-vocabulary segmentation removes that bottleneck, but stops at pixels, scores, and 2D boxes — not the 3D pose and dimensions a planner or grasp module needs.

Pure geometric pipelines have the opposite failure. Without a strong instance prior, depth fitting cannot decide what to measure. Even with a mask, single-view clouds usually show one or two faces of a box; PCA-style orientation tilts across those faces instead of aligning with them. Fitting in 3D alone also ignores whether the shape still matches the image silhouette once projected.

Approach

SAM3 answers what is in the scene from free-text prompts — new objects cost a phrase, not a dataset. Classical RGB-D geometry answers where it is and how large it is, with fitting built for the partially observed, single-view case robots actually see. The metric half stays inspectable: masks and depth can be logged and geometry re-run offline without the GPU segmenter.

Visible-face–aware box initialization — orientation from face planes, not PCA on partial clouds.
Joint refinement against the point cloud and the mask silhouette, so depth noise and segmentation errors check each other.
Soft temporal priors and motion-aware tracking so weakly observed axes stay stable as the camera or robot moves.

Takeaway

Open-vocabulary recognition and metric geometry can be engineered as one onboard loop for tabletop primitives — oriented boxes and cylinders on a single aligned RGB-D stream — without waiting for closed-set detectors or end-to-end black-box pose networks.

Keywords

ROS 2 · SAM3 · PCL / RANSAC · Nonlinear optimization