PoseRefer: Pathway-Local Parameters for Semantically Grounded Reference Resolution
Anna Deichler · 2026
Read PoseRefer: Pathway-Local Parameters for Semantically Grounded Reference Resolution by Anna Deichler (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24622.txt.
Introduction
Anna Deichler evaluates pose-language fusion for robots resolving spoken references, using co-speech gesture data from dyadic virtual-reality interaction. Her decoupled late-fusion design gives pose and text pathways no shared parameters. Category, pose, and text contributions can then be isolated by ablation. Fusion with frozen MiniLM category embeddings beats pose alone and the best text-only route on all reference types, reaching 31.9 percent top-1. A scalar gate flips between policies depending on category access, warning that fusion-accuracy claims may reflect category artifacts unless pathways are decoupled.
Catalog record
| CREATOR | Anna Deichler |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22437 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.24622
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PoseRefer: Pathway-Local Parameters for Semantically Grounded Reference Resolution Anna Deichler Abstract A robot resolving “put the cup on that one” must fuse gesture, language, and scene geometry, yet 3D grounding benchmarks only partially capture this regime: descriptions are written post-hoc, gestures are templated, or pointing is staged for the camera. MM-Conv captures natural co-speech gesture from dyadic VR interaction alongside full-body motion capture and 3D scene graphs. We use it to evaluate pose–language fusion with a decoupled late-fusion architecture in which pose and text pathways share no learned parameters. The two choices together make category, pose, and text contributions easier to isolate through controlled ablations. Fusion with frozen MiniLM category embeddings exceeds pose alone and the best text-only pathway on every reference type, reaching 31.9% top-1. The learned scalar gate flips between opposing policies depending on whether the text pathway has category access. This is a reliability diagnostic: fusion-accuracy claims for semantic grounding systems are indistinguishable from category-representation artifacts unless pathways are architecturally decoupled. I Introduction When a person points at a chair and says “put the cup on that one,” a service robot observing from a third-person view must integrate gesture, language, and scene geometry to identify the referent. This problem, exocentric reference resolution, has received less attention than its 2D image-grounded cousin, despite being a regime central to many household and assistive robotics scenarios. Existing 3D grounding benchmarks substitute one or more of the naturalistic ingredients: post-hoc descriptions on static scans (ScanRefer [ 2 ] , Sr3D/Nr3D [ 1 ] ), template-generated language with synthetic 2D pose (Ges3ViG [ 11 ] ), or single-user pointing performed for the camera (YouRefIt [ 3 ] ). MM-Conv [ 4 ] differs in kind: references arose from dyadic VR interaction where two participants worked through a referential task in a shared 3D scene, with full-body motion capture, word-level speech alignment, and ground-truth scene graphs. It contains \sim\!4{,}000 references across 5 rooms, with \sim\!55\% accompanied by a detectable pointing or pointing-variant gesture and \sim\!45\% without. Reliable grounding of such references is a prerequisite for any downstream action selection that depends on semantic scene understanding. Fig. 1: PoseRefer in a 3D scene. The SMPL-X mesh shows the speaker’s pose; the pointing ray and head direction define angular proximity to candidate objects (bounding boxes). On naturalistic gesture data, two confounds become visible that synthetic data hides. First , per-object category embeddings entangle with all three pathways, pose, text, and the fusion gate, through shared encoders and shared parameter tables. Ablating category in such architectures simultaneously alters every signal and every learned representation, making fusion comparisons hard to interpret. Second , the representation chosen for those per-object categories matters substantially: in our experiments, a learned 16-dimensional category embedding is insufficient at MM-Conv’s per-category training counts, and the fusion gate inherits this noise. Both problems are invisible unless the architecture allows direct ablation of individual signals and the category representation itself is varied. Contributions. 1. Position: naturalistic referential gesture data is an important testbed for claims about semantic grounding in multimodal perception. Nearly half of such references occur without clear pointing, a regime synthetic benchmarks largely exclude. 2. Method: a decoupled late-fusion architecture with pathway-local category embedding tables (no shared parameters between pose and text), and frozen pretrained semantic embeddings for per-object category on the text pathway. The two choices together reduce category, pose, and text to controllable ablations. 3. Empirical findings: on MM-Conv, (a) the fusion gate commits to opposing policies depending on whether category is available on the text pathway ( \alpha\!=\!0.84\rightarrow 0.22 ), a diagnostic reproducible across cross-validation folds; (b) pose and text are asymmetrically complementary: pose dominates pointing tiers, text dominates non-pointing tiers; (c) with pathway-local parameters and frozen MiniLM category embeddings, scalar fusion exceeds both singletons across every reference type, reaching 31.9\% aggregate top-1, a gain of 13.1 points over pose alone and 6.9 over the strongest text-only configuration. 4. Reliability implication: fusion gains in architectures with shared encoders or shared category embeddings may reflect the category-representation choice rather than properties of fusion itself. Our decoupled setup makes this directly testable and motivates reporting both learned and frozen category representations when evaluating multimodal fusion for semantic grounding. II Related Work 3D visual grounding operates on post-hoc scene descriptions without the speaker’s body (ScanRefer [ 2 ] , Sr3D/Nr3D [ 1 ] , BUTD-DETR [ 9 ] , EDA [ 14 ] ); Ges3ViG [ 11 ] adds template language with synthetic 2D pose and YouRefIt [ 3 ] captures real but single-user 2D reference. 3D scene graphs have emerged as a compact semantic environment representation for embodied tasks [ 7 , 6 ] ; MM-Conv’s ground-truth scene graphs let us isolate pose–language fusion from scene-perception error. Wang et al. [ 13 ] use a distance field between skeleton joints and scene points for language-guided motion generation; we adapt the idea to angular distance over discrete candidates. Deichler et al. [ 5 ] generate context-aware pointing gestures for embodied agents; we address the inverse problem of interpreting observed gestures for reference resolution. Related embodied fusion work explores task-adaptive gated routing [ 10 ] and exocentric reference benchmarks [ 8 ] . Frozen pretrained language spaces appear for semantic matching in open-vocabulary pipelines [ 15 ] , but typically as part of a broader vision–language stack rather than an isolated per-object handle in a dedicated fusion pathway. III Dataset and Setup MM-Conv contains approximately 4{,}200 referring expressions from 6.7 h of dyadic VR interaction across 5 AI2-THOR rooms with 42–61 candidate objects per room, synchronized speech (word-level timestamps), full-body SMPL-X parameters at 30 fps, and 3D scene graphs. Expressions are 38\% exact noun phrases, 14\% partitives, 48\% pronominals (e.g., “that one”, “this one”). After filtering for valid targets, text features, and valid temporal windows we evaluate on 3{,}764 references. We adopt a third-person robot camera view; scene-graph ground truth is unchanged. Category vocabulary. Scene-graph labels are lowercased, compound labels split, and color/material/size modifiers stripped (“RedVase” \rightarrow “vase”), addressing training-signal sparsity in the raw vocabulary. The normalized vocabulary contains 50 categories. Stripping discards potentially grounding-relevant attributes (e.g., color), a systematic ablation of modifier retention is deferred to future work. Gesture classification. A random-forest classifier on 16 SMPL-X kinematic features separates clear pointing from no-gesture (AUC 0.97 ). Classifier scores stratify references into four confidence tiers, T1–T4, plus a post-hoc T5 for two-arm pointing; we use the binary partition pointing (T1 \cup T2 \cup T5, n\!=\!2{,}068 , 55\% ) vs. non-pointing (T3 \cup T4, n\!=\!1{,}696 ) throughout. Tiers are descriptive only, never used as model input. Per-tier counts are reported in Table II . IV Method: Decoupled Late Fusion Fig. 2: PoseRefer architecture. Pose and text pathways operate independently with no shared learned parameters. The text pathway encodes the utterance via a frozen MiniLM-L6 model ( \to 384D) before a learned linear projection to 128D. Each pathway has its own per-object category embedding table: learned 16D on the pose side; on the text side, either learned 16D or (default in PT {}_{\text{minilm}} ) a frozen MiniLM encoding of the canonical category name. Per-pathway scores are z -normalized and combined by a learned scalar gate \alpha . We score each candidate object by its angular proximity