Direction-aware 3D Large Multimodal Models Quan Liu Weihao Xuan Junjue Wang Naoto Yokoya Ling Shao Shijian Lu Abstract 3D large multimodal models (3D LMMs) rely heavily on ego poses for enabling directional question-answering and spatial reasoning. However, most existing point cloud benchmarks contain rich directional queries but lack the corresponding ego poses, making them inherently ill-posed in 3D large multimodal modelling. In this work, we redefine a new and rigorous paradigm that enables direction-aware 3D LMMs by identifying and supplementing ego poses into point cloud benchmarks and transforming the corresponding point cloud data according to the identified ego poses. We enable direction-aware 3D LMMs with two novel designs. The first is PoseRecover , a fully automatic pose recovery pipeline that matches questions with ego poses from RGB-D video extrinsics via object–frustum intersection and visibility check with Z-buffers. The second is PoseAlign that transforms the point cloud data to be aligned with the identified ego poses instead of either injecting ego poses into textual prompts or introducing pose-encoded features in the projection layers. Extensive experiments show that our designs yield consistent improvements across multiple 3D LMM backbones such as LL3DA, LL3DA-SONATA, Chat-Scene, and 3D-LLAVA, improving ScanRefer mIoU by 30.0 % 30.0\% and Scan2Cap LLM-as-judge accuracy by 11.7 % 11.7\% . In addition, our approach is simple, generic, and training-efficient, requiring only instruction tuning while establishing a strong baseline for direction-aware 3D-LMMs. † † footnotetext: † Corresponding author: Shijian.Lu@ntu.edu.sg † † footnotetext: Code is available at https://github.com/liuQuan98/PoseAlign3D 1 Introduction Generalist point cloud 3D large multimodal models (3D-LMMs) pursue broad competence in 3D scene understanding and reasoning, targeting key tasks such as object grounding 56 ; 58 ; 25 ; 49 , referring 5 ; 1 ; 60 , question answering 2 ; 55 , ego position reasoning 33 , object captioning 10 ; 59 , route planning 43 , segmentation 44 ; 18 , detection 45 , etc. Such enabling represents a key step towards the visual-cognitive core of future embodied agents 22 ; 14 . In particular, an ideal generalist 3D-LMM should possess persistent awareness of ego poses and spatial directions in the world coordinate for accurate understanding of 3D input data 41 ; 52 , robust and alias-free spatial reasoning 57 ; 14 , and safely navigating in 3D environments 22 . Figure 1 : Ego pose is critical in spatial reasoning and understanding. (a) Direction-agnostic 3D LMMs are struggling to reason spatial directions due to the absence of ego-pose information. (b) Incorporating ego pose resolves directional ambiguity, enabling consistent and robust spatial reasoning. Ego-pose awareness entails distinguishing lateral directions (e.g., ‘left’, ‘right’, ‘front’, ‘back’) that are defined according to a reference frame 15 . Two forms of directions have been widely adopted: 1) Egocentric Direction that defines spatial relations relative to the agent’s own body axes ( e.g. , ‘the object is on my left’) and 2) Allocentric Direction that defines spatial relations relative to an external anchor ( e.g. , ‘the man is on the left of his car’). For indoor scenarios where the anchor has no canonical axes, like a plate or a table, the reference of the allocentric direction falls back to the ego agent 15 . Hence, an ego pose is essential to reconcile the two forms of directions to achieve consistent spatial reasoning and understanding by 3D LMMs. However, most existing 3D indoor datasets such as ScanRefer 5 , Multi3DRefer 61 , ScanQA 2 , Scan2Cap 10 , and Nr3D 1 contain many allocentric directional queries but provide no explicit ego poses, positioning spatial reasoning ill-posed and hindering the acquisition of persistent ego pose awareness as illustrated in Figure 1 . This omission largely stems from their assumption of a global third-person rather than an egocentric first-person perspective of the scene. Prior studies address this omission mostly by creating new datasets instead of rectifying existing ones, requiring 3D LMMs to infer ego poses while responding to spatial queries 33 ; 57 . Actually, inferring ego poses as a latent variable is conceptually redundant, as ego poses are already available when an embodied agent collects indoor point clouds via simultaneous localization and mapping (SLAM) 35 . To this end, we propose to directly incorporate the readily available pose data as model inputs due to two factors: (1) it rigorously resolves the directional ambiguity in existing 3D indoor benchmarks; (2) it does not affect the application scope of 3D LMMs by leveraging the ‘free-lunch’ pose in the practical embodied workflow. More discussion is available in section A.1 . For efficient implementation, we enable the new paradigm with two new designs. The first is PoseRecover , an automatic pose generation pipeline that addresses the lack of ego poses in existing benchmarks. Specifically, PoseRecover retrieves question-related camera poses from ScanNet RGB-D sequences by matching camera frustums with relevant object annotations such as segmentation masks, bounding boxes, and location annotators, leading to a list of candidate camera poses for online training or inference. During the forward pass, candidates undergo various screenings to purge opposite-view errors and ensure authenticity while maintaining a certain degree of variety. The second is PoseAlign , a simple solution that incorporates the identified ego pose into point cloud data to enable their interpretation by existing 3D LMMs. Specifically, PoseAlign repositions point clouds to align with the identified ego poses, enabling universal direction-awareness across existing 3D LMMs of different architectures. Thanks to the coordinate sensitivity of pretrained point cloud encoders, the direction awareness can be boosted without even tuning the encoder of 3D LMMs. The contributions of this work can be summarized in three major aspects. First, we identify that most existing 3D LMM benchmarks suffer from ill-posed directions and propose a new paradigm that mitigates this problem effectively by incorporating an ego pose. Second, we design PoseRecover and PoseAlign, the former being a pose generation pipeline that addresses the ill-posed problem in 3D LMM benchmarks by recovering mission-critical pose data, and the latter being a simple yet effective modifier that enables direction-awareness for existing 3D LMMs by injecting pose data into the point cloud. Third, extensive experiments show that our approach improves the direction awareness substantially and consistently across 4 different benchmarks and all 3D LMM architectures. 2 Related Work 2.1 Indoor 3D LMM Benchmarks Following the huge success of 2D visual understanding benchmarks 32 ; 31 and the richly-annotated indoor datasets 12 ; 4 ; 49 , the first batch of 3D understanding benchmarks 1 ; 55 ; 60 take similar approaches to link objects with text descriptions in 3D with expert human annotators, achieving a stunning variety and volume of tasks, e.g. , referral 5 , captioning 10 , and question-answering 2 . While one could argue that their human-centric pipelines implied the viewing direction must be from a human standing inside the room, not explicitly defining such poses causes their questions to be inherently ill-posed 57 . Often completed by crowdsourcing, these benchmarks could never again accurately recover those poses. Later attempts to improve directional awareness 25 ; 33 ; 17 ; 16 ; 49 ; 57 have recognized this problem but failed to fully address it. SQA3D 33 provides a text description of an ego situation and a QA pair that requires spatial reasoning. While the text description implicitly conveys the ego pose, this approach introduces textual ambiguity and is not practically meaningful in real-world applications, where ego pose descriptions are not always available. View2Cap 57 proposes the Situation Grounding module, which takes in a frustum-cropped partial point cloud and regresses the associated camera pose. However, pose regression is far from textual reasoning with direction awareness, thus its effect is limited. Scene-LLM 16 enhances egocentric awareness using a two-step inference scheme, with first an egocentric frustum-cropped point cloud and then a scene-level point cloud. However, this approach is unnecessarily complex and uses proprietary code and data. Furthermore, these methods all set new benchmarks rather than altering existing ones, which still suffer from the problem of ill-posed directional definition. 2.2 3D LMMs Figure 2 : The offline data generation pipeline for PoseRecover . (a) Object annotations and camera poses are obtained from ScanNet-v2 12 . Camera poses and objects are downsampled for visibility. Zoom in for details. (b) PoseRecover exhaustively calculates the intersection rates between objects and camera frustums. (c) Visibility of the intersection is further validated with a z-buffer. (d) These intersection rates are saved and later sampled during training or inference to supplement ego poses to models. 2D-based 3D LMMs. Based on powerful 2D-pretrained networks, e.g. , CLIP 42 , DINOv2 39 , and LLAVA-Video 62 , this line of research focuses on projecting RGB-D image features back to 3D for understanding 19 ; 16 ; 63 ; 53 . 3D-LLM 19 explored a mixture of projection methods, including direct projection, SLAM 36 , and NeRF 34 . Video-3D-LLM 63 and Spatial-MLLM 53 parse the scene as an RGB-D video with pretrained video- or vision-language-models (VLMs). These methods introduce implicit information of ego pose in the images, therefore can be seen as natural implementations of our method. Detector-based 3D LMMs. To reconcile between computational complexity and loss of task-centric information, these methods choose 3D detection 45 or instance segmentation 44 to encode objects as LLM input tokens 21 ; 50 ; 22 ; 9 ; 18 ; 64 . For example, Chat-Scene 21 extracts object-centric features via a pretrained detector and passes these object features as tokens using a unified object identifier token. However, these methods are inherently limited by the detector capability and ignore background information, which is essential for reasoning in real scenarios. 3D-backbone-based 3D LMMs. This line of method processes the scene-level point cloud in a single forward pass which is then compressed and sent to LLM for reasoning 7 ; 14 ; 23 . LL3DA 7 encodes scene-level features through a Q-Former 30 for natural human interaction. Inspired by superpoint transformers 28 ; 29 ; 46 , 3D-LLAVA 14 completes point cloud processing and compression simultaneously using the cluster centers, i.e. , superpoints, as tokens. Due to natural processing of scene contexts, these methods achieve state-of-the-art performance but are highly contingent on the point cloud encoder design. 3 Data Pipeline Existing point cloud benchmarks, including ScanQA, ScanRefer, Multi3DRefer, SQA3D, Scan2Cap, and Nr3D, contain 40% to 95% direction-critical queries according to our analysis, yet they omit the camera pose information required to determine the ego agent’s directional context. As analyzed in Section 1, this omission renders a significant portion of questions ill-posed with respect to egocentric or allocentric direction understanding. To address this, we introduce PoseRecover, a fully automatic pipeline that recovers ego poses for all text–scene pairs. We first introduce the question analysis to establish our motivation, then introduce the offline PoseRecover pipeline as depicted in Figure 2 , which matches the question-specific ground-truth object annotation with the camera frustums, and finally introduce two pose selection strategies during online training or inference. 3.1 Question-Level Analysis We first quantify the degree of directional ambiguity in existing datasets. An open-source large language model (GPT-OSS-20B 38 ) is prompted with a series of question–answer pairs and tasked to determine whether answering the question requires explicit lateral direction reasoning (e.g., “left”, “behind”, “in front of”) 51 . We kindly refer readers to C for the full prompt and B.2 for detailed results. This binary judgment establishes the direction-critical subsets of existing datasets, which guide the downstream pose recovery stage. Across datasets, we find that between 40% to 95% of the questions require directional reasoning and are therefore ill-posed without ego-pose supervision. 3.2 Pose Supplementation via PoseRecover Given the intrinsic and extrinsic parameters of the raw RGB-D sequences in ScanNet-v2, we compute the 3D camera frustums (a visible pyramid with near and far cutoff) and measure their spatial intersection with question-related objects, and save all normalized intersection values into a pose list. Multiple forms of object annotations are supported: Figure 3 : Three viable designs for PoseAlign . We explore three mutually exclusive designs to incorporate ego poses into the vanilla model in (a): 1) PoseAlign-Transform that shifts point clouds to the ego reference frame in (b); 2) PoseAlign-Embed that encodes ego poses into point cloud features in (c); 3) PoseAlign-Prompt that integrates ego poses into the text prompt in (d). The projection layer and the LoRA 20 weights of the LLM are trained with instruction-tuning. Segmentation-based. For segmentation masks, every point in the point cloud is projected into the image plane using the camera intrinsic and extrinsic, forming a Z-buffer following Equations 1 - 3 : \displaystyle\begin{aligned} \qquad(x^{\prime}_{i},y^{\prime}_{i},z^{\prime}_{i})^{\intercal}=\mathcal{R}^{-1}(p_{i}-t),\end{aligned} (1) \displaystyle\begin{aligned} \qquad(u_{i},v_{i},1)^{\intercal}=\lfloor K(x^{\prime}_{i},y^{\prime}_{i},z^{\prime}_{i})^{\intercal}/z^{\prime}_{i}\rfloor,\end{aligned} (2) \displaystyle\begin{aligned} \qquad&Z_{\mathcal{P}}^{u_{i},v_{i}}=\min_{j|(u_{j},v_{j})=(u_{i},v_{i})}{(z^{\prime}_{j})},\\ \qquad&\text{s.t.}\ p_{i},p_{j}\in\mathcal{P},0\leq u_{i} ’ token quality generated by the LLM 14 . We conclude that, given their potential to improve performance, ego-poses are undoubtedly good additions to current 3D LMM pipelines. Is LLM-as-judge a good metric? As discussed in section 5.1 , a good language-task metric should be consistent under verbal randomness as confirmed by Table 1 . LLM-as-judge accuracy is consistently higher across all our modifications than the respective baselines, while other metrics tend to fluctuate. It is also robust to the judgment model, as shown in Tables 1 and 6 . We conclude that LLM-as-judge is a sufficiently good metric for this task. Design ScanRef Multi3DRef ScanQA Scan2Cap mIoU \uparrow mIoU \uparrow L-A \uparrow L-A \uparrow Baseline 42.6 48.1 45.7 28.1 Baseline PoseAlign-T(Top) 37.5 41.5 43.0 23.5 Random Pose 39.0 44.3 44.7 25.8 PoseAlign-T(Clip X=0.3) 55.4 54.3 47.3 31.4 PoseAlign-T(Top) 68.5 60.2 47.5 29.7 PoseAlign-E(Top) 43.2 49.3 43.4 23.5 PoseAlign-P(Top) 44.2 49.4 44.9 28.1 Table 2 : Ablation experiment on 3D-LLAVA . ‘Baseline PoseAlign-T’ is the performance of the baseline model on PoseAlign-T data, where the input point cloud is transformed to the camera location following Equation 8 . ‘Random Pose’ uses random camera poses instead of those found by PoseRecover. 5.3 Other Results Ablation on 3D-LLAVA. We compare the effect of different PoseAlign designs on 3D-LLAVA in Table 2 . Incorporating ego-pose information clearly benefits spatial reasoning, as the proposed PoseAlign-Transform consistently outperforms all other variants on refer segmentation tasks, improving mIoU by up to 68.5\% (\Delta 60.8\%) on ScanRefer and 60.2\% (\Delta 25.2\%) on Multi3DRefer. The Clip strategy ( X=0.3 ) for PoseAlign-Transform further stabilizes performance metrics across benchmarks by purging inconsistent views. In contrast, encoding poses as text (PoseAlign-Prompt) or projection features (PoseAlign-Embed) offers limited or inconsistent gains. Additionally, test performance of a well-trained 3D-LLAVA baseline model on PoseAlign-Transform data (Baseline PoseAlign-T) significantly degrades all metrics, which gives two important insights: 1) the segmentation model never uses the pose data as a shortcut because it is frozen, and 2) the current performance increments are brought purely by better ‘ < SEG > ’ tokens. Using random poses also decreases model performance slightly below baseline, showcasing the importance of PoseRecover design. We conclude that PoseAlign-Transform with the Clip strategy where X=0.3 is the most balanced and robust way of introducing directional awareness into 3D LMMs. ScanRef Multi3DRef ScanQA Scan2Cap X mIoU \uparrow mIoU \uparrow L-A \uparrow L-A \uparrow 0.0 55.1 54.2 47.3 30.1 0.1 55.2 54.3 46.6 30.8 0.2 54.8 54.3 47.3 31.2 0.3 55.4 54.3 47.3 31.4 0.4 55.4 54.3 46.9 30.9 0.45 55.2 54.3 46.5 31.4 0.49 55.1 53.9 46.9 31.1 Table 3 : Parameter tuning experiment for Clip Ratio X of PoseAlign-Transform on 3D-LLAVA. Parameter tuning. Table 3 analyzes the sensitivity of the Clip Ratio X in PoseAlign-Transform, which discards the top and bottom X\% of views based on object–frustum intersection during selection of the mission-critical pose. Despite theoretical concerns about opposite-view errors discussed in Figure 4 , the model performance remains stable across datasets for 0.0\leq X\leq 0.49 with a slight hill in the mediocre Clip Ratio ranging from 0.2\leq X\leq 0.4 , demonstrating the robustness of the method to this hyperparameter. This is because a smaller variance in viewing directions indeed enhances pose coherence but destroys data diversity, while higher variance in viewing directions adds noise to the model but also serves as effective data augmentation as discussed in section 5.1 . Moderate clipping (around X=0.3 ) yields the best balance, slightly improving results on ScanQA and Scan2Cap to 47.3\% and 31.4\% , respectively. Therefore, X=0.3 is adopted as the default value for all other experiments. Model Evaluated on ScanQA ( 46.7\% Direction-critical) Scan2Cap ( 89.7\% Direction-critical) C \uparrow B-4 \uparrow M \uparrow R \uparrow L-A \uparrow C \uparrow B-4 \uparrow M \uparrow R \uparrow L-A \uparrow 3D-LLAVA Full Val set 95.4 16.3 18.9 44.6 45.7 77.4 36.4 26.93 57.4 28.1 Direction-critical subset 91.6 18.0 17.8 41.0 37.8 76.8 36.8 26.94 57.4 28.0 Complementary subset 97.6 11.9 20.1 47.7 52.4 82.7 32.6 26.88 56.9 29.6 3D-LLAVA+ PoseAlign-T Full Val set 99.8 (+4.4) 17.3 (+1) 19.7 (+0.8) 46.5 (+1.9) 47.3 (+1.6) 76.1 (-1.3) 37.1 (+0.7) 27.11 (+0.18) 57.6 (+0.2) 31.4 (+3.3) Direction-critical subset 96.1 (+4.5) 19.6 (+1.6) 18.7 (+0.9) 43.1 (+2.1) 40.3 (+2.5) 75.9 (-0.9) 37.5 (+0.7) 27.12 (+0.18) 57.7 (+0.3) 31.3 (+3.3) Complementary subset 101.8 (+4.2) 11.5 (-0.4) 20.8 (+0.7) 49.4 (+1.7) 53.3 (+0.9) 77.8 (-4.9) 33.4 (+0.8) 27.10 (+0.22) 56.9 (+0.0) 32.6 (+3.0) Table 4: Performance on direction-critical question subset. Figure 5 : Qualitative results of direction-critical questions for 3D-LLAVA baseline (top row) and PoseAlign-Transform (bottom row). The XYZ axes of the world coordinate frame are colored with red , green , and blue , respectively. The baseline paradigm uses default world coordinates of ScanNet-v2, which are non-informative. Instead, the PoseAlign paradigm aligns the coordinate frame to the recovered ego pose, providing an anchor for robust spatial reasoning. Red text highlights wrong answers and green text highlights correct answers. Performance on direction-critical subsets. Table 4 compares the baseline 3D-LLAVA and its pose-aware variant (PoseAlign-Transform) on full validation sets, direction-critical subsets, and their respective complementary direction-agnostic subsets of ScanQA and Scan2Cap. Firstly, the direction-critical subsets identified via LLMs exhibit a notable performance drop in the baseline model, confirming that existing 3D LMMs struggle with questions requiring spatial orientation. Applying PoseAlign-Transform consistently improves all metrics, especially the LLM-as-judge accuracy, narrowing down the gap between the two subsets. The gains are especially pronounced on ScanQA, where PoseAlign-T achieves 96.1 (\Delta 4.9\%) CiDEr and 40.3\% (\Delta 6.6\%) LLM-as-judge accuracy. While the direction-critical subset is the best-effort judgment of a powerful LLM, some direction-critical questions may still leak into the complementary subset, causing performance to rise slightly there as well. We conclude that our PoseAlign design indeed enhances directional reasoning based on the substantial improvements, especially on the direction-critical subsets. 5.4 Qualitative Results As displayed in Figure 5 , baseline models struggle to perceive spatial relationships, where the default world coordinate only adds confusion and complicates spatial reasoning. In contrast, PoseAlign effectively avoids directional ambiguity thanks to the recovered ego pose, enabling robust and consistent spatial reasoning for 3D LMMs. 5.5 Limitations Despite being lightweight and generic, our method has several limitations, including assuming SLAM accuracy, the variety of training views, and a static environment. We kindly refer readers to section A.3 for a detailed discussion. 6 Conclusion We have proposed PoseRecover and PoseAlign, a pair of lightweight yet powerful techniques that transform existing 3D LMMs towards a rigorously defined direction-aware paradigm. To fix existing benchmarks, PoseRecover automatically reconstructs mission-critical ego poses by aligning object annotations with camera frustums derived from ScanNet RGB-D extrinsics, effectively correcting ill-posed direction-critical queries across existing datasets. Building upon these recovered poses, PoseAlign enables persistent directional awareness for 3D LMMs by transforming the input point cloud or encoded point cloud embeddings into the camera reference frame, thereby resolving ambiguity in ego direction and substantially improving spatial reasoning performance. Extensive experiments on multiple benchmarks and model architectures prove that PoseRecover and PoseAlign together unlock the latent potential of current 3D LMM architectures, achieving consistent performance gains without retraining the point cloud encoder. We advocate this framework as a simple, generic, and effective paradigm for advancing direction-aware 3D-language understanding of LMMs. Supplementary Material Appendix A Additional Notes A.1 Explanations Ill-posed questions without poses. At first glance, it may seem odd that questions involving left/right directions are ill-posed when ego pose is not present, especially to researchers in the fields of image or VLM research. A key distinction between images and 3D point clouds is that images are captured from an ego pose, which inherently defines left/right as the image left/right; However, a room-level point cloud is reconstructed using RGB-D video or even with 360 degree view LiDARs. Once the points are put into free 3D space, the concept of an ego reference frame disappears, now that the ego pose can no longer be trivially inferred from the room-level point cloud data itself. However, existing 3D LMMs receive only such a room-level point cloud and are asked to answer questions containing directional words, which is an ill-posed problem we aim to conquer in this paper. From implicit ego pose inference to explicit ego pose input. We could infer from past research that they want the 3D LMM to implicitly infer its ego position first, then derive the related answers based on that inferred ego pose. However, this regime is both (1) inherently ill-posed, and (2) adds unnecessary complexity for the 3D LMM to infer its ego position, while the robot’s ego pose is trivially available in realistic applications such as SLAM-capable embodied intelligence operating indoors. We therefore conclude that adding such pose data as additional input to the 3D LMMs does not change its application potential at all, and such conversion of regime is both practical and theoretically grounded. From an ego pose trajectory to a mission-critical pose. In practice, even when the ego trajectory is a free lunch, the agent does not always face the target object of the query and the mission-critical pose has to be assessed on-the-fly. However, we argue that such a mission-critical pose should be retrieved with other techniques rather than with the 3D LMM itself, as the task is essentially a cross-modal alignment problem with well-established solutions. For example, the agent can localize the mission-critical area by comparing the text embedding of the queried object with an offline precomputed semantic map of the indoor point cloud, e.g., using CLIP 42 or OpenScene 40 . After that, using our PoseRecover script, the agent can locate the most probable ego position from the ego pose trajectory, which can then be used to reason consistently and unambiguously. In this work, we are more interested in the prior problem of whether a mission-critical pose retrieved from ground truth is a necessary and beneficial addition to 3D LMM input, to rectify the ill-posed direction definitions , rather than a full end-to-end implementation; Our work is purely exploratory and there must exist some better way of integrating those poses into a coherent pipeline than with our PoseAlign. A.2 Discussions Compatibility with image-based methods. We deem all image-based methods to have implicitly used ego pose trajectory during the camera projection of the input images. Therefore, these image-based methods are already implicit implementations of the PoseAlign-Transform method and do not need further modifications. Compatibility with SQA3D. Notably, PoseAlign receives negative results on SQA3D as listed in Tables 7 - 8 . This is because among all training datasets, SQA3D is the only dataset that does not provide any object annotations, which means that PoseRecover pipeline cannot determine authentic poses for the questions, yielding random poses instead. This results in a slightly decreased performance compared to the baseline. We list supplementing object annotations and poses to the SQA3D benchmark as a major future work. Difference compared to the setting of LEO 22 . While both PoseRecover and LEO can utilize the ground truth object label, we want to point out that they are different settings. PoseRecover aims to rule out the directional ambiguity problem which requires a pose that can only be retrieved using object annotations. PoseAlign only feeds a pose facing the object into the point cloud encoder and the point cloud encoder is not tuned at all, which forbids formation of any direct object feature shortcuts like presented in LEO. We are fully aware that the performance increment of PoseAlign is brought by the normally unavailable pose provided by PoseRecover, but we see this as a testament to our hypothesis that a mission-critical pose derived prior to 3D LMMs is a great tool for ruling out reasoning ambiguity. The system in real action should be composed of two stages, where the agent first ‘imagines’ itself to be at the mission location and then derives the answers based on that reference frame. Furthermore, the way PoseRecover generates the pose is non-deterministic and the target pose is randomly chosen from all poses that can see the target object, which makes it a lot more realistic and forgiving than LEO setup as shown in Table 1 . More explanation can be found in section A.1 . ScanRefer Multi3DRef ScanQA Scan2Cap Model Noise Bound mIoU \uparrow mIoU \uparrow B-4 \uparrow B-4 \uparrow 3D-LLAVA / 42.6 48.1 16.3 36.4 3D-LLAVA+ PoseAlign-T (Top) 0°, 0m 68.5 60.2 17.3 37.1 15°, 0.2m 62.5 57.7 17.1 36.9 30°, 0.5m 53.2 55.1 17.0 36.6 Table 5 : Robustness to added uniform pose error. Figure 6 : Distribution of objects in world (L) and PoseAlign ego (R) coordinates on X-Y plane. While clustered in front view, objects still follow non-trivial distribution. Applicability to RGB-D data without camera extrinsics or LiDAR data. In practical embodied workflows, camera extrinsics are a free lunch from SLAM. For datasets lacking extrinsics, they can be recovered offline using ORB-SLAM2 or similar methods. We simulated SLAM pose noises in Table 5 which reveals the robustness of our method. For single-frame RGB-D or LiDAR data without extrinsics, which are too sparse for SLAM, we operate on single-frame point clouds, which naturally lie in the ego frame and thus constitute a natural application of our method. Does our setup introduce information leakage? First, we clarify that our benchmark is intentionally defined as an ego-conditioned understanding task, rather than standard free-view understanding. As such, ego alignment is part of the task specification rather than an unintended shortcut. Second, to reduce unintended bias, we avoid selecting the single best-matching viewpoint and instead sample from a diverse candidate set, resulting in non-trivial object spatial distributions in the horizontal plane (see Figure 6 ). Third, we freeze the point cloud encoder to prevent learning geometric shortcuts, further limiting overfitting to pose priors. Fourth, a PoseAlign-trained 3D-LLAVA model tested on random camera extrinsics results in ScanRefer mIoU dropping from 55.4% to 42.9% but still above vanilla 42.6%. So target facing is crucial to directional awareness but is not a shortcut, otherwise the performance would drop below vanilla. Interpretation of the performance superiority of PoseAlign-T. The pretrained 3D encoder is trained on cartesian coordinates which internalizes a concept of distance and directions based on the input coordinates. Compared to PoseAlign-T which guarantees consistency at input, PoseAlign-E and PoseAlign-P variants cannot guarantee alignment with (or even fight against) the encoder’s concept of direction, causing deterioration. A.3 Limitations While PoseRecover and PoseAlign provide a lightweight and general solution for introducing directional awareness into existing 3D-LMMs, several limitations remain. First, our method relies on the assumption that the 3D LMM is applied on embodied agents, where the availability and accuracy of camera extrinsics are guaranteed. Noise from inaccurate sensor calibration or synchronization is not considered. Second, the recovered poses are limited to the discrete set of views present in ScanNet and may not fully represent all possible viewpoints implied by the language queries, e.g., ‘standing on the kitchen counter’ or ‘crouching under the desk’. Third, although PoseAlign improves directional reasoning without retraining the point cloud encoder, it still assumes the encoder’s coordinate sensitivity is preserved, which may not hold for certain architectures employing rotation-invariant designs. Finally, our approach focuses on static indoor environments; extending it to dynamic or outdoor scenarios where ego motion and scene layout change continuously remains an open direction. Design ScanQA SQA3D Scan2Cap L-A \uparrow L-A \uparrow L-A \uparrow LL3DA 35.2 - 23.3 LL3DA+PoseAlign-T 35.4 - - LL3DA-S 36.3 - - LL3DA-S+PoseAlign-T 36.5 - - Chat-Scene 44.1 56.9 38.6 Chat-Scene+PoseAlign-E 44.7 56.8 39.7 3D-LLAVA 45.9 58.1 39.3 3D-LLAVA+PoseAlign-T 48.3 57.3 40.7 Table 6 : Comparison between baselines and respective PoseAlign variants using GPT-5-mini for LLM-as-judge. X ScanQA (val) SQA3D (test) Scan2Cap (val) C B-4 M R L-A EM EM-R L-A C@0.5 B-4@0.5 M@0.5 R@0.5 L-A None 85.6 15.6 17.8 40.4 43.9 52.8 56.1 56.9 74.0 34.5 26.8 56.4 26.6 0.0 83.8 14.2 17.4 39.7 43.1 53.4 56.2 57.2 37.2 27.9 23.4 52.0 9.76 0.1 86.7 14.9 17.9 40.8 44.3 52.6 55.7 56.6 74.1 34.7 26.9 56.5 27.0 0.2 87.2 15.1 18.1 41.1 44.7 53.3 56.1 57.1 75.5 35.0 27.1 56.7 26.9 0.3 87.5 15.5 17.9 40.9 44.8 53.3 56.2 57.1 74.9 34.7 26.9 56.5 26.9 0.4 82.7 14.3 17.0 39.2 42.7 52.8 55.4 56.1 27.5 25.6 22.4 50.6 6.2 0.45 84.3 14.0 17.5 39.9 42.9 52.8 55.4 56.3 28.2 26.4 22.7 51.1 6.8 0.49 83.6 15.4 17.4 39.5 42.8 52.8 55.5 56.3 60.7 32.2 25.8 54.9 19.0 Table 7 : Full parameter tuning experiment for Clip Ratio X of PoseAlign-Embed on Chat-Scene. X ScanQA (val) SQA3D (test) Scan2Cap (val) C B-4 M R L-A EM EM-R L-A C@0.5 B-4@0.5 M@0.5 R@0.5 L-A None 95.4 16.3 18.9 44.6 45.7 54.9 57.3 57.9 77.4 36.4 26.9 57.4 28.1 0.0 98.7 16.3 19.3 46.2 47.3 55.1 57.6 56.1 75.5 36.6 26.9 57.2 30.1 0.1 99.4 17.4 19.4 46.1 46.6 52.6 55.0 55.6 76.6 37.0 27.1 57.5 30.8 0.2 99.8 17.3 19.5 46.5 47.3 53.4 56.2 56.6 74.7 36.9 27.0 57.5 31.2 0.3 99.8 17.3 19.7 46.5 47.3 54.2 56.7 57.4 76.1 37.1 27.1 57.6 31.4 0.4 96.9 15.6 19.0 45.6 46.9 53.5 56.1 55.9 73.6 36.4 26.8 57.4 30.9 0.45 97.8 16.0 19.2 45.8 46.5 53.9 56.0 56.6 75.7 36.9 27.0 57.4 31.4 0.49 98.2 16.6 19.3 46.1 46.9 54.2 56.7 55.8 76.3 37.3 27.1 57.5 31.1 Table 8 : Full parameter tuning experiment for Clip Ratio X of PoseAlign-Transform on 3D-LLAVA. Appendix B Additional Experiments B.1 Other Experimental Setup Diversity of backbone architectures. We adopt four baseline models, including LL3DA 7 , LL3DA-SONATA, Chat-Scene 21 and 3D-LLAVA 14 . Their encoder backbones have distinct architectures ranging from BERT-style Vote2Cap-DETR operating on point patches 8 , self-supervised PointTransformerV3 with serialized voxels 54 , sparse convolutional neural network with sparse voxels 44 , and Omni Superpoint Transformer operating on superpoints 14 . They include both object-wise 7 ; 21 and point-wise 54 ; 14 feature projection, both with or without a Q-Former 13 . Their base LLMs include OPT-1.3B, Vicuna-v1.5-7B, and the LLM section of LLAVA-v1.5-7B. Consistent performance improvements despite such diversity proves the universal applicability of the PoseAlign modification. Hardware specification. All comparison methods and respective improved versions are trained on 4\times RTX 4090 24G. Batch sizes are trimmed to the same value within the same backbone to enable fair comparison. PoseAlign-Embed implementation. For the only backbone enhanced with PoseAlign-Embed, Chat-Scene has been a special case among all object-centric 3D LMMs because it passes both the location and the feature of an object into the projection module. In that sense, encoding the ego-pose feature in Chat-Scene reduces to transforming the object locations following Equation 8 . For other backbones where the object location is not passed to the LLM, similar modifications can be done by adding a sinusoidal position embedding of the transformed object location onto the projection feature. LLM-as-judge configuration. For GPT-OSS-20B, we use the officially-recommended vllm package, which packs information into harmony format. The reasoning effort is set to medium in the system prompt. A stronger assessment suite is enabled with GPT-5-mini using minimal reasoning budget. B.2 Additional Results Direction-critical question distribution. We apply the template in section C on the validation sets of three existing point cloud language datasets, ScanQA, Scan2Cap, and SQA3D. The respective portions of direction-critical questions are 46.7% on ScanQA, 89.7% on Scan2Cap, and 95.2% on SQA3D, respectively. The high ratio for the latter two datasets are due to their strong dependence on allocentric spatial relationships to define the caption target or the ego position. We conclude that a significant portion of existing 3D LMM benchmarks contain direction-critical questions which are ill-posed without ego poses. Results for LLM-as-judge with GPT-5. In addition to reporting accuracy judged by GPT-OSS-20B, we also provide judgements from the powerful GPT-5 with the same set of prompts, which are listed in Table 6 . Similar trends are observed as found in Table 1 , where PoseAlign variants improve LLM-as-judge metrics consistently across all baselines and the majority of datasets. Notably, PoseAlign achieves 48.3\% (\Delta 5.0\%) and 40.7(+1.4)\% (\Delta 3.6\%) accuracy on ScanQA and Scan2Cap, and achieving top-tier accuracy of 57.3\% on SQA3D. We also note that the GPT-OSS-20B scores (Table 1 ) are consistent with the GPT-5 results and both follow the same order, which means that these metrics are consistent and reliable. Full parameter tuning experiment on Chat-Scene and 3D-LLAVA. We provide all performance metrics of tuning the clip ratio X for two representative backbones, Chat-Scene and 3D-LLAVA, to examine the effect of X on directional alignment. As shown in Tables 7 and 8 , performance first improves with moderate clip ratios and then gradually declines beyond X=0.3 , exhibiting a clear unimodal trend except on Scan2Cap. Extremely small values ( X{<}0.1 ) cause overfitting to marginal pose deviations, while large values ( X{>}0.4 ) discard too many candidate poses, reducing diversity and robustness. Both models achieve their best balance between stability and discriminability near X{=}0.3 , where directional understanding improves across ScanQA, SQA3D, and Scan2Cap without sacrificing general reasoning ability. This consistency across two architectures validates the robustness of the proposed cut-ratio mechanism and supports the choice of X{=}0.3 as the default setting in all main experiments. Appendix C Prompt Designs Template for Direction-critical question analysis. The following template is for ScanQA dataset, while the prompts for other datasets are similarly constructed with different examples but the same prompt structure. ⬇ Your job is to look at a question asking about the interior of a room and a bunch of associated ground truth answers , and then assign the problem as either [ A : " NEED_LATERAL_DIRECTION ", B : " DO_NOT_NEED_LATERAL_DIRECTION "] in order to answer . First , I will give examples of each class , and then you will classify a new question . The following are examples of DO_NOT_NEED_LATERAL_DIRECTION questions . ‘‘‘ Question 1: What color is the chair in the kitchen ? Answer 1: " brown ", " dark brown ". Question 2: What is the top of the table ? Answer 2: " tv ", " television ". Question 3: What is placed next to the fridge ? Answer 3: " door ", " the beige door ". Question 4: Where are the two chairs in front of one another ? Answer 4: " end table closest to door ", " across from each other over table " ‘‘‘ These questions all DO_NOT_NEED_LATERAL_DIRECTION because : - Either they ask a different trait of the object than its direction ( e . g ., color , material , shape , size , etc .). - Or the direction is dependent on an absolute direction ( e . g ., " top ", " bottom ", " next to ", " between ", " end ") which can be implied without additional direction information . - Or like Question 4 where the reference frame is not needed or can be implied despite the existence of directional words like " in front of ", because two chairs facing each other will be facing each other no matter where you look at them from . - While the question may contain spatial words , it does not necessarily need the directional information to answer . The following are examples of question that NEED_LATERAL_DIRECTION . ‘‘‘ Question 1: How many brown chairs are on the left of the brown table ? Answer 1: "4", "4" Question 2: Where is the tall chair on the right of the table ? Answer 2: " to left of narrow table tv ", " next to 2 other chairs as third chair on right " Question 3: What part of the room has a fridge to its right ? Answer 3: " right corner wall adjacent to tv ", " corner with tv alongside same wall " Question 4: Where is the beige wooden desk placed ? Answer 4: " up against wall ", " at front of class " Question 5: What is on the front of the brown table ? Answer 5: " tv ", " chair " ‘‘‘ These predicted answers all NEED_LATERAL_DIRECTION because : - A relative direction word ( e . g ., " left ", " right ", " front ", " back ", " against ") is present in the question or any of the answers . - Even if the question asks about directions relative to an object , in some cases the reference frame still cannot be implied , e . g ., cylindrical or symmetric objects like dishes , tables , beds , etc , just like Question 5 above . - The question may be asking about a different trait of the object than its direction ( e . g ., color , material , shape , size , etc .), but the question cannot be answered without the directional information . Also note the following things : - For classifying the questions where the answers or question may seem odd , ignore that and just focus on whether the question can be answered without directional information . - The answers may not coincide with each other , but that is okay . There can be multiple objects to the left or right of something . Just focus on whether the question can be answered without directional information . Here is a new question . Simply reply with either NEED_LATERAL_DIRECTION or DO_NOT_NEED_LATERAL_DIRECTION . Do not ask any more questions to the user . ‘‘‘ Question : { question } Answers : { gt_answer } ‘‘‘ Classify the question as one of : A : NEED_LATERAL_DIRECTION B : DO_NOT_NEED_LATERAL_DIRECTION Just return the letters " A " or " B ", with no text around it . Template for LLM-as-judge. The following template is for LLM-as-judge on the result of ScanQA dataset, while the prompts for other datasets are similarly constructed with different examples but the same prompt structure. The templates can be found in the open-sourced code. ⬇ Your job is to look at a question , a ground - truth answer , and a predicted answer , and then assign a grade of either [" CORRECT ", " INCORRECT "]. First , I will give examples of each grade , and then you will grade a new example . The following are examples of CORRECT predicted answers . ‘‘‘ Question 1: What type of stool is to the left of the door ? Ground truth : " bar stool ", " black topped wooden stool " Predicted answer 1: wooden stool . Predicted answer 2: black topped bar stool . Predicted answer 3: wooden bar stool . Question 2: Where is the door of the refrigerator located ? Ground truth : " next to microwave ", " on refrigerator to right of door " Predicted answer 1: next to the microwave . Predicted answer 2: on the refrigerator to the right of the door . Question 3: What is in the bin next to the door ? Ground Truth : " trash for recycling ", " recycling bins " Predicted answer 1: trash can . Predicted answer 2: litter waiting for recycling . Predicted answer 3: waste . ‘‘‘ These predicted answers are all CORRECT because : - They fully contain the important information in one of the ground truths . Aliases and synonyms are acceptable . - They do not contain any information that contradicts the ground truth . - Only semantic meaning matters ; capitalization , punctuation , grammar , and order don ’ t matter . - Hedging and guessing are permissible , provided that the ground truth is fully included and the response contains no incorrect information or contradictions . The following are examples of INCORRECT predicted answers . ‘‘‘ Question 1: What is to the right of the refrigerator and above the kitchen cabinet ? Ground truth : " microwave ", " microwave " Predicted answer 1: kitchen counter . Predicted answer 2: kitchen sink . Predicted answer 3: water tap . Question 2: Where is the door of the refrigerator located ? Ground truth : " next to microwave ", " on refrigerator to right of door " Predicted answer 1: on the microwave . Predicted answer 2: on the door next to the refrigerator . Predicted answer 3: to right of refrigerator . Predicted answer 4: on the microwave above the refrigerator . Predicted answer 5: on the refrigerator to left of the door . Question 3: How much larger is the blue recycling bin compared to the beige trash can ? Ground truth : " twice as wide ", " about double size " Predicted answer 1: larger . Predicted answer 2: blue recycling bin is larger than beige trash can . Predicted answer 3: blue recycling bin is much larger than beige trash can . ‘‘‘ These predicted answers are all INCORRECT because : - None of the information in the ground truth is included in the answer . - Or the predicted answer contradicts facts contained in the question and the ground truth . Pay extra attention to object relations , e . g ., the door of the refrigerator cannot be on the microwave or another door . Having the correct words is not enough ; the meaning must be correct . - The answer shies away from the question and does not provide any useful information . Also note the following things : - For grading questions where the ground truth contains a number , the predicted answer needs to be correct to the counting figure in the ground truth , but can be relaxed by +-1 when describing rough figures like relative size . For example , consider a question " How much larger is the brown cabinet compared to the purple stool ?" with ground truth "5 times larger ". - Predicted answers "5", "4", and "6" are all CORRECT . - Predicted answers "2", "3" and "8" are INCORRECT . - Predicted answers " larger " and " more than 2" are considered INCORRECT because they provide no useful information . - Do not punish predicted answers if they omit information that would be clearly inferred from the question . - For example , consider the question " What surface texture is the wooden cardboard ?" and the gold target " wooden matte ", " brown flat ". The predicted answer " flat " would be considered CORRECT , even though it does not include " wooden ", since it is clear from the question that the cardboard is wooden . Here is a new example . Simply reply with either CORRECT or INCORRECT . ‘‘‘ Question : { question } Ground truth : { ground_truth } Predicted answer : { predicted_answer } ‘‘‘ Grade the predicted answer of this new question as one of : A : CORRECT B : INCORRECT Just return the letters " A " or " B ", with no text around it .