Pose Estimation for the Inspecta Project
Williams, Kyle Adrian, Smartt, Heidi Anne · 2025
Read Pose Estimation for the Inspecta Project by Williams, Kyle Adrian, Smartt, Heidi Anne (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/doe-osti-3018752-pose-estimation-for-the-inspecta-project.txt.
Introduction
Kyle Williams, a Georgia Tech doctoral candidate working at Sandia, wrote this December 2025 workflow report, SAND2026-16910R, with Heidi A Smartt. The Inspecta project inspects sealed objects, and pose estimation tells the software where each photograph was taken from. Williams's pipeline starts with recorded video of a metal cup seal. COLMAP recovers camera positions from the frames. Multi-view stereo then builds the scene geometry, and axes are placed on the model. A matching CAD model is generated for reference.
SAM2 segmentation isolates the object in each frame. Neural radiance field training follows, with dataset interpolation filling gaps. The product is 308 photographs labeled with position and orientation relative to the seal. Inspectors get training data whose geometry is known, not guessed. The report is written as a repeatable recipe, step by step, for the next analyst who must pose a new object.
Catalog record
| CREATOR | Williams, Kyle Adrian, Smartt, Heidi Anne |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| LICENSE | Public Domain |
| RIGHTS | PUBLIC DOMAIN |
| SOURCE | DOE OSTI |
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SAND2026-16910R Pose Estimation for the Inspecta Project Kyle Williams Georgia Institute of Technology (current PhD candidate) Sandia National Laboratories Tracking Number: 1795463 December 2025 Sandia National Laboratories is a multimission laboratory managed and operated by National Technology and Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525. 2 CONTENTS 1. Introduction ................................................................................................................................................. 5 2. Record Video ............................................................................................................................................... 6 3. Run COLMAP ............................................................................................................................................ 7 4. Run Multi-view Stereo and Place Axes .................................................................................................... 9 4.1. CAD Generation............................................................................................................................... 9 4.2. Axis Placement ................................................................................................................................ 10 5. SAM2 Segmentation ................................................................................................................................. 12 6. NERF Training and Dataset Interpolation ........................................................................................... 13 6.1. Neural Radiance Fields .................................................................................................................. 13 6.2. Dataset Generation ......................................................................................................................... 13 7. Pose Estimator Training .......................................................................................................................... 15 7.1. Pose Representation ....................................................................................................................... 15 7.2. Network Architecture .................................................................................................................... 15 7.3. Results............................................................................................................................................... 15 8. Conclusions ................................................................................................................................................ 19 LIST OF FIGURES Figure 1: Selected frames from the input video. ........................................................................................... 7 Figure 2: Example COLMAP reconstruction views. .................................................................................... 8 Figure 3: Multi-view stereo derived CAD file..............................................................................................10 Figure 4: CAD axis placement. ......................................................................................................................10 Figure 5: COLMAP photos aligned to object-centered reference frame. ...............................................11 Figure 6: Labeled and segmented photos captured by camera and processed with SAM2. .................12 Figure 7: Two synthetic NERF rendered views of the metal cup seal.....................................................13 Figure 8: Synthetic dataset. .............................................................................................................................14 Figure 9: Training and testing loss curves. ...................................................................................................16 Figure 10: Training set performance. ............................................................................................................16 Figure 11: Testing set performance. ..............................................................................................................17 Figure 12: Validation set performance. .........................................................................................................18 4 1. INTRODUCTION In this addendum report we take the lessons learned from our initial generic object pose estimation research described in (Smartt, et al., 2025)1 and apply it to metal cup seals, which are attached to items after an international nuclear safeguards inspection (i.e., metal cup seals may be attached to containers of spent nuclear fuel after an inspector has measured and verified the material). We have also refined our software pipeline and training techniques since the final Inspecta report and will describe that process here. The objective of this task is to allow a robot to acquire images of the engraved seal characters to confirm that the correct seal is attached to the correct container. However, the seal may be attached in any orientation and therefore the robot manipulator arm with a camera needs to align with the normal face of the seal. 1 Note that the pose estimation R&D began at SNL, but the lead researcher started a PhD program at the Georgia Institute of Technology. Thus, the pose estimation task was moved to the Georgia Institute of Technology under contract to Professor Sehoon Ha and PhD candidate Kyle Williams. 5 2. RECORD VIDEO Our pipeline first takes a cellphone video of the object of interest as input. Backgrounds with 3D structure and fine grain texture produce the best results downstream, hence the colorful hand pan behind our metal cup seal. See Figure 1 for fifteen of the initial video-derived photos. 6 3. RUN COLMAP We next run COLMAP (COmputer vision Localization and MAPping)2 on the input video. We use the COLMAP GUI or the following command from nerfstudio3: ns-process-data video --data ./seal3.mp4 --output-dir ./records3/output Four views of the 3D scene generated by COLMAP are shown in Figure 2. The red pyramidal objects in the scene represent the position and orientation from which each frame of the input video was captured. The reference frame of this scene is meaningless however, hence the need for additional steps. Figure 1: Selected frames from the input video. 2 COLMAP is an open-source Structure-from-Motion (SfM) and Multi-View Stereo (MVS) software that creates 3D models and estimates camera positions and poses from a collection of still images. 3 Nerfstudio is an open-source modular framework for creating and training Neural Radiance Fields (NERFs). https://docs.nerf.studio 7 Figure 2: Example COLMAP reconstruction views. 8 4. RUN MULTI-VIEW STEREO AND PLACE AXES 4.1. CAD Generation Given the images and their position and orientation of capture, a crude CAD file can be generated using multi-view stereo (MVS)4. We use the COLMAP GUI or the following commands: 1. Undistort images colmap image_undistorter \ --image_path images \ --input_path colmap/sparse/0 \ --output_path dense \ --output_type COLMAP 2. Multi-view stereo (PatchMatchStereo) colmap patch_match_stereo \ --workspace_path dense \ --workspace_format COLMAP \ --PatchMatchStereo.geom_consistency true 3. Fuse depth maps into dense point cloud colmap stereo_fusion \ --workspace_path dense \ --workspace_format COLMAP \ --input_type geometric \ --output_path dense/fused.ply 4. Poisson meshing colmap poisson_mesher \ --input_ path dense/fused.ply \ --output_path dense/mesh.ply The CAD file is shown in Figure 3. The image on the left shows the raw CAD file generated by the MVS process. The image on the right shows a trimmed version, performed in MeshLab5. Notice the overall structure is sound but the visual quality of the CAD file is too low to be directly used for pose estimation. 4 Multi-view stereo (MVS) is a computer vision technique that creates a dense 3D model of a scene from multiple 2D images taken from different viewpoints. 5 MeshLab is a free, open-source software used to process and edit 3D triangular meshes. https://www.meshlab.net 9 Figure 3: Multi-view stereo derived CAD file. 4.2. Axis Placement The axes of the CAD file are at an arbitrary location selected by COLMAP. They must be manually placed at the origin of the metal cup seal and aligned to its axes. The exact placement is up to the discretion of the user. This is the only manual step in this otherwise automated pose estimation pipeline. The unaligned (left) and aligned (right) CAD files are shown in Figure 4. Figure 4: CAD axis placement. The homogeneous (H) matrix used to rotate the CAD file to its new object centric reference frame is exported from MeshLab and shown here. 0.923433 −0.366397 −0.114119 0.0235866 −0.235673 −0.776134 0.584871 0.276465 H= −0.302867 −0.513195 −0.803058 3.65347 0 0 0 1 We then use Python to apply this H matrix directly to the COLMAP camera positions and orientations. We apply it to the object point cloud as well, though this is only for visual debugging purposes. The important part is to arrive at a dataset of photos, which are labeled with their position and orientation relative to the metal cup seal. We have 308 such photos in our dataset. This is shown in Figure 5. Each red frame describes the position and orientation of one of the photos exemplified in Figure 1. 10 Figure 5: COLMAP photos aligned to object-centered reference frame. 11 5. SAM2 SEGMENTATION In this section we use the foundational Segment Anything Model 2 (SAM2)6 from Meta to segment out the object from its background. SAM2 typically requires a mouse click from the user to identify the object of interest. We use the object center identified in the previous step to automate this process. We need some background texture for the downstream steps in the pipeline, so we don’t do a pure background elimination. Rather, we set the red channel of the background to one-fourth its original value, while maxing out the green and blue channels. The results are shown in Figure 6 for select images. Figure 6: Labeled and segmented photos captured by camera and processed with SAM2. After SAM2, we also use a basic OpenCV style segmentation to