Computer Vision for Objects used in Group Work: Challenges and Opportunities
Changsoo Jung, Sheikh Mannan, Jack Fitzgerald, Nathaniel Blanchard · 2025
Read Computer Vision for Objects used in Group Work: Challenges and Opportunities by Changsoo Jung, Sheikh Mannan, Jack Fitzgerald, Nathaniel Blanchard (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-00224.txt.
Introduction
Changsoo Jung and three co-authors study how computers can track hand-held objects while students work in groups, in a 2025 paper. Hands-on group tasks help schoolchildren learn, and smart classroom tools need to follow what students do with real objects. That calls for estimating each object's full position and tilt in 3D from video, a task called 6D pose estimation.
The team built a video data set, FiboSB, of trios solving a task with small cubes and a balance scale. Filming whole groups from afar makes the cubes tiny in frame, which strains current methods. The team tested four leading pose estimators on the set and mapped where they fail. The paper frames the data set as both a benchmark and a call to build trackers that can keep up with busy, real classrooms.
Catalog record
| CREATOR | Changsoo Jung, Sheikh Mannan, Jack Fitzgerald, Nathaniel Blanchard |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 15 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-00224
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Computer Vision for Objects used in Group Work: Challenges and Opportunities Changsoo Jung, Sheikh Mannan, Jack Fitzgerald, Nathaniel Blanchard Abstract Interactive and spatially aware technologies are transforming educational frameworks, particularly in K-12 settings where hands-on exploration fosters deeper conceptual understanding. However, during collaborative tasks, existing systems often lack the ability to accurately capture real-world interactions between students and physical objects. This issue could be addressed with automatic 6D pose estimation, i.e., estimation of an object’s position and orientation in 3D space from RGB images or videos. For collaborative groups that interact with physical objects, 6D pose estimates allow AI systems to relate objects and entities. As part of this work, we introduce FiboSB, a novel and challenging 6D pose video dataset featuring groups of three participants solving an interactive task featuring small hand-held cubes and a weight scale. This setup poses unique challenges for 6D pose because groups are holistically recorded from a distance in order to capture all participants — this, coupled with the small size of the cubes, makes 6D pose estimation inherently non-trivial. We evaluated four state-of-the-art 6D pose estimation methods on FiboSB, exposing the limitations of current algorithms on collaborative group work. An error analysis of these methods reveals that the 6D pose methods’ object detection modules fail. We address this by fine-tuning YOLO11-x for FiboSB, achieving an overall m A P 50 mAP_{50} of 0.898. The dataset, benchmark results, and analysis of YOLO11-x errors presented here lay the groundwork for leveraging the estimation of 6D poses in difficult collaborative contexts. 1 Introduction Figure 1: Training image comparison between MegaPose and our FiboSB dataset. The left two images are from the synthetic training data of MegaPose [ 18 ] , while the images on the right represent our FiboSB training data. The right-most image under FiboSB is zoomed in for illustration, and evaluations were conducted on the original images. Recently, extensive research has shown the feasibility of an AI agent for collaborative groups in K-12 education [ 12 , 22 ] . However, these breakthroughs are typically driven by dialogic understanding [ 8 , 21 ] and largely ignore physical interactions across students and physical objects in these real-world settings. This gap is especially visible in collaborative activities that require hands-on manipulation, where solutions often rely on limited modalities such as verbal and textual input data rather than spatial and temporal clues. As a result, any AI system can only receive partial insight into group dynamics and task progress, limiting its ability to provide timely interventions and personalized support. From an educational context, effective collaborative learning involves not only individual problem-solving but also group cooperation and physical interaction. Interactions during teamwork can include pointing, assembling, and manipulating objects to describe and prove concepts for elaborating deeper understanding between peers. Here, an AI-powered agent can play a crucial role; by monitoring how students handle and position objects, an agent can offer context-aware guidance as feedback and assessment during group work. Particularly in K-12 settings, where students vary in their developmental levels and learning styles, an agent that “sees” and “understands” these interactions can adapt instructions, prompt collaborative discussions, and highlight suggestions by each student’s progress in the task. Beyond facilitating collaboration, a precise awareness of physical space and object relationships is essential for understanding many concepts in education, especially in spatial reasoning. 6D pose estimation entails the ability to track the 3D position and orientation of objects 1 1 1 The term 6D comes from the need to predict the 3D translation and the 3D rotation of the object’s pose , enabling detailed monitoring, such as how objects move or rotate and how they relate to each other in the environmental setting. This capability is particularly beneficial for younger or lower-grade students, who often depend on visual and tactile experiences to understand geometrical concepts: distance, shape, and scale. By recognizing real-time positions and orientations of objects, we can better measure students’ performance, observe their decision-making processes, and provide elaborate feedback. Educational studies have started incorporating tactile objects and 3D elements in lessons, benefiting students who lack spatial reasoning skills [ 1 , 23 ] , providing a compelling opportunity to integrate 6D pose estimation into collaborative learning. In this paper, we investigate the potential of 6D pose estimation in supporting K-12 collaborative tasks and outline how this technique can enhance spatial understanding, foster teamwork, and ultimately improve the overall learning experience. To complement and extend the modalities studied in prior work [ 3 , 16 , 25 ] , we introduce a novel 6D pose dataset called Fibonacci Small Blocks (FiboSB), which is adapted from the Weight Task Dataset (WTD) [ 15 ] . In summary, the key contributions of this study are as follows. • Collection of a novel 6D pose dataset specifically designed for educational settings involving collaborative group tasks. • Exploration of baseline performance of multiple state-of-the-art 6D pose methods on our dataset, highlighting the distinct challenges posed by collaborative group scenarios. 2 Dataset: Fibonacci Small Blocks (FiboSB) To demonstrate the potential of using 6D pose estimation for collaborative group work in an educational setting, we introduce the FiboSB dataset. FiboSB is based on the Weights Task Dataset (WTD) [ 15 ] , which involves a group of triads interacting with six colored blocks (two 1.5 inch^{3} and four 2.0 inch^{3} ) and a weight scale to determine the weights of each block. The group is initially given the weight of one block, and then they are tasked with finding the weights of the rest of the colored blocks, which follows a Fibonacci sequence. In FiboSB, we annotate the 6D poses of the colored blocks in the WTD so that we can train and evaluate 6D pose estimation models. Predicaments in FiboSB Detecting small blocks in the collaborative group work scene is a challenge (the two images on the right of Figure 1 ). Occlusions among the blocks make the visibility of each block worse, as shown in the last column of Figure 1 , which frequently occur during the group task. The FiboSB dataset contains 25,381 annotated frames across 10 groups, with the number of frames per group ranging from 1,257 to 3,967. From the annotated frames, 133,263 object instances were annotated in total. On average, each frame contains 5.25 objects, indicating that multiple colored blocks frequently appear together. The blocks are often placed close to other blocks, leading to frequent occlusions that increase the complexity of the dataset. This obstacle leads to difficulties in estimating the exact object positions and their orientations. In addition, one pixel off on an annotation or prediction results in a huge error for small objects. For these reasons, recognizing the blocks and estimating their precise 6D pose predictions are critical requirements in the FiboSB dataset. FiboSB vs. Other 6D Pose Datasets To our knowledge, FiboSB is the first 6D pose dataset aimed at collaborative tasks in educational settings. Many 6D pose datasets [ 2 , 4 , 9 ] are targeted at various obstacles such as textureless and transparent objects [ 11 , 14 ] . Figure 1 shows examples of synthetic training data for MegaPose; the images contain everyday objects, such as clocks, bottles,