Can ChatGPT Learn My Life From a Week of First-Person Video?

Keegan Harris Machine Learning Department Carnegie Mellon University [email protected] · 2025

Read Can ChatGPT Learn My Life From a Week of First-Person Video? by Keegan Harris Machine Learning Department Carnegie Mellon University [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-03857.txt.

Introduction

Keegan Harris, a doctoral student in machine learning at Carnegie Mellon University, tests whether a chatbot can learn his life from a week of first-person video. He wore a camera headset for fifty-four hours across one week. Summaries of the footage were made at minute, hour, and day scales.

Two models were then fine-tuned on that summary ladder. Harris probed the tuned models with questions about himself. Both learned basics such as rough age and sex. The larger model also deduced his city, his university, his handedness, and his pet cat. Yet both made up names for people in the footage. The study shows both the reach and the rot of personal learning by machines. It is a caution for the coming wave of wearable recorders.

Catalog record

CREATORKeegan Harris Machine Learning Department Carnegie Mellon University [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-03857

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Can ChatGPT Learn My Life From a Week of First-Person Video? Keegan Harris Machine Learning Department Carnegie Mellon University [email protected] Abstract Motivated by recent improvements in generative AI and wearable camera devices (e.g. smart glasses and AI-enabled pins), I investigate the ability of foundation models to learn about the wearer’s personal life through first-person camera data. To test this, I wore a camera headset for 54 hours over the course of a week, generated summaries of various lengths (e.g. minute-long, hour-long, and day-long summaries), and fine-tuned both Gpt-4o and Gpt-4o-mini on the resulting summary hierarchy. By querying the fine-tuned models, we are able to learn what the models learned about me. The results are mixed: Both models learned basic information about me (e.g. approximate age, gender). Moreover, Gpt-4o correctly deduced that I live in Pittsburgh, am a PhD student at CMU, am right-handed, and have a pet cat. However, both models also suffered from hallucination and would make up names for the individuals present in the video footage of my life. 1 Introduction The rise of wearable technologies such as smart glasses, AI-enabled pins, and other always-on camera devices signals a shift in how individuals might interact with artificial intelligence in daily life. Companies like Google and Meta are not only producing wearable hardware that captures egocentric video, but are also at the forefront of developing the foundation models that might consume this data. As these technologies mature, a natural question arises: What can AI learn about a person from passively collected, first-person video footage? This paper explores the above question through an experiment in personal data collection and model fine-tuning. Specifically, I wore a camera headset for roughly one week and used the resulting footage to fine-tune Gpt-4o and Gpt-4o-mini . The goal of this experiment is twofold: First, I wanted to assess whether an off-the-shelf language model could learn meaningful information about a person from a relatively small amount of first-person camera data. Second, I wanted to determine whether this process could be done cheaply, as companies will only have an incentive to learn from individuals’ personal video data if it can be done in a simple and cost-effective way. As a result, I set a total budget of \$100 for this project and used only my laptop for data preprocessing. In order to adhere to the \$100 budget constraint, I used OpenAI’s API to fine-tune Gpt-4o and Gpt-4o-mini on time-stamped text summaries generated from the raw camera footage. By doing next-token prediction on minute-long, 10 minute-long, hour-long, and day-long summaries of my life, the models were able to learn basic facts about me without being explicitly told any information. By examining the outputs of these personalized models (which I collectively refer to as KeeganGPT), we can evaluate what they did and did not learn about my life, how they hallucinate about personal data, and what this implies about the future of AI-powered wearables. Paper Outline. After discussing related work in Section 1.1 , I overview how video data was collected, how the hierarchy of summaries was generated, and how the models were fine-tuned in Section 2 . In Section 3 , I detail what happened when I queried the fine-tuned versions of Gpt-4o and Gpt-4o-mini . Finally, Section 4 discusses directions for future research and implications for the future of AI-powered wearable technology. 1.1 Related Work Figure 1 : Left: First-person view of me petting my cat. Right: Driving later that night. Both images were captured using the ORDRO EP8 action camera. Egocentric Video and Lifelogging. Prior work in computer vision has explored how first-person video can be used to understand human activity, intention, and context. Datasets such as Ego4D [ 4 ] and EPIC-KITCHENS [ 2 ] have enabled research in action recognition, video summarization, and object interaction in everyday settings. However, these efforts often focus on short, labeled clips and are not personalized. In contrast, this work uses egocentric video from a single individual’s life over one week as the dataset, and examines what can be learned without explicit annotation. LLMs as Personal Memory. Recent research has begun to explore the use of large language models as memory systems that can accumulate and recall personal information over time. Notable efforts include projects like Rewind [ 8 ] and MemGPT [ 9 ] , which integrate LLMs with retrieval systems to build persistent, evolving memories from user interactions. KeeganGPT differs by eschewing retrieval systems and relying solely on fine-tuning via next-token prediction on temporally structured summaries, with an emphasis on minimal compute, cost, and effort. Fine-Tuning for Personalization. While instruction tuning [ 7 ] and reinforcement learning from human feedback [ 1 ] are common methods for aligning models with broad user preferences, there has been less work which explores directly fine-tuning language models on a single person’s daily experience. Efforts like QLoRA [ 3 ] aim to make personalization feasible, but often require more supervision or higher compute budgets than used here. In contrast, KeeganGPT demonstrates that even lightweight fine-tuning on auto-generated summaries can teach LLMs some basic facts about a person’s life. As personalized models become more common, so too do questions about what they remember—and misremember. Prior studies on model hallucination (e.g. [ 5 , 6 ] ) highlight the risks of fine-tuning without careful curation. In this work, hallucinations ranged from harmless (e.g. wrong birthday) to uncanny (e.g. inventing names for real people). 2 Experimental Setup To continuously record first-person camera data, I wore an ORDRO EP8 action camera, connected to an Anker power bank stored in a small bag worn around my waist. See Figure 1 for sample snapshots captured using the ORDRO EP8. I wore the headset more-or-less continuously for one week, with a few exceptions: I removed the headset while sleeping, when entering spaces where recording would be illegal or unethical (e.g. while on private property), and on occasions when I simply forgot to wear it. The footage was recorded to a 256 GB microSD card, which I backed up to my laptop every night. In total, I wore the headset for 62 hours over the course of a week, for an average of 8.86 hours each day. Unfortunately, a camera malfunction caused no video to be captured for 8 of those hours, resulting in a total of 54 hours of recorded video footage. The recorded footage was a reasonable representation of a normal week of my life. Activities recorded include watching television, playing Sudoku on my phone, doing research, baking a pizza in the oven, and driving to my mom’s house for dinner. All individuals who appeared in the videos gave explicit consent to being recorded. After backing up all video to my computer, I extracted one frame every 30 seconds, for a total of 6,480 still images. I used Gpt-4o-mini to generate a text summary of each image, using the system prompt “Describe this image from Keegan’s camcorder in 3-4 sentences, focusing on details and context.” I generated minute-long summaries using Gpt-4o-mini by including both image summaries in each minute in the prompt, along with instructions to combine the two text logs into one summary. Ten minute summaries were generated by giving Gpt-4o-mini ten one-minute summaries in

Read the complete text