Naturalistic driving data extraction and processing for studying driver head scanning behavior at intersections Authors: Shrinivas Pundlik, Seonggyu Choe, Patrick Baker, Chen-Yuan Lee, Naser Al-Madi, Alex R. Bowers, Gang Luo Published: 2026 ABSTRACT Naturalistic driving (ND) studies offer key insights about real-world driver behavior compared to lab and on-road studies along a fixed course. We describe an ND data processing pipeline for characterizing the scene and driver behavior at intersections. The in-car recording system recorded the vehicle’s speed, location, and videos of the cabin and the front scene. Intersection locations were approximately marked from the drive log, and corresponding cabin and scene videos were clipped for ± 100 m segments around the marked intersection locations. Scene videos were processed to extract intersection scene characteristics including the type of signage, vehicle maneuver, traffic density, and to precisely estimate the vehicle’s entry and exit from an intersection. Driver head pose in cabin videos was estimated using a custom-developed AI model for wide angle head turns. Evaluation involved 365 manually-annotated intersection instances from 6 vehicles driven in 3 US states. Signage and maneuvers were correctly detected in 99% and 95% of cases, respectively. Median[IQR] intersection entry error was 1.1[0.3–4.2] meters, 0.23 [0.07–0.63] seconds; overlap between estimated and ground truth intersection bounds was 0.88[0.8–0.94]. There was strong positive correlation between estimated and annotated traffic density (R = 0.73, p < 0.001). Head pose estimation over a range of ± 135° had a mean absolute error of 6.75 ± 5.0° and an RMSE of 8.4°. Our approach can potentially allow automated processing of a vast amount of ND data while enabling the study of specific driver behaviors that were previously not achieved. Introduction Navigating intersections is a key part of everyday driving. They may represent only a small portion of a person’s total driving, yet more than 25% of fatal traffic crashes and about 50% of all crashes occur at intersections1,2. It is, therefore, important to study driving behaviors at intersections to understand factors affecting crash risk. Fundamental questions include whether the driver scanned (looked) to check for hazards before entering the intersection3 and whether they scanned again after entering the intersection4 to check for hazards that might suddenly materialize during the intersection maneuver. Detailed characterization of the intersection scenario along with the driver’s scanning behavior could help answer some of these questions. Scanning is especially important in the context of drivers with visual field loss who need to scan to see in the areas obscured by their vision loss. For example, at a T-intersection, a driver with hemianopia (the loss of either the left or the right side of the field in both eyes) would have to scan laterally almost 90° to see all of the intersection on the side of the field loss, requiring substantial head rotation5,6. To date, most behavioral data regarding gaze scanning by visually impaired people come from driving simulator studies, where head/gaze movements could be studied in controlled scenarios5,7–15. However, driving simulators do not replicate all the complexities of the traffic situations and environmental conditions encountered in real-world driving. Therefore, studying real-world driving behavior could provide critical insight into a variety of behavioral, traffic safety and vision science questions related to driving with visual field loss. Naturalistic driving (ND) study is one of the ways to analyze driving (driver) behavior in real-world settings. In the driving literature, one frequently encounters terms such as “on road study” or “instrumented vehicles” that may imply driving in natural conditions; for example, driving over a fixed course (open or closed course) with observers/instructors present in the vehicle16–19 or driving a study vehicle(s) instrumented for data collection20,21. However, they are not typically considered ND studies. An ND study can be defined as a study in which the participants drive their own vehicles that are instrumented to collect data (multimodal, multi-sensor), without any overt supervision, as they would usually do as part of their normal routine22,23. The data collection happens typically over a long duration (of the order of multiple days/weeks/months) and analysis is retrospective. Compared to other methods of understanding driving behavior, findings of well-designed and executed ND studies are more likely to be generalizable. The improved generalizability however comes with the tradeoff of increased complexity of data acquisition, processing, and analysis. ND studies also tend to be extensive in the amount and diversity of the data involved. Therefore, efficient methods and tools need to be developed to acquire, process, reduce, and analyze the data for a meaningful understanding of driver behavior. While there have been multiple different large-scale ND data collection efforts around the world24–26, many of them are based on the Virginia Tech Transportation Institute (VTTI) system27,28 which relies on extensive vehicle instrumentation for recording a vast array of data, including 5 channel videos and various vehicle state and kinematic parameters29,30. The collected data are reduced based on pre-established sensor cutoffs to generate interesting events that are manually reviewed by trained humans to code driver behavior and the scene conditions, among other details31. The VTTI system has been used for ND data collection and its subsequent analysis for studies addressing a variety of traffic safety and driving-related research questions32–45. One of the limitations of the VTTI system is that acquisition of new data for new studies can be challenging due to the overall complexity of data acquisition and processing. Depending on the specific study questions, the data may need to be customized, adding to the complexity. An alternative is to assemble a data acquisition system based on consumer grade hardware. With the increased availability of inexpensive dash-cam devices, it is possible to perform ND data acquisition using off-the-shelf consumer devices46,47. with the advantages of flexibility and convenience in system installation and data recording. Yet, the challenge here is that custom data processing and analysis workflows still need to be established. We describe the technical details of processing ND data acquired using commercial off-the-shelf recording devices. The specific processing goal for this study was to facilitate the analysis of driver head scanning behavior at intersections in ND. We describe methods for data processing and reduction via automated detection of driver head scans and the corresponding scene details, with the goal of aiding analysis of real-world driving/driver behavior at intersections. The effectiveness of the various aspects of the processing methodology is evaluated in a custom curated dataset of intersections. Naturalistic driving data acquisition The data used in this project were collected as a part of a multisite naturalistic driving project evaluating head scanning at intersections of drivers with visual field loss compared to drivers without field loss. The research adhered to the principles of the Declaration of Helsinki, and all participants provided written informed consent before participation. The study protocol was approved by the Institutional Review Board at Mass General Brigham. Driving data were recorded using a commercial fleet-management telematics solution by Zubie (Bloomington MN, USA), consisting of a Raven + dashcam recording device (Fig. 1; Ottawa ON, Canada) with a road-facing wide dynamic range scene camera (1920 × 1080, 30 Hz, 140° field of view) to capture the scene through the windshield and a high dynamic range driver-facing cabin camera (1280 × 720, 30 Hz, 130° field of view) to record the driver, among other on-board hardware. The cabin camera had an IR switcher and LED to facilitate nighttime recording of driver activity. We added a physical shutter for the cabin camera which could block the camera from recording inside the cabin (e.g., if the driver did not want video to be recorded or there was a child under 18 years of age in the car). The dashcam was securely placed on the vehicle dashboard or windshield by a member of the research team in a position which did not obstruct the driver’s view through the windshield (Fig. 1).Fig. 1Data recording system. (A) The dashcam is typically attached on the dashboard or on the front windshield of the vehicle, with the scene camera (blue cone) pointed outwards towards the road and the cabin camera (green cone) facing inwards towards the driver. The position of the cabin camera may be slightly different in each car, depending on the model and driver preference. The cabin camera axis may be somewhat offset from the driver’s straight-ahead gaze (shown by the red arrow). (B) Screenshots from cabin and scene cameras during data collection. (C) Example of GPS waypoints (blue circles) logged by the device during a drive (only select waypoints at intersections are shown). Map generated using Matlab Mapping Toolbox (https://www.mathworks.com/products/mapping.html). Data recording system. (A) The dashcam is typically attached on the dashboard or on the front windshield of the vehicle, with the scene camera (blue cone) pointed outwards towards the road and the cabin camera (green cone) facing inwards towards the driver. The position of the cabin camera may be slightly different in each car, depending on the model and driver preference. The cabin camera axis may be somewhat offset from the driver’s straight-ahead gaze (shown by the red arrow). (B) Screenshots from cabin and scene cameras during data collection. (C) Example of GPS waypoints (blue circles) logged by the device during a drive (only select waypoints at intersections are shown). Map generated using Matlab Mapping Toolbox (https://www.mathworks.com/products/mapping.html). The dashcam was connected to the vehicle OBD2 port, from which the vehicle speed was obtained and embedded in the cabin and scene camera video frames. The device recorded the GPS location at a frequency of about 0.5 Hz (once every 2 s). Recording started as soon as the ignition was turned on and stopped when the ignition was turned off. All data and videos from both cameras were recorded in an on-board storage card. Data were monitored remotely on a weekly basis to check the dashcam was functioning properly, confirm that video was being captured, and verify that the recorded data had not exceeded the capacity of the on-board storage card. When the storage card was close to being full the participant returned to the study site for the dashcam to be removed or for a new storage card to be installed. Data were transferred from the storage cards to secure network servers for further processing and analysis. Naturalistic driving data processing: problem statement & solution overview For studying driver head scanning and driving behavior at intersections, our ND data processing methodology needed to address the following aspects: (i) broadly determining the intersection locations within a large amount of on-road driving data, (ii) extracting relevant intersection segments for further processing, (iii) detecting driver head scans and selecting only those which occur within specific intersection regions (for example, just before entering the intersection or when within the intersection), (iv) extraction of driving variables within the intersection segments such as speed, acceleration/ deceleration, stops, distance traveled etc., v) characterizing the intersection—such as the type of traffic control device (“signage”), geometry, maneuver etc., and (vi) characterizing the conditions at the given intersection such as the traffic density, time of the day, weather etc. Intersections typically represent a relatively small part of the overall driving data recorded, and such intersection segments of interest could theoretically be identified and extracted from the overall driving data based on the vehicle speed and geolocation data (GPS coordinates). However, there were some major limitations of using vehicle speed and location information alone: (i) GPS location was not accurate—the error could be over many meters, (ii) there was some variation in the frequency of waypoint logging (so it was not exactly 2 s), (iii) there were delays in vehicle speed update affecting computation of distance travelled at a given instance, and (iv) intersection locations and their characteristics (such as signage type) were not reliably and clearly marked in mapping services. Moreover, characterizing the driver’s head scanning behavior just before entering an intersection or within the intersection required a very precise determination of the vehicle’s entry into and exit from the intersection, which was not possible using vehicle speed and location data alone. It was not feasible to manually review each intersection segment to annotate or code the required data elements, given the large quantities of driving data that was collected (typically more than 100 h and 2500 intersection events per participant). Therefore, our solution framework (Fig. 2) involved fusing information obtained from processing the scene videos with the vehicle speed and location information to extract the following 4 key pieces of information: intersection entry and exit points, driver behavior in/near the intersection segments, intersection characteristics, and scenario details.Fig. 2A solution framework that fuses a variety of information from diverse sources to precisely determine intersection bounds and its characteristics. A solution framework that fuses a variety of information from diverse sources to precisely determine intersection bounds and its characteristics. The system data log was stored in terms of trips, where a trip was defined as the interval between starting and shutting down of the engine. The system recorded 2 independent video streams (cabin and scene videos), as well as vehicle speed (embedded in the video frames) and vehicle location as set of waypoints. Each waypoint represented a timestamp, GPS-latitude, and GPS-longitude. The approximate location of intersections was interactively selected using a custom designed tool that loaded the location data from the selected trips of a given vehicle and displayed them on a map. This provided the flexibility to select a particular set of trips or locations depending on the aims of the investigation. The approximate center of each intersection of interest was manually marked, and the tool snapped to the nearest recorded waypoint, which was then added to the list of selected intersections. For each selected intersection, a segment spanning ± 100 m from the marked center was defined as the intersection segment of interest and the waypoint timestamps were used to extract the corresponding cabin and scene video clips. Workflow was developed to process each selected intersection segment (Fig. 3). Processing of cabin and scene videos proceeded independently of each other initially. Before processing, the date-time (YYYMMDD, HHMMSS format) and speed (in mph) embedded in both the videos was extracted. The extracted date-time information was used for frame-wise synchronization of cabin and scene videos.Fig. 3Data processing flow diagram showing extraction of ready-to-analyze head scans data at intersections, along with intersection/scene characteristics, from raw data recorded from the in-car recording device. Data processing flow diagram showing extraction of ready-to-analyze head scans data at intersections, along with intersection/scene characteristics, from raw data recorded from the in-car recording device. For each selected intersection instance, the cabin video was processed to detect head pose and head scans with respect to the driver’s straight-ahead direction. The scene video was processed to detect (i) scene objects—stop signs, traffic lights, vehicles, and pedestrians, (ii) stop line(s), and (iii) vehicle self-motion patterns for detection of turns. Also, GPS location data was used to determine vehicle maneuvers through the intersection independently of video processing to serve as an initialization step for refining the turn detection, as well as for registering the intended intersection location of the video clip if it contained multiple intersections (common in urban areas where there are multiple intersections a very short distance apart). The information thus gathered from various sources (GPS, video data, vehicle speed) was input to a rule-based decision making algorithm to determine the intersection and scene characteristics, as well as precise estimates of the vehicle’s entry and exit from the intersection (relative to the start of the video clip). Based on the precise intersection entry point, head scans just before entering and from within the intersection could be extracted. Detection of maneuvers using GPS data The method for turn detection using GPS coordinates was based on the change of the heading direction over the course of navigating through the intersection segment and was adopted from the approach previously described by Luo & Peli48,49. GPS data was saved as waypoints, recorded about every 2 s, within each trip that typically included multiple intersections. The first step was to smooth the recorded latitude and longitude values over the entire trip and remove the coordinates that were very close to each other (within a meter). Since the approximate center of the intersection was known (manually identified), the GPS data for the given intersection was segmented (± 100 m from the center coordinate) to loosely correspond to the video clips for the same intersection. Because the vehicle’s speed was not constant, the waypoints were not uniformly distributed. The coordinates were resampled so that they were spaced ≈5 m apart. Resampling was done by iteratively performing the following 3 operations (see Appendix A for details): i) finding the bearing angle between the current point and the next waypoint, ii) determining the latitude and longitude of the resampled point based on the bearing angle and the current coordinates, and (iii) finding the distance between the new resampled point and the next waypoint. Then the current point was set as the newly resampled point, and the above procedure was repeated until the distance was smaller than the resampling threshold (5 m); otherwise the next waypoint was set as the current point and the above procedure was repeated. Once the coordinates were resampled, the change in bearing angle was computed by looking ahead 10 steps. If the direction change was above a threshold limit (± 25°) then a turn was detected (start, peak, and end) and corresponding time stamps were recorded for comparison with video-based detections (Fig. 4A,B).Fig. 4Turn detection. (A) Direction change signal derived from GPS coordinates can identify approximate turn spans and their direction (left is positive and right is negative). (B) Original GPS coordinates (magenta circles) along with the interpolated locations (blue crosses) are overlaid with the start (green square) and the end (red triangle) of the detected turns. The solid blue arrows show the direction of travel. Map generated using Matlab Mapping Toolbox (https://www.mathworks.com/products/mapping.html). (C) Turn detection based on video motion. The left and right turns are indicated by positive and negative values, respectively. Screenshots associated with the start, peak, and end of the turns are included. The detected turn spans via video motion tend to a lot tighter than those detected based on the GPS coordinates. Turn detection. (A) Direction change signal derived from GPS coordinates can identify approximate turn spans and their direction (left is positive and right is negative). (B) Original GPS coordinates (magenta circles) along with the interpolated locations (blue crosses) are overlaid with the start (green square) and the end (red triangle) of the detected turns. The solid blue arrows show the direction of travel. Map generated using Matlab Mapping Toolbox (https://www.mathworks.com/products/mapping.html). (C) Turn detection based on video motion. The left and right turns are indicated by positive and negative values, respectively. Screenshots associated with the start, peak, and end of the turns are included. The detected turn spans via video motion tend to a lot tighter than those detected based on the GPS coordinates. Scene video processing Vehicle self-motion based on optical flow Vehicle self-motion was computed based on the image motion between successive scene video frames, which was also used for detecting turn maneuvers. When the vehicle was moving in a straight line (translation along the camera axis with minimal rotation), the typical radial motion pattern was observed around the focus of expansion (FOE)—with smaller motion magnitudes closer to the FOE that became larger towards the periphery of the image. When traveling straight, the FOE was typically close to the center of the image frame. When the vehicle turned, the rotational motion of the vehicle induced a strong horizontal image motion component near to the center part of the image frame. The median horizontal image motion component near the center of the scene video frame, obtained over the course of the video clip, was used as a signal for turn detection. The signal was further smoothed to reduce the effects of noise or other distracting motions, for example from the windshield wipers during rain. Turning maneuvers were detected by thresholding the average horizontal image motion component near the central regions of the scene video. For turn detection, peaks of the normalized image motion signal were determined with the following parameters: minimum peak height of 0.25, minimum peak prominence of 0.175, and with a spacing > 3 s between successive peaks. The detected turn candidates were further pruned based on the minimum distance spanned during the turn (4.5 m) and minimum speed during the peak turn (5 miles per hour). The start and end of each turn were determined from the extent of the peak signal from the baseline. The sign of the detected peaks indicated whether it was a left (+ ve) or right (-ve) turn (Fig. 4C). When no peaks were detected, it was assumed that the vehicle went straight through the intersection. The above parameters were determined based on heuristics—depending on the general placement of the camera with respect to the outside scene in our dataset, to account for a variety of intersection scenarios and to include tight and wide turns. Changing these numbers is not likely to affect the detection of the turn maneuvers themselves (as the underlying image motion signal during turns is quite strong), but could affect the location of the start or end of the turn maneuvers. The maneuver spans detected using the GPS data were compared with the turns detected based on image motion analysis. Agreement between the two maneuver detection methods (the type and when it occurred) was evaluated for confirmation. Compared to the turning spans detected by the GPS signal, the image motion signal tended to provide narrower or more precise turn spans. In practice, if an overlap existed between GPS and image motion based maneuvers, then a valid turn was detected. Otherwise, the intersection was flagged for manual review for reconciliation. Stop line detection Crossing the stop line can be an important indicator of when the vehicle entered the intersection. Presence of a stop line was detected by monitoring the changes in image intensity within a fixed region of interest (ROI) just in front of the vehicle (above/in-front of the hood) in the scene video. The intensity change associated with the presence of stop line usually first appears at the top of the ROI (farther from the vehicle’s hood) and shifts towards the bottom of the ROI (closer to the hood of the vehicle) as the vehicle approaches the stop line. Once the vehicle passes the stop line, intensity changes are no longer detected in the ROI. The maximum intensity difference in the ROI (500 × 100 pixels), rotated from 4° clockwise to 8° counter-clockwise with a step-size of 1°, was computed for each scene video frame. The magnitude and the spatio-temporal consistency of the intensity change was used to filter out spurious detections. Detections that first appeared in the top half of the ROI and then moved gradually towards the bottom half of the ROI over at least 5 consecutive frames and over a distance of at least 1 m during the ego vehicle’s approach, were selected as valid stop lines (Fig. 5). A buffer of 1 m was added to the final frame in a given stop line sequence to determine the point when the vehicle crossed the stop line.Fig. 5Stop line detection. A fixed ROI is placed in the scene video frame adjacent to the vehicle’s hood, which serves as the stop line detection region.Pixel intensity changes associated with the stop line are monitored within the ROI. Using the progression of the intensity change in the ROI for a small temporal segment, a stop line density curve (normalized) is built. The curve indicates the likelihood of how proximate the stop line is. The corresponding frame grabs show the first (#1) and the last frame (#2) from the stop line detection sequence. Based on this, it can be determined when the vehicle crossed the stop line. Stop line detection. A fixed ROI is placed in the scene video frame adjacent to the vehicle’s hood, which serves as the stop line detection region.Pixel intensity changes associated with the stop line are monitored within the ROI. Using the progression of the intensity change in the ROI for a small temporal segment, a stop line density curve (normalized) is built. The curve indicates the likelihood of how proximate the stop line is. The corresponding frame grabs show the first (#1) and the last frame (#2) from the stop line detection sequence. Based on this, it can be determined when the vehicle crossed the stop line. Scene object detection The YOLO object detection model50 trained on the COCO dataset51 was used for detecting scene objects. The COCO dataset consists of more than 90 common object classes, but only a few are relevant in a traffic scenario: vehicles (car, truck, bus, bicycle, and motorcycle), stop sign, traffic light, and person (pedestrian). Detections for the rest of the object classes were ignored. The output was a list of objects (bounding box, class label, confidence) per frame. The type of traffic control signage at the intersection was determined based on the detected objects in the video. In our dataset, there were 3 types of intersection signage –stop sign, traffic light, or none (when neither were present). Stop signs detected in the right half of the scene video frame that corresponded to the right side of the vehicle were considered as valid signage present in the direction of travel. For traffic lights, valid detections were within the central 70% of the scene video frame. The bounding box size of the detected traffic signs was used for generating a density function for signage detection. Thus, the density function changed as the vehicle approached the signs before passing them. Traffic lights were processed somewhat differently than stop signs, because unlike a stop sign, there could be single or multiple arrays of traffic lights in the direction of travel that were visible to the scene camera for the given intersection. The YOLO object detection model detected the traffic lights regardless of their orientation, so there could be multiple traffic light bounding boxes simultaneously present in each frame that corresponded to the same intersection. For example, when approaching a 4-way intersection, there could be one traffic light array facing the vehicle, one with its back to the vehicle, and 2 appearing sideways (Fig. 6). Depending on the maneuver, the bounding box areas of one array could increase or decrease as the vehicle passes through the intersection. Therefore, the largest bounding box per frame was selected to construct the traffic light density function, which could show multiple peaks, depending on the number of traffic light arrays present.Fig. 6Intersection signage detection and its utility in determining the vehicle’s progress through the intersection. Left: A density curve for a traffic-light intersection with 2 light arrays (two peaks), on which key time points are marked. Right: Scene video frames corresponding to the time points indicated on the plot. The curve is bimodal (unlike a stop sign or traffic light with a single light array), showing the approach and passing of the 2 light arrays. Intersection signage detection and its utility in determining the vehicle’s progress through the intersection. Left: A density curve for a traffic-light intersection with 2 light arrays (two peaks), on which key time points are marked. Right: Scene video frames corresponding to the time points indicated on the plot. The curve is bimodal (unlike a stop sign or traffic light with a single light array), showing the approach and passing of the 2 light arrays. Traffic density was defined as the average number of vehicle bounding boxes detected per second in the scene video. This number included all detected vehicles, including those not on the roadway (for example, vehicles parked on the side) and was not a count of unique vehicles seen in the video. The vehicle bounding boxes were further classified based on their aspect ratio as those belonging to cross-traffic (more rectangular bounding boxes) or those in the same direction of travel as the ego-vehicle (more square shaped). Further characterizing the traffic by tracking vehicles on the roadway and counting unique vehicles will be addressed in future work. Vehicle’s entry and exit from intersection Precise entry and exit from the intersection were determined by combining, in a hierarchical rule-based procedure, various pieces of information obtained from scene video processing: vehicle speed (distance derived from the speed, stops, acceleration/deceleration), stop line crossing, type of intersection signage (stop sign, traffic light, or none), and the maneuver (right turn, left turn, or straight). Depending on the type of intersection signage, entering an intersection was determined using objective criteria. For intersections with stop signs, the frame when the detected sign just moved out of the field of view of the scene video was defined as the point of entry into the intersection. For traffic light intersections, the situation was complicated by intersection geometry (for example, four way, or T etc.), status of traffic lights (green or red), and traffic in front of the vehicle. At a red light, it was possible that either the ego vehicle came to a halt before crossing the stop line or just after crossing the stop line. In the former scenario, entry into the intersection was determined as the point of crossing the stop line after the light turned to green. In the latter case (halt after crossing the stop line), it was generally safe to assume that there was no lead vehicle, and the entry into the intersection was when the vehicle first started moving after the halt (Fig. 7).Fig. 7Flow diagrams for determining intersection entry and exit points for intersections with traffic lights. Depending on the presence/detection of scene elements, the method of choosing the entry and exit points may vary. The entry and exit from intersection with stop signs or with no signage relied on more simplified versions of decision flowcharts. Flow diagrams for determining intersection entry and exit points for intersections with traffic lights. Depending on the presence/detection of scene elements, the method of choosing the entry and exit points may vary. The entry and exit from intersection with stop signs or with no signage relied on more simplified versions of decision flowcharts. In situations where the stop line could not be detected, the criteria were modified to consider when the vehicle passed by the traffic lights (when the traffic lights go out of the field of view of the scene camera). For intersections with a single traffic light array, the entry point was set at 15 m before the vehicle passed the traffic lights array (because the array was likely in the middle of the intersection). For intersections with multiple arrays of traffic lights, the entry point was when the vehicle passed the array closest to the vehicle’s approach (seen as the local minima between the two peaks of the density function). Compared to intersection entry, determining the exit point was somewhat less objective because there were no clear or consistent landmarks to use as reference points. Whenever turn maneuvers were involved, completion of the turn was the point of exit from the intersection. When no turns were involved, then the exit point was set at 2.5 m after passing the last traffic light array when multiple light arrays were involved, or 15 m for intersections with a single light array (because it was likely that the single array was in the middle of the road and the vehicle had to travel a longer distance to completely exit the intersection). In the case of stop signs without turns, the exit was estimated as the point when the vehicle had traveled 30 m from the entry point as there were no other indicators to mark the exit. The pre-set distances used in our approach were based on observations (as there is no single nationwide standard for intersection design). In actual analysis, these could be changed based on the geographical location. If the intersection did not have a stop sign or a traffic light (when it was labeled as “none”), then the only objective way to mark the entry and exit from the intersection was based on the start and end of the turning maneuver. If no turns were detected, then it was not possible to identify the entrance and exit points. Such intersection scenarios (no sign and going straight) were excluded from further processing and analysis. Cabin video processing The cabin video was cropped around the driver’s head/face (400 × 300 pixels). A fixed cropping window location was used for each participant since the camera position was fixed for a given vehicle. Contrast-limited adaptive histogram equalization was performed to enhance the face image of the driver to mitigate the effects of illumination changes (sunlight, glare or shadow, low light conditions etc.). Driver head pose was detected in the enhanced images using a custom wide-angle head pose detection model. Because the head orientation at intersections can be large (> 90° yaw rotation relative to the camera), existing head pose models were not effective in tracking across large head orientations. Adding fiducial markers on a driver’s head may simplify the processing52, but it is a burden to human subjects and not practical for ND studies. Our deep learning algorithm involved 2 components: precise head detection and wide-angle head-pose estimation. The head was localized by using a custom trained YoloV6 model that can reliably detect human heads wearing hats and/or glasses across a wide range of head poses. Once localized, the head pose was estimated by using a custom fine-tuned wide range head pose estimation model, 6DRepNet36053. The fine tuning was conducted based on a combination of the CMU Panoptic dataset54, 300W-LP55, and AFWL200056. Head scans, defined as the movement of the head away from the straight-ahead direction before returning back towards the straight-ahead direction, were detected using an algorithm that was previously evaluated in the context of walking mobility57. Because small yaw head movements do not play an important role in scanning a large field of view (e.g., 180°) at intersections3 only those scans where the peak head position exceeded a minimum threshold magnitude of 20° from the straight-ahead direction of the driver were retained for analysis10. In the case of compound scans, where the head moved farther away before fully returning back within 20° of the straight-ahead direction, the peak position was retained as the scan magnitude (Fig. 8). The offset between the cabin camera and the driver’s straight-ahead direction was corrected before detecting the head scans.Fig. 8Head scan detection at a busy T intersection with stop sign and left turn maneuver. The horizontal axis shows head angle and the vertical axis shows time (increasing from bottom to top of the plot, ≈ 37 s total clip duration). The blue curve shows the head orientation in the horizontal direction. The black dashed and dotted line at 0° indicates the straight-ahead direction. The black dashed lines at ± 20° eccentricity indicate the minimum head orientation threshold for scan detection employed in this example. The red circles indicate peak head scans (7 total detected). The corresponding screenshot of the cabin video are shown, with the head axis overlaid. The points of entry, exit, halt, and acceleration are shown. Head scan detection at a busy T intersection with stop sign and left turn maneuver. The horizontal axis shows head angle and the vertical axis shows time (increasing from bottom to top of the plot, ≈ 37 s total clip duration). The blue curve shows the head orientation in the horizontal direction. The black dashed and dotted line at 0° indicates the straight-ahead direction. The black dashed lines at ± 20° eccentricity indicate the minimum head orientation threshold for scan detection employed in this example. The red circles indicate peak head scans (7 total detected). The corresponding screenshot of the cabin video are shown, with the head axis overlaid. The points of entry, exit, halt, and acceleration are shown. Intersection detection evaluation We present experimental evaluation of processing methods for (i) characterizing the intersections, (ii) detection of intersection bounds, and (iii) comparison with GPS-based maneuvers. ND intersection dataset A dataset was created to evaluate the intersection detection and characterization approach described above. The dataset consisted of intersection segments from ND recordings, spanning about ± 100 m distance around the manually marked GPS waypoints. The timestamp associated with the intersection center waypoint was used to obtain the corresponding frame range in the scene videos. A custom visualization and annotation software was developed for annotating intersection characteristics (signage type, maneuver, and traffic density) and for playing the scene videos frame-by-frame. Within the clipped scene videos, intersection entry and exit were manually marked based on the criteria described above by annotating the respective frame numbers, as the ground truth. Marking exit frames was based on the best judgment of the reviewer (because, in contrast to the entry point, there were fewer consistently available landmarks to specify criteria to determine a vehicle’s exit from the intersection). Since the speed of the vehicle was known, the ground truth distance of the vehicle entry and exit from the intersection with reference to the beginning of the clip could be computed. Data from a total of 365 intersection instances from 6 vehicles were annotated (see Table 1 for details).Table 1Details of the evaluation dataset of ND intersections. VFL: visual field loss. IQR: inter-quartile range.Intersections365Vehicles6 (2 drivers with VFL)States within the USMA, CA, COSettingsUrban, suburban, & ruralIntersection geometry4-way (X or +): 252 (69%); 3-way (T or Y): 113 (31%)Clip duration; median [IQR]26 [21–41] sIntersection entry distance (from the start of the clip); median [IQR]87 [75–100] mIntersection span; median [IQR]34 [25–39] mTime within intersection; median [IQR]5.1 [3.8–7.2] s Table 1Details of the evaluation dataset of ND intersections. VFL: visual field loss. IQR: inter-quartile range.Intersections365Vehicles6 (2 drivers with VFL)States within the USMA, CA, COSettingsUrban, suburban, & ruralIntersection geometry4-way (X or +): 252 (69%); 3-way (T or Y): 113 (31%)Clip duration; median [IQR]26 [21–41] sIntersection entry distance (from the start of the clip); median [IQR]87 [75–100] mIntersection span; median [IQR]34 [25–39] mTime within intersection; median [IQR]5.1 [3.8–7.2] s Details of the evaluation dataset of ND intersections. VFL: visual field loss. IQR: inter-quartile range. For each test intersection clip, our algorithm predicted the entry and exit points in the scene video along with the corresponding distances. We employed the following outcome measures: i) difference between estimated entry time (from the start of the clip) with ground truth, ii) difference in the estimated entry distance (from the start of the clip) with ground truth, iii) the estimated overlap (exit—entry) of intersection bounds with the ground truth bounds (using dice coefficient), iv) difference between the number of head scans detected within actual and estimated intersection bounds, and v) correlation between estimated and rated traffic density. Nonparametric tests of comparison were used to evaluate differences among intersections with different sign types and maneuvers. Manual rating of traffic density (none, light, moderate, heavy) roughly corresponded to the number of vehicles observed in the scene. These ratings were assigned for traffic from different directions: front, left, and right. For correlation analysis, the traffic density ratings in different directions were combined over a single rating scale by assigning scores to each level (0, 1, 2, 3). Intersection detection results Intersection signage type and maneuver were correctly identified in 99% (361 out of 365) and 95% (352 of 365) of the instances, respectively. The overall median absolute difference [IQR] between the estimated and ground truth entry time was 0.23 [0.07–0.63] seconds, with an RMSE of 2.1 s. The entry time difference was within a 1 s margin for 86% (313 of 365) of the instances, and 95% of cases were within 3.1 s (Fig. 9A). The entry time was off by more than 5 s in only 2.7% of the instances (10 out of 365).Fig. 9Accuracy of estimating intersection bounds compared to the ground truth (manually annotated). Histograms show (A) the error in the intersection entry time point (from the beginning of the intersection clip), (B) the error in the distance to the entry point, and (C) the overlap between ground truth and estimated intersection region. The dashed vertical line shows the median values of the respective distributions. Accuracy of estimating intersection bounds compared to the ground truth (manually annotated). Histograms show (A) the error in the intersection entry time point (from the beginning of the intersection clip), (B) the error in the distance to the entry point, and (C) the overlap between ground truth and estimated intersection region. The dashed vertical line shows the median values of the respective distributions. The median absolute distance between the estimated and actual entry point was 1.1[0.3–4.2] meters, with an RMSE of 8.3 m (after exclusion of 1 case where the speed log was incorrect). The estimated entry point was within 10 m of the ground truth in 92% of the instances, and 95% of the instances recorded an error below 13 m (Fig. 9B). The intersection entry distance was within 5 m, which is the average length of a mid-size car, for 286 out of 365 (78%) of the instances. The median overlap proportion between ground truth and estimated intersection bounds was 0.88[0.8–0.94], with 83% of intersection bounds recording an overlap > 0.75 and 95% instances with overlap > 0.45 (Fig. 9C). The number of driver head scans counted within the ground truth and estimated intersection bounds were identical for 302 (83%) intersections, with a single scan difference in 49 of the 63 remaining cases. Thus, the scan count differed by more than 1 for only about 4% of the intersections. Accuracy of signage and maneuver detection, as well as intersection bounds detection varied for different intersection signage types and for different maneuvers (Table 2). There was a significant difference among the 3 signage types for entry time error (χ2 = 11,df = 2, p = 0.005), entry point distance error (χ2 = 25,df = 2,p < 0.001), and intersection segment overlap (χ2 = 44,df = 2,p < 0.001). Detection errors were lowest and the overlap was the highest (statistically significantly) for intersections with stop signs among the three signage types. There was also a significant difference among the 3 kinds of maneuvers for entry distance error (χ2 = 8,df = 2, p = 0.02), entry point distance error (χ2 = 8,df = 2,p = 0.02), and intersection segment overlap (χ2 = 34,df = 2,p < 0.001). However, after posthoc analysis, there was no clear consistent pattern of any one kind of maneuver being best or worst. Intersections with left turns recorded significantly higher overlap than Right or Straight maneuvers.Table 2Intersection detection results for different signage types and maneuvers.VariableCategoriesGround truth countCorrect detectionsMedian [25th—75th percentile]Entry point time error (seconds)Entry point distance error (m)Intersection segment overlapSignage typeNone4646 (100%)0.33 [0.18–0.81]3.0 [1.2–5.8]0.82 [0.68–0.89]Stop sign9692 (96%)0.13 [0.07–0.40]0.6 [0.2–1.3]0.94 [0.88–0.97]Traffic light223223 (100%)0.27 [0.07–0.70]1.2 [0.4–5.1]0.87 [0.80–0.92]Maneuver performedLeft113111(98%)0.27 [0.07–0.63]0.7 [0.1–3.2]0.93 [0.85–0.97]Right118114 (97%)0.30 [0.10–0.83]1.4 [0.4–5.0]0.89 [0.76–0.94]Straight134127 (95%)0.20 [0.07–0.49]1.2 [0.4–4.3]0.85 [0.80–0.91] Table 2Intersection detection results for different signage types and maneuvers.VariableCategoriesGround truth countCorrect detectionsMedian [25th—75th percentile]Entry point time error (seconds)Entry point distance error (m)Intersection segment overlapSignage typeNone4646 (100%)0.33 [0.18–0.81]3.0 [1.2–5.8]0.82 [0.68–0.89]Stop sign9692 (96%)0.13 [0.07–0.40]0.6 [0.2–1.3]0.94 [0.88–0.97]Traffic light223223 (100%)0.27 [0.07–0.70]1.2 [0.4–5.1]0.87 [0.80–0.92]Maneuver performedLeft113111(98%)0.27 [0.07–0.63]0.7 [0.1–3.2]0.93 [0.85–0.97]Right118114 (97%)0.30 [0.10–0.83]1.4 [0.4–5.0]0.89 [0.76–0.94]Straight134127 (95%)0.20 [0.07–0.49]1.2 [0.4–4.3]0.85 [0.80–0.91] Intersection detection results for different signage types and maneuvers. Purely based on GPS data, the accuracy of maneuver estimation with respect to the ground truth was 96% (349 out of 365 total); 93% for left turns (105 out of 113), 97% for right turns (115 out of 118), and 96% when going straight (129 out of 134). In 220 instances where a turn was correctly detected based on GPS data, median [IQR] overlap of the intersection bounds was 0.56 [0.39—0.66]. The overlap score was low because the GPS turn bounds tended to be much wider than the ground truth bounds. The GPS estimated turn span completely covered the ground truth intersection bounds in 195 out of the 220 instances (89%). There was an agreement between GPS-based and video-based processing on the estimated maneuver in the case of 341 instances (93%), and both agreed with the ground truth maneuvers in 339 instances (93%). In 217 instances where there was an agreement about presence of turn between GPS-based and video-based processing, median [IQR] overlap with the estimated intersection bounds was 0.59[0.36—0.73]. The GPS estimated turn span completely covered the intersection bounds estimated via video processing in 187 out of the 217 instances (83%). Traffic density ratings were available for a subset of our dataset (276 out of 365, 76%). The median [IQR] traffic density score (after conversion from manual rating) was 2.0[1.0—4.0], with a max score of 8.5 (max possible score was 10.5). The estimated average vehicle density from video analysis was 3.2 [1.4—6.0] vehicles bounding boxes detected/second, with a maximum value of 16.1 (no upper limit). There was strong positive correlation between estimated vehicle density and the assigned traffic density score (R = 0.73, 95% CI of R: [0.68–0.78], t = 18, df = 274, p < 0.001). Evaluation of head pose estimation Experimental setup The experiment for head pose accuracy measurement was performed in a 20 × 20 feet (6 × 6 m) room. Three subjects with normal vision, seated approximately in the middle of the room, turned their heads over a wide angular range of ≈ 135° on each side of a central reference point. Fifteen targets were attached to the walls around the seating location. One of the targets was close to the straight-ahead direction (or the 0° reference) and 7 targets each were placed on the left and right with increasing eccentricity from the center target. The subjects wore a head mounted laser pointer strapped to the center of their forehead (middle of the two eyes) (Fig. 10A) to help with aiming their head at the targets on the walls. Starting with the reference (straight-ahead) target, the subject successively aimed their head at the targets on the right until they reached the most extreme target. Then they reversed the order and revisited each target on the right, passing the center, and continued aiming their head at the targets on the left side all the way through, before returning to the center target. Subjects raised their hand to indicate that they had aligned with the target. Each alignment event lasted about 1 to 3 s. A video camera (iPhone) set up on a tripod was placed around 28 inches (71 cm) away from the subject’s face that recorded the full sequence of the head turns. The video was processed by the head pose detection algorithm. Video frames associated with raised hand gestures at each target alignment were extracted and the corresponding yaw angles were obtained. Ground truth was computed based on the known positions of the targets with respect to the subject’s location and compared to the estimated head orientation.Fig. 10Wide head pose estimation. A: Screenshots showing the ability of the model to detect head turns > 90° relative to the straight ahead direction. The laser point strapped to the forehead allowed alignment of the head with fixed targets on the wall. B: In an experiment involving 3 subjects viewing 15 fixed targets, the estimated and ground truth head angles were consistent (different markers show data for the 3 subjects). The mean absolute error was 6.75° ± 5.0°. Wide head pose estimation. A: Screenshots showing the ability of the model to detect head turns > 90° relative to the straight ahead direction. The laser point strapped to the forehead allowed alignment of the head with fixed targets on the wall. B: In an experiment involving 3 subjects viewing 15 fixed targets, the estimated and ground truth head angles were consistent (different markers show data for the 3 subjects). The mean absolute error was 6.75° ± 5.0°. Head pose evaluation results The head pose algorithm was able to track the head successfully for angles > 90° with respect to the straight ahead viewing direction (Fig. 10A). Compared to the ground truth, the estimated yaw pose was highly consistent with the ground truth for all 3 subjects, as indicated by the slope and R2 of the trend line being very close to 1 (Fig. 10B). The mean absolute error (± standard deviation) across all 3 subjects and across all target locations was 6.75 ± 5.0°. The root mean squared error (RMSE) was 8.4°. Given that our primary goal is to detect large angle head rotations at intersections, the magnitude of the error is acceptable. For context, the error is roughly equivalent to the half angle spanned by a typical road lane 10 m (33feet) away from the driver’s position. There was no significant difference in the errors between the 3 subjects. The error did not increase significantly with the increasing target eccentricity (no significant correlation). Comparatively, the mean absolute error (± standard deviation) for FSA-Net head pose model58 was 19.9 ± 29.2°, with an RMSE of 25.7°. This large error with FSA-Net was primarily because of the incorrect head pose estimation for head pose angles > 60°. Utility of quantifying head scanning at intersections Here, we show how the head scanning data at intersections obtained via the processing pipeline described above could be useful for studying driver behaviors. For example, one of the various research questions arising out of this study is to understand the driver head scanning patterns in relation to different maneuvers performed at the intersections. Specifically, whether the rates of scans to the left and right side were associated with the maneuver performed and whether there were any differences between drivers with vision impairment and normal vision drivers. For this analysis we used the head scans detected within the predicted intersection bounds (vehicle’s entry to exit from the intersection based on our algorithm processing). Out of the 6 drivers whose data was included in our evaluation dataset, 2 drivers (ages 63 and 52 years, both male) had partial visual partial field loss on the left side (VI). The other 4 (3 male ages 62, 47, and 58 years; 1 female age 51 years) had normal vision (NV). Out of 365 total intersection instances, 179 (49%) instances were from VI drivers with 44 left turn, 57 right turn, and 78 straight or no turn maneuvers as determined by the processing algorithm. In 186 intersection instances belonging to the NV drivers, there were 67 left, 65 right, and 54 straight maneuvers detected. A total of 653 head scans were recorded within the detected intersection bounds over the 365 total intersection instances, with VI drivers overall recording 269 head scans (153 to left and 116 towards the right side) and NV drivers recording 384 head scans (214 left and 170 right). The median[IQR] duration of presence within the intersection was 5.3[3.3–7.5] seconds. The data was analyzed to determine associations between the rate of head scans and the various potential predictors listed below. The number of head scans normalized by the duration of the vehicle’s presence within the intersection bounds (scaled to the order of minutes) was the dependent variable (scan rate). The interaction of three fixed effects: the direction of the head scan (leftward or rightwards), the maneuver performed (left turn, right turn, or no turn), and the vision status of the driver (VI or NV) was examined in a mixed effects negative binomial regression model, with the vehicle/driver serving as the random factor. Based on the regression model, the VI drivers scanned more frequently towards than away from the direction of the turn maneuver (Fig. 11). As would be expected in the case of left turns, the rate of leftward head scans was significantly higher than rightward scans for both VI (mean[95%CI] scans per minute; Left: 14.4[10.2–20.5], Right: 5.1[3.3–7.9], p < 0.001) and NV drivers (Left: 11.6[8.5–15.9], Right: 4.8[3.3–6.9], p < 0.001). For right turns, a significant difference in the rate of scanning to the left and right was found only in VI drivers who scanned more frequently to the right side (Left: 4.4[2.8–7.0], Right: 9.1[6.3–13.3], p = 0.001). Also, there was a statistically significant difference in leftward scan frequency between VI and NV drivers on right turns with NV scanning more frequently than VI (mean[95%CI] scans per minute; NV: 8.6[6.1–12.0], VI: 4.4[2.8–7], p < 0.02). There was no significant difference in head scan rates towards the left and right for maneuvers without turns. VI drivers also made significantly larger leftward gaze scans than rightward scans (mean ± SE Left—Right magnitude 21° ± 6.4°, p < 0.001) compared to NV drivers (4° ± 5°, p = 0.4).Fig. 11Head scan rate within intersection bounds for the 3 different maneuvers. Leftward (L) and rightward (R) head scans are compared within normal vision (NV) and visually impaired (VI) drivers. VI drivers scanned differently from NV drivers during right turn maneuvers. Head scan rate within intersection bounds for the 3 different maneuvers. Leftward (L) and rightward (R) head scans are compared within normal vision (NV) and visually impaired (VI) drivers. VI drivers scanned differently from NV drivers during right turn maneuvers. The analyses were repeated using head scans extracted from the videos based on data from ground truth labeled maneuvers and intersection bounds. The results were similar to those reported above, indicating that our processing approach resulted in computation of valid outcome measures. Discussion We describe methodological details of collecting and processing naturalistic driving (ND) data with the goal of studying head scanning behaviors of drivers at intersections. Major contributions of our approach are: (i) the use of off-the-shelf recording hardware for ND data capture, (ii) custom data processing pipeline that combined vehicle speed, location information, and scene video processing data to determine intersection characteristics, (iii) precise detection of vehicle’s entry and exit from the intersection to help study driver behavior, and (iv) a custom developed wide head pose detection algorithm for detecting large head rotations at intersections. The proposed data processing methods were able to determine the intersection bounds and scene characteristics with high accuracy in our evaluation dataset. Our approach can potentially allow automated processing of a vast amount of ND data while enabling the study of specific driver behaviors that were previously not achieved. In many of the previous large-scale studies involving VTTI systems, the focus of automated processing was to detect driving events largely based on vehicle dynamics, with some automated video processing; for example, lane monitoring29,30. ND studies related to intersection driving have typically relied on GPS location mapping and extensive use of manual coding of intersection characteristics32–34,39,40,59,60. The automated detection and characterization of intersections presented in the current paper makes it feasible to efficiently process large amounts of data without the need for time-consuming manual review (which would be impossible for thousands of intersections), opening up the possibility for detailed analysis of scanning behaviors and factors affecting scanning across several months of recorded driving data. Neural networks, computer vision algorithms, and multi-sensor data fusion approaches have been previously used for driving scene characterization21,61,62 In our approach, however, data fusion has the specific goal of precise determination of intersection bounds and characteristics. The precision of the automated detection of the objectively defined intersection entry point is pivotal to enabling analysis of scanning behaviors across many diverse intersections (i.e. it provides a common reference point across intersection types and maneuvers). There were only 4 out of 365 instances of incorrect signage detection, all involving intersections with a stop sign. The incorrect detections were due to glare or occlusions. The main reasons for misidentification of maneuvers by the video processing method included curving roads near the intersection (maneuvers labeled as straight in the ground truth dataset), erroneous or noisy motion signals from the video, or completion of the turn maneuver in a piecemeal manner that led to a sub-threshold motion signal. The main reason for incorrect turn detection using the GPS data was low-quality, inaccurate GPS coordinate logging. There were also some instances where large errors in intersection entry point estimation were seen. Some of the possible reasons for large intersection bounds detection errors include incorrect detection or failure to detect the stop line, adverse weather like rain or snow, incorrect speed recording, or delayed speed update by the recording device. For the sake of discussion, if large errors are defined as intersection entry time > 3 s and intersection entry distance > 10 m, then there were 9 such cases (2%). The main reasons for incorrect detection for these 9 cases were failure to detect the stop line (4 instances), slow speed update (2 instances), glare affecting the scene camera (1 case), incorrect detection of turn maneuver start (1 instance), and one case of a rolling stop that was not correctly handled. When determining intersection bounds in our evaluation, intersections without signage had the highest errors in terms of entry point and distance, as well as the lowest overlap between estimated bounds and ground truth. Since signage detection is a robust way to determine the intersection entry, absence of signage meant that bounds detection relied only on turn detection. Turn initiation and completion are somewhat more subjective, therefore there may be more arbitrariness to the ground truth annotation, which may lead to seemingly larger errors. Annotating the intersection exit point is always arbitrary regardless of the type of intersection; therefore, we see relatively smaller overlap of intersection bounds even when the entry point is estimated with a reasonable accuracy (for example, ≈40% of intersections with entry error smaller than the median value had less than the median overlap—this indicates that the exit annotation could have contributed to the overlap error). There are some limitations of the presented work. First, we did not evaluate some of the components of intersection scene understanding such as pedestrian detection and detection of adverse weather conditions. This will be future work. Second, our evaluation dataset was limited to video clips with typically only one intersection each. Third, we did not include intersections without signage and without a turn maneuver (i.e., no- sign, no-turn cases). Finally, our processing is currently restricted to daytime videos (including those captured during twilight). Processing nighttime videos would require some additional steps to handle glare and low light regions of the scene video. Such challenging cases could be handled with more advanced scene understanding functionalities (such as identifying different traffic signs or better use of GPS information), which is future work. Despite these limitations, the algorithm was able to extract head scans from intersection segments in a large ND dataset acquired with off-the-shelf hardware. While we included only a preliminary analysis of driver head scan behavior at intersections to demonstrate the potential utility of our processing methods, these initial findings provide some intriguing questions for future analyses. The finding that VI drivers scanned more frequently to the right side than their left side during a right turn was surprising given that they had left-side visual field loss, and it might have been expected that they would scan at least as frequently to the left as to the right before a right turn. These findings suggest that the VI drivers seemed to not pay enough attention to the left side when turning right compared to the NV drivers. However, other factors such as intersection types, traffic density etc. might account for these differences. Interestingly, the VI drivers made substantially larger scans to the left than the right, whereas there was little difference in left and right scan magnitudes for drivers with NV. These results suggest that the VI drivers were using a compensatory strategy, making larger scans to the side of their field loss63. Future work will involve more extensive analysis in a larger sample of drivers using the data acquisition and processing methodology described here to further understand driving behaviors.