BRIEFING HIGHLIGHTS:

• Although virtual and augmented reality (VR and AR) are emerging technologies in transportation without return on investment (ROI) data available yet, they are expected to help address transportation challenges in the near future.
• VR and AR could improve driver performance, help with trip planning, improve the transit experience, assist with highway construction, and better protect vulnerable road users.
• Some state Departments of Transportation (DOTs) are beginning to explore the potential of these technologies.

Introduction

The increasing use of three-dimensional (3D) model-based workflows and the rapid advancement in computer-interface design and hardware have enabled technologies like virtual reality (VR) and augmented reality (AR) to support a variety of transportation use cases [1]. While VR provides a complete virtual replacement to the visual world, AR overlays or combines virtual and real elements in real time to produce a 3D, interactive view [1]. Milgram and Kishino (1994) defined the relationship between AR and VR along the reality-virtuality continuum, summarized in Figure 1.

There are two basic categories of display types for these systems: handheld mobile devices, such as smartphones and tablets, and head-mounted display (HMD) devices, such as headsets or glasses [2], as shown in Figure 2.

Figure 2. Sample hand-held mobile device tablet (left) and sample head-mounted display device (right) with augmented reality. Source: iStock

Benefits

The National Science Foundation (NSF) has recently offered Small Business Innovation Research (SBIR) grants to support the development of Augmented and Virtual Reality and help bring them to the public sphere across multiple fields [3]. In transportation, AR and VR technologies are becoming increasingly available and could offer promising benefits. These technologies could improve driver performance, help with accessible trip planning, improve the transit passenger experience, assist with highway construction, and better protect vulnerable road users (e.g., pedestrians and bicyclists).

In terms of driver performance, AR enhances a driver’s perception by superimposing critically important real-world contextual information in minimal yet sufficient detail in a driver’s line of sight [4]. The AR could allow the passing driver to see road conditions ahead of the slower-moving vehicle it is trying to pass before it attempts a potentially dangerous maneuver [4]. This capability could be beneficial in preventing head-on collisions from passing at high speeds on undivided highways (see Figure 3), which leads to many roadway fatalities each year.

As VR and AR mature, there will be opportunities to use them to support accessible trip planning. These technologies can help users safely experience a route virtually before making their trip [5]. For example, users can learn the locations of transit stops, shops, and other landmarks as they move through the virtual environment. Additionally, VR can help users learn new routes that are not part of their regular routines, such as emergency evacuation routes from their apartment or office building, sidewalk detours during construction, or new environments while traveling outside their home area [5]. The Federal Transit Administration (FTA) aids SBIR projects, one of which created an AR visual component with mild cognitive impairment adaptations to support route creation as part of their Guided Augmented Independence Travel Aid (GAIT-Aid) [6].

In addition to making trip planning more accessible, AR and VR could make public transit more accessible and enjoyable. The Southeastern Pennsylvania Transportation Authority (SEPTA) recently hosted a “SEPTA for All Challenge” calling innovators to create AR tools to make public transit more accessible for people with disabilities [7]. Once in route, VR and AR headsets offer many potential advantages over current seatback and mobile displays that could improve the transit rider experience for all [8]. They have the potential to allow passengers to make better use of their time across modes by making travel more productive, entertaining, enjoyable, private, and immersed [8].

VR and AR can also help address challenges in management of highway infrastructure assets [1], [2]. Challenges in highway construction, management, and field operations include the lack of real-time and/or integrated information, gaps between planned solutions and practical implementations, quality assurance, and effective project communications [1]. AR provides enriched content that can help project managers and engineers deliver projects faster, safer, and with greater accuracy and efficiency, allowing them to catch errors before actual construction begins and potentially improve design [1]. In addition, AR has the potential to reduce construction costs and improve delivery time [1]. Managers may also be able to use these technologies for training, certification, inspection, and stakeholder outreach [1], [2]. For example, “providing project visualization to property owners to better understand project impacts and design options” was identified as a top potential future direction for AR in transportation construction and asset management [2]. As a user-friendly tool, AR helps to lower the level of software skill needed to navigate and interpret model-based design and construction (MBDC) data [9]. AR can also be used to speed up data collection and analysis processes. For example, AR can measure areas of newly installed concrete pads to calculate contractor payment amounts, see Figure 4 [2].

As a final example, AR and VR can help protect vulnerable road users, see Figure 5. The data below, which are derived from several evaluation projects, highlight some of the benefits of AR and VR for pedestrian safety.

Augmented Reality (AR)

• In a study performed with sixteen drivers and actual pedestrians in a parking lot environment, using AR heads up displays (HUDs) of pedestrian collision warnings in vehicles reduced maximum deceleration and stopping distance, resulting in larger gaps leading up to the pedestrians (2016- 01121).

Virtual Reality (VR)

• A study conducted in Germany leveraged VR technology to compare the effectiveness of various vehicle-pedestrian communication implementations in automated vehicles (AVs). AVs issuing a “high-content” message of “I’m stopping, you can cross” were rated as being the most trustworthy and reassuring by a focus group of pedestrians with vision impairments (2020-01458).
• A study in China demonstrated how VR training can be used to reduce risky pedestrian behaviors among school-aged children (2020-01445).

Costs

VR and AR systems are composed of hardware and software components as well as sensors. A variety of VR and AR devices (and their individual components) are becoming commercially available. For example, one AR headset, which supports audio and gesture commands and controls [1], costs $3,500 per headset device (2021-00493). Originally intended for gaming, entertainment, and enterprise applications [1], the freight and logistics industry has considered headsets like these for vehicle maintenance training.

Handheld tablet devices, as opposed to headsets, will most likely be the platform on which AR applications will be developed in the near future because of their low cost, simplicity, and ability to be used outdoors [1]. Additionally, tablet devices appear to be making more rapid advances for highway use cases because they have fewer limitations in outdoor display and viewing, see Figure 6 [1]. Tablets and handheld mobile devices also have the advantage of allowing multiple users to concurrently see the same view on a single device [1]. One negative aspect of using tablets and smartphone devices, however, is their need to be handheld, thus eliminating the opportunity for hands-free operation [1]. Another disadvantage of using tablets centers on the accuracy of the tracking technology; the overlay of virtual to real-world imagery might not be as accurately aligned [1].

The lack of return on investment (ROI) to justify AR implementation (whether handheld or head-mounted) remains a key challenge for its deployment in transportation [1]. While the lack of ROI is a traditional barrier to new technologies, independent field trials have helped address this concern [1]. Additional objective field trials that examine the benefits and costs of implementing AR technology in transportation based on empirical results will further alleviate the ROI challenge [1].

Best Practices

AR technology is rapidly changing and improving in terms of hardware, applications, and workflows with the integration of AR devices into existing modeling workflows expected in the near term [1]. Additionally, FHWA’s Every Day Counts (EDC) Program—which identifies and deploys proven, yet underutilized innovations to accelerate the delivery and cost effectiveness of highway projects, enhance safety, reduce congestion, and protect the environment [10]—has been supporting deployment of 3D MBDC workflows since EDC-2 (“3D Engineered Models for Construction”) beginning in 2013 [11]. These 3D MBDC data and workflows will help enable AR and VR technologies.

As emerging technologies in transportation, VR and AR have yet to offer many documented lessons learned or best practices. However, a few state Departments of Transportation (DOTs), including Utah DOT (see Case Study on pg. 5), Florida DOT, and Michigan DOT [2], are beginning to explore the potential of these technologies and will likely be able to offer lessons learned in the near future. Additionally, two lessons learned with respect to AR in transportation are presented below.

Augmented Reality (AR)

• Sixteen Human Factors researchers from a wide range of international institutions offered input on how AR can improve interactions between vulnerable road users and AVs. While AR functionalities were noted to have potential drawbacks—related to both technological issues and accessibility concerns—they may be useful in providing customized interpretations of signs and signals for each user (2021-01011).
• Include pedestrian collision warnings on in-vehicle AR heads-up displays to reduce reaction time, maximum deceleration, and, in some cases, stopping distance and braking time (2017-00775).

Case Study

In late 2018 and early 2019, the Utah Department of Transportation (UDOT) conducted a prototype field study using a handheld AR mobile device [1]. The project included MBDC requirements, a key enabler for AR systems [9]. One of the vendors UDOT partnered with used the project MBDC data to demonstrate AR as a potential tool for highway infrastructure asset planning, design, implementation, and management [9]. Infrastructure elements were transferred as 3D model elements into the AR device and then could be viewed in real time overlaid on the real-world context [1], [9].

The UDOT study demonstrated the basic 3D-model data flows that are required for any AR application [1]. Even without all the important attributes (e.g., striping color, sign size, draining box dimensions) included yet in the AR device’s view, crews commented on the value and were excited at the prospect of using AR with all 3D design model attributes included [2]. The AR demonstration also showed that stakeholders can use this technology to investigate roadway disruptions and their potential impacts to residents, businesses, and motorists [9].

For UDOT, the most valuable use for AR will probably be in locating underground utilities during design, construction, and asset management and operations [2]. Although UDOT anticipates AR to be useful and valuable, there will be shortterm gaps with most, if not all, information in 3D design models [2]. Until a fully geospatial utility database is populated, the precise location of angle points, valves, and other details often will not be known during the design phase of the project [2]. UDOT is currently working toward populating a utility database to mitigate these issues and allow for more robust highway planning, construction, and ongoing asset management [2].

References
[1] J. Mallela, K. Gilson, P. Goodrum and J. Steen, "Leveraging Augmented Reality for Highway Construction," FHWA Office of Infrastructure Research and Development, McLean, 2020. Available: https://www.fhwa.dot.gov/publications/research/infrastructure/pavements/20038/20038.pdf
[2] H. Azari and K. Gilson, "Can Augmented Reality Address Highway Construction Challenges?," Public Roads, Summer 2020. Available: https://highways.dot.gov/public-roads/public-roads-magazine-summer-2020-edition
[3] SBA, "Augmented and Virtual Reality (AV)," 2021. [Online]. Available: https://www.sbir.gov/node/1897375
[4] A. Mahmood, B. Butler and B. Jennings, "Potential of Augmented Reality for Intelligent Transportation Systems," In: Lee N. (eds) Encyclopedia of Computer Graphics and Games. Springer, Cham, June 2018. doi:10.1007/978-3-319-08234-9_274-1
[5] M. Vasudevan, H. Townsend, T. Nhi Dang, A. O'Hara, C. Burnier and K. Ozbay, "Identifying RealWorld Transportation Applications Using Artificial Intelligence (AI): Summary of Potential Application of AI in Transportation," U.S. Department of Transportation Intelligent Transportation Systems Joint Program Office (ITS JPO), Washington, D.C., 2020. Available: https://rosap.ntl.bts.gov/view/dot/50651
[6] E. A. Rodriguez, "FTA Annual Report on Public Transportation Innovation Research Projects for Fiscal Year 2020," Federal Transit Administration, Washington, D.C., 2021. Available: https://www.transit.dot.gov/sites/fta.dot.gov/files/2021-01/FTA-Report-No-0181.pdf
[7] City of Philadelphia, "SEPTA for All: Augmenting Transit with Augmented Reality," 2021. [Online]. Available: https://www.phila.gov/spotlight/septa-for-all-augmenting-transit-with-augmented-reality/
[8] M. McGill, J. Williamson, A. Ng, F. Pollick and S. Brewster, "Challenges in passenger use of mixed reality headsets in cars and other transportation," Virtual Reality, vol. 24, pp. 583-603, 2020. Available: https://doi.org/10.1007/s10055-019-00420-x
[9] FHWA, "e-Construction & Partnering Case Study: Partnering for Technology-Immersed Process Improvement," October 2019. [Online]. Available: https://www.fhwa.dot.gov/construction/econstruction/edc4/hif20007.pdf
[10] FHWA, "Every Day Counts," 6 August 2021. [Online]. Available: https://www.fhwa.dot.gov/innovation/everydaycounts/ [Accessed 11 August 2021].
[11] FHWA, "EDC-2 Innovations," 25 March 2021. [Online]. Available: https://www.fhwa.dot.gov/innovation/everydaycounts/edc-2.cfm