Clearly communicate requirements and testing procedures to connected vehicle device developers, and allow for industry input and iteration for less mature devices.

Experience from fostering the development of connected vehicle devices for the Safety Pilot Model Deployment in Ann Arbor, Michigan.

Date Posted

Safety Pilot Model Deployment: Lessons Learned and Recommendations for Future Connected Vehicle Activities

Summary Information

The Connected Vehicle Safety Pilot was a research program that demonstrated the readiness of DSRC-based connected vehicle safety applications for nationwide deployment. The vision of the Connected Vehicle Safety Pilot Program was to test connected vehicle safety applications, based on vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communications systems using dedicated short-range communications (DSRC) technology, in real-world driving scenarios in order to determine their effectiveness at reducing crashes and to ensure that the devices were safe and did not unnecessarily distract motorists or cause unintended consequences.

The Connected Vehicle Safety Pilot was part of a major scientific research program run jointly by the U.S. Department of Transportation (USDOT) and its research and development partners in private industry. This research initiative was a multi-modal effort led by the Intelligent Transportation Systems Joint Program Office (ITS JPO) and the National Highway Traffic Safety Administration (NHTSA), with research support from several agencies, including Federal Highway Administration (FHWA), Federal Motor Carrier Safety Administration (FMCSA), and Federal Transit Administration (FTA). This one-year, real-world deployment was launched in August 2012 in Ann Arbor, Michigan. The deployment utilized connected vehicle technology in over 2,800 vehicles and at 29 infrastructure sites at a total cost of over $50 million dollars in order to test the effectiveness of the connected vehicle crash avoidance systems. Overall, the Safety Pilot Program was a major success and has led the USDOT to initiate rulemaking that would propose to create a new Federal Motor Vehicle Safety Standard (FMVSS) to require V2V communication capability for all light vehicles and to create minimum performance requirements for V2V devices and messages.

Given the magnitude of this program and the positive outcomes generated, the Volpe National Transportation Systems Center conducted a study sponsored by the ITS JPO to gather observations and insights from the Safety Pilot Model Deployment. This report represents an analysis of activities across all stages of the Safety Pilot Model Deployment including scoping, acquisitions, planning, execution, and evaluation. The analysis aimed to identify specific accomplishments, effective activities and strategies, activities or areas needing additional effort, unintended outcomes, and any limitations and obstacles encountered throughout the Model Deployment. It also assessed the roles of organizations and the interactions among these organizations in the project. Findings were used to develop recommendations for use in future deployments of connected vehicle technology. Information for this analysis was gathered from a combination of over 70 participant interviews and a review of program documentation. It is anticipated that findings from this study will be valuable to future USDOT research programs and early adopters of connected vehicle technology.

The report contains numerous lessons across many topics, including program management, outreach and showcase, experiment setup, DSRC device development, device deployment and monitoring, and data management.

Lessons Learned

The USDOT developed and implemented a collaborative process through which the DSRC device development community would eventually be capable of supplying devices for the Safety Pilot Model Deployment (SPMD). The USDOT identified two key activities in an effort to engage and develop this relatively immature community of developers.

First, the USDOT funded a subset of the research and development costs through a set of procurements that resulted in multiple awards to device developers. The developers were tasked with producing a small number of prototype devices, roughly 2 – 5. These devices were then subjected to functional testing by the USDOT. The USDOT returned the devices to the developers with the test results for refinement and retesting. This activity was important for engaging the device development community because the majority of device developers were small organizations, and they did not have the research and development funding to develop these devices without USDOT support. Also the business model for developing some of these devices was not well established as a relatively small number of devices were needed for SPMD, and the profit made from this activity likely would not cover the development costs. These USDOT funded contracts attracted a relatively large number of device developers to the process.

For the second activity, the USDOT encouraged an open exchange of information across the development community through the use of weekly technical coordination meetings, sample functional testing events, plugfests, and other activities aimed at rapidly increasing the developers’ experience. The primary purpose of this open exchange was to ensure that the devices produced by the developers were interoperable with each other. This required all developers to interpret and implement the requirements in the same manner. The weekly open meetings shared lessons learned and other vital information across the development community, as well as documentation, e.g., meeting notes, example code, for the community to utilize in their development processes.

The USDOT developed the initial version of the device requirements for Vehicle Awareness Devices (VADs), Aftermarket Safety Devices (ASDs), and Roadside Units (RSUs) and provided them as input into the Product Development Phase, as illustrated in Figure 4. During the Product Development Phase, the device requirements were iteratively revised based on the results from the functional testing of the prototype devices and on input from the device developer community. At the completion of this iterative process, a final set of the requirements was generated for each of the devices deployed in the SPMD. This final set of requirements was used to qualify devices for the research Qualified Product List (rQPL). The collective set of requirements for each type of device was identified in the published specification document.

The USDOT sought to mature the requirements to a point that the final set would be able generate devices that would operate throughout the SPMD with only minimal manual intervention from the Test Conductor or device supplier. As it was, the majority of devices deployed in the SPMD were still operational after one year of data collection, however, there were a number of gaps in the device requirements that impacted the Test Conductor’s ability to monitor and maintain the devices throughout the Model Deployment. For example, the final requirements lacked test functionality typically needed for less mature or prototype devices, including diagnostic functions, reset capabilities, detailed status messages and status indicator lights. As a result, the Test Conductor had to utilize a variety of manual methods to monitor both the in-vehicle and infrastructure devices deployed in the field to identify device failures. In some cases, when a failed device was discovered, without having reset capabilities the Test Conductor had to physically remove the device and ship it back to the vendor to perform the device reset. This resulted in additional downtime for the affected devices which reduced the volume of data collected and increased the amount of Test Conductor support required.

Related recommendations made in the source report include:

  • Determine the most efficient and effective activities, i.e., conference calls, workshops, working meetings, “plugfests”, to engage the device community and increase industry knowledge to ensure a common understanding and interoperability of devices, applications, and infrastructure to support project objectives.
  • Tailor device requirements on a project by project basis depending on the intended use of the devices and current device maturity level. Clearly communicate requirements to device developers and incorporate the appropriate functionality for less mature devices, such as test capabilities and functions to monitor device up-time.
  • Utilize an iterative process for developing device requirements that incorporates cycles of industry input and device testing. Complete the requirement generation and Qualified Product Lists (QPLs) prior to all device and system use. If it is anticipated that the requirements will be updated after testing based on field deployment, incorporate enough time in the schedule to procure updated components and retest.
  • Institutionalize and formalize all testing procedures for the QPL. Provide clear and unambiguous qualification test procedures, with pass/fail criteria, as part of the test plan.
  • Provide testing specifications covering the performance of all components, to allow common understanding and expectation, and allow sufficient time for iteration.
  • Develop application tests that are specific to the type of technology being deployed, and place additional emphasis on false positive scenarios.
  • Evaluate impacts of procuring devices from various vendors included on the QPL. Balance the cost of increased procurement complexity of having more device types with the cost of possible schedule impacts due to possible device production delays. Consider the impacts of having to conduct interoperability testing between large numbers of devices and schedule implications for any needed hardware or software updates.