Safety Pilot Model Deployment: Lessons Learned and Recommendations for Future Connected Vehicle Activities
The Connected Vehicle Safety Pilot was a research program that demonstrated the readiness of DSRC-based connected vehicle safety applications for nationwide deployment. The vision of the Connected Vehicle Safety Pilot Program was to test connected vehicle safety applications, based on vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communications systems using dedicated short-range communications (DSRC) technology, in real-world driving scenarios in order to determine their effectiveness at reducing crashes and to ensure that the devices were safe and did not unnecessarily distract motorists or cause unintended consequences.
The Connected Vehicle Safety Pilot was part of a major scientific research program run jointly by the U.S. Department of Transportation (USDOT) and its research and development partners in private industry. This research initiative was a multi-modal effort led by the Intelligent Transportation Systems Joint Program Office (ITS JPO) and the National Highway Traffic Safety Administration (NHTSA), with research support from several agencies, including Federal Highway Administration (FHWA), Federal Motor Carrier Safety Administration (FMCSA), and Federal Transit Administration (FTA). This one-year, real-world deployment was launched in August 2012 in Ann Arbor, Michigan. The deployment utilized connected vehicle technology in over 2,800 vehicles and at 29 infrastructure sites at a total cost of over $50 million dollars in order to test the effectiveness of the connected vehicle crash avoidance systems. Overall, the Safety Pilot Program was a major success and has led the USDOT to initiate rulemaking that would propose to create a new Federal Motor Vehicle Safety Standard (FMVSS) to require V2V communication capability for all light vehicles and to create minimum performance requirements for V2V devices and messages.
Given the magnitude of this program and the positive outcomes generated, the Volpe National Transportation Systems Center conducted a study sponsored by the ITS JPO to gather observations and insights from the Safety Pilot Model Deployment. This report represents an analysis of activities across all stages of the Safety Pilot Model Deployment including scoping, acquisitions, planning, execution, and evaluation. The analysis aimed to identify specific accomplishments, effective activities and strategies, activities or areas needing additional effort, unintended outcomes, and any limitations and obstacles encountered throughout the Model Deployment. It also assessed the roles of organizations and the interactions among these organizations in the project. Findings were used to develop recommendations for use in future deployments of connected vehicle technology. Information for this analysis was gathered from a combination of over 70 participant interviews and a review of program documentation. It is anticipated that findings from this study will be valuable to future USDOT research programs and early adopters of connected vehicle technology.
The report contains numerous lessons across many topics, including program management, outreach and showcase, experiment setup, DSRC device development, device deployment and monitoring, and data management.
Setting up the experiment, including selection of the test area and coordinating, preparing, and managing the vehicle fleets and drivers was of critical importance to the Safety Pilot Model Deployment. These elements were key components of the Experimental Plan, which was developed by the Test Conductor in conjunction with the Independent Evaluator. The Experimental Plan also included the Test Conductor’s plans for collecting all data and ensuring data quality as required by the Independent Evaluator to perform its independent evaluation in support of the 2013 NHTSA agency decision.
Prior to developing the statement of work for the SPMD Test Conductor, the USDOT needed to determine what size test was necessary to generate sufficient data for the evaluation of the V2V safety applications. The test size depended on numerous parameters, including the number test participants, number of V2V-equipped vehicles, and the test duration. All could have major impacts on the Test Conductor scope of work and budget required. Therefore it was critical to get an accurate size estimate prior to starting the experiment.
During the Model Deployment, there were two instances where the risk response plans were initiated as a result of lower than expected volumes of interactions. First, due to a delay in the vehicle awareness devices (VAD) deployment as discussed above, the interactions at the start of Model Deployment were less than projected. The USDOT authorized the Test Conductor to utilize staff to drive VAD equipped vehicles throughout the Model Deployment area to increase potential exposure of integrated vehicles to the VAD equipped vehicles. Second, several months into the test, the 3 integrated heavy vehicles were not generating as many interactions as were needed for evaluation purposes. This was due to the vehicles not being driven within the SPMD area as often as anticipated. As a result, the USDOT authorized the Test Conductor to utilize existing staff members with valid commercial driver’s licenses (CDLs) to drive the vehicles within the Model Deployment Geographic Area on predefined routes to generate more data.
Related recommendations made in the source report include:
- Conduct an analysis using prior field test data to properly scope the size of a new test to meet the data collection and evaluation objectives. Do this early in the planning process as test size can have major impacts on scope and budget, and can be difficult to modify once the experiment has started.
- Consider adding more vehicles or devices to act as ‘spares’ to ensure that the minimum number of devices required will be operational during the entire test.
- Be prepared to conduct an iterative process to select the test site. Analyze the characteristics of the test area, i.e., historical crash and traffic volume data, against the performance requirements of the test. Ensure that the test site characteristics match the needs of all aspects of the pilot, particularly if involving commercial vehicles or transit. These vehicle types may be constrained by geographic area, operations and scheduling, or commercial and union restrictions.
- Closely coordinate the test area selection process with the driver recruitment strategy to ensure that selected participants will be driving primarily in the test area. This approach is likely to produce better results when examining interactions between equipped vehicles and interactions between equipped vehicles and infrastructure.
- Develop quantitative performance measures with intermediate targets for the environment that can be utilized throughout the test to evaluate progress and success towards project goals.
- Develop risk response plans in the case that intermediate performance targets are not being met. Determine trigger points to determine when each response plan should be executed.
- When identifying volunteer drivers, utilize metrics that focus on the volume of trips or time spent within the test area boundaries. Using metrics that focus exclusively on the total number of miles driven may not always be representative of the amount of time a driver spends within the test area. It may also be difficult to identify particular drivers within certain age or gender groups if there is a minimum miles driven requirement.
- Replicate the community-based driver recruitment strategy used in the SPMD. Understand what motivational factors are important to the community (e.g., education, jobs, safety) and incorporate these factors in the recruitment process. Utilize in-person recruitment activities whenever possible.
- Recruit participants iteratively to align with the planned device deployment schedule. This will reduce significant delays between recruitment and initial contact. Lags between recruitment and participation could cause participants to reconsider their willingness to participate for a variety of reasons or volunteers could move away from the test area making them non-viable candidates.
- Utilize actual driving data (i.e. GPS, surveys) over self-reported driving data, if possible when selecting and prioritizing volunteer drivers from the recruitment pool. SPMD demonstrated that there was a significant advantage to using the BSM data logged from VADs to identify participants for future phases.
- Take into account the geographic area where these vehicles will be operated prior to procurement of specific types of vehicles. There may be constraints that impact the types or configuration of the vehicles purchased, particularly in smaller cities.
- Ensure that contingency plans are in place in the event that an insufficient amount of data is being generated when conducting naturalistic testing with heavy vehicles. This is especially important with a small sample size and when the geographic area limits the exposure of the heavy vehicles to other V2V equipped vehicles.
- Ensure the vehicles will be exposed to the target populations addressed by the safety applications when deploying transit-specific safety applications.