Commit to testing the new systems thoroughly, develop an acceptance matrix to document status of testing, and perform verification and validation before introducing them to support agency’s transportation operations.
Chattanooga Area Regional Transportation Authority's experience in deploying transit ITS
Made Public Date


United States

A Case Study on Applying the Systems Engineering Approach: Best Practices and Lessons Learned from the Chattanooga SmartBus Project


Chattanooga, Tennessee is a city of about 170,000 people (about 500,000 in the metropolitan area) located near the Tennessee-Georgia border. The Chattanooga Area Regional Transportation Authority's (CARTA) provides transit services for the City of Chattanooga and portions of nearby counties. CARTA serves this area by providing fixed-route bus service (16 routes), curb-to-curb transit for people with disabilities (Care-A-Van), a free electric shuttle in the downtown area, an incline railway up historic Lookout Mountain, several parking garages, and management for much of the on-street parking in the downtown area. It is a moderate-sized transit organization in a moderate-sized community. In 2003, CARTA undertook an ITS project, SmartBus, which entailed introduction of many interdependent technologies across the entire range of CARTA operations:

  • Various network technologies were deployed to provide connectivity across CARTA's fixed and mobile assets
  • Technologies were deployed to help automate and modernize many field operations, such as automatic passenger counters and new bus fare boxes
  • Technologies were deployed to help automate and modernize many back office operations, such as new dispatch and revenue management systems
  • A data warehouse was developed to consolidate data collected during CARTA operations, and reporting tools were created to take advantage of this data warehouse

The deployment was challenging and susceptible to risks of failure. Effectively managing the risks, CARTA successfully implemented the SmartBus technologies over a period of 6 years, from 2003 to 2009, with most of the deployment completed. In November 2009, the Intelligent Transportation Systems Joint Program Office (ITS JPO) of the United States Department of Transportation (U.S. DOT) published an independent evaluation report documenting CARTA’s experiences in planning and implementing the SmartBus project. Presented below are lessons learned from CARTA’s experience that could be beneficial to other mid-size transit agencies’ planning for implementation of ITS program.

Lessons Learned

The Chattanooga Area Regional Transportation Authority's (CARTA) SmartBus ITS program offers valuable guidance on performing the testing, verification, and validation of new technological systems when implementing ITS at a mid-size transit agency. Key lessons learned include:

  • Develop an acceptance matrix to document the status and amendments resulting from testing, verification, and validation of requirements for ITS projects. A feature of CARTA's systems engineering approach was a strong commitment to thoroughly testing the ITS technologies before introducing them to support operations. CARTA's general approach worked as follows. All the requirements for a system were documented in an acceptance matrix. This matrix included: an identification number (ID) for each requirement, a description of and amendments to each requirement, and remarks from verification and validation exercises. For example, the acceptance matrix for the computer aided dispatch / automatic vehicle location (CAD/AVL) deployment was maintained in an Excel® worksheet that included nearly 800 lines of requirements information. The deployments themselves were divided up into a series of incremental demonstrations, with each demonstration exhibiting more of the final functionality required. Before each demonstration, the contractor was required to submit for CARTA approval an acceptance test procedure that identified the requirements to be met and the method used to verify that the requirements were met. CARTA and the contractor would follow the test procedures during the demonstration, and CARTA would update the requirements matrix with the result and status of each requirement tested. This approach provided a mechanism for (a) ensuring that all requirements were satisfactorily met and (b) monitoring progress towards meeting the requirements.
  • Commit to thoroughly testing systems before introducing them to operations. Having a mechanism for testing is not sufficient without a commitment to conducting thorough testing, and there were numerous examples that demonstrated CARTA's commitment. First was CARTA's recognition of the importance of testing. During discussions with the evaluators, CARTA identified testing as an important part of their system engineering process. Second was CARTA's willingness, if necessary, to change plans based on testing results. For example, during testing of the new operations software for its flex-route operations, CARTA determined that the planned new software would not provide the expected benefits. So, CARTA stopped the introduction of that software, selected an alternative that would better meet operational needs, and rescheduled the deployment to take place after the CAD/AVL system was in place. An example of the thoroughness of CARTA's testing is evident in the rollout of the next-stop announcement system. In late 2008, CARTA completed the onboard database and was capable of activating onboard audio and dynamic message sign (DMS) announcements. Rather than proceeding with system-wide operations after verifying the general functionality of the system, CARTA elected to conduct a comprehensive test of the system's functionality for each route. CARTA, using a test bus set up with the test database, drove each route variation in the system with an observer onboard to note any errors in the messages displayed and announcements made. During this testing, CARTA discovered and corrected several errors in the system, including incorrect announcements and announcements triggered at incorrect locations.
  • During thorough testing, identify enhancements to the system that would improve its operations. The test-bus end-use testing narrated above also allowed CARTA to identify enhancements to the system that would improve its operation. The system was originally configured to display a stop announcement first, then display the date/time, on the interior DMS prior to displaying information about the next stop. During testing, CARTA discovered that the time spent displaying the date/time sometimes delayed the next-stop information display, particularly when stops were closely spaced. Thus, performing end-use testing of the system prevented exposing the public to these errors, which could have reduced public confidence. Another example of CARTA's commitment to testing was the 2009 roll out of the Automated Vehicle Monitoring (AVM) system. The core infrastructure needed to support AVM—the onboard equipment, the Wireless Local Area Networks (WLAN) for daily data uploads, and the AVM software—was in place early in 2008. However, CARTA decided to focus on implementing systems that would deliver the most direct and visible benefits to riders, such as the next-stop announcements and the next-arrival-time predictions. After these systems came online in December 2008, CARTA shifted focus to rolling out AVM. By January 2009, the AVM system elements were integrated enabling the system database to receive daily uploads of data from the buses. CARTA then conducted internal testing of the AVM system to confirm it was operating correctly before releasing it for use by the mechanics in March 2009.
  • Beware of contractor’s resistance to extra time required for thorough testing. Initially, some contractors were resistant to the extra time CARTA required for testing. This resistance was typically overcome during the initial subsystem implementations, when CARTA first applied its thorough testing processes with that contractor. CARTA required that test procedures fully demonstrate all the specified requirements for the systems. CARTA reinforced its commitment to testing by refusing to accept systems that did not pass the documented test procedures. After working with CARTA's testing process, most contractors recognized that this approach, while it might delay final acceptance, helped prevent errors from being found after the system was in use when corrections would have been costlier and more difficult to make.

Developing an acceptance matrix for testing the requirements is a useful exercise. CARTA's strong commitment to thoroughly testing the requirements allowed the agency to identify errors early on and fix them in advance of introducing the systems to support operational efficiency and mobility in its transit services.

A Case Study on Applying the Systems Engineering Approach: Best Practices and Lessons Learned from the Chattanooga SmartBus Project

A Case Study on Applying the Systems Engineering Approach: Best Practices and Lessons Learned from the Chattanooga SmartBus Project
Publication Sort Date
Haas, R.; E. Perry; J. Rephlo
U.S. DOT Federal Highway Administration

(Our website has many links to other organizations. While we offer these electronic linkages for your convenience in accessing transportation-related information, please be aware that when you exit our website, the privacy and accessibility policies stated on our website may not be the same as that on other websites.)

Goal Areas

Focus Areas Taxonomy: