The main goal of the SIRIS field operational test (FOT) was to collect data to evaluate the performance of the prototype system and to determine the viability of such a system for use in CMV enforcement. From March 2010 to September 2010, Oak Ridge National Laboratory (ORNL) facilitated the SIRIS FOT at the Greene County Inspection Station (IS) in Greeneville, Tennessee.
During the course of the field operational test (FOT), 413 commercial motor vehicles (CMVs) were given a North American Standard (NAS) Level-1 inspection. Of those 413 CMVs, 384 were subjected to a SIRIS screening. A total of 36 (9.38%) of the vehicles screened by SIRIS were flagged by the system as having one or more thermal issues, with brakes issues making up 33 (91.67%) of those. Of the 36 vehicles flagged as having thermal issues, 31 (86.11%) were found to have a violation and 30 (83.33%) of those vehicles were placed out-of-service (OOS).
Overall, the enforcement personnel who have used SIRIS for screening purposes have given positive feedback on the potential of SIRIS. With needed improvements in detection algorithms and stability, the system will be beneficial to the CMV enforcement community and increase overall trooper productivity by accurately identifying a higher percentage of CMVs to be placed OOS with minimal error. Key lessons learned are presented below.
- Beware that performance of a Smart InfraRed Inspection System (SIRIS) is negatively affected by heavy precipitation or cold weather.
During the course of the FOT, there were 12 recorded instances that SIRIS became inoperable or performed in a manner that made it unusable by IS personnel. ORNL staff and inspection personnel were asked to keep track of any problems with SIRIS by logging them onto the “SIRIS Problem Log Sheet.” Not all problems were logged due to the busy nature of the IS. However, the majority of the problems recorded was similar in nature and easily corrected with a system reboot. There was a period of time in which SIRIS was nonfunctioning due to a hardware malfunction; no testing was performed during this time. The situation was resolved in a timely manner by IEM (International Electronic Machine).
The major problem with SIRIS throughout the FOT was the effect of weather on the image quality and functionality of the system. Whenever there was cold weather, the incidence of false positive flags increased dramatically based on anecdotal evidence from the troopers and ORNL staff, especially related to tires. When there was heavy precipitation present, the image quality was reduced, which caused fewer flagged vehicles to be observed and the system would randomly take pictures even if no vehicle was present. It should be noted that enforcement personnel typically do not inspect vehicles in harsh weather, thus, weather anomalies did not negatively affect the results during the FOT. The troopers also noted on the log sheets downtime of the system due to power surges during storms or unexpected malfunctions during normal use.
Another issue with SIRIS was related to the software and/or hardware inside the County IS. Occasionally the SIRIS graphical user interface (GUI) would lock up and not inspect vehicles until the power to the cameras was cycled or the computer was rebooted. This usually occurred on a regular basis during the FOT. It was noted that this phenomenon typically occurred after a vehicle passed too quickly through SIRIS.
- Recognize that SIRIS has promise for increasing productivity of inspection personnel but not yet ready for national deployment due to lack of accuracy in flagging vehicles with potential defects.
At the end of the evaluation, enforcement personnel familiar with SIRIS were asked questions regarding the functionality of SIRIS, and how they see SIRIS being used in the future. ORNL received five completed questionnaires. In general, the feedback received was positive. SIRIS is believed to have great potential in the enforcement community relative to increased productivity when used as a screening tool. However, all of the troopers agreed that in its current condition, SIRIS was not ready for national deployment with the main reasons being stability and accuracy in flagging vehicles with potential defects. Many of the troopers were not happy with the level of downtime for SIRIS and the number of times that the cameras had to be reset in order for vehicles to be detected. Also, due to the amount of false-alarms, troopers mentioned that in some cases the SIRIS alarms were not as helpful as the actual temperatures on the SIRIS GUI in determining if a vehicle needed to be inspected. In these situations, the troopers used their experience with SIRIS to determine whether or not a vehicle should be pulled in for inspection following an alert.
While it would be impossible to flag every single vehicle with a possible defect, the likelihood of a vehicle having a defect if flagged cannot be overlooked. Since not all vehicles can be inspected due to limited resources, using SIRIS as a screening tool to determine if a vehicle should be inspected is a great improvement over the traditional method of choosing vehicles.
- Consider developing functional and performance specifications that can be used by the states interested in deploying SIRIS.
From a qualitative standpoint, the SIRIS device, if deployed, could focus the limited resources of commercial vehicle inspection agencies to inspect vehicles with a high probability of having a brake or tire defect. From a quantitative standpoint, the inability of the current SIRIS device to remain operational within the ramp-side environment precludes its value to enforcement. Work must be done to overcome the stability and operational issues with the overall system for SIRIS to become a viable mainstream tool. These stability and operational issues should not overshadow the fact that the current optical system and decision-making algorithm produced results that could clearly have a positive effect on the OOS (out of service) rates of commercial vehicles and the related accident, personal injury, and death rates.
The next step for creating a nationally-deployable infrared screening tool similar to SIRIS is to create functional and performance specifications that can be used by a state interested in deploying this type of equipment. The specification should be based on the performance and capabilities of SIRIS and should require a more rugged system than the current prototype.
The SIRIS bears the promise of becoming an effective, productivity-enhancing tool in screening the commercial vehicles that are in need of further inspection. The SIRIS FOT evaluators suggest that all first-order testing, validation and certification of future infrared inspection systems be done at the Greene County IS to allow the testing and performance measurement of these systems to be done in the same operational and vehicle stream environment (I-81 southbound) as the SIRIS proof-of-concept, pilot test, and FOT.
(Our website has many links to other organizations. While we offer these electronic linkages for your convenience in accessing transportation-related information, please be aware that when you exit our website, the privacy and accessibility policies stated on our website may not be the same as that on other websites.)