This lesson is drawn from the experiences of the Field Operation Test (FOT) of a Roll Advisor and Control (RA&C) system. The experiences from this FOT demonstrate that when doing this type of FOT based study, it is important to carefully consider the experimental design, the data collection plan, and the FOT sample size to improve the chances of having a successful FOT. The following discussion illustrates these lessons learned in more detail.
- Develop a sound experimental design.
There are a number of important things to consider when developing an experimental design. Selecting a long enough period for data collection is important to help eliminate seasonal effects from the main phenomena being studied. In the current study, there were two ways that the experimental design limited the strength of the conclusions. The FOT was conducted essentially within a one-year period, with one phase conducted as the weather was warming and the other phase conducted as the days shortened. Separating the seasonal effects from the RA&C’s effects could not be done with absolute certainty. Secondly, the demographics of the drivers were uniform. All were experienced, and all were within a twenty-year age span. A better test of the educational benefits of the RSA would have been to study two similar groups of novice drivers, one with the system and one without. Battelle, the evaluator of the FOT, understands that any experiment must be planned within constraints, and the partnership worked around these constraints as well as possible.
- Automate data collection as much as possible.
In this study, data collection was largely automatic. UMTRI reviewed the data to make sure all was working well. On ordinary days it took no help from Praxair personnel. In the cases where valid data were missing or the data collected were invalid, the partnership could readily trace the path through the server at the terminal to diagnose and fix the problem.
- When planning the FOT size, carefully consider the trade-off between ease of data collection and statistical power.
During this FOT, the use of a single terminal kept logistics to a manageable level. There were two occasions when a Battelle staff member interviewed the drivers. In both instances, nearly all of them were seen in a single trip of only a couple days. A benefit of the limited delivery area was that there were many road segments through which each driver passed many times. There were, however, disadvantages to the selection of a single terminal. Only six tractors, driven for six months each in the control and treatment phases, produced a fairly small number of serious events to analyze. By the nature of statistics, confidently estimating small changes, as were expected and observed in this FOT, requires large numbers of near-rollover incidents, which were simply not available. In this sense, the compact size of the FOT fleet was largely responsible for the lack of statistical significance in the final safety benefits estimate. Also, while the convenience of the terminal’s proximity to both UMTRI and Battelle had its benefits, the RA&C was tested exclusively in the Great Lakes region of the country, which is fairly free of hills and curves. As with many experiments, there is an inherent trade-off: a larger experiment provides richer data, but a smaller one is easier to conduct.
The lessons learned discussed above suggest that the creation of a sound experimental design, automating data collection as much as possible, and carefully considering the optimal size of the FOT can have a significant impact on the success of the FOT. Following these guidelines will assist FOT planners and stakeholders in helping to maximize the success and effectiveness of their FOT efforts.
(Our website has many links to other organizations. While we offer these electronic linkages for your convenience in accessing transportation-related information, please be aware that when you exit our website, the privacy and accessibility policies stated on our website may not be the same as that on other websites.)