Crowdsourcing Study Paired Participants with Simulated Driving Video Footage to Evaluate Human Decision-Making Capabilities.
Nationwide, United States
Machine Learning, Human Factors and Security Analysis for the Remote Command of Driving: An MCity Pilot
Summary Information
The development of hybrid intelligence systems that accurately and rapidly address sudden and rare critical events is important to both human drivers and automated vehicles. To increase the speed of human operator decision making, this study introduces a new crowdsourcing workflow. When an autonomous system encounters an unfamiliar dangerous situation where it cannot determine which action to take, control is handed off to an instantaneous crowdsourcing workflow that uses collective human ability to rapidly predict which objects in a driving scenario will be dangerous. In this study, crowd workers (participants who were recruited to view the simulated driving scene video footage) watched simulated driving scenarios including annotations in various formats on the screen and were asked to predict whether or not there was going to be a collision. Participants were crowd workers recruited from a reputable crowdsourcing company who were shown simulated driving videos either with or without a car crash.
- Select all relevant labels from the perspective of others when annotating for efficiently estimating data distributions. In ambiguous or subjective domains, multiple valid interpretations may exist, especially if there is insufficient contextual information available to annotators. This study found in these cases, most fine-grained and time-consuming methods were not the most accurate. Instead, answer distributions could be a better representation of the data than a single answer and capture a more nuanced representation of an ambiguous scenario. This approach was also faster ꟷ reduced human time required by 21.4 percent while maintaining the same level of accuracy.
- Combine human computation and vision algorithms that do not require any data to accelerate reliable segmentation for image recognition by the Automated Vehicles. With vision algorithms, object images with pixel information can be pre-segmented and a large group of crowd workers could be recruited to decide whether each pre-segmentation belongs to a target object or not. It is important to strike a balance between quantity and quality during this process.
- Choose a proper annotation input method as it influences latency of crowd workers predicting whether a dangerous object is present in a driving scene. In this particular study, the one-hand-one-key condition (crowd workers clicked on left and right arrow keys) had the most robust recall scores, but precision and accuracy with this input method dropped when examining team performance as opposed to individual performance. Reliable performance could be achieved in one second under the two keys conditions. Therefore, it is important to have the crowd workers use the most optimum input method to make a faster and more precise prediction about a dangerous driving scene.
