Research In Action
Research In Action
CIRP recently published a research brief on Ohio’s implementation of the virtual driving assessment (VDA) within the Ohio Bureau of Motor Vehicles (OBMV) driver licensing workflow.
The 14-month pilot was initially called the Ohio Portable Driver Simulator System (now called Ohio- Ready, Test, Drive!) and it demonstrated that the system worked as designed, was feasible, and acceptable to both the OBMV staff and their customers.
Importantly, the VDA was able to predict with accuracy which new license applicants were likely to fail their on-road exam (ORE). It could also specify which driving errors on the VDA were associated with failing the ORE and with serious safety concerns, which can help with provision of customized feedback and improved overall driver training.
The technology used in the pilot is called Ready-Assess™ from Diagnostic Driving, Inc.
As background, a research team at CHOP developed and validated a laboratory-based driving assessment that safely exposes drivers to common serious crash scenarios and measures skill in avoiding crashes. CHOP invested in a new spin-out company, called Diagnostic Driving, Inc., which developed this technology as a commercial-grade mobile delivery system that doesn’t require expensive fixed-based simulators. The core component that drives Ready-Assess™ is its cloud-based software application.
The process of adapting Ready-Assess™ for use in OBMV licensing centers was a true partnership with OBMV’s IT staff and driving examiners, the Diagnostic Driving, Inc. team, and CIRP researchers. We are now in the next phase and expansion of this work with our partners in the Ohio Traffic Safety Office.
What Did We Pilot?
Over a 14-month period and in five different OBMV licensing centers, 20,000 new license applicants took the VDA as a 15-minute workflow right before their scheduled licensing on-road examination.
Each applicant was guided by OBMV staff to a workstation set up with a standard PC with a monitor, mouse, keyboard, headphones, and an off-the-shelf USB-based steering wheel and pedals (supplied by Logitech). In the completely self-guided workflow, applicants put on headphones; went through an orientation video and an acclimation drive; took the VDT; and then could opt in to receive automated feedback sent to their email address—all in less than 15 minutes.
There were 10 variations of driving routes deployed at random -- each incorporating the nationally recognized common serious crash scenarios, as well as a couple of supplementary tasks defined by the OBMV.
During the pilot, applicants and driving examiners were blinded to their results so as not to bias the ORE testing. In addition, regardless of whether they passed or failed the VDT, all applicants proceeded to take the ORE so that the dataset could be used to develop a scoring algorithm that predicted ORE results.
Data on the first 4,643 VDA results were linked to their corresponding ORE results to comprise the analytical dataset on which the scoring algorithm was developed. .
What Did We Find?
Metrics concerning operations, feasibility, ease-of-use, and scalability were all met:
- 94% of applicants who attempted the virtual driving test were able to complete it. Of note, those who could not complete it were also more likely to fail their ORE.
- Compared to those who passed their ORE, applicants who failed the VDA were 4.15 times more likely to fail their ORE.
- Our algorithm for scoring the VDA could identify individual-level performance errors and found that specific serious driving errors predicted failure of the ORE.
- When we added in customized feedback about those errors, 40% of applicants opted to receive it via email and 25% opened their email within 24 hours of getting it. Results from a focus group of applicants receiving the feedback revealed they felt it was easy, useful, and appreciated that the OBMV provided it. With a little more education and awareness about the program, these rates could easily increase.
Next Steps
As announced by Governor Mike DeWine on November 21, 2019, the VDA is being adapted as Ohio- Ready, Test, Drive! Over the next year, all of Ohio’s 57 driver examination locations and 350 of Ohio’s driver training schools will be outfitted with the driver assessment systems and the Ready-Assess™ software.
We at CIRP will partner with the ODPS to collect non-identifiable driver performance data from the virtual driving assessments linked with drivers’ future crash information to better inform licensing and driver training curriculum. Examining new drivers’ pre- and post-licensure performance and crash data will allow Ohio to make data-driven enhancements to the statewide training curriculum to better prepare future drivers. The goal is better prepared new drivers who crash less. I will be talking about this more in my next blog post to publish next week.
There is much more detail and graphics about every stage of the pilot in the Research Brief. Read Ohio Portable Driver Simulator System Pilot: Implementation of a Virtual Driving Test Within Ohio’s Driver Licensing System Workflow