‘Item Response Theory’ Model Has Potential to Reduce Crashes, FMCSA Says
ORLANDO, Fla. — A new complex statistical model that the Federal Motor Carrier Safety Administration is exploring would utilize data to measure a motor carrier’s “safety culture,” rather than attempt to predict its likelihood of a crash, a top agency official said.
The agency has begun studying how to use the so-called Item Response Theory model to improve its Compliance, Safety, Accountability Safety Measurement System that has been the subject of criticism ever since it went live in December of 2010.
The IRT model was recommended by an elite academic panel of the National Academy of Sciences that studied the CSA program for more than a year.
MCE COVERAGE: All stories | Live blog | Photo gallery
The panel recommended that the agency over a two-year period study and possibly shift to the more “rigorous” data-driven IRT model that would improve crash data and dig deeper into a variety of data that could measure the performance of individual trucks and buses, not just at the motor carrier level.
“What the National Academy said was, ‘FMCSA, the model that you have is pretty good. However, what we would like to see [is] a better statistical model behind SMS,’ ” Joe DeLorenzo, FMCSA’s director of the Office of Compliance and Enforcement, told an Oct. 22 educational session at American Trucking Associations’ Management Conference & Exhibition meeting here.
DeLorenzo said that carrier safety practices are related to crashes, and that violations are indications of unsafe practices.
“The focus is on prevention of crashes, not prediction,” he said.
To get a better picture of a carrier’s safety culture might require collection of such data as vehicle miles traveled, driver turnover rates and driver pay, the study said.
DeLorenzo compared the IRT model to educators finding out what questions are the most important on the Scholastic Aptitude Test. But because those in the freight transportation business who have worked with an IRT model are next-to-impossible to find, the agency will have to seek outside assistance from statisticians, DeLorenzo said.
“The good news is that if we apply this model the way the national academy outlined it, it will give us good data and a good understanding of what is happening,” DeLorenzo said.
“This is one of those things that because it’s so data intensive, it’s going to take some time to run,” DeLorenzo added. “We’re going to take it pieces at a time and we’re going to refine it.”
Bryan by Joseph Terry/Transport Topics
Speaker Steve Bryan, president of data analysis firm Vigillo, Inc., said he supports the IRT model, but called it a “heavy lift.”
“The IRT model is not going to predict the next crash,” Bryan said. “It’s going to put a light on the cultural weaknesses of motor carriers that can be remedied and those cultural weaknesses that lead to higher crash rates.”
“I think this model can get us to a much better safety scoring system,” he added.
But so far, agency officials have had a difficult time explaining the IRT model to carriers. And if the Oct. 22 session was any indicator, many remain confused.
During his talk, Bryan asked the group of carrier executives to raise their hands if they understood how it worked. No hands went up.
“By the time we leave this room, everyone will be able to raise their hands,” Bryan said.
After a demonstration analyzing the answers of a 10-question-quiz he put on Vigillo’s web site, he again ask for a show of hands from those who understand the IRT model.
One hand went up.
