Complex Proposed CSA Replacement May Take Time to Decipher
A possible replacement for the much-maligned Compliance, Safety, Accountability framework the Federal Motor Carrier Safety Administration uses to measure motor carriers’ safety performance has been recommended by an elite panel of academics, but its complexity may prove challenging to the trucking industry, experts said.
“It’s going to take time for the consumers of the information and motor carriers to really understand,” Sharon-Lise Normand, panel co-chairman and professor of health care policy at Harvard Medical School, told Transport Topics about Item Response Theory, a data measurement model that has been used to analyze a range of social issues, including medical care and aptitude tests. Now it is the recommended replacement for CSA, which relies on seven Behavior Analysis and Safety Improvement Categories, or BASICs, to establish carriers’ safety scores.
It is still too early to draw any specific conclusion about the IRT model.
Sharon Worthy, FMCSA spokeswoman
FMCSA authorized a study that was conducted over 15 months by the panel, several of whom specialize in statistics and transportation safety policy. The 132-page study was released in June by the National Academy of Sciences and cost an estimated $1 million, a NAS spokeswoman said.
If Item Response Theory, or IRT, helps identify the safety culture of the nation’s motor carriers, the 12-member academic panel said FMCSA should use it to replace the current CSA system. But it should take its time, Normand said.
“We recommended a two-year timeline to develop the model so you’re not falling off a cliff,” she said.
“It is still too early to draw any specific conclusion about the IRT model,” FMCSA spokeswoman Sharon Worthy told TT. “Additionally, FMCSA has begun its review of the data and related recommendations from the NAS. Going forward, FMCSA is committed to a transparent process and will be providing opportunities for public input as the recommendations are developed.”
The agency is in the process of finalizing a corrective action plan for CSA for submission to Congress, as specified in the FAST Act, Worthy said.
Normand and other panel members said IRT would require drilling down on statistical data that could be used to actually create new safety rating categories based on the data, rather than relying on expert opinion.
In an attempt to simplify the concept, they often compare it to the SAT tests used in high schools. A student’s SAT score is not based on the sheer number of correct answers. Rather, those who have a mastery of a subject such as math or science are determined by how many of the difficult questions they can answer correctly, Normand said.
“There are some very difficult questions where fewer students get a question correct and there will be easy questions that everybody should get right,” Normand said.
Similarly, with the IRT model the agency’s safety culture ranking could be restructured based on an analysis of violation numbers using the violation data to create rating categories.
For example, if everybody gets hit with a specific violation, such as speeding or tail lights out, that’s not helpful in distinguishing the carrier with a good safety culture from the one with a bad safety culture, Normand said. However, if only a few carriers are cited with a specific violation, it could be an indication of a safety culture problem for the carrier who received the citation.
“If everyone fails on something — it’s kind of like the easy math question — it might not tell us that much,” said Daniel McCaffrey, an experienced researcher and member of the Academy of Sciences panel that conducted the CSA study. “For example, if you have a driver who’s under the influence, that one might be pretty rare. But it also might be the kind of thing that’s a strong indicator that you’ve got a serious safety problem. So you want to give that a lot of weight.”
He said IRT should work for the trucking industry. “It embodies a lot of the ideas that were behind the BASICs to start with in the sense that there’s a lot of criteria, and the different criteria are all telling us something, but they aren’t all equal.”
With the IRT model carriers can be more confident that their safety scores are accurate since they’re based on the data, and the model and framework will be flexible, he said.
