FMCSA: Decision on Adoption of New Item Response Theory Model a Year Away
[Stay on top of transportation news: Get TTNews in your inbox.]
Federal Motor Carrier Safety Administration officials will not make a decision until September 2020 whether to adopt a new, complex statistical model known as the Item Response Theory to better evaluate the safety of motor carriers, according to a new audit released by the U.S. Department of Transportation Inspector General.
But even if the agency does decide to adopt the IRT model recommended by an academic panel of the National Academy of Sciences, FMCSA officials are concerned they won’t be able to explain the model to the trucking industry or collect additional data recommended for the new model to succeed.
The DOT IG’s Sept. 25 audit was conducted to comply with a mandate under the FAST Act to review FMCSA’s June 2018 “corrective action plan,” which was intended to revamp the Safety Measurement System that underpins the agency’s Compliance, Safety, Accountability Program, which is used to distinguish unsafe from safe motor carriers.
The panel’s study analyzed the ability of FMCSA’s current SMS to discriminate between low- and high-risk carriers, assessed the public usage of SMS, reviewed the data and methodology used to calculate the measures, and provided advice on additional data to improve collection and safety assessment methodologies.
The audit also drilled down on FMCSA’s early plan of attack to incorporate a slate of recommendations for CSA from the national academy panel, which studied the CSA program for 15 months.
“According to FMCSA, if the [IRT] model performs well and can be reasonably explained to stakeholders, it will replace the [CSA program] Safety Measurement System, as the National Academy of Sciences recommended,” the audit said. “Adopting the IRT model could help FMCSA address the methodological challenges inherent in SMS and enhance the credibility of the CSA program.”
However, the audit concluded that the agency may not be able to update the system for prioritizing motor carriers in an “accurate, efficient and timely manner.”
In a written response to the audit, FMCSA said it already has developed a variety of models, including a “full-scale” IRT model. It also said it determined that much of the data the panel requested it collect was not readily available, and said it had completed the aggregation and consolidation of 899 current potential violations down to 381 violations.
However, the audit noted that FMCSA “remains concerned” about its staff’s ability to explain the model to the motor carrier industry and the public because the agency does not have an in-house IRT expert.
While the national academy panel’s June 2017 study lauded FMCSA for its development of the CSA program, it said the agency should improve the program with a more data-driven way of collecting and compiling carrier performance obtained from sources such as roadside inspections, crash reports, compliance and reviews.
In the nearly nine years since CSA was first introduced, motor carriers and government auditors have been critical of the CSA’s data collection and manipulation, although the program is widely believed to have raised the safety consciousness of the industry.
The IG audit was mixed in its findings.
“FMCSA’s plan to make its carrier prioritization program more transparent and improve its assessment of motor carrier safety rankings does not clearly address the deficiencies and opportunities identified by the National Academy of Sciences and is missing an implementation plan,” the audit said.
The audit also said the agency does not plan to address the academy panel’s recommendation regarding the use of relative and absolute safety measures until it makes a decision on adopting the IRT model.
In its corrective action plan, FMCSA said it would work more closely with states to collect data, improve the uniformity of police accident reports data, and investigate ways to collect data on motor carrier characteristics such as driver turnover rate, type of cargo, method and level of driver compensation, and carrier vehicle miles traveled.
However, the audit says now FMCSA has determined it does not plan to follow up on the panel’s recommendations to collect additional data.
“Regarding the NAS recommendation on improving the quality of carrier data, FMCSA determined that much of the data to be collected do not exist,” the audit said. “As a result, FMCSA no longer plans to collect additional data and thus will not identify the associated costs as the FAST Act requires.”
“FMCSA’s plan to make its carrier prioritization program more transparent and improve its assessment of motor carrier safety rankings does not clearly address the deficiencies and opportunities identified by NAS and is missing an implementation plan. Finally, FMCSA does not plan to address the NAS recommendation regarding the use of relative and absolute safety measures — until it makes a decision on adopting the IRT model.”
Want more news? Listen to today's daily briefing:
