Study Questions if CSA Scores Reflect Fleet Safety Performance

This story appears in the July 19 print edition of Transport Topics.

Preliminary results from an independent review of the Federal Motor Carrier Safety Administration’s CSA safety monitoring system are raising questions about its ability to accurately identify high-risk carriers.

Paul Green, a researcher with the University of Michigan Transportation Research Institute, said that the CSA system is not showing a correlation between fleets that score well in two of the seven primary safety categories, called BASICs by FMCSA, and reduced accident levels.

“Most of their BASICs are associated with crash rates in a nice way but there are two that are troublesome,” driver fitness and load securement, Green told Transport Topics. “They don’t correlate well with crash rates, and our gold standard for safety is crash rates.”

These issues have led to at least one industry group to issue a new call to delay implementation of the CSA program until after the review — being conducted for FMCSA — is completed in December.

According to a presentation of UMTRI’s preliminary findings, a high score in the driver fitness BASIC, which includes training, licensing violations and a variety of health-related infractions, has an inverse relationship to crashes. That means that the higher the rate of violations, the lower the number of crashes experienced.

For the load securement BASIC, which includes violations relating to hazardous materials packaging as well as improper marking and tying down of cargo, Green’s data shows a bell-curve relationship: the fleets with the most and least number of violations having roughly the same risk of crash.

Green said load securement and driver fitness BASICs may “be measuring something else, but that doesn’t mean that it’s not important.”

“It may be something like paperwork or something that a carrier should be doing that they’re not doing that is not necessarily related to crash rates, but may be something FMCSA wants them to be doing,” he said.

FMCSA spokeswoman Candice Tolliver said the agency was aware of UMTRI’s preliminary results, but that those results “in no way suggest the driver fitness and load securement BASICs are not indicators of a safety risk.”

“In general, it does identify high-risk carriers, but it is not perfect,” Green said. But he warned that in the driver fitness and load securement areas the system may ultimately target safe carriers by mistake.

“Let’s say a carrier just exceeded [the BASIC] in one of those two that would become questionable,” he said. “They would get pinged and that may not necessarily be related to a safety issue.”

To address that, Green suggested changing how the agency weighs the BASICs relative to each other.

“Maybe [high scores] should not be such a serious thing if its one of those two; maybe that would be the correct thing to do,” he said.

The UMTRI report was the subject of scrutiny during a June congressional hearing as both industry officials and members of Congress pushed the agency to wait until the report is finalized before rolling out the CSA program (6-28, p. 3; click here for previous story).

Last week, Rob Abbott, vice president of safety for American Trucking Associations, renewed the group’s call to hold off on full implementation of CSA until UMTRI completes its review.

“If parts of CSA 2010 measure compliance with rules that do not have a firm relationship to risk reduction, the system will publicly label some carriers as safety-deficient — even if they have better-than-average crash rates,” Abbott said. He added the “program is very important and the results of the UMTRI evaluation should be known before implementation.”

Abbott said ATA was worried that without further study or examination, FMCSA “risks targeting the wrong carriers and erroneously labeling responsible carriers as unsafe.”

Green told TT that he didn’t think delaying CSA implementation was possible, and his interpretation of discussions with FMCSA about the study was that “their opinion was that our evaluation was not designed to in any way influence the rollout of the project.”

“FMCSA will use the findings of the UMTRI study as well as other data from the operational model test to continually strengthen the CSA 2010 program,” Tolliver said.

Trucking industry consultant John Hill said the agency “has a significant responsibility to go back and evaluate, maybe not change, but at least evaluate and make adjustments as necessary” to its program based on the UMTRI findings.

Hill was formerly FMCSA administrator and one of the architects of the program.

Tolliver defended CSA, saying “the test results to date, including data that shows increased oversight efficiency, capacity to reach a greater number of carriers and a more comprehensive safety measurement system will enable FMCSA to launch CSA 2010 in the fall.”