GAO Says SafeStat Data Better but Urges New Analysis Method
By Sean McNally, Senior Reporter
This story appears in the June 18 print edition of Transport Topics. Click here to subscribe today.
In the first of a pair of reviews of the Transportation Department’s truck-safety rating system, the Government Accountability Office said DOT’s Federal Motor Carrier Safety Administration has improved the quality of data in its SafeStat program but should change the way it calculates truck-fleet safety rankings.
SafeStat, which is used by FMCSA to determine which carriers to target for safety enforcement actions, has been a controversial subject for several years, with industry officials and government investigators citing the program for poor data quality and questioning the way it generates its safety rankings (2-23-04, p. 1).
GAO’s June 11 report said FMCSA could improve its ability to identify carriers most at risk of having crashes by 9% if it would use a “binomial regression model” to analyze raw safety data.
“While SafeStat does a better job of identifying motor carriers that pose high crash risks than does a random selection, regression models GAO applied do an even better job,” the report said.
SafeStat currently includes four pieces of data: driver information, vehicle safety information, safety management information and a company’s crash record, Carl Barden, a senior mathematical statistician with GAO, told Transport Topics. Those pieces are simply added together to produce a score, with the crash piece weighted the heaviest.
“Their formula never changes,” regardless of whether past crashes are the best predictor of future ones, Barden said.
“But in the regression model, it’s whatever fits best with the accident,” said Jim Ratzenberger, GAO assistant director. “It allows for change over time.”
In its findings, GAO said the agency “could more effectively” reduce truck-related fatalities if it “targeted those carriers that pose the greatest crash risks.”
Using the new statistical formula would be “more effective targeting of compliance reviews to the set of carriers that pose the greatest crash risks,” GAO said.
The agency said the statistical model “experienced 9,500 more crashes than those identified by the SafeStat model over an 18-month follow-up period.”
That improvement led the congressional watchdog to recommend FMCSA “apply a negative binomial regression model, such as the one discussed in this report, to enhance the current SafeStat methodology.”
GAO also said it expected “shortly” to issue a study on how FMCSA identifies and takes action against carriers that are “egregious safety violators” and “how thoroughly and consistently FMCSA conducts compliance reviews.”
FMCSA said the GAO study “provided useful insights and offered a potential avenue for further improving the effectiveness of FMCSA’s efforts to reduce crashes involving motor carriers.”
Trucking and enforcement officials said they hoped the report would lead to improved safety.
“This is positive for the industry and for highway safety,” said Dave Osiecki, vice president of safety, security and operations for American Trucking Associations. “I’d be surprised if FMCSA doesn’t follow this recommendation.”
“The good thing is that SafeStat is working,” said Steve Keppler, director of policy and programs for the Commercial Vehicle Safety Alliance. “Yes, it has some recommendations for improving it, but we always want to improve the tools we have.”
While pushing FMCSA to improve its scoring system, GAO did say the agency had made strides.
According to GAO, the median number of days it took for states to report crashes dropped from 225 in 2001 to just 57 in 2005, and the number of crashes reported within 90 days rose to 89% in the 2006 fiscal year from 32% in fiscal 2000.
The DOT’s inspector general office also is conducting a review of SafeStat’s data accuracy. After a 2004 report by the DOT’s inspector general, FMCSA removed the scores generated by SafeStat from its Web site (5-31-04, p. 1).
In 2005, GAO said the system for collecting data had improved, but the data’s quality still needed strengthening (11-28-05, p. 3).
Last year, FMCSA Administrator John Hill said he supported putting the safety rankings back up for public viewing, but would wait until the inspector general’s review.
FMCSA spokeswoman Melissa Mazzella DeLaney said Hill “still wants to wait for the OIG report before making a decision on reposting the SafeStat scores.”
