Opinion: FMCSA Should Accept NAS Recommendations for CSA
In late June of this year, 18 months after the FAST Act removed CSA Scores from public view, the National Academies of Science, Engineering and Medicine, or NAS, published its preliminary findings outlining recommendations to reform the Compliance, Safety, Accountability program. For industry stakeholders, each of the four categories in the NAS report is worth a closer look.
For example, in its assessment of the current Safety Measurement System, or SMS, NAS noted that while SMS is structured in a reasonable way, and that its method of identifying motor carriers for alert status is defensible, much of what is now done is ad hoc and based on subject matter expertise that has not been sufficiently empirically validated. This argues for the Federal Motor Carrier Safety Administration adopting a more statistically principled approach in SMS.
Bryan
While NAS gives more credit than I would to the subject matter expertise on which CSA originally was developed, and correctly points out that there has never been an identifiable statistical model that FMCSA has relied upon, I agree that the concept behind CSA is sound. Something like CSA makes sense; it just needs to have sound science behind it.
Discussing a more Natural Statistical Model, NAS recommends that FMCSA develop an Item Response Theory, or IRT, data testing model over the next two years. And if this model demonstrates it can perform well in identifying motor carriers for alerts, it should replace SMS in a manner akin to the way SMS replaced SafeStat, NAS said.
I believe that the science of the IRT model is far superior to the ad hoc pseudo-science of the current methodology. Therefore, IRT can improve CSA, even with data that is not as rich as we’d like it to be, and even in a complex environment such as trucking, with 50 enforcement regimes, millions of trucks, differing cargo types and tasks, and a huge variety of operating environments.
Data issues covered by NAS, in a report section on Improvement of MCMIS (Motor Carrier Management Information System) data, point out two specific elements that require immediate attention: carrier exposure and crash data. Specifically, the current exposure data are at low-frequency sampling, and data that are collected are likely of unsatisfactory quality. Crash data also are missing too often, and information is available from police reports that currently is not represented on MCMIS. That data could be helpful in understanding the contributing factors in a crash and could help validate assumptions linking violations to crash frequency. MCMIS captures data from field offices and is the authoritative source for inspection, crash, compliance review, safety audit and registration data.
NAS recommends that FMCSA continue to collaborate with states and other agencies to improve the quality of MCMIS data in support of SMS. As the quality of existing data reported today focuses on vehicle miles traveled, or VMT, and truck count (APU), as NAS correctly points out, “the impact of flawed or out-of-date data on SMS percentile ranks may not be fully appreciated by the carriers.” Better reporting of these two data elements would improve the data quality in CSA today and make the future IRT model more trustworthy.
NAS also suggests that VMT by state, cargo type, driver pay and pay model, and driver retention rates be reported to FMCSA and that this data could be available as a result of the ELD mandate. However, while providing more data is a worthy goal statistically, that becomes problematic in practice.
In terms of Transparency, Reproducibility and Public Disclosure of Safety Rankings, NAS recommends that FMCSA should structure a user-friendly version of the MCMIS data file for use as input to SMS without any personally identifiable information to facilitate its use by external parties, such as researchers, and by carriers. In addition, NAS says FMCSA should make a user-friendly computer code to compute SMS elements available to individuals in accordance with reproducibility and transparency guidelines.
FMCSA must strike a more collaborative relationship with the industry, and demonstrate absolute transparency in how it is collecting, analyzing and taking action on a new CSA data model. Perhaps a less combative, more inclusive partnership can be developed, and we can reboot from the very controversial launch of CSA in 2010.
NAS rightly compliments FMCSA for the work it has done on CSA and thanks everyone who contributed to the project. Credit is given to more than 125 individuals representing every stakeholder in trucking, including FMCSA, law enforcement, insurance, ATA and other trucking associations, bus and driver representatives, statisticians, lawyers, safety advocates and software developers.
Going forward, however, there needs to be a sound statistical model applied to replace the ad hoc nature of how CSA is formulated today. More and better quality data need to be gathered, and the data utilized need to be more transparent and software developed to make such data more useful to all stakeholders. Only then should CSA scores be released to the public.
It is my hope FMCSA accepts the recommendations of NAS and seriously begins to work on the next phase of CSA.
Vigillo, a SambaSafety company based in Portland, Ore., is the provider of CSA Daylight Suite, a diagnostic and management tool, and CSA data monitoring solutions.
