In mid-January 2019, FMCSA held a public briefing in Washington, part of which focused on the Agency’s preliminary testing of a small scale IRT model. You may recall that IRT was recommended by the National Academy of Sciences in 2017 as an alternative to the CSA Safety Measurement System (SMS) scoring method. At its briefing, FMCSA staff described some preliminary results from a ‘small scale’ IRT model run using violation data from a sample of 10,000 motor carriers. In this sample, only 212 violations (of a possible 899) were cited frequently enough to be used meaningfully in the model, according to FMCSA. Here are some of the preliminary results and Agency observations:

1. Scoring results are “not drastically different…” from results FMCSA is seeing from the longstanding SMS
   a. carriers with the worst safety measures are still appearing at the top of the IRT list, and carriers with the best safety measures are still at the bottom;
   b. IRT gives a “greater level of detail on these measures, and a higher level of certainty in how they are ranked.”
2. IRT is giving FMCSA a “more detailed view of the impact of a violation on safety”;
3. Due to the granularity of the data provided, IRT “may offer FMCSA more opportunities to address carriers with less data;
4. IRT appears to allow FMCSA to account for the impact of geographic, enforcement-related disparities; and
5. IRT might allow FMCSA to move away from rigid Agency-set thresholds and, instead, opt for a more empirical/verifiable approach to setting them.

It’s clear that the CSA improvement process will take more time than what some might have hoped or expected. FMCSA noted that the National Academy of Sciences recommended that the Agency take “two years…” to finalize improvements “…due to complexity and amount of data in the model.” There will be near-term next steps, though. In Spring 2019, FMCSA is planning to release the results of its small-scale model and seek feedback from the industry and other stakeholders.