New Autonomous Mileage Reports Are Out, but Is the Data Meaningful?
[Stay on top of transportation news: Get TTNews in your inbox.]
Alphabet’s Waymo and General Motors-backed Cruise saw their autonomous vehicles require less help from human drivers on California’s public roads in 2019.
Waymo drove 13,219 miles on average before a human safety driver had to take control of the vehicle and Cruise saw its autonomous cars drive 12,221 miles before the autonomous system was disengaged, according to data released Feb. 26 by the California Department of Motor Vehicles.
The autonomous vehicle disengagement reports are published annually to monitor the performance of self-driving systems by many of the biggest companies pursuing this technology. But in recent years, it has drawn criticism from those very companies, who question the value of the data.
Two key statistics are published: total miles driven autonomously with no input from a human driver; and disengagements, meaning deactivation of the autonomous mode and/or intervention by a safety driver. Companies holding a permit to test self-driving technology on California roads are required to self-report the figures to the DMV. It is a selective snapshot of how far companies’ autonomous ambitions have progressed, but it is the only standardized measure of this technology available to the public in the U.S.
Host Seth Clevenger went to CES 2020 in Las Vegas and met with Rich Mohr of Ryder Fleet Management Solutions and Stephan Olsen of the Paccar Innovation Center to discuss how high-tech the industry has become. Listen to a snippet above, and to hear the full episode, go to RoadSigns.TTNews.com.
In recent years, Waymo, Cruise and startup Zoox Inc. have led the way in autonomous testing, based on miles driven and miles per disengagement. Zoox reported doubling its public road testing to more than 60,000 miles last year, while its disengagement rate stayed roughly the same at about one every 1,700 miles. Apple Inc. drove around 118 miles per disengagement on average, up from just one mile per disengagement a year earlier. However, the company also significantly cut the number of autonomous testing miles. All told, the 62 companies required to report drove nearly 2.9 million autonomous miles during the reporting period.
Industry leaders, academics and automotive analysts have questioned the utility and reliability of the data, because the companies operate under varying circumstances. Some say it should not be used to benchmark one company against another.
“We appreciate what the California DMV was trying to do when creating this requirement,” Waymo posted on Twitter after the report’s release, “but the disengagement metric does not provide relevant insights into the capabilities of the Waymo driver or distinguish its performance from others in the self-driving space.”
1/7 We appreciate what the California DMV was trying to do when creating this requirement, but the disengagement metric does not provide relevant insights into the capabilities of the Waymo Driver or distinguish its performance from others in the self-driving space. — Waymo (@Waymo) February 26, 2020
Kyle Vogt, the founder and chief technology officer of Cruise, in a January blog post described the report as “woefully inadequate for most uses beyond those of the DMV.” He wrote: “The idea that disengagements give a meaningful signal about whether an AV is ready for commercial deployment is a myth.” Cruise’s miles per disengagement rose 134% in 2019; the company almost doubled the number of test miles.
Vogt displays an articulating radar during a GM Cruise reveal event in San Francisco on Jan. 21. (David Paul Morris/Bloomberg News)
The reports “are not intended to compare one company with another or reach broad conclusions on technological capabilities,” said DMV spokesman Marty Greenstein. “Permit holders all have different goals and business models, and are testing in different ways, locations and conditions with different amounts of vehicles.”
What counts as a disengagement varies. Different testing environments yield different results, with many disengagements occurring in congested urban areas. Human intervention typically occurs less frequently on highways and in less populated areas.
“It comes down to standardization,” said Pete Kelly, managing director of researcher LMC Automotive. He believes policymakers should spell out more clearly the circumstances in which a safety driver should or should not take control from the autonomous system. A better measure of the technology’s evolution will come only when legislators require more detailed reports, with fuller descriptions of the vehicle’s behavior during disengagement.
Industry executives including Cruise’s Vogt don’t like that the DMV’s disengagement metric is used to infer whether self-driving technology is ready for commercial deployment. Vogt suggests that more qualitative analysis of autonomous testing, such as reviews of large volumes of raw video material, could better inform companies and regulators of the reasons for disengagement.
Urmson by David Paul Morris/Bloomberg News
Aurora Innovation Inc. CEO Chris Urmson, a former head of Google’s self-driving team, says that using the disengagement rates as a benchmark of progress is misguided and that what counts is the quality of autonomous miles driven.
Aurora is moving away from collecting the data generated from testing on public roads. The company drove the majority of its 39,729 test miles on public roads in 2019, with a driver in control of the car; only 13,429 of those miles were fully autonomous. Instead, Aurora has focused on simulated testing, running more than 750,000 tests in a virtual lab.
In a recent blog post, CEO Urmson wrote that a better approach for the DMV would be to join other state and federal agencies and regularly communicate with the companies developing autonomous technology. “If the company or the public is measuring by disengagement rate, then you create this kind of odd incentive,” Urmson said. “The drivers in the vehicles will be like, ‘Oh geez, I don’t know if I want to just engage at that moment, because it’s a pressure.’ ”
Jim McPherson, a transportation lawyer specializing in autonomy, said some of the objections are valid, “but some of them also come across as disingenuous.” McPherson believes companies are not adequately explaining the circumstances of each disengagement. California’s regulations do require companies to add written details — such as weather conditions, traffic conditions or surrounding construction — in enough detail to explain what triggered the disengagement.
At stake is billions of dollars in potential revenue tied to a change in traffic systems globally. BloombergNEF expects 27 million robotaxis on the road globally by 2040, while Cruise CEO Dan Ammann has claimed there will be a $1 trillion addressable market in the U.S. alone for autonomous ride hailing. Waymo, seen as a front-runner to deploy a commercial service first, has been valued by some analysts at more than $100 billion.
Cruise does all of its California testing in San Francisco, a city of rolling hills, complex road systems and multiple vehicle types such as cable cars. Other companies like Waymo, Zoox and Aurora are testing both in San Francisco and in more suburban settings such as Mountain View or on the freeway, as well as in additional states, where regulations are different. For example, Arizona and Florida have become hubs for testing, in part because they don’t have disclosure requirements. Waymo operates a small autonomous ride service in suburban Phoenix for around 1,500 paying users. The vehicles usually have human safety drivers behind the wheel. In California, the California Public Utilities commission is running a more limited pilot, with five companies giving rides to the public but not being able to charge fares.
Vehicles traveling along Highway 101 in San Francisco. (David Paul Morris/Bloomberg)
One notable company missing from the pack of leaders in the DMV reports is Tesla. The automaker is working toward deploying fully autonomous vehicles by the end of this year, having said it plans to issue a fully self-driving version of its Autopilot feature. The deadline initially was set for the end of 2019. CEO Elon Musk also said Tesla customers would be able to put their cars into a shared robotaxi network this year, but clarified recently that this will depend on regulatory approval.
Tesla, whose valuation has swelled on optimism of its technological lead, holds a California DMV testing permit but does not actively run autonomous systems on California’s public roads. The company says it drove a single 12-mile autonomous journey around the perimeter of its Palo Alto headquarters in April 2019; it did not report any disengagements. It is developing full self-driving by gathering data from its vehicles globally and using it to build and train neural networks.
“The winner in an AI project has the most data and the highest quality data,” Cathie Wood, the CEO of Ark Investment, said recently on Bloomberg Television.
Want more news? Listen to today's daily briefing:
