As we became part of the Woven Planet family in July 2021, we’re retiring the Woven Planet Level 5 Medium account. We can now be found on the Woven Planet Medium account.

At Woven Planet, a subsidiary of Toyota, we’re reimagining mobility. Our promise: deliver the world’s safest, smartest and most human-centric mobility — in all its forms, for today, tomorrow and decades to come. We do it with Trust — building safety into all our technologies. We do it with Toyota — leveraging the competitive advantage of a partnership with the world’s biggest vehicle maker. We do it with Arene — our advanced software platform that, combined with Toyota’s hardware expertise, will create software-defined vehicles on a global scale.

Toyota’s long history of success in building ever-better vehicles has laid a strong foundation for its evolution into a more broadly conceived mobility company. Woven Planet was created to help catalyze that transformation through software — including a flexible, robust automotive operating system; safe, reliable AD/ADAS technology; and dynamic, scalable mapping.With a relentless focus on continued advances in safety technology, Toyota is strongly committed to reducing the number of traffic fatalities to zero.

We’ll keep publishing articles to the Woven Planet Medium account that highlight our technology development progress and provide more in-depth insight into our work. We’d like to share our progress with the community, so please follow us and stay up to date on what’s happening at Woven Planet!

The latest milestones in our data-driven approach to autonomous driving

At Woven Planet Level 5, we’re using machine learning (ML) to build an autonomous driving system that improves as it observes more human driving. This is based on our Autonomy 2.0 approach, which leverages machine learning and data to solve the complex task of driving safely. This is unlike traditional systems, where engineers hand-design rules for every possible driving event.

Last year, we took a critical step in delivering on Autonomy 2.0 by using an ML model to power our motion planner, the core decision-making module of our self-driving system. We saw the ML Planner’s performance improve as we trained it on more human driving data. However, at the time we were collecting the driving data using traditional, expensive autonomous vehicle (AV) sensor suites (including lidar and radar).

Today, we are excited to share that we’ve gone two steps further. First, our ML Planner has advanced to a point where it no longer needs expensive sensors to learn from human driving; it can learn effectively from stereo camera data. Second, we developed a camera-only collection device that is 90% cheaper than the sensors we were using before. This device can be easily installed in fleets of passenger cars to gather the data we need to power autonomy development at both low cost and global scale.

Proving that camera data can power our ML Planner

We needed to prove the hypothesis that lower-fidelity camera data could be used for training our ML Planner as effectively as the high-quality data gathered by our AV sensor suite. We began by picking a fairly common scenario — stopping behind a lead vehicle — to see if we could improve its performance with low-cost data from stereo cameras. Previously, we only used data from AV sensor suites to train the ML Planner on this scenario. We mined our camera data to find examples of this scenario type, and added them to the AV data training set.

Proving our hypothesis took some work. The first time we tried using camera data, the ML Planner’s performance was 15% worse compared with training it on high-cost data only. That’s because the output of our perception algorithms was not accurate enough for the ML Planner to learn to drive from the camera data, confirming our intuition that this would be much harder to do with these lower-fidelity sensors. The Level 5 team focused on iteratively improving our perception algorithms for camera data in a tight loop with the ML Planner’s comfort and safety metrics. That allowed us to target improvements to the perception system that had the highest impact on ML Planner performance. After nine months of work, the ML Planner’s performance improved by 42% on the “stop behind lead” scenario thanks to the camera data, and by 5% across all driving scenarios. We now had hard evidence that camera data was useful for training our ML Planner.

Achieving a 5% overall performance improvement was a promising start, but the camera data was a small proportion of the total training data. Next, we wanted to know what happens if the majority of the training data comes from cameras. We increased the proportion of camera data, and witnessed a critical breakthrough: the ML Planner’s performance was comparable to when it was trained exclusively on high-cost data.

The significance of this result is profound. Not only can we add targeted amounts of camera data to improve the ML Planner on specific scenarios, we can also train the ML Planner effectively even when camera data is our primary source.

We are now confident that our ML-based approach can learn from low-cost cameras installed on human-driven cars. Alongside this, we began scaling up our ability to collect camera-based data.

Meet the device collecting data at scale

To begin collecting data at scale, we built our own in-house device equipped with stereo cameras. The cameras are housed in a roof pod that makes it easy to non-invasively mount them on fleets of human-driven passenger vehicles. This allows us to deploy them at scale — we’ll be rolling them out in the Bay Area and beyond in 2022.

Our low-cost sensor suite is made up of only stereo cameras.
These vehicles were selected prior to our joining Woven Planet, and we look forward to transitioning the devices to a fleet of Toyota vehicles shortly.

Accelerating forward with a scalable approach to AV development

By design, we can train our ML Planner on a combination of data collected from different sensor configurations with varying levels of fidelity.

So far, this allowed us to use a mix of data from both AV sensors and camera sensors. It also gives us the opportunity to leverage Woven Planet’s unique advantage: access to diverse driving data at unprecedented scale through Toyota, one of the largest automakers in the world. Access to this sheer amount of human driving data will enable us to bring our ML-first Autonomy 2.0 approach to production.

Interested in joining our team to help solve the automated driving challenge? We’re hiring. Check out our open roles to join a team of passionate engineers making transportation safer and more accessible to people everywhere.

Powering Data-Driven Autonomy at Scale with Camera Data was originally published in Woven Planet Level 5 on Medium, where people are continuing the conversation by highlighting and responding to this story.

The challenge with current model evaluation

In any machine learning (ML) project, examining and improving models is critical to achieving a goal. At Level 5, part of Woven Planet, our goal is safe autonomous driving. This is one of the biggest challenges being tackled using ML, and we need to be able to inspect and debug our models with precision and granularity.

Machine learning researchers and practitioners usually rely on metrics averaged over all data points to compare ML models and select the best among them. For example, in computer vision, “mean average precision” (mAP) is heavily used to summarize and compare multi-class detection models. But average metrics like mAP mainly fit the purpose of improving the model architecture in an academic setting where performance trade-offs on individual data samples are not a major concern. In the autonomous vehicle (AV) industry, relying only on these kinds of metrics is insufficient. We need to be able to:

1. Understand corner cases because they’re crucial to safely deploying AV models.
2. Derive actionable insights by studying failure cases and identifying patterns.
3. Compare inference results side-by-side to analyze improvements and regressions.

We can rethink this!

Level 5’s model introspection framework allows our ML engineers to carefully debug their models. This framework makes it possible to log all the desired metrics and metadata for each sample in the evaluation dataset. In less than a minute, we can sort or filter the entire set using any desired combinations of the logged metadata.

We now can understand corner cases better by sorting hundreds of thousands of samples — and exploring their corresponding metrics, ground-truth, and model-prediction visualizations — all via a web interface (we typically use interactive Weight and Biases (W&B) tables).

Our framework enables us to find large-scale, data-driven insights on every trained model by utilizing distributed W&B tables. This allows us to interactively combine, filter or group different metrics until an actionable pattern is found. For example, as we inspected our recent object detection and classification model, we used the web interface to filter for pedestrian false negatives, looked at a few examples, and ran a few quick one-minute queries to explore all samples. When we sorted by ground-truth bounding box area, we discovered that misdetections are more severe for smaller bounding boxes. Then, we compared bounding box area distribution for false negatives versus correctly classified pedestrians. We saw a significant imbalance where it was clear that our next step would be data balancing (illustrated in the figure below).

Once we observe a pattern, it’s easy to attach a dynamic snapshot of these interactions to a sharable interactive report. Team members can apply the same explorations on top of another model’s output simply by changing the model name, all instantaneously in the web browser. Even better, it’s easy to turn summaries of these explorations into trackable metrics for all models.

How to scale and productionalize

In order for our model introspection framework to be useful beyond a few experiments, it needs to be productionalized and scalable. Here is how we addressed both requirements:

1. We developed the ability to record sample data for any intermediate steps during training in the GPU cluster.
2. We developed the ability to perform post-process statistical aggregations on the recorded samples and publish all these samples into a visualization tool. (Note that the CPU-intensive aggregation logic will be offloaded to the CPU cluster.)

Data collection

The framework provides a simple API for users to record all samples with additional metadata and model output such as tensors, images, and videos. These intermediate data samples can be either part of the ground-truth or generated by model prediction. The data will persist and will transparently upload to the cloud in the background. The framework has these data properties:

• Integrity. The framework offers strong data integrity by enforcing schema to the data format. Considering the fluidity of AV data, this helps maintain consistent data between different experiments and models.
• Scalability. Samples are gathered in chunks that support columnar data-access patterns and random access. This is critical to make both reading and writing efficient. Because there can be millions of tiny samples (such as tensors), as well as larger ones (like images and video), it’s critical that reading and writing is efficient. The persister splits large binary columns, such as images and video, into separated binary blobs. That ensures the large binary contents won’t interfere with our access to the tiny samples enabling us to read sample metrics in orders of magnitude less time.
• Durability. The gathered chunks or blobs are uploaded to the cloud in the background. The framework hides the details of the storage media by providing a thin abstraction layer for writing, retrieving and reading the samples.

Sample writer interface

To record interesting samples, there are two main API calls: write_sample and commit. For each sample, the user calls write_sample to publish the sample to local storage. This call is parallelizable, which means that the data can be published concurrently from multiple processes. Once all samples are written, making a commit call reorganizes the recorded samples. The reorganized samples then upload to the cloud so the data can be accessible from anywhere.

Sample reader interface

Because the data is accessible from anywhere, the logged samples can be retrieved by providing the dataset name and version to the API, as shown below. Once the logged samples are retrieved, they can be filtered to expose the aggregate samples to the desired datasets.

Scaling metrics aggregation

Users have to claim a fixed set of GPU nodes for a training task and thus the number of GPU/CPU cores are predefined. It is impossible to scale metrics aggregation in a training environment. Processing of millions of rows of sample data may require a lot more CPU resources than the training environment can provide.

Though training/data collection and metrics aggregation execute in different environments, we provide a seamless experience so all the development and testing happen in the same repo with the same interface. The framework makes handling sample data, offloading, and launching the metrics aggregation transparent to users. By doing so, we can keep our training cluster efficient while being able to process and analyze large amounts of data in the CPU cluster. Other limitations for conducting metrics aggregation in GPU cluster are:

• Restrictive data format. In the training environment, data is synced among nodes as tensors. This makes it less efficient and flexible for CPU-intensive compute logic.
• Expensive GPU cycles. Offloading metrics aggregation improves the efficiency of the GPU cluster and thus reduces costs.

Here’s how it works:

At the end of an ML training task, the framework triggers the metrics aggregation job in the CPU cluster by only invoking the specific entrypoint. End users will see the same image and programming paradigm when developing the training and metrics aggregation logics.

Users can implement the metrics aggregation logic, by leveraging the following set of APIs. They can specify the desired number of CPU workers, without worrying about where/when/how the logic will be executed.

• num_workers specifies the number of workers to distribute the aggregation tasks.
• run_metric_worker contains the main logic for metrics aggregation. Users can pull data via the sample reader API, as mentioned in the above session.
• commit_metrics will be invoked by the framework, after completing all the aggregation tasks. It will be called only once to finally commit the data or notify about the status of the current job.

Case Study: Improving ML Planner with safe stopping

Our data-driven ML Planner learns to predict the next few seconds of AV trajectory using the current frame and past historical information, instead of using hand-tuned heuristics. When we were evaluating the model, our overall metrics indicated that the planner had a lot of front collisions. By using model introspection, we were able to see that many of the front collisions were due to the AV failing to stop behind the lead vehicle.

Unaware of which cases were causing the front collisions, we logged thousands of scenes, along with metadata about each scene. The three main types of metadata are scene visualization, scene category, (e.g., stopping in front of a lead vehicle) and model performance according to the evaluator (e.g., number of failed frames for front collision).

Since we knew the planner was struggling with front collisions, we sorted the table by scenes that had the most failed front collision frames. We noticed that there was a lead vehicle in many of the scenes, which indicated that the planner was having a difficult time stopping with a lead vehicle.

Armed with that insight, we considered the cause. One hypothesis was that we don’t have a sufficient amount of examples that teach the model to stop behind a lead vehicle. So we curated more scenes where the ego vehicle slows down with a lead vehicle. Then, after we trained a model with this additional data and used model introspection to check the results, we found we were able to reduce front collisions by 3x by adding more scenes for the specific scenario..

The previous process was laborious and time-consuming; to find scenes prone to collisions, we had to manually look through a limited amount of randomly logged data until we would find an interesting case. And, we would have to curate general scenes to improve collision metrics instead of finding germane scenes.

By using model introspection, we were able to sort through all the inference samples in under a minute and quickly identify cases where the failures were occurring. Interestingly, it takes only a couple more minutes for the framework to gather and publish the data to the cloud.

The use of model introspection not only increased our ML engineering productivity, but also allows us to easily and efficiently identify failure cases. Since the framework is generalizable and scalable, we could apply it to different ML models and datasets across our organization.

Conclusion

We developed a model introspection framework that enables team members across our organization, including machine learning engineers, to quickly understand and debug their learned models. This framework scales to all the models trained daily across our team with very efficient resource consumption and a large degree of flexibility to understand new problems with trained models. We achieved this with architecture that is very entangled with our distributed training and evaluation setting.

Interested in solving problems like this? We’re hiring. Check out our open roles and join a team of passionate engineers building world-class ML ecosystems that will help scale the development of autonomous vehicles.

Improving Machine Learning Models for Autonomous Vehicles was originally published in Woven Planet Level 5 on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Moritz Niendorf, Senior Engineering Manager; and Alan Agon, Product Manager

At Level 5, part of Woven Planet, we believe our machine learning-first approach to motion planning for autonomous vehicles (AVs) will chart the course for the future of self-driving technology. Our Machine-Learned (ML) Planner is now the default planner operating on our autonomous vehicles driving in San Francisco and Palo Alto.

Today, we’re excited to share examples of our machine-learned motion planning system in action and insight into how we’re building it.

In with Autonomy 2.0

The motion planner acts as the decision-making center for autonomous vehicles. It sits in the middle of the AV’s technology stack, leveraging input from the vehicle’s sensors to generate the best trajectory for the AV, amidst uncertainty and often contradictory rules.

An AV planner has to consistently choose the right driving behaviors to ensure safety, legality, comfort, and route efficiency. Trying to summarize the complexity of the real world with hand-crafted rules may be limiting the speed and scalability of AV development. Instead, we believe a fully machine-learned system that’s trained using human driving data will be a catalyst for the next wave of advancements in autonomous driving.

This ML-first approach is called Autonomy 2.0. We believe that it can overcome the challenges inherent to the traditional rule-based Autonomy 1.0 approach that requires engineers to hand-code rules for every conceivable event. You can dig deeper into Autonomy 2.0 in our white paper, but simply put, it’s based on:

• An end-to-end, fully differentiable AV stack trainable from human demonstrations
• Closed-loop, data-driven reactive simulation
• Large-scale, low-cost data collection to enable scalability

Following this approach, we began applying machine learning algorithms to planning, allowing us to learn from complex driving scenarios. We’ve confirmed that continuously adding more datasets and iteratively improving algorithms and frameworks improves the performance of our AV planner.

Our fully machine-learned planner in action

Our current ML Planner takes in a representation of the environment by using our perception stack and map, and then generates a trajectory — that is, a sequence of positions, velocities, accelerations, orientations, and steering commands — using a single neural network. A lightweight rule-based secondary layer enables us to guarantee certain behaviors (e.g., always stopping a minimum distance away from a red traffic light should the ML planner choose to stop closer), allowing us to accelerate development while we continuously improve the ML Planner.

This planner has a drastically simplified architecture and doesn’t require encoding any prior knowledge into it like the old rule-based approach demands. Instead, it learns from a large number of real-world examples.

This ML Planner is now driving on public roads in San Francisco and handling a variety of complex driving scenarios.

In this video, our AV successfully:

• Stops smoothly at a red light and behind a lead vehicle
• Evaluates whether to drive through or stop at a yellow light
• Accelerates and decelerates appropriately for vehicles cutting in and out of its lane
• Nudges laterally around obstacles in its lane

Instead of investing effort to hand-code the desired behavior, our planner handles all of these scenarios well simply by being trained on a large amount of data.

We’re sure. More data leads to better performance.

By powering our machine learning approach with large, curated datasets, we’re learning from expert driving data at scale to rapidly improve performance and exhibit more nuanced behaviors.

We observed this phenomenon early on when the ML Planner struggled to perform good turning behavior. Our hypothesis was that turns were underrepresented in the training data set, so we added an additional dataset targeted at turning. Sure enough, we found that the model’s performance improved significantly after adding the additional training data. As we tested the planner on new routes, we also saw the ML Planner generalize well to new environments.

This example gave us a playbook to address performance issues by using more data rather than through time-consuming hand engineering. We’ve used this approach to tackle many scenarios on San Francisco’s streets, and as we continue to gather robust datasets from Lyft’s fleet and system data and eventually Toyota, we’ll be able to continuously improve our ML Planner’s performance.

What are the components of a road-ready ML Planner?

The best practices on how to build an ML-based planner are not yet established, so we’re in uncharted territory as we experiment, learn, and quickly iterate.

The figure above shows the typical workflow for our ML Planning engineers. It consists of an inner loop that’s fully contained within our ML training framework and an outer loop that allows us to develop and test ML models that are integrated into the AV environment and with different levels of simulation fidelity.

In order to put an ML-based planner on the road, we found it critical to develop easy-to-use tools and comprehensive metrics to enable fast development iteration. There’s a finite number of both AV engineers and hours in a day to test a planner version against every possible scenario — not to mention the cost required to continuously test and iterate. We need to be able to run a large number of simultaneous tests with different inputs and configurations at the push of a button, and then easily inspect and compare the results and dive deeper into interesting scenarios. Ultimately this method of evaluation allows engineers to review performance in aggregate using a statistical lens rather than custom, rigid test cases.

To achieve this, we built the following core components of Level 5’s ML Planner:

Modeling and training framework

The ML training framework we built is flexible enough to let us build models for both perception and planning. It gives us the velocity and scale needed for our team to train complex models in hours — not days — and enables us to evaluate metrics and deploy deep learned models into the AV planner.

Simulation framework

Our engineering team runs thousands of large-scale simulations every day at the lowest possible cost in order to continuously test the performance of the software that powers our self-driving systems. Within our simulation framework, we can test how the AV planner will perform based on side-by-side comparisons of different versions or configurations that might have different weights, parameters, or model architectures.
We can also switch between different levels of fidelity in order to evaluate the ML model in isolation, its integration with the secondary layer, or the integration of the ML Planner and the motion controller.

Evaluation framework

To evaluate a version of the planner in simulation, we need to predict how well the planner would perform on the road. This requires a high-quality signal that enables us to reliably predict safety-critical and comfort-related events in simulation (some of which may only happen once in several thousands of miles or more) so developers can focus on improving failure cases. However, we also need to minimize false alerts from simulation so that our engineering team can spend their effort focusing on the real issues.

Building the evaluation framework can be as difficult and complex as building AV driving systems themselves. We took a three-fold approach:

1. We built granular metrics for targeted datasets that allow us to compute specialized metrics for specialized scenarios, e.g., stopping at a red traffic light.
2. We built general metrics that can be applied to large-scale datasets that allow us to measure the impact of model changes across the full breadth of driving scenarios.
3. We built a data-driven evaluation system that scales with the performance of the planner and learns from the safety driver’s disengagement and notes. Using safety driver feedback, the system can learn when the AV does well, but also when it doesn’t perform optimally.

Result visualization and inspection framework

With the ability to simulate and evaluate the performance of the stack at scale, our visualization and introspection framework allows us to quickly visualize simulation experiments, analyze the results in aggregate, and zoom into individual metrics and scenarios.

Our engineering team is then able to review, correlate, and cross-reference failure cases with the specific models and data used for each iteration. This enables our engineers to identify failure case instances before returning to the modeling process.

Dataset curation & balancing framework

A large volume of data alone is not enough to continuously iterate and improve the AV planner. We also have to supply it with the right data to handle an ever-growing set of scenarios and edge cases (like something falling out of a truck’s flatbed in front of the AV). We test our planner against a very large data set, and leverage our evaluation framework to identify scenarios in which the planner does not perform well. Then, we add these scenarios to the training data set for the planner to improve in such cases.
Much like we’re able to study simulated edge cases with this approach, we’ll eventually be able to apply learnings from specific long-tail events that occur daily via our partnership with Lyft’s rideshare fleet to inform the development of our AV planner — a core differentiator for Level 5’s AV approach.

Automation framework

We heavily leverage workflow automation to increase development speed. This automation allows us to go through one full loop of the development cycle with few commands or a single click in a GUI. We also automatically train and evaluate our model stand-alone and integrated with the stack on a regular cadence to monitor our progress, track key metrics, and detect unwanted regressions.

Together, all of these components enable us to rapidly iterate on the machine learning-first planner, with a fast turnaround time from modeling to evaluation, followed by introspection.

Driving ahead

Deploying this fully machine-learned motion planner on our test fleet in an environment as challenging as downtown San Francisco marks a significant proof point for our Autonomy 2.0 approach. This deployment also sets us up to leverage data at scale, and take advantage of the progress made in the machine learning field to accelerate our development.

At Level 5, we’re uniquely positioned to train our machine learning models on real-world data from Lyft’s fleet and system data and Toyota, giving us access to vast amounts of data covering different geographies and use cases. We’re looking for talented machine learning engineers to help us on this journey. If you’d like to join us, we’re hiring.

Deploying a Machine-Learned Planner for Autonomous Vehicles in San Francisco was originally published in Woven Planet Level 5 on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Luc Vincent, President, Level 5

We’re thrilled to announce we’ve officially joined the Woven Planet family. Woven Planet is a software-first subsidiary of Toyota Motor Corporation, and their vision is to create mobility of people, goods, and information that everyone can enjoy and trust. We’re aligned in vision and values, prioritizing safety and improving mobility for all.

We recognize there is significant work to be done to get commercial autonomous vehicles (AVs) to scale, and AVs are taking a longer path than the industry initially predicted. To create safe, autonomous mobility and deploy it on a global scale, an AV company needs a combination of attributes rarely seen in the industry. Having joined Woven Planet, Level 5 now has that rare combination.

A rare set of attributes

Level 5 was already a fast-moving startup with a talented team. Woven Planet expands the R&D resources, reach, and applications for Level 5’s technology. Level 5 now has a unique set of assets to deliver on our mission:

• Startup agility as an independent group within Woven Planet tasked with rapid software innovation
• Financial backing from Toyota Motor Corporation, one of the world’s largest and most profitable automakers
• Access to data from Lyft’s rideshare business to power our data-driven approach to autonomy
• Talent and automotive expertise to continue driving toward L4 AVs with the know-how to scale
• A path to scaled deployment through our ongoing relationship with Lyft as well as Toyota’s global business
• Access to best-in-class vehicles, hardware, and automotive-grade software through Toyota Motor Corporation, Toyota Research Institute, and Woven Planet

We’re still Level 5

Not everything has changed around here. Our full Palo Alto and London teams joined Woven Planet, so we maintain the same great leadership and relationships. And we’re continuing our work in the same offices, labs, garage, and our Silicon Valley test track.

The goal remains to build L4 AVs to be deployed at scale, and we’ve kept a valuable component of our relationship with Lyft to help us achieve that. We’ll continue to use the Lyft system and fleet data to advance our data-driven approach to autonomy.

We’ve also expanded our opportunity for impact. Woven Planet plays a key role in supplying Toyota Motor Corporation with advanced technology across multiple levels of autonomy. While Level 5 is now Woven Planet’s core group for autonomous driving, we also have an exciting opportunity to help save lives and generate revenue by incorporating cutting-edge elements of our L4 technology into L2/3 products to be embedded into Toyota and others’ vehicles.

With this expansion of opportunities comes the need for an expanded team. We’re doubling down on our growth plans and have compelling opportunities for engineers on nearly every team at Level 5.

A strong foundation and path ahead

As part of Lyft, we built a mission-driven, values-oriented team focused on developing safe autonomous driving accessible by everyone. We’re grateful to Lyft for the great culture we take with us to Woven Planet, and we’re energized by the similarities we see in Woven Planet’s culture.

I believe Level 5 is in a stronger position for long-term success than ever. I like to say that with Woven Planet, we’ve become a dream team. We have the financial backing, robust resources, and technical and automotive expertise to bring to life our shared vision of safe mobility for all.

Visit our careers page to see open roles at Level 5, and follow this blog and @WPLevel5 on Twitter for updates.

Level 5’s Exciting Path Ahead at Woven Planet was originally published in Woven Planet Level 5 on Medium, where people are continuing the conversation by highlighting and responding to this story.

Nearly a year ago, Level 5’s autonomous vehicles (AVs) drove our first autonomous miles in San Francisco. We were excited to discover that the stack we built for Palo Alto generalizes well to new environments, tackling dense city traffic, lane changes, U-turns, and merges without any specific tuning. Since then, we have begun employee test rides, expanded San Francisco routes to cover more than 100 miles, and advanced our latest generation vehicle platform from testing to driving daily on these new routes.

In the clip below (sped up 3x), a Level 5 AV drives a seven-mile stretch of San Francisco in full autonomy mode. Our perception system “sees” 150 meters ahead and plans its trajectory, avoiding jaywalkers, conducting unprotected turns, changing lanes, and more.

Despite driving substantially more complex environments in San Francisco, we were able to significantly improve our key performance metrics and our overall number of disengagements year-over-year.

Over the last year, we demonstrated the advanced capabilities of our autonomy stack and ability to scale our operations and mapping efforts. As our program grows, we look forward to further improving our autonomy performance by fueling our data-driven approach to autonomy with more data.

If this challenge sounds interesting to you, we’re hiring! See our open roles.

Navigating San Francisco in Full Autonomy was originally published in Woven Planet Level 5 on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Robert Morgan, Engineering Director and Mason Lee, Technical Program Manager

To build an autonomous vehicle (AV), we need to be able to safely and efficiently modify and test its software and hardware stacks. While on-road testing may seem like an effective way to do this, it’s simply unrealistic. It’s estimated that it would take more than 10 billion miles to collect enough data to fully validate a self-driving vehicle — that’s 400,000 trips around the Earth.

Let’s put this into perspective by considering the level of engineering required for a self-driving vehicle to complete a frequent and intrinsic action for human drivers: lane changing.

Human drivers instinctively consider the variables that impact their decision to change lanes: their own speed, the speed and distance between other vehicles on the road, congestion, whether a lane change even makes sense given an upcoming turn in the route, and others. For an autonomous vehicle to make the same decision, models for each of these variables must be developed and tested over and over (and over) again, resulting in an astronomical amount of time and resources required if solely tested on-road.

Whether making lane changes or executing more complex scenarios, it’s essential to have a scalable, virtual option to effectively validate AVs as we accelerate toward a self-driving future.

The digital co-pilot to AV road testing

At Level 5, we supplement on-road testing with virtual validation to continuously test our self-driving vehicles’ software and hardware systems. This method involves running a high volume of virtual missions in a representative simulation while measuring performance along the way. We then evaluate the results to make necessary modifications to our software and hardware stacks, and feed the learnings back into the virtual validation system to improve autonomy performance, simulation, evaluation, and test coverage over time.

In addition to eliminating the need to complete miles of on-road missions to gather data for endless scenarios, virtual validation offers a number of other benefits:

• Safety. It allows us to test our system in a safe, virtual setting without taking on the risks that come with physical testing. We can also virtually test against rare edge cases — like a car slamming on its brakes in front of the AV — allowing us to proactively improve the performance and safety of our stack.
• Focus. In simulation, we are in control of the scenarios that the AV encounters. This means we can skip uneventful miles and focus on challenging scenarios that offer us more critical learning opportunities to improve our systems. We can also easily alter and test specific variables within a scene, like the types of traffic agents around us or the speed of oncoming vehicles. This allows us to extract more value from our testing, as opposed to spending time collecting uneventful data.
• Reproducibility. We have the resources to safely reproduce scenarios with vehicles on-road; however, they’re often time-consuming and would require a lot of manual setup. With virtual validation, reproducibility is built-in and we can easily study scenario variations in parallel. Disengagements that occur in simulation can be easily played back, altered, replayed, and shared.

Level 5’s approach to virtual validation

Virtual validation is not just a “final exam” we use to evaluate the performance of our software and hardware stacks for real-world deployment. It’s an approach we leverage and fuse with on-road vehicle testing throughout the AV development process.

Virtual validation is a rigorous, continuous process at Level 5. Every week, we collect our individual teams’ latest releases into a unified software update that we refer to as the “candidate.” Before we deploy the candidate on the AV fleet, it must run through a suite of virtual tests:

• Planning and Integration Regression Tests. This collection of tests includes feature-specific scenarios as well as replications of previous events and issues. This ensures that what has been working continues to work.
• Virtual Intervention Tests. At Level 5, we measure the number of human interventions per 1,000 miles to evaluate the high-level performance of our autonomy stack both on-road and in simulation. The Virtual Intervention Test consolidates scenarios seen in previous releases and replays the logs using the updated autonomy stack. The key goal in this stage is to detect potential issues in simulation using a high volume of real-world data.
• Full System Tests. At this stage, we evaluate all the software and hardware components of an AV, without the physical car itself. In our hardware-in-the-loop (HIL) testbed environment, we’re able to run full system tests for systemic functions separate from the actual behavior of the AV, including execution performance and timing, memory usage, and latency.

Once the stages of virtual testing are complete, the surfaced issues are sent to our Triage team for evaluation. They review the virtual disengagements, failing tests and diagnostics, and monitor the various automated metrics (like road law violations) for each candidate. Once the issues from virtual testing are addressed, we deploy the candidate onto our AVs for physical world testing. This allows us to not only validate the results from virtual testing, but also to feed these new real-world scenarios back into the virtual testing pipeline for future candidates.

Leveraging real-world data to focus on the right scenarios

Virtual validation enables us to simulate specific scenarios at scale and chip away at the target of 10 billion miles driven. But, how do we identify and address the challenging scenarios that require additional testing and virtual validation? The answer to this question lies with real-world datasets.

At Level 5, we have the opportunity to tap into data from Lyft’s nationwide rideshare network to study real-world driving scenarios that can inform the development of our own self-driving technology. This data can help us focus on what and how we test, like interesting event discoveries such as an object falling out of a truck bed; or naturalistic maneuvers like whether the AV changes lanes in a way that’s comfortable for passengers.

Virtual validation is essential to the development of self-driving technology. It’s safer, faster, and produces more complete answers about the performance and safety of AVs in various scenarios. By combining virtual validation with on-road testing and real-world data, Level 5 can accelerate autonomous driving technology for years to come.

Follow this blog for updates and follow @LyftLevel5 on Twitter to continue to learn about Lyft’s road to autonomy. Visit our careers page to join Lyft in the self-driving movement.

Virtual Validation: A Scalable Solution to Test & Navigate the Autonomous Road Ahead was originally published in Woven Planet Level 5 on Medium, where people are continuing the conversation by highlighting and responding to this story.

An introduction and tutorial for training machine learning motion prediction models using Lyft Level 5’s Prediction Dataset

By Luca Bergamini, Software Engineer; Vladimir Iglovikov, Software Engineer; Filip Hlasek, Engineering Manager; and Peter Ondruska, Head of Level 5 Research

Predicting the behavior of traffic agents around an autonomous vehicle (AV) is one of the key unsolved challenges in reaching full self-driving autonomy. With our Prediction Dataset and L5Kit, you can start building motion prediction models in a free afternoon or weekend — even if you have no prior AV experience. If successful, you can help propel the self-driving industry forward as a whole, and even win part of our $30,000 prize pool if your solution lands among the best in our Kaggle competition.

What is motion prediction and why is it important?

Let’s start with how the self-driving car works. Within the AV stack, the first step toward building a self-driving system is perception (identifying what’s around us). The next two tasks are prediction (determining what will happen next) and planning (deciding what the AV is going to do in the future). We’re focusing on this second task.

AVs need to be able to make predictions about the future — something drivers do subconsciously all the time. Imagine an AV trying to turn left while another car is approaching from the opposite direction. In order for the AV to perform this maneuver safely, it needs to determine whether the other car will turn right or continue driving straight and interfere with the left turn. This is exactly what motion prediction is about.

While perception is considered production-ready within the industry, prediction and planning still need improvement. Why are these later parts of the AV stack still unsolved for AVs?

Today, models for motion prediction and planning are mainly built using rule-based systems. However, the future is uncertain and rules may not always scale well with uncertainty. As you add other agents into the mix, the number of rules and costs grow exponentially. A deep supervised learning approach could address that, but you need a lot of data to properly capture rare and unexpected behaviors of the road. Good news: our Prediction Dataset is the largest of its kind and includes these rare behaviors.

Training your model

To train your model, you need to:

1. Download Lyft Level 5 Prediction Dataset
2. Get input and output for the Task
3. Define the Model
4. Train the Model

We prepared a Jupyter notebook to make these steps simple.

1. Download Level 5 Prediction Dataset

Visit our Prediction Dataset webpage to register and download the dataset. This dataset includes more than 1,000 hours of driving data over 16,000 miles collected by our fleet of AVs in Palo Alto, California. We’ve already run the dataset through our internal production perception stack since building prediction models is hard to do efficiently when starting with raw data. This means you can immediately get started building motion prediction models for traffic agents already identified in our perception outputs, and measure your progress against our ground truth.

In addition to the data, we’ve also included an aerial map and high-definition semantic map annotated by Level 5 researchers. You can read about it more here.

2. Getting input and output for the task

The Prediction Dataset registers the world around the AV at different timestamps. Each timestamp includes:

1. A frame
   A frame is a record of the AV itself. It contains its location and rotation, as well as a list of all the agents and traffic lights detected around it in that instant.
2. Agents
   An agent is a movable entity in the world. Agents are labeled with a class (car, pedestrian, etc.) and position information. Agents also have unique IDs, which are tracked between consecutive frames.

Getting information for frames and agents is as simple as running this snippet of code:

A common choice when working with AV data is to use Bird’s-Eye View (BEV) rasterization for the system’s input, which consists of top-down views of a scene. This simplifies building your models because the coordinate spaces of the input and output are the same.

Getting BEVs and output trajectories ready for training a DCNN is a matter or few lines:

Input BEVs overlaid on our HD semantic maps looks like this:

3. Define the model

Define a model to train using your inputs and outputs. You can create a simple baseline by adapting a standard CNN architecture (e.g. ResNet50) to your needs. While you can leave the central part of the network as-is, you’ll need to change its input and output layers to match your setting.

To do this, match the number of channels in the first convolutional layer to the one in the BEV. A 3-channel convolutional layer is not enough to rasterize different semantic information in different layers. Next, make sure the number of outputs matches your future prediction horizon multiplied by each timestep element (XY displacements are used in the example below). For a horizon of 50 steps, you’ll need a total of 100 neurons in the last layer of our network.

In PyTorch this looks like:

4. Train the model

You’re now ready to train the baseline model on the data. The model will be fed with BEV inputs and asked to predict future trajectories. You can set up a Mean Squared Error (MSE) criterion and optimize the network using the ADAM optimizer. In PyTorch this looks like:

We can also plot the predicted trajectories on BEV rasters centered around the AV after a few iterations.

We hosted a live demo (skip to 20:19) if you want a deeper dive on our codebase and dataset.

Take your model a step further

Here are some ideas and tips to improve and experiment with your model:

• Simply replace the first and last layer if you’re striving for speed and want to replace ResNet with the lighter EfficientNet.
• Want to see if an agents’ history can improve performance? That’s just a matter of changing a configuration value and everything works out-of-the-box.
• You can increase the raster’s resolution and even change its aspect ratio if it looks too coarse.
• Is one trajectory per agent not enough to capture the uncertainty? We already provide metrics to score multi-modal predictions, so you can know straight away if your model improves.

Enter the Kaggle Competition

Now that you have your model ready, are you up for a challenge? Our Kaggle competition is live and participants are already breaking records on the leaderboard. Submissions are increasing rapidly and there are still 2 months to compete for the $30,000 prizes.

Here are some tips if you’ve never submitted to Kaggle:

• Add your phone number to unlock GPU training if you’re not already registered on Kaggle.
• Our L5Kit is already included in the Kaggle environment, so you can still use your local workflow.
• If you’re curious about top scorer solutions, chances are they have already shared them in the notebooks section. It’s a good place to start if you’re looking for some inspiration.
• We have a specific TPU workflow included if you happen to be a TPU star!

We’re excited to be able to empower the research community to innovate, illuminate important unsolved challenges in self-driving, and propel the self-driving industry forward as a whole with this dataset and competition. We wish you the best of luck in the competition and can’t wait to see what you come up with as you build your models!

Follow @LyftLevel5 on Twitter and this blog for more information about self-driving at Lyft, and visit our careers page to see open roles at Level 5.

How to Build a Motion Prediction Model for Autonomous Vehicles was originally published in Woven Planet Level 5 on Medium, where people are continuing the conversation by highlighting and responding to this story.

Over the last few years, machine learning has become a core part of self-driving development. It has unlocked significant progress in perception, but it remains limited in its use across the rest of the autonomy stack, particularly within planning. Successful behavior planning is currently blocking the next wave of development for autonomous vehicle autonomy, and we believe that applying machine learning is the key to resolving this. Such machine learning approaches need large behavioral datasets, but where will these datasets come from?

Employing machine learning across the self-driving stack

First, let’s examine where machine learning is or isn’t being used successfully to address the fundamental questions for all modern self-driving systems:

Perception: What’s around the car? What’s the position of other cars, pedestrians, and cyclists? AV developers have successfully taught self-driving vehicles to see by applying the same machine learning approach that ImageNet pioneered to train convolutional deep learning neural networks for image classification. By leveraging a large dataset of human-labelled visual examples, AVs can see cars and pedestrians via a stream of camera and lidar data. For this application of machine learning, developers just need a large-enough dataset of human-labelled data.

Motion Prediction: What will likely happen next? Teaching an AV to anticipate how a situation may develop is much more nuanced than teaching it to see. For a time, this was solved using simple approaches, like extrapolating future positions based on current velocity. Of late, it’s proving more effective to use historical data in machine learning systems to predict the future. To go deeper on prediction, see our Motion Prediction competition.

Trajectory Planning: What should an AV do? Trajectory planning is how an AV decides what to do at any given moment. It’s the most difficult remaining engineering challenge in self-driving, and one where, so far, very little machine learning is used across the industry. Instead, AVs make decisions based on hard-coded rules for what to do in every scenario, with predetermined tradeoffs. These tradeoffs could be between factors like the smoothness of the ride and negotiations with other cars, pedestrians, or cyclists. This works for simple scenarios but fails in rare ones as these rules are becoming harder and harder to define.

Today, the industry is limited by existing rules-based systems as it relates to trajectory planning and motion prediction — which represent the missing pieces of the self-driving brain. We’ve reached a turning point in how AV programs address these challenges.

At Lyft, we believe real-world driving behavior examples play a major role in finding the answer. Just as with human drivers, only with true “experience” will vehicles be able to both see and drive with fully-realized autonomy.

Exploring a novel machine learning approach to planning

To make self-driving a reality at scale, we need a radically new approach — one that can learn from human driving demonstrations and experiences instead of complex hand-crafted rules.

Take for example DeepMind’s AlphaGo, a program that beat the world champion in Go. Instead of relying on an approach that follows dictated rules, AlphaGo replaced rules with a deep neural network and algorithms to create learning systems based on experiences and self-play. This is known as imitation learning or inverse reinforcement learning, and a similar approach can be used to learn from human experiences. In essence, this is the difference between learning the rules of the road via a DMV handbook and experiencing driving first-hand to learn from real-world experiences.

This approach is novel to self-driving technology and it’s what we’re developing at Level 5. It involves two core components:

1. New algorithms that apply next-generation machine learning to address the problems of planning and prediction.
2. An immense amount of human driving examples to feed them with a vast dataset of real experiences they can learn and emulate.

Building new datasets to train the vehicles

Once these algorithms exist, the remaining challenge is how to acquire data to feed them. This requires a completely new class of datasets that detail real-world driving scenarios at scale. We believe these datasets can be acquired through a rideshare network that captures what’s happening on the road every day, ranging from basic driving behavior to rare and challenging scenarios. Through Lyft, we’re able to tap into their nationwide rideshare network to study real-world driving examples to inform the development of our own self-driving technology and machine learning systems.

We’ve learned through our own experience that this paradigm works, and that the more behavioral examples applied to our planning system, the better our self-driving system performs.

With access to data from one of the largest rideshare networks in the world, we’re building the behavioral-based dataset needed to feed machine learning for our planning system. We’ve also shared the largest prediction dataset in the industry, and we invite the research community to join us in solving this challenge.

If you’re interested in learning more about how Lyft is navigating the autonomous road ahead, continue to follow this blog for updates and follow @LyftLevel5 on Twitter. To join Lyft in our self-driving development, visit our careers page.

The Next Frontier in Self-Driving: Using Machine Learning to Solve Motion Planning was originally published in Woven Planet Level 5 on Medium, where people are continuing the conversation by highlighting and responding to this story.

By: Robert Morgan, Director of Engineering and Sameer Qureshi, Director of Product Management

We’re excited to announce that our autonomous vehicles (AVs) are back on the road — and that during the shelter in place we continued to make progress by doubling down on simulation. Simulation is an important part of our testing program, enabling us to test beyond road miles.

Road Miles are not Enough

Testing AVs in the real world is necessary, but can also be limiting. Training inputs like weather and pedestrian behavior are limited to what’s happening in the world at each moment, and it can be unpredictable when you encounter a rare obstacle a second time. If reliant upon on-road miles, it may take some number of billions of miles to test everything. Simply put, the scale makes it impractical to rely only on road miles.

Therefore, we supplement our on-road testing with simulation which gives us a cost effective way to create additional control, repeatability, and safety. It also allows us to test our work without vehicles, without leaving our desks, and for the last few months, without leaving our homes.

Simulation Takes us Further

Simulation is an expansive domain: the focus of the simulation must pair with what you’re trying to develop. At Level 5, we’re developing an AV and have a purpose-built simulation catered to supporting our autonomous technology. This includes two dominant modes:

• Log replay: Replays the data from previous road testing to see how new software behaves using real data from a real vehicle. We leverage “closed loop” replay (within its limits) to test the effects of the vehicle commands that are sent by the software in simulation.
• Synthetic simulation: Creates a virtual world for the autonomous vehicle that includes the static world as described by maps and dynamic elements such as pedestrians, vehicles, and traffic lights. Synthetic simulation allows us to create an exact situation and explore a specific scenario we may not have encountered on the road.

With these tools, we can provide actionable feedback to the development team, ultimately resulting in greater confidence each time we push a release to the vehicle.

Using Rideshare Data to Power the Machine

The real amplifier is the data used to feed the simulation. Without the right data, teams might focus on solving problems in the wrong order, or the wrong problems entirely. At Level 5, we are able to use data from Lyft to develop accurate and useful synthetic simulation, enabling us to focus on solving the right problems in the right order.

• Learning from real rides: We know the routes, road complexity, and more about daily trips, focusing our feature development and capabilities on what’s most useful for people. This will allow us to get closer to developing a consumer product, sooner.
• Cameras: We placed cameras on a subset of rideshare vehicles in order to capture real-world data about the scenarios drivers and AVs face daily and how often they occur. For example, to better our planning, we can learn how often other drivers “cut in” for a last minute lane change at a particular intersection. The rideshare camera data also enables us to discover rare long tail events at scale to improve our system.

Improving Simulation with Feedback

We can combine all of these efforts to create a virtuous feedback machine that validates simulation performance or identifies opportunities for improvement.

The feedback pinpoints:

• Coverage: Our insights from Lyft data inform us of additional test space and the frequency of occurrence. This feedback helps us discover any unknown unknowns.
• Confidence: Comparing results between simulation and on-road AV performance allows us to understand the correlation and the predictive power of our simulation. This drives improvements in the fidelity of our simulation technology.
• Capability: The measurements of our AVs’ behavior in both simulation and the real world drive refinement and improvement of our AV software.

Maintaining Level 5’s Momentum

While road testing remains a critical aspect of our program, simulation allows us to leverage existing on-road data in many more ways, and multiple times over, to help improve and validate our software. With Lyft’s unique data and Level 5’s advancements in simulation, we believe we’re reducing the road miles needed by several orders of magnitude. Our focus on simulation over the last few months allowed us to maintain Level 5’s momentum toward our goal to improve access to safe and reliable transportation for millions of Lyft riders everywhere.

For more information about Lyft’s self-driving approach, follow @LyftLevel5 on Twitter and this blog. To see open roles at Level 5, visit our careers page.

Continued Momentum Through Simulation was originally published in Woven Planet Level 5 on Medium, where people are continuing the conversation by highlighting and responding to this story.