Invented by Chenggang Liu, John Evan Bares, Uatc LLC
The Uatc LLC invention works as follows
The present disclosure describes systems and methods of automatic event detection and classification for autonomous vehicles. In one example, a computing device can be used to obtain vehicle data that describes the vehicle conditions of an autonomous vehicle. The method involves extracting a number of features from vehicle data by one or more computing device. The method involves determining a classification by one or multiple computing devices, using a machine-learned algorithm, for each of the candidate events, based, at least in part, on the features associated with those candidate events. The method involves associating the vehicle data with the classification determined by the computing devices for each of the candidate events.
Background for Machine Learning for Event Detection and Classification in Autonomous Vehicles
An autonomous car is a vehicle capable of sensing and navigating its environment with little or no human intervention. An autonomous vehicle, in particular, can use a variety sensors to observe the surrounding environment and then attempt to understand the environment through various processing techniques. The autonomous vehicle, with its knowledge of the surrounding environment can determine an appropriate path to travel through that environment.
Aspects, and benefits of embodiments of this disclosure will be described in part.
One aspect of the disclosure is a computer-implemented technique to detect unpleasant driving events by autonomous vehicles. One or more computing devices are used to obtain vehicle data that describes the vehicle conditions of an autonomous vehicle in an autonomous driving session. The method involves extracting, using one or more computing systems, a number of features from vehicle data. The method involves determining, using machine-learned classification, a classifier for each of the candidate events, based on at least part of the features associated with those candidate events. The method involves associating the classification determined by the computing devices for each of the candidate events to the vehicle data.
Another aspect of the disclosure is a computer system.” The computer system contains one or more processors. The computer system contains one or more tangible non-transitory computer-readable media which collectively store a vehicle data log collected during an autonomous vehicle driving session. The vehicle data is descriptive of the vehicle conditions that were associated with an autonomous car during a previous autonomous driving session. The vehicle data is annotated at various times with a plurality event labels. The computer-readable media store collectively instructions that, when executed, cause the computer to: extract one or multiple features from the data log of the vehicle for each of a plurality times; associate each event with the extracted one or several features for the time corresponding to the event label; and train a classification model to perform events classification based, at least in part, on the plurality event labels and one or two features associated therewith.
Another aspect of the disclosure is directed at a computer system.” The computer system contains: one processor, a machine-learned classification model, and one tangible, nontransitory computer-readable medium that stores instructions that when executed by one or multiple processors cause them to perform operations. The operations include retrieving vehicle data that describes vehicle conditions for an autonomous vehicle in a driving session. Extraction of a number of features from vehicle data is part the operations. The operations include identifying multiple candidate events. For each of the multiple candidate events, input the features into the machine-learned classification model. As an output from the machine-learned classification model, the operations include receiving a classification of each of the multiple candidate events. The operations include associating each of the candidate events’ classifications with vehicle data.
Another aspect of the present disclosure is directed to various systems and apparatuses, nontransitory computer-readable mediums, user interfaces, electronic devices, and other systems.
These and other features, aspects and benefits of different embodiments of this disclosure will be better understood if you refer to the following description. These accompanying drawings are included in and form a part this specification and illustrate examples of the present disclosure. They also serve to explain related principles.
The present disclosure focuses on systems and methods that automatically detect and classify events in autonomous vehicles. The systems and methods disclosed herein can include one or more machine-learned classifier models which can be used to perform event classification based on various features extracted from vehicle information, such as vehicle sensor data. The features can be extracted, for example, by using a continuous-wavelet transform on vehicle data or another feature extraction technique. A machine-learned classifier model can be trained using existing vehicle data logs containing event labels that were manually entered by a passenger. The classifier model, for example, can be trained by using an objective function which describes the difference between the predictions made by a classifier on the existing vehicle log and the event label provided by a human passenger and associated to the vehicle datalog.
The continuous testing of autonomous vehicles is a key aspect of developing, operating and refining the technology of autonomous vehicle. One metric that can be used to measure the quality of the motion control of an autonomous vehicle is the number and types of special events which occur during the autonomous driving session. Events can include unwanted driving events, such as high acceleration or high deceleration. They could also be juke movements, jerks, weavings motions or other undesirable events.
The number and type events that occur in an autonomous driving session by an autonomous car can give an indication of how well the vehicle controls its motion. Data that describes these events can be used to test, score, or refine the operation of an autonomous vehicle.
Detection and classification of events is also useful for detecting and marking instances of poor performance of various components or system of the autonomous vehicle. (e.g. systems that are responsible for determining the motion plan in the environment, or for perceiving it). A detected event may, for example, help a triage system or team to route an instance of autonomous vehicle failure identified by the team.
One way to collect data on the type and number of events that happen during an autonomous driving session is by having a co-pilot ride with the vehicle. They can then record when certain events happen. If, for example, the autonomous vehicle makes an uncomfortably rapid deceleration during a certain time, the passenger could make a note (e.g. an electronic record) indicating that such an event took place. The record created by a human can be linked to the data collected at that time by the autonomous vehicle (e.g. vehicle data).
However the technique above that relies only on human passengers to detect, classify and categorize events has several drawbacks. The use of human passengers may make such a technique expensive and difficult to scale. The use of human passengers can also lead to inconsistent and/or poor detection of events due to the subjective definitions used by different humans. The use of a human to complete and create event records may cause a delay in the time between the event and the creation of the record, resulting in inaccurate timestamps. The above technique, which relies on human passengers solely to detect and categorize events, is not applicable to detecting and identifying events in simulated driving sessions.
Another possible technique to collect the number and types of events that happen during a driving sessions performed by an autonomous car is by applying various predefined metrics on vehicle data collected during driving sessions. To identify high-deceleration events, for example, you can apply a predefined threshold of deceleration or another metric. However, certain events may not have metrics (e.g. juke, weave). In these cases, a new measurement will have to be developed, tested and refined for every type of event. Even for events where predefined metrics are in place, they may not be easily transferred from one vehicle type or data type to another. A metric that identifies high deceleration for a sedan might not be applicable to a sports utility car. Even predefined metrics may require tweaking of thresholds or design parameters in order to make them consistent with human subjective feelings.
The present disclosure offers systems and methods to automatically detect and classify events in autonomous vehicles. The systems and methods disclosed herein can include one or more machine-learned classifier models which can be used to perform event classification based on various features extracted from vehicle information, such as vehicle sensor data. The features can, for example, be extracted using a continuous-wavelet transform or another feature extraction technique. A machine-learned classifier model can be trained using existing vehicle data logs, which include event labels that were recorded by human passengers (e.g. as described above). The classifier model, for example, can be trained by using an objective function which describes the difference between the predictions made by a classifier on the existing vehicle log and the associated event labels.
Once the classifier model is trained, systems and methods in the present disclosure will assist in evaluating an automated vehicle motion control system based solely on sensor feedback or other vehicle data. This eliminates the need for passengers to manually record events. The systems and methods disclosed herein can also provide more accurate and consistent results than human copilots, and can be used with both real-world vehicle data and simulated data. The systems and methods described in the present disclosure are applicable in many scenarios. For example, they can be used to detect and classify events in logs of vehicle data from previous autonomous driving sessions, e.g. for triaging vehicle failures during those sessions, or for providing real-time feedback or ride quality scores during vehicle operation.
More specifically, the systems or methods disclosed in the present disclosure are capable of performing automatic event detection and classifying for an autonomous car based on the vehicle data associated with that vehicle. The vehicle data could, for example, be descriptive of the conditions that a vehicle is experiencing during an autonomous driving session. In some implementations the vehicle data may include data that is collected or received by various sensors in the autonomous vehicle. The vehicle data may include, for example, data that describe a vehicle’s speed (also known as its velocity) or steering angle. Another example is that additional vehicle data descriptive of a vehicle’s lateral acceleration can be generated or obtained. In some implementations, the lateral acceleration of the autonomous vehicle can be determined by the speed and steering angle. In some implementations, data on lateral acceleration can be generated using speed and steering angle information.
As further examples, vehicle data may include various types descriptive of vehicle conditions. These can include, for instance: light detection data, radio detection data, imagery captured by one or multiple cameras onboard an autonomous vehicle, accelerometer data, positioning system data, gyroscopes data, throttle position data, engine torque data, exhaust oxygen data, engine airflow data, engine RPM data, vehicle control data related to a vehicle controller, and/or other vehicle data (such as data collected or produced by sensors aboard the vehicle). The vehicle data can therefore include discrete data types that can be referred as data channels in some cases (e.g. a steering angle channel versus speed data channel). As mentioned above, systems and methods disclosed herein can detect and classify events based on vehicle data.
In some implementations, vehicle data may include logs of completed driving sessions. As an example, a computing system or autonomous vehicle can collect and store data on the vehicle as it executes a session. A log of vehicle data can then be transferred from the autonomous vehicle to a computer system, which will perform event detection and classification using the techniques described in this document. The vehicle data log can include descriptive information about the detected events. The vehicle data log can, for example, include a record that an event was detected at a certain timestamp. The vehicle data log, which includes automatically generated event records, can then be used to analyze or score the performance of an autonomous vehicle during a driving session. Triage teams or systems can, for example, use the vehicle log and newly created event records to identify “failures” in a vehicle. Triage these failures and send them to the appropriate team for investigation or resolution.
In some implementations, vehicle data can also be received in real time and analyzed as the autonomous vehicle is operating. In some implementations the event detection systems and classification systems can be physically located onboard an autonomous vehicle. They can receive vehicle data, detect events and classify them in real-time. A small embedded computing device can implement and include the event detection systems and classification systems described in the present disclosure. “Real-time event classification and detection can be used for real-time feedback on vehicle operation or ride quality.
In some implementations, vehicle data may also include simulated vehicle information. For example, the motion planner system or another system in the autonomous vehicle could generate simulated vehicle information that corresponds with a simulation for a candidate movement plan. The simulated vehicle data can be used to detect and classify simulated events. At least partially, the candidate motion plan’s score can be based on detected simulated events. For example, simulated events that cause a high rate of deceleration may increase the cost associated with a candidate motion plan. This will encourage the motion planners to choose a plan which includes fewer unwanted driving events. Simulated vehicle information can be generated and analyzed as part of offline development activities. In various implementations, vehicle data may be real-time, logged, or simulated data.
