Patent for “User interface to display internal state of autonomous driving systems”

Autonomous Vehicles – Andrew Timothy Szybalski, Luis Ricardo Prada Gomez, Philip Nemec, Christopher Paul Urmson, Sebastian Thrun, Waymo LLC

Search Patent for “User interface to display internal state of autonomous driving systems”

Abstract for “User interface to display internal state of autonomous driving systems”

“Autonomous vehicles use a variety of computing systems to transport passengers. To safely maneuver the vehicle to its destination, a control computer sends messages. A control computer might display information on an electronic screen to enable the passenger to see what the vehicle is doing in the near future. This information may be provided to the passenger using icons or images.

Background for “User interface to display internal state of autonomous driving systems”

“Field of Invention”

“The invention is generally applicable to user interface applications for autonomous-driving systems.” In particular, the invention provides user interfaces that display the status of an autonomous driving system.

“Description of Related Art.”

“Autonomous vehicles use a variety of computing systems to transport passengers between locations. Some autonomous vehicles require input from an operator such as a pilot or driver. Others may need continuous input. Autopilot systems and other systems may only be used when the system is engaged. In this case, the operator can switch between a manual mode and an autonomous mode, where the vehicle drives itself.

The perception system is a key component of autonomous vehicles. It allows the vehicle’s surroundings to be perceived and interpreted during trips. The autonomous system can be engaged to make different decisions throughout the trip. It may speed up, slow down or stop. Operators may not be aware of calculations or the?reasoning. The reasoning behind the autonomous vehicle taking a particular action. Operators may be anxious to know what the vehicle will do in the near future to feel confident and safe.

“Navigation systems can include electronic displays that zoom in and out depending on the vehicle’s speed to allow the user to determine where the vehicle is within a few seconds. These systems may provide real-time information via satellite or radio signals about traffic. These systems don’t display the speed or exact location of any other vehicles or obstacles.

“An aspect is a vehicle. A vehicle is described by a plurality control apparatuses that include a braking device, acceleration apparatus and a steering mechanism. The processor is programmed with the following functions: receive destination information; identify a route from the destination; determine the current geographical location of vehicle using location information from the geographic component and the stored map data; identify an object and type based upon object information received by the object detection apparatus; select images to display based on the action taken and the identified item.

“In one instance, the identified object refers to a second vehicle. The icon that will be displayed is an icon of the second vehicle.”

“In one case, the processor is further programed to display the selected items on the electronic display.”

“In one example, an icon representing the second vehicle was selected based upon the type of vehicle.”

“In one case, the processor is programmed to display on the display an image that indicts a portion the route the vehicle will take in the next few seconds.”

“In one instance, the identified object (or vehicle) is a second vehicle. The action to take is to maintain a safe following distance from the vehicle.”

“In one case, the identified object is a vehicle. The action to take is to avoid a headroom zone in front the second vehicle.”

“In one instance, the action to take is waiting. The method further includes displaying text that indicates that the vehicle is still waiting.”

“In one case, the processor further programs the processor to display a geographical area that is larger for vehicles moving faster than others and smaller areas where vehicles are moving slower.

“In one case, the processor further programs the processor to display a geographical area. A larger area is displayed if the roadway has a high speed limit and a smaller area if the roadway has a low speed limit.

“In one instance, the action is to stop at an intersection. The selected images include an icon that indicates where the vehicle will stop at that intersection.

“In one instance, the action to take is waiting. The method also includes displaying an icon that indicates that the vehicle is in waiting.

“In one case, the processor is programmed to determine which geographic area will be displayed based upon the action taken. If the action taken is a turn then the geographic area includes a wider view in the opposite direction.

“In one instance, the identified object may be a traffic sign. The selected images contain an icon indicating a signal. Further, the processor can program the display to show the icon at the closest location to the traffic signal.

“In one example, a traffic sign is identified and selected images contain an icon indicating that traffic signal. The selected images also include an icon indicating the state of the traffic signal. Further, the processor can be programmed to determine the current state of the signal and determine the appropriate action based on that signal’s state.

“In one example, you would change lanes. The selected images also include an icon that indicates a turn signal.

Another aspect of the invention relates to a method of selecting images to be displayed on a vehicle’s display device. The method involves receiving destination information from a device user inputs; identifying a route to that destination; accessing geographic information including roads, signals and intersections; identifying an object on a roadway and an item type using object information received by an object detection apparatus; determining the appropriate action to take, which is determined based upon the identified object, current geographical location of vehicle and route; then selecting images to display on the apparatus based both the action to taken and the object.

“In one example, the process further includes displaying selected images on a display apparatus.”

“In one instance, the method also includes the determination of a geographic area to display so that a larger area is displayed when the vehicle is moving more quickly and a smaller area where it is moving less slowly.”

“Aspects of the invention, including features and advantages, will be apparent when viewed in conjunction with the following description and accompanying figures. Different drawings might use the same reference numbers to identify similar or identical elements. The following description is not intended to limit the invention; the appended claims, equivalents and limitations define the invention’s scope.

“As shown at FIG. “An autonomous driving system 100 according to one aspect of the invention comprises a vehicle 101 with many components. One or more computers may be used to control the vehicle, such as computer 110, which contains a processor 120 and memory 130, along with other components that are common in general-purpose computers.

The memory 130 contains information that is accessible by processor 120. This includes instructions 132 and data 134. These can be executed by the processor 120. The memory 130 can be any type of storage that is capable of accessing information by the processor. This includes a computer-readable medium or another medium that can store data that can be read using an electronic device such as a hard drive, memory card, ROM RAM or DVD, as well other write-capable or read-only memories. Different combinations of the above may be used to create systems and methods that store different parts of the instructions or data on different media.

“Instruments 132 can be any set or instructions that are to be executed by the processor directly (such a machine code) and indirectly (such a script). The instructions can be stored on a computer-readable medium as computer code, for example. The terms “instructions” and “programs” are used in this context. In this regard, the terms?instructions? und?programs?? may be interchangeably used. These terms may be interchangeable. Instructions may be stored in object-code format for processing by the processor or in any other computer language, including scripts and collections of independent source codes modules that can be interpreted on request or compiled in advance. Below are detailed explanations of functions, methods, and routines for the instructions.

According to the instructions 132, processor 120 may retrieve, store or modify data 134. The system and method are not restricted by any particular data structure. However, the data can be stored in computer registers or in a relational data base as a table with a variety of fields and records, XML files, or flat files. Data can also be stored in any computer-readable format. Image data can also be stored in any computer-readable format, such as bitmaps that contain grids of pixels. These bitmaps may be compressed or uncompressed, lossless, or lossy (e.g. JPEG), bitmap, vector-based (e.g. SVG), or instructions for drawing graphics. Any information necessary to identify the data, including numbers, descriptive text and proprietary codes, references or data stored in different areas (including other locations on the network), or information used by a function that calculates the relevant data, may be included.

“The processor 120 could be any standard processor, such processors from Intel Corporation and Advanced Micro Devices. Alternately, the processor could be a specific device like an ASIC. FIG. FIG. Memory could be, for example, a hard drive or another storage media that is located in a server farm within a data centre. Referring to a processor or computer means that it can refer to any collection of processors, computers, or memories that may or not operate in parallel.

“Computer 110 could include all the components usually used in connection with computers such as memory (e.g. RAM and internal hard drives), storing data 134, and instructions such a web browser. An electronic display 142 includes a monitor having a display, small LCD touch-screens or other electrical devices that are operable to display information) and user input (e.g. a keyboard, touch screen and/or microphone).

“Computer 110 could also include a geographical position component 144, which can be used to determine the device’s geographic location. Computer 110 might include a GPS receiver that determines the device’s altitude, longitude, and/or latitude. You may also use other location systems like laser-based localization, inertial-aided GSM, or camera-based locization.

The “Computer 110” may include additional features such as an accelerometer, GPS, or other acceleration device 146 that determine the direction the device is directed. The acceleration device can determine the pitch, roll, or changes to it relative to gravity, or a plane perpendicular. It is understood that the computer may provide location and orientation data to the user automatically, to other computers on the network, or both.

“Computer 110 could also have an object detection component (148) to detect and identify movement and location (e.g. relative speed) of objects like other vehicles, road obstacles, signs and traffic signals. Lasers, sonar radar, cameras, or other detection methods may be used in the detection system. An imaging device may be used to determine whether a traffic signal is yellow or other colors. Computer 110 can use this information to tell the vehicle’s braking system to apply the brakes.

“Data 134 could include different types of information used to compute 110. Detail map information 136 could include maps that identify lane lines, intersections and speed limits as well as traffic signals, buildings, signs and other information. Computer 110, for example, may access detailed map information 136, in order to locate the lane lines on a highway and adjust vehicle 101’s speed or direction accordingly. Computer 110 can also access display images 138, such as intersections and roadways, to give a passenger of vehicle 101 an idea of the future actions that vehicle 101 will take.

Computer 110 could be an autonomous driving computing device that can communicate with the vehicle’s internal computers such as 160. Computer 160 could be set up in a similar way to computer 110. It may include a processor 170 and memory 172, instructions 175, and data 176. Computer 110 can send and receive information from various vehicle 101 systems, such as the acceleration 182, signaling184 and navigation 186 systems, in order to control speed and movement. Vehicle 101. You will understand that even though vehicle 101 shows various systems and computers 110, 160, it is possible for these elements to be outside of vehicle 101.

“FIG. “FIG. 2” shows an example of an interior design for an autonomous vehicle. One way that vehicle 101 can display information to passengers is in the following ways: Vehicle 101 might have a display 242 to show the autonomous driving systems and an indicator 245 which indicates whether the system is active. The vehicle 101 might also display the current speed of travel 245 or audibly. As shown in the figure vehicle 101 may include a second display 286 to show navigation or entertainment systems.

Vehicle 101 may have one or more input devices for the driver, such as device 244. This allows them to input information into the autonomous driving system 110. For example, a user may input a destination, (e.g. The navigation system may also allow users to input a destination (e.g., 123 Oak Street). The navigation system might generate a route from the current location to the destination. Computer 110 can request or automatically get route information from the navigation systems if the autonomous driving system has been engaged. The autonomous driving system can drive the vehicle to its destination once a route has been established.

“FIG. 3. This is an example screen shot of the display 242 from computer 110. This display can be used to show the driver the current speed, position, orientation, and so on. Vehicle 101 relative to the relevant stretch of roadway305. The icon or box 310 depicts vehicle 101 on the screen. Any object that can be used to identify the vehicle may be used by the user. A line that extends from the vehicle may indicate the route the vehicle is on. FIG. FIG. 3 shows that the vehicle is moving forward. Route line 315 runs from the front of vehicle. Displays may display speed indicator 345 and the approximate route of the vehicle in the next few seconds. This is shown as path 355

“As shown at FIG. “As shown in FIG. 4, the display can also identify other relevant objects. Computer 110 may identify another vehicle on the roadway and display it as an icon. For example, box 410 on the display at the exact location of the roadway might be used. Computer 110 may choose the size and shape of the icon based on actual vehicle size. A compact vehicle might appear smaller than a tractor-trailer.

“As vehicle 101 travels along the roadway, it may detect objects and change the features of the road. These changes can be displayed to let the user know that vehicle 101 is constantly monitoring the condition of vehicles, roads and other objects. FIG. 4. Vehicle box 310 is seen driving along the road towards box 410. Vehicle 101 is seen moving closer to the second vehicle. 5 shows that the distance between box 310 and box410 has decreased. Bar 525 could indicate that both vehicles are approaching an intersection.

Computer 110 might provide additional information as vehicle box 310 nears an intersection. Display may include, for example, a crossroads 605 and additional vehicles (boxes 620 and box 630). These new vehicles can also be displayed in boxes as the computer recognizes them on roadway 305.

“Computer 110 could also slow down vehicle 101 as it approaches an intersection. Therefore, vehicle 101’s current speed may be constantly updated. FIGS. 4, and 5, vehicle 101 moves at 34 miles an hour, as indicated in speed indicator 345. FIG. FIG. 6 shows that vehicle 101 is closer to the intersection. Therefore, computer 110 could slow vehicle 101’s speeds, so speed indicator 345 now shows a speed of 27 mph.

Computer 110 can also zoom and pan the display depending on the type of roadway and speed limit. FIG. 7. Vehicle 101 is shown as box 310 and moving at 65 mph along roadway 705. Computer 110 shows a large stretch of road 705. FIG. Returning to FIG. The speed limit may also affect the zoom and pan options. Another example is that if vehicle 101 moves along a highway with cross traffic it may zoom out more than when vehicle 101 stops at a?four-way? intersection. Stop intersection as the passenger may only require a short distance to see the intersection. Another example is vehicle 101 making a left-hand turn unprotected. The display may zoom out to an extended distance so the passenger can see information from all directions.

“As vehicle 101 approaches other cars, computer 110 may keep a safe distance and display this information for the passenger. FIG. 8. Box 310 is traveling along roadway 805 at 33 mph. Box 310 is moving towards box 810 at 33 mph. Path 855 may be cut to show this information to the passenger. Box 810 could include a tail 865. You may display the tail 865 and path 855 as complementary shapes. For example, two arrow sections that may be combined. FIG. FIG. 9. Box 310 is now closer to box 801, and paths 855 and tail 865 are combined. These two shapes are closing may indicate that box310 is within a safe distance of box 810. Based on vehicle 101’s speed, box 310 may be at a safe following distance behind 101.

Computer 110 can determine that vehicle 101 is not following at a safe distance if the vehicle represented in box 810 slows. Computer 110 could determine that vehicle 101 needs to slow down to correct the distance. This information can be displayed to the passenger. FIG. FIG. 10 shows that the paths 855 and 865 have changed directions. This could indicate that vehicle 101 may not be located at a safe distance from the next vehicle, or box 810. Computer 110 could reduce vehicle 101’s speed and notify the user. FIG. 9 shows vehicle 101 moving at 28 miles per hour. FIG. FIG. 9. Vehicle 101 moves at 28 miles an hour while FIG. 10. vehicle 101 moves at 22 miles per hour. 10 Vehicle 101 is moving at 22 mph per hour. FIG. 9. Computer 110 can display an icon such as double arrows with speed indicator 345. The icon in the example is two arrows pointed upwards. FIG. FIG. 10 shows that computer 101 has decreased the speed of vehicle 101. The icon may be modified, such as the direction the arrows are directed downwards.

“Computer 110 can determine the safe following distance using vehicle speed 101 and display the distance and zoom level accordingly. FIGS. 10, 11 and 12, where the vehicle 110 (box 310) is moving at 22 mph per hour, the safe following range, 855 or 865 is significantly less than the safe distance, 1155 or 1165 where the vehicle is traveling at 65 mph per hour. Comparing FIGS. FIGS. 10 and 11 may show a smaller area of roadway and a longer length if the vehicle moves at a slower speed. Computer 110 might display a larger area and length of roadway if the vehicle moves faster. Another example is that the roadway speed limit may determine the safe following distance.

“Computer 110 might also show information to the passenger about intersections. FIG. 12 Box 310 is approaching an intersection marked with a stop sign. As an example, bar 1225 could indicate an intersection. Computer 110 may display an icon such as stop sign 1285 on roadway 1205 to indicate that 101 is near a stop sign. The section 1275 beyond bar 1225 on path 1255 may indicate that computer 110 will stop vehicle 101 for a time when it reaches the bar 1225 location. Section 1275, for example, may be displayed differently to the rest of path 1255. This could include stationary or moving stripes. Computer 110 can also display information about other vehicles, such as box 1210.

“As shown at FIG. 13: When vehicle 101 reaches bar 1225, vehicle 101 can stop and wait until it is safe for them to cross the intersection. The striped path 1275 indicates this. Box 1210 is seen moving along roadway 1305 near the intersection. FIG. FIG. 14 shows both boxes 310 & 1210. Computer 110 might display an icon such as a ticking timer to indicate that box 1210 has right of way. Computer 110 will then wait for box 1210’s response. Computer 110 can also wait for box 1210 to move. If box 1210 does not move, computer 110 will determine that box 120 is still waiting for vehicle 110. In this case, computer 110 will move vehicle 110 according to the route. FIG. FIG. 15 shows box 1210 moving through the intersection. Now that the intersection is clear, vehicle 101 is moving through it and box 310 is also moving through it. The path 1255 is shown as a solid line, which could indicate that computer 110 has not yet stopped or is waiting.

“In FIG. 16: Box 310 is waiting at an intersection for a right turn. Speed 345 might display information such as “waiting” or the text “waiting”. Computer 110 has paned the display to the correct vehicles. Box 1610 moving on roadway 1605 to the right side of box 315 is more important than objects moving along the roadway 1607 to right of box 311. The path 1675 could also indicate that vehicle 101 is waiting to make a right turn at the intersection. FIG. 17. As box 1610 nears box 310, computer110 may display a tail 1765 that complements the path 1675. As shown in FIG. To indicate vehicle 101 is ready for the turn, path 1875 can be marked as solid. As mentioned above, vehicle 101 may make a turn and path 1875 and tail1765 may merge. Computer 110 may keep a safe distance behind box 1610.

“FIG. 19 shows a more complex left hand turn that crosses several lanes of traffic. Similar to FIGS. Computer 110 may delay turning until the intersection is safe. Computer 110 may not turn until both boxes 1910-1920 have cleared the path 1975. FIG. FIG. 20 shows that box 1910 has passed path 1975. However, box 1920 has not. Therefore, computer 110 could continue to display the “waiting?” message. text, and path 1975 as striped. Computer may decide that it is safe for you to start turning when box 1920 reaches path 1975. FIG. 21. Path 1975 is now shown as path 2155 and box 1920 has a tail 2165. Speed 345 might no longer display the waiting text. This could indicate that vehicle 101 has been moving at 10 miles an hour, for instance. FIG. 22. Display may move to the right to indicate the direction change. As shown in FIGS. 23 and 24 The complementary shapes could close together when vehicle 101 is at a safe distance.

“Computer 110 can also recognize and display traffic signal statuses. FIG. 25. Vehicle 101 is approaching an intersection, and computer 110 intends to turn left. A 2557 inverted arrow icon may be displayed to indicate computer 110 has caused vehicle 101 to slow down due to the intersection ahead. Computer 110 could determine, using the detailed map, that the intersection contains a number traffic signals. FIG. FIG. 25 shows that computer could have identified a traffic sign but not determined its current status. This information may be displayed by computer 110 to the passenger as an icon, such as icon 2585. It is understood that the icon of the question mark and polygon are just examples and may not be used. FIG. 26. Computer 110 has determined the status of the traffic signal and may indicate it at icon 2685. If computer 110 determines that the traffic signal has gone yellow or red, path 2555 could include a section 2675 beyond bar 2525. A striped section 2675 indicates that computer 110 may stop vehicle number 310 at bar 25,255, and wait at the intersection until the situation permits them to turn left.

“Computer 110 might also show a headroom zone in front boxes that indicate other vehicles to notify the passenger that it will not move into this area. FIG. 27. Vehicle 101 is nearing an intersection as shown in FIG. Computer 110 will instruct vehicle 101 to make a left turn by following the path 2755. The striped section 2775 of path 275 indicates that computer 110 will wait until vehicle 101 has reached the intersection before making a left turn. Computer 110 has identified the vehicle in this example. It is displayed as box 2710 and will pass through path 2755. Computer 110 might display a headroom zone 2735 that computer 110 will not ignore in order to safely move through the intersection.

“As shown at FIG. 28. Box 2710 is closer to box 3310. Striped section 2775 has been removed because computer 110 has decided that it is safe to turn left. Computer 110 could increase vehicle 101’s speed. Computer 110 might not display the headroom zone 2735 because box 310 is unlikely to move into box 2710. A new vehicle is displayed in box 2810 and is near path 2755. Computer 110 might also show a headroom area 2835 to inform the passenger that it will not move vehicle 101 within this area. If computer 110 decides it safe to turn, even though the headroom area 2835 of box 2810 is still close, FIG. 29. Computer 110 will direct vehicle 101 towards the left turn.

“Computer 110 could also use the display for passenger information, such as to inform them that vehicle 101 is changing lanes. FIG. FIG. 30 shows vehicle 101 moving along route 3015 behind another vehicle. Box 3010 is the second vehicle. The position of tail 3065 and path 3055 may indicate that vehicle 101 follows a safe distance along route 3015 behind the second car. Computer 110 may decide that a change in lanes is necessary as shown in FIG. 31. The computer may display a turning signal icon 3195 to indicate that the vehicle will move to the lane right of box310. This is for route 3015, which requires the vehicle take an exit or turn. Computer 110 might also display a turning area 3175 similar to a path marked with stripes to indicate that it is still waiting to be allowed to enter the zone. Computer 110 is waiting as the headroom area 3135 of the third vehicle (box 3020) is within the turning circle 3175. Computer 110 might also show a tail 3165 in box 3020 if it has determined that vehicle 101 will be following behind the third vehicle. FIG. FIG. 32 shows that as vehicle 101 slows down and moves towards the tail 3165 the display of headroom area 3135 may fade. This could cause computer 110 to stop displaying the headroom zones 3135 as it is no longer needed. FIG. 33.”

The computer could also show the passenger that the vehicle has turned. If vehicle 101 is behind box 3020, then computer 110 will no longer show the turning area 3175. 34. Route 3015 could be moved to indicate that vehicle 101 will move in the lane right of box 331. FIG. 35. Computer 110 may move vehicle 101 to the right lane, and then on the display, move the ends of the complementary ends of path 3165 of box 2020 towards each other as shown in FIG. Computer 110 may display the turn signal icon 3195 even though vehicle 101 has not yet changed lanes. As shown in FIG. 36. When vehicle 101 is completely in the lane, computer110 may remove or fade turn signal icon 3195.

“Computer 110 can display additional information about turning information in different ways. FIG. Computer 110 may decide that the vehicle must move one lane left to avoid vehicles 3710, 3720, or 3730. Computer 110 might display a small icon 3795 near box 310 to indicate that computer 110 plans to move vehicle 101 one lanes to the left. FIG. 38 Computer 110 can also move along route 3755 or route 3715 in the left lane to indicate vehicle 101’s future path.

“Computer 110” may also display information like the estimated time to reach a specific location. FIG. 39. Vehicle 101 is traveling along roadway 3905. Route 3915 requires that vehicle 101 turn at intersection with roadway 3907. Computer 110 might display the estimated time it will take for vehicle 101 reach a specific location. This could be the intersection of roads 3905 and 3907 or the end of path 3955.

“Computer 110 could use the display to communicate other pertinent information to the passenger. FIG. 40 shows vehicle 101 moving over an overpass. Computer 110 might display box 310 as it moves along a roadway 4005 instead of roadway 4007. Computer 110 might display both the vehicles and the roadway beneath vehicle 101 so that they are clearly visible. To indicate that the objects below vehicle 101 are not relevant, the roadway 4007 may be shown with the vehicles, such as boxes 4010 or 4020.

“As these and many other combinations and variations of the features discussed can be used without departing from this invention as described by the claims,” the above description of exemplary embodiments should not be considered a limitation of the invention as claimed by the claims. It is also understood that the specification of examples of invention (as with clauses such as “such as”), as well as clauses such as “such as,” are not limitations of the invention. ?e.g. ?, ?including? (and the like) should be understood as not limiting the invention’s scope to these specific examples. Rather, the examples are meant to illustrate only a few of the many possible aspects.

It will also be understood that the examples of values, icons, types, and configurations of data shown in the figures are only for illustration purposes. Systems and methods according to aspects of the invention can include different physical attributes, data value, data types, and configurations. They may also be provided and received by different entities at different times.

Summary for “User interface to display internal state of autonomous driving systems”