Invented by Aditya S. KAMINI, Nathaniel H. WILLIAMS, Richard Elswick, GM Global Technology Operations LLC

The market for Augmented Reality Vehicle User Interface In recent years, the automotive industry has witnessed a significant shift towards incorporating advanced technologies into vehicles. One such technology that has gained immense popularity is Augmented Reality (AR). Augmented Reality Vehicle User Interface (ARVUI) is a cutting-edge technology that enhances the driving experience by overlaying digital information onto the real-world environment. The market for ARVUI is rapidly growing, driven by the increasing demand for advanced features and functionalities in vehicles. With the rise of electric vehicles and autonomous driving, there is a need for innovative user interfaces that provide drivers with real-time information and enhance their overall driving experience. One of the key advantages of ARVUI is its ability to provide drivers with important information without distracting them from the road. Traditional vehicle user interfaces, such as dashboards and infotainment systems, require drivers to take their eyes off the road to access information. ARVUI, on the other hand, projects information directly onto the windshield or heads-up display, allowing drivers to keep their eyes on the road while accessing important data. ARVUI also offers a wide range of features that enhance safety and convenience. For example, it can display navigation instructions directly on the windshield, eliminating the need for drivers to look at a separate screen or listen to voice instructions. It can also provide real-time information about traffic conditions, weather updates, and nearby points of interest, making driving more efficient and enjoyable. The market for ARVUI is not limited to passenger vehicles. Commercial vehicles, such as trucks and buses, can also benefit from this technology. ARVUI can provide truck drivers with valuable information about their cargo, fuel consumption, and maintenance schedules. It can also assist bus drivers in monitoring passenger counts, ensuring safety, and providing real-time updates to passengers. The market for ARVUI is expected to witness significant growth in the coming years. According to a report by MarketsandMarkets, the global market for AR in automotive is projected to reach $5.53 billion by 2025, growing at a CAGR of 30.31% from 2018 to 2025. This growth can be attributed to the increasing adoption of ARVUI by automotive manufacturers and the rising demand for advanced driver assistance systems. Several automotive manufacturers have already started incorporating ARVUI into their vehicles. For example, BMW has introduced its AR Head-Up Display, which projects information such as speed, navigation instructions, and warnings onto the windshield. Jaguar Land Rover has also developed an AR windshield system that displays real-time information about the road, surroundings, and potential hazards. In conclusion, the market for Augmented Reality Vehicle User Interface is witnessing significant growth due to the increasing demand for advanced features and functionalities in vehicles. ARVUI offers a range of benefits, including enhanced safety, convenience, and efficiency. As automotive manufacturers continue to invest in this technology, we can expect to see more vehicles equipped with ARVUI in the near future.

The GM Global Technology Operations LLC invention works as follows

The method and system for operating a car using virtual controls includes: capturing images or videos from an area inside the vehicle’s interior; sending virtual vehicle graphics requests to the vehicle; receiving virtual graphics from the vehicle.

Background for Augmented Reality Vehicle User Interface

The invention provides a method for operating a car using virtual controls. This method includes: capturing images or videos from an area inside the vehicle’s interior; sending virtual vehicle graphics requests to the vehicle; receiving virtual graphics from the vehicle.

The following features may be included in this method according to different embodiments or any combination of them that is technically feasible:

the captured image data or video is captured by an AR/VR camera, where the camera is located on the device and faces a specific area in front of the user when the device’s display is facing the user

The AR/VR Device is a pair AR/VR goggles that can be worn by a user to present augmented reality and virtual reality media content. When the AR/VR Device is worn by the wearer, the AR/VR display faces the wearer and the AR/VR camera captures a region in front of them.

Receiving an advertisement for virtual vehicle control graphics that shows virtual vehicle controls are available to use by the AR/VR devices;

establishing a link between the AR/VR devices and the wireless communications devices included in the vehicle electronics using Bluetooth? Wi-Fi? or other IEEE 802.11 technology and the virtual vehicle control graphics being received via this established connection

The virtual vehicle graphics response contains virtual vehicle graphics location information, which indicates the vehicle component on which the virtual graphics will be displayed or the location of the virtual graphics on the display device.

Receiving vehicle control inputs from users at an AR/VR device and sending them to the vehicle. The vehicle then performs one or more actions in response to the inputs.

receiving from the vehicle a second virtual control graphics response after sending vehicle input, wherein the second control graphics includes updated control graphics for the virtual vehicle and presenting these updated control graphics to the display of the VR/AR device

The virtual graphics of the vehicle controls are updated according to vehicle input.

The invention also provides a method for controlling a vehicle by using virtual controls. This method includes: establishing a connection with an AR/VR (augmented reality/virtuality) device that includes a display, a camera and is configured to capture video or image data; determining if the device is authorized to receive and/or use virtual vehicle graphics; sending a response of virtual vehicle graphics to the device using the established SRWC link; the device being further configured to display the virtual graphics over the captured video or image data; a

The following features may be included in this method according to different embodiments or any combination of them that is technically feasible:

The virtual vehicle graphics response contains virtual vehicle graphics location information, which indicates the vehicle component on which the virtual graphics will be displayed or a specific location in the display of the VR/AR device at the location where the graphics will be displayed.

The vehicle component corresponds with a capacitive touchscreen sensor array in the vehicle’s electronics, wherein receiving the sensor location information from the capacitive touchscreen sensor array further includes receiving a sensor location indication from the capacitive touchscreen sensor array.

the sensor location input is used to determine whether the vehicle control input indicates selection or modification for one or more components of the vehicle-user interface;

The AR/VR device receives the vehicle control input.

The vehicle control input is first received at the AR/VR devices using a controller for the AR/VR devices;

The vehicle control input initially is received at the AR/VR camera, and the vehicle control is communicated to the AR/VR via the gestures that the user creates in the field of view of camera.

authorizing the virtual vehicle control graphics request based on authorization data contained in the vehicle input or the virtual vehicle control input;

modifying the operation of one or multiple vehicle system modules (VSMs), in response to receiving vehicle control input, wherein the modifying is based on vehicle control input

after modifying the operation of one or multiple vehicle system modules (VSMs), send a vehicle input response that includes updated graphics for virtual vehicle control; and/or

Click here to view the patent on Google Patents.