Software Technology for “Perception-based predictive tracking for head mounted displays.”

Metaverse – Steve LaValle, Peter Giokaris, Meta Platforms Technologies LLC

Abstract for “Perception-based predictive tracking for head mounted displays.”

“A method and apparatus for predictive tracking of a head-mounted display are disclosed. The method involves obtaining three-dimensional velocity measurements from a sensor that monitors the head-mounted display. This is then used to set a prediction interval. When the head-mounted display is stationary, the prediction interval decreases up to a predetermined interval. Method also includes creating a three-dimensional orientation prediction for the head mount display, which generates a rendered image that corresponds to the predicted orientation.

Background for “Perception-based predictive tracking for head mounted displays.”

Field

“This disclosure is about predictive motion tracking for head mounted displays.”

“Description of Related Art”

Head mounted displays have been used for augmented and virtual reality since long. Virtual reality systems typically enclose the wearer’s eyes completely, and replace a?virtual reality? Virtual reality systems can substitute reality for reality. These virtual reality environments can be crude, either by design or because the virtual reality system is not capable of doing so. Virtual reality environments can be interactive, detailed and complex. They may include virtual people, conversations, and experiences. A video game that allows a player to interact with the world may be an example of a virtual environment. Virtual environments do not have to be games. They can be educational experiences, group activities, or simply sitting with an avatar representing a friend in a virtual space and having a conversation.

“Augmented reality systems provide an overlay semitransparent or transparent screen in front of the wearer’s eyes so that reality is?augmented?” Additional information, graphical representations or supplementary data. Augmented reality can, for instance, superimpose “virtual?” Augmented reality may overlay items, people, cars, rooms and spaces, as well as signs, on the reality of a viewer. An augmented reality system may provide information about the area or scene being viewed (e.g. Temperature, upcoming appointments, speed, location GPS, etc. may be provided by simple augmented reality systems. Complex augmented reality systems might superimpose “virtual?”. There may be tangible objects, such as people, artwork and similar elements, that are added to a scene. They can also be updated in real time so that images displayed on an augmented reality display appear to exist within a specific location for a wearer.

“In both systems, the wearer of the headset’s movement may be tracked to update and react to the images being displayed. Tracking uses sensors such as accelerometers and magnetometers. In some cases, color sensors or cameras can be used to generate data about the headset’s position, motion, orientation, and other details. This data can be used for information like angular velocity, linear acceleration and gravitational data, which may then be used to adjust display settings of the headset to suit wearer movements.

“Predictive motion has been integrated into virtual reality headsets and augmented reality headsets in past. However, sample rates for the sensors identified above have typically been quite long?on the order tens of milliseconds?relative to the acuity of human vision. These predictions are often called “overshoot” because of the long time between samples. If a prediction is too close to the actual head position or orientation, or must be smoothed so as not to cause other problems, it can lead to predictions that are sufficiently inaccurate to simply not predict movement.

“DESCRIPTION DU DRAWINGS”

“FIG. “FIG.

“FIG. “FIG.

“FIG. “FIG.

“FIG. “FIG.

“FIG. “FIG.5” 5 refers to a user who is wearing a virtual reality headset.

“FIG. “FIG. Predictive motion tracking produces jitter in 6A and 6B.

“FIG. “FIG.7, which is made up of FIGS. 7A, 7B, and 7C are examples of predictive motion tracking in comparison to actual movement.

“FIG. “FIG.8” is a flowchart that shows perceptually tuned filtering applied to a predictive movement tracking process.

“Elements appearing in figures have three-digit reference designers. The most significant digit is the number of the figure, and the two lowest significant digits are the specific element. A figure that does not include an element may assume that the element has the same characteristics or function as the previously described element.

“Dynamic motion prediction can be used to account for many factors such as jitter, latency, and overall responsiveness of virtual reality systems. This will significantly reduce problems associated with predictive movement tracking. For example, predictive movement can only be applied to headsets moving to a suitable size for that movement. This will result in better predictive accuracy than simply enabling predictive tracking of all movements to the same degree (over the exact same prediction interval).

Smoothing, for instance, regression-based averaged motions may be applied more aggressively if motion data indicates that the headset is motionless but less aggressively if motion data shows that the headset is moving at an angular velocity that is high or increasing in acceleration. Because human eyes are sensitive to what is called ‘jitter? When the head is not moving at all. Jitter is caused by tiny variations in sensor measurements that cause rendered video to “bounce”. This jitter can be seen when a head is still. In motion, jitter can be almost totally ignored by the human eye.

These types of dynamic scaling of smoothing and motion prediction can be called perceptually-tuned filtering, as they are based on discoveries about human perception along with an understanding the technical limitations of hardware that is available for virtual reality systems.

“Description of Apparatus.”

“Referring to FIG. “Referring now to FIG. 1, an overview is given of a virtual reality 100 system. The system comprises a user system 110, an environment server 120 connected via a network 150 and a VR headset 130. The system also shows a user 135 but the user 135 could be treated as a separate entity from the 100.

“Usually, the user system 110 will run software. This includes operating system software as well as virtual environment software that creates images of virtual environments for display on the VR head-mounted display 130. For example, virtual environment software could be video game software that creates images of a 3-dimensional world so that the VR headset 130 makes it appear that the wearer is ‘in? The virtual environment of the game.

“The environment server 120 can be described as a computing device. Depending on the implementation of virtual reality system 100, the environment server 120 could be optional. The environment server 120 can provide some data to the user system 110 for multi-user and multiparty virtual environments.

“For instance, the environment server 120 could maintain and communicate via the network 150 data pertaining to other users connected to the virtual environment being rendered through the user system 110. This allows for user avatars to be created by a group of VR headset users, such as VR headset 130. One user “moves” within the virtual environment. The environment server 120 can receive the data from that user and pass it to other user systems like user system 110. The environment server 120 can allow a group to interact in a virtual environment.

“In certain embodiments, especially those that don’t include multi-user experiences or multiparty experiences,” the environment server 120 might not be necessary. In these cases, the environment server 120 may not be necessary.

The VR headset 130 is a computing device similar to the user system 110 but with some special characteristics. The VR headset 130 has at least one integrated display. It also includes one or more sensors that can be used to generate motion and/or position data for the VR headset 130. The sensors transmit motion data to the user system 110 when the VR headset 130 is worn. The display integrated into the VR headset 130 will show one or more images returned by the user system 110.

The VR headset 130 could include some or all the functionality of the user-system 110. These systems are still distinct using current technology. This is because they require different processing and storage capabilities to create convincing virtual environments and perform motion extrapolation.

“In certain instances, such that the user system 110 is a smartphone, tablet, or mobile phone, the sensor of the mobile device or the display of the device could replace all or part of the VR headset130 functionality. The display on the mobile device could be replaced by the VR headset 130 display.

The VR headset 130 is worn by the user 135 The headset is worn by the user 135 in a manner that allows the user to see the integrated display 130. To further control the user system 110, the 135 user may use other peripherals (not shown), such as a keyboard, mouse, headphones, speakers or microphones to interact with the headset 130.

“The network 150 is a wireless or wired network that connects user system 110 to environment server 120. The network 150 could be the Internet, a local network (LAN), or a wide-area network (WAN). To facilitate communication between user systems 110 and 120, the network 150 can use standard protocols like Ethernet, TCP/IP, Ethernet, and other large-scale networking protocols. You can also use custom protocols. Broadcast protocols may be used to communicate from the environment server 120 with the user system 110 and other user systems. This allows the environment server 120 to update the user system 110’s data without having to wait for multiple user systems to confirm receipt.

“Turning to FIG. 2. A block diagram of a computing system 200 is shown. This is representative of all the client devices, server computers and other computing devices described herein. Computing devices include the environment server 120 and user system 110.

“The computing device 200 could include software or hardware to provide functionality and features discussed herein. A computing device 200 could include memory, digital circuits and analog circuits. It may also contain software, firmware, processors and other components. The hardware and firmware components may include various specialized units and circuits as well as software and interfaces to provide the functionality and features discussed herein.

“The computing device 200 includes a processor 210, coupled to a storage 214, memory 212, and an I/O interface 218, as well as a network interface (216) and a memory interface 216. One or more processors may include integrated specific function processors (such video processors), storage 214, a network interface 216 and an I/O interface 218.

The memory 212 can include RAM, DRAM, RAM, SRAM, MRAM and MRAM. It may also include firmware such as static data and fixed instructions, BIOS and configuration data. The memory 212 can also store data and instructions that are associated with applications or data processed by the processor.

“The storage214 allows for non-volatile bulk, long-term or volatile storage of data and instructions within the computing device 200. Storage 214 can be in the form of a tape, disk, CD, DVD or SSD (solid state drive), or any other reasonably high-capacity addressable or serial storage medium. Multiple storage devices can be made available or provided for the computing device 200. These storage devices could be external to the 200 computing device, such as cloud-based storage or network storage.

“The network interface 216, which includes an interface to network 150 (FIG. 1).”

“The I/O interface 218 interfaces with the processor 210 and peripherals (not illustrated), such as the VR headset 130 displays, keyboards mice controllers, USB devices, and other peripherals.

“FIG. “FIG. 2. FIG. 2 is a summary of the elements that were discussed previously. 2. will not be repeated. The virtual reality headset 300 computing device may be implemented entirely or in part as a system on a chip.

The processor 312 might be less powerful than the full-size computer. One package may contain the memory 314, storage 318, network interface 316, and input/output 320. This may allow for quick response times and a small instruction set.

The sensors 322 are used to generate motion and position data. These sensors can include gyroscopes and accelerometers as well as magnetometers, video cameras and color sensors. Sub-portions of the sensors 322 could also contain active or passive markers, which can be viewed externally using a color sensor or camera to generate motion, location, and orientation data. A virtual reality headset might have a number of markers on its exterior, such as reflectors and lights (e.g. infrared, visible light) that can be viewed externally by a camera or illuminated with a light (e.g. infrared, visible light) to provide one or more points for interpretation by software to generate motion, location, and orientation data. These markers are not sensors in this sense. They are a sub-part a sensor system used to generate motion and position data.

The sensors 322 can operate at high frequencies to deliver high-quality sensor data. Sensor data can be generated at 1000 Hz, or 1 sensor reading per 1 millisecond. This means that one thousand readings per second are taken. This is because sensors can generate so much data (or even more) at such a rapid rate that it makes it possible to predict motion over very short time periods of time, on the order of the tens or hundreds of milliseconds.

“The virtual reality headset has a display buffer 324, where image data is stored before it is displayed on the display. This buffer may include or be combined with the memory 314 that is available to the processor 312. This allows the display to respond quickly to inputs from the processor 312.

“The display (not illustrated) of the virtual reality headset300 may include one or more displays. For example, the display could be placed directly in front of the 130 VR headset wearer’s eyes. While the display might have a lower resolution, it may still provide a large visual area for the wearer. The display shows the virtual environment as rendered by user system 110.

“Turning towards FIG. 4. A functional diagram of the virtual reality system 400 is shown. Virtual reality system 400 includes a user-system 410, environment server 420, and a VR headset. These could correspond to the environment server 120, user system 110 and VR headset 130 in FIG. 1. Another user 435 is shown, but it is not part of the system.

“The user system410 contains an operating system 411 and network input/output software 412. Virtual reality drivers 414 and motion predictor 415.

“The network input/output protocol 412 allows the user system (410) to communicate with the environment server, 420 (when available). The network input/output connector 412 can include standard network drivers as well as software specific to the environment software 413 which enables network communication.

“The environment software 413 works in conjunction with other components to create a virtual reality environment. Environment software could be video game software, which creates a 3-D world to display on the VR headset 430. Environment software 413 could include special software that allows it to render multiple displays simultaneously from different angles. This is common in VR headsets, which can have two or more displays, or even two images at once. One for each eye. Software designed to work with the VR headset 430 may be integrated into the environment software 413.

The environment software 413 does not have to be videogame software. It can be any type or virtual reality, augmented reality environment software. This software can generate virtual reality locations. It may include historical locations, fictional locations, educational locations, and almost any immersive environment that mimics reality.

“The virtual reality drivers 414 could be considered a software overlay between operating system 411, environment software 413 and VR headset 430. The virtual reality driver 414 receives input from the environment 413 and sends it to the VR headset 433. To speed up the transmission and use environment data, the virtual reality drivers 414 can interact with the VR helmet 430 at one or more abstraction levels below the software level. This is especially true if the user system 410 has a higher processor (e.g. The VR headset 430 has a faster processor. The VR headset430 may thus accept more raw data and display it directly on the VR headset430.

“The motion predictor 415 and smoother 415 are software that can accurately predict movement, orientation and position of the VR headset. The virtual reality drivers 414 may include the motion predictor.

The motion predictor and smoother 415 can accept data from the VR headset 433. This data is used to calculate the VR headset 433, position, orientation, and movement. To predict the future position, orientation and movement of the VR headset 430, the motion predictor 415 and smother 415 can, for instance, average the angular velocity measurements within a very short time period (e.g., 10 measurements in the last ten seconds). The motion predictor 415 and smother415 can also differentiate over a series angular velocity measurements in order to derive an angle acceleration and extrapolate into future time to predict future movement, orientation and position. Below, we will describe the prediction process in greater detail with reference to FIG. 8.”

“In addition, smoother 415 and motion predictor may perform?smoothing? operation on sensor data. The raw motion, orientation, or position data from the VR headset 430 could be slightly off-base. With sample rates in the range of one per millisecond, it is possible for tiny changes in orientation, movement or position to result in incremental sensor readings which do not reflect actual movement, orientation, and position.

The sensors can still detect minor variations even when a VR headset 430 user is sitting still. These data may result in images that do not accurately reflect real changes in the VR headset. So-called “jitter” may be introduced. In the VR headset’s display, 430 might be present. This is what the VR headset 430 user sees as an image “bouncing”. These tiny variations in data from the VR headset 430 sensors cause tiny changes in movement.

Smoothing may be applied to data received by the motion predictor or smoother 415 to address jitter and similar errors. Smoothing could, for instance, average a number of sensors readings to determine the data needed to generate the movement, orientation and position data. This data is then used to create the next image on the VR headset. You can also use nonlinear or linear filtering to smoothen sensor data over a period of time and remove outliers.

The environment server 420 contains an operating system 421, an interface for network input/output 422, and environment software 423. Except for the environment server 420, the operating system 421 is almost identical to the one described above. Operating system 421 will likely be an operating system that supports multiple user connections via a network. The network input/output interface 422, which may be faster or better for multiple connections, could also provide faster connections to a network.

The purpose of the environment software 423 is different from the one 413 in the user system 410. The environment software does not render any three-dimensional world to users. It may instead maintain a state for multiple users connected to the environment servers 420. If hundreds of users connect to the environment server, 420 simultaneously, environment software 423 could maintain an environment database 424 which indicates each user’s location, actions, movements, and other data. These locations may be stored in the environment database 424 and dynamically updated by it. This data can then be transmitted to user systems such as user system 413 so that its visual display can include that data. This could be, for instance, the location in the three-dimensional environment rendered by environment software of an avatar representative.

The VR headset 430 could be substantially identical to the VR headset 130 in FIG. 1. and the computing device 300 in FIG. 3. A VR headset 430 can be worn by 435 users to experience virtual reality and augmented reality virtual environments.

“Turning Now to FIG. 5. A user 535 is shown wearing a virtual reality headset530. User 535 is wearing the VR headset 530 over his eyes. The user 535’s head can be considered to be at the center of a three dimensional axis. It has axes of pitch 535, roll 550, and yaw 560. Pitch 540 can be considered an x axis. Roll 550 can be considered a Z axis. Yaw 560 in a three-dimensional Cartesian coordinate scheme is a y.

“Movement in the VR headset 530 (user motion) can be expressed as pitch, roll, and 550, or yaw, 560. Quaternions are one way to express a three-dimensional orientation. Because any three-dimensional orientation is represented by quaternions in the form q,? ?????= (cos(??/2), vx(?,2), and vy(?/2) respectively. In which case q(v?,?) The same rotation. Quaternions allow manipulation of rotations with fewer parameters, while keeping geometry (for instance, Haar measure) under algebraic operation. This is extremely useful for prediction in three-dimensional spaces.

The Haar measure is used to make sure multiple rotations of an object within three-dimensional space are related. A third rotation applied to two previous rotations would, in a preferred scenario, still result in the two earlier ones being separated by the same distance. This is possible by keeping consistent with the Haar measure. It also works well with quaternions.

“FIG. “FIG. FIG. 6A and 6B show an example of jitter caused by predictive motion tracking. FIG. FIG. 6A displays a scene consisting of two images 620, 622 and presented on a VR headset. To create depth, two images (620 and 622) are displayed, one for each eye, and from slightly different perspectives. The individual 630 may be seen in different angles in each of the images 620 and 622. The mind perceives the individual 630 as having depth of field. This enhances immersion.

“FIG. 6B depicts a different scene, including images 640 & 642. The individual 650 has “moved?” The scene shows the individual 650 moving upwards. This type of movement can occur when predictive tracking is used and the sensor detection drifts a little. A minor fluctuation in sensors, which could indicate that the wearer is “looking up”, may cause predictive motion tracking systems to exaggerate this motion data into a dramatic movement that isn’t there.

“In FIG. “In FIG. 6B, a minor fluctuation was extrapolated to create motion data that does not exist. The wearer’s head remains substantially stationary. The effect can be quite dramatic if this happens multiple times in a matter of seconds. The image in FIG. 6B is slightly exaggerated to make it more easily recognisable. 6B. A wearer of a motionless VR helmet experiences minor jitter as very non-immersive. This is because the wearer’s brain thinks it is stationary. Visually perceiving movement data, even tiny jitter, does not correspond to actual head movement or non-movement. This is not consistent with the wearer’s remaining perception.

“FIG. “FIG.7, which is made up of FIGS. FIGS. 7A, 7B, and 7C show an example of predictive motion tracking relative to actual movement. FIG. FIG. 7A shows the exact same scene as FIG. 6A, with images 722, 720 and individual 730.

“FIG. 7B is a scene that uses predictive motion tracking to project a wearer?s movements over time into the future. Individual 750 seems to have moved to one side in the updated images 740 and 742. Visual perception suggests that this could be because the wearer turned his or her head to one side, which caused the individual 750’s view to shift to the left.

“The predictive motion tracking has predicted where the individual 750 will be located (and the orientation for the entire scene), based on motion, orientation and position data extrapolated using sensor data generated by the VR headset. The motion, orientation, or position data of the VR headset may indicate, for example, that it is moving in a specific direction and at a certain angular velocity (or possibly linear velocity, angular, or linear acceleration). 7B could be extrapolated or predicted over a time period.

“FIG. 7C is a scene that includes two images 760, 762 and individual 770. This corresponds to the actual motion and orientation of a VR headset. Take care to examine FIGS. FIGS. 7B and 7C show that the motion predicted was slightly off. The individual 750 didn’t move enough to the left in the view field compared to actual motion, orientation and position in images 762 and 760. The individual 770 is therefore slightly left of the scene. These errors can be seen by comparing the predicted distances 744 & 746 with the actual distances 764 & 766.

These minor errors are almost non-existent when the methods described herein are used. Predictive motion tracking is not perfect. It predicts a wearer?s movements in advance of them happening. The wearer may not be able to see the difference if the errors are small. This is especially true in virtual reality systems.

“Description of processes”

“Referring to FIG. 8 A flowchart that shows perceptually tuned filtering applied to predictive motion tracking. Although the flow chart shows a start 805 as well as an end 895, it is actually cyclical. The process almost always iterates as long as the VR headset is worn on a person’s head. The process could iterate for each frame of rendered video at least once, and possibly several times.

Sensor measurements are taken from all sensors (including those in the VR headset), at 810. These sensors could include accelerometers and gyroscopes as well as magnetometers. Mounting the cameras on the headset may allow for markers to be placed in the environment. Or, markers can be mounted on headsets and external cameras may track them. These sensors provide precise data that can be used to make predictions about motion, orientation, and position. The sensors can be placed anywhere and provide data to the user system, allowing them to start the process of prediction based on perception.

“Next, sensor-fusion takes place at 820. This process can be complicated or simple. In a simple system, raw data (e.g. velocity and acceleration) can be combined into a standard form that is suitable for environment software to render a three-dimensional environment. The data might be “smoothed” in more complicated systems. As mentioned above, data extrapolating possible predictions for headset movement may be generated at the sensor-fusion stage and then passed on to subsequent stages.

“A determination is made using the virtual reality drivers 414 to determine whether the headset is turning at 8.25. This is a simple task. A series of angular velocity measurements can be taken in a very short time (e.g. The sensors can generate a series of angular velocity measurements in a short time period (e.g., 10 milliseconds). These measurements can be combined to determine if the angle velocity exceeds a threshold. For example, 1-5 degrees per second. This will make it impossible to perceive a minor movement or a few sensor errors as “movement”. ”

A VR headset can’t be worn by a human head. However, a threshold of movement can be set so that it can be determined if the headset is turning 825. A head that is not turning is considered substantially stationary. The expression “substantially stationary” is used herein. The headset is not moving in a way that would cause the virtual reality system to re-render the scene. Specifically, ?substantially stationary? This does not necessarily mean that the VR headset is in a stationary position. To the extent that a person can keep their head’still’ without moving, but not necessarily to zero movement.

Smoothing can be disabled at 830 if the headset is turning. Smoothing, as mentioned above, is primarily designed to counteract sensor artifacts and minor movements. Sensor drift (linear, exponential, or linear drift in the accuracy of sensors) is also used. This ensures that the headset does not cause jitter or random movement. Studying human perception has shown that virtual reality environments are perceived differently by people. They don’t experience any jitter when their heads turn. The mind can either ignore jitter, or it is more used to seeing some visual irregularities in a scene when a head turns. It automatically compensates for them.

Smoothing can be disabled temporarily if the headset is turning. This allows processor cycles to be used for other tasks, but it also ensures that predictive motion tracking is not affected by pre- or post-calculation smoothing. Smoothing too much can make rendered frames appear slower than actual movements. This can be caused by over-weighting or undershooting of prior data, which can indicate smaller (or no) movements. Disabling smoothing can be used to detect if the headset is turning. This will result in a more responsive rendering of scene details as the headset turns.

“Next (or substantially simultaneously), an angle velocity of the headset at 840 is determined. This measurement can be one or more data sets generated by the sensorfusion at 820. The angular speed is best determined using one of several options, such as an average of a prior set of angular momentum measurements (e.g. The most recent 10 measurements to a weighted mean of similar angular velocity measurements with additional weighting applied for the most current set of measurements (e.g. The most recent 1-3 measurements to determine the linear acceleration by differentiation, extrapolation, or acceleration over a time period.

The next step is to determine the prediction interval for motion prediction at 850. The ‘prediction interval’ is the time that the prediction of motion will take place. The prediction interval is the time that the motion prediction is made. If the prediction interval is 40ms, then the predictive motion tracking will predict the position, orientation, and motion of the headset at the 40ms mark. The prediction interval of zero means that the predictive motion tracker predicts the position, orientation and motion of the headset at the moment. This effectively eliminates predictive motion tracking. The prediction interval allows the predictive motion tracking system dynamically to respond to headset movements of any size or scope.

This prediction interval could be very small for small movements. A prediction interval of 5 milliseconds may be used for movements that are less than 10 degrees per second. These situations are where the angular velocity measurements, regardless of their method, are applied for a time period and used in lieu of actual data to provide a user system with the ability to render the associated video frames. Based on the prediction, the next frame of rendered footage is produced.

“In cases where the angular velocity measurement (e.g. 500 degrees/second) is very large, the prediction interval within which the system predicts movement is increased. The system will set the prediction interval between 30 and 60 microseconds for fast movements, as an example. These situations are where the motion prediction data is passed to a renderer. It predicts the wearer’s position, orientation, and motion at the end the prediction interval. The renderer then delivers the next frame of video based on that prediction.

“The latency interval is well-known in well-studied systems. “Latency interval” is a term that refers to the time between when motion data is generated by sensors and when it is delivered to the renderer. The latency interval is the time between the moment motion data is generated by sensors and the delivery of a rendered video frame incorporating that motion data. The latency interval in the current system is approximately 40 milliseconds.

“This latency interval can be used to set an upper limit for the prediction interval. The prediction interval should not exceed the latency interval. Because the headset’s head movement causes the next rendered frame of video to be rendered at the next latency interval.

“Accordingly, it is not possible to predict beyond this time frame and more likely will result in overshoot (rendering a too large movement or a stop when movement was just starting) and disconnect from actual wearer movements. In the same way, additional data can also be made available in the described system, including tens of additional position, motion, and orientation data points that can all be used to render next frame of video. Predicting motion to the latency interval can be used to attempt to predict user movements with the next frame of rendered video.

The process could work in a way that determines the angular velocity of wearer movement. The time for prediction increases linearly from the threshold rate of 0.3 radians/second to the latency interval. For example, a prediction interval increase of +2 milliseconds per radion per second could be considered. A headset that turns at 1 radians per sec would use a prediction interval of 2 miliseconds in the future. A headset that turns at 10 radians per seconds would use a prediction interval 20 milliseconds into the future. You can also use linear increase slopes, with a maximum prediction interval equaling the latency interval (e.g. “40 milliseconds into future

If the angular velocity (e.g. The angular velocity is 0.03 to 0.03 radians/second. In order to determine the orientation in three dimensions, smoothing is used. It relies on the current position and orientation and does not predict. Smoothing is disabled if the angular velocity exceeds substantially zero. The position, or place in three-dimensional space, is calculated using the last measured position and the linear velocity over the chosen prediction interval.

To summarize, movement (angular velocity of the headset) is measured to determine if it is substantially zero (0 to 0.33 radians per sec). Smoothing is activated and the prediction interval is set either to substantially zero or zero. Smoothing is disabled if the headset moves at an angular velocity greater than substantially zero (greater that 0.3 radians/second), and the prediction interval is then set. If the headset is moving at an increasing angular velocity, the prediction interval will rise by a prediction interval variance (e.g. 2 milliseconds/radian per second per reading up until a maximum latency interval (e.g. 40 milliseconds).

The linear increase in prediction interval (up until the maximum latency interval) could be linear. This corresponds to linear movement of a headset. Therefore, faster movement can be compensated by a linear increase in the prediction interval. A smoothstep algorithm can also be used. Smoothstep algorithms take the generalized form smoothstep(t)=3t2?2t3 where t is the variable to be smoothed and the results of the generalized form fall between 0 and 1. Smoothstep is sloped towards zero at the high and low ends. This allows Smoothstep to be rounded off, which is a different method from linear interpolation. The ends. Wearers of such headsets may believe that their prediction interval is slowing approaching the maximum or minimum, without abrupt stops or starts at either the beginning or end. The prediction interval can be?smoothly’. The maximum latency interval can be approached or it can approach zero.”

Smoothing can be activated at 860 if the headset stops turning at 825. This indicates that the user’s head remains substantially still. Smoothing is most useful when the headset isn’t moving or making small movements, as discussed above. It helps avoid excessive or exaggerated movements of the scene such as jitter when a headset remains substantially still. These issues can be reduced by allowing smoothing to occur when the headset is substantially still.

“Similarly, at 870, the prediction interval has been set to nearly zero. If the head is stationary, it is impossible to make an aggressive prediction for any time period. Although some predictive motion tracking is possible, it is limited to a short time period, like 5 milliseconds. The prediction interval can be set to zero in some cases. This helps avoid predictive motion tracking which may exaggerate small movements.

“Next, the headset will be able to tell whether it is turning at 825. After setting the prediction interval at 870 and 850, the prediction of an orientation and angular velocity at 880 will be made. This predicts the position, orientation, and motion of the headset at the beginning of the prediction interval. This can be expressed as a quaternion as mentioned above. However, it may also be expressed in any other way.

Finally, the display is updated at each rendered frame to correspond with the predicted position, orientation, motion based on the prediction interval at 890. If the prediction interval was 40ms, then the VR headset’s next rendered frame will render as though the wearer’s head is in a predetermined position 40ms away. If the prediction interval was zero, then the VR headset will render the next rendered frame as though the wearer’s head is in a predetermined position. Actual data, however, is not required. This means that there is no prediction, or even a very limited prediction.

“Both these options and extremes indicate that the head is turning at 825 or not. The process is over once the display has been updated. The process is repeated for each sensor update (or for any interval of sensor updates) as explained above. Every 10 to 20 sensor updates. This allows the rendering of the next frame of rendered footage to be done with smoothing or with no smoothing, and with or with out a prediction interval at, or near, a latency interval.

“Closing Comments”

“The embodiments and examples presented in this description should not be interpreted as limitations or restrictions on the disclosed and claimed apparatuses and procedures. While many examples herein are specific combinations of system elements or method acts, it is important to understand that these elements and acts can be combined in different ways to achieve the same goals. You can add or subtract steps to flowcharts. The steps shown in this document may also be combined with other steps. The acts, elements and features described only in relation to one embodiment do not exclude others from playing a similar role.

“As used herein, ?plurality? It can refer to two or more. A?set? is a group of items. A?set? may be a collection of one or more items. The terms ‘comprising? and ‘including? are used in this claim. Only the transitional phrases “consisting of” and “consisting essentially of?” are closed or semi-closed transitional phrases. The transitional phrases?consisting essentially? and?first? are respectively closed or semi-closed phrase with respect to claims. The use of ordinal terms like?first? or?second? in claims to modify claim elements does not in itself indicate any priority, precedence or order of claim elements over others or the temporal order by which acts of a methodology are performed. They are used only as labels to differentiate one claim element with a particular name from another element with a similar name (but for the use of the ordinal phrase) to distinguish claim elements. As used herein, ?and/or? The expression?and/or? means that the items listed are alternatives to the items, but that all options can be used in addition to the items listed.

Summary for “Perception-based predictive tracking for head mounted displays.”

Field

“This disclosure is about predictive motion tracking for head mounted displays.”

“Description of Related Art”

Head mounted displays have been used for augmented and virtual reality since long. Virtual reality systems typically enclose the wearer’s eyes completely, and replace a?virtual reality? Virtual reality systems can substitute reality for reality. These virtual reality environments can be crude, either by design or because the virtual reality system is not capable of doing so. Virtual reality environments can be interactive, detailed and complex. They may include virtual people, conversations, and experiences. A video game that allows a player to interact with the world may be an example of a virtual environment. Virtual environments do not have to be games. They can be educational experiences, group activities, or simply sitting with an avatar representing a friend in a virtual space and having a conversation.

“Augmented reality systems provide an overlay semitransparent or transparent screen in front of the wearer’s eyes so that reality is?augmented?” Additional information, graphical representations or supplementary data. Augmented reality can, for instance, superimpose “virtual?” Augmented reality may overlay items, people, cars, rooms and spaces, as well as signs, on the reality of a viewer. An augmented reality system may provide information about the area or scene being viewed (e.g. Temperature, upcoming appointments, speed, location GPS, etc. may be provided by simple augmented reality systems. Complex augmented reality systems might superimpose “virtual?”. There may be tangible objects, such as people, artwork and similar elements, that are added to a scene. They can also be updated in real time so that images displayed on an augmented reality display appear to exist within a specific location for a wearer.

“In both systems, the wearer of the headset’s movement may be tracked to update and react to the images being displayed. Tracking uses sensors such as accelerometers and magnetometers. In some cases, color sensors or cameras can be used to generate data about the headset’s position, motion, orientation, and other details. This data can be used for information like angular velocity, linear acceleration and gravitational data, which may then be used to adjust display settings of the headset to suit wearer movements.

“Predictive motion has been integrated into virtual reality headsets and augmented reality headsets in past. However, sample rates for the sensors identified above have typically been quite long?on the order tens of milliseconds?relative to the acuity of human vision. These predictions are often called “overshoot” because of the long time between samples. If a prediction is too close to the actual head position or orientation, or must be smoothed so as not to cause other problems, it can lead to predictions that are sufficiently inaccurate to simply not predict movement.

“DESCRIPTION DU DRAWINGS”

“FIG. “FIG.

“FIG. “FIG.

“FIG. “FIG.

“FIG. “FIG.

“FIG. “FIG.5” 5 refers to a user who is wearing a virtual reality headset.

“FIG. “FIG. Predictive motion tracking produces jitter in 6A and 6B.

“FIG. “FIG.7, which is made up of FIGS. 7A, 7B, and 7C are examples of predictive motion tracking in comparison to actual movement.

“FIG. “FIG.8” is a flowchart that shows perceptually tuned filtering applied to a predictive movement tracking process.

“Elements appearing in figures have three-digit reference designers. The most significant digit is the number of the figure, and the two lowest significant digits are the specific element. A figure that does not include an element may assume that the element has the same characteristics or function as the previously described element.

“Dynamic motion prediction can be used to account for many factors such as jitter, latency, and overall responsiveness of virtual reality systems. This will significantly reduce problems associated with predictive movement tracking. For example, predictive movement can only be applied to headsets moving to a suitable size for that movement. This will result in better predictive accuracy than simply enabling predictive tracking of all movements to the same degree (over the exact same prediction interval).

Smoothing, for instance, regression-based averaged motions may be applied more aggressively if motion data indicates that the headset is motionless but less aggressively if motion data shows that the headset is moving at an angular velocity that is high or increasing in acceleration. Because human eyes are sensitive to what is called ‘jitter? When the head is not moving at all. Jitter is caused by tiny variations in sensor measurements that cause rendered video to “bounce”. This jitter can be seen when a head is still. In motion, jitter can be almost totally ignored by the human eye.

These types of dynamic scaling of smoothing and motion prediction can be called perceptually-tuned filtering, as they are based on discoveries about human perception along with an understanding the technical limitations of hardware that is available for virtual reality systems.

“Description of Apparatus.”

“Referring to FIG. “Referring now to FIG. 1, an overview is given of a virtual reality 100 system. The system comprises a user system 110, an environment server 120 connected via a network 150 and a VR headset 130. The system also shows a user 135 but the user 135 could be treated as a separate entity from the 100.

“Usually, the user system 110 will run software. This includes operating system software as well as virtual environment software that creates images of virtual environments for display on the VR head-mounted display 130. For example, virtual environment software could be video game software that creates images of a 3-dimensional world so that the VR headset 130 makes it appear that the wearer is ‘in? The virtual environment of the game.

“The environment server 120 can be described as a computing device. Depending on the implementation of virtual reality system 100, the environment server 120 could be optional. The environment server 120 can provide some data to the user system 110 for multi-user and multiparty virtual environments.

“For instance, the environment server 120 could maintain and communicate via the network 150 data pertaining to other users connected to the virtual environment being rendered through the user system 110. This allows for user avatars to be created by a group of VR headset users, such as VR headset 130. One user “moves” within the virtual environment. The environment server 120 can receive the data from that user and pass it to other user systems like user system 110. The environment server 120 can allow a group to interact in a virtual environment.

“In certain embodiments, especially those that don’t include multi-user experiences or multiparty experiences,” the environment server 120 might not be necessary. In these cases, the environment server 120 may not be necessary.

The VR headset 130 is a computing device similar to the user system 110 but with some special characteristics. The VR headset 130 has at least one integrated display. It also includes one or more sensors that can be used to generate motion and/or position data for the VR headset 130. The sensors transmit motion data to the user system 110 when the VR headset 130 is worn. The display integrated into the VR headset 130 will show one or more images returned by the user system 110.

The VR headset 130 could include some or all the functionality of the user-system 110. These systems are still distinct using current technology. This is because they require different processing and storage capabilities to create convincing virtual environments and perform motion extrapolation.

“In certain instances, such that the user system 110 is a smartphone, tablet, or mobile phone, the sensor of the mobile device or the display of the device could replace all or part of the VR headset130 functionality. The display on the mobile device could be replaced by the VR headset 130 display.

The VR headset 130 is worn by the user 135 The headset is worn by the user 135 in a manner that allows the user to see the integrated display 130. To further control the user system 110, the 135 user may use other peripherals (not shown), such as a keyboard, mouse, headphones, speakers or microphones to interact with the headset 130.

“The network 150 is a wireless or wired network that connects user system 110 to environment server 120. The network 150 could be the Internet, a local network (LAN), or a wide-area network (WAN). To facilitate communication between user systems 110 and 120, the network 150 can use standard protocols like Ethernet, TCP/IP, Ethernet, and other large-scale networking protocols. You can also use custom protocols. Broadcast protocols may be used to communicate from the environment server 120 with the user system 110 and other user systems. This allows the environment server 120 to update the user system 110’s data without having to wait for multiple user systems to confirm receipt.

“Turning to FIG. 2. A block diagram of a computing system 200 is shown. This is representative of all the client devices, server computers and other computing devices described herein. Computing devices include the environment server 120 and user system 110.

“The computing device 200 could include software or hardware to provide functionality and features discussed herein. A computing device 200 could include memory, digital circuits and analog circuits. It may also contain software, firmware, processors and other components. The hardware and firmware components may include various specialized units and circuits as well as software and interfaces to provide the functionality and features discussed herein.

“The computing device 200 includes a processor 210, coupled to a storage 214, memory 212, and an I/O interface 218, as well as a network interface (216) and a memory interface 216. One or more processors may include integrated specific function processors (such video processors), storage 214, a network interface 216 and an I/O interface 218.

The memory 212 can include RAM, DRAM, RAM, SRAM, MRAM and MRAM. It may also include firmware such as static data and fixed instructions, BIOS and configuration data. The memory 212 can also store data and instructions that are associated with applications or data processed by the processor.

“The storage214 allows for non-volatile bulk, long-term or volatile storage of data and instructions within the computing device 200. Storage 214 can be in the form of a tape, disk, CD, DVD or SSD (solid state drive), or any other reasonably high-capacity addressable or serial storage medium. Multiple storage devices can be made available or provided for the computing device 200. These storage devices could be external to the 200 computing device, such as cloud-based storage or network storage.

“The network interface 216, which includes an interface to network 150 (FIG. 1).”

“The I/O interface 218 interfaces with the processor 210 and peripherals (not illustrated), such as the VR headset 130 displays, keyboards mice controllers, USB devices, and other peripherals.

“FIG. “FIG. 2. FIG. 2 is a summary of the elements that were discussed previously. 2. will not be repeated. The virtual reality headset 300 computing device may be implemented entirely or in part as a system on a chip.

The processor 312 might be less powerful than the full-size computer. One package may contain the memory 314, storage 318, network interface 316, and input/output 320. This may allow for quick response times and a small instruction set.

The sensors 322 are used to generate motion and position data. These sensors can include gyroscopes and accelerometers as well as magnetometers, video cameras and color sensors. Sub-portions of the sensors 322 could also contain active or passive markers, which can be viewed externally using a color sensor or camera to generate motion, location, and orientation data. A virtual reality headset might have a number of markers on its exterior, such as reflectors and lights (e.g. infrared, visible light) that can be viewed externally by a camera or illuminated with a light (e.g. infrared, visible light) to provide one or more points for interpretation by software to generate motion, location, and orientation data. These markers are not sensors in this sense. They are a sub-part a sensor system used to generate motion and position data.

The sensors 322 can operate at high frequencies to deliver high-quality sensor data. Sensor data can be generated at 1000 Hz, or 1 sensor reading per 1 millisecond. This means that one thousand readings per second are taken. This is because sensors can generate so much data (or even more) at such a rapid rate that it makes it possible to predict motion over very short time periods of time, on the order of the tens or hundreds of milliseconds.

“The virtual reality headset has a display buffer 324, where image data is stored before it is displayed on the display. This buffer may include or be combined with the memory 314 that is available to the processor 312. This allows the display to respond quickly to inputs from the processor 312.

“The display (not illustrated) of the virtual reality headset300 may include one or more displays. For example, the display could be placed directly in front of the 130 VR headset wearer’s eyes. While the display might have a lower resolution, it may still provide a large visual area for the wearer. The display shows the virtual environment as rendered by user system 110.

“Turning towards FIG. 4. A functional diagram of the virtual reality system 400 is shown. Virtual reality system 400 includes a user-system 410, environment server 420, and a VR headset. These could correspond to the environment server 120, user system 110 and VR headset 130 in FIG. 1. Another user 435 is shown, but it is not part of the system.

“The user system410 contains an operating system 411 and network input/output software 412. Virtual reality drivers 414 and motion predictor 415.

“The network input/output protocol 412 allows the user system (410) to communicate with the environment server, 420 (when available). The network input/output connector 412 can include standard network drivers as well as software specific to the environment software 413 which enables network communication.

“The environment software 413 works in conjunction with other components to create a virtual reality environment. Environment software could be video game software, which creates a 3-D world to display on the VR headset 430. Environment software 413 could include special software that allows it to render multiple displays simultaneously from different angles. This is common in VR headsets, which can have two or more displays, or even two images at once. One for each eye. Software designed to work with the VR headset 430 may be integrated into the environment software 413.

The environment software 413 does not have to be videogame software. It can be any type or virtual reality, augmented reality environment software. This software can generate virtual reality locations. It may include historical locations, fictional locations, educational locations, and almost any immersive environment that mimics reality.

“The virtual reality drivers 414 could be considered a software overlay between operating system 411, environment software 413 and VR headset 430. The virtual reality driver 414 receives input from the environment 413 and sends it to the VR headset 433. To speed up the transmission and use environment data, the virtual reality drivers 414 can interact with the VR helmet 430 at one or more abstraction levels below the software level. This is especially true if the user system 410 has a higher processor (e.g. The VR headset 430 has a faster processor. The VR headset430 may thus accept more raw data and display it directly on the VR headset430.

“The motion predictor 415 and smoother 415 are software that can accurately predict movement, orientation and position of the VR headset. The virtual reality drivers 414 may include the motion predictor.

The motion predictor and smoother 415 can accept data from the VR headset 433. This data is used to calculate the VR headset 433, position, orientation, and movement. To predict the future position, orientation and movement of the VR headset 430, the motion predictor 415 and smother 415 can, for instance, average the angular velocity measurements within a very short time period (e.g., 10 measurements in the last ten seconds). The motion predictor 415 and smother415 can also differentiate over a series angular velocity measurements in order to derive an angle acceleration and extrapolate into future time to predict future movement, orientation and position. Below, we will describe the prediction process in greater detail with reference to FIG. 8.”

“In addition, smoother 415 and motion predictor may perform?smoothing? operation on sensor data. The raw motion, orientation, or position data from the VR headset 430 could be slightly off-base. With sample rates in the range of one per millisecond, it is possible for tiny changes in orientation, movement or position to result in incremental sensor readings which do not reflect actual movement, orientation, and position.

The sensors can still detect minor variations even when a VR headset 430 user is sitting still. These data may result in images that do not accurately reflect real changes in the VR headset. So-called “jitter” may be introduced. In the VR headset’s display, 430 might be present. This is what the VR headset 430 user sees as an image “bouncing”. These tiny variations in data from the VR headset 430 sensors cause tiny changes in movement.

Smoothing may be applied to data received by the motion predictor or smoother 415 to address jitter and similar errors. Smoothing could, for instance, average a number of sensors readings to determine the data needed to generate the movement, orientation and position data. This data is then used to create the next image on the VR headset. You can also use nonlinear or linear filtering to smoothen sensor data over a period of time and remove outliers.

The environment server 420 contains an operating system 421, an interface for network input/output 422, and environment software 423. Except for the environment server 420, the operating system 421 is almost identical to the one described above. Operating system 421 will likely be an operating system that supports multiple user connections via a network. The network input/output interface 422, which may be faster or better for multiple connections, could also provide faster connections to a network.

The purpose of the environment software 423 is different from the one 413 in the user system 410. The environment software does not render any three-dimensional world to users. It may instead maintain a state for multiple users connected to the environment servers 420. If hundreds of users connect to the environment server, 420 simultaneously, environment software 423 could maintain an environment database 424 which indicates each user’s location, actions, movements, and other data. These locations may be stored in the environment database 424 and dynamically updated by it. This data can then be transmitted to user systems such as user system 413 so that its visual display can include that data. This could be, for instance, the location in the three-dimensional environment rendered by environment software of an avatar representative.

The VR headset 430 could be substantially identical to the VR headset 130 in FIG. 1. and the computing device 300 in FIG. 3. A VR headset 430 can be worn by 435 users to experience virtual reality and augmented reality virtual environments.

“Turning Now to FIG. 5. A user 535 is shown wearing a virtual reality headset530. User 535 is wearing the VR headset 530 over his eyes. The user 535’s head can be considered to be at the center of a three dimensional axis. It has axes of pitch 535, roll 550, and yaw 560. Pitch 540 can be considered an x axis. Roll 550 can be considered a Z axis. Yaw 560 in a three-dimensional Cartesian coordinate scheme is a y.

“Movement in the VR headset 530 (user motion) can be expressed as pitch, roll, and 550, or yaw, 560. Quaternions are one way to express a three-dimensional orientation. Because any three-dimensional orientation is represented by quaternions in the form q,? ?????= (cos(??/2), vx(?,2), and vy(?/2) respectively. In which case q(v?,?) The same rotation. Quaternions allow manipulation of rotations with fewer parameters, while keeping geometry (for instance, Haar measure) under algebraic operation. This is extremely useful for prediction in three-dimensional spaces.

The Haar measure is used to make sure multiple rotations of an object within three-dimensional space are related. A third rotation applied to two previous rotations would, in a preferred scenario, still result in the two earlier ones being separated by the same distance. This is possible by keeping consistent with the Haar measure. It also works well with quaternions.

“FIG. “FIG. FIG. 6A and 6B show an example of jitter caused by predictive motion tracking. FIG. FIG. 6A displays a scene consisting of two images 620, 622 and presented on a VR headset. To create depth, two images (620 and 622) are displayed, one for each eye, and from slightly different perspectives. The individual 630 may be seen in different angles in each of the images 620 and 622. The mind perceives the individual 630 as having depth of field. This enhances immersion.

“FIG. 6B depicts a different scene, including images 640 & 642. The individual 650 has “moved?” The scene shows the individual 650 moving upwards. This type of movement can occur when predictive tracking is used and the sensor detection drifts a little. A minor fluctuation in sensors, which could indicate that the wearer is “looking up”, may cause predictive motion tracking systems to exaggerate this motion data into a dramatic movement that isn’t there.

“In FIG. “In FIG. 6B, a minor fluctuation was extrapolated to create motion data that does not exist. The wearer’s head remains substantially stationary. The effect can be quite dramatic if this happens multiple times in a matter of seconds. The image in FIG. 6B is slightly exaggerated to make it more easily recognisable. 6B. A wearer of a motionless VR helmet experiences minor jitter as very non-immersive. This is because the wearer’s brain thinks it is stationary. Visually perceiving movement data, even tiny jitter, does not correspond to actual head movement or non-movement. This is not consistent with the wearer’s remaining perception.

“FIG. “FIG.7, which is made up of FIGS. FIGS. 7A, 7B, and 7C show an example of predictive motion tracking relative to actual movement. FIG. FIG. 7A shows the exact same scene as FIG. 6A, with images 722, 720 and individual 730.

“FIG. 7B is a scene that uses predictive motion tracking to project a wearer?s movements over time into the future. Individual 750 seems to have moved to one side in the updated images 740 and 742. Visual perception suggests that this could be because the wearer turned his or her head to one side, which caused the individual 750’s view to shift to the left.

“The predictive motion tracking has predicted where the individual 750 will be located (and the orientation for the entire scene), based on motion, orientation and position data extrapolated using sensor data generated by the VR headset. The motion, orientation, or position data of the VR headset may indicate, for example, that it is moving in a specific direction and at a certain angular velocity (or possibly linear velocity, angular, or linear acceleration). 7B could be extrapolated or predicted over a time period.

“FIG. 7C is a scene that includes two images 760, 762 and individual 770. This corresponds to the actual motion and orientation of a VR headset. Take care to examine FIGS. FIGS. 7B and 7C show that the motion predicted was slightly off. The individual 750 didn’t move enough to the left in the view field compared to actual motion, orientation and position in images 762 and 760. The individual 770 is therefore slightly left of the scene. These errors can be seen by comparing the predicted distances 744 & 746 with the actual distances 764 & 766.

These minor errors are almost non-existent when the methods described herein are used. Predictive motion tracking is not perfect. It predicts a wearer?s movements in advance of them happening. The wearer may not be able to see the difference if the errors are small. This is especially true in virtual reality systems.

“Description of processes”

“Referring to FIG. 8 A flowchart that shows perceptually tuned filtering applied to predictive motion tracking. Although the flow chart shows a start 805 as well as an end 895, it is actually cyclical. The process almost always iterates as long as the VR headset is worn on a person’s head. The process could iterate for each frame of rendered video at least once, and possibly several times.

Sensor measurements are taken from all sensors (including those in the VR headset), at 810. These sensors could include accelerometers and gyroscopes as well as magnetometers. Mounting the cameras on the headset may allow for markers to be placed in the environment. Or, markers can be mounted on headsets and external cameras may track them. These sensors provide precise data that can be used to make predictions about motion, orientation, and position. The sensors can be placed anywhere and provide data to the user system, allowing them to start the process of prediction based on perception.

“Next, sensor-fusion takes place at 820. This process can be complicated or simple. In a simple system, raw data (e.g. velocity and acceleration) can be combined into a standard form that is suitable for environment software to render a three-dimensional environment. The data might be “smoothed” in more complicated systems. As mentioned above, data extrapolating possible predictions for headset movement may be generated at the sensor-fusion stage and then passed on to subsequent stages.

“A determination is made using the virtual reality drivers 414 to determine whether the headset is turning at 8.25. This is a simple task. A series of angular velocity measurements can be taken in a very short time (e.g. The sensors can generate a series of angular velocity measurements in a short time period (e.g., 10 milliseconds). These measurements can be combined to determine if the angle velocity exceeds a threshold. For example, 1-5 degrees per second. This will make it impossible to perceive a minor movement or a few sensor errors as “movement”. ”

A VR headset can’t be worn by a human head. However, a threshold of movement can be set so that it can be determined if the headset is turning 825. A head that is not turning is considered substantially stationary. The expression “substantially stationary” is used herein. The headset is not moving in a way that would cause the virtual reality system to re-render the scene. Specifically, ?substantially stationary? This does not necessarily mean that the VR headset is in a stationary position. To the extent that a person can keep their head’still’ without moving, but not necessarily to zero movement.

Smoothing can be disabled at 830 if the headset is turning. Smoothing, as mentioned above, is primarily designed to counteract sensor artifacts and minor movements. Sensor drift (linear, exponential, or linear drift in the accuracy of sensors) is also used. This ensures that the headset does not cause jitter or random movement. Studying human perception has shown that virtual reality environments are perceived differently by people. They don’t experience any jitter when their heads turn. The mind can either ignore jitter, or it is more used to seeing some visual irregularities in a scene when a head turns. It automatically compensates for them.

Smoothing can be disabled temporarily if the headset is turning. This allows processor cycles to be used for other tasks, but it also ensures that predictive motion tracking is not affected by pre- or post-calculation smoothing. Smoothing too much can make rendered frames appear slower than actual movements. This can be caused by over-weighting or undershooting of prior data, which can indicate smaller (or no) movements. Disabling smoothing can be used to detect if the headset is turning. This will result in a more responsive rendering of scene details as the headset turns.

“Next (or substantially simultaneously), an angle velocity of the headset at 840 is determined. This measurement can be one or more data sets generated by the sensorfusion at 820. The angular speed is best determined using one of several options, such as an average of a prior set of angular momentum measurements (e.g. The most recent 10 measurements to a weighted mean of similar angular velocity measurements with additional weighting applied for the most current set of measurements (e.g. The most recent 1-3 measurements to determine the linear acceleration by differentiation, extrapolation, or acceleration over a time period.

The next step is to determine the prediction interval for motion prediction at 850. The ‘prediction interval’ is the time that the prediction of motion will take place. The prediction interval is the time that the motion prediction is made. If the prediction interval is 40ms, then the predictive motion tracking will predict the position, orientation, and motion of the headset at the 40ms mark. The prediction interval of zero means that the predictive motion tracker predicts the position, orientation and motion of the headset at the moment. This effectively eliminates predictive motion tracking. The prediction interval allows the predictive motion tracking system dynamically to respond to headset movements of any size or scope.

This prediction interval could be very small for small movements. A prediction interval of 5 milliseconds may be used for movements that are less than 10 degrees per second. These situations are where the angular velocity measurements, regardless of their method, are applied for a time period and used in lieu of actual data to provide a user system with the ability to render the associated video frames. Based on the prediction, the next frame of rendered footage is produced.

“In cases where the angular velocity measurement (e.g. 500 degrees/second) is very large, the prediction interval within which the system predicts movement is increased. The system will set the prediction interval between 30 and 60 microseconds for fast movements, as an example. These situations are where the motion prediction data is passed to a renderer. It predicts the wearer’s position, orientation, and motion at the end the prediction interval. The renderer then delivers the next frame of video based on that prediction.

“The latency interval is well-known in well-studied systems. “Latency interval” is a term that refers to the time between when motion data is generated by sensors and when it is delivered to the renderer. The latency interval is the time between the moment motion data is generated by sensors and the delivery of a rendered video frame incorporating that motion data. The latency interval in the current system is approximately 40 milliseconds.

“This latency interval can be used to set an upper limit for the prediction interval. The prediction interval should not exceed the latency interval. Because the headset’s head movement causes the next rendered frame of video to be rendered at the next latency interval.

“Accordingly, it is not possible to predict beyond this time frame and more likely will result in overshoot (rendering a too large movement or a stop when movement was just starting) and disconnect from actual wearer movements. In the same way, additional data can also be made available in the described system, including tens of additional position, motion, and orientation data points that can all be used to render next frame of video. Predicting motion to the latency interval can be used to attempt to predict user movements with the next frame of rendered video.

The process could work in a way that determines the angular velocity of wearer movement. The time for prediction increases linearly from the threshold rate of 0.3 radians/second to the latency interval. For example, a prediction interval increase of +2 milliseconds per radion per second could be considered. A headset that turns at 1 radians per sec would use a prediction interval of 2 miliseconds in the future. A headset that turns at 10 radians per seconds would use a prediction interval 20 milliseconds into the future. You can also use linear increase slopes, with a maximum prediction interval equaling the latency interval (e.g. “40 milliseconds into future

If the angular velocity (e.g. The angular velocity is 0.03 to 0.03 radians/second. In order to determine the orientation in three dimensions, smoothing is used. It relies on the current position and orientation and does not predict. Smoothing is disabled if the angular velocity exceeds substantially zero. The position, or place in three-dimensional space, is calculated using the last measured position and the linear velocity over the chosen prediction interval.

To summarize, movement (angular velocity of the headset) is measured to determine if it is substantially zero (0 to 0.33 radians per sec). Smoothing is activated and the prediction interval is set either to substantially zero or zero. Smoothing is disabled if the headset moves at an angular velocity greater than substantially zero (greater that 0.3 radians/second), and the prediction interval is then set. If the headset is moving at an increasing angular velocity, the prediction interval will rise by a prediction interval variance (e.g. 2 milliseconds/radian per second per reading up until a maximum latency interval (e.g. 40 milliseconds).

The linear increase in prediction interval (up until the maximum latency interval) could be linear. This corresponds to linear movement of a headset. Therefore, faster movement can be compensated by a linear increase in the prediction interval. A smoothstep algorithm can also be used. Smoothstep algorithms take the generalized form smoothstep(t)=3t2?2t3 where t is the variable to be smoothed and the results of the generalized form fall between 0 and 1. Smoothstep is sloped towards zero at the high and low ends. This allows Smoothstep to be rounded off, which is a different method from linear interpolation. The ends. Wearers of such headsets may believe that their prediction interval is slowing approaching the maximum or minimum, without abrupt stops or starts at either the beginning or end. The prediction interval can be?smoothly’. The maximum latency interval can be approached or it can approach zero.”

Smoothing can be activated at 860 if the headset stops turning at 825. This indicates that the user’s head remains substantially still. Smoothing is most useful when the headset isn’t moving or making small movements, as discussed above. It helps avoid excessive or exaggerated movements of the scene such as jitter when a headset remains substantially still. These issues can be reduced by allowing smoothing to occur when the headset is substantially still.

“Similarly, at 870, the prediction interval has been set to nearly zero. If the head is stationary, it is impossible to make an aggressive prediction for any time period. Although some predictive motion tracking is possible, it is limited to a short time period, like 5 milliseconds. The prediction interval can be set to zero in some cases. This helps avoid predictive motion tracking which may exaggerate small movements.

“Next, the headset will be able to tell whether it is turning at 825. After setting the prediction interval at 870 and 850, the prediction of an orientation and angular velocity at 880 will be made. This predicts the position, orientation, and motion of the headset at the beginning of the prediction interval. This can be expressed as a quaternion as mentioned above. However, it may also be expressed in any other way.

Finally, the display is updated at each rendered frame to correspond with the predicted position, orientation, motion based on the prediction interval at 890. If the prediction interval was 40ms, then the VR headset’s next rendered frame will render as though the wearer’s head is in a predetermined position 40ms away. If the prediction interval was zero, then the VR headset will render the next rendered frame as though the wearer’s head is in a predetermined position. Actual data, however, is not required. This means that there is no prediction, or even a very limited prediction.

“Both these options and extremes indicate that the head is turning at 825 or not. The process is over once the display has been updated. The process is repeated for each sensor update (or for any interval of sensor updates) as explained above. Every 10 to 20 sensor updates. This allows the rendering of the next frame of rendered footage to be done with smoothing or with no smoothing, and with or with out a prediction interval at, or near, a latency interval.

“Closing Comments”

“The embodiments and examples presented in this description should not be interpreted as limitations or restrictions on the disclosed and claimed apparatuses and procedures. While many examples herein are specific combinations of system elements or method acts, it is important to understand that these elements and acts can be combined in different ways to achieve the same goals. You can add or subtract steps to flowcharts. The steps shown in this document may also be combined with other steps. The acts, elements and features described only in relation to one embodiment do not exclude others from playing a similar role.

“As used herein, ?plurality? It can refer to two or more. A?set? is a group of items. A?set? may be a collection of one or more items. The terms ‘comprising? and ‘including? are used in this claim. Only the transitional phrases “consisting of” and “consisting essentially of?” are closed or semi-closed transitional phrases. The transitional phrases?consisting essentially? and?first? are respectively closed or semi-closed phrase with respect to claims. The use of ordinal terms like?first? or?second? in claims to modify claim elements does not in itself indicate any priority, precedence or order of claim elements over others or the temporal order by which acts of a methodology are performed. They are used only as labels to differentiate one claim element with a particular name from another element with a similar name (but for the use of the ordinal phrase) to distinguish claim elements. As used herein, ?and/or? The expression?and/or? means that the items listed are alternatives to the items, but that all options can be used in addition to the items listed.

Click here to view the patent on Google Patents.

How to Search for Patents

A patent search is the first step to getting your patent. You can do a google patent search or do a USPTO search. Patent-pending is the term for the product that has been covered by the patent application. You can search the public pair to find the patent application. After the patent office approves your application, you will be able to do a patent number look to locate the patent issued. Your product is now patentable. You can also use the USPTO search engine. See below for details. You can get help from a patent lawyer. Patents in the United States are granted by the US trademark and patent office or the United States Patent and Trademark office. This office also reviews trademark applications.

Are you interested in similar patents? These are the steps to follow:

1. Brainstorm terms to describe your invention, based on its purpose, composition, or use.

Write down a brief, but precise description of the invention. Don’t use generic terms such as “device”, “process,” or “system”. Consider synonyms for the terms you chose initially. Next, take note of important technical terms as well as keywords.

Use the questions below to help you identify keywords or concepts.

• What is the purpose of the invention Is it a utilitarian device or an ornamental design?
• Is invention a way to create something or perform a function? Is it a product?
• What is the composition and function of the invention? What is the physical composition of the invention?
• What’s the purpose of the invention
• What are the technical terms and keywords used to describe an invention’s nature? A technical dictionary can help you locate the right terms.

2. These terms will allow you to search for relevant Cooperative Patent Classifications at Classification Search Tool. If you are unable to find the right classification for your invention, scan through the classification’s class Schemas (class schedules) and try again. If you don’t get any results from the Classification Text Search, you might consider substituting your words to describe your invention with synonyms.

3. Check the CPC Classification Definition for confirmation of the CPC classification you found. If the selected classification title has a blue box with a “D” at its left, the hyperlink will take you to a CPC classification description. CPC classification definitions will help you determine the applicable classification’s scope so that you can choose the most relevant. These definitions may also include search tips or other suggestions that could be helpful for further research.

4. The Patents Full-Text Database and the Image Database allow you to retrieve patent documents that include the CPC classification. By focusing on the abstracts and representative drawings, you can narrow down your search for the most relevant patent publications.

5. This selection of patent publications is the best to look at for any similarities to your invention. Pay attention to the claims and specification. Refer to the applicant and patent examiner for additional patents.

6. You can retrieve published patent applications that match the CPC classification you chose in Step 3. You can also use the same search strategy that you used in Step 4 to narrow your search results to only the most relevant patent applications by reviewing the abstracts and representative drawings for each page. Next, examine all published patent applications carefully, paying special attention to the claims, and other drawings.

7. You can search for additional US patent publications by keyword searching in AppFT or PatFT databases, as well as classification searching of patents not from the United States per below. Also, you can use web search engines to search non-patent literature disclosures about inventions. Here are some examples:

• Add keywords to your search. Keyword searches may turn up documents that are not well-categorized or have missed classifications during Step 2. For example, US patent examiners often supplement their classification searches with keyword searches. Think about the use of technical engineering terminology rather than everyday words.
• Search for foreign patents using the CPC classification. Then, re-run the search using international patent office search engines such as Espacenet, the European Patent Office’s worldwide patent publication database of over 130 million patent publications. Other national databases include:
• European Patent Office (EPO) provides esp@cenet to access a network of Europe’s patent databases with access to machine translation of European patents. 
• Japan Patent Office (JPO) – with access to machine translations of Japanese patents.
• World Intellectual Property Organization (WIPO) offers PATENTSCOPE with a full-text search of published international patent applications and machine translations for some documents, as well as a list of international patent databases.
• Korean Intellectual Property Rights Information Service (KIPRIS)
• State Intellectual Property Office (SIPO) with machine translation of Chinese patents.
• Other International Intellectual Property Offices with online patent databases include Australia, Canada, Denmark, Finland, France, Germany, Great Britain, India, Israel, Netherlands, Norway, Sweden, Switzerland, and Taiwan.
• Search non-patent literature. Inventions can be made public in many non-patent publications. It is recommended that you search journals, books, websites, technical catalogs, conference proceedings, and other print and electronic publications.

To review your search, you can hire a registered patent attorney to assist. A preliminary search will help one better prepare to talk about their invention and other related inventions with a professional patent attorney. In addition, the attorney will not spend too much time or money on patenting basics.

Download patent guide file – Click here