Metaverse – James M. Powderly, Savannah Niles, Frank Hamilton, Marshal A. Fontaine, Rony Abovitz, Alysha Naples, Magic Leap Inc

Abstract for “Selecting virtual objects within a three-dimensional space”

“Systems and methods of interfacing with virtual objects in three-dimensional space using wearable systems are disclosed. Programming the wearable system to allow user interaction with interactable objects within a user’s field of view (FOR) can be done. The FOR is a part of the environment that can be perceived by the AR system. The system can identify a set of interactable objects within the FOR of the user, and then determine the pose of the user. Based on the user’s pose or field of view (FOV), the system can update a subgroup from interactable objects located within the FOV. The system will then select a target interactable object in that subgroup. The system can trigger a selection event for the target interactable item.

Background for “Selecting virtual objects within a three-dimensional space”

Modern computing and display technology have made it possible to create systems that can be called “virtual reality”, “augmented reality?”, or “mixed reality?” Digitally reproduced images and portions of them are presented to users in a way that makes them appear real or can be perceived as such. A virtual reality (or?VR?) scenario usually presents digital or virtual information without transparency to any other real-world visual input. An augmented reality (or?AR?) scenario typically presents digital or virtual information as an enhancement to the visualization of the real world around the user. A mixed reality (or?MR?) scenario involves merging real and digital worlds to create new environments in which physical and virtual objects coexist and interact in real-time. It turns out that the human visual system is complex. Therefore, it is difficult to create a VR, AR or MR technology that allows for a natural-feeling, rich presentation and interaction of virtual images elements with real-world or virtual imagery elements. The systems and methods described herein address different challenges in VR, AR, and MR technology.

“Some embodiments disclose a system that allows the user to change their input mode on a wearable device. The system may include a display system for the wearable device that presents a three-dimensional (3D), view to the user. This view can contain interactable objects. A user input device is configured to receive user input. A sensor is configured to collect data about the pose of the user. Finally, there’s a hardware processor communicating with the user input device. The hardware processor can be programmed in order to determine whether the current user input method for interfacing with the interactable objects is either a first or second user input mod. In this case, the first mode relies at least partially on the pose of the user while the second mode relies at least partly upon the input of the user. If the current user interface mode is determined to be the first, the hardware processor can use the sensor to monitor the user’s pose and present via the display system, based at minimum partly on that pose, a focus indicator in the first shape associated to the first user mode in a direction related the user’s pose; receive a first indication of switching to the second mode; and then switch the current input mode to the second mode in response to the first indication. If the current user mode is determined to be the second, the hardware processor can monitor user input using the sensor; present via display system, based at minimum partly on the monitored pose, the focus indicator in a second shape associated to the second user mode; receive a second signal to switch to that mode; and then switch the current input mode to that mode in response to the second indication.

“In certain embodiments, there is a method of changing the user input mode of a wearable device. This method can be carried out under the control of a wearable device with a processor. A wearable device may be designed to allow interaction between interactable objects and users in a field. The FOR is a section of the environment that can be perceived by the user through a display system. The method involves: determining the pose of a person; displaying via the display a first indicator associated to a target interactable item in a direction that is related to the user’s pose; receiving a selection from the target interactable objects; displaying via the system the plurality virtual objects; updating the second indicator via the display device based at most partly on user input.

“In some embodiments, the disclosure includes a wearable system and a method of selecting a virtual object in a three-dimensional (3D), space. The wearable system may include a display system that displays virtual objects in 3D space, a non-transitory storage data store for interactable objects in 3D space, a sensor to determine the pose of the user, and a processor programed to communicate with both the display system, data store and the sensor. The method and wearable system can detect the pose and data from the sensor. They also determine a field view (FOV), which is a section of the environment that the user perceives at any given time. They identify a group interactable objects within the FOV and identify the target object.

“Details about one or more implementations are provided in the accompanying drawings as well as the description. The claims, drawings, and description will reveal other features, aspects, or advantages. This summary and the detailed description below do not attempt to limit or define the scope of the inventive subject matter.

“Overview”

A virtual interface allows users to select and target objects in three-dimensional space. A user could select a virtual object by using body poses, such as touching, touching, or physically approaching the objects. You can also point at the virtual objects and click on them with virtual beams or rays. These techniques can lead to fatigue, and it can be difficult for the user to select objects accurately.

The present disclosure gives examples of wearable devices that can address any or all of these issues. One example is that the user might move his head to look at a group object. To transport the highlighted object from one object to the next, the user can use a user input device (e.g. by tapping on a touchscreen) or move his head to highlight another object. You can confirm that the target object has been selected by activating the input device once more (e.g. Touch the touchscreen to confirm. The AR user interface can allow the user to perform additional actions upon the target object once it has been selected (e.g. display or select from a menu, perform an action related to a game in which the target object appears, etc.). This is a great way to select the objects you are interested in, while also reducing fatigue. Because head poses can be difficult to control accurately, this is why it’s so useful. The wearable system will identify objects of interest based upon the user’s head position and, in the meantime allow the user to select objects using hand gestures.

“Some implementations allow for multiple virtual objects to be interacted with an interactable object. A virtual user interface plane could include multiple virtual apps, such as a video streaming app, a virtual classroom, a weather app, a game, or an astronomy program. Based on the properties of the interactable object, the wearable system can support different user input methods. The wearable system might allow users to interact with interactable objects by using poses, for example, if they are large user interface planes. The wearable system might instead choose the user input mode as the default input mode for small objects to enable precise interaction with them. These options may prove to be more advantageous as moving and targeting large objects may require less precision, while selecting and moving small objects may require precision.

“The context information can be used by the wearable system to determine the user’s input mode. The wearable system can, for example, determine the location of virtual objects within the user’s environment. The wearable system can detect a dense clustering of virtual objects along the user’s gaze direction and give the user the ability to switch from head control to hands. The user can interact more precisely with virtual objects this way. Another example is that the AR system might detect the orientation of an object (e.g. vertical or horizontal) to provide the appropriate interactions for the users (e.g. volume controls for a TV application that appears vertically in front of the user, or typing controls for virtual keyboards that appear horizontally on the user’s desk).

“The wearable device can allow a user to share virtual contents with others (also wearing wearable systems), for example by sharing a world map of their environment or by communicating the virtual information (or updates to it) among the wearable devices via a network.

“Examples 3D Display”

“FIG. FIG. 1 shows an illustration of a mixed-reality scenario, with certain virtual objects and some physical objects that can be viewed by a human. FIG. FIG. 1. A MR scene 100 depicts a user of MR technology. The scene shows a person imagining a park-like setting 110 with people, trees, buildings, and a concrete platform 120. The MR technology user also perceives these items. A robot statue 130 stands upon the real-world platform 120. A cartoon-like avatar character 140 is flying past, which appears to be the personification of a Bumble Bee.

It may be beneficial for 3D displays to have an accommodative response that corresponds to the virtual depth of each point. The human eye can experience accommodation conflicts if the accommodative response of a display point doesn’t correspond to its virtual depth as determined by stereopsis and binocular depth cues. This could lead to unstable imaging, headaches, and even complete absence of surface depth.

Display systems that provide images corresponding to a variety of depth planes to viewers can offer VR, AR, and MR experiences. Each depth plane may have different images. This allows the viewer to see depth cues by observing differences in image features or the accommodation required to bring them into focus. These depth cues, as discussed in the past, provide credible perceptions about depth.

“FIG. “FIG. 2” illustrates an example wearable system 200. The wearable system 200 comprises a display 220 and various electronic and mechanical modules that support display 220. Display 220 can be connected to frame 230. This frame is usable by the user, wearer or viewer 210. The display 220 may be placed in front of the user’s eyes 210. The display 220 can show AR/VR/MR content. A head-mounted display (HMD), which is attached to the display 220, can be used. A speaker 240 can be attached to the frame 230 in some embodiments and placed next to the user’s ear canal. (In some embodiments, an additional speaker, not shown here, is placed adjacent to the user’s other ear canal to allow for stereo/shapeable sounds control).

“The wearable device 200 may include an outward-facing imaging unit 464 (shown at FIG. 4) that observes the environment around the user. A wearable system 200 may also include an inward-facing image system 462 (shown at FIG. 4), which can track the eye movements. Inward-facing imaging systems can track the movements of one or both eyes. The frame 230 may have an inward-facing image system 462 attached. It may also be electrically connected to the processing modules 265 and 270. These modules may process image data acquired by the inward facing imaging system to determine the pupil sizes or orientations, eye movements, or user’s eye position 210.

“As an example the wearable device 200 can use either the outward-facing or inward-facing imaging systems 464 to capture images of the user’s pose. Images can be still images, frames, or combinations of images.

“The display 220 may be operatively coupled 250 such as by wired connectivity or wireless connectivity to a local processing module 260. It can be fixedly attached 230 to a helmet/hat worn by the user, embedded into headphones or other removable attachments to the user 210 (e.g. backpack-style configuration or belt-coupling style configuration).

“The local processing module and data module 265 may include a hardware processor as well as digital memories such as flash memory and non-volatile memory. Both of these may be used to aid in data processing, caching, storage, and processing. Data may include data from sensors (which could be, e.g.., operatively coupled with the frame 230, or otherwise attached to user 210), as well as microphones, inertial measuring units (IMUs), compasses and global positioning system units (GPS), radio devices or gyroscopes. Alternatively, data can be acquired using remote processing module (270) or remote data repository (280) for passage to display 220. Local processing and data modules 260 can be operatively connected via communication links 262 and 264 to remote processing module (270) or remote data repository (280), so that remote modules are available for local processing and data 260. Remote processing module 280 or remote data repository 282 may also be operatively coupled.

“In some cases, the remote processing module (270) may include one or more processors that are capable of processing data and/or images. The remote data repository 280 could include a digital storage facility that can be accessed via the internet or another networking configuration. Resource configuration. Some embodiments store all data and perform all computations in the local processing module. This allows for fully autonomous use from remote modules.

The human visual system is complex and it is difficult to perceive depth accurately. It is possible that viewers may perceive an object as three-dimensional due to vergence and accommodation. The movement of vergence (i.e. rolling of the pupils towards or away from one another to converge the lines in the eyes to fixate on an object) is closely related to focusing (or?accommodation?). The lenses of the eyes. Under normal conditions, changing the focus of the lenses of the eyes, or accommodating the eyes, to change focus from one object to another object at a different distance will automatically cause a matching change in vergence to the same distance, under a relationship known as the ?accommodation-vergence reflex.? Under normal conditions, a change of vergence will also trigger a matching change to accommodation. Display systems that match accommodation and vergence better may create more comfortable and realistic simulations of three-dimensional imagery.

“FIG. 3. illustrates aspects of a method for simulating three-dimensional imagery using multiple depth plans. Referring to FIG. FIG. The eyes 302 & 304 use particular accommodated states to focus objects at different distances along z-axis. A particular accommodated condition may be associated with one of the depth planes 306, and has an associated focal distance. This means that objects or parts of objects in particular depth planes are in focus when the eye moves in that accommodated state. Three-dimensional imagery can be created in some embodiments by providing different images for each eye 302 and 304 and different versions of each image corresponding to each depth plane. Although the fields of view are shown separately for illustration purposes, it is possible for the eyes 302 or 304 to overlap as the distance along the Z-axis increases. For illustration purposes, the contours of depth planes are shown flat. However, it is possible for them to curve in space so that all of the features of the depth planes are focused with the eye in an accommodated state. The theory does not limit the possibility that the human eye can perceive depth in a finite number depth planes. Therefore, it is possible to create a convincing simulation of depth perception by showing the eye different images that correspond to each of the limited depth planes.

“Waveguide Stack Assembly”

“FIG. “FIG. 4 shows an example of a waveguide stack that outputs image information to a user. Wearable system 400 may include a stack of waveguides or stacked waveguide assemblies 480. These waveguides can be used to provide three-dimensional perception for the eye/brain by using a plurality waveguides 432, 434, 436, 436, 438, 438, 438, 438, 438, 438, 400 b. The wearable system 400 could be compared to the 200 in FIG. 2 with FIG. 4, schematically showing more parts of the wearable system 200. In some embodiments, for example, the waveguide assembly (480) may be integrated into FIG. 2.”

“With reference to FIG. 4. The waveguide assembly 480 could also contain a number of features 458, 456, 454, or 454, 452, between the waveguides. The features 458, 456, 454, and 454, 452, may be lenses in some embodiments. Other embodiments may not allow for the features 458, 456, 454 or 454, 452 to be lenses. They may be spacers, such as cladding layers, or structures that form air gaps.

Waveguides 432, 434, 436, 438, 438, 438, 440 b and the plurality lenses 458, 456, 436, 438, 438, 438, 438, 438, 438, 438, 438, 438, 440 b may be set up to transmit image information to the eye at different levels of light ray divergence or wavefront curvature. Each waveguide level can be associated with a specific depth plane, and may be configured so that it outputs image information that corresponds to that depth. The image injection devices 420 to 432, 424, 426 and 428 can be used to inject image information into waveguides 440, 438, 436, 434, 434, 432, 428. Each of these devices may be designed to distribute light across the waveguides, for output towards the eye 410. The light exits from an output surface of image injection devices 420-422, 424-426, 428, 428 and is injected into the waveguides’ corresponding input edges 440 b to 438 b. 436 b. 434 b. 432 b. One beam of light, such as a collimated beam, may be injected into each waveguide in order to produce an entire field of collimated collimated beams directed towards the eye 410 at specific angles and divergence corresponding to the depth plan associated with that waveguide.

“In some embodiments the image injection devices 422, 424 and 426 are discrete displays that produce image information to be injected into a waveguide 440, 438, 436, 434, 434, 432, and 432 b. Other embodiments include the image injection device 420 to 422, 424 and 426. These devices may be output ends of a single multiplexed monitor that can pipe image information through one or more optical conduits (such fiber optic cables) into each of the image injectors 420, 422, 422, 424 and 426.

“A controller 460 controls operation of the stacked wafer assembly 480 and image injection devices 424, 426. 428. The controller 460 contains programming (e.g. instructions in a nontransitory computer-readable media) that controls the timing and provision image information to waveguides 440, 438, 436, 436, 434, 434, 434, 432, 432 b. The controller 460 can be either a single integrated device or a distributed system connected via wired or wireless communication channels. The controller 460 could be part of one or more processing modules 260 and 270 (illustrated at FIG. 2) in certain embodiments.

“The waveguides 438 b and 436 b can be set up to transmit light through total internal reflection (TIR) within their respective waveguides. Waveguides 440, 438, 436, 434, 434, 432 and 432 b can be either planar or curved, with major top- and bottom surfaces, and edges that extend between them. The illustrated configuration of the waveguides 440, 438, 436, 434, and 432 B may all include light extracting optical components 440a,436 b., 436, b., 434, b., 432, b. These elements are designed to extract light from a waveguide by redirecting light within the waveguide, out of each waveguide to provide image information to the eye 434 a., 432a. Outcoupling optical elements and extracted light can also be called outcoupled. The waveguide outputs a beam of light at the locations where the light from the waveguide strikes a light redirecting component. For example, light extracting optical elements (440a, 438a, 436, 434 a., 432a, 442 a., 432) may be reflective or diffractive optical components. For ease of description and drawing, the light extracting optic elements are shown at the bottom of the waveguides. The light extracting optical element 440a, 438a, 436, 434a, and 432a can be formed in a layer made of material attached to a transparent substrate. This will form the waveguides. Other embodiments allow the waveguides 440, 438, 436, 434, b and 432 to be formed in a single piece of material. The light extracting optical element 440a,438 a., 436, 436, 434, b., 432b. may also be formed on a surface, or within that piece.

“With reference to FIG. “With continued reference to FIG. The waveguide 432b closest to the eye could be set up to deliver collimated lighting, such as that injected into waveguide 432b to the eye 410. The collimated light could be representative of optical infinity focal plan. The next waveguide may send out collimated beams through the first lens 452 (e.g. a negative lens) before reaching the eye 410. The first lens 452 could be set up to create a convex wavefront curvature, so the eye/brain perceives the light coming from the next waveguide 434 b as coming in from a first focal point closer inward towards the eye 410 from optical infinite. The third up waveguide 436b also passes its output light through the second lens 454 and first lens 452, before reaching the eye. Combining the optical power of the first two lenses 452 and 444, may create an incremental amount of wavefront curve so that the eye/brain perceives light coming out of the third waveguide 436 B as coming from a second focal point that is closer inward towards the person than light from the next waveguide, 434 b.

The other waveguide layers (e.g. waveguides 438b, 440b) and lenses (e.g. lenses 456, 458) are similarly configured. The highest waveguide, 440b in the stack, sends its output through all lenses between it, the eye, for an aggregate focal power that is representative of the nearest focal plane to the person. A compensating lens layer (430) may be placed at the top to offset the effect of the lens stack 458, 456, 454 and 452 on viewing/interpreting light from the world 470. This configuration can provide as many perceived focal points as possible, regardless of the number of waveguide/lens pairs. The light extracting optical elements and the focusing parts of the lenses can be either static or not dynamic. Alternate embodiments may include one or both of these elements, with electro-active features.

“With reference to FIG. “With continued reference to FIG. 4, the light extracting optic elements 440a, 438a, 436, 436a, 434a, 443, 434a, and 432a can be configured to redirect light out of respective waveguides and to produce this light with the correct amount of divergence/colimation for the particular depth plane associated to the waveguide. Waveguides with different depth planes can have different configurations for light extracting optical elements. These elements may output light with different amounts of divergence or collimation depending on the depth plane. As discussed above, light extracting optical element 440 a and 438 a are volumetric or surface features that can be set up to produce light at certain angles. The light extracting optical components 440 a. 438 a. 436 a. 434 a. 432 a. 432 a. may be volume or surface holograms and/or diffraction gratings. U.S. Patent Publication No. 2015/0178939 published June. 25th of June 2015, is included by reference in its entirety.”

“In certain embodiments, light extracting optical elements (440 a), 438 a. 436 a. 436 a. 434 a. 432 a. 432 a) are diffractive features that create a diffraction patterns or?diffractive optic element?. Also known as a “DOE” The DOE should have a low diffraction efficiency, so that only a small portion of the beam is deflected toward the eye (410 at each intersection). The rest of the beam continues to travel through the waveguide via total internal reflect. This allows for the division of the light carrying image information into several related exit beams, which exit the waveguide at a variety of locations. The result is a relatively uniform pattern of exit emission towards the eye 304 in this particular collimated beam that bounces around within a waveguide.

“In some embodiments, one of the DOEs can be switched between?on and?off? state in which they actively diffract and?off?” They do not significantly diffract. A switchable DOE could be a layer of polymer-dispersed liquid crystal. The microdroplets may be arranged in a diffraction pattern in the host medium. In this case, the refractive indice of the microdroplets can either be changed to match the host material’s or to an index that is not as high as the host medium’s (in which case they actively diffract incident sunlight).

“In some embodiments the distribution and number of depth planes (or depth of field) may be dynamically adjusted based on the pupil sizes of the viewers eyes. The size of a viewer’s pupils may affect the depth of field. The depth of field may increase as the size of the pupil decreases. This means that a plane that isn’t discernible due to its location being beyond the depth-of-focus of the eyes can become more discernible. With a reduction in pupil size, and in proportion with an increase in depth, the field depth will also change. The decreased pupil size may also affect the number of depth planes that are used to show different images to the viewers. A viewer might not be able see details of both a depth plane and another depth plane at the same pupil size. This could lead to confusion. However, these two depth planes can be in focus simultaneously to the user at a different pupil size.

“In certain embodiments, the display system can alter the number of waveguides that receive image information based on pupil size and orientation or upon receiving signals indicative of particular pupil sizes or orientations. If the user is unable to distinguish between the depth planes of two waveguides, the controller 460 can be programmed or configured to stop providing image information to that waveguide. This may increase the system’s responsiveness and reduce the system’s processing load. If the DOEs of a waveguide can be switched between the on or off states, they may be switched to off when the waveguide receives image information.

“In certain embodiments, it might be desirable for an exit beam to meet the condition that its diameter is less than the diameter the viewer’s eye. This condition can be difficult due to the variable size of viewers’ pupils. This condition can be met in some embodiments by changing the size and shape of the exit beam according to the size of the pupil. The size of the exit beam could decrease as the pupil grows, for example. In certain embodiments, the exit beam size can be adjusted using a variable aperture.

The wearable system 400 may include an outward-facing image system 464 (e.g. a digital camera) that captures a portion the world 470. This area of the world 470 is often referred to by the term “field of view” (FOV), and the 464 imaging system is sometimes called an FOV camera. The field of regard (FOR) is the entire area that is available for viewing and imaging by a viewer. There may be 4 FOR. The FOR may include 4? Other contexts may require the wearer to move more tightly, so the FOR of the wearer may have a smaller solid angle. Images taken with the outward-facing imaging device 464 can be used for tracking gestures (e.g. hand or finger gestures), detecting objects in the world, 470, and so on.

The wearable system 400 may also include an inward facing imaging system 466 (e.g. a digital camera) that monitors the movements of the user such as their eye movements and facial movements. To determine the size and/or orientation the pupil of the eyes 304, the inward-facing imaging device 466 can be used. Inward-facing imaging system 466 may be used to capture images that can be used for biometric identification (e.g. via iris identification) or to determine the user’s gaze direction. One camera may be used for each eye in some embodiments. This allows each eye to determine its pupil size and position independently. Each eye can then be presented with image information that is dynamically tailored to it. Other embodiments allow for the determination of the pupil diameter and orientation of one eye 410. This allows the presentation of image information to each eye to be dynamically tailored. The inward-facing imaging device 466 can analyze the images to determine the user?s mood or eye position. This information can then be used by the wearable technology 400 to determine which audio or visual content to present to the user. Wearable system 400 can also detect head pose (e.g. head position or orientation) by using sensors such IMUs, accelerometers and gyroscopes.

The wearable system 400 may include a user input device 466 through which the user can input commands 460 to the controller to interact with the wearable systems 400. The user input device 466 may include a trackpad and touchscreen, as well as a joystick, multiple degree of freedom (DOF), controller, capacitive sensing devices, a game controller (D-pad), keyboard, mouse, a directional pad, a wand, and a haptic device. It can also function as a virtual user interface device (e.g., a totem). Sometimes, the user may press or swipe with a finger (e.g. a thumb) on a touch sensitive input device to input the wearable systems 400. This could be to provide user input to the user interface provided by wearable system 405. During the wearable system 400, the user may hold the user’s input device 466 in their hand. The user input device 466 may be wired or wirelessly connected to the wearable system 400.

“FIG. “FIG.5” 5 illustrates an example of exit beams produced by a waveguide. Although one waveguide is shown, it can be seen that many waveguides within the waveguide assembly480 could function in a similar fashion. The waveguide 432b is infected at the input edge 432c of the waveguide 432, and the light propagates through the waveguide 422, b via TIR. A portion of the light 520 exits the waveguide as the exit beams 510 at points where it impedes on the DOE 432. Although the exit beams 510 appear to be substantially parallel, they can also be directed to propagate to eye 410 at an angle (e.g. forming divergent exit beacons), depending upon the depth plane associated to the waveguide 432b. You will see that the exit beams 510 are substantially parallel. This could be a sign of a waveguide that uses light extracting optical elements to outcouple light, forming images that appear to be set at a deep plane that is far away (e.g. optical infinity). Waveguides with other light extracting optical elements or waveguides may produce a divergent exit beam pattern. This would require the eye to adjust to a greater distance to focus the image on the retina. The brain would interpret this as light coming from a farther distance than the eye 410.

“FIG. “FIG. An optical system may include a waveguide device, an optical coupler subsystem that optically couples light to or from waveguide apparatus, as well as a control subsystem. Multi-focal volumetrics, images, and light fields can all be generated by the optical system. One or more primary planar wavesguides 632a can be included in the optical system (only one is shown at FIG. 6) and one (or more) DOEs 632b that are associated with at least one of the primary waveguides (632 a). The planar waveguides 632b can be identical to those waveguides 432b, 434b, 436b, 436b, 438b, and 438b. 4. An optical system could use a distribution waveguide device to transmit light along a first direction (vertical or the Y-axis, as shown in FIG. 6.) and increase the light’s effective escape pupil along the first direction (e.g., the Y-axis). Distribution waveguide apparatus could include, for instance, a distribution waveguide 622b and at most one DOE 622a (illustrated with double dash-dot lines) that is associated with the distribution waveguide 622b. The distribution planar wavesguide 622b could be identical or similar to the primary planar waveguide 642 b in at least some aspects, but with a different orientation. The DOE 622 a could be identical or similar to the DOE 632. The distribution planar waveguide 622b and DOE 622a could be made of the same materials, or the primary planar wavesguide 632b or DOE 622 a, for example. FIG. 6 shows embodiments of the optical display 600. 6 can be integrated in the wearable system 200 as shown in FIG. 2.”

“The relayed or exit-pupil extended light may be optically coupled to the distribution waveguide apparatus into one or more primary plane waveguides 632b. The primary planar waveguide 632b can relay light along another axis, but it should be orthogonal to the first axis (e.g. horizontal or X-axis as shown in FIG. 6). The second axis may be non-orthogonal to the first. The primary planar waveguide 632b expands light’s exit pupil along the second axis (e.g. X-axis). The distribution planar wavesguide 622 b, for example, can expand and relay light along the vertical axis. It then passes that light on to the primary planar waveguide 642, which can expand and relay light along its horizontal axis.

“The optical system can include one or more sources for colored light (e.g. red, green and blue laser light)610, which may be optically connected to a proximal tip of a single-mode optical fiber 640. The distal end may be threaded through or received through a hollow tube 642 made of piezoelectric materials. As a flexible, fixed-free cantilever 644, the distal end protrudes out of the tube 642 Four quadrant electrodes can be attached to the piezoelectric tube 642. For example, the electrodes can be plated on the outer, outer, or outer periphery of the tube 642. Unillustrated core electrodes may be found in the core, inner periphery, or inner diameter of the tube 642.

“Drive electronics 655, for example, electrically coupled via wires 665, drive opposing electrodes to bend piezoelectric tube 642 in 2 axes independently. Mechanical modes of resonance are found at the distal tip 644 of the optical fibre. The frequency of resonance depends on the diameter, length, as well as the material properties of optical fiber 644. The fiber cantilever 644 vibrates when the piezoelectric tube 642 is vibrated near the first mode of mechanical resonance. It can also sweep through large deflections by vibrating the 642.

“By stimulating resonance vibration in two directions, the tip 644 of the fiber cantilever is scan biaxially in an area that fills two-dimensional (2D). scan. An image can be created by modulating the intensity of light source(s), 610, in sync with the scan of fiber cantilever 644. U.S. Patent Publication No. 2014/0003762 is incorporated herein in its entirety.

“A component of an optical subsystem that can collimate light from the scanning fibre cantilever 644 can be called an optical coupler subsystem. Mirror surface 648 can reflect the collimated light into the narrow distribution plane waveguide 622b, which contains at least one diffractive optic element (DOE 622a). Relatively to FIG. 6) along the distribution plane waveguide 622b by TIR and intersects repeatedly with the DOE 622a. The DOE 622a is preferred to have a low coefficient of reflection. This can result in a fraction (e.g. 10%) of light being diffracted towards an edge of the primary planar waveguide 642 b at each intersection with the DOE622 a and a fraction to continue its original trajectory along the length of distribution planar waveguide 642 b via TIR.

“At every intersection with the DOE 622a, additional light may be diffracted towards the primary waveguide 632b. The DOE 4 can expand vertically the exit pupil by dividing the incoming beam into multiple outcoupled sets. This vertically expanded light can be coupled out of distribution waveguide 622b to enter the edge the primary planar wavesguide 632b.

“Light entering the primary waveguide 632b can propagate horizontally (relatively to FIG. 6) along primary waveguide 632b via TIR. The light crosses DOE 632a at multiple points and propagates horizontally along at most a portion the primary waveguide 632b via TIR. To produce deflection and focus of light, the DOE 632a can be advantageously designed or configured with a phase profile. This is a combination of a linear and radially symmetric diffractive patterns. The DOE 632a may have a lower diffraction efficiency (e.g. 10%) so that only a small portion of the beam is deflected towards the eye with each intersection of DOE 632a, while the rest propagates through the primary waveguide 632b via TIR.

“A fraction of the light is scattered toward the face of primary waveguide 632b at each intersection of the propagating light with the DOE 632a. This allows the light to escape from the TIR and emerge from primary waveguide 642 b. The DOE 632a radially symmetric pattern of diffraction imparts an additional focus level to diffracted light. This allows the light wavefront to be shaped (e.g., giving a curvature) and the beam to be directed at an angle that matches the intended focus level.

These different paths can result in the light being coupled out of the primary plane waveguide 632b by a multiplicity OFEs 632a at different angles, focus level, and/or fill patterns at exit pupil. To create light fields with multiple depth planes, different fill patterns can be used at the exit pupil. Each layer of the waveguide assembly, or a group of layers (e.g. 3 layers), may be used to produce a particular color, such as red, blue and green. A first set of three layers adjacent may be used to produce red, blue, and green light at a particular focal depth. To produce red, green and blue light at a second depth, a second set of three layers may be used. Multiple sets can be used to create a full 3D and 4D color image lightfield with different focal depths.

“Other Components”

In many cases, other components may be added to or substituted for the components of the wearable systems described above. For example, the wearable system could include one or more haptic components or devices. A user may feel a tactile sensation from the haptic components or devices. The haptic components or devices may be able to provide a tactile sensation such as pressure or texture for touching virtual content (e.g. virtual objects, tools, or other constructs). The tactile sensation could replicate the feel of a real object that a virtual object represents or it may mimic an imagined object or character (e.g. a dragon) that the virtual content represents. Some implementations allow the user to wear haptic components or devices (e.g., a glove for users). Some implementations allow the user to hold haptic components or devices.

“The wearable device may include, for instance, one or more physical objects that can be controlled by the user to enable input or interaction with it. These objects can be called totems. Totems can take the form inanimate objects such as a piece or plastic of metal, a wall, or a table surface. Some totems might not have any input structures, such as keys, triggers or joysticks, trackballs, joysticks, rocker switches, etc. The totem might provide a physical surface and the wearable device may create a user interface that makes it appear the user is on the totem’s surfaces. The wearable system might render an image of a trackpad and keyboard to make it appear that they are on a specific surface of the totem. The wearable system could make a virtual keyboard and trackpad appear on the surface of an aluminum plate that is thin enough to be used as a totem. The rectangular plate itself does not have any trackpad, sensors or physical keys. The wearable system can detect user interaction with the rectangular plate, such as touch or manipulation via the virtual keyboard. FIG. 466 shows the user input device. 4) may be an embodiment of a totem, which may include a trackpad, a touchpad, a trigger, a joystick, a trackball, a rocker or virtual switch, a mouse, a keyboard, a multi-degree-of-freedom controller, or another physical input device. The totem can be used by a user to interact with the wearable device or other users, either alone or in combination.

U.S. Patent Publication No. 0016777 describes “Examples haptic devices or totems usable alongside the wearable devices HMD and display systems of this disclosure.” 2015/0016777 is incorporated herein in its entirety.

“Examples of Wearable Systems, Environments and Interfaces”

A wearable system might use different mapping techniques to obtain high depth of field in rendered light fields. It is important to be able to map out the virtual world and to understand the details of the real world in order to accurately depict virtual objects relative to the real one. By adding new photos that provide information about different features and points of the real world, FOV images can be captured by users of the wearable device. The wearable system can, for example, collect a number of map points (such 2D or 3D points) from users and then find new map points to create a more precise version of the world model. A first user’s world model can be shared (e.g. over a network like a cloud network) with a second user to allow the second user to experience the world around the first user.

“FIG. 7. This is a block diagram showing an example of an MR Environment 700. The MR environment 700 can be configured to receive input. This could include visual input 702 from the wearable system, stationary input 704 like room cameras, sensory input 706 coming from various sensors, gestures and eye tracking, as well as input from the user input device 504. One or more user wearable devices (e.g. wearable system 200, display system 220), or stationary room systems, (e.g. room cameras, etc.). Wearable systems can be equipped with various sensors such as accelerometers and gyroscopes. They can also use temperature sensors, movement sensors or GPS sensors. To determine the location of the user and other environmental attributes. These data may be further supplemented by information from stationary cameras within the room, which may provide images or other cues from another point of view. The data from the cameras, such as the room cameras or the outward-facing imaging systems cameras, may be reduced to a number of mapping points.

“One or more object recognitions 708 can scan through the received data (e.g. the collection of points), and identify or map points, tag images or attach semantic information to objects using a map database 710. The map database 710 can contain various points and the corresponding objects. Through a network, the various devices and map database can be connected. To access the cloud

Based on the information and the collection of points in a map database, object recognizers 708a-708n may recognize objects and add semantic information to make them more real. If the system recognizes a set points as a door, it may add semantic information to the objects (e.g. the hinge has a 90-degree movement around the hinge). The system might attach semantic information to a set that is identified as a mirror if it recognizes the points as mirrors. This may include the fact that the mirror’s reflective surface can reflect images from objects within the room. As more data is added to the map database (which can be local or accessible via a wireless network), the system will grow. The information can be sent to one or more wearable devices once the objects have been identified. The MR environment 700 could include information about a California scene. One or more New Yorkers may receive the environment 700. The object recognizers and the other software components can use data from the FOV camera to map the points and recognize objects. This allows the scene to be?passed over’ accurately. To another user who might be in a different area of the world. Environment 700 can also use topological maps for localization purposes.

“FIG. “FIG.8 is a process flow diagram for an example of a method 800 that renders virtual content in relation with recognized objects. A method 800 shows how a virtual scene can be presented to the wearable device user. The scene may not be accessible to the user. The user could be in New York but want to see a scene in California or go on a walk together with a friend who lives in California.

“At block 810 the AR system can receive input from the user or other users about the environment. This can be done using various input devices and knowledge from the map database. Block 810 is populated with information from the user’s FOV camera and sensors, GPS, eye track, and other devices. Based on the information at block 820, the system can determine sparse point. These sparse points can be used to determine pose data (e.g. head pose, eye position, or hand gestures). This information can be used to display and understand the orientation and location of objects in the environment. These points can be used by object recognition 708 a-708n to crawl through and identify one or more objects from a map database at block 8.30. The information may then be sent to block 840 and displayed at block 855. The desired virtual scene, e.g. user in CA, may be displayed in the right orientation and position in relation to various objects and other surrounding users in New York.

“FIG. “FIG. 9 is a block diagram for another example of a wearable systems. The wearable system 900 in this example includes a map that may contain map data for the entire world. The map can reside partly locally on the wearable device, while other parts may reside in networked storage locations (e.g., cloud systems) that are accessible via wired or wireless networks. The wearable computing architecture may execute a pose process 910. This uses data from the map to determine the user’s position and orientation. Pose data can be calculated from data that is collected as the user interacts with the system and operates in the real world. Data may include images, data from sensors (such inertial measuring units, which usually comprise accelerometers and gyroscope parts), and surface information relevant to objects in the real and virtual environments.

A sparse point representation could be the result of simultaneous localization (SLAM) or V-SLAM processes. This refers to a configuration in which the input is visual only. It is possible to configure the system to find out not only where the components are located in the world, but also what the world is made from. Pose can be used to accomplish many goals. It may also be used to populate the map with data.

“One embodiment shows that a sparse point location may not be sufficient on its own. Additional information may be required to create a multifocal AR, VR or MR experience. This gap may be filled at most in part using dense representations. Dense representations generally refer to depth map information. This information can be obtained from Stereo 940. In Stereo 940, depth information is calculated using techniques such as triangulation and time-of-flight sensoring. Stereo process 940 may also use image information and active patterns, such as infrared patterns made with active projectors. It is possible to combine a lot of depth map information, and some of it may be combined with a surface representation. Mathematically definable surfaces can be more efficient than large point clouds and provide digestible inputs for other processing devices, such as game engines. The stereo output (e.g., depth map) 940 can be combined with the fusion process 930. Pose could also be used as an input for this fusion process 930. The output of fusion 930 can then be used to populate the map process 920. To create larger surfaces, sub-surfaces can connect, as in topographical mapping. The map then becomes a large mixture of points and surfaces.

Multiple inputs can be used to resolve different aspects of a mixed reality process (960). FIG. 9 illustrates an example of such inputs. 9. Game parameters can be inputs that determine whether the user is playing a game of monster fighting with one or more monsters at different locations, monsters running away or dying under certain conditions (such as if they are shot by the user), walls at various locations or any other objects. To add value to mixed reality, the world map might include information about where these objects are located relative to one another. The input of pose relative to the world is also important and plays an important role in almost all interactive systems.

The wearable system 900 also accepts inputs and controls from the user. User inputs include visual input, gestures and totems, audio input and sensory input. To move around, or to play a game, the user might need to tell the wearable system 9000 what he or her wants. There are many user controls that can be used to move oneself around in space. In one embodiment, a totem (e.g. A totem (e.g. The system should be able to identify that the user holds the item and track the interaction.

“Hand gesture recognition or tracking may provide input information. The wearable system may be configured to interpret and track hand gestures such as stop, grab, hold, grab, left, right, grab, or stop. In one configuration, the user might want to flip through emails, a calendar, or even do a “fist bump”. With another player or person. The wearable system may be set up to use a minimal amount of hand gestures, which may or not be dynamic. The gestures could be static gestures such as open hand for stop, thumbs down to not ok, thumbs up to ok; or a flip of the hand right, left, or right for direction commands.

Eye tracking is another input. It is used to track the user’s desire to have the display technology render at a particular depth or range. In one embodiment, vergence of the eyes may be determined using triangulation, and then using a vergence/accommodation model developed for that particular person, accommodation may be determined.”

The FIG. 9 example of a wearable system 900 shows how cameras work. Nine can have three types of cameras. One pair may be a passive SLAM or relative wide FOV camera, and another pair oriented in front to record stereo imaging process 940. The other pair could be positioned in front to capture hand gestures as well as totem/object tracking. The stereo imaging system 940, which includes the FOV cameras and the pair for the stereo process 940, may also include the outward-facing imaging systems 464 (shown at FIG. 4). Eye tracking cameras can be included in the wearable system 900. They may be part of an inward facing imaging system 462 as shown in FIG. 4) are oriented towards the eyes of the user to calculate eye vectors and other information. One or more textured light projectsors (such infrared (IR), projectors) may be included in the wearable system 900 to add texture to a scene.

“FIG. “FIG. The user can interact with a totem in this example. Multiple totems may be owned by the user. One totem could be used for social media, while another can be used for games. Block 1010 may be where the wearable system detects a movement of a totem. The outward-facing system may detect the movement of the totem or sensors may (e.g., haptic gloves, image sensors, hand tracking cameras, head pose sensors, etc )

The wearable system, based at least partially on the detected gesture, eye position, head pose or input through the device, detects the totem’s position, orientation and/or movement relative to block 1020. The reference frame could be a set or map points that the wearable system uses to translate the user’s movement to an action or command. Block 1030 is where the system maps the user’s interaction to the totem. The system calculates the input of the user at block 1040 based on the mapping of user interaction to the reference frame 1020.

To move a totem, or other physical object, the user can turn a page by moving it back and forth. This could be used to indicate moving to a new page or from one user interface (UI), screen to another. Another example is that the user can move their head or gaze to view different virtual or real objects in FOR. The input may be the real or virtual object if the user’s gaze is fixed on a specific real or virtual object for more than a threshold amount of time. In some implementations, the vergence of the user’s eyes can be tracked and an accommodation/vergence model can be used to determine the accommodation state of the user’s eyes, which provides information on a depth plane on which the user is focusing. The wearable system may use raycasting techniques in some instances to determine real and virtual objects that are located along the direction of the user?s eye or head poses. The ray casting techniques may include casting pencil-thin rays that have a very small transverse width, or casting rays that are large in transverse (e.g. cones or fustums).

“The display system described herein may project the user interface (such as the display 220 shown in FIG. 2). You can also display it using other methods, such as one or several projectors. Projectors can project images onto physical objects such as a globe or canvas. One or more cameras may be used to track interactions with the user interface (e.g. using the inward-facing image system 462 or outward-facing imagery system 464)

“FIG. “FIG. 11. is a process flow diagram for an example of a 1100 method of interfacing with a virtual user interface. The wearable system described in this article may perform the method 1100.

“At block 1110 the wearable system might identify a specific UI. The user may choose the type of UI they want. A user input may be used to tell the wearable system which UI should be filled. Block 1120 may be used to generate data for the virtual interface. Data such as the shape, structure, and confines of the virtual UI may be generated. The wearable system could also determine the coordinates of the user?s physical location to display the UI relative to that location. If the UI is body-centric, for example, the wearable device may determine the coordinates and head position of the user so that a ring UI or planar UI can either be displayed around them or on a wall in front of them. The map coordinates of the user?s hands can be determined if the UI is hand-centric. These map points can be obtained from data collected through FOV cameras, sensory input or any other type data.

“Block 1130: The wearable system can send data to the display either from the cloud, or from a local database. Based on the data sent, the UI will be displayed to the user at block 1140. A light field display, for example, can project the virtual interface into the eyes of one or both users. The wearable system can wait for a user command to create more virtual content at block 1150 once the virtual UI is created. The UI could be a ring that wraps around the body of the user, for example. The wearable system might then wait for the command (e.g., a gesture, head or eye movement, etc.). If the command is recognized (block 1106), virtual content associated to it may be displayed to users (block 1170). The wearable system might wait for the user to make hand gestures before mixing multiple steam tracks.

U.S. Patent Publication No. 0016777 also describes additional examples of AR systems, UI and user experiences (UX). 2015/0016777 is incorporated herein by reference in its entirety.

“Examples of Objects in Field of Regard (FOR), and Field of View(FOV)”

“FIG. “FIG. 12” schematically illustrates a virtual object in a view field (FOV) or a view field (FOR). Referring to FIG. 4 The FOR is a part of the environment that can be perceived by the wearable system. FIG. FIG. 12. The FOR 1200 can include a group of objects (e.g. The FOR 1200 can contain a group of objects (e.g. 1210, 1220 and 1230), which can all be perceived by the user using the wearable system. For 1200, the objects may be physical or virtual. The FOR 1200 user may have a physical object like a chair, sofa, wall, or other items. Virtual objects can include objects from the operating system such as a recycle bin to delete files, a terminal for entering commands, a file manager or file manager for accessing files and directories, icons, menus, applications for audio or video streaming, notifications from operating systems, etc. Virtual objects can also include objects within an application, such as avatars, virtual objects used in games, graphics, or images, and so on. Virtual objects can exist as both an operating system object or an object within an application. The wearable system may add virtual objects to existing physical objects in some instances. The wearable system could add a virtual menu to a room with a TV. This virtual menu would allow the user to change or turn on the channels using the wearable.

Virtual objects can be three-dimensional (3D), one-dimensional (1D), or two-dimensional (2D). FIG. 16. The virtual object could be a 3D coffee cup 1636, which may act as a virtual control for an actual coffee maker. A virtual object can also be a 2D graphic representation of a clock 1634, which displays the current time to the user. One or more virtual objects can be displayed within, or associated with, another virtual object in some implementations. FIG. 13 shows the virtual coffee cup 1636 inside the user interface plane 1504, although it appears that the virtual coffee cup is 3D within this virtual 2D space.

“The objects found in the FOR of the user can be part a world map described with reference to FIG. 9. Data associated with objects (e.g. location, semantic information, properties, etc.) You can store information in many data structures, including arrays, lists and trees, as well as hashes, graphs and other semantic information. If applicable, the index of each object stored can be determined by its location. The data structure might index objects using a single coordinate, such as how far the object is from a fiducial location (e.g. how far the object is from the fiducial positions top, bottom, left, or depth wise). The user’s position, such as their head position, may determine the fiducial position. A virtual or physical object, such as a target interactable item, may determine the fiducial position. This allows the user to collapse the 3D space into a 2D user interface, where virtual objects are placed according to their distance from the fiducial location.

“In the FOR 1200, the area that a user perceives at any given time is called the FOV 1250. (e.g. the FOV 1250 could include the FOR portion that the user is currently looking towards). FIG. 12 shows the FOV 1250. FIG. 12. The FOV 1250 schematically illustrated with dashed line 1252 is FIG. The FOV 1250 can be used to perceive multiple objects, including the object 1242, object 1244 and a portion the object 1230. The size and optical characteristics of a wearable device can affect the FOV. An AR display might have optics that provide AR functionality only when the user is looking through a specific portion of it. The FOV 1250 could be the angle the user can see when they look through AR displays such as the stacked waveguide assembly (FIG. 4) or the planarwaveguide 600 (FIG. 6).”

“As the user changes their pose (e.g., head position or eye pose), the FOV 1250 and objects within it 1250 may change. In FIG. 1, the map 1210 is in the FOV of the user. 12. The map 1210 can be seen if the user is looking at it. If the object 1230 is not in the FOV 1250 of the user, then the map 1210 could move into their FOV 1250. The wearable system can keep track of objects in the FOR 1200 and FOV 1250, as will be explained here.

“Examples for Interactable Objects”

“In FIG. FIG. 12 shows how a user can interact directly with a subset the objects within the user’s FOR1200. Sometimes, this subset of objects is called interactable objects. Some implementations may allow interactable objects to include all objects (virtual or physical) within the user’s environmental. In other implementations, however, only a small portion of objects can be included in the environment.

“The wearable system is capable of identifying a subset of interactable objects (e.g. 1242, 1244 and 1230 are all within the user’s FOV 1250. Because the user can perceive them currently, and can choose to activate, move, activate, or retrieve information about them, the subgroup of interactable objects within the FOV is often referred to as selectable. The user’s FOV may change if he moves his head, body or eyes. Some objects will not move outside the FOV and some objects will stay inside the FOV. Other objects will move within the FOV and become selectable. The wearable system can therefore update the FOV subgroup of interactable objects based on user’s body, head or eye position.

The wearable system could identify a target object that is interacted with within the user’s FOV. The user may want to interact with the target interactable objects or it could be the object the wearable device expects the user to interact with (e.g. the interactable object towards which the user is looking, or the one closest to the center of the FOV). The user’s pose, preference or location can all help identify the target interactable object. The wearable system can, for example, choose the object nearest to the center of its FOV as the target interactable item. To make the target interactable object, the wearable system can choose to use either the leftmost or rightmost object in the FOV. The wearable system can also use the inward-facing imagery system 462 (shown at FIG. The IMUs can be used alone or together to determine the user?s gaze direction. The wearable system will identify any object that collides with the user’s gaze and mark it as the target interactable item.

“In certain implementations, the AR software can automatically position the target interactable objects so that their normal faces the user. A virtual TV screen might initially be oriented toward the ceiling. The AR system will automatically rotate the virtual television screen so that it faces the user once it has determined that the user is looking towards the screen.

“Examples for a Focus Indicator”

The wearable system might assign a focus indicator (or a color) to the target object in order for the user to be able to perceive it more easily. The user can see the focus indicator. The focus indicator could include a halo or a color that causes the target object to appear smaller or larger depending on whether it is selected. It can also be used to draw attention. You can also use audible and tactile effects like vibrations, ringtones, beeps, or other visual effects to focus your attention.

“In certain embodiments, the wearable device may identify an object as the target interactable objects based upon the rules herein. Then it could change the target interactable objects to another object based the user’s poses. The user’s pose may change and the focus indicator could move from one object towards the other.

The cursor may correspond to the current position of the wearable system. You can choose from a number of different shapes for the cursor, including a cone, beam of ray or a reticle. An arrow, a reticle or arrow. The cursor can be displayed in the same format as the focus indicator. The cursor could have the same visual, auditory, or tactile effect of the focus indicator. The cursor could be a reticle that corresponds to the user?s head position. Another example is that the cursor could be shaped like an arrow and correspond to the current position of a user input device. The cursor can change its position or move according to the user’s actions. As the user moves about, the cursor can point to one or more objects or empty spaces in his environment. FIG. 16. The AR system can move cursors on the virtual user interface 1514 from one position to another, such as from 1620 to 1624 or 1622 to 1620.

“The cursor can be presented by the wearable system in addition or as an alternative to the focus indicator. FIG. FIG. 15 shows an example of how the wearable system could display a reticle, which can correspond to the user?s direction of gaze, or a light blue circle as a focus indicator. The cursor can be used as an example of the focus indicator in some implementations. The cursor may be used to indicate the focus indicator when the user stares at a virtual object. These visual indicators may indicate the target object the user wants to interact with as well as the current position of the user.

“Examples of Interactions with Interactable Items”

“The wearable system allows users to interact with interactable objects in their FOR 1200, and more specifically with objects in their FOV 1250. The virtual object 1230 could be a graph showing the price change over time. The virtual object 1230 can be selected by the user to allow them to interact with it. This includes stock quotes, buying or selling stock, information about the company and more. The wearable system can display menus, toolbars and other information associated with the virtual objects, which allow the user to perform different actions (e.g., getting a stock quote).

The user can interact with objects within the FOV by opening menus or toolbars, selecting them, moving them, choosing new objects or choosing from a set. You can interact with interactable objects by using hand gestures (see FIG. 466 for an example of user input device 466). 4.) The user may interact with the interactable objects using hand gestures (see e.g. user input device 466 in FIG. Interactable objects can also be accessed by the user using their head, eyes, hands, feet, or other body positions, such as gazing at or pointing an arm at an object, tapping on a touch pad, touching a capacitive button, pressing a key on a keyboard or game controller (e.g., a 5-way d-pad), pointing a joystick, wand, or totem toward an object, blinking for a specified number of times, or tapping foot The AR system can initiate a selection event by using the hand gestures and other poses, such as tapping foot, blinking eyes, or tapping the foot.

“The AR system can assign a focal indicator to a target interactable item in the user’s FOV upon initiation of the selection events using the rules herein. FIG. 12. The AR system could assign a focus indicator for the object 1244, because it is closest to the FOV’s midpoint.

“During the selection event, a user can modify the target object interactable by using any of the hand gestures described in this document. FIG. 12 shows an example of this. FIG. 12 shows how the AR system can change the target object by allowing the user to swipe left on a touch screen. The AR system can transport the visible focus indicator between the object 1244 and the object 1230 in accordance with its settings.

Summary for “Selecting virtual objects within a three-dimensional space”

Modern computing and display technology have made it possible to create systems that can be called “virtual reality”, “augmented reality?”, or “mixed reality?” Digitally reproduced images and portions of them are presented to users in a way that makes them appear real or can be perceived as such. A virtual reality (or?VR?) scenario usually presents digital or virtual information without transparency to any other real-world visual input. An augmented reality (or?AR?) scenario typically presents digital or virtual information as an enhancement to the visualization of the real world around the user. A mixed reality (or?MR?) scenario involves merging real and digital worlds to create new environments in which physical and virtual objects coexist and interact in real-time. It turns out that the human visual system is complex. Therefore, it is difficult to create a VR, AR or MR technology that allows for a natural-feeling, rich presentation and interaction of virtual images elements with real-world or virtual imagery elements. The systems and methods described herein address different challenges in VR, AR, and MR technology.

“Some embodiments disclose a system that allows the user to change their input mode on a wearable device. The system may include a display system for the wearable device that presents a three-dimensional (3D), view to the user. This view can contain interactable objects. A user input device is configured to receive user input. A sensor is configured to collect data about the pose of the user. Finally, there’s a hardware processor communicating with the user input device. The hardware processor can be programmed in order to determine whether the current user input method for interfacing with the interactable objects is either a first or second user input mod. In this case, the first mode relies at least partially on the pose of the user while the second mode relies at least partly upon the input of the user. If the current user interface mode is determined to be the first, the hardware processor can use the sensor to monitor the user’s pose and present via the display system, based at minimum partly on that pose, a focus indicator in the first shape associated to the first user mode in a direction related the user’s pose; receive a first indication of switching to the second mode; and then switch the current input mode to the second mode in response to the first indication. If the current user mode is determined to be the second, the hardware processor can monitor user input using the sensor; present via display system, based at minimum partly on the monitored pose, the focus indicator in a second shape associated to the second user mode; receive a second signal to switch to that mode; and then switch the current input mode to that mode in response to the second indication.

“In certain embodiments, there is a method of changing the user input mode of a wearable device. This method can be carried out under the control of a wearable device with a processor. A wearable device may be designed to allow interaction between interactable objects and users in a field. The FOR is a section of the environment that can be perceived by the user through a display system. The method involves: determining the pose of a person; displaying via the display a first indicator associated to a target interactable item in a direction that is related to the user’s pose; receiving a selection from the target interactable objects; displaying via the system the plurality virtual objects; updating the second indicator via the display device based at most partly on user input.

“In some embodiments, the disclosure includes a wearable system and a method of selecting a virtual object in a three-dimensional (3D), space. The wearable system may include a display system that displays virtual objects in 3D space, a non-transitory storage data store for interactable objects in 3D space, a sensor to determine the pose of the user, and a processor programed to communicate with both the display system, data store and the sensor. The method and wearable system can detect the pose and data from the sensor. They also determine a field view (FOV), which is a section of the environment that the user perceives at any given time. They identify a group interactable objects within the FOV and identify the target object.

“Details about one or more implementations are provided in the accompanying drawings as well as the description. The claims, drawings, and description will reveal other features, aspects, or advantages. This summary and the detailed description below do not attempt to limit or define the scope of the inventive subject matter.

“Overview”

A virtual interface allows users to select and target objects in three-dimensional space. A user could select a virtual object by using body poses, such as touching, touching, or physically approaching the objects. You can also point at the virtual objects and click on them with virtual beams or rays. These techniques can lead to fatigue, and it can be difficult for the user to select objects accurately.

The present disclosure gives examples of wearable devices that can address any or all of these issues. One example is that the user might move his head to look at a group object. To transport the highlighted object from one object to the next, the user can use a user input device (e.g. by tapping on a touchscreen) or move his head to highlight another object. You can confirm that the target object has been selected by activating the input device once more (e.g. Touch the touchscreen to confirm. The AR user interface can allow the user to perform additional actions upon the target object once it has been selected (e.g. display or select from a menu, perform an action related to a game in which the target object appears, etc.). This is a great way to select the objects you are interested in, while also reducing fatigue. Because head poses can be difficult to control accurately, this is why it’s so useful. The wearable system will identify objects of interest based upon the user’s head position and, in the meantime allow the user to select objects using hand gestures.

“Some implementations allow for multiple virtual objects to be interacted with an interactable object. A virtual user interface plane could include multiple virtual apps, such as a video streaming app, a virtual classroom, a weather app, a game, or an astronomy program. Based on the properties of the interactable object, the wearable system can support different user input methods. The wearable system might allow users to interact with interactable objects by using poses, for example, if they are large user interface planes. The wearable system might instead choose the user input mode as the default input mode for small objects to enable precise interaction with them. These options may prove to be more advantageous as moving and targeting large objects may require less precision, while selecting and moving small objects may require precision.

“The context information can be used by the wearable system to determine the user’s input mode. The wearable system can, for example, determine the location of virtual objects within the user’s environment. The wearable system can detect a dense clustering of virtual objects along the user’s gaze direction and give the user the ability to switch from head control to hands. The user can interact more precisely with virtual objects this way. Another example is that the AR system might detect the orientation of an object (e.g. vertical or horizontal) to provide the appropriate interactions for the users (e.g. volume controls for a TV application that appears vertically in front of the user, or typing controls for virtual keyboards that appear horizontally on the user’s desk).

“The wearable device can allow a user to share virtual contents with others (also wearing wearable systems), for example by sharing a world map of their environment or by communicating the virtual information (or updates to it) among the wearable devices via a network.

“Examples 3D Display”

“FIG. FIG. 1 shows an illustration of a mixed-reality scenario, with certain virtual objects and some physical objects that can be viewed by a human. FIG. FIG. 1. A MR scene 100 depicts a user of MR technology. The scene shows a person imagining a park-like setting 110 with people, trees, buildings, and a concrete platform 120. The MR technology user also perceives these items. A robot statue 130 stands upon the real-world platform 120. A cartoon-like avatar character 140 is flying past, which appears to be the personification of a Bumble Bee.

It may be beneficial for 3D displays to have an accommodative response that corresponds to the virtual depth of each point. The human eye can experience accommodation conflicts if the accommodative response of a display point doesn’t correspond to its virtual depth as determined by stereopsis and binocular depth cues. This could lead to unstable imaging, headaches, and even complete absence of surface depth.

Display systems that provide images corresponding to a variety of depth planes to viewers can offer VR, AR, and MR experiences. Each depth plane may have different images. This allows the viewer to see depth cues by observing differences in image features or the accommodation required to bring them into focus. These depth cues, as discussed in the past, provide credible perceptions about depth.

“FIG. “FIG. 2” illustrates an example wearable system 200. The wearable system 200 comprises a display 220 and various electronic and mechanical modules that support display 220. Display 220 can be connected to frame 230. This frame is usable by the user, wearer or viewer 210. The display 220 may be placed in front of the user’s eyes 210. The display 220 can show AR/VR/MR content. A head-mounted display (HMD), which is attached to the display 220, can be used. A speaker 240 can be attached to the frame 230 in some embodiments and placed next to the user’s ear canal. (In some embodiments, an additional speaker, not shown here, is placed adjacent to the user’s other ear canal to allow for stereo/shapeable sounds control).

“The wearable device 200 may include an outward-facing imaging unit 464 (shown at FIG. 4) that observes the environment around the user. A wearable system 200 may also include an inward-facing image system 462 (shown at FIG. 4), which can track the eye movements. Inward-facing imaging systems can track the movements of one or both eyes. The frame 230 may have an inward-facing image system 462 attached. It may also be electrically connected to the processing modules 265 and 270. These modules may process image data acquired by the inward facing imaging system to determine the pupil sizes or orientations, eye movements, or user’s eye position 210.

“As an example the wearable device 200 can use either the outward-facing or inward-facing imaging systems 464 to capture images of the user’s pose. Images can be still images, frames, or combinations of images.

“The display 220 may be operatively coupled 250 such as by wired connectivity or wireless connectivity to a local processing module 260. It can be fixedly attached 230 to a helmet/hat worn by the user, embedded into headphones or other removable attachments to the user 210 (e.g. backpack-style configuration or belt-coupling style configuration).

“The local processing module and data module 265 may include a hardware processor as well as digital memories such as flash memory and non-volatile memory. Both of these may be used to aid in data processing, caching, storage, and processing. Data may include data from sensors (which could be, e.g.., operatively coupled with the frame 230, or otherwise attached to user 210), as well as microphones, inertial measuring units (IMUs), compasses and global positioning system units (GPS), radio devices or gyroscopes. Alternatively, data can be acquired using remote processing module (270) or remote data repository (280) for passage to display 220. Local processing and data modules 260 can be operatively connected via communication links 262 and 264 to remote processing module (270) or remote data repository (280), so that remote modules are available for local processing and data 260. Remote processing module 280 or remote data repository 282 may also be operatively coupled.

“In some cases, the remote processing module (270) may include one or more processors that are capable of processing data and/or images. The remote data repository 280 could include a digital storage facility that can be accessed via the internet or another networking configuration. Resource configuration. Some embodiments store all data and perform all computations in the local processing module. This allows for fully autonomous use from remote modules.

The human visual system is complex and it is difficult to perceive depth accurately. It is possible that viewers may perceive an object as three-dimensional due to vergence and accommodation. The movement of vergence (i.e. rolling of the pupils towards or away from one another to converge the lines in the eyes to fixate on an object) is closely related to focusing (or?accommodation?). The lenses of the eyes. Under normal conditions, changing the focus of the lenses of the eyes, or accommodating the eyes, to change focus from one object to another object at a different distance will automatically cause a matching change in vergence to the same distance, under a relationship known as the ?accommodation-vergence reflex.? Under normal conditions, a change of vergence will also trigger a matching change to accommodation. Display systems that match accommodation and vergence better may create more comfortable and realistic simulations of three-dimensional imagery.

“FIG. 3. illustrates aspects of a method for simulating three-dimensional imagery using multiple depth plans. Referring to FIG. FIG. The eyes 302 & 304 use particular accommodated states to focus objects at different distances along z-axis. A particular accommodated condition may be associated with one of the depth planes 306, and has an associated focal distance. This means that objects or parts of objects in particular depth planes are in focus when the eye moves in that accommodated state. Three-dimensional imagery can be created in some embodiments by providing different images for each eye 302 and 304 and different versions of each image corresponding to each depth plane. Although the fields of view are shown separately for illustration purposes, it is possible for the eyes 302 or 304 to overlap as the distance along the Z-axis increases. For illustration purposes, the contours of depth planes are shown flat. However, it is possible for them to curve in space so that all of the features of the depth planes are focused with the eye in an accommodated state. The theory does not limit the possibility that the human eye can perceive depth in a finite number depth planes. Therefore, it is possible to create a convincing simulation of depth perception by showing the eye different images that correspond to each of the limited depth planes.

“Waveguide Stack Assembly”

“FIG. “FIG. 4 shows an example of a waveguide stack that outputs image information to a user. Wearable system 400 may include a stack of waveguides or stacked waveguide assemblies 480. These waveguides can be used to provide three-dimensional perception for the eye/brain by using a plurality waveguides 432, 434, 436, 436, 438, 438, 438, 438, 438, 438, 400 b. The wearable system 400 could be compared to the 200 in FIG. 2 with FIG. 4, schematically showing more parts of the wearable system 200. In some embodiments, for example, the waveguide assembly (480) may be integrated into FIG. 2.”

“With reference to FIG. 4. The waveguide assembly 480 could also contain a number of features 458, 456, 454, or 454, 452, between the waveguides. The features 458, 456, 454, and 454, 452, may be lenses in some embodiments. Other embodiments may not allow for the features 458, 456, 454 or 454, 452 to be lenses. They may be spacers, such as cladding layers, or structures that form air gaps.

Waveguides 432, 434, 436, 438, 438, 438, 440 b and the plurality lenses 458, 456, 436, 438, 438, 438, 438, 438, 438, 438, 438, 438, 440 b may be set up to transmit image information to the eye at different levels of light ray divergence or wavefront curvature. Each waveguide level can be associated with a specific depth plane, and may be configured so that it outputs image information that corresponds to that depth. The image injection devices 420 to 432, 424, 426 and 428 can be used to inject image information into waveguides 440, 438, 436, 434, 434, 432, 428. Each of these devices may be designed to distribute light across the waveguides, for output towards the eye 410. The light exits from an output surface of image injection devices 420-422, 424-426, 428, 428 and is injected into the waveguides’ corresponding input edges 440 b to 438 b. 436 b. 434 b. 432 b. One beam of light, such as a collimated beam, may be injected into each waveguide in order to produce an entire field of collimated collimated beams directed towards the eye 410 at specific angles and divergence corresponding to the depth plan associated with that waveguide.

“In some embodiments the image injection devices 422, 424 and 426 are discrete displays that produce image information to be injected into a waveguide 440, 438, 436, 434, 434, 432, and 432 b. Other embodiments include the image injection device 420 to 422, 424 and 426. These devices may be output ends of a single multiplexed monitor that can pipe image information through one or more optical conduits (such fiber optic cables) into each of the image injectors 420, 422, 422, 424 and 426.

“A controller 460 controls operation of the stacked wafer assembly 480 and image injection devices 424, 426. 428. The controller 460 contains programming (e.g. instructions in a nontransitory computer-readable media) that controls the timing and provision image information to waveguides 440, 438, 436, 436, 434, 434, 434, 432, 432 b. The controller 460 can be either a single integrated device or a distributed system connected via wired or wireless communication channels. The controller 460 could be part of one or more processing modules 260 and 270 (illustrated at FIG. 2) in certain embodiments.

“The waveguides 438 b and 436 b can be set up to transmit light through total internal reflection (TIR) within their respective waveguides. Waveguides 440, 438, 436, 434, 434, 432 and 432 b can be either planar or curved, with major top- and bottom surfaces, and edges that extend between them. The illustrated configuration of the waveguides 440, 438, 436, 434, and 432 B may all include light extracting optical components 440a,436 b., 436, b., 434, b., 432, b. These elements are designed to extract light from a waveguide by redirecting light within the waveguide, out of each waveguide to provide image information to the eye 434 a., 432a. Outcoupling optical elements and extracted light can also be called outcoupled. The waveguide outputs a beam of light at the locations where the light from the waveguide strikes a light redirecting component. For example, light extracting optical elements (440a, 438a, 436, 434 a., 432a, 442 a., 432) may be reflective or diffractive optical components. For ease of description and drawing, the light extracting optic elements are shown at the bottom of the waveguides. The light extracting optical element 440a, 438a, 436, 434a, and 432a can be formed in a layer made of material attached to a transparent substrate. This will form the waveguides. Other embodiments allow the waveguides 440, 438, 436, 434, b and 432 to be formed in a single piece of material. The light extracting optical element 440a,438 a., 436, 436, 434, b., 432b. may also be formed on a surface, or within that piece.

“With reference to FIG. “With continued reference to FIG. The waveguide 432b closest to the eye could be set up to deliver collimated lighting, such as that injected into waveguide 432b to the eye 410. The collimated light could be representative of optical infinity focal plan. The next waveguide may send out collimated beams through the first lens 452 (e.g. a negative lens) before reaching the eye 410. The first lens 452 could be set up to create a convex wavefront curvature, so the eye/brain perceives the light coming from the next waveguide 434 b as coming in from a first focal point closer inward towards the eye 410 from optical infinite. The third up waveguide 436b also passes its output light through the second lens 454 and first lens 452, before reaching the eye. Combining the optical power of the first two lenses 452 and 444, may create an incremental amount of wavefront curve so that the eye/brain perceives light coming out of the third waveguide 436 B as coming from a second focal point that is closer inward towards the person than light from the next waveguide, 434 b.

The other waveguide layers (e.g. waveguides 438b, 440b) and lenses (e.g. lenses 456, 458) are similarly configured. The highest waveguide, 440b in the stack, sends its output through all lenses between it, the eye, for an aggregate focal power that is representative of the nearest focal plane to the person. A compensating lens layer (430) may be placed at the top to offset the effect of the lens stack 458, 456, 454 and 452 on viewing/interpreting light from the world 470. This configuration can provide as many perceived focal points as possible, regardless of the number of waveguide/lens pairs. The light extracting optical elements and the focusing parts of the lenses can be either static or not dynamic. Alternate embodiments may include one or both of these elements, with electro-active features.

“With reference to FIG. “With continued reference to FIG. 4, the light extracting optic elements 440a, 438a, 436, 436a, 434a, 443, 434a, and 432a can be configured to redirect light out of respective waveguides and to produce this light with the correct amount of divergence/colimation for the particular depth plane associated to the waveguide. Waveguides with different depth planes can have different configurations for light extracting optical elements. These elements may output light with different amounts of divergence or collimation depending on the depth plane. As discussed above, light extracting optical element 440 a and 438 a are volumetric or surface features that can be set up to produce light at certain angles. The light extracting optical components 440 a. 438 a. 436 a. 434 a. 432 a. 432 a. may be volume or surface holograms and/or diffraction gratings. U.S. Patent Publication No. 2015/0178939 published June. 25th of June 2015, is included by reference in its entirety.”

“In certain embodiments, light extracting optical elements (440 a), 438 a. 436 a. 436 a. 434 a. 432 a. 432 a) are diffractive features that create a diffraction patterns or?diffractive optic element?. Also known as a “DOE” The DOE should have a low diffraction efficiency, so that only a small portion of the beam is deflected toward the eye (410 at each intersection). The rest of the beam continues to travel through the waveguide via total internal reflect. This allows for the division of the light carrying image information into several related exit beams, which exit the waveguide at a variety of locations. The result is a relatively uniform pattern of exit emission towards the eye 304 in this particular collimated beam that bounces around within a waveguide.

“In some embodiments, one of the DOEs can be switched between?on and?off? state in which they actively diffract and?off?” They do not significantly diffract. A switchable DOE could be a layer of polymer-dispersed liquid crystal. The microdroplets may be arranged in a diffraction pattern in the host medium. In this case, the refractive indice of the microdroplets can either be changed to match the host material’s or to an index that is not as high as the host medium’s (in which case they actively diffract incident sunlight).

“In some embodiments the distribution and number of depth planes (or depth of field) may be dynamically adjusted based on the pupil sizes of the viewers eyes. The size of a viewer’s pupils may affect the depth of field. The depth of field may increase as the size of the pupil decreases. This means that a plane that isn’t discernible due to its location being beyond the depth-of-focus of the eyes can become more discernible. With a reduction in pupil size, and in proportion with an increase in depth, the field depth will also change. The decreased pupil size may also affect the number of depth planes that are used to show different images to the viewers. A viewer might not be able see details of both a depth plane and another depth plane at the same pupil size. This could lead to confusion. However, these two depth planes can be in focus simultaneously to the user at a different pupil size.

“In certain embodiments, the display system can alter the number of waveguides that receive image information based on pupil size and orientation or upon receiving signals indicative of particular pupil sizes or orientations. If the user is unable to distinguish between the depth planes of two waveguides, the controller 460 can be programmed or configured to stop providing image information to that waveguide. This may increase the system’s responsiveness and reduce the system’s processing load. If the DOEs of a waveguide can be switched between the on or off states, they may be switched to off when the waveguide receives image information.

“In certain embodiments, it might be desirable for an exit beam to meet the condition that its diameter is less than the diameter the viewer’s eye. This condition can be difficult due to the variable size of viewers’ pupils. This condition can be met in some embodiments by changing the size and shape of the exit beam according to the size of the pupil. The size of the exit beam could decrease as the pupil grows, for example. In certain embodiments, the exit beam size can be adjusted using a variable aperture.

The wearable system 400 may include an outward-facing image system 464 (e.g. a digital camera) that captures a portion the world 470. This area of the world 470 is often referred to by the term “field of view” (FOV), and the 464 imaging system is sometimes called an FOV camera. The field of regard (FOR) is the entire area that is available for viewing and imaging by a viewer. There may be 4 FOR. The FOR may include 4? Other contexts may require the wearer to move more tightly, so the FOR of the wearer may have a smaller solid angle. Images taken with the outward-facing imaging device 464 can be used for tracking gestures (e.g. hand or finger gestures), detecting objects in the world, 470, and so on.

The wearable system 400 may also include an inward facing imaging system 466 (e.g. a digital camera) that monitors the movements of the user such as their eye movements and facial movements. To determine the size and/or orientation the pupil of the eyes 304, the inward-facing imaging device 466 can be used. Inward-facing imaging system 466 may be used to capture images that can be used for biometric identification (e.g. via iris identification) or to determine the user’s gaze direction. One camera may be used for each eye in some embodiments. This allows each eye to determine its pupil size and position independently. Each eye can then be presented with image information that is dynamically tailored to it. Other embodiments allow for the determination of the pupil diameter and orientation of one eye 410. This allows the presentation of image information to each eye to be dynamically tailored. The inward-facing imaging device 466 can analyze the images to determine the user?s mood or eye position. This information can then be used by the wearable technology 400 to determine which audio or visual content to present to the user. Wearable system 400 can also detect head pose (e.g. head position or orientation) by using sensors such IMUs, accelerometers and gyroscopes.

The wearable system 400 may include a user input device 466 through which the user can input commands 460 to the controller to interact with the wearable systems 400. The user input device 466 may include a trackpad and touchscreen, as well as a joystick, multiple degree of freedom (DOF), controller, capacitive sensing devices, a game controller (D-pad), keyboard, mouse, a directional pad, a wand, and a haptic device. It can also function as a virtual user interface device (e.g., a totem). Sometimes, the user may press or swipe with a finger (e.g. a thumb) on a touch sensitive input device to input the wearable systems 400. This could be to provide user input to the user interface provided by wearable system 405. During the wearable system 400, the user may hold the user’s input device 466 in their hand. The user input device 466 may be wired or wirelessly connected to the wearable system 400.

“FIG. “FIG.5” 5 illustrates an example of exit beams produced by a waveguide. Although one waveguide is shown, it can be seen that many waveguides within the waveguide assembly480 could function in a similar fashion. The waveguide 432b is infected at the input edge 432c of the waveguide 432, and the light propagates through the waveguide 422, b via TIR. A portion of the light 520 exits the waveguide as the exit beams 510 at points where it impedes on the DOE 432. Although the exit beams 510 appear to be substantially parallel, they can also be directed to propagate to eye 410 at an angle (e.g. forming divergent exit beacons), depending upon the depth plane associated to the waveguide 432b. You will see that the exit beams 510 are substantially parallel. This could be a sign of a waveguide that uses light extracting optical elements to outcouple light, forming images that appear to be set at a deep plane that is far away (e.g. optical infinity). Waveguides with other light extracting optical elements or waveguides may produce a divergent exit beam pattern. This would require the eye to adjust to a greater distance to focus the image on the retina. The brain would interpret this as light coming from a farther distance than the eye 410.

“FIG. “FIG. An optical system may include a waveguide device, an optical coupler subsystem that optically couples light to or from waveguide apparatus, as well as a control subsystem. Multi-focal volumetrics, images, and light fields can all be generated by the optical system. One or more primary planar wavesguides 632a can be included in the optical system (only one is shown at FIG. 6) and one (or more) DOEs 632b that are associated with at least one of the primary waveguides (632 a). The planar waveguides 632b can be identical to those waveguides 432b, 434b, 436b, 436b, 438b, and 438b. 4. An optical system could use a distribution waveguide device to transmit light along a first direction (vertical or the Y-axis, as shown in FIG. 6.) and increase the light’s effective escape pupil along the first direction (e.g., the Y-axis). Distribution waveguide apparatus could include, for instance, a distribution waveguide 622b and at most one DOE 622a (illustrated with double dash-dot lines) that is associated with the distribution waveguide 622b. The distribution planar wavesguide 622b could be identical or similar to the primary planar waveguide 642 b in at least some aspects, but with a different orientation. The DOE 622 a could be identical or similar to the DOE 632. The distribution planar waveguide 622b and DOE 622a could be made of the same materials, or the primary planar wavesguide 632b or DOE 622 a, for example. FIG. 6 shows embodiments of the optical display 600. 6 can be integrated in the wearable system 200 as shown in FIG. 2.”

“The relayed or exit-pupil extended light may be optically coupled to the distribution waveguide apparatus into one or more primary plane waveguides 632b. The primary planar waveguide 632b can relay light along another axis, but it should be orthogonal to the first axis (e.g. horizontal or X-axis as shown in FIG. 6). The second axis may be non-orthogonal to the first. The primary planar waveguide 632b expands light’s exit pupil along the second axis (e.g. X-axis). The distribution planar wavesguide 622 b, for example, can expand and relay light along the vertical axis. It then passes that light on to the primary planar waveguide 642, which can expand and relay light along its horizontal axis.

“The optical system can include one or more sources for colored light (e.g. red, green and blue laser light)610, which may be optically connected to a proximal tip of a single-mode optical fiber 640. The distal end may be threaded through or received through a hollow tube 642 made of piezoelectric materials. As a flexible, fixed-free cantilever 644, the distal end protrudes out of the tube 642 Four quadrant electrodes can be attached to the piezoelectric tube 642. For example, the electrodes can be plated on the outer, outer, or outer periphery of the tube 642. Unillustrated core electrodes may be found in the core, inner periphery, or inner diameter of the tube 642.

“Drive electronics 655, for example, electrically coupled via wires 665, drive opposing electrodes to bend piezoelectric tube 642 in 2 axes independently. Mechanical modes of resonance are found at the distal tip 644 of the optical fibre. The frequency of resonance depends on the diameter, length, as well as the material properties of optical fiber 644. The fiber cantilever 644 vibrates when the piezoelectric tube 642 is vibrated near the first mode of mechanical resonance. It can also sweep through large deflections by vibrating the 642.

“By stimulating resonance vibration in two directions, the tip 644 of the fiber cantilever is scan biaxially in an area that fills two-dimensional (2D). scan. An image can be created by modulating the intensity of light source(s), 610, in sync with the scan of fiber cantilever 644. U.S. Patent Publication No. 2014/0003762 is incorporated herein in its entirety.

“A component of an optical subsystem that can collimate light from the scanning fibre cantilever 644 can be called an optical coupler subsystem. Mirror surface 648 can reflect the collimated light into the narrow distribution plane waveguide 622b, which contains at least one diffractive optic element (DOE 622a). Relatively to FIG. 6) along the distribution plane waveguide 622b by TIR and intersects repeatedly with the DOE 622a. The DOE 622a is preferred to have a low coefficient of reflection. This can result in a fraction (e.g. 10%) of light being diffracted towards an edge of the primary planar waveguide 642 b at each intersection with the DOE622 a and a fraction to continue its original trajectory along the length of distribution planar waveguide 642 b via TIR.

“At every intersection with the DOE 622a, additional light may be diffracted towards the primary waveguide 632b. The DOE 4 can expand vertically the exit pupil by dividing the incoming beam into multiple outcoupled sets. This vertically expanded light can be coupled out of distribution waveguide 622b to enter the edge the primary planar wavesguide 632b.

“Light entering the primary waveguide 632b can propagate horizontally (relatively to FIG. 6) along primary waveguide 632b via TIR. The light crosses DOE 632a at multiple points and propagates horizontally along at most a portion the primary waveguide 632b via TIR. To produce deflection and focus of light, the DOE 632a can be advantageously designed or configured with a phase profile. This is a combination of a linear and radially symmetric diffractive patterns. The DOE 632a may have a lower diffraction efficiency (e.g. 10%) so that only a small portion of the beam is deflected towards the eye with each intersection of DOE 632a, while the rest propagates through the primary waveguide 632b via TIR.

“A fraction of the light is scattered toward the face of primary waveguide 632b at each intersection of the propagating light with the DOE 632a. This allows the light to escape from the TIR and emerge from primary waveguide 642 b. The DOE 632a radially symmetric pattern of diffraction imparts an additional focus level to diffracted light. This allows the light wavefront to be shaped (e.g., giving a curvature) and the beam to be directed at an angle that matches the intended focus level.

These different paths can result in the light being coupled out of the primary plane waveguide 632b by a multiplicity OFEs 632a at different angles, focus level, and/or fill patterns at exit pupil. To create light fields with multiple depth planes, different fill patterns can be used at the exit pupil. Each layer of the waveguide assembly, or a group of layers (e.g. 3 layers), may be used to produce a particular color, such as red, blue and green. A first set of three layers adjacent may be used to produce red, blue, and green light at a particular focal depth. To produce red, green and blue light at a second depth, a second set of three layers may be used. Multiple sets can be used to create a full 3D and 4D color image lightfield with different focal depths.

“Other Components”

In many cases, other components may be added to or substituted for the components of the wearable systems described above. For example, the wearable system could include one or more haptic components or devices. A user may feel a tactile sensation from the haptic components or devices. The haptic components or devices may be able to provide a tactile sensation such as pressure or texture for touching virtual content (e.g. virtual objects, tools, or other constructs). The tactile sensation could replicate the feel of a real object that a virtual object represents or it may mimic an imagined object or character (e.g. a dragon) that the virtual content represents. Some implementations allow the user to wear haptic components or devices (e.g., a glove for users). Some implementations allow the user to hold haptic components or devices.

“The wearable device may include, for instance, one or more physical objects that can be controlled by the user to enable input or interaction with it. These objects can be called totems. Totems can take the form inanimate objects such as a piece or plastic of metal, a wall, or a table surface. Some totems might not have any input structures, such as keys, triggers or joysticks, trackballs, joysticks, rocker switches, etc. The totem might provide a physical surface and the wearable device may create a user interface that makes it appear the user is on the totem’s surfaces. The wearable system might render an image of a trackpad and keyboard to make it appear that they are on a specific surface of the totem. The wearable system could make a virtual keyboard and trackpad appear on the surface of an aluminum plate that is thin enough to be used as a totem. The rectangular plate itself does not have any trackpad, sensors or physical keys. The wearable system can detect user interaction with the rectangular plate, such as touch or manipulation via the virtual keyboard. FIG. 466 shows the user input device. 4) may be an embodiment of a totem, which may include a trackpad, a touchpad, a trigger, a joystick, a trackball, a rocker or virtual switch, a mouse, a keyboard, a multi-degree-of-freedom controller, or another physical input device. The totem can be used by a user to interact with the wearable device or other users, either alone or in combination.

U.S. Patent Publication No. 0016777 describes “Examples haptic devices or totems usable alongside the wearable devices HMD and display systems of this disclosure.” 2015/0016777 is incorporated herein in its entirety.

“Examples of Wearable Systems, Environments and Interfaces”

A wearable system might use different mapping techniques to obtain high depth of field in rendered light fields. It is important to be able to map out the virtual world and to understand the details of the real world in order to accurately depict virtual objects relative to the real one. By adding new photos that provide information about different features and points of the real world, FOV images can be captured by users of the wearable device. The wearable system can, for example, collect a number of map points (such 2D or 3D points) from users and then find new map points to create a more precise version of the world model. A first user’s world model can be shared (e.g. over a network like a cloud network) with a second user to allow the second user to experience the world around the first user.

“FIG. 7. This is a block diagram showing an example of an MR Environment 700. The MR environment 700 can be configured to receive input. This could include visual input 702 from the wearable system, stationary input 704 like room cameras, sensory input 706 coming from various sensors, gestures and eye tracking, as well as input from the user input device 504. One or more user wearable devices (e.g. wearable system 200, display system 220), or stationary room systems, (e.g. room cameras, etc.). Wearable systems can be equipped with various sensors such as accelerometers and gyroscopes. They can also use temperature sensors, movement sensors or GPS sensors. To determine the location of the user and other environmental attributes. These data may be further supplemented by information from stationary cameras within the room, which may provide images or other cues from another point of view. The data from the cameras, such as the room cameras or the outward-facing imaging systems cameras, may be reduced to a number of mapping points.

“One or more object recognitions 708 can scan through the received data (e.g. the collection of points), and identify or map points, tag images or attach semantic information to objects using a map database 710. The map database 710 can contain various points and the corresponding objects. Through a network, the various devices and map database can be connected. To access the cloud

Based on the information and the collection of points in a map database, object recognizers 708a-708n may recognize objects and add semantic information to make them more real. If the system recognizes a set points as a door, it may add semantic information to the objects (e.g. the hinge has a 90-degree movement around the hinge). The system might attach semantic information to a set that is identified as a mirror if it recognizes the points as mirrors. This may include the fact that the mirror’s reflective surface can reflect images from objects within the room. As more data is added to the map database (which can be local or accessible via a wireless network), the system will grow. The information can be sent to one or more wearable devices once the objects have been identified. The MR environment 700 could include information about a California scene. One or more New Yorkers may receive the environment 700. The object recognizers and the other software components can use data from the FOV camera to map the points and recognize objects. This allows the scene to be?passed over’ accurately. To another user who might be in a different area of the world. Environment 700 can also use topological maps for localization purposes.

“FIG. “FIG.8 is a process flow diagram for an example of a method 800 that renders virtual content in relation with recognized objects. A method 800 shows how a virtual scene can be presented to the wearable device user. The scene may not be accessible to the user. The user could be in New York but want to see a scene in California or go on a walk together with a friend who lives in California.

“At block 810 the AR system can receive input from the user or other users about the environment. This can be done using various input devices and knowledge from the map database. Block 810 is populated with information from the user’s FOV camera and sensors, GPS, eye track, and other devices. Based on the information at block 820, the system can determine sparse point. These sparse points can be used to determine pose data (e.g. head pose, eye position, or hand gestures). This information can be used to display and understand the orientation and location of objects in the environment. These points can be used by object recognition 708 a-708n to crawl through and identify one or more objects from a map database at block 8.30. The information may then be sent to block 840 and displayed at block 855. The desired virtual scene, e.g. user in CA, may be displayed in the right orientation and position in relation to various objects and other surrounding users in New York.

“FIG. “FIG. 9 is a block diagram for another example of a wearable systems. The wearable system 900 in this example includes a map that may contain map data for the entire world. The map can reside partly locally on the wearable device, while other parts may reside in networked storage locations (e.g., cloud systems) that are accessible via wired or wireless networks. The wearable computing architecture may execute a pose process 910. This uses data from the map to determine the user’s position and orientation. Pose data can be calculated from data that is collected as the user interacts with the system and operates in the real world. Data may include images, data from sensors (such inertial measuring units, which usually comprise accelerometers and gyroscope parts), and surface information relevant to objects in the real and virtual environments.

A sparse point representation could be the result of simultaneous localization (SLAM) or V-SLAM processes. This refers to a configuration in which the input is visual only. It is possible to configure the system to find out not only where the components are located in the world, but also what the world is made from. Pose can be used to accomplish many goals. It may also be used to populate the map with data.

“One embodiment shows that a sparse point location may not be sufficient on its own. Additional information may be required to create a multifocal AR, VR or MR experience. This gap may be filled at most in part using dense representations. Dense representations generally refer to depth map information. This information can be obtained from Stereo 940. In Stereo 940, depth information is calculated using techniques such as triangulation and time-of-flight sensoring. Stereo process 940 may also use image information and active patterns, such as infrared patterns made with active projectors. It is possible to combine a lot of depth map information, and some of it may be combined with a surface representation. Mathematically definable surfaces can be more efficient than large point clouds and provide digestible inputs for other processing devices, such as game engines. The stereo output (e.g., depth map) 940 can be combined with the fusion process 930. Pose could also be used as an input for this fusion process 930. The output of fusion 930 can then be used to populate the map process 920. To create larger surfaces, sub-surfaces can connect, as in topographical mapping. The map then becomes a large mixture of points and surfaces.

Multiple inputs can be used to resolve different aspects of a mixed reality process (960). FIG. 9 illustrates an example of such inputs. 9. Game parameters can be inputs that determine whether the user is playing a game of monster fighting with one or more monsters at different locations, monsters running away or dying under certain conditions (such as if they are shot by the user), walls at various locations or any other objects. To add value to mixed reality, the world map might include information about where these objects are located relative to one another. The input of pose relative to the world is also important and plays an important role in almost all interactive systems.

The wearable system 900 also accepts inputs and controls from the user. User inputs include visual input, gestures and totems, audio input and sensory input. To move around, or to play a game, the user might need to tell the wearable system 9000 what he or her wants. There are many user controls that can be used to move oneself around in space. In one embodiment, a totem (e.g. A totem (e.g. The system should be able to identify that the user holds the item and track the interaction.

“Hand gesture recognition or tracking may provide input information. The wearable system may be configured to interpret and track hand gestures such as stop, grab, hold, grab, left, right, grab, or stop. In one configuration, the user might want to flip through emails, a calendar, or even do a “fist bump”. With another player or person. The wearable system may be set up to use a minimal amount of hand gestures, which may or not be dynamic. The gestures could be static gestures such as open hand for stop, thumbs down to not ok, thumbs up to ok; or a flip of the hand right, left, or right for direction commands.

Eye tracking is another input. It is used to track the user’s desire to have the display technology render at a particular depth or range. In one embodiment, vergence of the eyes may be determined using triangulation, and then using a vergence/accommodation model developed for that particular person, accommodation may be determined.”

The FIG. 9 example of a wearable system 900 shows how cameras work. Nine can have three types of cameras. One pair may be a passive SLAM or relative wide FOV camera, and another pair oriented in front to record stereo imaging process 940. The other pair could be positioned in front to capture hand gestures as well as totem/object tracking. The stereo imaging system 940, which includes the FOV cameras and the pair for the stereo process 940, may also include the outward-facing imaging systems 464 (shown at FIG. 4). Eye tracking cameras can be included in the wearable system 900. They may be part of an inward facing imaging system 462 as shown in FIG. 4) are oriented towards the eyes of the user to calculate eye vectors and other information. One or more textured light projectsors (such infrared (IR), projectors) may be included in the wearable system 900 to add texture to a scene.

“FIG. “FIG. The user can interact with a totem in this example. Multiple totems may be owned by the user. One totem could be used for social media, while another can be used for games. Block 1010 may be where the wearable system detects a movement of a totem. The outward-facing system may detect the movement of the totem or sensors may (e.g., haptic gloves, image sensors, hand tracking cameras, head pose sensors, etc )

The wearable system, based at least partially on the detected gesture, eye position, head pose or input through the device, detects the totem’s position, orientation and/or movement relative to block 1020. The reference frame could be a set or map points that the wearable system uses to translate the user’s movement to an action or command. Block 1030 is where the system maps the user’s interaction to the totem. The system calculates the input of the user at block 1040 based on the mapping of user interaction to the reference frame 1020.

To move a totem, or other physical object, the user can turn a page by moving it back and forth. This could be used to indicate moving to a new page or from one user interface (UI), screen to another. Another example is that the user can move their head or gaze to view different virtual or real objects in FOR. The input may be the real or virtual object if the user’s gaze is fixed on a specific real or virtual object for more than a threshold amount of time. In some implementations, the vergence of the user’s eyes can be tracked and an accommodation/vergence model can be used to determine the accommodation state of the user’s eyes, which provides information on a depth plane on which the user is focusing. The wearable system may use raycasting techniques in some instances to determine real and virtual objects that are located along the direction of the user?s eye or head poses. The ray casting techniques may include casting pencil-thin rays that have a very small transverse width, or casting rays that are large in transverse (e.g. cones or fustums).

“The display system described herein may project the user interface (such as the display 220 shown in FIG. 2). You can also display it using other methods, such as one or several projectors. Projectors can project images onto physical objects such as a globe or canvas. One or more cameras may be used to track interactions with the user interface (e.g. using the inward-facing image system 462 or outward-facing imagery system 464)

“FIG. “FIG. 11. is a process flow diagram for an example of a 1100 method of interfacing with a virtual user interface. The wearable system described in this article may perform the method 1100.

“At block 1110 the wearable system might identify a specific UI. The user may choose the type of UI they want. A user input may be used to tell the wearable system which UI should be filled. Block 1120 may be used to generate data for the virtual interface. Data such as the shape, structure, and confines of the virtual UI may be generated. The wearable system could also determine the coordinates of the user?s physical location to display the UI relative to that location. If the UI is body-centric, for example, the wearable device may determine the coordinates and head position of the user so that a ring UI or planar UI can either be displayed around them or on a wall in front of them. The map coordinates of the user?s hands can be determined if the UI is hand-centric. These map points can be obtained from data collected through FOV cameras, sensory input or any other type data.

“Block 1130: The wearable system can send data to the display either from the cloud, or from a local database. Based on the data sent, the UI will be displayed to the user at block 1140. A light field display, for example, can project the virtual interface into the eyes of one or both users. The wearable system can wait for a user command to create more virtual content at block 1150 once the virtual UI is created. The UI could be a ring that wraps around the body of the user, for example. The wearable system might then wait for the command (e.g., a gesture, head or eye movement, etc.). If the command is recognized (block 1106), virtual content associated to it may be displayed to users (block 1170). The wearable system might wait for the user to make hand gestures before mixing multiple steam tracks.

U.S. Patent Publication No. 0016777 also describes additional examples of AR systems, UI and user experiences (UX). 2015/0016777 is incorporated herein by reference in its entirety.

“Examples of Objects in Field of Regard (FOR), and Field of View(FOV)”

“FIG. “FIG. 12” schematically illustrates a virtual object in a view field (FOV) or a view field (FOR). Referring to FIG. 4 The FOR is a part of the environment that can be perceived by the wearable system. FIG. FIG. 12. The FOR 1200 can include a group of objects (e.g. The FOR 1200 can contain a group of objects (e.g. 1210, 1220 and 1230), which can all be perceived by the user using the wearable system. For 1200, the objects may be physical or virtual. The FOR 1200 user may have a physical object like a chair, sofa, wall, or other items. Virtual objects can include objects from the operating system such as a recycle bin to delete files, a terminal for entering commands, a file manager or file manager for accessing files and directories, icons, menus, applications for audio or video streaming, notifications from operating systems, etc. Virtual objects can also include objects within an application, such as avatars, virtual objects used in games, graphics, or images, and so on. Virtual objects can exist as both an operating system object or an object within an application. The wearable system may add virtual objects to existing physical objects in some instances. The wearable system could add a virtual menu to a room with a TV. This virtual menu would allow the user to change or turn on the channels using the wearable.

Virtual objects can be three-dimensional (3D), one-dimensional (1D), or two-dimensional (2D). FIG. 16. The virtual object could be a 3D coffee cup 1636, which may act as a virtual control for an actual coffee maker. A virtual object can also be a 2D graphic representation of a clock 1634, which displays the current time to the user. One or more virtual objects can be displayed within, or associated with, another virtual object in some implementations. FIG. 13 shows the virtual coffee cup 1636 inside the user interface plane 1504, although it appears that the virtual coffee cup is 3D within this virtual 2D space.

“The objects found in the FOR of the user can be part a world map described with reference to FIG. 9. Data associated with objects (e.g. location, semantic information, properties, etc.) You can store information in many data structures, including arrays, lists and trees, as well as hashes, graphs and other semantic information. If applicable, the index of each object stored can be determined by its location. The data structure might index objects using a single coordinate, such as how far the object is from a fiducial location (e.g. how far the object is from the fiducial positions top, bottom, left, or depth wise). The user’s position, such as their head position, may determine the fiducial position. A virtual or physical object, such as a target interactable item, may determine the fiducial position. This allows the user to collapse the 3D space into a 2D user interface, where virtual objects are placed according to their distance from the fiducial location.

“In the FOR 1200, the area that a user perceives at any given time is called the FOV 1250. (e.g. the FOV 1250 could include the FOR portion that the user is currently looking towards). FIG. 12 shows the FOV 1250. FIG. 12. The FOV 1250 schematically illustrated with dashed line 1252 is FIG. The FOV 1250 can be used to perceive multiple objects, including the object 1242, object 1244 and a portion the object 1230. The size and optical characteristics of a wearable device can affect the FOV. An AR display might have optics that provide AR functionality only when the user is looking through a specific portion of it. The FOV 1250 could be the angle the user can see when they look through AR displays such as the stacked waveguide assembly (FIG. 4) or the planarwaveguide 600 (FIG. 6).”

“As the user changes their pose (e.g., head position or eye pose), the FOV 1250 and objects within it 1250 may change. In FIG. 1, the map 1210 is in the FOV of the user. 12. The map 1210 can be seen if the user is looking at it. If the object 1230 is not in the FOV 1250 of the user, then the map 1210 could move into their FOV 1250. The wearable system can keep track of objects in the FOR 1200 and FOV 1250, as will be explained here.

“Examples for Interactable Objects”

“In FIG. FIG. 12 shows how a user can interact directly with a subset the objects within the user’s FOR1200. Sometimes, this subset of objects is called interactable objects. Some implementations may allow interactable objects to include all objects (virtual or physical) within the user’s environmental. In other implementations, however, only a small portion of objects can be included in the environment.

“The wearable system is capable of identifying a subset of interactable objects (e.g. 1242, 1244 and 1230 are all within the user’s FOV 1250. Because the user can perceive them currently, and can choose to activate, move, activate, or retrieve information about them, the subgroup of interactable objects within the FOV is often referred to as selectable. The user’s FOV may change if he moves his head, body or eyes. Some objects will not move outside the FOV and some objects will stay inside the FOV. Other objects will move within the FOV and become selectable. The wearable system can therefore update the FOV subgroup of interactable objects based on user’s body, head or eye position.

The wearable system could identify a target object that is interacted with within the user’s FOV. The user may want to interact with the target interactable objects or it could be the object the wearable device expects the user to interact with (e.g. the interactable object towards which the user is looking, or the one closest to the center of the FOV). The user’s pose, preference or location can all help identify the target interactable object. The wearable system can, for example, choose the object nearest to the center of its FOV as the target interactable item. To make the target interactable object, the wearable system can choose to use either the leftmost or rightmost object in the FOV. The wearable system can also use the inward-facing imagery system 462 (shown at FIG. The IMUs can be used alone or together to determine the user?s gaze direction. The wearable system will identify any object that collides with the user’s gaze and mark it as the target interactable item.

“In certain implementations, the AR software can automatically position the target interactable objects so that their normal faces the user. A virtual TV screen might initially be oriented toward the ceiling. The AR system will automatically rotate the virtual television screen so that it faces the user once it has determined that the user is looking towards the screen.

“Examples for a Focus Indicator”

The wearable system might assign a focus indicator (or a color) to the target object in order for the user to be able to perceive it more easily. The user can see the focus indicator. The focus indicator could include a halo or a color that causes the target object to appear smaller or larger depending on whether it is selected. It can also be used to draw attention. You can also use audible and tactile effects like vibrations, ringtones, beeps, or other visual effects to focus your attention.

“In certain embodiments, the wearable device may identify an object as the target interactable objects based upon the rules herein. Then it could change the target interactable objects to another object based the user’s poses. The user’s pose may change and the focus indicator could move from one object towards the other.

The cursor may correspond to the current position of the wearable system. You can choose from a number of different shapes for the cursor, including a cone, beam of ray or a reticle. An arrow, a reticle or arrow. The cursor can be displayed in the same format as the focus indicator. The cursor could have the same visual, auditory, or tactile effect of the focus indicator. The cursor could be a reticle that corresponds to the user?s head position. Another example is that the cursor could be shaped like an arrow and correspond to the current position of a user input device. The cursor can change its position or move according to the user’s actions. As the user moves about, the cursor can point to one or more objects or empty spaces in his environment. FIG. 16. The AR system can move cursors on the virtual user interface 1514 from one position to another, such as from 1620 to 1624 or 1622 to 1620.

“The cursor can be presented by the wearable system in addition or as an alternative to the focus indicator. FIG. FIG. 15 shows an example of how the wearable system could display a reticle, which can correspond to the user?s direction of gaze, or a light blue circle as a focus indicator. The cursor can be used as an example of the focus indicator in some implementations. The cursor may be used to indicate the focus indicator when the user stares at a virtual object. These visual indicators may indicate the target object the user wants to interact with as well as the current position of the user.

“Examples of Interactions with Interactable Items”

“The wearable system allows users to interact with interactable objects in their FOR 1200, and more specifically with objects in their FOV 1250. The virtual object 1230 could be a graph showing the price change over time. The virtual object 1230 can be selected by the user to allow them to interact with it. This includes stock quotes, buying or selling stock, information about the company and more. The wearable system can display menus, toolbars and other information associated with the virtual objects, which allow the user to perform different actions (e.g., getting a stock quote).

The user can interact with objects within the FOV by opening menus or toolbars, selecting them, moving them, choosing new objects or choosing from a set. You can interact with interactable objects by using hand gestures (see FIG. 466 for an example of user input device 466). 4.) The user may interact with the interactable objects using hand gestures (see e.g. user input device 466 in FIG. Interactable objects can also be accessed by the user using their head, eyes, hands, feet, or other body positions, such as gazing at or pointing an arm at an object, tapping on a touch pad, touching a capacitive button, pressing a key on a keyboard or game controller (e.g., a 5-way d-pad), pointing a joystick, wand, or totem toward an object, blinking for a specified number of times, or tapping foot The AR system can initiate a selection event by using the hand gestures and other poses, such as tapping foot, blinking eyes, or tapping the foot.

“The AR system can assign a focal indicator to a target interactable item in the user’s FOV upon initiation of the selection events using the rules herein. FIG. 12. The AR system could assign a focus indicator for the object 1244, because it is closest to the FOV’s midpoint.

“During the selection event, a user can modify the target object interactable by using any of the hand gestures described in this document. FIG. 12 shows an example of this. FIG. 12 shows how the AR system can change the target object by allowing the user to swipe left on a touch screen. The AR system can transport the visible focus indicator between the object 1244 and the object 1230 in accordance with its settings.

Click here to view the patent on Google Patents.