Software Technology for “Interactions using 3D virtual objects using poses, multiple-DOF controllers”

Metaverse – James M. Powderly, Savannah Niles, Frank Alexander Hamilton, IV, Marshal Ainsworth Fontaine, Paul Armistead Hoover, Magic Leap Inc

Abstract for “Interactions using 3D virtual objects using poses, multiple-DOF controllers”

A wearable system may include a display system that displays virtual content in three-dimensional space, a device for user input to receive input and one or more sensors that detect the user’s pose. Based on context information, the wearable system allows for various user interactions with objects within the user’s environment. The contextual information can be used to adjust the size of the aperture of a virtual cone in a cone cast. Another example is that the wearable system can adjust how much movement virtual objects have in response to an actuation of a user input device. This is based on contextual information.

Background for “Interactions using 3D virtual objects using poses, multiple-DOF controllers”

Modern computing and display technology have made it possible to create systems that can be called “virtual reality”, “augmented reality?”, or “mixed reality?” Digitally reproduced images and portions of them are presented to users in a way that makes them appear real or can be perceived as such. A virtual reality (or?VR?) scenario usually presents digital or virtual information without transparency to any other real-world visual input. An augmented reality (or?AR?) scenario typically presents digital or virtual information as an enhancement to the visualization of the real world around the user. A mixed reality (or?MR?) scenario involves merging real and digital worlds to create new environments in which physical and virtual objects coexist and interact in real-time. It turns out that the human visual system is complex. Therefore, it is difficult to create a VR, AR or MR technology that allows for a natural-feeling, rich presentation and interaction of virtual images elements with real-world or virtual imagery elements. The systems and methods described herein address various issues related to VR, AR, and MR technology.

“In one embodiment, there is a system that allows users to interact with objects on a wearable device. The system includes a display system for a wearable device that presents a three-dimensional (3D), view to the user and allows interaction with objects within a field (FOR). The FOR may include a part of the environment that can be perceived by the user through the display system. A sensor that acquires data about the user’s pose and a processor communicating with the hardware processor and the display system can be included in the system. The hardware processor can be programmed to: Determine a user’s pose based on data from the sensor; start a cone casting operation on FOR objects. This involves casting a virtual cone with an opening in a direction determined at least partially by the user’s pose; analyze context information about the user’s environment; adjust the aperture of virtual cone based at minimum partly on that contextual information; and render visual representations of the cone for the cone casting.

“Another embodiment discloses a method of interfacing with objects for wearable devices. The method involves receiving a selection from a target virtual objects that is displayed to a user in a first position in 3D space. It also includes receiving an indication of movement for the target object. Based at least partly upon the context information, calculating a multiplier for movement of the target object. Displaying the target virtual objects at a second location, determined at least in part by the first position and the movement quantity to the user.

“Another embodiment discloses a system that allows objects to interact with a wearable device. The display system for a wearable device is configured to show a 3D view to the user. This view can include a target virtual object. A hardware processor can be connected to the system. The hardware processor can be programmed to: Receive an indication of movement for the target object; analyze context information associated with target object; calculate a multiplier that will be applied to the object’s movement based at minimum partly on the indicator of movement and multiplier; display the target object at a second location by the display system, based at most in part on the first and second positions.

“Details about one or more implementations are provided in the accompanying drawings as well as the description. The claims, drawings, and description will reveal other features, aspects, or advantages. This summary and the detailed description below do not attempt to limit or define the scope of the inventive subject matter.

“Overview”

A wearable system can display virtual content in an AR/VR/MR setting. A wearable system allows a user interaction with virtual or physical objects within the environment. The user can interact with objects by moving them around, using poses, or by activating an input device. The user can move the input device for a distance, and the virtual object will follow that user and move the same distance. Cone casting may also be used by the wearable system to enable a user to target or select the virtual object using poses. The wearable system is able to target and select virtual objects within the user’s field by moving his head.

If objects are placed far apart, these approaches can lead to user fatigue. To move the virtual object from one location to another or reach the desired object, the user must move the input device or increase body movement (e.g. increasing arm or head movement) to achieve the desired distance. It can also be difficult to accurately position a distant object because it is hard to see tiny adjustments at far away locations. However, users may prefer to interact with objects closer together and have more precise positioning.

The wearable system is able to automatically adjust the user interface operations according to contextual information. This helps reduce user fatigue and facilitate dynamic interactions with the system.

The wearable system can update the cone’s aperture in cone casting using contextual information. This is an example of dynamic user interaction based upon contextual information. The wearable system can automatically reduce the cone aperture if the user points her head in a direction that has a high density or selectable objects. The wearable system can also automatically increase the cone aperture if the user points her head in a direction with low objects density. This could allow for more objects to be included within the cone, or decrease the movement required to overlap it with virtual objects.

The wearable system may also provide a multiplier that can convert the movement of the input device (and/or user’s movements) into a greater movement of the virtual objects. The user doesn’t have to move far to move the virtual objects to their desired locations if they are located far away. The multiplier can be set to 1 when the virtual object is within reach of the user, e.g., if it is within their hand reach. The wearable system allows for one-to-one interaction between the user’s movements and the movement of the virtual object. This could allow the user to interact more precisely with the virtual object nearby. Below are examples of user interactions that use contextual information.

“Examples for 3D Display of a Wearable Device”

A wearable system, also known as an AR (augmented reality) system, can present virtual images in 2D and 3D to the user. Images can be still images, frames from a video or combinations of both. A wearable device can be included in the wearable system that allows for interaction with VR, AR, and MR environments. A wearable device may be a head-mounted (HMD) device.

“FIG. FIG. 1 shows an illustration of a mixed-reality scenario, with certain virtual objects and some physical objects that can be viewed by a human. FIG. FIG. 1. A MR scene 100 depicts a user of MR technology. The scene shows a person imagining a park-like setting 110 with people, trees, buildings, and a concrete platform 120. The MR technology user also perceives these items. A robot statue 130 stands upon the real-world platform 120. A cartoon-like avatar character 140 is flying past, which appears to be the personification of a Bumble Bee.

It may be beneficial for 3D displays to have an accommodative response that corresponds to the virtual depth of each point. The human eye can experience accommodation conflicts if the accommodative response of a display point doesn’t correspond to its virtual depth as determined by stereopsis and binocular depth cues. This could lead to unstable imaging, headaches, and even complete absence of surface depth.

Display systems that provide images corresponding to a variety of depth planes to viewers can offer VR, AR, and MR experiences. Each depth plane may have different images. This allows the viewer to see depth cues by observing differences in image features or the accommodation required to bring them into focus. These depth cues, as discussed in the past, provide credible perceptions about depth.

“FIG. “FIG. 2” illustrates an example wearable system 200. The wearable system 200 comprises a display 220 and various electronic and mechanical modules that support display 220. Display 220 can be connected to frame 230. This frame is usable by the user, wearer or viewer 210. The display 220 may be placed in front of the user’s eyes 210. The display 220 can show AR/VR/MR content. A head-mounted display (HMD), which is attached to the display 220, can be used. A speaker 240 can be attached to the frame 230 in some embodiments and placed next to the user’s ear canal. (In some embodiments, an additional speaker, not shown here, is placed adjacent to the user’s other ear canal to allow for stereo/shapeable sounds control).

“The wearable device 200 may include an outward-facing imaging unit 464 (shown at FIG. 4) that observes the environment around the user. A wearable system 200 may also include an inward-facing image system 462 (shown at FIG. 4), which can track the eye movements. Inward-facing imaging systems can track the movements of one or both eyes. The frame 230 may have an inward-facing image system 462 attached. It may also be electrically connected to the processing modules 265 and 270. These modules may process image data acquired by the inward facing imaging system to determine the pupil sizes or orientations, eye movements, or user’s eye position 210.

“As an example the wearable device 200 can use either the outward-facing or inward-facing imaging systems 464 to capture images of the user’s pose. Images can be still images, frames, or combinations of images.

“The display 220 may be operatively coupled 250 such as by wired connectivity or wireless connectivity to a local processing module 260. It can be fixedly attached 230 to a helmet/hat worn by the user, embedded into headphones or other removable attachments to the user 210 (e.g. backpack-style configuration or belt-coupling style configuration).

“The local processing module and data module 265 may include a hardware processor as well as digital memories such as flash memory and non-volatile memory. Both of these may be used to aid in data processing, caching, storage, and processing. Data may include data from sensors (which could be, e.g.., operatively coupled with the frame 230, or otherwise attached to user 210), as well as microphones, inertial measuring units (IMUs), compasses and global positioning system units (GPS), radio devices or gyroscopes. Alternatively, data can be acquired using remote processing module (270) or remote data repository (280) for passage to display 220. Local processing and data modules 260 can be operatively connected via communication links 262 and 264 to remote processing module (270) or remote data repository (280), so that remote modules are available for local processing and data 260. Remote processing module 280 or remote data repository 282 may also be operatively coupled.

“In some cases, the remote processing module (270) may include one or more processors that are capable of processing data and/or images. The remote data repository 280 could include a digital storage facility that can be accessed via the internet or another networking configuration. Resource configuration. Some embodiments store all data and perform all computations in the local processing module. This allows for fully autonomous use from remote modules.

The human visual system is complex and it is difficult to perceive depth accurately. It is possible that viewers may perceive an object as three-dimensional due to vergence and accommodation. The movement of vergence (i.e. rolling of the pupils towards or away from one another to converge the lines in the eyes to fixate on an object) is closely related to focusing (or?accommodation?). The lenses of the eyes. Under normal conditions, changing the focus of the lenses of the eyes, or accommodating the eyes, to change focus from one object to another object at a different distance will automatically cause a matching change in vergence to the same distance, under a relationship known as the ?accommodation-vergence reflex.? Under normal conditions, a change of vergence will also trigger a matching change to accommodation. Display systems that match accommodation and vergence better may create more comfortable and realistic simulations of three-dimensional imagery.

“FIG. 3. illustrates aspects of a method for simulating three-dimensional imagery using multiple depth plans. Referring to FIG. FIG. The eyes 302 & 304 use particular accommodated states to focus objects at different distances along z-axis. A particular accommodated condition may be associated with one of the depth planes 306, and has an associated focal distance. This means that objects or parts of objects in particular depth planes are in focus when the eye moves in that accommodated state. Three-dimensional imagery can be created in some embodiments by providing different images for each eye 302 and 304 and different versions of each image corresponding to each depth plane. Although the fields of view are shown separately for illustration purposes, it is possible for the eyes 302 or 304 to overlap as the distance along the Z-axis increases. For illustration purposes, the contours of depth planes are shown flat. However, it is possible for them to curve in space so that all of the features of the depth planes are focused with the eye in an accommodated state. The theory does not limit the possibility that the human eye can perceive depth in a finite number depth planes. Therefore, it is possible to create a convincing simulation of depth perception by showing the eye different images that correspond to each of the limited depth planes.

“Waveguide Stack Assembly”

“FIG. “FIG. 4 shows an example of a waveguide stack that outputs image information to a user. Wearable system 400 may include a stack of waveguides or stacked waveguide assemblies 480. These waveguides can be used to provide three-dimensional perception for the eye/brain by using a plurality waveguides 432, 434, 436, 436, 438, 438, 438, 438, 438, 4400 b. The wearable system 400 could be compared to the wearable device 200 in FIG. 2 with FIG. 4, schematically showing more parts of the wearable system 200. In some embodiments, for example, the waveguide assembly (480) may be integrated into FIG. 2.”

“With reference to FIG. 4. The waveguide assembly 480 could also contain a number of features 458, 456, 454, or 454, 452, between the waveguides. The features 458, 456, 454, and 454, 452, may be lenses in some embodiments. Other embodiments may not allow for the features 458, 456, 454 or 454, 452 to be lenses. They may be spacers, such as cladding layers, or structures that form air gaps.

Waveguides 432, 434, 436, 438, 438, 438, 440 b and the plurality lenses 458, 456, 436, 438, 438, 438, 438, 438, 438, 438, 438, 438, 440 b may be set up to transmit image information to the eye at different levels of light ray divergence or wavefront curvature. Each waveguide level can be associated with a specific depth plane, and may be configured so that it outputs image information that corresponds to that depth. The image injection devices 420 to 432, 424, 426 and 428 can be used to inject image information into waveguides 440, 438, 436, 434, 434, 432, 428. Each of these devices may be designed to distribute light across the waveguides, for output towards the eye 410. The light exits from an output surface of image injection devices 420-422, 424-426, 428, 428 and is injected into the waveguides’ corresponding input edges 440 b to 438 b. 436 b. 434 b. 432 b. One beam of light, such as a collimated beam, may be injected into each waveguide in order to produce an entire field of collimated collimated beams directed towards the eye 410 at specific angles and divergence corresponding to the depth plan associated with that waveguide.

“In some embodiments the image injection devices 422, 424 and 426 are discrete displays that produce image information to be injected into a waveguide 440, 438, 436, 434, 434, 432, and 432 b. Other embodiments include the image injection device 420 to 422, 424 and 426. These devices may be output ends of a single multiplexed monitor that can pipe image information through one or more optical conduits (such fiber optic cables) into each of the image injectors 420, 422, 422, 424 and 426.

“A controller 460 controls operation of the stacked wafer assembly 480 and image injection devices 424, 426. 428. The controller 460 contains programming (e.g. instructions in a nontransitory computer-readable media) that controls the timing and provision image information to waveguides 440, 438, 436, 436, 434, 434, 434, 432, 432 b. The controller 460 can be either a single integrated device or a distributed system connected via wired or wireless communication channels. The controller 460 could be part of one or more processing modules 260 and 270 (illustrated at FIG. 2) in certain embodiments.

“The waveguides 438 b and 436 b can be set up to transmit light through total internal reflection (TIR) within their respective waveguides. Waveguides 440, 438, 436, 434, 434, 432 and 432 b can be either planar or curved, with major top- and bottom surfaces, and edges that extend between them. The illustrated configuration of the waveguides 440, 438, 436, 434, and 432 B may all include light extracting optical components 440a,436 b., 436, b., 434, b., 432, b. These elements are designed to extract light from a waveguide by redirecting light within the waveguide, out of each waveguide to provide image information to the eye 434 a., 432a. Outcoupling optical elements and extracted light can also be called outcoupled. The waveguide outputs a beam of light at the locations where the light from the waveguide strikes a light redirecting component. For example, light extracting optical elements (440a, 438a, 436, 434 a., 432a, 442 a., 432) may be reflective or diffractive optical components. For ease of description and drawing, the light extracting optic elements are shown at the bottom of the waveguides. The light extracting optical element 440a, 438a, 436, 434a, and 432a can be formed in a layer made of material attached to a transparent substrate. This will form the waveguides. Other embodiments allow the waveguides 440, 438, 436, 434, b and 432 to be formed in a single piece of material. The light extracting optical element 440a,438 a., 436, 436, 434, b., 432b. may also be formed on a surface, or within that piece.

“With reference to FIG. “With continued reference to FIG. The waveguide 432b closest to the eye could be set up to deliver collimated lighting, such as that injected into waveguide 432b to the eye 410. The collimated light could be representative of optical infinity focal plan. The next waveguide may send out collimated beams through the first lens 452 (e.g. a negative lens) before reaching the eye 410. The first lens 452 could be set up to create a convex wavefront curvature, so the eye/brain perceives the light coming from the next waveguide 434 b as coming in from a first focal point closer inward towards the eye 410 from optical infinite. The third up waveguide 436b also passes its output light through the second lens 454 and first lens 452, before reaching the eye. Combining the optical power of the first two lenses 452 and 444, may create an incremental amount of wavefront curve so that the eye/brain perceives light coming out of the third waveguide 436 B as coming from a second focal point that is closer inward towards the person than light from the next waveguide, 434 b.

The other waveguide layers (e.g. waveguides 438b, 440b) and lenses (e.g. lenses 456, 458) are similarly configured. The highest waveguide, 440b in the stack, sends its output through all lenses between it, the eye, for an aggregate focal power that is representative of the nearest focal plane to the person. A compensating lens layer (430) may be placed at the top to offset the effect of the lens stack 458, 456, 454 and 452 on viewing/interpreting light from the world 470. This configuration can provide as many perceived focal points as possible, regardless of the number of waveguide/lens pairs. The light extracting optical elements and the focusing parts of the lenses can be either static or not dynamic. Alternate embodiments may include one or both of these elements, with electro-active features.

“With reference to FIG. “With continued reference to FIG. 4, the light extracting optic elements 440a, 438a, 436, 436a, 434a, 443, 434a, and 432a can be configured to redirect light out of respective waveguides and to produce this light with the correct amount of divergence/colimation for the particular depth plane associated to the waveguide. Waveguides with different depth planes can have different configurations for light extracting optical elements. These elements may output light with different amounts of divergence or collimation depending on the depth plane. As discussed above, light extracting optical element 440 a and 438 a are volumetric or surface features that can be set up to produce light at certain angles. The light extracting optical components 440 a. 438 a. 436 a. 434 a. 432 a. 432 a. may be volume or surface holograms and/or diffraction gratings. U.S. Patent Publication No. 2015/0178939 published June. 25th of June 2015, is included by reference in its entirety.”

“In certain embodiments, light extracting optical elements (440 a), 438 a. 436 a. 436 a. 434 a. 432 a. 432 a) are diffractive features that create a diffraction patterns or?diffractive optic element?. Also known as a “DOE” The DOE should have a low diffraction efficiency, so that only a small portion of the beam is deflected toward the eye (410 at each intersection). The rest of the beam continues to travel through the waveguide via total internal reflect. This allows for the division of the light carrying image information into several related exit beams, which exit the waveguide at a variety of locations. The result is a relatively uniform pattern of exit emission towards the eye 304 in this particular collimated beam that bounces around within a waveguide.

“In some embodiments, one of the DOEs can be switched between?on and?off? state in which they actively diffract and?off?” They do not significantly diffract. A switchable DOE could be a layer of polymer-dispersed liquid crystal. The microdroplets may be arranged in a diffraction pattern in the host medium. In this case, the refractive indice of the microdroplets can either be changed to match the host material’s or to an index that is not as high as the host medium’s (in which case they actively diffract incident sunlight).

“In some embodiments the distribution and number of depth planes (or depth of field) may be dynamically adjusted based on the pupil sizes of the viewers eyes. The size of a viewer’s pupils may affect the depth of field. The depth of field may increase as the size of the pupil decreases. This means that a plane that isn’t discernible due to its location being beyond the depth-of-focus of the eyes can become more discernible. With a reduction in pupil size, and in proportion with an increase in depth, the field depth will also change. The decreased pupil size may also affect the number of depth planes that are used to show different images to the viewers. A viewer might not be able see details of both a depth plane and another depth plane at the same pupil size. This could lead to confusion. However, these two depth planes can be in focus simultaneously to the user at a different pupil size.

“In certain embodiments, the display may alter the number of waveguides that receive image information based on the determination of pupil size and orientation or upon receiving electrical signals indicating particular pupil size and orientation. If the user is unable to distinguish between the depth planes of two waveguides, the controller 460 can be programmed or configured to stop providing image information to that waveguide.

This may be advantageous as it reduces the processing load on the system and increases the system’s responsiveness. If the DOEs of a waveguide can be switched between the on or off states, they may be switched to off when the waveguide receives image information.

“In certain embodiments, it might be desirable for an exit beam to meet the condition that its diameter is less than the diameter the viewer’s eye. This condition can be difficult due to the variable size of viewers’ pupils. This condition can be met in some embodiments by changing the size and shape of the exit beam according to the size of the pupil. The size of the exit beam could decrease as the pupil grows, for example. In certain embodiments, the exit beam size can be adjusted using a variable aperture.

The wearable system 400 may include an outward-facing image system 464 (e.g. a digital camera) that captures a portion the world 470. This area of the world 470 is often referred to by the term “field of view” (FOV), and the 464 imaging system is sometimes called an FOV camera. The field of regard (FOR) is the entire area that is available for viewing and imaging by a viewer. There may be 4 FOR. The FOR may include 4? Other contexts may require the wearer to move more tightly, so the FOR of the wearer may have a smaller solid angle. Images taken with the outward-facing imaging device 464 can be used for tracking gestures (e.g. hand or finger gestures), detecting objects in the world, 470, and so on.

The wearable system 400 may also include an inward facing imaging system 466 (e.g. a digital camera) that monitors the movements of the user such as their eye movements and facial movements. To determine the size and/or orientation the pupil of the eyes 304, the inward-facing imaging device 466 can be used. Inward-facing imaging system 466 may be used to capture images that can be used for biometric identification (e.g. via iris identification) or to determine the user’s gaze direction. One camera may be used for each eye in some embodiments. This allows each eye to determine its pupil size and position independently. Each eye can then be presented with image information that is dynamically tailored to it. Other embodiments allow for the determination of the pupil diameter and orientation of one eye 410. This allows the presentation of image information to each eye to be dynamically tailored. The inward-facing imaging device 466 can analyze the images to determine the user?s mood or eye position. This information can then be used by the wearable technology 400 to determine which audio or visual content to present to the user. Wearable system 400 can also detect head pose (e.g. head position or orientation) by using sensors such IMUs, accelerometers and gyroscopes.

The wearable system 400 may include a user input device 466 through which the user can input commands 460 to the controller to interact with the wearable systems 400. The user input device 466 may include a trackpad and touchscreen as well as a joystick, multiple degree of freedom (DOF), controller, a capacitive sensor device, a game controller (D-pad), a keyboard, mouse, a direction pad (D-pad), a keyboard, a mouse (K-pad), a wand or haptic device), and a totem (e.g. functioning as a virtual input device). Multi-DOF controllers can detect user input in any or all translations (e.g. forward/backward or up/down), or rotations (e.g. yaw/pitch, roll, etc.) of the controller. Multi-DOF controllers that support translation movements are called 3DOF, while multi-DOF controllers that support translations and rotations can be called 6DOF. Sometimes, the user may swipe or press a touch-sensitive input device with a finger (e.g. thumb) to input information to the wearable systems 400. During the use of wearable system 400, the user may hold the user input device 466 in their hand. The user input device 466 may be wired or wirelessly connected to the wearable system 400.

“FIG. “FIG.5” 5 illustrates an example of exit beams produced by a waveguide. Although one waveguide is shown, it can be seen that many waveguides within the waveguide assembly480 could function in a similar fashion. The waveguide 432b is infected at the input edge 432c of the waveguide 432, and the light propagates through the waveguide 422, b via TIR. A portion of the light 520 exits the waveguide as the exit beams 510 at points where it impedes on the DOE 432. Although the exit beams 510 appear to be substantially parallel, they can also be directed to propagate to eye 410 at an angle (e.g. forming divergent exit beacons), depending upon the depth plane associated to the waveguide 432b. You will see that the exit beams 510 are substantially parallel. This could be a sign of a waveguide that uses light extracting optical elements to outcouple light, forming images that appear to be set at a deep plane that is far away (e.g. optical infinity). Waveguides with other light extracting optical elements or waveguides may produce a divergent exit beam pattern. This would require the eye to adjust to a greater distance to focus the image on the retina. The brain would interpret this as light coming from a farther distance than the eye 410.

“FIG. “FIG. An optical system may include a waveguide device, an optical coupler subsystem that optically couples light to or from waveguide apparatus, as well as a control subsystem. Multi-focal volumetrics, images, and light fields can all be generated by the optical system. One or more primary planar wavesguides 632a can be included in the optical system (only one is shown at FIG. 6) and one (or more) DOEs 632b that are associated with at least one of the primary waveguides (632 a). The planar waveguides 632b can be identical to those waveguides 432b, 434b, 436b, 436b, 438b, and 438b. 4. An optical system could use a distribution waveguide device to transmit light along a first direction (vertical or the Y-axis, as shown in FIG. 6.) and increase the light’s effective escape pupil along the first direction (e.g., the Y-axis). Distribution waveguide apparatus could include, for instance, a distribution waveguide 622b and at most one DOE 622a (illustrated with double dash-dot lines) that is associated with the distribution waveguide 622b. The distribution planar wavesguide 622b could be identical or similar to the primary planar waveguide 642 b in at least some aspects, but with a different orientation. The DOE 622 a could be identical or similar to the DOE 632. The distribution planar waveguide 622b and DOE 622a could be made of the same materials, or the primary planar wavesguide 632b or DOE 622 a, for example. FIG. 6 shows embodiments of the optical display 600. 6 can be integrated in the wearable system 200 as shown in FIG. 2.”

“The relayed or exit-pupil extended light may be optically coupled to the distribution waveguide apparatus into one or more primary plane waveguides 632b. The primary planar waveguide 632b can relay light along another axis, but it should be orthogonal to the first axis (e.g. horizontal or X-axis as shown in FIG. 6). The second axis may be non-orthogonal to the first. The primary planar waveguide 632b expands light’s exit pupil along the second axis (e.g. X-axis). The distribution planar wavesguide 622 b, for example, can expand and relay light along the vertical axis. It then passes that light on to the primary planar waveguide 642, which can expand and relay light along its horizontal axis.

“The optical system can include one or more sources for colored light (e.g. red, green and blue laser light)610, which may be optically connected to a proximal tip of a single-mode optical fiber 640. The distal end may be threaded through or received through a hollow tube 642 made of piezoelectric materials. As a flexible, fixed-free cantilever 644, the distal end protrudes out of the tube 642 Four quadrant electrodes can be attached to the piezoelectric tube 642. For example, the electrodes can be plated on the outer, outer, or outer periphery of the tube 642. Unillustrated core electrodes may be found in the core, inner periphery, or inner diameter of the tube 642.

“Drive electronics 655, for example, electrically coupled via wires 665, drive opposing electrodes to bend piezoelectric tube 642 in 2 axes independently. Mechanical modes of resonance are found at the distal tip 644 of the optical fibre. The frequency of resonance depends on the diameter, length, as well as the material properties of optical fiber 644. The fiber cantilever 644 vibrates when the piezoelectric tube 642 is vibrated near the first mode of mechanical resonance. It can also sweep through large deflections by vibrating the 642.

“By stimulating resonance vibration in two directions, the tip 644 of the fiber cantilever is scan biaxially in an area that fills two-dimensional (2D). scan. An image can be created by modulating the intensity of light source(s), 610, in sync with the scan of fiber cantilever 644. U.S. Patent Publication No. 2014/0003762 is incorporated herein in its entirety.

“A component of an optical subsystem that can collimate light from the scanning fibre cantilever 644 can be called an optical coupler subsystem. Mirror surface 648 can reflect the collimated light into the narrow distribution plane waveguide 622b, which contains at least one diffractive optic element (DOE 622a). Relatively to FIG. 6) along the distribution plane waveguide 622b by TIR and intersects repeatedly with the DOE 622a. The DOE 622a is preferred to have a low coefficient of reflection. This can result in a fraction (e.g. 10%) of light being diffracted towards an edge of the primary planar waveguide 642 b at each intersection with the DOE622 a and a fraction to continue its original trajectory along the length of distribution planar waveguide 642 b via TIR.

“At every intersection with the DOE 622a, additional light may be diffracted towards the primary waveguide 632b. The DOE 4 can expand vertically the exit pupil by dividing the incoming beam into multiple outcoupled sets. This vertically expanded light can be coupled out of distribution waveguide 622b to enter the edge the primary planar wavesguide 632b.

“Light entering the primary waveguide 632b can propagate horizontally (relatively to FIG. 6) along primary waveguide 632b via TIR. The light crosses DOE 632a at multiple points and propagates horizontally along at most a portion the primary waveguide 632b via TIR. To produce deflection and focus of light, the DOE 632a can be advantageously designed or configured with a phase profile. This is a combination of a linear and radially symmetric diffractive patterns. The DOE 632a may have a lower diffraction efficiency (e.g. 10%) so that only a small portion of the beam is deflected towards the eye with each intersection of DOE 632a, while the rest propagates through the primary waveguide 632b via TIR.

“A fraction of the light is scattered toward the face of primary waveguide 632b at each intersection of the propagating light with the DOE 632a. This allows the light to escape from the TIR and emerge from primary waveguide 642 b. The DOE 632a radially symmetric pattern of diffraction imparts an additional focus level to diffracted light. This allows the light wavefront to be shaped (e.g., giving a curvature) and the beam to be directed at an angle that matches the intended focus level.

These different paths can result in the light being coupled out of the primary plane waveguide 632b by a multiplicity OFEs 632a at different angles, focus level, and/or fill patterns at exit pupil. To create light fields with multiple depth planes, different fill patterns can be used at the exit pupil. Each layer of the waveguide assembly, or a group of layers (e.g. 3 layers), may be used to produce a particular color, such as red, blue and green. A first set of three layers adjacent may be used to produce red, blue, and green light at a particular focal depth. To produce red, green and blue light at a second depth, a second set of three layers may be used. Multiple sets can be used to create a full 3D and 4D color image lightfield with different focal depths.

“Other Components”

In many cases, other components may be added to or substituted for the components of the wearable systems described above. For example, the wearable system could include one or more haptic components or devices. A user may feel a tactile sensation from the haptic components or devices. The haptic components or devices may be able to provide a tactile sensation such as pressure or texture for touching virtual content (e.g. virtual objects, tools, or other constructs). The tactile sensation could replicate the feel of a real object that a virtual object represents or it may mimic an imagined object or character (e.g. a dragon) that the virtual content represents. Some implementations allow the user to wear haptic components or devices (e.g., a glove for users). Some implementations allow the user to hold haptic components or devices.

“The wearable device may include, for instance, one or more physical objects that can be controlled by the user to enable input or interaction with it. These objects can be called totems. Totems can take the form inanimate objects such as a piece or plastic of metal, a wall, or a table surface. Some totems might not have any input structures, such as keys, triggers or joysticks, trackballs, joysticks, rocker switches, etc. The totem might provide a physical surface and the wearable device may create a user interface that makes it appear the user is on the totem’s surfaces. The wearable system might render an image of a trackpad and keyboard to make it appear that they are on a specific surface of the totem. The wearable system could make a virtual keyboard and trackpad appear on the surface of an aluminum plate that is thin enough to be used as a totem. The rectangular plate itself does not have any trackpad, sensors or physical keys. The wearable system can detect user interaction with the rectangular plate, such as touch or manipulation via the virtual keyboard. FIG. 466 shows the user input device. 4) may be an embodiment of a totem, which may include a trackpad, a touchpad, a trigger, a joystick, a trackball, a rocker or virtual switch, a mouse, a keyboard, a multi-degree-of-freedom controller, or another physical input device. The totem can be used by a user to interact with the wearable device or other users, either alone or in combination.

U.S. Patent Publication No. 0016777 describes “Examples haptic devices or totems usable alongside the wearable devices HMD and display systems of this disclosure.” 2015/0016777 is incorporated herein in its entirety.

“Examples of Wearable Systems, Environments and Interfaces”

A wearable system might use different mapping techniques to obtain high depth of field in rendered light fields. It is important to be able to map out the virtual world and to understand the details of the real world in order to accurately depict virtual objects relative to the real one. By adding new photos that provide information about different features and points of the real world, FOV images can be captured by users of the wearable device. The wearable system can, for example, collect a number of map points (such 2D or 3D points) from users and then find new map points to create a more precise version of the world model. A first user’s world model can be shared (e.g. over a network like a cloud network) with a second user to allow the second user to experience the world around the first user.

“FIG. “FIG.7” is a block diagram showing an example of an MR Environment 700. The MR environment 700 can be configured to receive input. This could include visual input 702 from the wearable system, stationary input 704 like room cameras, sensory input 706 from various sensors, gestures and eye tracking, as well as input from the user input device 466. One or more user wearable devices (e.g. wearable system 200, display system 220), or stationary room systems, (e.g. room cameras, etc.). Wearable systems can be equipped with various sensors such as accelerometers and gyroscopes. They can also use temperature sensors, movement sensors or GPS sensors. To determine the user’s location and other environmental attributes. These data may be further supplemented by information from stationary cameras within the room, which may provide images or other cues from another point of view. The data from the cameras, such as the room cameras or the outward-facing imaging systems cameras, may be reduced to a number of mapping points.

“One or more object recognitions 708 can scan through the received data (e.g. the collection of points), and identify or map points, tag images or attach semantic information to objects using a map database 710. The map database 710 can contain various points and the corresponding objects. Through a network, the various devices and map database can be connected. To access the cloud

Based on this information, and the collection of points in a map database, object recognizers 708a-708n can recognize objects in an environmental. The object recognizers are able to recognize faces, people, windows, walls and user input devices. They can also recognize televisions and other objects within the environment. One or more object recognition devices may have a specific focus on objects with particular characteristics. One example is the object recognizer 708a which can be used to recognize faces and another to recognize totems.

“Object recognitions can be made using many computer vision techniques. The wearable system can, for example, analyze images taken by the outward-facing image system 464 (shown at FIG. 4) to perform scene reconstruction, event detection, video tracking, object recognition, object pose estimation, learning, indexing, motion estimation, or image restoration, etc. These tasks may be performed by one or more computer vision algorithms. Examples of computer vision algorithms that may be used include: Scale invariant feature transformation (SIFT), speeded-up robust features (SURF), oriented fast and rotated BRIEFs (ORB), binary robust and scalable keypoints(BRISK), fast and reliable retina keypoints (FREAK), Viola Jones algorithm, Eigenfaces algorithm, Lucas-Kanade algorithm and Horn-Schunk algorithm. Mean-shift algorithm. Visual simultaneous location and mapping (vSLAM), techniques, Kalman filter and extended Kalman filter. ), bundle adjustment, Adaptive thresholding (and other thresholding techniques), Iterative Closest Point (ICP), Semi Global Matching (SGM), Semi Global Block Matching (SGBM), Feature Point Histograms, various machine learning algorithms (such as e.g., support vector machine, k-nearest neighbors algorithm, Naive Bayes, neural network (including convolutional or deep neural networks), or other supervised/unsupervised models, etc. ), and so on.

“A variety of machine learning algorithms can be used to recognize objects. The HMD can store the machine learning algorithm once it has been trained. Machine learning algorithms include either supervised or unsupervised, such as Ordinary Least Squares Regression, instance-based algorithms like Learning Vector Quantization, and decision tree algorithms like, for instance, Bayesian algorithms. These algorithms can be stored by the HMD once they are trained. Individual models can be tailored to specific data sets in some embodiments. The wearable device may store or generate a base model. A base model can be used to create additional models for a specific data type (e.g. a user in a telepresence sessions), a data set (e.g. a set additional images taken by the user during the session), or conditional situations. The wearable HMD may be configured to use a variety of methods to generate models to analyze the aggregated data. You may also use pre-defined thresholds and data values.

Based on the information and the collection of points in a map database, object recognizers 708a-708n may recognize objects and add semantic information to make them more real. If the system recognizes a set points as a door, it may add semantic information to the objects (e.g. the hinge has a 90-degree movement around the hinge). The system might attach semantic information to a set that is identified as a mirror if it recognizes the points as mirrors. This may include the fact that the mirror’s reflective surface can reflect images from objects within the room. As more data is added to the map database (which can be local or accessible via a wireless network), the system will grow. The information can be sent to one or more wearable devices once the objects have been identified. The MR environment 700 could include information about a California scene. One or more New Yorkers may receive the environment 700. The object recognizers and the other software components can use data from the FOV camera to map the points and recognize objects. This allows the scene to be?passed over’ accurately. To another user who might be in a different area of the world. Environment 700 can also use topological maps for localization purposes.

“FIG. “FIG.8 is a process flow diagram for an example of a method 800 that renders virtual content in relation with recognized objects. A method 800 shows how a virtual scene can be presented to the wearable device user. The scene may not be accessible to the user. The user could be in New York but want to see a scene in California or go on a walk together with a friend who lives in California.

“Block 810 allows the wearable system to receive input from users and other users about the environment. This can be done using various input devices and information already stored in the map database. Block 810 is populated with information from the user’s FOV camera and sensors, GPS, eye track, and other devices. Based on the information at block 820, the system can determine sparse point. These sparse points can be used to determine pose data (e.g. head pose, eye position, or hand gestures). This information can be used to display and understand the orientation and location of objects in the environment. These points can be used by object recognition 708 a-708n to crawl through and identify one or more objects from a map database at block 8.30. The information may then be sent to block 840 and displayed at block 855. The desired virtual scene, e.g. user in CA, may be displayed in the right orientation and position in relation to various objects and other surrounding users in New York.

“FIG. “FIG. 9 is a block diagram for another example of a wearable systems. The wearable system 900 in this example includes a map that may contain map data for the entire world. The map can reside partly locally on the wearable device, while other parts may reside in networked storage locations (e.g., cloud systems) that are accessible via wired or wireless networks. The wearable computing architecture may execute a pose process 910. This uses data from the map to determine the user’s position and orientation. Pose data can be calculated from data that is collected as the user interacts with the system and operates in the real world. Data may include images, data from sensors (such inertial measuring units, which usually comprise accelerometers and gyroscope parts), and surface information relevant to objects in the real and virtual environments.

A sparse point representation could be the result of simultaneous localization (SLAM) or V-SLAM processes. This refers to a configuration in which the input is visual only. It is possible to configure the system to find out not only where the components are located in the world, but also what the world is made from. Pose can be used to accomplish many goals. It may also be used to populate the map with data.

“One embodiment shows that a sparse point location may not be sufficient on its own. Additional information may be required to create a multifocal AR, VR or MR experience. This gap may be filled at most in part using dense representations. Dense representations generally refer to depth map information. This information can be obtained from Stereo 940. In Stereo 940, depth information is calculated using techniques such as triangulation and time-of-flight sensoring. Stereo process 940 may also use image information and active patterns, such as infrared patterns made with active projectors. It is possible to combine a lot of depth map information, and some of it may be combined with a surface representation. Mathematically definable surfaces can be more efficient than large point clouds and provide digestible inputs for other processing devices, such as game engines. The stereo output (e.g., depth map) 940 can be combined with the fusion process 930. Pose could also be used as an input for this fusion process 930. The output of fusion 930 can then be used to populate the map process 920. To create larger surfaces, sub-surfaces can connect, as in topographical mapping. The map then becomes a large mixture of points and surfaces.

Multiple inputs can be used to resolve different aspects of a mixed reality process (960). FIG. 9 illustrates an example of such inputs. 9. Game parameters can be inputs that determine whether the user is playing a game of monster fighting with one or more monsters at different locations, monsters running away or dying under certain conditions (such as if they are shot by the user), walls at various locations or any other objects. To add value to mixed reality, the world map might include information about where these objects are located relative to one another. The input of pose relative to the world is also important and plays an important role in almost all interactive systems.

The wearable system 900 also accepts inputs and controls from the user. User inputs include visual input, gestures and totems, audio input and sensory input. To move around, or to play a game, the user might need to tell the wearable system 9000 what he or her wants. There are many user controls that can be used to move oneself around in space. In one embodiment, a totem (e.g. A totem (e.g. The system should be able to identify that the user holds the item and track the interaction.

“Hand gesture recognition or tracking may provide input information. The wearable system may be configured to interpret and track hand gestures such as stop, grab, hold, grab, left, right, grab, or stop. In one configuration, the user might want to flip through emails, a calendar, or even do a “fist bump”. With another player or person. The wearable system may be set up to use a minimal amount of hand gestures, which may or not be dynamic. The gestures could be static gestures such as open hand for stop, thumbs down to not ok, thumbs up to ok; or a flip of the hand right, left, or right for direction commands.

Eye tracking is another input. It is used to track the user’s desire to have the display technology render at a particular depth or range. In one embodiment, vergence of the eyes may be determined using triangulation, and then using a vergence/accommodation model developed for that particular person, accommodation may be determined.”

The FIG. 9 example of a wearable system 900 shows how cameras work. Nine can have three types of cameras. One pair may be a passive SLAM or relative wide FOV camera, and another pair oriented in front to record stereo imaging process 940. The other pair could be positioned in front to capture hand gestures as well as totem/object tracking. The stereo imaging system 940, which includes the FOV cameras and the pair for the stereo process 940, may also include the outward-facing imaging systems 464 (shown at FIG. 4). Eye tracking cameras can be included in the wearable system 900. They may be part of an inward facing imaging system 462 as shown in FIG. 4) are oriented towards the eyes of the user to calculate eye vectors and other information. One or more textured light projectsors (such infrared (IR), projectors) may be included in the wearable system 900 to add texture to a scene.

“FIG. “FIG. The user can interact with a totem in this example. Multiple totems may be owned by the user. One totem could be used for social media, while another can be used for games. Block 1010 may be where the wearable system detects a movement of a totem. The outward-facing system may detect the movement of the totem or sensors may (e.g., image sensors, hand-tracking devices, head as, etc )

The wearable system, based at least partially on the detected gesture, eye position, head pose or input through the device, detects the totem’s position, orientation and/or movement relative to block 1020. The reference frame could be a set or map points that the wearable system uses to translate the user’s movement to an action or command. Block 1030 is where the system maps the user’s interaction to the totem. The system calculates the input of the user at block 1040 based on the mapping of user interaction to the reference frame 1020.

To move a totem, or other physical object, the user can turn a page by moving it back and forth. This could be used to indicate moving to a new page, or from one user interface (UI), to another. Another example is that the user can move their head or gaze to view different virtual or real objects in FOR. The input may be the real or virtual object if the user’s gaze is fixed on a specific real or virtual object for more than a threshold amount of time. In some implementations, the vergence of the user’s eyes can be tracked and an accommodation/vergence model can be used to determine the accommodation state of the user’s eyes, which provides information on a depth plane on which the user is focusing. The wearable system may use ray casting techniques in some instances to determine real and virtual objects that are located along the direction of the user?s eye or head poses. The ray casting techniques may include casting pencil-thin rays that have a very small transverse width, or casting rays that are large in transverse (e.g. cones or fustums).

“The display system described herein may project the user interface (such as the display 220 shown in FIG. 2). You can also display it using other methods, such as one or several projectors. Projectors can project images onto physical objects such as a globe or canvas. One or more cameras may be used to track interactions with the user interface (e.g. using the inward-facing image system 462 or outward-facing imagery system 464)

“FIG. “FIG. 11. is a process flow diagram for an example of a 1100 method of interfacing with a virtual user interface. The wearable system described in this article may perform the method 1100.

“At block 1110 the wearable system might identify a specific UI. The user may choose the type of UI they want. A user input may be used to tell the wearable system which UI should be filled. Block 1120 may be used to generate data for the virtual interface. Data such as the shape, structure, and confines of the virtual UI may be generated. The wearable system could also determine the coordinates of the user?s physical location to display the UI relative to that location. If the UI is body-centric, for example, the wearable device may determine the coordinates and head position of the user so that a ring UI or planar UI can either be displayed around them or on a wall in front of them. The map coordinates of the user?s hands can be determined if the UI is hand-centric. These map points can be obtained from data collected through FOV cameras, sensory input or any other type data.

“Block 1130: The wearable system can send data to the display either from the cloud, or from a local database. Based on the data sent, the UI will be displayed to the user at block 1140. A light field display, for example, can project the virtual interface into the eyes of one or both users. The wearable system can wait for a user command to create more virtual content at block 1150 once the virtual UI is created. The UI could be a ring that wraps around the body of the user, for example. The wearable system might then wait for the command (e.g., a gesture, head or eye movement, etc.). If the command is recognized (block 1106), virtual content associated to it may be displayed to users (block 1170). The wearable system might wait for the user to make hand gestures before mixing multiple steam tracks.

U.S. Patent Publication No. 0016777 also describes additional examples of wearable systems, UIs and user experiences (UX). 2015/0016777 is incorporated herein by reference in its entirety.

“Overview of User Interactions Based On Contextual Information”

“The wearable system supports various user interactions with objects within the FOR based upon contextual information. The wearable system can adjust cone aperture size when a user interacts using cone casting. Another example is that the wearable system can adjust virtual objects’ movement based on contextual information. Below are detailed examples of these interactions.

“Example Objects”

A user’s FOR may contain a collection of objects that can be perceived via the wearable device. Virtual and/or physical objects can be contained within the FOR. Virtual objects can include objects from the operating system such as a recycle bin to delete files, a terminal to input commands, a file manager or file manager for accessing files and directories, icons, menus, applications for audio or video streaming, notifications from operating systems, and so forth. Virtual objects can also include objects within an application, such as avatars, graphics, images, or virtual objects in games. Virtual objects can exist as both an operating system object or an application object. The wearable system may add virtual objects to existing physical objects in some instances. The wearable system could add a virtual menu to a room with a TV. This virtual menu would allow the user to change or turn on the channels using the wearable.

“The objects found in the FOR of the user can be part a world map described with reference to FIG. 9. Data associated with objects (e.g. location, semantic information, properties, etc.) You can store information in many data structures, including arrays, lists and trees, as well as hashes, graphs and other semantic information. If applicable, the index of each object stored can be determined by its location. The index of each object may be determined by the location of the object. Some implementations include a light field display which can show virtual objects at different depths relative to the user. You can organize the interactable objects into multiple arrays at different fixed depth planes.

“A user can interact only with certain objects in their FOR. Sometimes, this subset of objects is called interactable objects. You can interact with objects using many different techniques. You can interact with objects using a variety of methods, such as selecting them, moving them, opening a menu or toolbar that is associated with the object or choosing a different set of interactable items. You can interact with the interactable objects using hand gestures (see e.g. user input device 466 in FIG. FIG. 4 shows a user input device 466. Interactable objects can also be accessed by the user using their head, eyes, or body poses, such as gazing at or pointing at an object for some time. The wearable system can initiate a selection event by using hand gestures or poses. This could include displaying a menu, gaming operations, and a display of information about the target object.

“Examples for Cone Casting”

“As explained herein, a user may interact with objects in his environment by using poses. A user might look in a room to see furniture, walls and even a television screen on one wall. The cone casting technique, which is described as a projection of an invisible cone in the direction that the user is looking, allows the wearable device to identify any objects that intersect with it. Cone casting is a method of casting one ray with no lateral thickness from the wearable device to physical or virtual objects. Ray casting is also referred to cone casting using a single ray.

A collision detection agent can be used to track the ray along with identifying if any objects intersect with it. The wearable system can monitor the user’s position (e.g. body, head or eye direction) and use inertial measurement units (e.g. accelerometers, eye-tracking cameras etc. to determine the direction that the user is looking. The wearable system can determine the direction of the ray by using the user’s posture. You can use the ray casting techniques with input devices 466 like a hand-held multi-degree of freedom (DOF), input device. To anchor the size or length of the ray, the user can use the multi-DOF input device. Another example is that the HMD cannot cast the ray, but the wearable system can cast it from the input device.

“In some embodiments, instead of casting a thin ray, the wearable system may cast a cone with a non-negligible aperture (transversely to a central beam 1224). FIG. FIG. 12A shows examples of cone casting using non-negligible apertures. Cone casting can create a conic or other shape volume 1220 using an adjustable aperture. A cone 1220 can be a geometric cone with a proximal and distal ends 1228 a, 1228b. The distal end 1228b of the cone can be larger than the aperture. A large aperture could correspond to a large area at the distal end 1228b of the cone, which is the part that is far from the HMD, user input device, or user. Another example is that a large aperture may correspond to a large distal diameter 1226 at the cone 1220. A small aperture could correspond to a small distal diameter 1226 at the cone 1220. Referring to FIG. 12A. The proximal tip 1228 a can be found at any position, including between the eyes, in the middle of the user?s ARD, or on one of his limbs (e.g. a finger on the hand), or in a user input device (e.g. a toy weapon).

The cone’s direction can be represented by the central ray 1224. The cone’s direction can be related to the user?s body position (such as head, hand, and arm gestures). The user’s gaze direction (also known as eye pose) can be indicated by the cone. FIG. 1206 shows an example. FIG. 12A shows cone casting with poses. The wearable system can determine direction 1224 of cone by using the user?s head position or eye pose. This illustration also shows a coordinate system to determine the head pose. Multiple degrees of freedom may be available for a head 1250. The head position will change as the head moves in different directions relative to the natural resting point 1260. FIG. FIG. 12A illustrates three degrees of freedom (e.g. The three angular degrees of freedom (e.g., pitch, roll, and yaw) can be used to measure the head position relative to the natural resting condition 1260. FIG. FIG. 12A shows that the head 1250 can be tilted forward or backward (e.g. pitching, turning left and right (e.g. yawing, tilting from side to side (e.g. rolling). Other methods or angular representations can be used in other implementations to measure head position, such as any type of Euler angle system. IMUs can be used to determine the head position of the wearable system. FIG. 462 shows the inward-facing imaging device 462 The user can use the 4 to determine their eye position.

“The 1204 example shows another example cone casting with poses. The wearable system can determine which direction the cone is going based on the user’s hand gestures. The cone 1220’s proximal tip 1228a is located at the user’s finger 1214. The user can move the cone 1220 and the central ray 1224 as he points his finger at another location.

The direction of the cone may also be determined by the orientation or position of the user’s input device. The direction of the cone could be determined by a user-drawn trajectory that is placed on the touch surface. To indicate the cone’s direction, the user can place his finger on the touch surface and move it forward. Another cone casting is possible using a user input device. The 1202 example illustrates this. The proximal end 1228a of this example is at the tip 1212 of a weapon-shaped user interface device. The cone 1220 can be moved with the user interface device 1212 as the cone 1224 and central ray 1224 are also movable.

The orientation or position of the HMD can also affect the direction of the cone. The cone can be cast in a direction where the HMD tilts, and at another direction when it is not.

“Initiation a Cone Cast

The wearable system can initiate the casting of a cone when the user 1210 taps on a touchpad, clicks on a mouse, taps on a touchpad, swipes on a touch screen or hovers over a capacitive button.

Summary for “Interactions using 3D virtual objects using poses, multiple-DOF controllers”

Modern computing and display technology have made it possible to create systems that can be called “virtual reality”, “augmented reality?”, or “mixed reality?” Digitally reproduced images and portions of them are presented to users in a way that makes them appear real or can be perceived as such. A virtual reality (or?VR?) scenario usually presents digital or virtual information without transparency to any other real-world visual input. An augmented reality (or?AR?) scenario typically presents digital or virtual information as an enhancement to the visualization of the real world around the user. A mixed reality (or?MR?) scenario involves merging real and digital worlds to create new environments in which physical and virtual objects coexist and interact in real-time. It turns out that the human visual system is complex. Therefore, it is difficult to create a VR, AR or MR technology that allows for a natural-feeling, rich presentation and interaction of virtual images elements with real-world or virtual imagery elements. The systems and methods described herein address various issues related to VR, AR, and MR technology.

“In one embodiment, there is a system that allows users to interact with objects on a wearable device. The system includes a display system for a wearable device that presents a three-dimensional (3D), view to the user and allows interaction with objects within a field (FOR). The FOR may include a part of the environment that can be perceived by the user through the display system. A sensor that acquires data about the user’s pose and a processor communicating with the hardware processor and the display system can be included in the system. The hardware processor can be programmed to: Determine a user’s pose based on data from the sensor; start a cone casting operation on FOR objects. This involves casting a virtual cone with an opening in a direction determined at least partially by the user’s pose; analyze context information about the user’s environment; adjust the aperture of virtual cone based at minimum partly on that contextual information; and render visual representations of the cone for the cone casting.

“Another embodiment discloses a method of interfacing with objects for wearable devices. The method involves receiving a selection from a target virtual objects that is displayed to a user in a first position in 3D space. It also includes receiving an indication of movement for the target object. Based at least partly upon the context information, calculating a multiplier for movement of the target object. Displaying the target virtual objects at a second location, determined at least in part by the first position and the movement quantity to the user.

“Another embodiment discloses a system that allows objects to interact with a wearable device. The display system for a wearable device is configured to show a 3D view to the user. This view can include a target virtual object. A hardware processor can be connected to the system. The hardware processor can be programmed to: Receive an indication of movement for the target object; analyze context information associated with target object; calculate a multiplier that will be applied to the object’s movement based at minimum partly on the indicator of movement and multiplier; display the target object at a second location by the display system, based at most in part on the first and second positions.

“Details about one or more implementations are provided in the accompanying drawings as well as the description. The claims, drawings, and description will reveal other features, aspects, or advantages. This summary and the detailed description below do not attempt to limit or define the scope of the inventive subject matter.

“Overview”

A wearable system can display virtual content in an AR/VR/MR setting. A wearable system allows a user interaction with virtual or physical objects within the environment. The user can interact with objects by moving them around, using poses, or by activating an input device. The user can move the input device for a distance, and the virtual object will follow that user and move the same distance. Cone casting may also be used by the wearable system to enable a user to target or select the virtual object using poses. The wearable system is able to target and select virtual objects within the user’s field by moving his head.

If objects are placed far apart, these approaches can lead to user fatigue. To move the virtual object from one location to another or reach the desired object, the user must move the input device or increase body movement (e.g. increasing arm or head movement) to achieve the desired distance. It can also be difficult to accurately position a distant object because it is hard to see tiny adjustments at far away locations. However, users may prefer to interact with objects closer together and have more precise positioning.

The wearable system is able to automatically adjust the user interface operations according to contextual information. This helps reduce user fatigue and facilitate dynamic interactions with the system.

The wearable system can update the cone’s aperture in cone casting using contextual information. This is an example of dynamic user interaction based upon contextual information. The wearable system can automatically reduce the cone aperture if the user points her head in a direction that has a high density or selectable objects. The wearable system can also automatically increase the cone aperture if the user points her head in a direction with low objects density. This could allow for more objects to be included within the cone, or decrease the movement required to overlap it with virtual objects.

The wearable system may also provide a multiplier that can convert the movement of the input device (and/or user’s movements) into a greater movement of the virtual objects. The user doesn’t have to move far to move the virtual objects to their desired locations if they are located far away. The multiplier can be set to 1 when the virtual object is within reach of the user, e.g., if it is within their hand reach. The wearable system allows for one-to-one interaction between the user’s movements and the movement of the virtual object. This could allow the user to interact more precisely with the virtual object nearby. Below are examples of user interactions that use contextual information.

“Examples for 3D Display of a Wearable Device”

A wearable system, also known as an AR (augmented reality) system, can present virtual images in 2D and 3D to the user. Images can be still images, frames from a video or combinations of both. A wearable device can be included in the wearable system that allows for interaction with VR, AR, and MR environments. A wearable device may be a head-mounted (HMD) device.

“FIG. FIG. 1 shows an illustration of a mixed-reality scenario, with certain virtual objects and some physical objects that can be viewed by a human. FIG. FIG. 1. A MR scene 100 depicts a user of MR technology. The scene shows a person imagining a park-like setting 110 with people, trees, buildings, and a concrete platform 120. The MR technology user also perceives these items. A robot statue 130 stands upon the real-world platform 120. A cartoon-like avatar character 140 is flying past, which appears to be the personification of a Bumble Bee.

It may be beneficial for 3D displays to have an accommodative response that corresponds to the virtual depth of each point. The human eye can experience accommodation conflicts if the accommodative response of a display point doesn’t correspond to its virtual depth as determined by stereopsis and binocular depth cues. This could lead to unstable imaging, headaches, and even complete absence of surface depth.

Display systems that provide images corresponding to a variety of depth planes to viewers can offer VR, AR, and MR experiences. Each depth plane may have different images. This allows the viewer to see depth cues by observing differences in image features or the accommodation required to bring them into focus. These depth cues, as discussed in the past, provide credible perceptions about depth.

“FIG. “FIG. 2” illustrates an example wearable system 200. The wearable system 200 comprises a display 220 and various electronic and mechanical modules that support display 220. Display 220 can be connected to frame 230. This frame is usable by the user, wearer or viewer 210. The display 220 may be placed in front of the user’s eyes 210. The display 220 can show AR/VR/MR content. A head-mounted display (HMD), which is attached to the display 220, can be used. A speaker 240 can be attached to the frame 230 in some embodiments and placed next to the user’s ear canal. (In some embodiments, an additional speaker, not shown here, is placed adjacent to the user’s other ear canal to allow for stereo/shapeable sounds control).

“The wearable device 200 may include an outward-facing imaging unit 464 (shown at FIG. 4) that observes the environment around the user. A wearable system 200 may also include an inward-facing image system 462 (shown at FIG. 4), which can track the eye movements. Inward-facing imaging systems can track the movements of one or both eyes. The frame 230 may have an inward-facing image system 462 attached. It may also be electrically connected to the processing modules 265 and 270. These modules may process image data acquired by the inward facing imaging system to determine the pupil sizes or orientations, eye movements, or user’s eye position 210.

“As an example the wearable device 200 can use either the outward-facing or inward-facing imaging systems 464 to capture images of the user’s pose. Images can be still images, frames, or combinations of images.

“The display 220 may be operatively coupled 250 such as by wired connectivity or wireless connectivity to a local processing module 260. It can be fixedly attached 230 to a helmet/hat worn by the user, embedded into headphones or other removable attachments to the user 210 (e.g. backpack-style configuration or belt-coupling style configuration).

“The local processing module and data module 265 may include a hardware processor as well as digital memories such as flash memory and non-volatile memory. Both of these may be used to aid in data processing, caching, storage, and processing. Data may include data from sensors (which could be, e.g.., operatively coupled with the frame 230, or otherwise attached to user 210), as well as microphones, inertial measuring units (IMUs), compasses and global positioning system units (GPS), radio devices or gyroscopes. Alternatively, data can be acquired using remote processing module (270) or remote data repository (280) for passage to display 220. Local processing and data modules 260 can be operatively connected via communication links 262 and 264 to remote processing module (270) or remote data repository (280), so that remote modules are available for local processing and data 260. Remote processing module 280 or remote data repository 282 may also be operatively coupled.

“In some cases, the remote processing module (270) may include one or more processors that are capable of processing data and/or images. The remote data repository 280 could include a digital storage facility that can be accessed via the internet or another networking configuration. Resource configuration. Some embodiments store all data and perform all computations in the local processing module. This allows for fully autonomous use from remote modules.

The human visual system is complex and it is difficult to perceive depth accurately. It is possible that viewers may perceive an object as three-dimensional due to vergence and accommodation. The movement of vergence (i.e. rolling of the pupils towards or away from one another to converge the lines in the eyes to fixate on an object) is closely related to focusing (or?accommodation?). The lenses of the eyes. Under normal conditions, changing the focus of the lenses of the eyes, or accommodating the eyes, to change focus from one object to another object at a different distance will automatically cause a matching change in vergence to the same distance, under a relationship known as the ?accommodation-vergence reflex.? Under normal conditions, a change of vergence will also trigger a matching change to accommodation. Display systems that match accommodation and vergence better may create more comfortable and realistic simulations of three-dimensional imagery.

“FIG. 3. illustrates aspects of a method for simulating three-dimensional imagery using multiple depth plans. Referring to FIG. FIG. The eyes 302 & 304 use particular accommodated states to focus objects at different distances along z-axis. A particular accommodated condition may be associated with one of the depth planes 306, and has an associated focal distance. This means that objects or parts of objects in particular depth planes are in focus when the eye moves in that accommodated state. Three-dimensional imagery can be created in some embodiments by providing different images for each eye 302 and 304 and different versions of each image corresponding to each depth plane. Although the fields of view are shown separately for illustration purposes, it is possible for the eyes 302 or 304 to overlap as the distance along the Z-axis increases. For illustration purposes, the contours of depth planes are shown flat. However, it is possible for them to curve in space so that all of the features of the depth planes are focused with the eye in an accommodated state. The theory does not limit the possibility that the human eye can perceive depth in a finite number depth planes. Therefore, it is possible to create a convincing simulation of depth perception by showing the eye different images that correspond to each of the limited depth planes.

“Waveguide Stack Assembly”

“FIG. “FIG. 4 shows an example of a waveguide stack that outputs image information to a user. Wearable system 400 may include a stack of waveguides or stacked waveguide assemblies 480. These waveguides can be used to provide three-dimensional perception for the eye/brain by using a plurality waveguides 432, 434, 436, 436, 438, 438, 438, 438, 438, 4400 b. The wearable system 400 could be compared to the wearable device 200 in FIG. 2 with FIG. 4, schematically showing more parts of the wearable system 200. In some embodiments, for example, the waveguide assembly (480) may be integrated into FIG. 2.”

“With reference to FIG. 4. The waveguide assembly 480 could also contain a number of features 458, 456, 454, or 454, 452, between the waveguides. The features 458, 456, 454, and 454, 452, may be lenses in some embodiments. Other embodiments may not allow for the features 458, 456, 454 or 454, 452 to be lenses. They may be spacers, such as cladding layers, or structures that form air gaps.

Waveguides 432, 434, 436, 438, 438, 438, 440 b and the plurality lenses 458, 456, 436, 438, 438, 438, 438, 438, 438, 438, 438, 438, 440 b may be set up to transmit image information to the eye at different levels of light ray divergence or wavefront curvature. Each waveguide level can be associated with a specific depth plane, and may be configured so that it outputs image information that corresponds to that depth. The image injection devices 420 to 432, 424, 426 and 428 can be used to inject image information into waveguides 440, 438, 436, 434, 434, 432, 428. Each of these devices may be designed to distribute light across the waveguides, for output towards the eye 410. The light exits from an output surface of image injection devices 420-422, 424-426, 428, 428 and is injected into the waveguides’ corresponding input edges 440 b to 438 b. 436 b. 434 b. 432 b. One beam of light, such as a collimated beam, may be injected into each waveguide in order to produce an entire field of collimated collimated beams directed towards the eye 410 at specific angles and divergence corresponding to the depth plan associated with that waveguide.

“In some embodiments the image injection devices 422, 424 and 426 are discrete displays that produce image information to be injected into a waveguide 440, 438, 436, 434, 434, 432, and 432 b. Other embodiments include the image injection device 420 to 422, 424 and 426. These devices may be output ends of a single multiplexed monitor that can pipe image information through one or more optical conduits (such fiber optic cables) into each of the image injectors 420, 422, 422, 424 and 426.

“A controller 460 controls operation of the stacked wafer assembly 480 and image injection devices 424, 426. 428. The controller 460 contains programming (e.g. instructions in a nontransitory computer-readable media) that controls the timing and provision image information to waveguides 440, 438, 436, 436, 434, 434, 434, 432, 432 b. The controller 460 can be either a single integrated device or a distributed system connected via wired or wireless communication channels. The controller 460 could be part of one or more processing modules 260 and 270 (illustrated at FIG. 2) in certain embodiments.

“The waveguides 438 b and 436 b can be set up to transmit light through total internal reflection (TIR) within their respective waveguides. Waveguides 440, 438, 436, 434, 434, 432 and 432 b can be either planar or curved, with major top- and bottom surfaces, and edges that extend between them. The illustrated configuration of the waveguides 440, 438, 436, 434, and 432 B may all include light extracting optical components 440a,436 b., 436, b., 434, b., 432, b. These elements are designed to extract light from a waveguide by redirecting light within the waveguide, out of each waveguide to provide image information to the eye 434 a., 432a. Outcoupling optical elements and extracted light can also be called outcoupled. The waveguide outputs a beam of light at the locations where the light from the waveguide strikes a light redirecting component. For example, light extracting optical elements (440a, 438a, 436, 434 a., 432a, 442 a., 432) may be reflective or diffractive optical components. For ease of description and drawing, the light extracting optic elements are shown at the bottom of the waveguides. The light extracting optical element 440a, 438a, 436, 434a, and 432a can be formed in a layer made of material attached to a transparent substrate. This will form the waveguides. Other embodiments allow the waveguides 440, 438, 436, 434, b and 432 to be formed in a single piece of material. The light extracting optical element 440a,438 a., 436, 436, 434, b., 432b. may also be formed on a surface, or within that piece.

“With reference to FIG. “With continued reference to FIG. The waveguide 432b closest to the eye could be set up to deliver collimated lighting, such as that injected into waveguide 432b to the eye 410. The collimated light could be representative of optical infinity focal plan. The next waveguide may send out collimated beams through the first lens 452 (e.g. a negative lens) before reaching the eye 410. The first lens 452 could be set up to create a convex wavefront curvature, so the eye/brain perceives the light coming from the next waveguide 434 b as coming in from a first focal point closer inward towards the eye 410 from optical infinite. The third up waveguide 436b also passes its output light through the second lens 454 and first lens 452, before reaching the eye. Combining the optical power of the first two lenses 452 and 444, may create an incremental amount of wavefront curve so that the eye/brain perceives light coming out of the third waveguide 436 B as coming from a second focal point that is closer inward towards the person than light from the next waveguide, 434 b.

The other waveguide layers (e.g. waveguides 438b, 440b) and lenses (e.g. lenses 456, 458) are similarly configured. The highest waveguide, 440b in the stack, sends its output through all lenses between it, the eye, for an aggregate focal power that is representative of the nearest focal plane to the person. A compensating lens layer (430) may be placed at the top to offset the effect of the lens stack 458, 456, 454 and 452 on viewing/interpreting light from the world 470. This configuration can provide as many perceived focal points as possible, regardless of the number of waveguide/lens pairs. The light extracting optical elements and the focusing parts of the lenses can be either static or not dynamic. Alternate embodiments may include one or both of these elements, with electro-active features.

“With reference to FIG. “With continued reference to FIG. 4, the light extracting optic elements 440a, 438a, 436, 436a, 434a, 443, 434a, and 432a can be configured to redirect light out of respective waveguides and to produce this light with the correct amount of divergence/colimation for the particular depth plane associated to the waveguide. Waveguides with different depth planes can have different configurations for light extracting optical elements. These elements may output light with different amounts of divergence or collimation depending on the depth plane. As discussed above, light extracting optical element 440 a and 438 a are volumetric or surface features that can be set up to produce light at certain angles. The light extracting optical components 440 a. 438 a. 436 a. 434 a. 432 a. 432 a. may be volume or surface holograms and/or diffraction gratings. U.S. Patent Publication No. 2015/0178939 published June. 25th of June 2015, is included by reference in its entirety.”

“In certain embodiments, light extracting optical elements (440 a), 438 a. 436 a. 436 a. 434 a. 432 a. 432 a) are diffractive features that create a diffraction patterns or?diffractive optic element?. Also known as a “DOE” The DOE should have a low diffraction efficiency, so that only a small portion of the beam is deflected toward the eye (410 at each intersection). The rest of the beam continues to travel through the waveguide via total internal reflect. This allows for the division of the light carrying image information into several related exit beams, which exit the waveguide at a variety of locations. The result is a relatively uniform pattern of exit emission towards the eye 304 in this particular collimated beam that bounces around within a waveguide.

“In some embodiments, one of the DOEs can be switched between?on and?off? state in which they actively diffract and?off?” They do not significantly diffract. A switchable DOE could be a layer of polymer-dispersed liquid crystal. The microdroplets may be arranged in a diffraction pattern in the host medium. In this case, the refractive indice of the microdroplets can either be changed to match the host material’s or to an index that is not as high as the host medium’s (in which case they actively diffract incident sunlight).

“In some embodiments the distribution and number of depth planes (or depth of field) may be dynamically adjusted based on the pupil sizes of the viewers eyes. The size of a viewer’s pupils may affect the depth of field. The depth of field may increase as the size of the pupil decreases. This means that a plane that isn’t discernible due to its location being beyond the depth-of-focus of the eyes can become more discernible. With a reduction in pupil size, and in proportion with an increase in depth, the field depth will also change. The decreased pupil size may also affect the number of depth planes that are used to show different images to the viewers. A viewer might not be able see details of both a depth plane and another depth plane at the same pupil size. This could lead to confusion. However, these two depth planes can be in focus simultaneously to the user at a different pupil size.

“In certain embodiments, the display may alter the number of waveguides that receive image information based on the determination of pupil size and orientation or upon receiving electrical signals indicating particular pupil size and orientation. If the user is unable to distinguish between the depth planes of two waveguides, the controller 460 can be programmed or configured to stop providing image information to that waveguide.

This may be advantageous as it reduces the processing load on the system and increases the system’s responsiveness. If the DOEs of a waveguide can be switched between the on or off states, they may be switched to off when the waveguide receives image information.

“In certain embodiments, it might be desirable for an exit beam to meet the condition that its diameter is less than the diameter the viewer’s eye. This condition can be difficult due to the variable size of viewers’ pupils. This condition can be met in some embodiments by changing the size and shape of the exit beam according to the size of the pupil. The size of the exit beam could decrease as the pupil grows, for example. In certain embodiments, the exit beam size can be adjusted using a variable aperture.

The wearable system 400 may include an outward-facing image system 464 (e.g. a digital camera) that captures a portion the world 470. This area of the world 470 is often referred to by the term “field of view” (FOV), and the 464 imaging system is sometimes called an FOV camera. The field of regard (FOR) is the entire area that is available for viewing and imaging by a viewer. There may be 4 FOR. The FOR may include 4? Other contexts may require the wearer to move more tightly, so the FOR of the wearer may have a smaller solid angle. Images taken with the outward-facing imaging device 464 can be used for tracking gestures (e.g. hand or finger gestures), detecting objects in the world, 470, and so on.

The wearable system 400 may also include an inward facing imaging system 466 (e.g. a digital camera) that monitors the movements of the user such as their eye movements and facial movements. To determine the size and/or orientation the pupil of the eyes 304, the inward-facing imaging device 466 can be used. Inward-facing imaging system 466 may be used to capture images that can be used for biometric identification (e.g. via iris identification) or to determine the user’s gaze direction. One camera may be used for each eye in some embodiments. This allows each eye to determine its pupil size and position independently. Each eye can then be presented with image information that is dynamically tailored to it. Other embodiments allow for the determination of the pupil diameter and orientation of one eye 410. This allows the presentation of image information to each eye to be dynamically tailored. The inward-facing imaging device 466 can analyze the images to determine the user?s mood or eye position. This information can then be used by the wearable technology 400 to determine which audio or visual content to present to the user. Wearable system 400 can also detect head pose (e.g. head position or orientation) by using sensors such IMUs, accelerometers and gyroscopes.

The wearable system 400 may include a user input device 466 through which the user can input commands 460 to the controller to interact with the wearable systems 400. The user input device 466 may include a trackpad and touchscreen as well as a joystick, multiple degree of freedom (DOF), controller, a capacitive sensor device, a game controller (D-pad), a keyboard, mouse, a direction pad (D-pad), a keyboard, a mouse (K-pad), a wand or haptic device), and a totem (e.g. functioning as a virtual input device). Multi-DOF controllers can detect user input in any or all translations (e.g. forward/backward or up/down), or rotations (e.g. yaw/pitch, roll, etc.) of the controller. Multi-DOF controllers that support translation movements are called 3DOF, while multi-DOF controllers that support translations and rotations can be called 6DOF. Sometimes, the user may swipe or press a touch-sensitive input device with a finger (e.g. thumb) to input information to the wearable systems 400. During the use of wearable system 400, the user may hold the user input device 466 in their hand. The user input device 466 may be wired or wirelessly connected to the wearable system 400.

“FIG. “FIG.5” 5 illustrates an example of exit beams produced by a waveguide. Although one waveguide is shown, it can be seen that many waveguides within the waveguide assembly480 could function in a similar fashion. The waveguide 432b is infected at the input edge 432c of the waveguide 432, and the light propagates through the waveguide 422, b via TIR. A portion of the light 520 exits the waveguide as the exit beams 510 at points where it impedes on the DOE 432. Although the exit beams 510 appear to be substantially parallel, they can also be directed to propagate to eye 410 at an angle (e.g. forming divergent exit beacons), depending upon the depth plane associated to the waveguide 432b. You will see that the exit beams 510 are substantially parallel. This could be a sign of a waveguide that uses light extracting optical elements to outcouple light, forming images that appear to be set at a deep plane that is far away (e.g. optical infinity). Waveguides with other light extracting optical elements or waveguides may produce a divergent exit beam pattern. This would require the eye to adjust to a greater distance to focus the image on the retina. The brain would interpret this as light coming from a farther distance than the eye 410.

“FIG. “FIG. An optical system may include a waveguide device, an optical coupler subsystem that optically couples light to or from waveguide apparatus, as well as a control subsystem. Multi-focal volumetrics, images, and light fields can all be generated by the optical system. One or more primary planar wavesguides 632a can be included in the optical system (only one is shown at FIG. 6) and one (or more) DOEs 632b that are associated with at least one of the primary waveguides (632 a). The planar waveguides 632b can be identical to those waveguides 432b, 434b, 436b, 436b, 438b, and 438b. 4. An optical system could use a distribution waveguide device to transmit light along a first direction (vertical or the Y-axis, as shown in FIG. 6.) and increase the light’s effective escape pupil along the first direction (e.g., the Y-axis). Distribution waveguide apparatus could include, for instance, a distribution waveguide 622b and at most one DOE 622a (illustrated with double dash-dot lines) that is associated with the distribution waveguide 622b. The distribution planar wavesguide 622b could be identical or similar to the primary planar waveguide 642 b in at least some aspects, but with a different orientation. The DOE 622 a could be identical or similar to the DOE 632. The distribution planar waveguide 622b and DOE 622a could be made of the same materials, or the primary planar wavesguide 632b or DOE 622 a, for example. FIG. 6 shows embodiments of the optical display 600. 6 can be integrated in the wearable system 200 as shown in FIG. 2.”

“The relayed or exit-pupil extended light may be optically coupled to the distribution waveguide apparatus into one or more primary plane waveguides 632b. The primary planar waveguide 632b can relay light along another axis, but it should be orthogonal to the first axis (e.g. horizontal or X-axis as shown in FIG. 6). The second axis may be non-orthogonal to the first. The primary planar waveguide 632b expands light’s exit pupil along the second axis (e.g. X-axis). The distribution planar wavesguide 622 b, for example, can expand and relay light along the vertical axis. It then passes that light on to the primary planar waveguide 642, which can expand and relay light along its horizontal axis.

“The optical system can include one or more sources for colored light (e.g. red, green and blue laser light)610, which may be optically connected to a proximal tip of a single-mode optical fiber 640. The distal end may be threaded through or received through a hollow tube 642 made of piezoelectric materials. As a flexible, fixed-free cantilever 644, the distal end protrudes out of the tube 642 Four quadrant electrodes can be attached to the piezoelectric tube 642. For example, the electrodes can be plated on the outer, outer, or outer periphery of the tube 642. Unillustrated core electrodes may be found in the core, inner periphery, or inner diameter of the tube 642.

“Drive electronics 655, for example, electrically coupled via wires 665, drive opposing electrodes to bend piezoelectric tube 642 in 2 axes independently. Mechanical modes of resonance are found at the distal tip 644 of the optical fibre. The frequency of resonance depends on the diameter, length, as well as the material properties of optical fiber 644. The fiber cantilever 644 vibrates when the piezoelectric tube 642 is vibrated near the first mode of mechanical resonance. It can also sweep through large deflections by vibrating the 642.

“By stimulating resonance vibration in two directions, the tip 644 of the fiber cantilever is scan biaxially in an area that fills two-dimensional (2D). scan. An image can be created by modulating the intensity of light source(s), 610, in sync with the scan of fiber cantilever 644. U.S. Patent Publication No. 2014/0003762 is incorporated herein in its entirety.

“A component of an optical subsystem that can collimate light from the scanning fibre cantilever 644 can be called an optical coupler subsystem. Mirror surface 648 can reflect the collimated light into the narrow distribution plane waveguide 622b, which contains at least one diffractive optic element (DOE 622a). Relatively to FIG. 6) along the distribution plane waveguide 622b by TIR and intersects repeatedly with the DOE 622a. The DOE 622a is preferred to have a low coefficient of reflection. This can result in a fraction (e.g. 10%) of light being diffracted towards an edge of the primary planar waveguide 642 b at each intersection with the DOE622 a and a fraction to continue its original trajectory along the length of distribution planar waveguide 642 b via TIR.

“At every intersection with the DOE 622a, additional light may be diffracted towards the primary waveguide 632b. The DOE 4 can expand vertically the exit pupil by dividing the incoming beam into multiple outcoupled sets. This vertically expanded light can be coupled out of distribution waveguide 622b to enter the edge the primary planar wavesguide 632b.

“Light entering the primary waveguide 632b can propagate horizontally (relatively to FIG. 6) along primary waveguide 632b via TIR. The light crosses DOE 632a at multiple points and propagates horizontally along at most a portion the primary waveguide 632b via TIR. To produce deflection and focus of light, the DOE 632a can be advantageously designed or configured with a phase profile. This is a combination of a linear and radially symmetric diffractive patterns. The DOE 632a may have a lower diffraction efficiency (e.g. 10%) so that only a small portion of the beam is deflected towards the eye with each intersection of DOE 632a, while the rest propagates through the primary waveguide 632b via TIR.

“A fraction of the light is scattered toward the face of primary waveguide 632b at each intersection of the propagating light with the DOE 632a. This allows the light to escape from the TIR and emerge from primary waveguide 642 b. The DOE 632a radially symmetric pattern of diffraction imparts an additional focus level to diffracted light. This allows the light wavefront to be shaped (e.g., giving a curvature) and the beam to be directed at an angle that matches the intended focus level.

These different paths can result in the light being coupled out of the primary plane waveguide 632b by a multiplicity OFEs 632a at different angles, focus level, and/or fill patterns at exit pupil. To create light fields with multiple depth planes, different fill patterns can be used at the exit pupil. Each layer of the waveguide assembly, or a group of layers (e.g. 3 layers), may be used to produce a particular color, such as red, blue and green. A first set of three layers adjacent may be used to produce red, blue, and green light at a particular focal depth. To produce red, green and blue light at a second depth, a second set of three layers may be used. Multiple sets can be used to create a full 3D and 4D color image lightfield with different focal depths.

“Other Components”

In many cases, other components may be added to or substituted for the components of the wearable systems described above. For example, the wearable system could include one or more haptic components or devices. A user may feel a tactile sensation from the haptic components or devices. The haptic components or devices may be able to provide a tactile sensation such as pressure or texture for touching virtual content (e.g. virtual objects, tools, or other constructs). The tactile sensation could replicate the feel of a real object that a virtual object represents or it may mimic an imagined object or character (e.g. a dragon) that the virtual content represents. Some implementations allow the user to wear haptic components or devices (e.g., a glove for users). Some implementations allow the user to hold haptic components or devices.

“The wearable device may include, for instance, one or more physical objects that can be controlled by the user to enable input or interaction with it. These objects can be called totems. Totems can take the form inanimate objects such as a piece or plastic of metal, a wall, or a table surface. Some totems might not have any input structures, such as keys, triggers or joysticks, trackballs, joysticks, rocker switches, etc. The totem might provide a physical surface and the wearable device may create a user interface that makes it appear the user is on the totem’s surfaces. The wearable system might render an image of a trackpad and keyboard to make it appear that they are on a specific surface of the totem. The wearable system could make a virtual keyboard and trackpad appear on the surface of an aluminum plate that is thin enough to be used as a totem. The rectangular plate itself does not have any trackpad, sensors or physical keys. The wearable system can detect user interaction with the rectangular plate, such as touch or manipulation via the virtual keyboard. FIG. 466 shows the user input device. 4) may be an embodiment of a totem, which may include a trackpad, a touchpad, a trigger, a joystick, a trackball, a rocker or virtual switch, a mouse, a keyboard, a multi-degree-of-freedom controller, or another physical input device. The totem can be used by a user to interact with the wearable device or other users, either alone or in combination.

U.S. Patent Publication No. 0016777 describes “Examples haptic devices or totems usable alongside the wearable devices HMD and display systems of this disclosure.” 2015/0016777 is incorporated herein in its entirety.

“Examples of Wearable Systems, Environments and Interfaces”

A wearable system might use different mapping techniques to obtain high depth of field in rendered light fields. It is important to be able to map out the virtual world and to understand the details of the real world in order to accurately depict virtual objects relative to the real one. By adding new photos that provide information about different features and points of the real world, FOV images can be captured by users of the wearable device. The wearable system can, for example, collect a number of map points (such 2D or 3D points) from users and then find new map points to create a more precise version of the world model. A first user’s world model can be shared (e.g. over a network like a cloud network) with a second user to allow the second user to experience the world around the first user.

“FIG. “FIG.7” is a block diagram showing an example of an MR Environment 700. The MR environment 700 can be configured to receive input. This could include visual input 702 from the wearable system, stationary input 704 like room cameras, sensory input 706 from various sensors, gestures and eye tracking, as well as input from the user input device 466. One or more user wearable devices (e.g. wearable system 200, display system 220), or stationary room systems, (e.g. room cameras, etc.). Wearable systems can be equipped with various sensors such as accelerometers and gyroscopes. They can also use temperature sensors, movement sensors or GPS sensors. To determine the user’s location and other environmental attributes. These data may be further supplemented by information from stationary cameras within the room, which may provide images or other cues from another point of view. The data from the cameras, such as the room cameras or the outward-facing imaging systems cameras, may be reduced to a number of mapping points.

“One or more object recognitions 708 can scan through the received data (e.g. the collection of points), and identify or map points, tag images or attach semantic information to objects using a map database 710. The map database 710 can contain various points and the corresponding objects. Through a network, the various devices and map database can be connected. To access the cloud

Based on this information, and the collection of points in a map database, object recognizers 708a-708n can recognize objects in an environmental. The object recognizers are able to recognize faces, people, windows, walls and user input devices. They can also recognize televisions and other objects within the environment. One or more object recognition devices may have a specific focus on objects with particular characteristics. One example is the object recognizer 708a which can be used to recognize faces and another to recognize totems.

“Object recognitions can be made using many computer vision techniques. The wearable system can, for example, analyze images taken by the outward-facing image system 464 (shown at FIG. 4) to perform scene reconstruction, event detection, video tracking, object recognition, object pose estimation, learning, indexing, motion estimation, or image restoration, etc. These tasks may be performed by one or more computer vision algorithms. Examples of computer vision algorithms that may be used include: Scale invariant feature transformation (SIFT), speeded-up robust features (SURF), oriented fast and rotated BRIEFs (ORB), binary robust and scalable keypoints(BRISK), fast and reliable retina keypoints (FREAK), Viola Jones algorithm, Eigenfaces algorithm, Lucas-Kanade algorithm and Horn-Schunk algorithm. Mean-shift algorithm. Visual simultaneous location and mapping (vSLAM), techniques, Kalman filter and extended Kalman filter. ), bundle adjustment, Adaptive thresholding (and other thresholding techniques), Iterative Closest Point (ICP), Semi Global Matching (SGM), Semi Global Block Matching (SGBM), Feature Point Histograms, various machine learning algorithms (such as e.g., support vector machine, k-nearest neighbors algorithm, Naive Bayes, neural network (including convolutional or deep neural networks), or other supervised/unsupervised models, etc. ), and so on.

“A variety of machine learning algorithms can be used to recognize objects. The HMD can store the machine learning algorithm once it has been trained. Machine learning algorithms include either supervised or unsupervised, such as Ordinary Least Squares Regression, instance-based algorithms like Learning Vector Quantization, and decision tree algorithms like, for instance, Bayesian algorithms. These algorithms can be stored by the HMD once they are trained. Individual models can be tailored to specific data sets in some embodiments. The wearable device may store or generate a base model. A base model can be used to create additional models for a specific data type (e.g. a user in a telepresence sessions), a data set (e.g. a set additional images taken by the user during the session), or conditional situations. The wearable HMD may be configured to use a variety of methods to generate models to analyze the aggregated data. You may also use pre-defined thresholds and data values.

Based on the information and the collection of points in a map database, object recognizers 708a-708n may recognize objects and add semantic information to make them more real. If the system recognizes a set points as a door, it may add semantic information to the objects (e.g. the hinge has a 90-degree movement around the hinge). The system might attach semantic information to a set that is identified as a mirror if it recognizes the points as mirrors. This may include the fact that the mirror’s reflective surface can reflect images from objects within the room. As more data is added to the map database (which can be local or accessible via a wireless network), the system will grow. The information can be sent to one or more wearable devices once the objects have been identified. The MR environment 700 could include information about a California scene. One or more New Yorkers may receive the environment 700. The object recognizers and the other software components can use data from the FOV camera to map the points and recognize objects. This allows the scene to be?passed over’ accurately. To another user who might be in a different area of the world. Environment 700 can also use topological maps for localization purposes.

“FIG. “FIG.8 is a process flow diagram for an example of a method 800 that renders virtual content in relation with recognized objects. A method 800 shows how a virtual scene can be presented to the wearable device user. The scene may not be accessible to the user. The user could be in New York but want to see a scene in California or go on a walk together with a friend who lives in California.

“Block 810 allows the wearable system to receive input from users and other users about the environment. This can be done using various input devices and information already stored in the map database. Block 810 is populated with information from the user’s FOV camera and sensors, GPS, eye track, and other devices. Based on the information at block 820, the system can determine sparse point. These sparse points can be used to determine pose data (e.g. head pose, eye position, or hand gestures). This information can be used to display and understand the orientation and location of objects in the environment. These points can be used by object recognition 708 a-708n to crawl through and identify one or more objects from a map database at block 8.30. The information may then be sent to block 840 and displayed at block 855. The desired virtual scene, e.g. user in CA, may be displayed in the right orientation and position in relation to various objects and other surrounding users in New York.

“FIG. “FIG. 9 is a block diagram for another example of a wearable systems. The wearable system 900 in this example includes a map that may contain map data for the entire world. The map can reside partly locally on the wearable device, while other parts may reside in networked storage locations (e.g., cloud systems) that are accessible via wired or wireless networks. The wearable computing architecture may execute a pose process 910. This uses data from the map to determine the user’s position and orientation. Pose data can be calculated from data that is collected as the user interacts with the system and operates in the real world. Data may include images, data from sensors (such inertial measuring units, which usually comprise accelerometers and gyroscope parts), and surface information relevant to objects in the real and virtual environments.

A sparse point representation could be the result of simultaneous localization (SLAM) or V-SLAM processes. This refers to a configuration in which the input is visual only. It is possible to configure the system to find out not only where the components are located in the world, but also what the world is made from. Pose can be used to accomplish many goals. It may also be used to populate the map with data.

“One embodiment shows that a sparse point location may not be sufficient on its own. Additional information may be required to create a multifocal AR, VR or MR experience. This gap may be filled at most in part using dense representations. Dense representations generally refer to depth map information. This information can be obtained from Stereo 940. In Stereo 940, depth information is calculated using techniques such as triangulation and time-of-flight sensoring. Stereo process 940 may also use image information and active patterns, such as infrared patterns made with active projectors. It is possible to combine a lot of depth map information, and some of it may be combined with a surface representation. Mathematically definable surfaces can be more efficient than large point clouds and provide digestible inputs for other processing devices, such as game engines. The stereo output (e.g., depth map) 940 can be combined with the fusion process 930. Pose could also be used as an input for this fusion process 930. The output of fusion 930 can then be used to populate the map process 920. To create larger surfaces, sub-surfaces can connect, as in topographical mapping. The map then becomes a large mixture of points and surfaces.

Multiple inputs can be used to resolve different aspects of a mixed reality process (960). FIG. 9 illustrates an example of such inputs. 9. Game parameters can be inputs that determine whether the user is playing a game of monster fighting with one or more monsters at different locations, monsters running away or dying under certain conditions (such as if they are shot by the user), walls at various locations or any other objects. To add value to mixed reality, the world map might include information about where these objects are located relative to one another. The input of pose relative to the world is also important and plays an important role in almost all interactive systems.

The wearable system 900 also accepts inputs and controls from the user. User inputs include visual input, gestures and totems, audio input and sensory input. To move around, or to play a game, the user might need to tell the wearable system 9000 what he or her wants. There are many user controls that can be used to move oneself around in space. In one embodiment, a totem (e.g. A totem (e.g. The system should be able to identify that the user holds the item and track the interaction.

“Hand gesture recognition or tracking may provide input information. The wearable system may be configured to interpret and track hand gestures such as stop, grab, hold, grab, left, right, grab, or stop. In one configuration, the user might want to flip through emails, a calendar, or even do a “fist bump”. With another player or person. The wearable system may be set up to use a minimal amount of hand gestures, which may or not be dynamic. The gestures could be static gestures such as open hand for stop, thumbs down to not ok, thumbs up to ok; or a flip of the hand right, left, or right for direction commands.

Eye tracking is another input. It is used to track the user’s desire to have the display technology render at a particular depth or range. In one embodiment, vergence of the eyes may be determined using triangulation, and then using a vergence/accommodation model developed for that particular person, accommodation may be determined.”

The FIG. 9 example of a wearable system 900 shows how cameras work. Nine can have three types of cameras. One pair may be a passive SLAM or relative wide FOV camera, and another pair oriented in front to record stereo imaging process 940. The other pair could be positioned in front to capture hand gestures as well as totem/object tracking. The stereo imaging system 940, which includes the FOV cameras and the pair for the stereo process 940, may also include the outward-facing imaging systems 464 (shown at FIG. 4). Eye tracking cameras can be included in the wearable system 900. They may be part of an inward facing imaging system 462 as shown in FIG. 4) are oriented towards the eyes of the user to calculate eye vectors and other information. One or more textured light projectsors (such infrared (IR), projectors) may be included in the wearable system 900 to add texture to a scene.

“FIG. “FIG. The user can interact with a totem in this example. Multiple totems may be owned by the user. One totem could be used for social media, while another can be used for games. Block 1010 may be where the wearable system detects a movement of a totem. The outward-facing system may detect the movement of the totem or sensors may (e.g., image sensors, hand-tracking devices, head as, etc )

The wearable system, based at least partially on the detected gesture, eye position, head pose or input through the device, detects the totem’s position, orientation and/or movement relative to block 1020. The reference frame could be a set or map points that the wearable system uses to translate the user’s movement to an action or command. Block 1030 is where the system maps the user’s interaction to the totem. The system calculates the input of the user at block 1040 based on the mapping of user interaction to the reference frame 1020.

To move a totem, or other physical object, the user can turn a page by moving it back and forth. This could be used to indicate moving to a new page, or from one user interface (UI), to another. Another example is that the user can move their head or gaze to view different virtual or real objects in FOR. The input may be the real or virtual object if the user’s gaze is fixed on a specific real or virtual object for more than a threshold amount of time. In some implementations, the vergence of the user’s eyes can be tracked and an accommodation/vergence model can be used to determine the accommodation state of the user’s eyes, which provides information on a depth plane on which the user is focusing. The wearable system may use ray casting techniques in some instances to determine real and virtual objects that are located along the direction of the user?s eye or head poses. The ray casting techniques may include casting pencil-thin rays that have a very small transverse width, or casting rays that are large in transverse (e.g. cones or fustums).

“The display system described herein may project the user interface (such as the display 220 shown in FIG. 2). You can also display it using other methods, such as one or several projectors. Projectors can project images onto physical objects such as a globe or canvas. One or more cameras may be used to track interactions with the user interface (e.g. using the inward-facing image system 462 or outward-facing imagery system 464)

“FIG. “FIG. 11. is a process flow diagram for an example of a 1100 method of interfacing with a virtual user interface. The wearable system described in this article may perform the method 1100.

“At block 1110 the wearable system might identify a specific UI. The user may choose the type of UI they want. A user input may be used to tell the wearable system which UI should be filled. Block 1120 may be used to generate data for the virtual interface. Data such as the shape, structure, and confines of the virtual UI may be generated. The wearable system could also determine the coordinates of the user?s physical location to display the UI relative to that location. If the UI is body-centric, for example, the wearable device may determine the coordinates and head position of the user so that a ring UI or planar UI can either be displayed around them or on a wall in front of them. The map coordinates of the user?s hands can be determined if the UI is hand-centric. These map points can be obtained from data collected through FOV cameras, sensory input or any other type data.

“Block 1130: The wearable system can send data to the display either from the cloud, or from a local database. Based on the data sent, the UI will be displayed to the user at block 1140. A light field display, for example, can project the virtual interface into the eyes of one or both users. The wearable system can wait for a user command to create more virtual content at block 1150 once the virtual UI is created. The UI could be a ring that wraps around the body of the user, for example. The wearable system might then wait for the command (e.g., a gesture, head or eye movement, etc.). If the command is recognized (block 1106), virtual content associated to it may be displayed to users (block 1170). The wearable system might wait for the user to make hand gestures before mixing multiple steam tracks.

U.S. Patent Publication No. 0016777 also describes additional examples of wearable systems, UIs and user experiences (UX). 2015/0016777 is incorporated herein by reference in its entirety.

“Overview of User Interactions Based On Contextual Information”

“The wearable system supports various user interactions with objects within the FOR based upon contextual information. The wearable system can adjust cone aperture size when a user interacts using cone casting. Another example is that the wearable system can adjust virtual objects’ movement based on contextual information. Below are detailed examples of these interactions.

“Example Objects”

A user’s FOR may contain a collection of objects that can be perceived via the wearable device. Virtual and/or physical objects can be contained within the FOR. Virtual objects can include objects from the operating system such as a recycle bin to delete files, a terminal to input commands, a file manager or file manager for accessing files and directories, icons, menus, applications for audio or video streaming, notifications from operating systems, and so forth. Virtual objects can also include objects within an application, such as avatars, graphics, images, or virtual objects in games. Virtual objects can exist as both an operating system object or an application object. The wearable system may add virtual objects to existing physical objects in some instances. The wearable system could add a virtual menu to a room with a TV. This virtual menu would allow the user to change or turn on the channels using the wearable.

“The objects found in the FOR of the user can be part a world map described with reference to FIG. 9. Data associated with objects (e.g. location, semantic information, properties, etc.) You can store information in many data structures, including arrays, lists and trees, as well as hashes, graphs and other semantic information. If applicable, the index of each object stored can be determined by its location. The index of each object may be determined by the location of the object. Some implementations include a light field display which can show virtual objects at different depths relative to the user. You can organize the interactable objects into multiple arrays at different fixed depth planes.

“A user can interact only with certain objects in their FOR. Sometimes, this subset of objects is called interactable objects. You can interact with objects using many different techniques. You can interact with objects using a variety of methods, such as selecting them, moving them, opening a menu or toolbar that is associated with the object or choosing a different set of interactable items. You can interact with the interactable objects using hand gestures (see e.g. user input device 466 in FIG. FIG. 4 shows a user input device 466. Interactable objects can also be accessed by the user using their head, eyes, or body poses, such as gazing at or pointing at an object for some time. The wearable system can initiate a selection event by using hand gestures or poses. This could include displaying a menu, gaming operations, and a display of information about the target object.

“Examples for Cone Casting”

“As explained herein, a user may interact with objects in his environment by using poses. A user might look in a room to see furniture, walls and even a television screen on one wall. The cone casting technique, which is described as a projection of an invisible cone in the direction that the user is looking, allows the wearable device to identify any objects that intersect with it. Cone casting is a method of casting one ray with no lateral thickness from the wearable device to physical or virtual objects. Ray casting is also referred to cone casting using a single ray.

A collision detection agent can be used to track the ray along with identifying if any objects intersect with it. The wearable system can monitor the user’s position (e.g. body, head or eye direction) and use inertial measurement units (e.g. accelerometers, eye-tracking cameras etc. to determine the direction that the user is looking. The wearable system can determine the direction of the ray by using the user’s posture. You can use the ray casting techniques with input devices 466 like a hand-held multi-degree of freedom (DOF), input device. To anchor the size or length of the ray, the user can use the multi-DOF input device. Another example is that the HMD cannot cast the ray, but the wearable system can cast it from the input device.

“In some embodiments, instead of casting a thin ray, the wearable system may cast a cone with a non-negligible aperture (transversely to a central beam 1224). FIG. FIG. 12A shows examples of cone casting using non-negligible apertures. Cone casting can create a conic or other shape volume 1220 using an adjustable aperture. A cone 1220 can be a geometric cone with a proximal and distal ends 1228 a, 1228b. The distal end 1228b of the cone can be larger than the aperture. A large aperture could correspond to a large area at the distal end 1228b of the cone, which is the part that is far from the HMD, user input device, or user. Another example is that a large aperture may correspond to a large distal diameter 1226 at the cone 1220. A small aperture could correspond to a small distal diameter 1226 at the cone 1220. Referring to FIG. 12A. The proximal tip 1228 a can be found at any position, including between the eyes, in the middle of the user?s ARD, or on one of his limbs (e.g. a finger on the hand), or in a user input device (e.g. a toy weapon).

The cone’s direction can be represented by the central ray 1224. The cone’s direction can be related to the user?s body position (such as head, hand, and arm gestures). The user’s gaze direction (also known as eye pose) can be indicated by the cone. FIG. 1206 shows an example. FIG. 12A shows cone casting with poses. The wearable system can determine direction 1224 of cone by using the user?s head position or eye pose. This illustration also shows a coordinate system to determine the head pose. Multiple degrees of freedom may be available for a head 1250. The head position will change as the head moves in different directions relative to the natural resting point 1260. FIG. FIG. 12A illustrates three degrees of freedom (e.g. The three angular degrees of freedom (e.g., pitch, roll, and yaw) can be used to measure the head position relative to the natural resting condition 1260. FIG. FIG. 12A shows that the head 1250 can be tilted forward or backward (e.g. pitching, turning left and right (e.g. yawing, tilting from side to side (e.g. rolling). Other methods or angular representations can be used in other implementations to measure head position, such as any type of Euler angle system. IMUs can be used to determine the head position of the wearable system. FIG. 462 shows the inward-facing imaging device 462 The user can use the 4 to determine their eye position.

“The 1204 example shows another example cone casting with poses. The wearable system can determine which direction the cone is going based on the user’s hand gestures. The cone 1220’s proximal tip 1228a is located at the user’s finger 1214. The user can move the cone 1220 and the central ray 1224 as he points his finger at another location.

The direction of the cone may also be determined by the orientation or position of the user’s input device. The direction of the cone could be determined by a user-drawn trajectory that is placed on the touch surface. To indicate the cone’s direction, the user can place his finger on the touch surface and move it forward. Another cone casting is possible using a user input device. The 1202 example illustrates this. The proximal end 1228a of this example is at the tip 1212 of a weapon-shaped user interface device. The cone 1220 can be moved with the user interface device 1212 as the cone 1224 and central ray 1224 are also movable.

The orientation or position of the HMD can also affect the direction of the cone. The cone can be cast in a direction where the HMD tilts, and at another direction when it is not.

“Initiation a Cone Cast

The wearable system can initiate the casting of a cone when the user 1210 taps on a touchpad, clicks on a mouse, taps on a touchpad, swipes on a touch screen or hovers over a capacitive button.

Click here to view the patent on Google Patents.

How to Search for Patents

A patent search is the first step to getting your patent. You can do a google patent search or do a USPTO search. Patent-pending is the term for the product that has been covered by the patent application. You can search the public pair to find the patent application. After the patent office approves your application, you will be able to do a patent number look to locate the patent issued. Your product is now patentable. You can also use the USPTO search engine. See below for details. You can get help from a patent lawyer. Patents in the United States are granted by the US trademark and patent office or the United States Patent and Trademark office. This office also reviews trademark applications.

Are you interested in similar patents? These are the steps to follow:

1. Brainstorm terms to describe your invention, based on its purpose, composition, or use.

Write down a brief, but precise description of the invention. Don’t use generic terms such as “device”, “process,” or “system”. Consider synonyms for the terms you chose initially. Next, take note of important technical terms as well as keywords.

Use the questions below to help you identify keywords or concepts.

• What is the purpose of the invention Is it a utilitarian device or an ornamental design?
• Is invention a way to create something or perform a function? Is it a product?
• What is the composition and function of the invention? What is the physical composition of the invention?
• What’s the purpose of the invention
• What are the technical terms and keywords used to describe an invention’s nature? A technical dictionary can help you locate the right terms.

2. These terms will allow you to search for relevant Cooperative Patent Classifications at Classification Search Tool. If you are unable to find the right classification for your invention, scan through the classification’s class Schemas (class schedules) and try again. If you don’t get any results from the Classification Text Search, you might consider substituting your words to describe your invention with synonyms.

3. Check the CPC Classification Definition for confirmation of the CPC classification you found. If the selected classification title has a blue box with a “D” at its left, the hyperlink will take you to a CPC classification description. CPC classification definitions will help you determine the applicable classification’s scope so that you can choose the most relevant. These definitions may also include search tips or other suggestions that could be helpful for further research.

4. The Patents Full-Text Database and the Image Database allow you to retrieve patent documents that include the CPC classification. By focusing on the abstracts and representative drawings, you can narrow down your search for the most relevant patent publications.

5. This selection of patent publications is the best to look at for any similarities to your invention. Pay attention to the claims and specification. Refer to the applicant and patent examiner for additional patents.

6. You can retrieve published patent applications that match the CPC classification you chose in Step 3. You can also use the same search strategy that you used in Step 4 to narrow your search results to only the most relevant patent applications by reviewing the abstracts and representative drawings for each page. Next, examine all published patent applications carefully, paying special attention to the claims, and other drawings.

7. You can search for additional US patent publications by keyword searching in AppFT or PatFT databases, as well as classification searching of patents not from the United States per below. Also, you can use web search engines to search non-patent literature disclosures about inventions. Here are some examples:

• Add keywords to your search. Keyword searches may turn up documents that are not well-categorized or have missed classifications during Step 2. For example, US patent examiners often supplement their classification searches with keyword searches. Think about the use of technical engineering terminology rather than everyday words.
• Search for foreign patents using the CPC classification. Then, re-run the search using international patent office search engines such as Espacenet, the European Patent Office’s worldwide patent publication database of over 130 million patent publications. Other national databases include:
• European Patent Office (EPO) provides esp@cenet to access a network of Europe’s patent databases with access to machine translation of European patents. 
• Japan Patent Office (JPO) – with access to machine translations of Japanese patents.
• World Intellectual Property Organization (WIPO) offers PATENTSCOPE with a full-text search of published international patent applications and machine translations for some documents, as well as a list of international patent databases.
• Korean Intellectual Property Rights Information Service (KIPRIS)
• State Intellectual Property Office (SIPO) with machine translation of Chinese patents.
• Other International Intellectual Property Offices with online patent databases include Australia, Canada, Denmark, Finland, France, Germany, Great Britain, India, Israel, Netherlands, Norway, Sweden, Switzerland, and Taiwan.
• Search non-patent literature. Inventions can be made public in many non-patent publications. It is recommended that you search journals, books, websites, technical catalogs, conference proceedings, and other print and electronic publications.

To review your search, you can hire a registered patent attorney to assist. A preliminary search will help one better prepare to talk about their invention and other related inventions with a professional patent attorney. In addition, the attorney will not spend too much time or money on patenting basics.

Download patent guide file – Click here