Invented by Richard St. Clair Bailey, Brian Everett Meaney, John Austin Day, Lorena Pazmino, James Cameron Petty, James M. Powderly, Savannah Niles, Magic Leap Inc
The Magic Leap Inc invention works as followsSystems and Methods for Displaying a Group of Virtual Objects and a Scrollbar in a Virtual, Augmented, or Mixed Reality Environment are Disclosed.” A group of virtual items can be scrolled and a virtual panel can indicate objects that will be upcoming on the scroll. Scrollbars can provide feedback in real time, such as the starting point of scrolling, the current point of scrolling, and the amount of virtual items that the user is currently viewing relative to total virtual objects.
Background for Scrollbar for Augmented Reality
Modern computing and display technology has facilitated the development systems for ‘virtual reality, augmented reality, or mixed reality. Digitally reproduced images, or portions thereof, are presented in such a way that they appear to be real, or can be perceived to be so. Virtual reality (VR) is a scenario where digital images or portions thereof are presented to a user in a way that they appear to be real. Augmented reality (AR) is presenting digital image data as an enhancement to the visualization of real world. It turns out that the human visual system is complex. Producing a VR technology, AR or MR which facilitates a rich, comfortable and natural presentation of virtual images amongst real or virtual imagery elements can be challenging. The systems and methods described herein address a variety of challenges related to VR technology, AR technology and MR.
A wearable display system may include an interface which presents a number of interactive virtual items in a grid of virtual content, such as icons, thumbnails or other graphics, at different depths. A thumbnail is a miniature version of the virtual content (e.g. a page from a document or an image), which can be used as a way to identify it. In some implementations selecting the thumbnail (e.g. clicking or double-clicking it) opens the content of a virtual item (e.g. by executing an app configured to play, view or edit virtual content). The thumbnail can be rendered to appear at a single depth (e.g. as a 2D thumb) or multiple depths (e.g. so that it looks 3D). The thumbnail of an item in the grid may be rendered in response to a moving cursor. This can include the following: increasing in size, adding a focus indicator, moving to another depth (e.g. to one that appears closer to the user), changing virtual content, or moving to a deeper depth. The thumbnails can be arranged according to one or several grouping criteria, such as alphabetically, by content type, date etc .).
The grid of thumbnails can be scrolled by the user using gestures (e.g. head, eye or body movements, or input from a user totem). The edges of the grid, such as in the direction of scrolling, may dynamically show indications of virtual contents that are next to be displayed during the scroll (e.g. upcoming content), e.g. semi-transparent thumbs at a depth different than the edge.
The user interface can have a scrollbar that provides real-time feedback in relation to the content scrolled. Scrollbars can have a bar moving within a trough. The bar’s position can indicate what content in the content library is being viewed by the viewer within the viewable area. The bar’s length can be used to represent a fraction of the content library being rendered in the viewable window. The bar can have a fixed edge to indicate the position of the virtual content where the user began scrolling from and a moving edge to show the current position. The temporarily fixed edge may become unfixed when scrolling stops and can move (or snap into) a final position. This will give the bar a length and position that is representative of the fractional amount of virtual content being rendered.
The disclosure, in various aspects, provides an ornamental design of a display screen, or a portion thereof, with virtual content, or with a graphic user interface that is transitional (e.g. animated). The display screen, or a portion thereof, can be a virtual, mixed or augmented reality display device.
Details about one or more implementations are provided in the accompanying drawings as well as the description. The claims, drawings, and description will reveal other features, aspects, or advantages. This summary and the detailed description below do not attempt to limit or define the scope of the inventive subject matter.
A wearable display can present an interactive VR/AR/MR world.” The VR/AR/MR environments can include data that the user may interact with through various poses such as head pose, eye gaze, body pose or user input via a user input devices. The system can render a visual aid on screen to help the user navigate and select objects within the VR/AR/MR environments.
In some cases, visual aids on screen can include a virtual mouse (sometimes referred to as a “reticle”) that responds to the user’s interaction (e.g. user input via a totem held in hand) and indicates (to the users) the position a moveable indicator which can be used to interact or select objects within the VR/AR/MR environments. The user can move their thumb along the touch-sensitive part of the totem in order to move the cursor within the 3D VR/AR/MR world. The user can select an object or interact with it (e.g. by pressing on the touch-sensitive part of the totem) when the cursor hovers or is close enough to the object. This may trigger context-dependent functionality from the wearable device. The user can move the cursor to a virtual display showing a movie, select it, and then get a menu with other movie options, volume controls, etc. In some cases the cursor will be displayed so the user can easily locate it in the environment. In environments with few objects, this may be the case. In some cases, the cursor may not be displayed to the users and instead, focus indicators (e.g. glows around objects), are used to give visual cues as to where the cursor should be located (e.g. the cursor will be positioned close to the object that has the brightest glow). This can happen in environments with a lot of objects, where the cursor may not be necessary or distracting.
A conventional cursor renders without taking into account scene content. As the cursor moves around the VR/AR/MR environments, it will move over the objects in the environment (e.g. be rendered in front of them). In the above example, the cursor could appear in front the virtual video display. This would not only obscure the display but also distract the user from what is being shown. The cursor may appear in front of content and cause the user to focus on it rather than the actual content.
When a cursor hovers over an object or when it is used to select that object, at least a part of the object is covered or occluded. This obstruction of the object’s view can have a significant impact on the user’s experience in the environment. The object may include text, images or other content. This can make it difficult for the user to select the object and view the content.
While these issues are present in 2D environments, they can be magnified in 3D environments. In a 2D space, for example, both the cursor and the objects do not have depth. The cursor is rendered in front of the object by rendering both the cursor as well as the object on the exact same plane. In a 3D space, the cursors and objects have depth in relation to the user. A cursor is not always at the same level of depth as an item in a 3D space. The cursor, for example, may be closer or further away from the user in relation to an object. This difference in depth can cause the user to see the cursor or object as blurry if they focus on the one. Even if a cursor or an object have the same depth in a 3D environment, relative to the user’s perspective, the cursor will still “roll over” the object. The system must adjust the depth of the mouse to prevent the cursor from appearing to move through an object. As shown in FIG. The system can move the cursor to be closer to the user, as shown in FIGS. The system can emphasize the cursor in relation to an object by bringing it closer to a user. This is undesirable because people are drawn to objects closer to them.
To solve these and other issues, embodiments can render a visual aid on screen that is aware of content. When a cursor overlaps with an object, the system may render the cursor in the background (rather then in front) of the object, or it may not render the mouse cursor (because it is hidden behind the object, and therefore not visible to users). The cursor will not obscure the object in the user’s view, nor will the system unintentionally emphasize the cursor, by bringing it closer to them. Eclipse cursors or eclipse reticles are some examples of this type of cursor. This is because the target object “eclipses” it. The cursor
When a cursor has been eclipsed by an item, it can be difficult for an user to know where they are pointing in the scene. Or where the cursor currently is. The cursor has been partially or completely blocked by the object. To continue to give the user a sense of the cursor?s position in the environment, the system may render an additional (or alternative) visual aid on the screen (e.g. a focus indication) to highlight the object when the cursor moves behind it (or within a certain distance threshold from it).
A focus indicator may include a halo or color, or a change in perceived size or depth (e.g. making the object appear larger or closer when selected), or shading, virtual rays or other graphical highlights emanating or associated with an object that tends to attract the user’s eye. The focus indicator could include, for example, a glow radiating outwards from an object as if there were a glowing source of light behind it (so the object “eclipses” the light source). The focus indicator can include a glow that appears to radiate outward from an object, as if a glowing light source were situated behind the object (so that the object?eclipses? The intensity of glow can be greater near the outer edge of an object, and less so at a distance from it. The focus indicator emphasizes the item instead of obscuring it (since it is usually rendered partially around the object). This is a more user-friendly and non-distracting option to the cursor for indicating which object is being interacted with.
In some cases, the proximity of a cursor to an object can affect the intensity or position of a focus marker or the cursor. The attractive effect can act as though the cursor (or object) and focus indicator were gravitationally or magnetically attracted. In some cases, the cursor’s position relative to an object or focus indicator can affect the intensity, size or location of a focus indication. As the cursor gets closer to an item, the focus indicators of the object may become brighter or more intense. They can also move towards the cursor (as if it were being pulled). The system can make the cursor appear as if it is being pulled behind an object while increasing the intensity of the focus indicators. This may allow the user to select objects more easily and naturally, as the cursor will be pulled towards (or snaps to) the nearest object when it gets close to the desired target object. The user does not have to make any fine adjustments in order to position the cursor. The cursor may act as if it has mass or inertia, so that the cursor moves in the direction initially selected. It is also pulled toward nearby objects by the attractive effect. The intensity of the focus indicator (s) can change as the cursor moves within the environment.
In some cases, the focus indicator may be assigned to more than one item. Or, it can be given a different intensity or glow that can fade in and out depending on the proximity of an object to the cursor. One or more focus indicator can provide positional feedback by highlighting one or several objects at different intensities. The intensity of the glow or varying intensity can change as input changes to provide accurate cursor positioning and sustained feedback.
The system can be arranged to include an interface which presents a number of interactive virtual items (e.g. icons, thumbnails etc.) arranged on a grid of virtual content. At one or more depths. The thumbnail for an item can be rendered in one or more ways in response to the cursor moving in front of it. These include: increasing in size, adding a focus indicator, moving to another depth (e.g. to one that appears closer to the user), and having different virtual contents (e.g. a higher-resolution image, a caption or sound, or playing a video, animation, or graphic). The thumbnails can be arranged according to one or several grouping criteria, such as alphabetically, by content type, date etc .).
The grid of thumbnails can be scrolled by the user, e.g. using eye, head or body gestures or input from a user totem. The edges of the grid, e.g. in the direction of scrolling, may dynamically display indications that virtual content is about to be displayed (e.g. as semi-transparent thumbs, optionally with a different depth from the edge of grid).
In some cases, the GUI may render a viewable box that only shows a small fraction of the virtual content. (The viewable box that contains the virtual content can be either the entire field of view or a part of it.) The virtual content displayed by the GUI may be a subset from a content library which includes hidden content (e.g. un-rendered content) that extends outside the boundaries of the viewable windows. Scrolling initiated by the user may bring one or more hidden portions into view. The GUI may include a scrollbar that provides real-time feedback in relation to the scrolled contents. Scrollbars can have a bar moving within a trough. The bar’s position can indicate what content in the content library is being viewed by the viewer within the viewable area. The bar’s length can be used to represent the percentage of the content library being rendered in the viewable window. The bar can have a fixed edge to indicate the position of the virtual content where the user began scrolling from and a moving edge to show the current position. The temporarily fixed edge may become unfixed when scrolling stops and can move (or snap into) a final position. This will give the bar a length and a position that are representative of the fractional amount of virtual content and its location.
The scrollbar can provide useful feedback to users, such as the position and amount of virtual content displayed in the viewable area relative to the total virtual content.Click here to view the patent on Google Patents.