Software Technology for “Spatial random acces enabled video system with three-dimensional viewing volume

Metaverse – Derek Pang, Colvin Pitts, Kurt Akeley, Google LLC

Abstract for “Spatial random acces enabled video system with three-dimensional viewing volume

An environment can be shown from a different viewpoint. One method is volumetric video data. This may include a tiled array of cameras. One viewing volume may have multiple vantage points. Each vantage may have its own video data, which can be used to create video data representing the environment. A viewpoint may be designated by a user. A subset of the viewing volume may be chosen from amongst the many vantages. The subset of video data may be combined to create viewpoint video data that depicts the environment from the viewpoint. Viewers can view the viewpoint video data to see the environment from the viewpoint chosen by them.

Background for “Spatial random acces enabled video system with three-dimensional viewing volume

“Display of a volume or positional tracking video can enable viewers to see a captured scene from any angle and location within a viewing volume. A viewpoint can be rebuilt using the data from such a system to correct view-dependent lighting and motion parallax. The user can experience a virtual presence in an environment by viewing the video through a virtual reality head-mounted monitor. This virtual reality experience can provide six degrees freedom and uncompromised stereooscopic perception at any distance.

Video with three-dimensional viewing volumes presents a major challenge. The data volume may be 100 times greater than the one for conventional two-dimensional video. As the viewing volume grows, so does the data required. It may prove difficult for viewers to store content on consumer-grade devices, or for distributors to send the content over the internet due to the large data requirements. For low-latency, high-fidelity playback on today’s computers, it may be difficult to meet memory, bandwidth and/or processing needs. These challenges can be addressed by a well-designed compression and playback scheme.

“A viewer of virtual reality or augmented realities experiences is limited to observing a specific field-of-view (FoV), at any one time. Practical systems may only retrieve and render the FoV needed from the video volume data. A spatial random access coding scheme and viewing scheme can be used to address the data and complexity challenges. This will allow viewers to have arbitrary access to their preferred FoV in a compressed volumetric video stream. To improve the efficiency of the system’s code, inter-vantage and inter-spatial-layer predictions could also be used.

Different spatial resolution layers can be used to represent different vantage points in order to meet the varied complexity, display, and/or bandwidth characteristics for client devices and/or networks. You can conceal network errors or spatial random access latency by using lower spatial layers. These layers can also be used to support multi-spatial resolution layer perceptual rendering techniques.

Inter-vantage prediction can be used to reduce storage and bandwidth requirements. It uses the scene’s geometric information and exploits redundancies between vantages. Inter-spatial layer predictions can be used to reduce storage requirements and/or improve error resilience when using multiple spatially-resolution layers.

Multiple methods are described for creating virtual views of light-field data and capturing image or video data. These embodiments allow for continuous, or almost continuous, light-field data to be captured from many or all directions away from the capture system. This may make it possible to generate virtual views that are more precise and/or give viewers greater viewing freedom.

“Definitions”

“For the purposes of this description, the following definitions will be used:

“In addition, to simplify nomenclature, we use the term ‘camera? “Camera” is used herein to denote an image capture device, or another data acquisition device. Any device or system that records, measures, estimates, determines, calculates, calculates, and/or computes data representative of a scene can be considered a data acquisition tool. This includes but is not limited to light-field data, two-dimensional data, three-dimensional data, and/or three-dimensional data. This data acquisition device can include optics, sensors, or image processing electronics to acquire data representative of a scene using well-known techniques. The disclosure can be used with many data acquisition devices. One skilled in the art will know that this disclosure does not limit to cameras. The term “camera” is used in this context. This disclosure is intended to be illustrative, exemplary and not limit its scope. Any use of this term should be understood to mean any device that can acquire image data.

“The following description describes several methods and techniques for processing light-field photos. These techniques and methods are easy to use, and can be used in combination.

“Problem Description”

Virtual reality is designed to provide a full immersive experience for users. Often, the goal is to create an experience that feels as real as possible. Most headsets have stereo sound and multidirectional sound. Onboard sensors can also measure position, accelerations and orientation. FIG. FIG. 27 is an image of an Oculus Rift Developer Kit headset that serves as an example for a virtual reality headset 2700. Viewers of virtual reality headsets and/or augmented realities may move their heads in any direction. They can move forward, backward and sideways. The user’s point of view may change depending on the movement of their head.

“FIG. “FIG.27” shows some of the components of the virtual reality headset 2270. The virtual reality headset 2700 could have a processor 2710 and memory 2720, as well as a data store 2730, user input 2740 and a display screen 2775. These components can be any device that is used in computing or virtual reality arts to process data, store data for short-term and long-term use, receive user input and display a view. One or more sensors may be used to detect the orientation and position of the virtual reality headset 2700 in some embodiments. A user, also known as a “viewer”, can move his or her head to view the virtual reality headset 2700. You can choose the view point and/or direction you want to see an environment from.

“The virtual reality headset 2700 could also include additional components that are not shown in FIG. 27. The virtual reality headset 2700 can be used either as a standalone device or in conjunction with a server to provide video, audio, and/or any other data to it. The virtual reality headset 2700 can also be used as a client computing device. Any of the components in FIG. 27 could be distributed between the virtual headset 2700, and a nearby computing unit, so that the virtual headset 2700, and the nearby computing unit, together, create a client computing machine.

Virtual reality content can be divided into two parts: synthetic content and real-world content. Synthetic content can include computer-generated movies or video games. Real-world content could include panoramic imagery or live action video, which are captured from actual places and events.

“Synthetic content can contain or be generated from a 3D model of the environment. This may also be used to create views that match the viewer’s actions. This could include changing views to accommodate head orientation or position and may also include adjustments for distances between eyes.

Real world content is harder to capture using known methods and systems. The hardware used to capture it is fundamentally limited. FIGS. FIGS. 7 and 8 illustrate exemplary capture systems 700, 800, and 800, respectively. Specifically, FIG. FIG. 7 shows a virtual reality capture method, or capture system 700 according to Jaunt’s prior art. The capture system 700 is made up of several traditional video capture cameras 710 that are arranged in a spherical fashion. The traditional video capture cameras 710 face outwards from the surface. FIG. FIG. 8 shows a stereo virtuality capture system (or capture system 800) according to the prior art. Eight stereo camera pairs 810 and one vertically facing camera 820 make up the capture system 800. The camera pairs 810 are placed facing outwards from a ring and capture image or video data. The capture system 700 and 800 limit the number of views that can be captured.

The viewer may not be able to see the real-world content captured with these systems accurately if they are only viewing it from one of the camera viewpoints. A intermediate viewpoint is required if the viewer views the scene from between cameras. Although there are many ways to create intermediate viewpoints, each has its limitations.

Another method is to try to create intermediate viewpoints from captured data. Then, interpolate between viewpoints using at least part of the generated model. Although this model allows for more freedom of movement, it is severely limited by the quality and limitations of the three-dimensional model. It is extremely difficult to model certain optical aspects such as specular reflections and partially transparent surfaces. This type of approach’s visual success is greatly dependent on how much interpolation is needed. This type of interpolation might work well for some content if the distances between cameras is very small. Any errors that are made in interpolation will be more obvious as the distance between the cameras increases.

“Another way to generate intermediate viewpoints is to include manual correction and/or art in the postproduction workflow. Although manual processes can be used to correct many kinds of issues, they are expensive and time-consuming.

“A capture system capable of capturing a continuous or almost continuous set viewpoints can remove or greatly reduce interpolation required for generating arbitrary viewpoints. This allows the viewer to move more freely within a given space.

“Tiled array of light-field cameras”

The present document describes several architectures and arrangements that enable the capture of light-field volume data from continuous, or almost continuous viewpoints. You can arrange the viewpoints to cover a surface, or a volume with tiled arrays light-field cameras. These systems are sometimes called “capture systems”. This document. An array of tiled light-field cameras can be combined and arranged to create a continuous, or almost continuous, light-field capture surface. This continuous capture surface can capture light-field volumes. You can use the tiled array to create any shape or size capture surface.

“FIG. 2. A conceptual diagram of a light field volume 200 according to one embodiment. FIG. FIG. 2 shows that the light-field volume 200 can be considered a spherical volume. Rays of light 220 that originate outside the light-field volumes 200 and intersect with the 200 light-field volumes 200 may have their intensity, color, intersection location and direction vector recorded. All rays and/or “ray bundles” are captured in a fully sampled volume of light-field. All rays and/or?ray bundles that originate outside of the light-field volume will be captured and recorded. A subset of intersecting rays can be recorded in a light-field volume that has been partially or sparsely sampled.

“FIG. “FIG. 2. The light-field volume could be a fully sampled volume of light-field light; therefore, all rays entering the light field volume 200 may have been captured. Any virtual viewpoint in the light-field volume 200 may be generated, regardless of direction.

“In FIG. “In FIG. These subviews may be presented to the viewer of a VR system, which shows the subject matter in the light-field volume 200. Each viewer may see one subview 300. Subview generation may not be possible due to sampling patterns, acceptance angles and the surface coverage of the capture systems.

Referring to FIG. 9 shows a capture system, 900 according to one embodiment. A set of light-field cameras 910 may be included in the capture system 900 to form a continuous, or almost continuous capture surface 920. Light-field cameras 910 can cooperate to capture part or all of a light field volume, such the light-field volume 200 in FIG. 2.”

The attached circuitry 930 controls and displays the readout of each light-field camera 910. This circuitry 930 can control operation of the attached light field camera 910 and can also read captured image or video data from light-field cam 910.

The capture system 900 could also include a user interface 940 to control the array. The user interface 940 can be attached to the rest of the capture systems 900 or remotely connected to the rest of the capture systems 900. The user interface 940 can include a graphical user interface and analog controls.

“The capture system 910 may have a primary controller (950) that communicates with all light-field cameras 910. The primary controller 950 could be used to control individual light-field camera 910 or synchronize them all.

“The capture system 910 may also contain data storage 960. This may include remote and onboard components to record the video and/or image data captured by the light-field camera 910. “The capture system 900 may also include data storage 960, which could include onboard and/or remote components for recording the captured video and/or image data generated by the light-field cameras 910.

“The capture system 970 may also contain data processing circuitry 970. This may be used to process image and/or videos as part of the capture device 900. Any type of processing circuitry may be included in the data processing circuitry 970, which could include one or more microprocessors. Alternate embodiments of the capture system 900 could simply collect and store raw information, which can be processed by another device, such as a computing device equipped with microprocessors or other data processing circuitry.

“In FIGS. “In FIGS. 23 and 24, nine light field cameras 2310 or 2410 are shown in every layer. It is important to understand that every layer could benefit from having more or less light-field devices 2310 and 24 respectively, depending on which light-field camera they are using. You can also use other camera arrangements, which could include additional layers. In certain embodiments, enough layers can be used to create or approximate a spherical arrangement for light-field cameras.

“In at most one embodiment, the tiled-light-field cameras are placed on the outside facing surface of a volume or sphere. FIG. FIG. 11 illustrates possible configurations of the tiled array. Specifically, FIG. FIG. 11A illustrates a tiling pattern 1100 from light-field cameras, which creates a cubic volume. FIG. FIG. 11B illustrates a pattern of tiling 1120 in which quadrilateral areas may be warped to approximate the surface a sphere. FIG. FIG. 11C shows a pattern for tiling 1140 that is based on a geodesic domed. The tile shape can alternate between hexagons and pentagons in the tiling pattern 1400. These tiling patterns will be highlighted in a darker color. The number of tiles used in the exemplary patterns is only an example. However, any system can use as many tiles as it needs. You can also create many other volumes or tiling patterns.

“Notably the tiles in the tiling patterns 1100, 1120 and 1140 represent the maximum light-field capturing area for one light-field camera within a tiled array. The physical capture surface might closely match the size of the tile in some embodiments. In some embodiments, the physical catch surface may be significantly smaller than the tile size.

“Size and Field of View for the Tiled Array”

“For many virtual reality or augmented reality viewing experiences the?human natural? is required. Viewing parameters are required. This is also known as “human natural”. Viewing parameters are specifically concerned with providing approximate human fields-ofview and inter-ocular distances. It is also desirable to be able to generate accurate image or video data for any viewpoint, as the viewer moves his/her head.

The output requirements and the fields-of-view for the objective lenses within the capture system may determine the physical size of the tiled array’s capture surface. FIG. FIG. 4. Conceptually, this shows the relationship between a physical surface or capture surface 400 and an acceptance or capture field-of-view (410) and a virtual fully-sampled light-field volume (420). A fully sampled volume of light-field is one that has all the incoming rays captured. This volume, for example, the sampled-light-field volume 420, allows you to generate any virtual viewpoint, in any direction and with any field of view.

“In one embodiment, the tiled array has sufficient size and captures enough field-of-view to allow generation of viewpoints that enable VR viewers to freely move heads within normal ranges of neck motion. This motion can include rotation, tilting, and/or translational movement of the head. For example, 100mm may be the radius you want for such a volume.

Referring to FIG. 34, it can be seen that the physical radius of the capture surface 400, r_surface, and the capture surface field-of-view half angle, surface_half_fov, may be related to the virtual radius of the fully sampled light-field volume, r_complete, by:\nr_complete=r_surface*sin(surface_half_fov)”

“To complete the example: the physical capture surface (or capture surface 400) may be designed to be at minimum 300 mm in diameter in order to accommodate system design parameters.

“In one embodiment, the tiled array light-field cameras captures enough field-of-view and is sufficiently large to permit viewers to view in any direction without translational motion. The fully sampled light field volume 420 might be large enough to create virtual views with sufficient separation to allow normal human viewing. One embodiment has a diameter of 60 mm for the fully sampled light field volume 420. This gives it a radius of 30. The radius of the capture surface 400 can be as small as 90 mm if you use the lenses in the above example.

“Tiled array of Plenoptic Light Field Cameras”

“Many types of cameras can be used in a tiled array of camera, as described herein. The light-field camera in the tiled array is, at least one embodiment of it, plenoptic light field cameras.

Referring to FIG. 1. A plenoptic light field camera 100 can capture a light-field with an objective lens 110 and a plenoptic microlens arrangement 120. The photosensor 130 is also included. An aperture may be used to position the objective lens 110. Each of the microlens of the plenoptic microlens assortment 120 could create an image of what is called the aperture on the photosensor 130. The plenoptic light field camera 100 can capture data about the direction at which light rays are received from the photosensor 130. This may allow for the creation of viewpoints in a sampled volume of light that are not aligned to any of the lens cameras of the capture system. We will explain this in more detail below.

“To generate physically accurate virtual views from any place on a physical capture area such as the capture face 400 in FIG. 4. The light-field can be captured as much as 400 pixels of the capture system. FIGS. 25A and 25B illustrate the relationship between a plenoptic camera, such as the plenoptic camera 100 in FIG. 1, and a virtual array of 2500 that are roughly optically equivalent.”

“In order to get as close to a continuous light field capture surface when spanning multiple cameras,” the entrance pupil of one light-field camera might be as near to the entrance pupil(s), from neighboring camera(s). FIG. FIG. 10 shows a tiled 1000 array in a ring configuration. The entrance pupils 1010 and objective lenses 1020 create a smooth surface on the tiled 1000 1000.

“In order to allow the entrance pupils 1010 of neighboring objective lenses 1020 create a nearly continuous surface the entrance pupil 1010 might be larger relative to each light-field camera 1030 in a tiled array 1000 as shown in FIG. 10. It may also be advantageous to choose a lens with a wide field of view to get large viewing angles in a small volume. A good lens design may have a large field-of-view and a large aperture. (Aperture size and the entrance pupil size are closely related).

“FIG. “FIG. 13” is a diagram 1300 that shows typical fields-ofview and aperture ranges of different lens designs. One embodiment uses a double Gauss design 1310 with low F-number as the objective lens. Alternate embodiments may use different types of lenses, such as those shown on the diagram 1300.

“FIG. 14 is a cross-section view of a double Gauss 1400 lens with a large aperture. Double Gauss lenses offer a desirable combination in terms of both field-of-view as well as a large entrance pupil. F/1.0 or lower is available for 50mm lenses, which are suitable for 35mm cameras. These lenses can use an aperture stop greater than 50mm for a sensor measuring approximately 35mm in width.

“In one embodiment, a tiled array could have plenoptic-light-field cameras. The aperture stop and entrance pupil are rectangular. The objective lenses’ entrance pupils create a continuous or almost continuous surface on the capture device. You can shape the aperture stop to permit gap-free tessellation. FIG. 10. The entrance pupil 1010 could be rectangular or square. One or more lenses elements can be cut (for instance, squared) in order to facilitate close bonding and match the shape the aperture stop. The layout and packing of microlens arrays, such as the plenoptic array 120 in FIG. 1, can be optimized to further optimize the design. 1. may be optimized to match the shape of the entrance pupils 1010. The plenoptic microlens array 120 could have a rectangular or square shape, and the packing may be adjusted to match the square or rectangular form of the entrance pupil 1010.

“In one embodiment, the objective lens is a lens with a large field-of-view and large entrance pupil. The lenses are placed as close as possible, while keeping the traditional round shape. A double Gauss type lens may be a good option for the objective lens. It could also have a large aperture.

FIG. 15. The objective lenses 1520 can be circular. This is also true for the entrance pupils 1510 from the light-field cameras 15.30. As shown in the side view to the right, the entrance pupils 1510 might not be continuous to one another. These types of objective lenses can be used in any type of tiling. Another embodiment uses two different lens diameters to create a geodesic dome with light-field cameras. FIG. 1140 shows the tiling pattern. 11C. This arrangement could help reduce the distance between entrance pupils 1510 to improve the continuity of light-field data.

“In one embodiment, one to three top-facing cameras can be used in conjunction with a tiled array within a ring configuration. FIG. FIG. 12 Conceptually, this depicts a tiled arrangement 1200 with light field cameras 1210 placed in a ring-shaped design with one light-field camera 1220 facing upwards. A second light-field camera 1220, not shown, may be placed on the opposite side to the tiled array 1200. It may be oriented in an opposite direction to the light-field cam 1220.

“Notably the upward or downward facing light field camera(s), 1220 could be a standard two-dimensional camera, light-field camera (s), or a combination thereof. These embodiments may capture incomplete light-field volume data directly below and above the tiled array 1200. However, they may provide significant cost savings and/or reduced complexity. Sometimes, views that are directly above or below the tiled array 1200 might be less important than those in other directions. A viewer might not need as much detail or accuracy looking up and down than when viewing images at the elevation.

“Changing the Rotational Position of the Tiled Array.”

“In at most one embodiment, the surface may be modified to allow it to rotate and capture different viewpoints at different times. Each frame can be used to capture portions that are not captured in the previous frame by changing its rotational position.

“Referring FIGS. Referring to FIGS. 16A-16C, a sensor array 1600 could be a sparsely populated circle of plenoptic-light-field cameras 1610. Each frame could capture a different set angles than the one before it.”

“Specifically, at the time A, a part of the light-field volume was captured. Rotating the ring to position A, the sensor array 1600 captures another portion of light-field volume. Rotating the ring once more to rotate the sensor array 1600, and another capture is made at time C.

This embodiment allows for finer sampling, complete sampling, and/or less hardware. The ring configuration shows the embodiments that have a changing rotational position. The principle can be applied to any tiled arrangement, however. Rotation can be performed around one axis as shown in FIGS. 16A through 16C can be rotated about one axis or several axes if needed. For example, a spherically-tiled configuration can be rotated around all three orthogonal directions.

“In one embodiment, the camera array rotates in a single direction between each capture. As shown in FIGS. 16A-16C Another embodiment of the camera array oscillates among two or more capture positions, and may change the direction in which rotation takes place between captures.

The overall frame rate of the system can be high enough to capture every rotation at a sufficient frame rate for video capture. If output video is required at 60 frames per seconds, and the capture system uses three distinct capture positions, the overall frame capture speed, including time for position changes, could be higher than or equal 180 frames per sec. This allows samples to be taken at each position, in sync with the desired frame rate.

“In at most one embodiment, the whole sensor array 1600 may be attached via a rotary joint. This allows the tiled array rotate independently from the rest of system and surrounding. To connect non-rotating components to rotating components of the system, the electrical connections can be made through a slipring or rotary electrical interface. A motor 1620 may drive the rotation and/or oscillation. It may be a stepper, DC motor or other suitable motor system.

“Changing Rotational Location of Light-Field Sensors

“In at most one embodiment, light-field sensors in the capture system can be rotated to capture different viewpoints at different times. However, objective lenses may remain fixed. Each frame can be used to capture portions that weren’t captured in the previous frame by changing the position of the sensors.

Referring to FIGS. 17A-17C: A sensor array 1700 could include a ring that includes a full set 1710 objective lenses and a small set 1720 light-field sensors. The sensor array 1700 can capture images from one subset of objective lenses 1710 at each time. The array of light-field sensor 1720 can rotate while the objective lenses 1710 will remain in a fixed position.

“At time A, a portion is captured that corresponds with the objective lenses 1710 active at that time (i.e. the objective lenses1710 that align with one of light-field sensors 1720). After the light field sensors 1720 have been rotated to position B, another portion of light-field volume is captured. This time, it corresponds with the objective lenses 1710 which are aligned with the light fields sensors 1720. Rotate the light-field sensors 1720 again and capture another portion at time C.

This embodiment allows for finer sampling, complete sampling, and/or less hardware. The ring configuration shows the embodiments that have a changing rotational position. The principle can be applied to any tiled arrangement, however. Rotation can be performed around one axis as shown in FIGS. 17A to 17C can be rotated about one axis. However, multiple axes may be used if necessary. For example, a spherically-tiled configuration can be rotated around all three orthogonal directions.

“In one embodiment, light-field sensor arrays rotate in the same direction between captures, as shown in FIGS. 17A-17C. Another embodiment of the light-field sensor array can oscillate between multiple capture positions, and may change the direction in which rotation occurs between captures. FIGS. 18A-18C

“FIGS. 18A through 18C show a sensor array 1800. This may include a ring that has a full set 1810 objective lenses and a small set 1820 light-field sensors, such as FIGS. 17A-17C The objective lenses 1810 can remain in a fixed position, while the array light-field sensors (1820) rotates. The array of light-field sensor 1820 may move clockwise starting at FIG. 18A to FIG. 18A to FIG. 18B to FIG. 18B to FIG. 18C, returning in FIG. 18A. 18A. The array of light field sensors 1820 could oscillate between two or three relative positions.

“In at most one embodiment, the arrays of light field sensors 1720 and/or 1820 may be attached directly to a rotary joint. This allows the arrays of lightfield sensors 1720 and arrays of tiled lightfield sensors 1820 to rotate independently from the rest of the capture systems and surrounding. To connect non-rotating parts to rotating components of the system, the electrical connections can be made through a slipring or rotary electrical interface. A stepper motor, DC motor or other suitable motor system can be used to drive the rotation and/or oscillation.

“Tiled array of Array Light Field Cameras”

According to the present disclosure, a wide range of cameras can be used in a tiled arrangement. The array light-field camera used in the tiled array is at least one embodiment. FIG. 6 is an example of such a configuration. 6.”

“FIG. “FIG. 6 can be tiled to create a virtually continuous capture surface 1900. The FIG. shows a ring-tiling pattern. 19 may be used with any tiling scheme, not just those shown in FIG. 11A, 11B and 11C.

“Changing the Rotational Position of a Tiled Array Of Array Light-Field Cameras.”

“Array light field cameras and/or their components may be rotated in order to capture a greater amount of light-field than with stationary components. FIGS. 16A through 16C,17A through 17C, or 18A through 18C can be applied to array-light-field cameras such as the array light field camera 600 of FIG. 6. These will be discussed in more detail in conjunction with FIGS. 32A through 32C, and FIGS. 10A to 10C.”

“In at most one embodiment, the surface of an array of light-field cameras can be changed to rotate and capture different viewpoints at different times. Each frame can be used to capture portions that were not captured in the previous frame by changing its rotational position, such as FIGS. 16A-16C

“Referring FIGS. “Referring to FIGS. 32A through 32C. A sensor array 3200 could be a sparsely-populated ring light-field cameras 3210. Each frame could capture a different set angles than the one before it.”

“Specifically, at the time A, a part of the light-field volume was captured. Rotating the ring to position B of the sensor array 3200, another portion is captured. Rotating the sensor array 3200 again by rotating the ring is done. Another capture at time C is taken.

This embodiment allows for finer sampling, complete sampling, and/or less hardware. You may also reap the benefits of using array light-field cameras. The embodiments that change rotational positions are shown in a ring configuration for clarity. The principle can be applied to any tiled arrangement, however. Rotation can be performed around one axis as shown in FIGS. 32A through 32C can be rotated about one axis. However, multiple axes may be used if necessary. For example, a spherically-tiled configuration can be rotated around all three orthogonal directions.

“In one embodiment, an array light-field camera array rotates in a single direction between each capture. This is the same as FIGS. 32A through 32C. Another embodiment of the array light-field cam array oscillates between multiple capture positions, and may alter the direction of rotation among captures.”

The overall frame rate of the system can be high enough to capture every rotation at a sufficient frame rate for video capture. If output video is required at 60 frames per seconds, and the capture system uses three distinct capture positions, the overall frame capture speed, including time for position changes, could be higher than or equal 180 frames per sec. This allows samples to be taken at each position, in sync with the desired frame rate.

“In at most one embodiment, the entire sensor array 3200 can be attached to a rotating joint. This allows the tiled array rotate independently from the rest of system and surrounding. To connect non-rotating components to rotating components of the system, the electrical connections can be made through a slipring or rotary electrical interface. A stepper motor, DC motor or other suitable motor system can be used to drive the rotation and/or oscillation.

“Changing Rotational Location of the Photosensors in Array Light-Field Cams”

“In at most one embodiment, array light-field camera light-field sensors may be rotated so that different viewpoints can be captured at different times. However, arrays of objective lenses may remain in the same fixed position. Each frame can be used to capture light-field volumes that have not been captured in the previous frame by changing the position of the sensors.

Referring to FIGS. 20A and 20B may contain a ring that includes a complete set of objective lenses 2010, with a small set of light-field sensors 2020. The sensor array 2000 can capture images from one subset of the arrays. While the arrays of objective lens 2010 may remain in a fixed location, 2020’s array of light-field sensor 2020 may move.

“At time A, a portion is captured that corresponds with the arrays objective lenses 2010 that were active at the time (i.e. the arrays objective lenses 2010 aligned with one of light-field sensors 2020). After the light field sensors 2020 have been rotated to the position shown in time B, another portion of light-field volume is captured. This time, it corresponds with the different arrays of objective lens 2010 that align with the 2020 light-field sensor. The light-field sensor 2020 is then rotated to reach the position at Time A. Capture may continue oscillating between the configurations at Time A and B. You can achieve this by continuous, unidirectional rotation (as shown in FIGS. 17A through 17C, or oscillating motion where rotation reverses the direction between captures as in FIGS. 18A-18C

This embodiment allows for finer sampling, complete sampling, and/or less hardware. You may also reap the benefits of using array light-field cameras. The embodiments that change rotational positions are shown in a ring configuration for clarity. The principle can be applied to any tiled arrangement, however. Rotation can be performed around one axis as shown in FIGS. 20A and 20B can be rotated around one axis. However, multiple axes may be used if necessary. For example, a spherically-tiled configuration can be rotated around all three orthogonal directions.

Summary for “Spatial random acces enabled video system with three-dimensional viewing volume

“Display of a volume or positional tracking video can enable viewers to see a captured scene from any angle and location within a viewing volume. A viewpoint can be rebuilt using the data from such a system to correct view-dependent lighting and motion parallax. The user can experience a virtual presence in an environment by viewing the video through a virtual reality head-mounted monitor. This virtual reality experience can provide six degrees freedom and uncompromised stereooscopic perception at any distance.

Video with three-dimensional viewing volumes presents a major challenge. The data volume may be 100 times greater than the one for conventional two-dimensional video. As the viewing volume grows, so does the data required. It may prove difficult for viewers to store content on consumer-grade devices, or for distributors to send the content over the internet due to the large data requirements. For low-latency, high-fidelity playback on today’s computers, it may be difficult to meet memory, bandwidth and/or processing needs. These challenges can be addressed by a well-designed compression and playback scheme.

“A viewer of virtual reality or augmented realities experiences is limited to observing a specific field-of-view (FoV), at any one time. Practical systems may only retrieve and render the FoV needed from the video volume data. A spatial random access coding scheme and viewing scheme can be used to address the data and complexity challenges. This will allow viewers to have arbitrary access to their preferred FoV in a compressed volumetric video stream. To improve the efficiency of the system’s code, inter-vantage and inter-spatial-layer predictions could also be used.

Different spatial resolution layers can be used to represent different vantage points in order to meet the varied complexity, display, and/or bandwidth characteristics for client devices and/or networks. You can conceal network errors or spatial random access latency by using lower spatial layers. These layers can also be used to support multi-spatial resolution layer perceptual rendering techniques.

Inter-vantage prediction can be used to reduce storage and bandwidth requirements. It uses the scene’s geometric information and exploits redundancies between vantages. Inter-spatial layer predictions can be used to reduce storage requirements and/or improve error resilience when using multiple spatially-resolution layers.

Multiple methods are described for creating virtual views of light-field data and capturing image or video data. These embodiments allow for continuous, or almost continuous, light-field data to be captured from many or all directions away from the capture system. This may make it possible to generate virtual views that are more precise and/or give viewers greater viewing freedom.

“Definitions”

“For the purposes of this description, the following definitions will be used:

“In addition, to simplify nomenclature, we use the term ‘camera? “Camera” is used herein to denote an image capture device, or another data acquisition device. Any device or system that records, measures, estimates, determines, calculates, calculates, and/or computes data representative of a scene can be considered a data acquisition tool. This includes but is not limited to light-field data, two-dimensional data, three-dimensional data, and/or three-dimensional data. This data acquisition device can include optics, sensors, or image processing electronics to acquire data representative of a scene using well-known techniques. The disclosure can be used with many data acquisition devices. One skilled in the art will know that this disclosure does not limit to cameras. The term “camera” is used in this context. This disclosure is intended to be illustrative, exemplary and not limit its scope. Any use of this term should be understood to mean any device that can acquire image data.

“The following description describes several methods and techniques for processing light-field photos. These techniques and methods are easy to use, and can be used in combination.

“Problem Description”

Virtual reality is designed to provide a full immersive experience for users. Often, the goal is to create an experience that feels as real as possible. Most headsets have stereo sound and multidirectional sound. Onboard sensors can also measure position, accelerations and orientation. FIG. FIG. 27 is an image of an Oculus Rift Developer Kit headset that serves as an example for a virtual reality headset 2700. Viewers of virtual reality headsets and/or augmented realities may move their heads in any direction. They can move forward, backward and sideways. The user’s point of view may change depending on the movement of their head.

“FIG. “FIG.27” shows some of the components of the virtual reality headset 2270. The virtual reality headset 2700 could have a processor 2710 and memory 2720, as well as a data store 2730, user input 2740 and a display screen 2775. These components can be any device that is used in computing or virtual reality arts to process data, store data for short-term and long-term use, receive user input and display a view. One or more sensors may be used to detect the orientation and position of the virtual reality headset 2700 in some embodiments. A user, also known as a “viewer”, can move his or her head to view the virtual reality headset 2700. You can choose the view point and/or direction you want to see an environment from.

“The virtual reality headset 2700 could also include additional components that are not shown in FIG. 27. The virtual reality headset 2700 can be used either as a standalone device or in conjunction with a server to provide video, audio, and/or any other data to it. The virtual reality headset 2700 can also be used as a client computing device. Any of the components in FIG. 27 could be distributed between the virtual headset 2700, and a nearby computing unit, so that the virtual headset 2700, and the nearby computing unit, together, create a client computing machine.

Virtual reality content can be divided into two parts: synthetic content and real-world content. Synthetic content can include computer-generated movies or video games. Real-world content could include panoramic imagery or live action video, which are captured from actual places and events.

“Synthetic content can contain or be generated from a 3D model of the environment. This may also be used to create views that match the viewer’s actions. This could include changing views to accommodate head orientation or position and may also include adjustments for distances between eyes.

Real world content is harder to capture using known methods and systems. The hardware used to capture it is fundamentally limited. FIGS. FIGS. 7 and 8 illustrate exemplary capture systems 700, 800, and 800, respectively. Specifically, FIG. FIG. 7 shows a virtual reality capture method, or capture system 700 according to Jaunt’s prior art. The capture system 700 is made up of several traditional video capture cameras 710 that are arranged in a spherical fashion. The traditional video capture cameras 710 face outwards from the surface. FIG. FIG. 8 shows a stereo virtuality capture system (or capture system 800) according to the prior art. Eight stereo camera pairs 810 and one vertically facing camera 820 make up the capture system 800. The camera pairs 810 are placed facing outwards from a ring and capture image or video data. The capture system 700 and 800 limit the number of views that can be captured.

The viewer may not be able to see the real-world content captured with these systems accurately if they are only viewing it from one of the camera viewpoints. A intermediate viewpoint is required if the viewer views the scene from between cameras. Although there are many ways to create intermediate viewpoints, each has its limitations.

Another method is to try to create intermediate viewpoints from captured data. Then, interpolate between viewpoints using at least part of the generated model. Although this model allows for more freedom of movement, it is severely limited by the quality and limitations of the three-dimensional model. It is extremely difficult to model certain optical aspects such as specular reflections and partially transparent surfaces. This type of approach’s visual success is greatly dependent on how much interpolation is needed. This type of interpolation might work well for some content if the distances between cameras is very small. Any errors that are made in interpolation will be more obvious as the distance between the cameras increases.

“Another way to generate intermediate viewpoints is to include manual correction and/or art in the postproduction workflow. Although manual processes can be used to correct many kinds of issues, they are expensive and time-consuming.

“A capture system capable of capturing a continuous or almost continuous set viewpoints can remove or greatly reduce interpolation required for generating arbitrary viewpoints. This allows the viewer to move more freely within a given space.

“Tiled array of light-field cameras”

The present document describes several architectures and arrangements that enable the capture of light-field volume data from continuous, or almost continuous viewpoints. You can arrange the viewpoints to cover a surface, or a volume with tiled arrays light-field cameras. These systems are sometimes called “capture systems”. This document. An array of tiled light-field cameras can be combined and arranged to create a continuous, or almost continuous, light-field capture surface. This continuous capture surface can capture light-field volumes. You can use the tiled array to create any shape or size capture surface.

“FIG. 2. A conceptual diagram of a light field volume 200 according to one embodiment. FIG. FIG. 2 shows that the light-field volume 200 can be considered a spherical volume. Rays of light 220 that originate outside the light-field volumes 200 and intersect with the 200 light-field volumes 200 may have their intensity, color, intersection location and direction vector recorded. All rays and/or “ray bundles” are captured in a fully sampled volume of light-field. All rays and/or?ray bundles that originate outside of the light-field volume will be captured and recorded. A subset of intersecting rays can be recorded in a light-field volume that has been partially or sparsely sampled.

“FIG. “FIG. 2. The light-field volume could be a fully sampled volume of light-field light; therefore, all rays entering the light field volume 200 may have been captured. Any virtual viewpoint in the light-field volume 200 may be generated, regardless of direction.

“In FIG. “In FIG. These subviews may be presented to the viewer of a VR system, which shows the subject matter in the light-field volume 200. Each viewer may see one subview 300. Subview generation may not be possible due to sampling patterns, acceptance angles and the surface coverage of the capture systems.

Referring to FIG. 9 shows a capture system, 900 according to one embodiment. A set of light-field cameras 910 may be included in the capture system 900 to form a continuous, or almost continuous capture surface 920. Light-field cameras 910 can cooperate to capture part or all of a light field volume, such the light-field volume 200 in FIG. 2.”

The attached circuitry 930 controls and displays the readout of each light-field camera 910. This circuitry 930 can control operation of the attached light field camera 910 and can also read captured image or video data from light-field cam 910.

The capture system 900 could also include a user interface 940 to control the array. The user interface 940 can be attached to the rest of the capture systems 900 or remotely connected to the rest of the capture systems 900. The user interface 940 can include a graphical user interface and analog controls.

“The capture system 910 may have a primary controller (950) that communicates with all light-field cameras 910. The primary controller 950 could be used to control individual light-field camera 910 or synchronize them all.

“The capture system 910 may also contain data storage 960. This may include remote and onboard components to record the video and/or image data captured by the light-field camera 910. “The capture system 900 may also include data storage 960, which could include onboard and/or remote components for recording the captured video and/or image data generated by the light-field cameras 910.

“The capture system 970 may also contain data processing circuitry 970. This may be used to process image and/or videos as part of the capture device 900. Any type of processing circuitry may be included in the data processing circuitry 970, which could include one or more microprocessors. Alternate embodiments of the capture system 900 could simply collect and store raw information, which can be processed by another device, such as a computing device equipped with microprocessors or other data processing circuitry.

“In FIGS. “In FIGS. 23 and 24, nine light field cameras 2310 or 2410 are shown in every layer. It is important to understand that every layer could benefit from having more or less light-field devices 2310 and 24 respectively, depending on which light-field camera they are using. You can also use other camera arrangements, which could include additional layers. In certain embodiments, enough layers can be used to create or approximate a spherical arrangement for light-field cameras.

“In at most one embodiment, the tiled-light-field cameras are placed on the outside facing surface of a volume or sphere. FIG. FIG. 11 illustrates possible configurations of the tiled array. Specifically, FIG. FIG. 11A illustrates a tiling pattern 1100 from light-field cameras, which creates a cubic volume. FIG. FIG. 11B illustrates a pattern of tiling 1120 in which quadrilateral areas may be warped to approximate the surface a sphere. FIG. FIG. 11C shows a pattern for tiling 1140 that is based on a geodesic domed. The tile shape can alternate between hexagons and pentagons in the tiling pattern 1400. These tiling patterns will be highlighted in a darker color. The number of tiles used in the exemplary patterns is only an example. However, any system can use as many tiles as it needs. You can also create many other volumes or tiling patterns.

“Notably the tiles in the tiling patterns 1100, 1120 and 1140 represent the maximum light-field capturing area for one light-field camera within a tiled array. The physical capture surface might closely match the size of the tile in some embodiments. In some embodiments, the physical catch surface may be significantly smaller than the tile size.

“Size and Field of View for the Tiled Array”

“For many virtual reality or augmented reality viewing experiences the?human natural? is required. Viewing parameters are required. This is also known as “human natural”. Viewing parameters are specifically concerned with providing approximate human fields-ofview and inter-ocular distances. It is also desirable to be able to generate accurate image or video data for any viewpoint, as the viewer moves his/her head.

The output requirements and the fields-of-view for the objective lenses within the capture system may determine the physical size of the tiled array’s capture surface. FIG. FIG. 4. Conceptually, this shows the relationship between a physical surface or capture surface 400 and an acceptance or capture field-of-view (410) and a virtual fully-sampled light-field volume (420). A fully sampled volume of light-field is one that has all the incoming rays captured. This volume, for example, the sampled-light-field volume 420, allows you to generate any virtual viewpoint, in any direction and with any field of view.

“In one embodiment, the tiled array has sufficient size and captures enough field-of-view to allow generation of viewpoints that enable VR viewers to freely move heads within normal ranges of neck motion. This motion can include rotation, tilting, and/or translational movement of the head. For example, 100mm may be the radius you want for such a volume.

Referring to FIG. 34, it can be seen that the physical radius of the capture surface 400, r_surface, and the capture surface field-of-view half angle, surface_half_fov, may be related to the virtual radius of the fully sampled light-field volume, r_complete, by:\nr_complete=r_surface*sin(surface_half_fov)”

“To complete the example: the physical capture surface (or capture surface 400) may be designed to be at minimum 300 mm in diameter in order to accommodate system design parameters.

“In one embodiment, the tiled array light-field cameras captures enough field-of-view and is sufficiently large to permit viewers to view in any direction without translational motion. The fully sampled light field volume 420 might be large enough to create virtual views with sufficient separation to allow normal human viewing. One embodiment has a diameter of 60 mm for the fully sampled light field volume 420. This gives it a radius of 30. The radius of the capture surface 400 can be as small as 90 mm if you use the lenses in the above example.

“Tiled array of Plenoptic Light Field Cameras”

“Many types of cameras can be used in a tiled array of camera, as described herein. The light-field camera in the tiled array is, at least one embodiment of it, plenoptic light field cameras.

Referring to FIG. 1. A plenoptic light field camera 100 can capture a light-field with an objective lens 110 and a plenoptic microlens arrangement 120. The photosensor 130 is also included. An aperture may be used to position the objective lens 110. Each of the microlens of the plenoptic microlens assortment 120 could create an image of what is called the aperture on the photosensor 130. The plenoptic light field camera 100 can capture data about the direction at which light rays are received from the photosensor 130. This may allow for the creation of viewpoints in a sampled volume of light that are not aligned to any of the lens cameras of the capture system. We will explain this in more detail below.

“To generate physically accurate virtual views from any place on a physical capture area such as the capture face 400 in FIG. 4. The light-field can be captured as much as 400 pixels of the capture system. FIGS. 25A and 25B illustrate the relationship between a plenoptic camera, such as the plenoptic camera 100 in FIG. 1, and a virtual array of 2500 that are roughly optically equivalent.”

“In order to get as close to a continuous light field capture surface when spanning multiple cameras,” the entrance pupil of one light-field camera might be as near to the entrance pupil(s), from neighboring camera(s). FIG. FIG. 10 shows a tiled 1000 array in a ring configuration. The entrance pupils 1010 and objective lenses 1020 create a smooth surface on the tiled 1000 1000.

“In order to allow the entrance pupils 1010 of neighboring objective lenses 1020 create a nearly continuous surface the entrance pupil 1010 might be larger relative to each light-field camera 1030 in a tiled array 1000 as shown in FIG. 10. It may also be advantageous to choose a lens with a wide field of view to get large viewing angles in a small volume. A good lens design may have a large field-of-view and a large aperture. (Aperture size and the entrance pupil size are closely related).

“FIG. “FIG. 13” is a diagram 1300 that shows typical fields-ofview and aperture ranges of different lens designs. One embodiment uses a double Gauss design 1310 with low F-number as the objective lens. Alternate embodiments may use different types of lenses, such as those shown on the diagram 1300.

“FIG. 14 is a cross-section view of a double Gauss 1400 lens with a large aperture. Double Gauss lenses offer a desirable combination in terms of both field-of-view as well as a large entrance pupil. F/1.0 or lower is available for 50mm lenses, which are suitable for 35mm cameras. These lenses can use an aperture stop greater than 50mm for a sensor measuring approximately 35mm in width.

“In one embodiment, a tiled array could have plenoptic-light-field cameras. The aperture stop and entrance pupil are rectangular. The objective lenses’ entrance pupils create a continuous or almost continuous surface on the capture device. You can shape the aperture stop to permit gap-free tessellation. FIG. 10. The entrance pupil 1010 could be rectangular or square. One or more lenses elements can be cut (for instance, squared) in order to facilitate close bonding and match the shape the aperture stop. The layout and packing of microlens arrays, such as the plenoptic array 120 in FIG. 1, can be optimized to further optimize the design. 1. may be optimized to match the shape of the entrance pupils 1010. The plenoptic microlens array 120 could have a rectangular or square shape, and the packing may be adjusted to match the square or rectangular form of the entrance pupil 1010.

“In one embodiment, the objective lens is a lens with a large field-of-view and large entrance pupil. The lenses are placed as close as possible, while keeping the traditional round shape. A double Gauss type lens may be a good option for the objective lens. It could also have a large aperture.

FIG. 15. The objective lenses 1520 can be circular. This is also true for the entrance pupils 1510 from the light-field cameras 15.30. As shown in the side view to the right, the entrance pupils 1510 might not be continuous to one another. These types of objective lenses can be used in any type of tiling. Another embodiment uses two different lens diameters to create a geodesic dome with light-field cameras. FIG. 1140 shows the tiling pattern. 11C. This arrangement could help reduce the distance between entrance pupils 1510 to improve the continuity of light-field data.

“In one embodiment, one to three top-facing cameras can be used in conjunction with a tiled array within a ring configuration. FIG. FIG. 12 Conceptually, this depicts a tiled arrangement 1200 with light field cameras 1210 placed in a ring-shaped design with one light-field camera 1220 facing upwards. A second light-field camera 1220, not shown, may be placed on the opposite side to the tiled array 1200. It may be oriented in an opposite direction to the light-field cam 1220.

“Notably the upward or downward facing light field camera(s), 1220 could be a standard two-dimensional camera, light-field camera (s), or a combination thereof. These embodiments may capture incomplete light-field volume data directly below and above the tiled array 1200. However, they may provide significant cost savings and/or reduced complexity. Sometimes, views that are directly above or below the tiled array 1200 might be less important than those in other directions. A viewer might not need as much detail or accuracy looking up and down than when viewing images at the elevation.

“Changing the Rotational Position of the Tiled Array.”

“In at most one embodiment, the surface may be modified to allow it to rotate and capture different viewpoints at different times. Each frame can be used to capture portions that are not captured in the previous frame by changing its rotational position.

“Referring FIGS. Referring to FIGS. 16A-16C, a sensor array 1600 could be a sparsely populated circle of plenoptic-light-field cameras 1610. Each frame could capture a different set angles than the one before it.”

“Specifically, at the time A, a part of the light-field volume was captured. Rotating the ring to position A, the sensor array 1600 captures another portion of light-field volume. Rotating the ring once more to rotate the sensor array 1600, and another capture is made at time C.

This embodiment allows for finer sampling, complete sampling, and/or less hardware. The ring configuration shows the embodiments that have a changing rotational position. The principle can be applied to any tiled arrangement, however. Rotation can be performed around one axis as shown in FIGS. 16A through 16C can be rotated about one axis or several axes if needed. For example, a spherically-tiled configuration can be rotated around all three orthogonal directions.

“In one embodiment, the camera array rotates in a single direction between each capture. As shown in FIGS. 16A-16C Another embodiment of the camera array oscillates among two or more capture positions, and may change the direction in which rotation takes place between captures.

The overall frame rate of the system can be high enough to capture every rotation at a sufficient frame rate for video capture. If output video is required at 60 frames per seconds, and the capture system uses three distinct capture positions, the overall frame capture speed, including time for position changes, could be higher than or equal 180 frames per sec. This allows samples to be taken at each position, in sync with the desired frame rate.

“In at most one embodiment, the whole sensor array 1600 may be attached via a rotary joint. This allows the tiled array rotate independently from the rest of system and surrounding. To connect non-rotating components to rotating components of the system, the electrical connections can be made through a slipring or rotary electrical interface. A motor 1620 may drive the rotation and/or oscillation. It may be a stepper, DC motor or other suitable motor system.

“Changing Rotational Location of Light-Field Sensors

“In at most one embodiment, light-field sensors in the capture system can be rotated to capture different viewpoints at different times. However, objective lenses may remain fixed. Each frame can be used to capture portions that weren’t captured in the previous frame by changing the position of the sensors.

Referring to FIGS. 17A-17C: A sensor array 1700 could include a ring that includes a full set 1710 objective lenses and a small set 1720 light-field sensors. The sensor array 1700 can capture images from one subset of objective lenses 1710 at each time. The array of light-field sensor 1720 can rotate while the objective lenses 1710 will remain in a fixed position.

“At time A, a portion is captured that corresponds with the objective lenses 1710 active at that time (i.e. the objective lenses1710 that align with one of light-field sensors 1720). After the light field sensors 1720 have been rotated to position B, another portion of light-field volume is captured. This time, it corresponds with the objective lenses 1710 which are aligned with the light fields sensors 1720. Rotate the light-field sensors 1720 again and capture another portion at time C.

This embodiment allows for finer sampling, complete sampling, and/or less hardware. The ring configuration shows the embodiments that have a changing rotational position. The principle can be applied to any tiled arrangement, however. Rotation can be performed around one axis as shown in FIGS. 17A to 17C can be rotated about one axis. However, multiple axes may be used if necessary. For example, a spherically-tiled configuration can be rotated around all three orthogonal directions.

“In one embodiment, light-field sensor arrays rotate in the same direction between captures, as shown in FIGS. 17A-17C. Another embodiment of the light-field sensor array can oscillate between multiple capture positions, and may change the direction in which rotation occurs between captures. FIGS. 18A-18C

“FIGS. 18A through 18C show a sensor array 1800. This may include a ring that has a full set 1810 objective lenses and a small set 1820 light-field sensors, such as FIGS. 17A-17C The objective lenses 1810 can remain in a fixed position, while the array light-field sensors (1820) rotates. The array of light-field sensor 1820 may move clockwise starting at FIG. 18A to FIG. 18A to FIG. 18B to FIG. 18B to FIG. 18C, returning in FIG. 18A. 18A. The array of light field sensors 1820 could oscillate between two or three relative positions.

“In at most one embodiment, the arrays of light field sensors 1720 and/or 1820 may be attached directly to a rotary joint. This allows the arrays of lightfield sensors 1720 and arrays of tiled lightfield sensors 1820 to rotate independently from the rest of the capture systems and surrounding. To connect non-rotating parts to rotating components of the system, the electrical connections can be made through a slipring or rotary electrical interface. A stepper motor, DC motor or other suitable motor system can be used to drive the rotation and/or oscillation.

“Tiled array of Array Light Field Cameras”

According to the present disclosure, a wide range of cameras can be used in a tiled arrangement. The array light-field camera used in the tiled array is at least one embodiment. FIG. 6 is an example of such a configuration. 6.”

“FIG. “FIG. 6 can be tiled to create a virtually continuous capture surface 1900. The FIG. shows a ring-tiling pattern. 19 may be used with any tiling scheme, not just those shown in FIG. 11A, 11B and 11C.

“Changing the Rotational Position of a Tiled Array Of Array Light-Field Cameras.”

“Array light field cameras and/or their components may be rotated in order to capture a greater amount of light-field than with stationary components. FIGS. 16A through 16C,17A through 17C, or 18A through 18C can be applied to array-light-field cameras such as the array light field camera 600 of FIG. 6. These will be discussed in more detail in conjunction with FIGS. 32A through 32C, and FIGS. 10A to 10C.”

“In at most one embodiment, the surface of an array of light-field cameras can be changed to rotate and capture different viewpoints at different times. Each frame can be used to capture portions that were not captured in the previous frame by changing its rotational position, such as FIGS. 16A-16C

“Referring FIGS. “Referring to FIGS. 32A through 32C. A sensor array 3200 could be a sparsely-populated ring light-field cameras 3210. Each frame could capture a different set angles than the one before it.”

“Specifically, at the time A, a part of the light-field volume was captured. Rotating the ring to position B of the sensor array 3200, another portion is captured. Rotating the sensor array 3200 again by rotating the ring is done. Another capture at time C is taken.

This embodiment allows for finer sampling, complete sampling, and/or less hardware. You may also reap the benefits of using array light-field cameras. The embodiments that change rotational positions are shown in a ring configuration for clarity. The principle can be applied to any tiled arrangement, however. Rotation can be performed around one axis as shown in FIGS. 32A through 32C can be rotated about one axis. However, multiple axes may be used if necessary. For example, a spherically-tiled configuration can be rotated around all three orthogonal directions.

“In one embodiment, an array light-field camera array rotates in a single direction between each capture. This is the same as FIGS. 32A through 32C. Another embodiment of the array light-field cam array oscillates between multiple capture positions, and may alter the direction of rotation among captures.”

The overall frame rate of the system can be high enough to capture every rotation at a sufficient frame rate for video capture. If output video is required at 60 frames per seconds, and the capture system uses three distinct capture positions, the overall frame capture speed, including time for position changes, could be higher than or equal 180 frames per sec. This allows samples to be taken at each position, in sync with the desired frame rate.

“In at most one embodiment, the entire sensor array 3200 can be attached to a rotating joint. This allows the tiled array rotate independently from the rest of system and surrounding. To connect non-rotating components to rotating components of the system, the electrical connections can be made through a slipring or rotary electrical interface. A stepper motor, DC motor or other suitable motor system can be used to drive the rotation and/or oscillation.

“Changing Rotational Location of the Photosensors in Array Light-Field Cams”

“In at most one embodiment, array light-field camera light-field sensors may be rotated so that different viewpoints can be captured at different times. However, arrays of objective lenses may remain in the same fixed position. Each frame can be used to capture light-field volumes that have not been captured in the previous frame by changing the position of the sensors.

Referring to FIGS. 20A and 20B may contain a ring that includes a complete set of objective lenses 2010, with a small set of light-field sensors 2020. The sensor array 2000 can capture images from one subset of the arrays. While the arrays of objective lens 2010 may remain in a fixed location, 2020’s array of light-field sensor 2020 may move.

“At time A, a portion is captured that corresponds with the arrays objective lenses 2010 that were active at the time (i.e. the arrays objective lenses 2010 aligned with one of light-field sensors 2020). After the light field sensors 2020 have been rotated to the position shown in time B, another portion of light-field volume is captured. This time, it corresponds with the different arrays of objective lens 2010 that align with the 2020 light-field sensor. The light-field sensor 2020 is then rotated to reach the position at Time A. Capture may continue oscillating between the configurations at Time A and B. You can achieve this by continuous, unidirectional rotation (as shown in FIGS. 17A through 17C, or oscillating motion where rotation reverses the direction between captures as in FIGS. 18A-18C

This embodiment allows for finer sampling, complete sampling, and/or less hardware. You may also reap the benefits of using array light-field cameras. The embodiments that change rotational positions are shown in a ring configuration for clarity. The principle can be applied to any tiled arrangement, however. Rotation can be performed around one axis as shown in FIGS. 20A and 20B can be rotated around one axis. However, multiple axes may be used if necessary. For example, a spherically-tiled configuration can be rotated around all three orthogonal directions.

Click here to view the patent on Google Patents.