KR20240121342A - Systems and methods for augmented reality - Google Patents

Abstract

A method for localizing a user operating a plurality of sensing components in an augmented or mixed reality environment is disclosed, the method comprising the steps of transmitting pose data from a fixed control and processing module and receiving the pose data at a first sensing component, wherein the pose data is then transformed into a first component relative pose in a coordinate frame based on the control and processing module. A display unit in communication with the first sensing component is updated with the transformed first component relative pose to render virtual content with improved environmental awareness.

Description

SYSTEMS AND METHODS FOR AUGMENTED REALITY

[0001] This application claims the benefit of and the benefit of U.S. Provisional Patent Application Serial No. 62/627155, filed February 6, 2018, the contents of which are hereby incorporated by reference in their entirety.

[0002] The present disclosure relates to systems and methods for localizing the position and orientation of one or more objects in the context of augmented reality systems.

[0003] Modern computing and display technologies have enabled the development of systems for so-called "virtual reality" or "augmented reality" experiences, in which digitally reproduced images or portions of images are presented to a user in a manner that makes them appear or can be perceived as real. Virtual reality or "VR" scenarios typically involve the presentation of digital or virtual image information without transparency to other actual real-world visual inputs; augmented reality or "AR" scenarios typically involve the presentation of digital or virtual image information as an augmentation to a visualization of the real world around the user.

[0004] For example, referring to FIG. 1, an augmented reality scene (4) is depicted in which a user of AR technology sees a real-world park-like location (6) featuring people, trees, buildings, and a concrete platform (1120) in the background. In addition to these items, the user of AR technology also perceives that he or she "sees" a robot statue (1110) standing on the real-world platform (1120), and a flying cartoon avatar character (2) that appears to be an anthropomorphic bumblebee, but these elements (2, 1110) do not exist in the real world. As it turns out, the human visual perception system is very complex, and creating VR or AR technology that allows for a rich, comfortable and natural-feeling presentation of virtual image elements among other virtual or real-world imagery elements is challenging.

[0005] For example, head-worn AR displays (or helmet-mounted displays, or smart glasses) are typically at least loosely coupled to the user's head, and thus move as the user's head moves. When the user's head motions are detected by the display system, the data being displayed can be updated to account for changes in head pose.

[0006] For example, when a user wearing a head-mounted display views a virtual representation of a three-dimensional (3D) object on the display and walks around an area in which the 3D object appears, the 3D object may be re-rendered for each viewpoint, thereby providing the user with the perception that he or she is walking around an object that occupies real space. When the head-mounted display is used to present multiple objects in a virtual space (e.g., a rich virtual world), measurements of head pose (i.e., the position and orientation of the user's head) may be used to re-render the scene to match the user's dynamically changing head position and orientation, thereby providing an increased sense of immersion in the virtual space.

[0007] In AR systems, detection or calculation of head pose may enable the display system to render virtual objects so that they appear to occupy real-world space in a manner that is perceptible to the user. In addition, detection of the position and/or orientation of a real-world object, such as a handheld device (a handheld device may also be referred to as a "totem"), a haptic device, or other real-world physical object, relative to the user's head or the AR system may also enable the display system to present display information to the user so that the user can efficiently interact with certain aspects of the AR system. As the user's head moves around the real-world, the virtual objects may be re-rendered as a function of the head pose, so that the virtual objects appear to remain stable relative to the real world. At least for AR applications, the placement of virtual objects in spatial relationship to physical objects (e.g., presented so that they appear spatially close to the physical objects in two or three dimensions) may be an important issue. For example, head movement may significantly complicate the placement of virtual objects in a view of the surroundings. This may apply whether the view is captured as an image of the surroundings and then projected or displayed to the end user, or whether the end user directly perceives the view of the surroundings. For example, head movement may cause the end user's field of view to change, which may require updates to the positions where various virtual objects are displayed in the end user's field of view.

[0008] Additionally, head movements can occur within a wide variety of ranges and speeds. Head movement speeds can vary not only between different head movements, but also within or across a range of a single head movement. For example, head movement speeds can initially increase (e.g., linearly or non-linearly) from a starting point and decrease as they reach an end point, achieving a maximum speed somewhere between the start and end points of the head movement. Rapid head movements can even exceed the ability of a particular display or projection technology to render images that appear to the end user as uniform and/or smooth motion.

[0009] Head tracking accuracy and latency (i.e., the time elapsed between the time a user moves his or her head and the time the image is updated and displayed to the user) have been challenges for VR and AR systems. In particular, for display systems that fill a significant portion of the user's visual field with virtual elements, it is critical that the head tracking accuracy be high and the overall system latency be very low from the initial detection of head motion to the update of the light delivered to the user's visual system by the display. If the latency is high, the system can create a mismatch between the user's vestibular system and the visual sensory system, creating a user perception scenario that can lead to motion sickness or simulator sickness. If the system latency is high, the apparent position of virtual objects will appear unstable during rapid head motions.

[00010] In addition to head-worn display systems, other display systems may benefit from accurate, low-latency head pose detection. These include head-tracked display systems, where the display is not worn on the user's body, but rather mounted on, for example, a wall or other surface. The head-tracked display acts as a window into the scene, and as the user moves his or her head relative to the "window," the scene is re-rendered to match the user's changing viewpoint. Other systems include head-worn projection systems, where the head-worn display projects light onto the real world.

[00011] Additionally, to provide a realistic augmented reality experience, AR systems may be designed to interact with the user. For example, multiple users may play a ball game with a virtual ball and/or other virtual objects. One user may "catch" the virtual ball and throw the ball back to another user. In another embodiment, a totem (e.g., a real bat communicatively coupled to the AR system) may be provided to the first user to hit the virtual ball. In other embodiments, a virtual user interface may be presented to the AR user to allow the AR user to select one of many options. The user may use totems, haptic devices, wearable components, or simply touch the virtual screen to interact with the system.

[00012] Detecting the user's head pose and orientation and detecting the physical locations of real objects in space can enable AR systems to display virtual content in an effective and enjoyable manner. However, these capabilities are key to AR systems, but are difficult to achieve. In other words, the AR system must recognize the physical locations of real objects (e.g., the user's head, a totem, a haptic device, a wearable component, the user's hand, etc.) and correlate the physical coordinates of the real objects to virtual coordinates corresponding to one or more virtual objects being displayed to the user. This requires highly accurate sensors and sensor recognition systems that can track the positions and orientations of one or more objects at high speed. Current approaches do not perform localization to satisfactory speed or accuracy standards.

[00013] Therefore, better localization systems are needed in the context of AR and VR devices.

[00014] The drawings illustrate the design and utility of various embodiments of the present invention. It should be noted that the drawings are not drawn to scale and that elements of similar structures or functions are represented by like reference numerals throughout the drawings. In order to better appreciate how the various embodiments of the present invention achieve the above-recited and other advantages and objects, a more detailed description of the inventions briefly described above will now be provided by reference to specific embodiments of the invention, which specific embodiments are illustrated in the accompanying drawings. Understanding that these drawings illustrate only typical embodiments of the invention and are therefore not to be considered limiting of the scope of the invention, the present invention will be described and illustrated with additional features and details by means of the accompanying drawings.
[00015] FIG. 1 illustrates a plan view of an AR scene displayed to a user of an AR system according to one embodiment.
[00016] Figures 2a-2d illustrate various embodiments of wearable AR devices.
[00017] FIG. 3 illustrates an exemplary embodiment of a wearable AR device interacting with one or more cloud servers of the system, according to embodiments of the present invention.
[00018] FIG. 4 illustrates an exemplary embodiment of a wearable AR device interacting with one or more handheld controller devices according to embodiments of the present invention.
[00019] FIG. 5 illustrates an example of a user operating a localization protocol within a physical environment according to embodiments of the present invention.
[00020] FIGS. 6A-6C illustrate examples of a user operating a localization protocol within a physical environment using one or more support modules according to embodiments of the present invention.
[00021] FIGS. 7a-7c illustrate examples of support modules according to embodiments of the present invention.
[00022] Figures 8a-8b illustrate examples of configurations utilizing embodiments of the present invention.
[00023] FIGS. 9A-9I illustrate examples of coupling means for support modules and wearable AR systems according to embodiments of the present invention.
[00024] Figures 10 and 11 illustrate a configuration for utilizing an ecosystem of localization tools according to embodiments of the present invention.
[00025] Figures 12a-12c illustrate examples of environment specific support modules according to embodiments of the present invention.
[00026] FIG. 13 illustrates an exemplary configuration for leveraging an ecosystem of localization tools including environment-specific support modules according to embodiments of the present invention.

[00027] Referring to FIGS. 2a-2d , some general componentry options are illustrated. In the detailed descriptions that follow the discussion of FIGS. 2a-2d , various systems, subsystems, and components are presented to address the goals of providing a high-quality, comfortably-perceived display system for human VR and/or AR.

[00028] As illustrated in FIG. 2a, a user (60) of an AR system is depicted wearing a head-mounted component (58) featuring a frame (64) structure coupled to a display system (62) positioned in front of the user's eyes. A speaker (66) is coupled to the frame (64) in the depicted configuration and positioned adjacent to the user's ear canal (in one embodiment, another speaker, not shown, is positioned adjacent to the other ear canal of the user to provide stereo/shapeable sound control). The display (62) is operatively coupled (68), for example, by a wired lead or wireless connectivity, to a local processing and data module (70), which may be mounted in a variety of configurations, such as fixedly attached to the frame (64), fixedly attached to a helmet or hat (80) as shown in the embodiment of FIG. 2b, embedded in headphones, removably attached to the torso (82) of the user (60) in a backpack-style configuration as shown in the embodiment of FIG. 2c, or removably attached to the hips (84) of the user (60) in a belt-coupled style configuration as shown in the embodiment of FIG. 2d.

[00029] The local processing and data module (70) may include a power-efficient processor or controller as well as digital memory, such as flash memory, both of which may be utilized to facilitate processing, caching, and storing data, which data is a) captured from sensors operatively coupled to the frame (64), such as image capture devices (e.g., cameras), infrared (IR) emitters and receivers, microphones, inertial measurement units, accelerometers, compasses, GPS units, radio devices, and/or gyros; and/or b) acquired and/or processed using a remote processing module (72) and/or remote data storage (74) (possibly for passage to the display (62) following such processing or retrieval). The local processing and data module (70) may be operatively coupled (76, 78) to a remote processing module (72) and a remote data store (74), for example, via wired or wireless communication links, such that these remote modules (72, 74) are operatively coupled to one another and available as resources to the local processing and data module (70).

[00030] In one embodiment, the remote processing module (72) may include one or more relatively powerful processors or controllers configured to analyze and process data and/or image information. In one embodiment, the remote data storage (74) may include a relatively large-scale digital data storage facility, which may be available via the Internet or other networking configuration of a "cloud" resource configuration. In one embodiment, all data is stored and all computations are performed in the local processing and data module, allowing for completely autonomous use from any remote module.

[00031] Now referring to FIG. 3 , a schematic diagram illustrates coordination between cloud computing assets (46) and local processing assets, which may reside, for example, on a head-mounted component (58) coupled to the user's head (120) and a local processing and data module (70) coupled to the user's belt (308) (and thus, the component (70) may also be referred to as a "belt pack" (70)), as shown in FIG. 3 . In one embodiment, cloud (46) assets, such as one or more server systems (110), are operatively coupled (115), such as via wired or wireless networking (wireless is preferred for mobility, wired is preferred for certain high-bandwidth or high-data-volume transfers that may be required), to one or both of the local computing assets, such as processor and memory configurations, coupled to the user's head (120) and belt (308) as described above (40, 42). These computing assets local to the user may also be operatively coupled to one another via wired and/or wireless connection configurations (44), such as the wired coupling (68) discussed below with reference to FIG. 8 . In one embodiment, to maintain the low inertia and small size of the subsystem mounted on the user's head (120), the primary transmission between the user and the cloud (46) may be via a link between the belt-mounted subsystem (308) and the cloud, with the head-mounted subsystem (120) being data-tethered to the belt-based subsystem, primarily using a wireless connection, such as an "ultra-wideband" (UWB) connection, such as is currently utilized in personal computing peripheral connectivity applications.

[00032] Using efficient local and remote processing coordination, and a suitable display device for the user, such as the user interface or user display system (62) illustrated in FIG. 2a, or variations thereof, aspects of the world relative to the user's current real or virtual location can be transmitted or "transmitted" to the user and updated in an efficient manner. In other words, a map of the world can be continually updated from a storage location that may partially reside on the user's AR system and partially reside on cloud resources.

[00033] The map (also referred to as a "passable world model") may be a large database containing raster imagery, 3-D and 2-D points, parametric information about the real world, and other information. As more and more AR users continue to capture information about their real-world environment (e.g., via cameras, sensors, IMUs, etc.), the map becomes increasingly accurate and complete.

[00034] In a configuration such as described above, where there is a world model that can reside on and be distributed from cloud computing resources, such a world can be "delivered" to one or more users in a relatively low bandwidth form, which is preferable to trying to pass around real-time video data, etc. The augmented experience of a person standing near a statue (i.e., as shown in FIG. 1 ) can be informed by a cloud-based world model, a subset of which can be passed down to the person and their local display device to complete the view. A person sitting at a remote display device, which can be as simple as a personal computer sitting on a desk, can efficiently download the same section of information from the cloud and render it on their display. In effect, a person physically present in a park near the statue can take a friend who is far away for a walk in the park, and the friend joins them via virtual and augmented reality. The system needs to know where there are roads, where there are trees, where there are statues, but once that information is in the cloud, the friend who joins can download aspects of the scenario from the cloud and then start walking along as an augmented reality of the local person actually in the park.

[00035] Features or otherwise 3-D points can be detected and captured or otherwise extracted from the environment, and the poses (i.e., vector and/or original positional information relative to the world) of the cameras capturing such images or points can be determined, and accordingly, these points or images can be "tagged" or associated with this pose information. Then, the points captured by the second camera (either physically by the second camera, or by the same camera but at a second time instant) can be utilized to determine the pose of the second camera. In other words, the second camera can be oriented and/or localized based on comparisons with the tagged images from the first camera. This knowledge can then be utilized to extract textures, create maps, and create a virtual copy of the real world (e.g., there are two cameras registered around it, or there are two time instants that can be used to compare commonly registered data).

[00036] Thus, at a base level, in one embodiment, the human-worn system can be utilized to capture both 2-D images and 3-D points that generate points, and these points and images can be transmitted to cloud storage and processing resources. These can also be cached locally along with embedded pose information (i.e., caching the tagged images); thus, the cloud can prepare the tagged 2-D images (i.e., tagged with 3-D poses) along with the 3-D points (i.e., within the available cache). If the user is observing something dynamic, the user can also transmit additional information about the motion to the cloud (e.g., if viewing another person's face, the user can take a texture map of the face and push it up at an optimized frequency, even if the rest of the surrounding world is essentially static).

[00037] In order to capture points that can be used to generate a "transferable world model", it is helpful to know the user's position, pose, and orientation relative to the world accurately. More specifically, the user's position should be localized to a granular degree, since it may be important to know the user's head pose as well as his hand pose (e.g., if the user is grasping a handheld component, making a gesture, etc.). In one or more embodiments, GPS and other localization information may be utilized as inputs to such processing. Highly accurate localization of the user's head, totems, hand gestures, haptic devices, etc. is important for displaying appropriate virtual content to the user.

[00038] Referring to FIG. 4, an AR system is illustrated featuring an operably coupled hand-held controller (606). In one or more embodiments, the hand-held controller may be referred to as a “totem” utilized in various user scenarios, such as gaming. In other embodiments, the hand-held controller may be a haptic device. The hand-held controller (606) may include a battery or other power supply for powering various interconnected components, which may include cameras, IMUs, and other devices, as described below. As described herein, the local processing and data module (70) may be operably coupled to cloud resources (630) and may be operably coupled to a head-mounted wearable component (58).

[00039] In various embodiments, it may be desirable to aggregate the computational, perceptual, and perceptual capabilities of various components to improve understanding of various issues related to the user's movements within a local space, such as where components are located and/or oriented relative to each other within the space, what type of local environment the user is dealing with (e.g., room geometry, positions and/or orientations of objects within the room, etc.), and/or what objects and features can be mapped (e.g., using SLAM analysis).

[00040] Referring to FIG. 5, a floor plan of a room (200) is shown, typically having a user (60) standing facing a desk (206), a coffee table (204), and a door (202) configured to swing open into the room when utilized. As described above, in various embodiments, the wearable computing system includes operably coupled components, such as a head-mounted component (58), a local processing and data module (70), and a hand-held component (606). Additionally, as described above, each of such components may include various sensors and processors and associated capabilities of their own, such as cameras, IMUs, and processor components, which may be configured to provide data to pose determination algorithms utilized by one or more processors to determine, among other things, where such components are relative to each other and/or aspects of the environment around them in position and/or orientation (so-called "pose" determination, as described above).

[00041] Referring again to FIG. 5, in one embodiment, the relative proximity of the handheld component (606) and the local processing and data module (70) to the head mounted component (58) may be advantageous in terms of determining positioning and/or orientation of such components relative to one another, but may not be optimal for gathering information about the environment surrounding the person and such components. For example, in some positions and orientations, the user's own body may obscure or otherwise interfere with the handheld component (606) and prevent it from communicating effectively with the head mounted component (58). Other non-user environment factors may also affect performance, such as the door (202) of FIG. 5 - which, due to the movable nature of the door as a functional door, may change position and/or orientation relative to the components (58, 70, and/or 606) - is nearly completely across the room from the operator (60) in the depicted configuration. The distance and angular motion of such a structure may not be readily detectable by the handheld component (606) or the head mounted component (58).

[00042] Referring to FIG. 6A , in various embodiments, it may be desirable to place one or more instrumented components, such as additional handheld component modules (606) that may be interconnected to the processing and storage resources of other components (58, 70, and/or 606) coupled to the operator (60) wirelessly, such as via Bluetooth or other wireless connectivity modalities, in a position near the door (202) that is deemed of interest, such as having instrumentation oriented toward the door. In other words, the additional handheld component module (606) may be positioned and oriented relative to the room or other objects (e.g., by being placed on a desk, floor, or other object, by being held by a mount coupled to another object, by being removably and/or fixedly attached to another object, etc.) rather than being actually held in the user's hand; in such configurations, the additional handheld component module (606) may be referred to as a "support" or "capture and processing" module (606). In one embodiment, the capture and processing module (606) depicted in FIG. 6A may be positioned in a temporary position/orientation configuration on the floor or some other structure, as depicted in the embodiment of FIG. 6B , or may be fixedly attached to the structure, with an additional capture and processing module (606) coupled to a wall, also coupled in a selected position and orientation to assist in collecting data regarding a nearby door (202), and further, the additional capture and processing module (606) operably coupled to other components (58, 70, 606) of the target system, such as via Bluetooth or other wireless connectivity configuration, such as those described above.

[00043] In such an embodiment, each component (58, 70, 606) physically coupled to the operator (60) may be configured to benefit from additional data capture and processing capabilities residing within an additional capture and processing module (606), which is illustrated here as an additional version of the component utilized as a handheld component. FIG. 6c illustrates that a plurality of additional capture and processing modules (606) may be positioned and oriented around the operator's environment to assist in processing and analyzing the world around them. It will be appreciated that the plurality of capture and processing modules may be individually optimized for different measurements. For example, a capture and processing module (606) proximate a door may be configured to update localization data for the system utilized by the user (60) only when the door (202) is opened and closed. While a ray cast or cone cast from a distance from a sensor on a head mounted component (58) or hand held component (606) may not be able to tell the difference between a semi-open and a closed door, a sensor closer to the door, having a time of flight sensor oriented in the angular direction of the door opening and closing, can more quickly and accurately determine the door (202) (and by extension the room (200)) with updated detectable feature positions.

[00044] Similarly, the capture and processing module (606) at the corner of the room (200) can activate and provide localization data when the capture and processing module (606) detects that the user (60) is approaching and that the head pose is at least partially directed toward the corner capture and processing module (606). An advantage of such an embodiment is that as the user approaches a physical limit of the space, such as a wall or a corner, there are fewer features for the sensors on the head-mounted component (50) or the hand-held component (606) to detect and extract localization, and the capture and processing module (606) can provide a “lighthouse” effect for localization.

[00045] Figures 7a and 7b illustrate embodiments of portable additional capture and processing modules (606) that may include components described above, such as but not limited to, handles (210), desktop stands (212), and embedded and/or mobile processors (128) suitably coupled to cameras (124), IR emitters and/or receivers, IMUs (102), WiFi communication devices (114), Bluetooth communication devices (115), and other sensors. FIG. 7c illustrates a close-up view of one embodiment of a camera module (124) including a plurality of (e.g., four) monochrome camera devices (125), a color camera (127), and a depth camera (129), as depicted, wherein the depth camera (129) is configured to utilize time-of-flight data to assist in positioning image elements within space relative to a capture interface of such depth camera (129).

[00046] Referring to FIG. 9a, a configuration such as that characterized in FIGS. 5 and 7a-7c may be utilized to determine relative poses of various components. A user may be wearing a head-mounted wearable component (58) operably coupled to a hand-held controller (606), which is further coupled to the operator, such as via the operator's hand (214). The head-mounted wearable component (58) may be configured to utilize sensors coupled thereto to determine certain aspects of the pose of the head-mounted wearable component relative to an environment around the head-mounted wearable component (58), such as a room and elements of the room, such as a position and/or orientation, and/or certain features of such environment, such as the shape of the room or objects present therein (216).

[00047] The handheld controller (606) may be configured to utilize sensors coupled thereto to determine certain aspects of the pose of the handheld controller (606) relative to the environment surrounding the handheld controller (606), such as position and/or orientation, and/or certain features of such environment, such as the shape of the room (218).

[00048] The handheld controller and the head-mounted wearable component may be configured to share determined pose information via their operable couplings (e.g., via a Bluetooth connection as described above) so that relative positioning and/or orientation between the handheld controller and the head-mounted wearable component may be determined (220). Such motions may be frequently updated (222) to maintain timely refresh information regarding relative positioning and/or orientation between the handheld controller and the head-mounted wearable component. The localization of a component may be provided as a function of a coordinate frame of the component. In other words, the handheld component may operate its sensors to determine its position and orientation in a "handheld component framework" or "coordinate frame" and may then be coupled with the head-mounted component to determine the relative position of the head-mounted component, thereby providing the head-mounted component with its position and orientation in the same framework. The converse is also true, in that the head-mounted component can determine its own position and orientation, and then determine the position and orientation of the controller for the head-mounted component, which in turn can provide localization for the hand-held component in the head-mounted component framework or head-mounted coordinate frame. Matrix transformations between the two, such as Euclidean distance matrices, quaternion orientations, or 3D rotation matrices, can be used to facilitate such coordinate frame transformations between the components.

[00049] In some embodiments, any one component may localize itself relative to a plurality of other components. For example, the handheld controller (606) may detect its position relative to the head-mounted component (58) and the capture and processing module (606). The means of relative sensing may also be subjective. The hand component (606) may utilize IR array tracking to detect IR signals on the head-mounted component (58), and may use visible light tracking (e.g., feature detection and extraction or fiducial detection) to detect the capture and processing module (606). In that regard, translational or rotational measurements between components of one pairing may replace or supplement image capture feature detections between components of other pairings to provide a user with an optimized set of localization data at any given time.

[00050] Referring to the previous example, if the user were to approach a blank wall without many detectable features, using image-based tracking would result in reduced accuracy or increased latency, as there would be fewer detected features for the system to utilize in its calculations. However, if the control and processing module (606) provided additional detectable data, such as IR signals, time-of-flight pulses, or electromagnetic transmissions, the user's rotation and translation relative to the control and processing module could at least supplement any localization logic to compensate for the lack of detectable features by the image-based tracking sensors.

[00051] Referring to FIG. 8b, configurations such as those described above with reference to FIGS. 6a-6c and 7a-7c may be utilized to determine relative or absolute poses of various components. As illustrated in FIG. 8b, a user may wear a head-mounted wearable component (58) operably coupled to a plurality of hand-held controllers (606) and/or fixedly mounted (8) controllers, one of which may be physically coupled to the operator's body (224), such as via a hand-held coupling. A head-mounted wearable component (58) may be configured to utilize sensors coupled thereto to determine certain aspects of a pose of the head-mounted wearable component (58) relative to an environment (e.g., a room) surrounding the head-mounted wearable component (58) (e.g., position and orientation), and/or certain features of such environment (e.g., the shape of the room) (226). A plurality of handheld controllers (606) — the plurality of handheld controllers (606) may be configured to rest on or be fixedly mounted to other objects, one of which may be physically coupled to an operator as in FIGS. 6A-6C and FIGS. 7A-7C — may be configured to utilize sensors coupled to them to determine particular aspects of the pose of such controllers (such as position and orientation) relative to an environment around such controllers (such as a room), and/or particular features of such environment (such as the shape of the room) (228). A plurality of handheld controllers (606)—the plurality of handheld controllers (606) may be fixedly mounted to other objects, placed on other objects, or coupled to an operator—and a head-mounted wearable component (58) may be configured to share determined pose information via their operable couplings (e.g., via a Bluetooth connection) such that relative positioning and/or orientation between one or more of the plurality of handheld controllers (606) and the head-mounted wearable component (58) may be determined (230).

[00052] Matrix transformations between each of the components can be optimized to provide absolute positions rather than simply relative positions between the components. For example, if the positions and orientations of the fixedly mounted components are known relative to the physical environment, the handheld component (606) or the head mounted component (58) can determine their individual poses relative to those fixedly mounted components, and by extension, determine "real world coordinate frame" or "environment coordinate frame" positions and orientations, which can be referred to as absolute poses.

[00053] As discussed above, it will be appreciated that other intermediate coordinate frames may exist or be useful. For example, a handheld controller that receives only position data (e.g., from an inertial measurement unit, accelerometer, or camera coupled to the handheld component) may determine its pose in a "handheld coordinate frame", and similarly its sensors detect and extract a "head-mounted coordinate frame" and pose information. In some embodiments, a localization system and protocol for determining position using the disclosed sensors and sensing components may need to operate only in an intermediate coordinate frame, and may not require absolute or global environmental coordinate frame information.

[00054] Such actions may be frequently updated to maintain timely information regarding the relative positioning and/or orientation between the multiple handheld controllers and/or fixedly mounted controllers and the head-mounted wearable component (232).

[00055] Referring to FIG. 9a, small fiducials (246), which may represent transmitting elements, such as infrared transmitters, or reflectors, such as a plurality of small mirrors, glass spheres, or other reflective elements, may be utilized to facilitate rapid determination of position and/or orientation between modules of the system having known or predetermined dimensions between one or more of the fiducials (246). For example, referring to FIG. 9b, in one embodiment of a head-mounted component (58) coupled with a plurality of fiducials (246), the relatively long known lengths between the fiducials (246) may be utilized to determine and track the relative position and/or orientation of such head-mounted component (58) in space relative to another component configured to track the fiducials using a camera or other device, or relative to a global coordinate system. FIG. 9c illustrates another head mounted component (58) configuration having additional reference points (246) to provide additional information regarding positioning and/or orientation of such a configuration.

[00056] Figure 9d illustrates a local processing and data module (70) configuration in which multiple reference points (246) are also coupled to aid in relative pose determination. Figure 9e illustrates another embodiment of a local processing and data module (70) configuration in which more reference points (246) are coupled to aid in relative pose determination.

[00057] Figure 9f illustrates a configuration of a handheld controller module (606) that is also coupled with a plurality of reference points (246) to aid in relative pose determination. Figure 9g illustrates another embodiment of a configuration of a handheld controller (606) that is coupled with a greater number of reference points (246) to aid in relative pose determination.

[00058] Figure 9h illustrates a configuration of a handheld controller module (606) coupled with a multi-fiducial array (248) to aid in relative pose determination; such fiducial arrays having known positions of sub-parts can be useful for pose determination when geometric structures for them can be provided, such as additional handheld controller modules (606) that can be placed on a coffee table or the like without as many geometric constraints as wearable components of a mobile computing system. Figure 9i illustrates a configuration where, in addition to a single handheld controller (606) already coupled to an operator (60), a plurality of additional controllers, such as handheld controller modules (606), can also be outfitted with fiducials (246) to aid in rapid pose determination of relative components relative to one another.

[00059] Referring to FIG. 10, in an embodiment such as that described with reference to FIG. 9a, a user (60) may be wearing a head-mounted wearable component (58) operably coupled to a handheld controller (606) (214). The head-mounted wearable component (58) may be configured to utilize sensors coupled thereto to determine certain aspects of a pose of the head-mounted wearable component (such as position and orientation) relative to an environment (such as a room) surrounding the head-mounted wearable component, and/or certain features of such environment (such as the shape of the room) (216). The handheld controller (606), such as a handheld controller (606) physically coupled to an operator's hand, and the head-mounted wearable component can be configured to utilize reference points, such as via transmitter/receiver pairings (such as infrared spectral beacons and sensors in single-lap configurations) or transceiver/reflector pairings (such as infrared spectral transmitter/receiver devices and reflectors in two-lap configurations) coupled thereto, to determine certain aspects (such as position and orientation) of the pose of the handheld controller relative to the pose of the head-mounted wearable component (234). The handheld controller (606) and the head-mounted wearable component (58) can be configured to share determined pose information via their operable coupling (e.g., via a Bluetooth connection) such that relative positioning and/or orientation between the handheld controller and the head-mounted wearable component can be determined (236). Such actions can be frequently updated to maintain refresh information regarding the relative positioning and/or orientation between the handheld controller and the head-mounted wearable component (238).

[00060] Referring to FIG. 11, for an embodiment such as that illustrated in FIG. 9i, a user (60) may be wearing a head-mounted wearable component (58) operably coupled to a plurality of hand-held style controllers (606), which may be physically coupled to the user or other objects in the surrounding room, such as walls (8) (224). The head-mounted wearable component (58) may be configured to utilize sensors coupled thereto to determine certain aspects of the pose of the head-mounted wearable component (58) relative to an environment (e.g., a room) surrounding the head-mounted wearable component (58), such as position and orientation, and/or certain features of such environment (e.g., a shape of the room) (226). The plurality of hand-held controllers (606) and the head-mounted wearable component (58) can be configured to utilize reference point configurations, such as transmitter/receiver pairings (such as infrared spectrum beacons and sensors in single-lap configurations) or transceiver/reflector pairings (such as infrared spectrum transmitter/receiver devices and reflectors in two-lap configurations) coupled thereto, to determine particular aspects (such as position and orientation) of the pose of the hand-held controllers (606) relative to the pose of the head-mounted wearable component (58) (240). The plurality of handheld controllers (606) and the head-mounted wearable component (58) may be configured to share determined pose information via their operable coupling (e.g., via a Bluetooth connection) such that relative positioning and/or orientation between the plurality of handheld controllers (606) and the head-mounted wearable component (58) may be determined (242). Such operations may be frequently updated (244) to maintain refresh information regarding the relative positioning and/or orientation between the plurality of handheld controllers and/or fixedly mounted controllers and the head-mounted wearable component.

[00061] Referring to FIG. 12a, another embodiment may be configured such that the periodic motion (248) of the handheld controller (606) may be utilized to further utilize the instrumentation included thereon for additional pose determination data from different positions/orientations of the handheld controller (606). In other words, if the operator (60) is particularly interested in the motion of the area surrounding the door, the operator (60) may waive the handheld controller (60) around to assist the controller in utilizing the sensors included therein to capture additional information about that local environment from different viewpoints, vectors, and/or orientations.

[00062] Such a hand-held periodic motion would require the operator to be closer to the targeted structures, which may be undesirable or impractical; the embodiment of FIG. 12b features the user (60) being relatively far away from the target door (202), but with two hand-held controllers (606) mounted in the area of interest - one mounted on a nearby wall and the other on the door (202) itself. FIG. 12c illustrates an embodiment featuring both the periodic motion (248) and additional sensing devices (606) as in FIG. 12b.

[00063] Figure 13 illustrates a configuration similar to that of Figure 11, with the addition that the user (60) moves one or more hand-held controllers around the object or volume of interest, such as a door or other object, to facilitate additional capture of information about the object or volume of interest, and positioning and/or orientation of the object or volume of interest relative to the hand-held controller (606) and other devices, such as one or more fixedly mounted controllers (606) and the head-mounted wearable component (58), such as through a periodic motion or movement. Such actions may be frequently updated (270) to maintain refresh information about the relative positioning and/or orientation between the plurality of hand-held controllers and/or fixedly mounted controllers and the head-mounted wearable component.

[00064] Various exemplary embodiments of the present invention are described herein. Reference is made to these examples in a non-limiting sense. The examples are provided to illustrate aspects of the invention that are more broadly applicable.

[00065] The present invention includes methods that can be performed using the devices of the present invention. The methods can include the act of providing such a suitable device. Such provisioning can be performed by an end user. In other words, the act of "providing" merely requires that the end user obtain, access, approach, position, set-up, activate, power-up, or otherwise perform the necessary device of the method of the present invention. The methods mentioned herein can be performed in any logically possible order of the mentioned events, as well as in the mentioned order of events.

[00066] Exemplary aspects of the present invention have been described above, together with details regarding material selection and manufacturing. As for other details of the present invention, these may be recognized in connection with the above-referenced patents and publications, as well as generally known or may be recognized by those skilled in the art. The same may also be true for method-based aspects of the present invention, with respect to additional operations, as is usually or logically employed.

[00067] It is also contemplated that any optional feature of the described variations of the invention may be described and claimed independently or in combination with any one or more of the features described herein. Reference to a singular item includes the possibility that there may be plural of the same items. More specifically, as used herein and in the claims associated therewith, the singular includes plural referents unless specifically stated otherwise. In other words, use of the singular allows for "at least one" of the subject item in the claims as well as in the description above associated with the present disclosure. It is further noted that such claims may be drafted to exclude any optional element. Accordingly, this statement is intended to serve as antecedent basis for use of such exclusive terminology as "solely," "only," etc., or use of a "negative" limitation in connection with the recitation of claim elements.

[00068] Without using such exclusive terminology, the term "comprising" in claims associated with this disclosure should allow for the inclusion of any additional elements, regardless of whether a given number of elements are recited in such claims or whether the addition of a feature can be considered to change the nature of an element recited in such claims. Except as specifically defined herein, all technical and scientific terms used herein are to be given as broad a commonly understood meaning as possible while maintaining claim validity.

Claims (20)

A method for localizing a user who operates multiple sensing components,
A step of transmitting aspects of relative position from one or more control and processing modules to a head-mounted component including one or more fiducials;
A step of receiving environmental characteristics representing location information from the head-mounted component;
A step of determining a head-mounted component pose in the environment coordinate frame of the head-mounted component based on the received environmental characteristics;
In the head-mounted component, a step of receiving aspects of a relative position of at least one handheld component with respect to the head-mounted component;
A step of determining a relative handheld component pose of said at least one handheld component in a head-mounted component coordinate frame utilizing said one or more reference points;
a step of sharing the head-mounted component pose in the above environmental coordinate frame with at least one handheld component, and
A step of sharing the relative handheld component pose in the head-mounted component coordinate frame with the head-mounted component.
Including,
method. In the first paragraph,
A method further comprising the step of updating the relative handheld component pose in the head-mounted component coordinate frame based on the head-mounted component pose in the environment coordinate frame. In the second paragraph,
A method further comprising the step of updating the head-mounted component pose in the environment coordinate frame based on the relative handheld component pose. In the first paragraph,
A method further comprising the step of transforming the relative handheld component pose in the head-mounted component coordinate frame into the handheld component pose in the environment coordinate frame. In the first paragraph,
A method further comprising the step of transmitting aspects of said relative position from said one or more control and processing modules to said at least one handheld component in said environmental coordinate frame. In paragraph 5,
A method further comprising the step of determining a handheld component pose in the above environmental coordinate frame. In Article 6,
A method further comprising the step of updating the relative handheld component pose shared with the head-mounted component using the handheld component pose in the environment coordinate frame. In the first paragraph,
A method wherein the step of receiving the above environmental characteristics includes image-based feature extraction and location determination. In the first paragraph,
A method comprising utilizing one or more of said reference points to transmit electromagnetic field signals. In Article 9,
The above electromagnetic field signal is in the visible light spectrum, method. In Article 10,
The above electromagnetic field signal is in the infrared spectrum, method. A method for positioning a user who operates multiple detection components, comprising:
A step of transmitting aspects of relative position from one or more control and processing modules to a head-mounted component including one or more reference points;
A step of receiving environmental characteristics representing location information from the head-mounted component;
A step of determining a pose of the head-mounted component in the environment coordinate frame of the head-mounted component based on the received environmental characteristics;
In the head-mounted component, a step of receiving aspects of the relative position of at least one fixed mounting controller with respect to the head-mounted component;
A step of determining a relative fixed mount controller pose of said at least one fixed mount controller in said head-mounted component coordinate frame utilizing said one or more reference points;
a step of sharing the head-mounted component pose in the above environmental coordinate frame with at least one fixed-mounted controller, and
A step of sharing the relative fixed mounted controller pose in the head mounted component coordinate frame with the head mounted component.
Including,
method. In Article 12,
A method further comprising the step of updating the relative fixed mounted controller pose in the head mounted component coordinate frame based on the head mounted component pose in the environment coordinate frame. In Article 13,
A method further comprising the step of updating the head-mounted component pose in the environment coordinate frame based on the relative fixed-mounted controller pose. In Article 12,
A method further comprising the step of transforming said relative fixed mounted controller pose in said head mounted component coordinate frame into said relative fixed mounted controller pose in said environment coordinate frame. In Article 12,
A method further comprising the step of transmitting aspects of said relative position from one or more control and processing modules to said relative fixed-mount controller pose in said environmental coordinate frame. In Article 16,
A method further comprising the step of determining the relative fixed mounting controller pose in the environmental coordinate frame. In Article 17,
A method further comprising the step of updating the relative fixed mounted controller pose shared with the head mounted component with the relative fixed mounted controller pose in the environmental coordinate frame. In Article 12,
A method wherein receiving said environmental characteristics comprises image-based feature extraction and location determination. In Article 12,
A method comprising utilizing one or more of said reference points to transmit electromagnetic field signals.