US20030040361A1

US20030040361A1 - Method and apparatus for generating tactile feedback via relatively low-burden and/or zero burden telemetry

Info

Publication number: US20030040361A1
Application number: US10/201,584
Authority: US
Inventors: Craig Thorner
Original assignee: Individual
Current assignee: Individual
Priority date: 1994-09-21
Filing date: 2002-07-23
Publication date: 2003-02-27
Also published as: US20130165226A1; US8328638B2; US20080280678A1; US20090069081A1; US8308558B2; US8834266B2

Abstract

A method and apparatus for providing tactile feedback for computer and video game systems is disclosed which provides real time tactile feedback to enhance a user's experience while interacting with a computer/video game or simulation. The method relies on application-generated telemetry that is generated by an executing application for the benefit of a telemetry processing method, and/or telemetry that is self-generated by said telemetry processor when no telemetry has been provided by said executing application. The application-generated telemetry is used to produce tactile feedback that is derived directly from the internal calculations of said application, where said telemetry is processed by a telemetry processor that calculates said derivative tactile feedback in real time by applying scaling values and algorithms to the raw telemetry. In the event that application-generated telemetry is not available or relatively limited, self-generated telemetry is utilized to supplement the provided telemetry. This telemetry can be used to scale in real time the results of an audio analysis tactile feedback generation method. The system's tactile sensation generators include independent groups of one or more actuators, e.g., electric motors or solenoids, that are embedded within or attached to various devices that can be connected to the system.

Description

RELATED APPLICATION DATA

This application is a continuation-in-part of U.S. patent application Ser. No. 08/935,762 (assigned U.S. Pat. No. 6,422,941 set to issue on Jul. 23, 2002) which itself is a continuation-in-part of U.S. Pat. No. 5,669,818 (filed Mar. 23, 1995, issued Sep. 23, 1997) and U.S. Pat. No. 5,684,722 (filed Sep. 21, 1994, issued Nov. 4, 1997). All teachings of the parent applications and/or patents are incorporated herein by reference.[0001]

FIELD OF THE INVENTION

This invention relates to tactile feedback systems for computer and video game, simulation, and training systems, and, more particularly, to a method for using telemetry to generate and/or shape the tactile feedback generated by said systems, such that real-time tactile feedback is provided to enhance a user's or spectator's experience while interacting with said systems, and at a zero burden or a heretofore impossibly low burden to software developers, yet producing very effective results.

BACKGROUND OF THE DISCLOSURE

The implementation of tactile feedback in computer gaming, simulation, and training systems, where “tactile feedback” refers to physical sensations that one feels while interacting with said systems, has heretofore been plagued by the fundamental limitation of poor developer support. Typically, software developers are pressed for both time and money, and they are under constant pressure to release their software into the marketplace as soon as is practicable. In this competitive field, it is difficult for any given developer to spend the time necessary to create thoughtful and artistic tactile feedback. With regard to tactile feedback, proper implementation is more of an art than a science. No right or wrong answer necessarily exists for how a particular tactile event should feel, such as driving a car on gravel or firing a fictional laser cannon in a fictional spacecraft. Therefore, much time can be spent (and from the developer's point of view, even wasted) in tweaking the subjective feel of the desired effects, ad infinitum. In the end, there is a vast difference between the quality of tactile feedback that was merely implemented into a given software application, and tactile feedback that was thoughtfully and artistically crafted to render effective results (given of course the limitations of the particular tactile feedback hardware in use). In cases where the code that renders tactile feedback is incorporated by any given developer into that developer's wares, it is difficult for that code to be upgraded at a later time once a given product is released, because developers are usually fighting the time constraints of the post-release cycle where updates to some given software are periodically released to fix bugs or add features that were left out of the original release (or are functioning with substantial deficits) due to time constraints in effect during the prerelease period. Ultimately, servicing the tactile feedback support is less than a priority to the developer, and the effectiveness of said tactile feedback suffers as a result. Furthermore, given the subjective nature of tactile feedback, its effectiveness and quality varies greatly in each title, application, and/or implementation.

With regard to the patent applications and patents for which this application is a continuation-in-part, much of the difficulty presented by lack of developer support has been overcome by AudioSense®, which is a sound analysis technology that generates tactile feedback in real time without developer support of any kind. However, AudioSense® has inherent limitations that are overcome and/or minimized by intelliVIBE®, which is the type of tactile feedback that has typically required developer support—and therefore, intelliVIBE® suffers from the very limitations described previously herein.

Until the innovation that is the subject matter of this patent application, software developers were relied upon to decide what events in their software required what sensations, and what the triggering mechanisms were for initiating those sensations. For example, Immersion Corporation of 801 Fox Lane, San Jose, Calif. 95131 USA (NASDAQ: IMMR) markets and licenses technologies collectively falling under the trademark name TouchSense™. In using TouchSense™, developers can utilize a toolkit known as “Immersion Studio” (as of this writing, currently in version 4.1.0). When using Immersion Studio, developers can more easily create tactile effects, but are relied on to determine the subjective artistic tactile feel of those effects, and when such effects are to be rendered. This is precisely the burden that this patent application is intended to relieve.

Therefore, a need exists in the art for a relatively quick and simple method by which any given software developer is required only to activate a simple shared data structure, which will provided real-time telemetry to an external executable, application, function, or code segment (known to the present invention as an “intelliVIBE® module”), which in turn will itself generate the necessary tactile feedback control signals via general purpose algorithms that are shaped by said telemetry, thereby relieving the developer of the most substantial burdens of supporting an interface such as intelliVIBE®, and/or Immersion Corporation's TouchSense™, and/or any other proprietary methodology for generating tactile feedback. For reference, throughout this application, the terms “tactile feedback” should be understood to include and be synonymous with the terms “force feedback”. In much the same way that AudioSense® audio analysis technology allows tactile feedback hardware to function with no support whatsoever from any given developer of any given software title, application, or hardware system, this telemetry based method allows intelliVIBE® (and other similar interfaces) to self-generate effective and thoughtful tactile feedback with minimal, simple, and one-time-only support by any given developer. Furthermore, due to the telemetry basis for the tactile feedback, the teachings herein can be applied to other areas, such as remotely piloted vehicles, remotely controlled vehicles, UAVs (Unmated Aerial Vehicles), spacecraft, and other types of tele-operated or tele-presense vehicles, where telemetry is available or can be provided, thereby producing effective tactile feedback for the operators or spectators of such vehicles.

SUMMARY OF THE INVENTION

The present invention provides a method and corresponding apparatus for providing tactile feedback for computer and video game, training, and simulation systems. More specifically, the tactile feedback system comprises a software application, a tactile feedback controller, and a plurality of tactile sensation generators. The innovative method herein provides that any given software application can provide simple telemetry about its current state to a shared memory data structure, and subsequently leave all the burden of producing tactile feedback to an externally operating thread that employs general purpose algorithms and graphically adjustable scaling factors and configuration options to yield effective and appropriate tactile feedback without further effort by the given software application. It is a simple one-time only operation for any given software developer to implement the telemetry support. Thusly, the tactile feedback so produced is achieved with very low burden to the software developer. Additionally, should no telemetry be provided by a given software application, the innovative method herein has the means to self-generate its own telemetry by examining the state of the host computer's control input devices (i.e., keyboard, mouse, joystick, etc.), and to use said self-generated telemetry as the basis for applying real-time directional scaling to tactile feedback simultaneously produced via real-time audio analysis. The present invention can produce tactile feedback via the combination of real-time audio analysis with simultaneous directional scaling produced by real-time self-generated telemetry in combination with pre-defined general-purpose algorithms. This capability is achieved at a zero burden, that is, with no effort whatsoever nor any acknowledgement of its existence from the perspective of the developers of any given software application, yet yields results that resemble very carefully and thoughtfully crafted tactile feedback.

More specifically, the tactile feedback method and apparatus is provided in a preferred embodiment comprised by (1) a software application (currently known as the Tactile Feedback System Control Panel or TFS Control Panel), which configures the Tactile Feedback System (TFS) and handles the intelliVIBE® modules, each of which is adapted to handle the telemetry provided by any given software title, application, or system, and provides a graphical user interface for adjusting the available tactile feedback parameters that descend from the provided telemetry, (2) a tactile feedback controller (the presently preferred embodiment is currently known as the TFS2 (Tactile Feedback System version 2.0) which incorporates cooperative AudioSense® and intelliVIBE® capabilities, and (3) a plurality of tactile sensation generators, the currently preferred embodiment of which is a tactile feedback seat known as the TFSU (Tactile Feedback Seating Unit) and a tactile feedback vest (U.S. Pat. No. 5,565,840) known by the trademark name FLAKjaket™.

BRIEF DESCRIPTION OF THE DRAWINGS

The teachings of the present invention can be readily understood by considering the following detailed description in conjunction with the accompanying drawings, in which: [0009]
FIG. 1 depicts a block diagram of an illustrative configuration of the present invention; [0010]
FIG. 2 is a flow chart for using illustrative application generated telemetry to generate tactile feedback for a driving simulation; [0011]
FIG. 3 is a flow chart for using illustrative application generated telemetry to generate tactile feedback for a flight simulation; [0012]
FIG. 4 is a flow chart for using illustrative self-generated telemetry to generate scaling factors for directionally scaling tactile feedback simultaneously produced via audio analysis; [0013]
FIG. 5 shows an illustrative group of graphical controls for adjusting various telemetry scaling factors; [0014]
FIG. 5A depicts the graphical controls of FIG. 5, specifically illustrative RPM telemetry scaling factors; [0015]
FIG. 5B depicts the graphical controls of FIG. 5, specifically illustrative SPEED telemetry scaling factors; [0016]
FIG. 5C depicts the graphical controls of FIG. 5, specifically illustrative TAKE OFF telemetry scaling factors; [0017]
FIG. 5D depicts the graphical controls of FIG. 5, specifically illustrative ROLL telemetry scaling factors; [0018]
FIG. 5E depicts the graphical controls of FIG. 5, specifically illustrative G-FORCE telemetry scaling factors; [0019]
FIG. 5F depicts the graphical controls of FIG. 5, specifically illustrative WEAPONS telemetry scaling factors; [0020]
FIG. 5G depicts the graphical controls of FIG. 5, specifically illustrative HOTKEYS telemetry scaling factors; [0021]
FIG. 5H depicts the graphical controls of FIG. 5, specifically illustrative AFTERBURNER telemetry scaling factors; [0022]
FIG. 6 depicts an aircraft and the 4 quadrants used as the basis for an innovative aircraft roll algorithm; [0023]
FIG. 7 depicts the actual mathematical algorithm used to generate an aircraft's roll scaling factor based upon the aircraft's current quadrant as in FIG. 6, and how to utilize said scaling factor once derived from the roll telemetry data;[0024]
To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures. [0025]

DETAILED DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts a block diagram of a universal [0026] tactile feedback system 100 interacting with a host computer 102. The host computer 102 is connected to the universal tactile feedback controller 200 by an audio connection 112 and/or digital connection 116. The universal tactile feedback controller 200 is connected to multiple independent tactile sensation generators 500 by connection 460. The tactile sensation generators 500 collectively refer to hardware that produces physical sensations that one can feel while interacting with the software application 104. In this illustrative embodiment, no specific limitation is intended for any or all of the connections 112, 116, 460 with regard to their particular form, whether analog, digital, wired, wireless, and so on. The host computer 102 is presumed to be the host for some desired software application 104. The software application 104 is the application for which the present invention will generate tactile feedback. If the developers of the given software application 104 have supported the simple telemetry requirement, there will exist application generated telemetry 600. This application-generated telemetry 600 is a simple data structure that contains information about the state of operation of said software application 104. For example, if the software application 104 is a car racing simulation or car racing game, typical telemetry would include the car's engine RPM, the car's MPH, the car's throttle position, lateral G-forces, acceleration, deceleration, brake pedal position, steering wheel deflection, current gear, surface type (i.e., is the car currently on asphalt, dirt, gravel, sand, wet asphalt, icy asphalt, snow, mud, etc.), and damage flags (i.e., are there any flat tires, deformations of the vehicle's body panels, or equipment failures, etc.). This illustrative telemetry is by no means comprehensive, but should be sufficient to illustrate the type of data common to car racing games and/or simulations. It is easy for developers to provide this type of telemetry because they already calculate these variables (and most often many more variables than these) to make their software function properly. Each different game or simulation will have a unique set of telemetry data. The telemetry is entered into a shared data structure by a common memory mapping technique readily understood by those skilled in the art. In the illustrative embodiment, the computer host is running on a Microsoft Windows platform wherein a predetermined shared memory area name is agreed and known to both the software application 104 and the telemetry processor 700. The software application 104 creates the telemetry area by opening a shared memory area with the predetermined shared memory are name. The telemetry processor detects the shared memory area by said predetermined name, and uses the standard Windows coding “MapViewOfFile” to map the application-generated telemetry 600 into the memory of the telemetry processor 700. This simple operation affords the telemetry processor 700 access to the real-time state data (telemetry) of the software application 104. In short, it allows the telemetry processor 700 to know what is going on inside the software application 104, to the extent and detail inherent in the data structure provided by the software application 104 for the benefit of the application generated telemetry 600 shared memory structure, and ultimately for the benefit of the owners of the universal tactile feedback system 100 and the users of the software application 104. Providing the application generated telemetry 600 is a very simple operation for any reasonably skilled software developer, and even if new to the entire concept, most often it is a process of only several tens of minutes to effectuate the shared memory area once the data structure that is to be mapped there is known.
Continuing with FIG. 1, if a given software developer does not provide the necessary application generated [0027] telemetry 600, the telemetry processor 700 can examine the state of the host computer 102 in real time and generate useful telemetry by eavesdropping on the state of the host computer's 102 keyboard, mouse, joystick, and the like. This raw self-generated telemetry 800 is an effective substitute for application-generated telemetry; however, the application-generated telemetry is much preferred if it is available. When there is no application generated telemetry available, the universal tactile feedback system runs in its host-independent audio analysis mode of operation, which is known by the registered trademark name AudioSense® (U.S. Trademark Registration Number 2261526, registered Jul. 14, 1999). In this case, AudioSense® autonomously generates useful “base level” tactile feedback, which is then “directionally scaled” by the self-generated telemetry 800 that is culled by the telemetry processor 700 from the host computer 102. AudioSense® has no capabilities as of this writing to apply left, and/or right, and/or front, and/or back directional scaling based upon its audio analysis. While an effective technology, it is greatly enhanced by directional scaling from self-generated telemetry 800. The presumption is that if the software application 104 provides application-generated telemetry 600 for the benefit of the telemetry processor, most often AudioSense® will ideally be disabled while such telemetry is available. However, AudioSense® can work simultaneously and cooperatively with application generated telemetry 600 as well, sometimes lending an “analog flavor” to the tactile feedback produced by the universal tactile feedback system 100.
Further in FIG. 1, the [0028] telemetry processor 700 provides a graphical user interface 900 for configuring and adjusting the way the available telemetry is utilized. The telemetry processor 700 is itself a software application that manages the available telemetry and performs the processing that converts said telemetry into a useful data stream for generating and/or shaping tactile feedback. More detail will be provided on this later in this disclosure.
Continuing on to FIG. 2, an illustrative flow chart is shown wherein the [0029] telemetry processor 700 will process telemetry from a typical car based game or simulation. The telemetry processor 700 begins at the START 702. In step 704, the telemetry processor 700 accepts the telemetry via a shared memory mapped data structure. In the illustrative data set, the telemetry processor 700 is given the following illustrative values: the simulated car's engine RPM, vehicle MPH, throttle position, lateral g-force, forward acceleration and deceleration, the car's brake position, the car's steering wheel deflection, the current transmission gear, the surface type that the vehicle is currently on, and damage flags that might indicate various damage typical of cars, such as flat tires, body damage, engine failure, transmission failure, and the like. All of the received telemetry is in a “raw” state, that is, in a format native to the software application 104 that generated it. In step 706, scaling factors are applied to “normalize” the telemetry. For example, in one car simulation, the engine might idle at 700 RPM and might redline at 5500 RPM. However, in another car simulation, the engine might idle at 1500 RPM and redline at 9000 RPM. A specific constant RPM scaling factor might be used to always scale the RPM range for any given simulated car to a range of 0 to 100, and then, a baseline constant may be added, for example, 20. This would mean as long as the engine is started, the lowest scaled RPM value would be 0+20, or 20, where the highest RPM value would be 100+20, or 120. In this way, the “raw” telemetry will have been scaled into a useful range of 20 to 120. If the raw RPM value is 0, however, the baseline RPM constant +20 would be skipped, thereby leaving the RPM at 0. In this way, a useful RPM value is created. Now, once the initial scaled RPM value is rendered, another scaling layer is applied from a graphical user interface 900 that is provided by the telemetry processor 700. The graphical user interface 900 allows the user to make the RPM value more or less prevalent in the tactile feedback that is perceived. Typically, the RPM scaling factor would be a floating-point decimal value from 0.00 up to 2.00, including all interim values. Let's say the user leaves the RPM scaling factor in the graphical interface 900 at 1.00. In that case, the final scaled RPM range would be 20-120, or zero if the simulated car's engine was off or disabled. If the user set the graphical RPM scaling value to 1.50, the final scaled RPM range would be 20 times 1.5, or 30, up to 120 times 1.5, or 180, giving a final range of 30 to 180. In this way, by adjusting a simple control, the original unadulterated raw RPM value as generated by the simulation is converted to a useful and preferred strength. A similar process is used for all the telemetry variables, such that the relative strength of each can be tweaked to produce useful results. However, all the final results descend directly from the raw telemetry as calculated by the original software application 104 as mapped into the application generated telemetry 600 shared memory area. More detail will be examined later in this disclosure.
Continuing with FIG. 2, in [0030] step 708, we determine the baseline feedback level from the fundamental telemetry variables engine RPM, vehicle MPH, and throttle position. These are non-directionally scaled forces, which is why they together comprise what is known as the “baseline” tactile feedback level. In much the same process as above, the final scaled engine RPM value is calculated. With regard to MPH, lets assume in one game the given car has a speed range of 0 to 130 MPH, and in another game or simulation, the car has a speed range of 0 to 236 MPH. In the case where the top speed is 130 MPH, a constant scaling factor would be the current speed always divided by 1.3 (or rather multiplied by 0.769) such that the speed range was reduced to 0 to 100. Likewise, in the case where the car's top speed was 236 MPH, the constant scaling factor would be the current speed divided by 2.36 (or rather multiplied by 0.424), thereby yielding the same range 0 to 100. This makes the scaled MPH range consistent from game to game and simulation to simulation. Then, in the graphical interface 900 within the telemetry processor 700 application, a MPH scaling factor is provided where the user can increase or decrease the importance of the MPH by adjusting a multiplier with a graphical scroll bar, where the scaled MPH range now from 0 to 100 would be multiplied by any selected number 0.00 to 2.00, in the same way was the RPM was previously.
The next step is to compare the final RPM result and MPH result, and to accept the highest value as the baseline tactile feedback value for that cycle through the [0031] telemetry processor 700, which typically runs at hundreds of cycles second. This way, if you are going 0 MPH but your simulated engine has produced a final RPM vibration value of 88, you'll feel tactile feedback at a “strength” of 88. Again, this is illustrative only. Likewise, if you are traveling at a velocity of 150 MPH, and your MPH scaling factor is 0.70, but you've just blown your engine up and are at 0 (zero) engine RPM, you'll still feel 150 MPH times 0.70, for a vibration “strength” of 105. In this way, RPM and MPH are constantly fighting each other depending on which is the higher value at each and every single pass through the code in the telemetry processor 700 (which may be running at several hundred cycles per second).
Now the telemetry processor factors in the throttle position, because RPM or MPH should both feel stronger if you are full on the throttle, or not on the throttle at all. For example, if your throttle position runs from 0.00 at no throttle to 1.00 at full throttle, we may set a first scaling factor of “plus 0.60”, that is, adding 0.60 to each end of the throttle range, which changes the throttle scaling factor from 0.00-1.00 to 0.60-1.60. Now, the graphical scaling factor is utilized to change that range yet again according to the user's preference. If the user sets the graphical scaling factor for the throttle to 0.82, the final throttle scaling factor becomes 0.82 times 0.60 (or 0.492) at the low end, and 0.82 times 1.60 (or 1.312) at the high end, so the final range is 0.492 to 1.312, depending on the position of the throttle. [0032]
From this very simple basis, the raw telemetry calculated by the [0033] software application 104 has produced tactile feedback that is indicative of the engine RPM, vehicle MPH, and driver's throttle position. This is the “baseline” tactile feedback. Were the application generated telemetry 600 not provided, the AudioSense® audio analysis mode of operation of the universal tactile feedback controller 200 would have provided this baseline tactile feedback level. The importance of this cannot be stressed enough, because directional scaling is completely worthless if it is not based on an effective “baseline”.
Continuing on to step [0034] 710, the baseline tactile feedback values are now scaled according to the lateral gforce, acceleration, and deceleration telemetry. The unique nature of the preferred embodiment is that the tactile feedback seat in the parent cases has multiple zones, that is, a back zone, a left leg zone, and a right leg zone. Much like stereo speakers can create the illusion of a sound originating from any point between only two speakers by biasing the amplitude of the broadcast sound toward one speaker, and reducing it at the other, the tactile feedback seat functions in much the same way. If we are in a left turn, simple math is applied to increase the tactile feedback felt in the right leg zone, and to decrease the tactile feedback in the left leg zone. For example, if the lateral g-force is 2.3 g's, the tactile feedback for the right leg is be multiplied by 2.3, while the left leg is divided by 2.3. The scaling factors would change the numerical range of the G's to bring them into a useful range. In this way, the driver of the simulated car feels tactile cues that indicate he or she is being thrown to the outside of the turn, in direct proportion to the lateral g-forces involved. Likewise, upon acceleration, tactile feedback in the back zone is increased (i.e., multiplied by the scaled telemetry acceleration value) while tactile feedback in the legs is simultaneously reduced (i.e., divided by the scaled telemetry acceleration value), such that the illusion is provided that the driver is being thrown backward into the seat. Likewise, on deceleration, tactile feedback is increased at the legs and simultaneously reduced at the back, such that the illusion is provided that the driver is being thrown forward in the seat, as previously in direct proportion to the g-force values involved. The first layer of scaling factors work in the same way to bring the raw telemetry into a useful range, while the graphical interface is used to set the user's preference as to how strong this directional scaling is to be perceived. By changing any given scaling factor, the math that is applied to the raw scaled data is change accordingly. In this way it is very easy to dial in excellent tactile feedback, because all the calculations are based on the mathematics that are already built into the simulation due to the nature of the shared telemetry.
Continuing on to step [0035] 712, tactile feedback texturing is applied to the now directionally scaled tactile feedback values. Texturing adds choppiness and other characteristics to the tactile feedback. For example, if a “damage flag” indicates that there is a flat tire on the left side of the car, there may be a “flat tire scaling factor” provided in the graphical user interface 900 within the telemetry processor 700 that sets the flat tire scaling factor as 1.5 times (1.5×). The texturing to the left leg zone in the tactile feedback seat (the side of the vehicle with the simulated flat tire) would now be periodically multiplied by 1.5× in some time period that is directly proportional to the current MPH of the car. Again, very simple math is used to create dramatic effects, all directly descending from the raw telemetry and the same general-purpose algorithms in place for most games or simulation of the car type genre. In other tactile feedback technologies, the effects for flat tires are canned periodic cycles of bumping that has no correspondence with the speed of the given vehicle because there is no telemetry shared between the application and the tactile feedback algorithms.
In the case of the car's current gear (the vehicle's transmission) telemetry data element, simple math can yield a brief skip in the tactile feedback throughout the tactile feedback seat any time the gear changes. For example, if the gear has just changed from 2 to 3, the telemetry processor may have two factors associated with gear changes, that is, a percent reduction value, and a time value in milliseconds. If the gear change scaling factors indicate that the tactile feedback throughout the seat should be multiplied by 0.25 and that condition should last for 100 milliseconds every time a gear changes, then each gear change will cause a direct self-generating effect of a 75% reduction in perceived tactile feedback for 0.10 seconds at every gear change. For the brake pedal position telemetry data, hard braking at fast speeds can make other very brief scaling factors take effect. Surface type can layer additional scaling factors upon the baseline vibration. The end result is that the telemetry data itself carries the information about what is happening, and all perceived tactile feedback descend directly from the actual state of the simulation or game. [0036]
In [0037] step 714, we transmit the current tactile feedback results to the universal tactile feedback controller 200 with the standard intelliVIBE® protocol, but no limitation should be inferred about which types of tactile feedback devices can benefit from this telemetry based approach.
Continuing on to FIG. 3, it is nearly identical to FIG. 2, but the telemetry carries data relevant to flight simulations instead of car simulations. [0038] Steps 722, 724, 726, 728 are analogous to steps 702,704,706,708 of FIG. 2. However, for step 730, we refer now to FIG. 6.
The major difference here is the way the telemetry processor handles g-forces, pitch, and roll, which are data typical of flight simulations. The innovative approach here allows two possibilities for rendering aircraft rolls, that is, when an aircraft such as a modem military jet rolls on its center axis in a continuous right or left turn. In reality, when a jet enters a continuous roll to the right, the passengers get pushed to the left side and stay there due to the g-forces involved that are stable for the duration of the rightward roll. This is calculated just like the lateral g-forces of the car of FIG. 2. However, an aircraft's roll can be divided into four quadrants, each of which affects the rendering of the roll in a different way. Rather than a continuous push to one side, which is a realistic and readily available but perhaps a dull representation, we can artificially induce additional aircraft orientation data in the tactile feedback that is produced. [0039]
Referring now to FIG. 6, an [0040] aircraft 740 is depicted with its tail 742 pointing straight up, that is, the aircraft is perfectly level. The current quadrant is determined by the orientation of the aircraft's tail 742. Whichever quadrant 1, 2, 3 or 4 the aircraft's tail 742 points to, that is the quadrant the aircraft is considered to be in for the purposes of rendering the “roll” effect upon the baseline vibration. Referring now to FIG. 7, the quadrants 1, 2, 3, and 4 are analogous to the same quadrants of FIG. 6. However, FIG. 7 explains the simple algorithm by which the right leg tactile feedback value and the left leg tactile feedback value are scaled depending in which quadrant the aircraft's tail is oriented. With this simple math applied to the telemetry derived “baseline” vibration values, the user gets a excellent feel of the aircraft's orientation while interacting with any given aircraft simulation or game. With the innovative algorithm of FIG. 7, a continual roll to the right in the aircraft does not produce a static leftward bias as would be the case if driving a car, but rather, the left-right leg bias actually rolls around the seat in direct proportion to the simulated aircraft's orientation. When fully inverted, that is, upside-down, the bias disappears and the legs tactile feedback scaling is nullified. This is a completely artificial effect, having no basis in reality; however, the effect is remarkable and highly effective.
When in any vehicle, positive and negative g-forces are experienced. Some vehicles induce more positive and negative g-forces than others, but nonetheless, because of the telemetry offered by a given simulation or game, the same mathematics apply and are effective across the board. When experiencing positive g-forces, one gets the feeling of weighing more than they actually weigh, in direct proportion to the positive g-forces that are experienced. Therefore, with the telemetry based system, positive g-forces are mathematically multiplied against the leg section (both left and right legs), thereby making the user feel “heavier”, while simultaneously used as a divisor to mathematically reduce the tactile feedback to the back zone. Most often the raw g-forces are scaled into the range of 1.000 to 2.000. Of course, as with all scaling factors, the users have additional scaling factors to augment of reduce the prevalence of the g-forces. Likewise, when experiencing negative g-forces, one gets the feeling of being more weightless than is normal. Therefore, when negative g-forces are present, the negative g-force value is multiplied by the back value to increase the back zone tactile feedback, while simultaneously used as a divisor to mathematically reduce the tactile feedback to the legs, making one feel like they are rising out of the tactile feedback seat. [0041]
When the roll of an aircraft is simultaneously layered with the positive and negative g-forces, one gets to experience a very tactile and informative representation of the aircraft's flight envelope. Similarly, pitch and yaw, and any and all other telemetry, can be layered into the calculations as well. The forces so calculated are not arbitrary, yet are based upon the actual math occurring in real time in the shared telemetry memory area. The only subjective element to all these forces are the user's adjustment of the graphical scaling factors in order to make any particular effect more or less prevalent in comparison to all the other forces for which telemetry data is available. [0042]
Referring now to FIG. 4, the [0043] telemetry processor 700 does not have the benefit of application-generated telemetry. Therefore, the universal tactile feedback controller 200 will run in its audio analysis mode of operation (AudioSense®). When in AudioSense®, the baseline tactile feedback is produced without capabilities for directional biasing, regardless as to the forces acting on the user's vehicle or person in any particular application. Simply put, the tactile feedback system has no awareness as to the specific calculations that are necessarily taking place in any particular application. When this is the case, the telemetry processor begins in step 782 and in step 784 examines the telemetry it can derive from the host computer's 102 control input devices. If, for example, the user of a non-telemetry supportive title (i.e., no intelliVIBE® support) is running in AudioSense®, they can give specific scaling instructions to the telemetry processor 700 so that it can generate real time tactile feedback zone-specific scaling factors that will introduce real time scaling upon the AudioSense® audio analysis results. For example, if the user is playing a car racing simulation that offers no intelliVIBE® support, the telemetry processor can track the movement of the player's joystick and can know its deflection from center on both the X and Y axes, that is, left and right (steering) and forward and back (acceleration and brake) of the simulated vehicle. Certainly this is not as good as real time telemetry, but the telemetry processor 700 will do the best it can with what data it can find. In this case, the user will adjust the joystick scaling factors such that the longer the accelerator axis (presumably the Y axis, but configured as such by the user) is pressed forward, the more speed the telemetry processor will assume will have been gained by the simulated vehicle. As the simulated vehicle is assumed to be going faster, it will automatically adjust the X-axis scaling factors (the car's steering) to make the lateral g-forces more prevalent. Likewise, if the Y axis indicates that the user has stepped on the brake of the simulated vehicle, the telemetry processor will indicate this in the stream of scaling factors that are being broadcast to the universal tactile feedback controller 200 such that a deceleration effect is produced by directionally biasing the tactile feedback in the seat to the legs section of the tactile feedback seating unit, and away from the back section.
The scaling factors are calculated in real time based upon assumed data, so they do not have the finesse of real-time accurate telemetry that would ideally be available. The scaling factors are 8 bit values, (0-255), wherein whatever value given as a scaling factor is divided by 32 within the universal [0044] tactile feedback controller 200. The scaling factors exist in a data table, where each single scaling factor is applied to only one of eight zones: back, left leg, right leg, chest, left hand, right hand, left foot, right foot. These zones are illustrative and not assumed to match this presently preferred embodiment. Sending a scaling factor of 32 yields no change to a particular tactile feedback zone because of the subsequent divide by 32 that occurs within the universal tactile feedback controller 200 (i.e., 32 divided by 32 equals 1.0). However, a scaling factor of 16 sent to the left leg will cut the tactile feedback there in half, because 16/32 yields a scaling factor of 0.5×. The divide by 32 algorithm is used to yield very fast mathematical performance by using a shortcut rather than full floating point math which would impede the performance of AudioSense® if the microcontroller inside the universal tactile feedback controller 200 does not offer full hardware multiply and divide.
By monitoring the joystick input over time, fairly accurate assumptions can be made about a particular game or simulation, but this requires more complicated help by way of pre-determined application-[0045] specific telemetry processor 700 configuration files. For example, in the real F16 Falcon military aircraft, the main cannon only has capacity enough for approximately 2.3 seconds of trigger time before it runs out of 30 mm cannon shells. Therefore, the telemetry processor 700 for an accurate F16 simulation with limited ammunition can assume from its configuration data that the main joystick trigger (or some corresponding keyboard key) fires the main cannon. Without proper telemetry being given, the telemetry processor will keep track of the time the cannon trigger is active. During the presumed activation of the cannon, an effect will be rendered according to the scaling parameters in effect for the cannon. After approximately 2.3 seconds of trigger time, the intelliVIBE® cannon effect will be disabled until reset by the user. However, AudioSense® will still do its best to render the cannon by interpreting the sound of it firing if the cannon is not yet depleted of ammunition. Additionally, if the throttle has been active and at a high power, and the joystick has tended to be pulled back, the telemetry processor can assume that the aircraft is airborne, and can then do its best to use the joystick position to send scaling factors to the left leg, right leg, and back zones of the tactile feedback seat in order to simulate the G-forces and bank roll effects that would ideally be more accurately prompted by fully supported telemetry. Even with the slightly less-than-ideal implementation of AudioSense® audio analysis and assumed game state data by way of brute force self-generated telemetry via game-specific control input analysis, the combination of cooperative AudioSense® with intelliVIBE® scaling is a very effective means of self-generating very appropriate tactile feedback, regardless of the lack of support by any given application.
Referring now to FIG. 5, a group of user adjustable telemetry scaling factors is presented. These controls are illustrative only for the purposes of understanding. Getting a closer look at FIG. 5A, the RPM baseline tactile feedback generator is presented. The baseline setting of 30 establishes the basic level of prevalence of engine RPM induced tactile feedback. The multiplier at the bottom shows 0.7, which is used to scale the 30 baseline setting. If the aircraft is known to be on the ground, the engine RPM setting is multiplied by 0.9, whereas if the aircraft is in the air, the engine RPM is multiplied by 0.6, thereby making the engine RPM feel weaker once in the air. [0046]
Referring now to FIG. 5B, the speed value is multiplied by 0.8 if the aircraft is on the ground, while it is multiplied by 0.08 if the aircraft is in the air. This has the effect of making the speed tactile effect ten times weaker when the aircraft is airborne, but the typical maximum speed when on the ground might be only 180 MPH, but when airborne, a modern military aircraft can travel at 1,300 MPH and higher. These scaling factors allow the tactile feedback to be matched to a useful range regardless as to the performance envelope of any particular aircraft, simply by adjusting the slider knob of the corresponding scaling factor control. [0047]
Looking now at FIG. 5C, we have a compound effect called ground texturing. Ground texturing is set to begin at 30 MPH and end at 250 MPH, which is well past a conservative take-off speed. Ultimately as the aircraft goes faster, the ground will create more and more texturing, that is, bumpiness in the tactile feedback. Upon takeoff, the ground texturing will be reduced to 25% (multiplied by 0.25×) until the take off speed is surpassed, which then nullifies the ground texturing, until the aircraft comes in for a landing, at which time the ground texturing will automatically take effect yet again. [0048]
FIG. 5D presents the bank scaling value. The 1.0 as shown lets the bank scaling take place just as the calculations in FIG. 7 indicate. However, if the user wants the bank scaling to be stronger, they only need to increase the corresponding 1.0 scale factor setting. To reduce bank scaling, setting bank scaling to toward 0 will nullify the effect accordingly. [0049]
FIG. 5E presents the positive and negative G-force multipliers. In positive g-force conditions, these settings multiply the legs by 1.3 times the g-force value, and multiply the back by 2.5 times the g-force value. In negative G-force conditions, the back is reduced by half and the legs are scaled to 80% for the duration of the negative G-force condition. [0050]
In FIG. 5F, the scaling factors for the main gun are available. During the firing of the main gun, 100 pulses per second will occur in the tactile feedback, alternating between a high pulse percentage of 100% and a low pulse percentage of 12% as determined by the scaling factors. The “use high intensity” check box allows the main gun to temporarily over-ride the current intensity setting for the duration of the main gun being fired. Knowing that the main gun is being fired can be given to us by application generated [0051] telemetry 600 or self-generated telemetry 800, same as with all the effects and scaling factors of FIG. 5A through FIG. 5H.
In FIG. 5G, the [0052] telemetry processor 700 is told what key press equals what known effect. This is configured by the user. For example, we can see in the upper left corner that the “G” key activates the landing gear. This way, if the user presses the G key to deploy the landing gear, the telemetry processor can run a simple effect representing the landing gear. If telemetry is not provided that indicates the true state of the landing gear, the telemetry processor 700 will fall back on its self-generating capability and keep track of the G key presses on its own and will do its best to stay in sync with the simulation in question.
In FIG. 5H, an illustrative afterburner effect is shown. Many modem jets have multiple stages of afterburner. In FIG. 5H, there are 3 stages of afterburner, where stages one, two, and three yield tactile “thumps” at 60%, 80%, and 100% strength, respectively, each lasting only 0.01 seconds, but also incorporating an RPM scaling factor of 1.33×, 1.5×, or 1.66× times respectively, which will increase the prevalence of the engine RPM according to what level of afterburner the aircraft is in, for as long as the afterburner is active. [0053]
Ultimately, the innovative methods and apparatus of the present invention allows for the most versatile tactile feedback results to be produced, both with and without support of any kind by the [0054] host computer 102 and the executing software application 104. The best condition is for fully detailed low-burden telemetry to be provided for the benefit of the universal tactile feedback controller 200 by a given developer. However, in the absence of given telemetry via a shared data structure by standard windows memory mapping, the telemetry processor 700 of the present invention is able to generate data on its own to supplement the AudioSense® audio analysis mode of operation with real time directional biasing of all tactile feedback, in the end producing effective zero-burden feedback from the point of the game developer . . . in the end eliminating the fundamental problem of lack of quality developer support!
Although various embodiments which incorporate the teachings of the present invention have been shown and described in detail herein, those skilled in the art can readily devise many other varied embodiments that still incorporate these teachings. Accordingly, the described embodiments are to be considered in all respects only as illustrative and not restrictive. Therefore, it is to be understood that nothing shall be construed as a limitation of the present invention, other than the structure recited in the appended claims. [0055]

Claims

What is claimed is:

1. In a computer gaming or simulation system, a method for producing tactile sensations in correspondence to the activity occurring on said system, said method comprising:

an executing software application that is providing telemetry indicative of its real-time state;

a real-time telemetry processor for processing said telemetry with various scaling factors and algorithms;

wherein said tactile sensations are derived from said telemetry processor.

2. The method of claim 1, where said tactile sensations are generated by:

a flexible pad;

a plurality of actuators, attached to said pad, for selectively generating tactile sensation.

3. The apparatus of claim 2, wherein said plurality of actuators are comprised by offset weights affixed to DC motors, such that when said motors are energized, said offset weights induce physical vibrations of varying intensities.

4. In a computer gaming or simulation system, a method for producing tactile sensations in correspondence to the activity occurring on said system, said method comprising:

an executing software application that is not providing telemetry, and therefore, providing no information indicative of its real-time state;

a real-time telemetry processor that will self-generate telemetry for its own use by examining the state of said computer gaming or simulation system; and

using said telemetry to directionally bias and/or supplement and otherwise enhance the tactile effects occurring as a result of simultaneous audio analysis.

5. The method of claim 4, where said tactile sensations are generated by:

a flexible pad;

6. The apparatus of claim 5, wherein said plurality of actuators are comprised by offset weights affixed to DC motors, such that when said motors are energized, said offset weights induce physical vibrations of varying intensities.