WO2015005959A1

WO2015005959A1 - A touchscreen capable of fingerprint recognition

Info

Publication number: WO2015005959A1
Application number: PCT/US2014/032598
Authority: WO
Inventors: Patrick Baudisch; Christian Holz
Original assignee: Patrick Baudisch; Christian Holz
Priority date: 2013-04-01
Filing date: 2014-04-01
Publication date: 2015-01-15

Abstract

An apparatus that includes a transparent element having a first surface configured to receive a finger input, a first light source configured to direct light to the first surface, an optical sensor optically coupled to the transparent element and configured to capture the light that is reflected off the first surface, and a display element configured to produce a visible image on the transparent element is presented.

Description

A TOUCHSCREEN CAPABLE OF FINGERPRINT RECOGNITION

RELATED APPLICATION

[0001] This application claims the benefit of U.S. Provisional Application No.

61/807,273 filed on April 1, 2013, the content of which is incorporated by reference herein in its entirety.

BACKGROUND

[0002] The present disclosure relates generally to touchscreens, biometrics-based authentication, and touchscreens with fingerprint recognition capability.

[0003] Today, electronic devices and computing devices play an important function in many aspects of one's life, including work, social life, and entertainment. Many of the functions that used to be executed by humans are now done by machines and computing devices. While machines and computers may be able to work faster and more accurately than humans in many aspects, they are deficient in others. One of the areas in which the machines and computers are deficient is "judgment," as the personal computing devices and electronics often do not incorporate capabilities to distinguish one user from another. Hence, in some ways, the ubiquitous use of machines and computers has created room for accidents and thefts (e.g., information thefts).

[0004] A technique that allows machines and computing devices to distinguish users and react differently to different users is desired.

SUMMARY

[0005] In one aspect, the inventive concept pertains to an apparatus that includes a transparent element having a first surface configured to receive a finger input, a first light source configured to direct light to the first surface, an optical sensor optically coupled to the transparent element and configured to capture the light that is reflected off the first surface, and a display element configured to produce a visible image on the transparent element. A first portion of the light that is incident on an area of the first surface that is covered by a valley of the finger reflects off the first surface and a second portion of the light that is incident on an area of the first surface that is covered by a ridge of the finger is frustrated.

BRIEF DESCRIPTION OF THE DRAWINGS

[0006] FIG. 1A and FIG. IB depict a prism-based optical fingerprint scanning arrangement.

[0007] FIG. 2A and FIG. 2B depict an waveguide-based fingerprint scanning arrangement.

[0008] FIG. 3A depicts a fiber optic -bundle based fingerprint-sensing touchscreen device in accordance with one embodiment of the inventive concept.

[0009] FIG. 3B is a schematic diagram of an embodiment of the fingerprint-sensing touchscreen device 10 of FIG. 3 A.

[0010] FIG. 3C depicts an example embodiment of an authentication process that may be used with the fingerprint scanning arrangements disclosed herein.

[0011] FIG. 4A, FIG. 4B, FIG. 4C, and FIG. 4D depict embodiments where multiple users are authenticated for one computing device.

[0012] FIG. 5A and FIG. 5B depict a transfuser that may be used to implement some embodiments of the inventive concept.

[0013] FIG. 6 illustrates a high-contrast fingerprint illuminated from behind the optical sensor.

[0014] FIG. 7A, FIG. 7B, FIG. 7C, and FIG. 7D explain the source of ring diffusion.

[0015] FIG. 8A and FIG. 8B illustrate that pointing a single layer at a fiber bundle

(transfuser) causes diffusion of incident light into a ring. [0016] FIG. 9 illustrates how a graded-index fiber bundle blurs ring diffusion.

[0017] FIG. 10A and FIG. 10B depict a fiber bundle (transfer)-based fingerprint scanner arrangement.

[0018] FIG. 11 depicts an example of a fiber bundle-based scanner capable of detecting fiducial markers.

[0019] FIG. 12A and FIG. 12B depict exemplary arrangements for overcoming formation of hotspots.

[0020] FIG. 13 depicts diffusion of light reflected inside a fiber-optic bundle into rings.

[0021] FIGs. 14A, 14B, 14C, and 14D depict images of an example fingerprint at different points in the processing pipeline.

[0022] FIGs. 15A, 15B, 15C, and 15D depict fingerprint-specific operations to enhance captured images.

[0023] FIGs. 16A, 16B, 17A, 17B, 18, 19A, 19B, 20A, 20B, 21, 22, 23, and 24 illustrate various alternative embodiments of the fingerprint-sensing touchscreen device in accordance with the inventive concept.

[0024] FIGs. 25A, 25B, 25C, and 25D depict different embodiments of a sensor system that may be used with a fingerprint-sensing touchscreen system that includes a display element and a transparent element in accordance with the inventive concept.

[0025] FIGs. 26A, 26B, 26C, and 26D depict different embodiments of a sensor system that may be used with a transparent element in accordance with the inventive concept.

[0026] FIG. 27 depicts an in-cell device that sense touch and displays an image, in accordance with one embodiment of the inventive concept.

[0027] FIG. 28 depicts an example application of the inventive concept.

[0028] FIGs. 29A, 29B, 29C, and 29D depict different embodiments of a sensor system that may be used with a transparent element in accordance with the inventive concept. DETAILED DESCRIPTION

[0029] In one aspect, the inventive concept disclosed herein pertains to a display element combined with an optical finger-print sensing element. The finger-print sensing element may include a transparent element such as a transfuser, a waveguide, or a prism, although this is not intended to be an exhaustive list of possibilities. The display element may be a flat display or a projection device. Several embodiments will be presented below.

[0030] As used herein, a first element being "on" a second element is intended to mean both the case where the first element and the second element are in direct contact with each other and the case where there is a gap or an intervening element between the first and second elements. Furthermore, a first element being "on" another element is not intended to imply that the apparatus is in any specific orientation, and covers the possibility of the apparatus being right way up, upside down, sideways, etc.

[0031] There have been numerous attempts at enabling computing devices and machines to distinguish between users. In one attempt, users are equipped with identification tokens that can be pressed against (or held near) a surface that incorporates a scanner or a reader of the identification information. For example, some systems can identify users by reading RFID (Radio-Frequency Identification) tags attached to the users. Other systems can identify users by observing worn accessories that emit unique patterns (e.g., rings that flash unique sequences of light, such as an IR (Infra-Red) Ring). Also, some systems can identify users by reading fiduciary markers attached to a glove on a user's hand. In some instances, the systems can identify users through identification tokens on the users' mobile phones.

[0032] The token-based identification techniques suffer from the obvious weakness that they work only if users carry the props/tokens with them. However, just like electronic devices, tokens can be exchanged, lost, stolen, etc. Hence, to free the users from the burden of keeping track of their tokens, tabletop systems with additional sensors have been proposed. For example, some systems can reliably identify users when they are touching the surface of the system, but require users to remain stationary during identification. An example is the DiamondTouch™ , which is a tabletop system that identifies users by electrically connecting their chairs to the tabletop, and closing a loop when a user touches the table. While the DiamondTouch™^' 's approach may be robust, it nonetheless requires users to remain stationary for the electrical loop to be closed. These built-in sensors free the users from having to track their tokens, but this freedom comes with sacrificed mobility.

[0033] Methods of identifying users based on the color of their shoes or reflections off their arms have also been proposed. For example, some tabletop systems can be instrumented with additional sensors to identify users based on their shoes or arm contours. However, those systems typically identify users less accurately, because the extracted features are not as robust and unique (for example, compared to features obtained in fingerprint sensing). As a result, recognition accuracy is sacrificed in those systems.

[0034] Biometric identification, in contrast to the above-described identification methods, allows users' identities to be determined accurately, without requiring the users to carry identification tokens, or having to instrument the systems with a large number of additional sensors. For example, some systems (e.g., HandsDown™) allow users to be recognized by their hand contours, but require that the users press their entire hand against the surface of the system. Besides hand contours, fingerprints are a biometric identification feature that is widely used, because fingerprints exhibit unique patterns of structural features. indexOptical fingerprint sensing

[0035] The inventive concept disclosed herein pertains to combining the principle of biometric identification with touch screen technology.

[0036] Many optical tabletop systems (e.g., Holowall™ and Microsoft Surface™ table) are based on the diffused illumination principle. Diffused illumination systems capture the light reflected by objects through the diffuser to detect touch or to recognize fiducial markers. These tabletop systems employ different materials to see through the diffuser, such as a Holoscreen that diffuses light only from a particular direction (TouchLight™) or a switchable diffuser (i.e., a screen that switches between clear and diffuse many times a second, for example

SecondLight™).

[0037] To design a touchscreen that is also capable of recording fingerprints, the device would have to obtain sufficiently crisp images of fingerprints, i.e., the device would have to first generate contrast between the ridges and valleys of the fingerprint. In the case of an optical system, such as those based on diffused illumination, this means they would have to generate optical contrast, i.e., valleys would have to be dark and ridges light, or vice versa. However, the existing diffused illumination systems do not produce such high contrast, because the user's finger reflects light in its entirety and the system's diffuser further blurs all details.

[0038] A thermo-plastic elastomer can allow the scanning of skin structure (e.g.,

GelSight™). Very high contrast in fingerprint sensing is typically achieved by fingerprint scanner devices that illuminate a reflective surface, such that the fingerprint ridges diffuse that reflection locally (i.e., they "frustrate" the reflection). The device then optically senses this specular reflections or the lack thereof to obtain an image of the fingerprint. This approach is herein referred to as the "scanner" approach, the "scanner" being a piece of transparent element such as glass or acrylic.

[0039] FIG. 1A depicts an example of a prism-based scanner. This particular embodiment uses a prism-based optical fingerprint scanning arrangement that is capable of producing high-contrast scans. As shown, the prism-based arrangement has a solid prism made of glass or similarly clear material, with a diffuser on one surface, a finger contact surface on another surface, and a camera on a third surface. Light is shined into the prism through a diffuser and propagates toward the finger contact surface. Due to the incident angle (see FIG. IB), the prism surface fully reflects the light into the camera. However, when a fingerprint ridge touches the surface, the light frustrates, and the finger's skin scatters the light, such that a negligible amount of light reaches the camera (see region "c" in FIG. IB). In contrast, where a ridge on the skin of the finger does not contact the prism surface (see region "b" in FIG. IB), most of the light is reflected toward the camera. Thus, fingerprint ridges appear dark in the image, while everything around the ridges appear bright due to the fully reflected light, thereby resulting in the images having high contrast.

[0040] However, the existing prism-based optical fingerprint scanner cannot form a touchscreen, because the touch surface on top of the prism does not produce visual output, nor could one project on it. Projection, in particular, requires a diffuse surface, whereas scanning fingerprints requires the top of the prism to be polished in order to obtain reflections.

[0041] A range of HCI (human-computer interaction) systems (e.g., optical tabletop systems) have exploited the ability of glass fibers to bend and still transmit light from one end to the other. For example, FiberBoard packs optical sensing into a small form factor by folding the optical paths of its camera system. Lumino™ channels light through tangibles, thereby allowing tabletop systems to sense stacked objects. To produce output, FUSA²™ projects onto plastic fiber bundles and senses hands that are hovering on the fibers' loose ends, using IR sources positioned between the fibers.

[0042] A number of input-only fingerprint scanners have been proposed that exploit

Fresnel reflection inside glass fibers. While the contrast between ridges and valleys is lower (in the glass-fibers based scanners) than in prism-based scanners, the main advantage of the glass- fibers based scanners is that the image sensor in the glass-fibers based scanners does not require any lens. Instead, the fibers (in the glass-fibers based scanners) guide the light directly onto the image sensor.

[0043] Some scanner setups employ slanted glass fibers and illuminate the fiber bundle from the side or below. Similar to prism-based scanning, light is reflected into the fibers and guided onto the sensor, except the light frustrates once a finger touches the fibers. Besides slanted glass fibers, straight glass fibers have also been used to record fingerprints by illuminating between fibers and capturing and guiding reflections onto the sensor with fibers, or by using solid bundles and illuminating from below and capturing reflections using a camera. However, the above scanner setups are only able to scan fingerprints with high contrast on a small area, due to limitations in the placement of the light source (in these scanner setups).

[0044] Furthermore, the glass-fibers based fingerprint scanners described above do not produce any visual output. Instead, the scanners merely use the glass-fiber bundles for scanning.

FTIR Touchscreens do not sense fingerprints

[0045] As described above with reference to glass-fibers based fingerprint scanners, light frustrates once a finger touches the transparent surface of an appropriately designed device. A related concept is used in creating FTIR touch screens - this design frustrated total internal reflection (FTIR) to sense touch input. It does not capture fingerprints, however, even if it were used in combination with a high-resolution camera.

[0046] FIGs. 2A and 2B show the waveguide-based setup of a touchscreen (shown without a compliant surface). A "waveguide," as used herein, is a clear solid material that allows light to travel through it. Unlike the prism-based scanner of FIG. 1A, which looks for the reflected light, this waveguide-based design looks for light diffused by frustration. Comparing FIG. 2A with FIG. 1A, a difference between the waveguide setup and the prism-based fingerprint scanner is that the camera in the waveguide setup sees the finger at all times. In the waveguide setup, when the finger touches the surface, the light escapes the waveguide and illuminates the fingerprint ridges. The skin on the finger, however, diffuses the light at a depth of about 1mm (see region "b" in FIG. 2B), which causes the light to spill over into adjacent valleys. The camera thus captures the diffused light reflected by the finger. In contrast, prism- based optical fingerprint scanners capture the light that reflects inside the prism (i.e., waveguide) and derive the contrast of fingerprint captures from observing these reflections directly.

Therefore, touchscreen waveguide produces very little contrast between ridges and valleys.

[0047] In addition, waveguide-based touchscreens typically use a compliant surface on top of the waveguide to act as a diffuser for projection. The compliant surface is comparably coarse, as it is designed to create a signal for touch input, not fingerprint input. Similar to diffused illumination systems, this compliant surface further impedes sensing fingerprints, as it acts as a diffuser, blurring the detail required to sense fingerprints. It also dampens touch and prevents ridges from leaving distinct impressions on the waveguide.

[0048] Because waveguide scanners are unable to create the desired level of contrast between the valleys and ridges of a fingerprint that is needed for capturing fingerprints, they cannot be integrated into diffused illumination systems for capturing fingerprints.

[0049] Hence, none of the above-mentioned techniques fulfill the need for a fingerprint- sensing touchscreen system, i.e. a system that allows simultaneous image projection and biometric (e.g., fingerprint) sensing. In one aspect, the inventive concept disclosed herein fills this void by enabling a touchscreen that can also read fingerprints for authentication. In some embodiments, the fingerprint-sensing touchscreen may be applied to a multi-touch display system.

[0050] In some cases, the inventive concept may be implemented in an interactive tabletop system that is designed to be used by multiple users at the same time, similarly to an optical tabletop system (e.g., Microsoft Surface™ table). Distinguishing simultaneously interacting users and associating each interaction to a particular user in real-time allows tabletop systems to personalize, log, and verify interaction. For example, a collaborative application in a tabletop system can manage users' private and public objects, and ensure that only the respective owners (or designated users) can access private objects. To do that, the tabletop system has to be able to first identify users using one or more methods of authentication.

[0051] In this disclosure, a multi-touch system that senses and authenticates users biometrically based on their fingerprints during each interaction is provided. The authentication process may be executed whenever the system requires, e.g., periodically or on a triggering event, such as on every touch; in between the system may track the finger using the same or a different sensor. Hence, even if an authenticated user successfully logged in, a different user would not be able to continue as the authenticated user while the authenticated user is away.

[0052] The display in the disclosed multi-touch system is touch-sensitive and capable of creating a desired level of contrast for sensing and capturing fingerprints. This disclosure presents several embodiments of an apparatus that includes a transparent element having a first surface configured to receive a finger input, a first light source configured to direct light to the first surface, a camera optically coupled to the transparent element and configured to capture the light that is reflected off the first surface, and a display element configured to produce a visible image on the transparent element.

[0053] One group of embodiments includes a fiber-optic bundle for simultaneous image projection and fingerprint sensing. The fiber-optic bundle diffuses incoming light, thereby acting as a projection surface and, at the same time, reflects light specularly, which enables sensing of fingerprints. This combination of touch surface + display device, is herein referred to as a "transfuser." The transfer includes a fiber-optic bundle and diffuses light on transmission while also offering a specular reflection.

[0054] As mentioned above, the transfuser includes a fiber-optic bundle that simultaneously implements specular reflection and diffuse transmission. The diffuse transmission allows the exemplary system to project image, while the specular reflection allows the exemplary system to sense fingerprints exploiting frustrated Fresnel reflection. By combining reflective and diffuse properties, the fiber-optic bundle thereby allows the exemplary system to image the projection and obtain optical frustration on touch.

[0055] FIG. 3A depicts a fiber optic -bundle based fingerprint-sensing touchscreen device 10 in accordance with one embodiment of the inventive concept. As shown, the fingerprint- sensing touchscreen 10 includes a sensing system 20 and a projection system 30. The sensing system 20 includes a transfuser 22 disposed on top of a projector 32, an infrared (IR) light source 24, and an optical sensor 26 (e.g., a camera). The projection system 30 may be any conventional, commercially available projection system. In the particular embodiment that is shown, the optical sensor 26 is mounted at an angle to avoid capturing the hotspots caused by the IR light source 24. As a result, the IR light source 24 shines light at the fiber-optic bundle from an angle. However, this is not a limitation of the invention and a person skilled in the art would know to adjust the exact positions of the components as he sees fit for the exact application.

[0056] The system in FIG. 3A uses a shared location for the optical sensor 26 and light source

24 . A "light source," as used herein, is intended to include any radiation source, such as infrared source.

[0057] The particular embodiment depicted in FIG. 3A uses a 10cmx5cm transfuser, onto which a MicroVision ShowWX+™ projects 848><480px images. The fiber optic bundle can be obtained from commercial suppliers (e.g., Schott). The camera is a PointGrey

FireflyMV™ and captures images at 1.3MP with 60fps. A set of LEDs is used around the camera lens to approximate a shared location of camera and light source. All components are mounted to the acrylic front using a GorillaPod™, which facilitates calibration.

[0058] FIG. 3B is a schematic diagram of an embodiment of the fingerprint-sensing touchscreen device 10 of FIG. 3 A. As shown, the fingerprint-sensing touchscreen device 10 includes a processing unit 40 that controls the sensing system 20 and the projection system 30. The processing unit 40 connects what is displayed by the projection system 30 with the touch detected by the sensing system 20, enabling the device 10 to know what area/button/ GUI object is "touched" so the command can be understood properly. Upon detecting a touch, the fingerprint is read and stored in local memory, perhaps indexed by user ID assigned to each user.

[0059] The fingerprint-sensing touchscreen device 10 has access to a fingerprint database that stores fingerprints, perhaps indexed by user ID and optionally indications of privilege level associated with each user ID. In one embodiment, the fingerprint database may be populated and managed by another party. In another embodiment, the fingerprint database may be populated through the fingerprint-sensing touchscreen device 10. One way in which this can be achieved is to ask a user to "register" by touching the screen. In some multi-touch embodiments, the user may be asked to touch the display with all of his fingers, simultaneously or sequentially, and all the fingerprints may be stored, e.g. in the local memory in the processing unit 40 or in a remote database. In some embodiments, the fingerprints may be stored in a remote database or a cloud and associated with a particular device identifier in addition to being stored locally on each device. This may be useful, for example, where a company issues tablets or smartphones to hundreds of employees.

[0060] In some embodiments, each fingerprint may be translated into a unique set of attributes. This way, the same fingerprint will be recognized regardless of the different pressure, angle, or surface area of the touch. More details about the image processing aspect of fingerprint matching will be provided below.

[0061] After the initial registration of the fingerprints, each time the device 10 detects a touch, an authentication process 50 is triggered. FIG. 3C depicts the authentication process 50. Upon detecting a triggering event (e.g., a touch, multi-touch, a gesture), a raw image of the touch surface is obtained (step 51). The points of contacts (touches) are identified in the triggering event and the touch coordinates (e.g., x and y) are determined (step 52). For each touch, a fingerprint is extracted (step 53a) and the touched GUI object is determined (step 53b). The minutiae of the touch, such as the coordinates of fingerprint features, are determined (step 54) and matched against fingerprint database (step 55) to identify the user and optionally his access level. If a match is found (step 56, "yes"), the user is associated with the touch (step 57a) and the touch is sent to the GUI object along with the user ID (step 57b). If there is no match found between the received fingerprint and the fingerprint database (step 56, "no"), the user may be treated as an unidentified touch (step 58a). Depending on what the user was trying to do, this could mean that his attempted action will be rejected, automatically translated into a low- privilege-level command, etc. The touch event is sent to the GUI object with no user ID associated (step 58b). The processing unit 40 reacts differently to the received touch command depending on the GUI object-user ID pairing.

[0062] FIG. 4A, FIG. 4B, FIG. 4C, and FIG. 4D depict embodiments where multiple users are authenticated for one computing device. In some embodiments where multiple users are authenticated, the users may be granted different levels of access or different functionalities. One aspect of the invention disclosed herein includes an interactive tabletop system that biometrically identifies users— without requiring additional instrumentation of the system, without requiring users to wear identification tokens or props, and without requiring users to remain stationary during identification.

[0063] FIG. 4A depicts an exemplary multi-touch system according to some embodiments. Specifically, a 19" rear-projected tabletop system that identifies/authenticates users based on their fingerprints during interaction is depicted. As mentioned above, multiple users may be authenticated for one fingerprint-sensing touchscreen device 10. The fingerprint- sensing touchscreen device 10 can associate the user ID and finger ID to each touch event. The device also enables multi-touch applications requiring fine-grained authentication.

[0064] The particular transfuser in the embodiment of FIG. 4A includes two 25cmx20cm fiber optic bundles, which create the 19" interaction and projection surface. The fiber optic bundle can be obtained from commercial suppliers (e.g., Incom). The projector is a Benq™ short-throw projector at 1024*768px resolution. While the system's setup is designed for any array of high-resolution cameras that observe the whole transfuser, the system can also use one camera that is placed at an arbitrary location under the surface. A PointGrey Flea3™ USB3 delivers 16-bit images with 4096x2160 pixels resolution at 1 1.5 fps. The camera observes an area of 20cmx 10.5cm. This roughly corresponds to 500dpi, common in fingerprint scanning. Given the restricted area, the table's footprint can be reduced by substituting the half-silvered mirror with two infrared light sources. A Sony PS3™ camera detects and tracks touches across the entire interaction surface at 640x480 pixels with 75 fps. To avoid interference between cameras and projector, a hot mirror is mounted in front of the projector and an IR pass filter in front of both cameras. All components are mounted in an aluminum frame.

[0065] FIG. 4B depicts a multi-user authentication screen where three different users are simultaneously touching the fingerprint-sensing touchscreen device 10. In this situation, the device 10 executes the authentication process 50 of FIG. 3C for each user to verify that each touch comes from an authorized user.

[0066] FIG. 4C depicts a multi-user computing device where different users are granted different functionality/access levels. In the particular situation that is depicted, User #1 is sorting through a stack of application documents, some of which are to be rejected and some of which are to be approved. User #1, however, does not have the final authority to approve any applications. He reviews the documents and, when he comes across a document that he believes should be approved, he slides it over to User #2, who has the authority to approve the application. While the fingerprint-sensing touchscreen device 10 recognizes both User #1 and User #2 as authenticated users, it will only recognize an approval command from user #2. If user #1 were to attempt to "stamp" a final approval on an application, the fingerprint-sensing touchscreen device 10 will not accept it.

[0067] FIG. 4D is a flowchart depicting one embodiment of a multi-level authentication process 60 in accordance with the inventive concept. The process 60 is triggered upon detection of multiple touches (step 61). Each fingerprint is read/extracted (step 62) and compared with the authenticated users' fingerprints that are stored (step 63). If any of the detected fingerprints does not match one of the stored fingerprints, that user is treated as a regular, low-privilege-level default user (step 64) and the touch is sent to the GUI object with no user ID. For each fingerprint that is authenticated, the computing device retrieves a user ID from the

memory/storage to determine which fingerprints belong to one user (step 65). Grouping the touches by user helps the computing device 40 understand the commands properly (step 66). For example, where there are two touches moving away from each other, knowing that the two touches are coming from one user will let the computing device 40 know that the user is trying to enlarge an image. On the other hand, if the two touches are coming from two different users, the computing device 40 will know that two users are moving documents in different directions.

[0068] Upon determining which command is being received from each user, the computing device verifies that the command being received is within the permitted range of functions or access granted to the user from which the command is received (step 67). The permitted function/access range may be stored in a local memory or database and indexed by user name or ID. If the command is within the permitted function/access range of the user issuing the command, it is executed normally (step 68). On the other hand, if a user is attempting to do something that he is not authorized to do (i.e., something that is not within the function/access range granted to him), the command will not be executed and a denial notice may be output (step 69).

[0069] In addition to user authentication, the fingerprint images sensed by the present invention can also be used to implement a wide range of interactive capabilities, such as distinguishing the hands, individual fingers, or parts of the fingers, e.g. to allow users to invoke different functions with different fingers. A system may also use this information to distinguish different multi-touch gestures and to distinguish gestures consisting of the same number, yet different fingers. The specific part of the observed fingerprint can be used to reconstruct the rotation of the finger in 3 -space; this can be used to compensate for inaccuracies in touch input (as demonstrated by Holz & Baudisch in Proceedings of HCI 2010).

The Transfuser

[0070] As previously mentioned, the transfuser differentiates the exemplary systems of

FIGs. 3A and 4A from a regular diffused illumination setup. The transfuser simultaneously fulfills two key functions. First, the transfuser serves as a projection surface, i.e., it diffuses the light coming from the projector, allowing users to see the image from all directions. Second, the fiber-optic bundle functions as a touch sensor. Unlike a regular diffused illumination device, the fiber-optic bundle produces a camera image that reveals the parts of the finger that are in physical contact with the surface.

[0071] In one aspect, the tabletop system has substantially no parallax between display and sensing layer, and supports applications of traditional diffused illumination systems, such as detecting hovering objects or fiducial markers.

[0072] FIG. 5A and FIG. 5B depict the transfuser 22, which is a bundle (e.g., millions) of optical fibers formed into a plate. The transfuser 22 may be an off-the-shelf fiber optic bundle manufactured by companies like Incom or Schott. A fiber bundle that offers a resolution of 508 dpi provides satisfactory coverage of the user's fingerprint features. As illustrated in FIG. 5A, since the system is based on fiber optics, the system's visual output appears not under, but on top of its screen, so that users can touch the output directly without parallax effect or blur. Similarly, the touch input appears to the camera without parallax. The transfuser also does not blur fingerprints.

[0073] FIG. 5B shows a close-up of an example fiber-optic bundle in the transfuser, comprising -40,000 glass fibers per square centimeter that run perpendicular to the surface. As shown in FIG. 4B, the underlying image appears on top of the bundle as each fiber transmits light down to the underlying newspaper and back.

[0074] The nature of the transfuser distinguishes the system of FIG. 3 A from a regular diffused illumination setup. The transfuser in FIG. 3A is a large fiber-optic bundle, which simultaneously performs diffuse transmission (required to display images), as well as specular reflection (needed to capture crisp fingerprints).

[0075] FIG. 6 illustrates a high-contrast fingerprint illuminated from behind the optical sensor 26. The fingerprint emerges due to a specific type of frustrated reflection. This frustrated reflection is not FTIR, but frustrated Fresnel reflection, as described later in the specification.

[0076] The key behind the exemplary devices of FIG. 3A and FIG. 4A is that its touch surface simultaneously serves as a diffuser for projection and as a reflective surface for fingerprint sensing.

[0077] As previously described, the existing scanning systems are unable to

simultaneously diffuse incoming light and reflect light specularly for fingerprint sensing.

Specifically, the two conflicting requirements are: (1) The projection system requires a diffuse screen to image; and (2) The touch surface needs to be specular in order to generate contrast for fingerprint sensing.

[0078] The fiber-optic bundle in the exemplary systems resolves the above conflict by diffusing light while conducting it inside the fibers. In other words, the light is not diffused via a diffuse surface. Both surfaces of the fiber-optic bundle are highly polished and specular, thereby preventing the light source from diffusely reflecting back into the camera. The light that exits the bundle at the other end, however, is diffuse because it diffuses inside the fibers and allows the projection to image. The diffusion of light inside the fibers will be as described as follows with reference to FIGs. 7-10. The following description pertains to how the transfuser diffuses incident light, which is not only key to projection, but also informs the arrangement of components in the table setup.

Glass fibers perform ring diffusion

[0079] In order to serve as a projection surface, a material has to diffuse incident light, i.e., scatter the light into all directions, thereby allowing an image to emerge. Glass fibers have two properties (ring diffusion, and diffusion in grade-index fibers) that have not been explored in conventional HCI systems. The exemplary system according to some embodiments exploits those two properties to accomplish projection.

[0080] FIG. 7A, FIG. 7B, and FIG. 7C explain the source of ring diffusion. FIGs. 7A and 7B show two parallel, slightly offset rays travelling down a fiber. Each ray describes the shape of a star polygon when viewed across the fiber's surface. The small difference in entry position, however, leads to a different path, resulting in a different exit direction. This explains the radial distribution of incident light.

[0081] FIG. 7C shows a step-index fiber reflecting rays in constant angles with respect to fiber's central axis. As shown in FIG. 7C, a ray's angle along the fiber's normal axis remains the same throughout all reflections at the same time, because the angle of incidence equals the angle of reflection within the fiber. Rays exiting the fiber are therefore spread along the ring of a cone, whose aperture is determined by the rays' incident angle. Therefore, optical fibers diffuse light along the azimuth angle, while maintaining the angle to the fiber's central axis. This causes the incident rays into diffuse into rings.

[0082] Referring to FIGs. 8A and 8B, light rays injected into a fiber optic bundle form a cone on exit, which manifests itself as a ring on a projection surface. In the example of FIG. 8A, the projection surface is a piece of paper located 5cm below the fiber optic bundle. The effect shown in FIG. 8A is the result of a single directed light source (e.g., a laser pointer) being directed into the fiber optic bundle. Specifically, FIG. 8A illustrates that pointing a single laser at a fiber bundle causes diffusion of incident light into a ring. The angle of incident light determines the size of the ring. For example, a smaller incident angle results in a smaller ring (FIG. 8A), and a larger incident angle results in a larger ring (FIG. 8B).

[0083] However, to use a fiber-optic bundle as a diffuse projection surface, light is required to be dispersed in all directions, such that projected images are visible from all sides. Ring diffusion alone (as shown in FIG. 8A and FIG. 8B) does not suffice to create such a projection surface, because light is only dispersed to locations on the ring.

[0084] To enable light diffusion across all solid angles, a bundle of graded-index fibers can be used. The index of refraction in graded-index fibers decreases continuously towards the outside, causing light rays to reflect smoothly as they approach the fibers' cladding. This causes light rays to take a sinusoidal path (see FIG. 7D), and allows rays to exit the fiber at different angles, resulting in the desired diffusion as shown in FIG. 9. Referring to FIG. 9, a graded-index fiber bundle blurs the ring diffusion, and disperses incoming light in all directions.

Sensing fingerprints using frustrated Fresnel reflections

[0085] Fresnel reflection occurs when light moves between two media of differing refractive indices (e.g., glass and air). Using Fresnel reflection, the exemplary system can capture fingerprints from touches. To produce Fresnel reflection, the system's light source shines light onto the fiber-optic bundle from below, as shown in FIG. 10A. Although some light is reflected, most of the light enters the fibers, as shown in FIG. 10B.

[0086] Referring to "c" in FIG. 10B, the Fresnel reflection at the top surface of the fiber bundle allows the exemplary system to sense touch and fingerprints. Although a large part of the light exits the fibers, a portion of the light is nonetheless reflected at the top surface and returns into the fiber-optic bundle to the camera. If light hits the top surface orthogonally, around 4% of the light is reflected at the top surface. A larger portion of the light is reflected if the light is incident at greater angles as described by the Fresnel equations. However, if a fingerprint ridge touches the top surface, the reflection at the top surface is frustrated and almost all the light exits the glass fibers, as shown in by "d" in FIG. 10B. When that happens, only a tiny fraction of light returns into the fiber, and as a result the fingerprint ridges will appear dark to the camera.

[0087] Using the setup shown in FIG. 10A, the camera can observe a fingerprint that is high in contrast. Similar to prism-based scanning, the high contrast results from the direct reflection of a portion of the light at the specular top surface back into the camera. As mentioned above, little light reaches the camera from the locations where the fingerprint ridges are in physical contact with the surface. Since the fingerprint ridges reflect only diffused light, the directly reflected light will have high intensity.

Placing the light source to obtain high contrast

[0088] To obtain a high-contrast fingerprint from a glass-fiber bundle (e.g., as that shown in FIG. 1 1), illumination needs to be strong, such that reflections into the fiber are high in contrast relative to the frustrated reflections at the locations of the ridges. See, e.g., FIG. 10B. [0089] Reflected light is subject to the same effect as projected light, i.e., the fiber-optic plate disperses the light into rings. Therefore, in order to capture the light reflections, the camera needs to be placed "on the same ring" as the light source, meaning that light reflections off the top surface are visible to the camera only at locations at the same angle as the light source with respect to the user's finger. Reflections are only visible at lower intensity at all other locations.

[0090] On conventional fiber-based fingerprint scanners with small surfaces, users touch a comparably small and predetermined area. This allows a camera to be placed underneath the surface, and the light sources carefully positioned to ensure optimal illumination and harvest maximum contrast across the small area. However, the above approach does not scale to larger surfaces. The camera perceives reflections from all locations across a large surface if the light from the light source logically originates from the location of the camera. The setup positions both the light source and camera at the same angle with respect to a finger no matter where the finger touches on the surface, i.e., the camera and light source always sit on the same ring.

[0091] One problem relating to illumination of large surfaces is that the light source can create hotspots, which can result in the camera being unable to resolve fingerprints with sufficient contrast. Although the areas around a hotspot provides optimal contrast, more distant locations receive only limited illumination, which can cause the camera to perceive little to no contrast.

Detection of Objects Other than Fingerprints

[0092] Being a diffused illumination device, the system can also detect hovering objects and/or fiducial markers (shown in FIG. 1 1), in addition to sensing fingerprints.

[0093] FIG. 12A and FIG. 12B illustrate two solutions to the above hotspot problem. As shown in FIG. 12A, the entire surface of the system is evenly illuminated by using (a) a half- mirror to reflect illuminations or (b) a sheet of Endlighten T below the bundle, to create one big reflection to recognize fingerprints across the whole surface.

[0094] As shown in FIG. 12A, a uniform area light source 24 is placed at the side of the table (Endlighten Twith mirror foil behind). The light source 24 directs light at the fiber-optic bundle 22 via a half-silvered mirror, and the sensor/camera 26 underneath the mirror captures the reflections. In FIG. 12A, the system produces one massive but evenly distributed reflection to capture the user's touches across the entire surface. Therefore the system can detect fingerprints within one even reflection. A working prototype of the solution of FIG. 12A is shown in FIG. 3A. Referring back to the system of FIG. 3 A, the light source 24 uniformly shoots light at the fiber-optic bundle from below, creating one evenly illuminated area. This ensures that light enters the fiber-optic bundle from all directions and that the camera can resolve fingerprints across the entire surface. Since light intensities are roughly constant across the surface, no hotspot occurs in the system (i.e., there is no oversaturation).

[0095] As mentioned above, the second solution to the hotspot problem is to use a sheet of Endlighten Tbelow the bundle. This is shown in FIG. 12B, where a layer of Endlighten T is placed below the fiber-optic bundle. While the solution in FIG. 12B is structurally more elegant (thinner form factor), the system in FIG. 12B nonetheless requires the camera's sensor to be very sensitive to varying levels of high light intensities. Since the layer shines light into all directions, the camera needs to see the reflections of the user's finger through the illuminated layer. A working prototype of the solution of FIG. 12B is shown in FIG. 3A. Referring to FIG. 3A, the system has a shared location for light source and camera, which is mounted at an angle to avoid hotspots. The light source, too, shines light at the fiber-optic bundle from an angle.

[0096] In both the solutions of FIGs. 12A and 12B, the half-silvered mirror and the transparent Endlighten T do not reduce the perceived quality of the projected image. Both the corresponding light setups (FIGs. 3A and 4A) work across the whole surface. That is, no matter where the camera is located, the camera will still see the reflection off the users' fingerprints.

[0097] The embodiment of this disclosure uses a single-spot light source that shares the location of the camera. This arrangement enables the camera to observe fingerprints across the whole surface of the fiber-optic bundle. As previously stated, the light emitted by the light source enters the fiber-optic bundle and reflects at the top surface. Since the light subsequently exits the fiber bundle in a ring shape, the camera needs to sit on the same ring as the light source for all locations across the surface. As shown in FIG. 13, an arrangement that satisfies this condition is a shared location, which maximizes the amount of light that reflects back into the camera.

[0098] Referring to FIG. 13, the light reflected inside the fiber-optic bundle diffuses into rings [(a), (b)], whose sizes depend on the incident angle. The camera thus only captures the reflections across the whole surface if it shares its location with the light source, i.e., they both sit on the same ring for all locations on the surface.

Image processing and Event Handling

[0099] In an exemplary system, the cameras and projector are calibrated to a shared coordinate system. During image processing, the system first corrects the distortions in the camera images and then feeds them into the processing pipeline. The web camera is used to extract and track touches purely based on the diffused illumination image. If a finger enters the region of the high-resolution camera, the system processes that touch separately and extracts the fingerprint. Both the touch processing and the fingerprint extraction are performed using the same camera. Finally, the system distorts the projector image to render the image correctly. All the processing steps (touch, fingerprint, matching, displaying output, etc.) run in parallel, such that touch recognition and tracking each works independently and the system continuously stays responsive to the user input.

[00100] To extract fingerprint features from the image, a processing pipeline traditionally used for improving and matching fingerprints is employed. Since the pipeline is computationally expensive but includes mainly image processing, all of the following processing described below can be implemented, for example, in CUDA 4.2 to run on the GPU (NVidia GTX 680™), so as to allow the system to process the images in real-time.

GPU Pipeline

[00101] The system extracts from the image the region of interest that contains the touch contact, which can be a 512*512 pixel image. Referring to FIG. 14A, the raw input image shows a bright fingerprint against a dark background. The fingerprint ridges appear as dark lines inside the bright area, but almost too faint to see with the naked eye. As a first step, the captured background is removed, and the contrast in the input image is maximized by stretching the histogram. FIG. 14B shows the image after histogram equalization. Next, Sauvola binarization is applied to bring out the contrast between ridges and valleys (in the fingerprint), to which an eroded mask of the original contour is applied. FIG. 14C shows the result of the Sauvola binarization. Application of the eroded mask discards the values around the edges, and a morphological opening further removes noisy pixels. The resulting image (shown in FIG. 14D) is then fed into the fingerprint pipeline.

[00102] To augment the resulting fingerprint image and extract the locations of the minutiae (i.e., ridge endings and bifurcations), the following operations can be applied to the binarized image. Referring to FIG. 15 A, a directional Gabor filter applied to the binarized image improves edge detection and smoothens noisy and interrupted ridges, which then produces the improved binarized image shown in FIG. 15B. Next, referring to FIG. 15C, thinning the binarized image allows the locations and orientations of minutiae (i.e., the bifurcations and ridge endings) to be extracted, which are the salient features of the fingerprint. FIG. 15D shows the result of the extraction. In some embodiments, the system may take about 23ms to extract the locations and types of minutiae.

[00103] To match two fingerprints based on their minutiae, the best spatial alignment of both point sets is determined. The matching score is then derived based on the number of minutiae that match in location, as well as their angular difference in orientation within a certain small range.

Additional Embodiments Using Prism-Based Scanner

[00104] FIG. 16A depicts an embodiment of the fingerprint-sensing touchscreen device 10 in accordance with the inventive concept. Unlike the embodiment of FIG. 3 A, this embodiment would not require a projector. As shown, this embodiment combines the prism- based scanner device 100 that is depicted in FIG. 1A and IB with a transparent active display, such as a transparent organic light emitting diode display (T-OLED) 160. The fingerprint scanner 100 may be made of any appropriate transparent material, such as glass or acrylic. The T-OLED 160 may be combined with the fingerprint scanner 100 by using heat and/or chemicals, or by injecting a clear adhesive between the two that has a close index of refraction as the components it is joining. An example of such adhesive is clear lacquer or silicone. The adhesive may be a liquid or a solid. If any of the affected components contain non-insulated electric components, a non-conductive bonding substance should be chosen. For all embodiments where LCDs or T-OLEDs are directly contacted, the embodiment may protect that LCD or T-OLED using an additional clear cover layer. To maintain the sensing functionality, this layer also may be bonded using clear substance.

[00105] In the particular embodiment of FIG. 16A, the transparent active display, such as a T-OLED 160, is placed on a first surface of the prism-based scanner 100, which corresponds to the top surface in FIG. 16A. The "first surface," as used herein, is the closest surface to where the finger contact occurs and the surface through which a finger input is received in an element. Although a finger may directly contact the first surface of an element, this is not a requirement and there may be another element between the surface that is contacted by the finger and the first surface. As the T-OLED 160 prevents the first surface of the fingerprint scanner 100 from being directly contacted by a finger, the T-OLED is optically integrated into the fingerprint scanner 100 in a way that minimizes or avoids light from being scattered or reflected at the boundary between the T-OLED 160 and the fingerprint scanner 100.

[00106] Additional measures may be taken to prevent the light output from the T-

OLED 160 from negatively affecting the function of the fingerprint scanner 100. For example, different light frequencies may be used for the fingerprint scanner 100 - for example, infrared, UV, or time multiplexing may be adopted. Alternatively, no image content may be displayed in the touched area, e.g. during scanning. As yet another alternative, the image may be processed by software to remove interfering signals.

[00107] FIG. 16B shows the path of light through the apparatus of FIG. 16A. As shown, light travels through the combined layers of fingerprint scanner 100, the adhesive, and the T-OLED 160. The non-visible wavelength that is emitted by the T-OLED 160 is absorbed by a touching object (e.g., a finger).

[00108] FIG. 17A depicts an embodiment of the fingerprint-sensing touchscreen device 10 in accordance with the inventive concept. This embodiment is a combination of the prism-based fingerprint scanner 100 with a passive flat display, such as a liquid crystal display (LCD) panel 170 and a backlight 172. There may be a clear liquid placed between the LCD panel 170 and the backlight 172 to prevent light reflection and/or loss. As shown, the backlight 172 may be disposed between the LCD panel 170 and the prism-based scanner 100, although this is not a limitation of the inventive concept. Separate light sources may be used for the fingerprint scanner 100 and the LCD 170, allowing different wavelengths to be used. For example, an invisible wavelength (e.g., infrared) may be used for the fingerprint scanner 100 and a visible light range may be used for the LCD. In some embodiments, the backlight 172 may simultaneously serve the LCD 170 and the fingerprint scanner 100. [00109] To achieve color images, a white light source may be used. If the backlight 172 is between the LCD panel 170 and the scanner 100, the backlight 172 is transparent, so that light reflected at the top surface is detectable by the light sensor 26.

Alternatively, the light source may be positioned along the side edge of the LCD 170 along with a diffuser to avoid interfering with light propagation toward the light sensor 26.

[00110] FIG. 17B depicts a variation of the embodiment shown in FIG. 17A.

Specifically, the embodiment of FIG. 17B includes a diffuser 174 and a visible light source 176 instead of the backlight 172. The diffuser 174 may be disposed between the fingerprint scanner 100 and the LCD 170. The light source 176, which may emit light in the visible wavelength range, may be positioned under the fingerprint scanner 100.

[00111] FIG. 18 depicts another embodiment of the fingerprint-sensing touchscreen device 10 in accordance with the inventive concept. This embodiment utilizes a projector 180 to project an image on a diffuser 182 placed on top of and optically coupled to the fingerprint scanner 100. Since most diffusers will interfere with the fingerprint scanning, a switchable diffuser might be used, so that projecting the image and scanning the fingerprint would take place at alternating moments in time, typically many times in a second. Although the figure illustrates an arrangement whereby the projector 180 is positioned on the same side of the fingerprint scanner 100 as the light sensor 26, this is not a limitation of the inventive concept.

Additional Embodiments using FTIR Waveguide

[00112] FIG. 19A depicts another embodiment of the fingerprint-sensing touchscreen device 10 in accordance with the inventive concept. As shown, this embodiment combines an FTIR waveguide 150 such as what is depicted in FIG. 2A with a display element, such as a T-OLED 190. The transparent element, which in this case is the waveguide 150, may be glass or acrylic, among other options. The T-OLED 190 is disposed on top of the waveguide 150, or on a first surface of the waveguide 150. The light source 24 is positioned on one side of the waveguide 150, and the light sensor 26 is positioned on another side of the prism 150 to receive the light that that is coming from the first surface. A clear substance 192, as in the embodiment of FIG. 16A, may be used between the waveguide 150 and the T-OLED 190 to minimize light loss and reflection at the boundary. The clear substance 192 may have adhesive properties. [00113] FIG. 19B shows the path of light through the fingerprint-sensing touchscreen device 10 of FIG. 19A. As shown, light travels through the combined layers of the device 10, the clear substance 192, and the OLED 190. The typically non-visible wavelength that is emitted by the light source 24 propagates through the waveguide 150 until it may or may not be by an object (e.g., a finger), causing it to diffuse and become visible to the light sensor 26.

[00114] FIG. 20A depicts another embodiment of the fingerprint-sensing touchscreen device 10 in accordance with the inventive concept. This embodiment combines waveguide 150 of FIG. 2A with an LCD panel 200 and a transparent backlight 202. As shown, the backlight 202 is disposed between the waveguide 150 and the LCD panel 200. The LCD panel 200, the backlight 202, and the waveguide 150 may all be bonded using a clear substance, as disclosed in one of the embodiments above.

[00115] FIG. 20B depicts an alternative embodiment where the waveguide 150 is used with the LCD panel 200, as in FIG. 20A. However, unlike in FIG. 20A, there is no backlight 202. A visible light source 29 and a diffuser 208 may be used instead for the display element.

[00116] FIG. 21 depicts yet another embodiment of the fingerprint-sensing touchscreen device 10 in accordance with the inventive concept. This embodiment utilizes a projector 210 and a diffuser 212. Optionally, a flexible, ultrathin, compliance layer (not shown) may be added to serve as a projection surface, allowing the projector 210 to display a higher- quality image. This flexible layer may be fixed to a first surface of the waveguide 150 with tiny, transparent dots of adhesive. Alternatively, the compliant surface may be created on the waveguide 150 by applying particles of a compliant material that adhere to the waveguide 150. Any fake signal created by the adhesive may be processed by the software, and surrounding image information may be intrapolated/extrapolated across any "dead spots."

[00117] Where the optional compliant surface is not used, the finger may come in direct contact with the waveguide 150. As explained above, this may cause the finger to be lit up as a whole, resulting in sub-optimal contrast. To compensate for this adverse effect, the software may process the image in the context of the designed based on fiber optic plate.

Alternatively or in addition, the illuminant and the light sensor may utilize a wavelength that diffuses less well inside the finger. A human finger has a high water content and conducts visible light effectively. Much shorter wavelengths, e.g. around 100 nm ultraviolet UV-C light or much longer wavelength, e.g. around 3000 nm mid-infrared light are absorbed quickly by water and may thus generate a stronger contrast between the valleys and ridges of the fingerprint. The light sensor 26 may be picked accordingly to be sensitive to the respective wavelength that is used.

[00118] FIG. 22 depicts yet another embodiment of the fingerprint-sensing touchscreen device 10 in accordance with the inventive concept. In this embodiment, the display element, in this case the T-OLED 190, is disposed between on the surface of the waveguide 150 that is opposite the first surface. More specifically, in the embodiment of FIG. 22, the T-OLED 190 is disposed between the waveguide 150 and the light sensor 26. There is an air gap 220 between the waveguide 150 and the T-OLED 190 in this particular configuration, unlike in the embodiment of FIG. 19 where the positions of the two elements are reversed. In the embodiment of FIG. 19, the presence of an air gap would blur the image due to parallex effect; however, that is not a concern when the display element is "under" the waveguide 150 from the user's perspective.

[00119] FIG. 23 depicts another embodiment of the fingerprint-sensing touchscreen device 10 in accordance with the inventive concept. In this embodiment, as in the embodiment of FIG. 22, the display element is positioned between the waveguide 150 and the light sensor 26. However, the display element in this embodiment is an LCD panel 200 used in conjunction with a backlight 202 (or a visible light source 29 and a diffuser). As illustrated, T- OLED and LCD can also be placed below the waveguide and coupled with it. However, if the elements inside of the OLED or LCD are less clear, it may be better to place them below the waveguide and not to couple with the waveguide 150.

[00120] The embodiments of FIG. 22 and FIG. 23 may be used with a transfuser as the transparent element instead of the waveguide 150 or the prism 100, as shown in FIG. 24. Where the transparent element is a transfuser, the display element is "under" the transfuser from the user's perspective. For example, the display element (e.g., T-OLED or LCD panel) may be disposed between the transfuser and the light sensor 26. The embodiments that do not include the projector (e.g., the embodiment including the T-OLED) have the benefit of allowing the end device to be thinner - e.g., if the light sensor 26 is replaced with an array of camera or cameras with short focal length. Applications

[00121] The exemplary system can enable a wide range of HCI applications on touchscreens, including access control, logging activity, high degree-of-freedom touch input, and detecting users' finger poses, as well as numerous other applications.

[00122] The system can enable at least three types of novel interactions on tabletops: (1) biometric authentication on every touch, even real-time; (2) access logging; and (3) high degree-of-freedom touch input.

(1) Biometric authentication on a per-touch basis

[00123] Unlike previous interactive systems, the exemplary system extracts fingerprints during interaction, i.e., it does not require users to perform a separate identification step. In particular, the system offers true biometric authentication during interaction and requires no explicit login procedure or other token of identification

(2) Logging object accesses and ensuring permissions

[00124] In a collaborative document sorting application, the system logs which user around the table accesses and interacts with which file during a meeting, including the final layout of documents on the screen. This can, for example, help participants to reflect after the meeting and see which user has brought up which document, which can assist in brainstorming sessions. Examples of multi-user embodiments described above in reference to FIGs. 4B and 4C illustrate how permission/access levels are checked.

(3) High degree-of-freedom touch input

[00125] In some embodiments, an exemplary system can exploit the

expressiveness of the user's hand (e.g.,, by allowing each finger to invoke a different operation). This can allow GUI applications to shrink in size (e.g., by overloading input controls). For example, the system may be able to infer input operations from a particular hand, finger, or part of a finger that touches the screen. For instance, a one-button music player can play music when touched with the index finger, while the middle finger skips to the next track, and the ring finger stops playback. In another example of a drawing application, users of the system can draw with a brush using the index finger, color with the middle, and erase with the ring finger. Alternative Embodiments

[00126] The embodiments described above focus on a single camera covering an entire screen. However, this is not a limitation of the inventive concept. In one embodiment, a camera may cover only a sub-area 61 of the screen 60 (FIG. 26A). In another embodiment, an array of cameras may be used to cover a screen 60, with each camera covering a sub-area of the screen, as shown in FIG. 26B. Other embodiments may include a camera guided to relevant touch locations using a steerable/movable mirror (FIG. 26C), or a moveable camera (FIG. 26D).

[00127] FIGs. 25A, 25B, 25C, and 25D illustrate similar arrangements of the camera system as in FIGs. 26A, 26B, 26C, and 26D except with a display element - transparent element combination device (e.g., embodiments of FIGs. 16A - FIGs. 24) instead of just the transparent element, as in FIG. 26A-D. FIGs. 29A, 29B, 29C, and 29D illustrate similar arrangements of the cameras as above, with a scanner 100.

[00128] In another implementation, the camera(s) in FIGs. 25A-D, FIGs. 26A-D, and FIGs. 29A-D may be replaced with an optical sensor or an optical sensor array.

[00129] The fingerprint-sensing touchscreen device 10 is not limited to being used with a horizontal screen. For example, the fingerprint-sensing touchscreen device 10 may be used with a vertically-positioned screen such as a display wall or a kiosk system, a slanted wall (e.g., in the form factor of a drafting table or a slanted kiosk system), or any other angle of touch surface. The touch surface is not limited to being planar, but maybe curved (e.g., cylindrically or spherically curved) to accommodate its purpose.

[00130] Using appropriately distorting optics for light sensor/camera/scanning and display system and/or curved fiber optic plates, and/or distortion compensation in software, the fingerprint-sensing touchscreen device 10 may work on such curved touch surfaces.

[00131] Additionally, the fingerprint-sensing touchscreen device 10 may use a display screen made from a single fiber optic plate or be a combination of multiple plates "tiled" or glued/fused together. To allow for large installations, a sheet of glass or similar material below may be used to add structural support. [00132] By combining multiple fingerprint-sensing touchscreen devices 10, multi- surface systems may be created to form a single, larger touchscreen, multiple surface of the same orientation, or multiple surfaces of different orientations.

[00133] The thickness of the fingerprint-sensing touchscreen device 10 may be controlled by shortening the optical path of the camera by using stronger lenses. Replacing the DLP projector with a laser projection would allow it to maintain focus. Alternatively, the optical paths between the components of the fingerprint-sensing touchscreen device 10 can be "folded" by use of mirrors.

[00134] Instead of illuminating the entire screen at once, one or more sub-areas may be illuminated at a time. There are many different ways to accomplish this. For example, a mechanical scanner design, i.e., a long thin illuminant + sensor array that swipes across the screen, may be used. Alternatively, a laser may be used to illuminate a single point at a time, obtaining the entire picture by scanning the surface with the illuminant.

[00135] As disclosed above, the projector 32 in the individual projector-based embodiments may be replaced with parts that are inherently flat, such as flat panel displays (e.g., LCD or T-OLED), e.g. placed under the transfuser 22. In addition, the rear-projection may be replaced with a top-projection configuration. Alternatively, the optical sensor 26 and the projector 32 may be replaced with a (high-resolution) sensing optical array. Alternatively, this sensing array could be integrated with the flat display into an in-cell device that senses touch and displays an image, such as the one in FIG. 27. FIG. 27 depicts a mobile device 77 using an in- cell screen to detect touch on the transfuser 22.

[00136] Thin form factors allow implementing the fingerprint-sensing touchscreen device 10 in a wide range of applications that desire mobility and portability, including but not limited to smart phones, media players, tablets, and touchscreen and touch keyboards in laptop computers.

[00137] The authentication function of the inventive concept is described above.

This authentication capability allows select users to access restricted information, such as restricted files, functions, links, etc. even when multiple users have access to the same physical device. As described above, the authentication is done real-time on a touch basis; hence, even when multiple users are accessing the same device at the same time, different levels of access may be granted to different users.

[00138] The touch-based authentication may be used to log usage, e.g., to allow a multi-user learning environment to monitor which user solved the math problem, which of the multiple players solved the puzzle or shot the bad guy, who ordered what from a restaurant, or who placed which bet in a casino/gambling game.

[00139] In addition, the fingerprints extracted by the fingerprint-sensing touchscreen device 10 allow implementing a range of additional functionality. In one embodiment, the fingerprint can be used to reconstruct the three-dimensional position of the finger in space. The reconstructed position, in turn, may be used to implement high-precision touch input by applying corrective offsets depending on finger angles and user ID.

[00140] In another embodiment, fingerprints can be used to distinguish a user's fingers. Different fingers may be assigned different functions. For example, rather than making a selection from a menu or an array of buttons/keys, a user may perform different functions by using a different finger or a different combination of fingers. This may be combined with single and multi-finger gesture input.

[00141] In yet another embodiment, the fingerprint information may be used to personalize the user interface. If multiple users are working on a system at the same time, different regions may be personalized for different users (e.g., a region may be personalized for the closest user).

[00142] When one user is touching, for example, an email icon, the user's email could come up. If a second user is touching that same icon, no email or the second user's email could come up.

[00143] For all embodiments where the sensing and the display subsystems use light of different wavelengths, the light source and the sensor/camera system have be picked so as to have matching frequencies. Furthermore, to prevent interference with the display system, light sources, projectors, and cameras may be complemented with filters. For an embodiment that uses infrared light for the sensing, for example, the sensor may be complemented with an Infrared pass filter, while the projector may be complemented with an infrared filter.

[00144] To prevent the invention from being spoofed using images of fingers or props of fingers an appropriate type of liveliness detection may be added. Approaches using extra hardware include components that measure temperature, pulse, blood pressure, or electric resistance of the object touching the device. Approaches without extra hardware include looking for skin deformation, pores, perspiration, etc. Any of can also be added to the present invention.

[00145] It is also possible to add one or more low resolution sensors for the purpose of obtaining touch recognition in some areas. This can help conserve power, computational effort, and/or hardware effort compared to a solution sensing fingerprints everywhere. A typical combination is to use the low-res sensor to detect touch and then to point the fingerprint sensing system attention to that area. Any of these embodiments may also allow adding additional cameras on the fly as necessary.

[00146]

[00147] Multiple users with different access privileges may work on the same device, as illustrated above in FIGs. 4A, 4B, 4C, and 4D. In the particular case that was illustrated above, a bank clerk collaborated with a bank manager. A similar scenario may include multiple doctors and nurses accessing health records, or multiple military personnel of different ranks gathered around an interactive table in a war room. Possible applications that take advantage of the multi-user interaction with different levels of access privileges, where users with different levels of access can work and interact without interfering with one another's access levels, are numerous.

[00148] The fingerprint-sensing capability of the embodiments disclosed herein may be useful in various entertainment-related applications. For example, in the gambling context, touchscreen games, roulette tables, slot machines, etc. may be operated specifically for the user who is playing. Monetary transactions, including but not limited to payments in both directions, may be set up to automatically happen with the correct player.

[00149] Users may also interact across multiple fingerprint-sensing touchscreen devices 10. Doctors and nurses, for example, may travel between different stations, each of which is equipped with an fingerprint-sensing touchscreen device 10. At each station, a doctor, a nurse, or other hospital staff members and contractors can accomplish what s/he intended without interfering with one another, as the authentication is performed on every touch. [00150] In addition to thinking of the fingerprint-sensing touchscreen device 10 as a touchscreen that can recognize fingerprints, it can also be used as a fingerprint scanner with display capabilities. In an airport or an immigration office, for example, the device 10 could display usage instructions to the user, such as "place left thumb here" with "here" being a displayed shape (see FIG. 28), "move hand to the left" or it could highlight which of several fingers was misrecognized, etc. The fingerprinting capabilities of the fingerprint-sensing touchscreen device 10 can be used to record fingerprints without serving an authentication function.

[00151] While most scenarios discussed above focus on applications where users are authenticated with every touch, the fingerprint-sensing touchscreen device 10 can be used in a more traditional one-off or access authentication, such as for accessing a building or part of a building, a bank, a car, bank accounts, ATMs, online payment or other transactions, restricted contents on television or online, confidential files (e.g., bank accounts, loan documents), voting records, immigration records, lockers, mail, email, will call, online accounts, hospitals and medical records, educational records, password files, remote login, creation of electronic signature, home appliances, or as part of in-classroom educational devices.

[00152] The methods described above may be implemented in a processing device. While the embodiments are described in terms of a method or technique, it should be kept in mind that the disclosure may also cover an article of manufacture that includes a non- transitory computer readable medium on which computer-readable instructions for carrying out embodiments of the method are stored. The computer readable medium may include, for example, semiconductor, magnetic, opto-magnetic, optical, or other forms of computer readable medium for storing computer readable code. Further, the disclosure may also cover apparatuses for practicing embodiments of the invention. Such apparatus may include circuits, dedicated and/or programmable, to carry out operations pertaining to embodiments.

[00153] Examples of such apparatus include a general purpose computer that includes a processor, a memory, and a user interface and/or a dedicated computing device when appropriately programmed and may include a combination of a computer/computing device and dedicated/programmable hardware circuits (such as electrical, mechanical, and/or optical circuits) adapted for the various operations pertaining to the embodiments. [00154] The description is not intended to be exhaustive or to limit the invention to the precise form disclosed. It should be understood that the invention can be practiced with modification and alteration.

Claims

What is claimed is:

1. An apparatus comprising: a transparent element having a first surface configured to receive a finger input; a first light source configured to direct light to the first surface such that a first portion of the light that is incident on an area of the first surface that is covered by a valley of the finger reflects off the first surface and a second portion of the light that is incident on an area of the first surface that is covered by a ridge of the finger is frustrated; an optical sensor optically coupled to the transparent element and configured to capture the light that is reflected off the first surface; and a display element configured to produce a visible image on the transparent element.

2. The apparatus of Claim 1, wherein the optical sensor is a camera.

3. The apparatus of Claim 1, wherein the display element is a transparent organic light emitting diode (T-OLED).

4. The apparatus of Claim 1, wherein the display element is a liquid crystal display (LCD) panel.

5. The apparatus of Claim 4, further comprising a second light source providing visible light to the LCD, wherein the second light source emits light of a different wavelength from the first light source.

6. The apparatus of Claim 1, wherein the transparent element comprises a transfuser that diffuses light during transmission while offering specular reflection.

7. The apparatus of Claim 6, wherein the transparent element is directionally transparent and comprises a fiber-optic bundle that includes fibers extending orthogonally to the first surface.

8. The apparatus of Claim 6, wherein the display element is a projection device that projects the visible image on a second surface of the transparent element.

9. The apparatus of Claim 6, wherein the display element is one of a transparent organic light emitting diode (T-OLED) and a liquid crystal display (LCD) overlapping the transparent element.

10. The apparatus of Claim 6, wherein optical sensor is an in-cell sensor and the transfuser comprises a fiber-optic bundle.

1 1. The apparatus of Claim 1, further comprising a second light source that generates light of a different wavelength from the first light source, wherein both first and second light sources generate light that is either about 100 nm or 3000 nm in wavelength.

12. The apparatus of Claim 1, wherein the optical sensor is configured with a mechanism that allows the light sensor to be pointed at different parts of the transparent element.

13. The apparatus of Claim 1, wherein the first surface is non-planar.

14. The apparatus of Claim 1, wherein the transparent element is a prism with a diffused light source.

15. The apparatus of Claim 14, wherein the display element comprises a diffuser coupled to the prism.

16. The apparatus of Claim 1, wherein the light sensor captures the reflected light simultaneously as the image is displayed on the display element.

17. The apparatus of Claim 1, wherein the transparent element comprises a first region that is configured to either display an image or sense fingerprint but not both.

18. The apparatus of Claim 1 further comprising a clear liquid between the transparent element and the display element, the clear liquid having an index of refraction close to the transparent element and the display element.

19. The apparatus of Claim 1, wherein the display element is on a side of the transparent element that is farthest away from the first surface, and wherein the display element is spaced apart from the transparent element.

20. The apparatus of Claim 1, wherein the transparent element is disposed between the display element and the optical sensor.