WO2005018129A2

WO2005018129A2 - Improved gesture recognition for pointing devices

Info

Publication number: WO2005018129A2
Application number: PCT/IB2004/051418
Authority: WO
Inventors: Victor Marten; Aakar Patel
Original assignee: Semtech Corporation
Priority date: 2003-08-15
Filing date: 2004-08-06
Publication date: 2005-02-24
Also published as: WO2005018129A3

Abstract

A method and algorithm for determination of gestures on pointing devices is disclosed. The invention provides for means to make the probability of erroneous operations arbitrarily small and variable, suitable to the present task at hand. The invention is applicable to a variety of different types of pointing devices. Historical data are maintained regarding the performance of gestures by a human operator, and are analyzed to develop heuristic thresholds distinguishing whether or not the gesture occurred.

Description

Description IMPROVED GESTURE RECOGNITION FOR POINTING DEVICES

[1] This application claims priority from US appl. no. 60/481,240, filed August 15, 2003, which application is incorporated herein by reference for all purposes. Background of Invention

[2] Modern electronic devices often make use of a cursor seen on the display, or control some parameter or position of a mechanical actuator, by means of a pointing device. One or more buttons or switches are usually associated with a pointing device, controlling the required actions once the cursor or actuator is positioned in an appropriate location. However, for the users, it is highly advantageous not to employ such buttons or switches, and affect an appropriate action utilizing the pointing device itself. This is desirable since the controlling digit or limb is already acting on the pointing device, and does not need to be moved to the button or switch (so that the operation can be conducted without the user actually looking at the pointing device), or the second digit or appendage does not need to come into play.

[3] Numerous previous art apparatus and patents describe means for achieving such functionality utilizing variety of pointing devices. However, each method is typically applicable only to a specific pointing device type, and is usually sensitive to a multiplicity of "noise" sources, be it internal electrical noise, or aberrations in the functioning of the human digits or limbs. Summary of Invention

[4] The objective of the current invention is to provide a robust method and an algorithm for determination of various gestures carried out on the pointing device, simultaneously applicable to a multiplicity of the pointing device types.

[5] For example, gesture determination on a touch pad requires timely and reliable information that the touch event is taking place and the duration of such touch event.

[6] According to the current invention, the desired resistance to noise sources (e.g. probability of "false positive" activations and probability of "false negative" behavior) can be readily adjusted and can be made variable in time, if different degrees of noise resistance are required at different moments in time.

[7] The determination of a gesture is made using past history of the data, with the amount of data (length of observation) suitable to the desired resistance to noise. Variable threshold levels are utilized when the higher/lower type of comparisons are made. The values of the thresholds are determined and constantly adjusted in time based on the amount of noise (electrical, mechanical or human behavior), expected in the future and observed in the past (for an appropriate period of time and including the present conditions).

[8] The algorithm can be further characterized, using the language of mathematics, as utilizing the cross-correlations of the parts of the data itself, and correlations of the data to variable templates based on the previous history of the data, human motor abilities, and specific mechanical configuration of the particular pointing device.

[9] Data from all mutually independent sources (for example X-position, Y-position, X-force, Y-force, Z-force, duration of Z-force event, Z-proximity, actions on other controls and/or keys/keyboard of the host device) are accumulated and factor into the final decision in order to reduce the possibility of erroneous and/or undesirable operation.

[10] The invention provides for means to prevent erroneous motion of the cursor or actuator while the determination of the gesture is taking place.

[11] The burden of logical and arithmetical calculations can be arbitrarily divided between the local controller of the pointing device and the driver processing the data from the pointing device, which is running on the host apparatus (if such exists).

[12] An arbitrarily small subset of the method and algorithm of the invention can be utilized, suitable to the operating conditions, specific mechanical configuration and desired functionality of a particular pointing device. In some of such cases, operations may be equivalent to the previously described in prior art and/or patents.

[13] Brief Description of Drawings

[14] Figure 1 shows a representation of information flow in the pointing device.

[15] Figure 2 is a depiction of Probability Density Function ("PDF" or "Bell Curve") and Determination of Threshold Value.

[16] Figure 3 illustrates Typical Gesture on Touch Pad Hold.

[17] Figure 4 illustrates Typical Gesture on Touch Pad Click.

[18] Figure 5 illustrates Typical Gesture on Touch Pad Double Click.

[ 19] Figure 6 illustrates Typical Gesture on Touch Pad Click and Hold. Detailed Description

[20] Any variable that requires a threshold to detect activation or de-activation (such as Z-force or Z-proximity for detection of touch, for example) is processed and a histogram (a representation of Probability Density Function or PDF) is accumulated. The data in the histogram includes past history and the current conditions. As the time progresses, some old data is removed from the histogram, and new data is added. These operations are carried out using an optimized algorithm that does not require recalculation of the complete array of data.

[21] Looking at Figure 2, once the median value 21 (V ) is determined, several values 22 around V are used to determine the shape of the Probability Distribution (with the Normal or Gaussian Distribution being the worst case). A threshold value 23 (V ) is then determined based on the acceptable error probability as described below.

[22] The total area under the PDF curve 20 includes all possible values of the variable and thus represents a probability of 1 (100%). A portion of the area 24 represents the probability of erroneous activation for a given threshold 23 (V ) for an arbitrarily selected case when the activation event requires the variable to exceed the threshold. The same considerations apply when the desired activation will result from the variable being smaller than the threshold, except the threshold value will probably be located to the left of the median value V o .

[23] Using Gaussian Distribution as the worst possible case, a value for threshold 23 can be calculated or selected from a table of values such that the probability of error is always smaller than or equal to the error represented by area 24.

[24] When appropriate, the determination of the PDF may also be done during a training session when the user sets various parameters according to personal abilities, and the system learns the user's behavior.

[25] For example, looking at Figure 3 through Figure 6, a typical gesture on a touch pad may require setting of value thresholds for parameters Z-force and Z-proximity (which are used for creation of logical variable "Processed Touch Event") and time thresholds for time intervals designated 40, 41, 50, 51, 52, 60 and 61. The time thresholds may at first be set to default values, then they may be further refined during a training session, and then they may be even further refined during the actual operations of a pointing device. The algorithm for refinement is the same as described above (building of a histogram, checking the shape of the distribution, and calculating a threshold according to the required probability of error).

[26] It will be appreciated that the benefits of the invention offer themselves with any of a multitude of types of pointing devices, among them force sticks, movable joy-sticks, FSR (force sensor resistance), and touch pads. Touch pad technologies with which the invention can offer its benefits include capacitive, resistive, infrared, or any other technology.

[27] Those skilled in the art will have no difficulty whatsoever devising myriad obvious variants and improvements of the invention, all of which are intended to be encompassed within the claims which follow.

Claims

[I] A method for use in recognizing a gesture made by a human at a human-input pointing device, the method comprising the steps of: performing the gesture a plurality of times; for each performance of the gesture, collecting items of data from the pointing device regarding the gesture; maintaining a historical record of the items of data; selecting one or more thresholds relating to the items of data from the historical record; and detecting the gesture by employing the one or more thresholds with respect to values detected at the pointing device.

[2] The method of claim 1 further comprising the step of deleting old data from the historical record.

[3] The method of claim 1 wherein the historical record comprises accumulated data from mutually independent sources selected from the set consisting of X- position, Y-position, X-force, Y-force, Z-force, duration of Z-force-event, and Z- proximity.

[4] The method of claim 1 wherein the historical record is a histogram.

[5] The method of claim 4 wherein the histogram is a probability distribution function.

[6] The method of claim 1 comprising extracting from the historical record a median value, analyzing several values around the median value to determine the shape of a probability distribution function, and selecting the threshold value based upon a predetermined acceptable error probability.

[7] The method of claim 1 wherein the gesture is "hold".

[8] The method of claim 1 wherein the gesture is "click".

[9] The method of claim 1 wherein the gesture is "double click".

[10] The method of claim 1 wherein the gesture is "click and hold".

[I I] The method of claim 1 wherein the performance of the gesture a plurality of times comprises a training session in which the user's behavior is learned.

[12] The method of claim 1 wherein the performance of the gesture a plurality of times comprises performing the gesture during the actual operations of a pointing device, and wherein the one or more thresholds are refined thereby.

[13] The method of claim 1 wherein the human-input pointing device is a force stick, a movable joy-stick, an FSR (force sensor resistance) device, or a touch pad.

[14] The method of claim 13 wherein the human-input pointing device is a touch pad and wherein the touch pad is capacitive, resistive, or infrared.

[15] Apparatus for use in recognizing a gesture made by a human at a human-input pointing device, the apparatus comprising: means responsive to performing the gesture a plurality of times for collecting items of data from the pointing device regarding the gesture; means responsive to the collecting items of data for maintaining a historical record of the items of data; means selecting one or more thresholds relating to the items of data from the historical record; and means detecting the gesture by employing the one or more thresholds with respect to values detected at the pointing device.

[16] The apparatus of claim 15 wherein the means responsive to the collecting items of data for maintaining a historical record of the items of data deletes old data from the historical record.

[17] The apparatus of claim 15 wherein the historical record comprises accumulated data from mutually independent sources selected from the set consisting of X- position, Y-position, X-force, Y-force, Z-force, duration of Z-force-event, and Z- proximity.

[ 18] The apparatus of claim 15 wherein the means selecting one or more thresholds relating to the items of data from the historical record extracts from the historical record a median value, analyzes several values around the median value to determine the shape of a probability distribution function, and selects the threshold value based upon a predetermined acceptable error probability.

[19] The apparatus of claim 15 wherein the gesture is "hold".

[20] The apparatus of claim 15 wherein the gesture is "click".

[21] The apparatus of claim 15 wherein the gesture is "double click".

[22] The apparatus of claim 15 wherein the gesture is "click and hold".

[23] The apparatus of claim 15 wherein the first, second, third, and fourth means comprise a local controller of the pointing device or a driver processing data from the pointing device, or a combination thereof.

[24] The apparatus of claim 15 wherein the first, second, third, and fourth means comprise a local controller of the pointing device.

[25] The apparatus of claim 15 wherein the pointing device is a touch pad.

[26] The apparatus of claim 25 wherein the touch pad is capacitive, resistive, or infrared. [27] The apparatus of claim 15 wherein the pointing device is a force stick, a movable joy-stick, an FSR (force sensor resistance) device, or a touch pad.