WO2002008881A2

WO2002008881A2 - Human-computer interface

Info

Publication number: WO2002008881A2
Application number: PCT/GB2001/003153
Authority: WO
Inventors: Christopher Worthington; Ian Duncan Campbell Andrew
Original assignee: Qinetiq Limited
Priority date: 2000-07-21
Filing date: 2001-07-19
Publication date: 2002-01-31
Also published as: GB0017793D0; AU7080801A; WO2002008881A3; AU2001270808A1

Abstract

A multimedia human-computer interface includes a position sensitive panel and a display wherein gestures drawn onto the panel by a user are interpreted by gesture processing software, and, according to the gesture input and the context in which it is used, the operating mode of software for the manipulation of images is affected. Typically the panel will be coincident with the display. A programmable library of gestures is stored in a compute system, each gesture corresponding to a particular software function. Gestures may be programmed or reprogrammed to control any suitable operation of the software. The invention provides for the switching of operating mode, as well as the control of parameters associated with a given operating mode. The invention also provides for speech recognition techniques to be combined with the gesture recognition to enhance the functionality still further.

Description

Human - Computer Interface.

This invention relates to a multimedia human-computer interface, and also to a method of quickly inputting information into a computer. More specifically, it relates to a means of entering commands and data into a computer system using gestures applied to a panel. The invention also allows the operator to use speech to enter commands into the computer. The invention provides an interface between the operator and software running on the computer system, the software generally being connected with image analysis or manipulation. However, any programs having a requirement for selecting and manipulating data displayed on a screen may benefit from use of the invention. Text selection and manipulation may also be performed.

Background And Prior Art

It is an aim of the designer of human - computer interfaces to make them as easy to use as possible. Differing requirements of different users, and applications of computers in very different environments have made the problem even greater. As well as on the office desktop, computers are being used in such diverse areas as operating theatres, factories, etc and by mobile users, with laptop, palmtop and handheld computers, and by people in the home sending messages whilst sat on the settee watching television. All these people have particular requirements for entering data into the computer. Some of these users may have a particular need to concentrate particularly hard on the computer display for some reason, and will find it an inconvenience to look away from the image presented to find a keyboard or mouse. This invention will be of particular use to these people.

It is the job of the image analyst to gather information from an image, and compile a report on it The analyst uses a computer in this task, viewing a digital representation of the image on the computer display. In carrying out their task there may be a great deal of manipulating of the image to be done. For example, the analyst will need to zoom in on areas to see them in more detail, pan around the image if it is too large to fit on the display device, select localised areas for image processing, such as inverting or filtering, move or copy areas to compare against each other or save as reference images, rotate areas, or perform any other function as may be necessary.

Modern computer systems usually have attached, as a means for inputting information and performing this manipulation, a keyboard and a mouse. These tools are fine for many tasks, and work well, but for some tasks other methods of interacting with the computer may be preferable. When working with images on a computer screen the keyboard can be very awkward. The mouse solves many of the problems associated with image manipulation, but introduces problems of its own. It is only when performing a great number of image manipulation tasks that these problems come to light. For example, when it is required to move an object from point A on the screen to point B using the mouse, there are a great many steps to go through, when the problem is broken down. These are:

Locate mouse on desk

Move arm to locate hand over mouse

Grasp mouse Move mouse to visually locate pointer on screen

By moving mouse, move pointer to locate it on object at A

Locate index finger on mouse button

Press mouse button to select object at A

Hold mouse button down to drag object Move mouse to move pointer, dragging object, near to B (ballistic movement)

Move mouse to move pointer to adjust position (fine) of object on to B

Release mouse button to drop object at B

Many of these tasks are trivial in nature, but when repeated often, they can get tiring, and lead to stress for the operator. Some tasks, such as image analysis or image manipulation may involve a great deal of time spent working in this way, staring at the screen and highlighting areas and moving them etc. One of the key aspects of the job of an image analyst is that, when manipulating the images, it is very inconvenient if they need to look away from the screen for ancillary purposes. This disturbs their concentration, and makes the task more of a chore.

One method of solving this problem is to use speech, as discussed in US patent 5600765. Here, the user can designate features on a display using touch and gesture, and, by speaking commands and indicating on the display, can enlarge, or move etc the item selected. Using speech alone for the inputting of commands can become wearisome and in high noise environments the speech recognition system can become confused by the speech input. This system uses gesture input for selecting and designating areas only, and speech only for the issuing of commands. This can be restrictive.

The current invention solves these problems, and alleviates the stress to the operator caused by the mouse and other operations discussed above. It aims to reduce the number of small interactions that the operator needs to do when manipulating images and compiling reports on the images. It aims to provide a quicker, more intuitive interface that is easier to learn and faster to operate.

Statements Of Invention And Advantages

According to the present invention there is provided a multimedia human computer interface that is useable on a computer that has installed software for enabling an operator the ability to directly manipulate images, comprising at least a position sensitive panel and a display, characterised in that one or more gestures applied anywhere on the panel are interpreted by gesture processing software according to the gesture made and are used to control operating modes of the image manipulation software.

Note that where this specification refers to the manipulation of images, it will be understood that an image can also be an image of textual information. Thus, unless stated otherwise, use of the word "image" covers both pictures and text displayed on a screen. The specification also refers to touch sensitive panels, and position sensitive panels. It will be understood that these terms refer to any panel that is capable of detecting the position of some pointer on, or adjacent, to it, and the terms "touch sensitive" and "position sensitive" should be regarded as equivalent.

The current invention allows the operator to select areas on the screen, and to process those areas with only minimal movement of his gaze away from the area of the screen upon which he is currently working. The invention also provides the operator with a choice of media in which to interface to the computer. Thus the operator is able to concentrate for longer periods of time on the main purpose of the task, and not have to waste time and effort on other tasks such as those mentioned above.

There are two categories of gesture that can be used with this invention. The first is known as a mode change gesture. This is used to switch the operating mode of the software from one to another. For example, if the display software is currently in zoom mode, such that it is being used to control the magnification of a portion of the image, and it was then switched into pan mode, this is called a mode change. A gesture used to do this is a mode change gesture. When in a particular mode, any gestures that are used to operate on a parameter of that mode are called mode operation gestures. This is the second type of gesture. Thus a gesture that, for example, selects the zoom factor, altering the magnification of the image, is a mode operation gesture. «...

A mode change gesture, in the context of this specification, can consist of placing one or more pointers, which can be the operator's fingers, in contact with the panel, and then moving either some, or all of the pointers. The movement can either substantially maintain contact with the panel, or it can be off from the panel. The latter case signifies that whatever action is being performed is now completed. This type of gesture can be thought of as having an equivalent effect to the menu items on more traditional human- computer interfaces, such as the Windows operating system from Microsoft^® but which of course are chosen in a completely different manner. A mode operation gesture can differ from a mode change gesture, in that it does not necessarily finish once the pointer or pointers are removed from the touch panel. For example, duririg a move operation, a gesture incorporating multiple pointers that is used to simulate picking up an area of the image, moving it, and putting it down elsewhere on the image can lead to the pointers losing contact with the screen during the move phase of this operation. See below for more information on use of multiple pointers.

Mode change gestures may be pre-set, or they may be changed by the user at will. Also, the gesture used to activate a particular function may be assigned by the user according to their preference. Different gestures can be programmed into the computer system by means of a training function. A library of currently programmed mode change gestures can then be created that holds samples of each gesture Here, a user would make several examples of the gesture currently being programmed. All would vary slightly from each other due to an inability to exactly reproduce a previous tracing on the touch panel. However, the gestures would be substantially alike. These gesture inputs would then be stored as templates. Any gesture then input after this training had been completed would then go to a gesture comparison engine, comprising a pattern matching algorithm. If the gesture were substantially similar, it would be able to recognise it as that trained gesture. The function currently assigned to that gesture could then be activated. Spatial information, i.e. the pattern of the reference gestures is stored in the library. Temporal information, relating to the time taken to draw the gesture can also be stored. This allows the software to differentiate between a mode change gesture input, and a mode operation gesture (in cases where the context of the input required does not define the interpretation of the gesture), so long as the latter does not share the same, or substantially similar temporal parameters. The only limitation on use of a mode change gesture is that it must be able to be drawn on the touch panel without removing the pointer, or all pointers if more than one is used, from the screen. The prior art interfaces have menu options on the screen in set places. Even though the user may be able to alter where these places are, by moving menu bars around etc. the user still has to locate the place where he has put the menu item on the screen each time he wishes to use it. The current invention allows the user to draw mode change gestures anywhere on the touch panel. This provides the benefit that he does not need to look away from the particular area of the image he is currently working on to locate a menu option

There are various methods that can be used to compare a gesture input to the system with those stored in the library. These include basic methods like Fourier transform matching, or correlation algorithms, both of which are established methods of pattern matching, and will be familiar to those skilled in the relevant art. Other techniques can be used that look at various parameters of the gesture. Details of the function of these is beyond the scope of this specification, but for more information on suitable algorithms see Rubine, D, "Specifying gestures by example", Computer Graphics, 329-327 ACM SIGGRAPH, Addison Wesley, July 1991. These methods will look at some or all of the parameters that go to making up a particular gesture, and, comparing these against a reference gesture, will come up with a measure of the similarity of the two gestures. This can be represented as a certainty value of the two gestures being a. match. A threshold can be set for the certainty of the match, in percentage terms, and if two gestures are compared and have a certainty value over this given threshold then they will be regarded being a match.

Before this invention, the operator was able to select areas on the screen using a touch sensitive display, but when he wanted to, say, move a selected area, he had to somehow select the operating mode that allowed items to be moved. This normally involved removing one's gaze from the selected area to look for the button or control that activated the move operating mode, or function, or speaking the command into the computer's microphone. The present invention enables, once the area has been selected, the move function to be chosen by means of the operator making a gesture on the screen that represents selecting the move function. Similarly, if the operator wants to copy the selected area, he will make a gesture that activated the copy function of the software. The number of functions that can be so represented is limited only by the number of unique, easily drawn gestures that the operator can devise. Functions that may advantageously be controlled by the present invention include, but are not limited to,

- Move and copy functions as described above

- Rotate functions, whereby the selected area can be rotated at an angle around a point.

- Pan, whereby when only a small part of an image is visible in some viewing window, that part actually visible may be changed, such that any part of the complete image may be viewed.

- Zoom, whereby some selected area can be expanded in size, such that more detail is visible.

- Select, used for the selection, or confirmation of a highlighted area on the screen.

- Image processing functions, such as filtering, inverting, intensifying etc

- Image annotation functions, such as adding labels to areas, or adding shapes such as rectangles or circles.

Note that the term "operating mode" and the noun "function" and their derivatives are used interchangeably throughout this specification.

Labels can be placed on to areas and defined as markers, that act like a bookmark in a book, such that the areas can be accessed by reference to the "bookmark" label.

Note that the starting point used when drawing a gesture can be used to make that gesture unique. For example, if a gesture consisted of drawing a horizontal line on the panel, then the computer system can detect whether that line is drawn from left to right, or from right to left. Depending on the one detected, one of two separate functions can be chosen. This can be achieved by examination of the temporal characteristics of the gesture. The start point will clearly have a timestamp earlier than that of the end point. This can also be achieved merely by looking at the co-ordinates of the points in the array data structure used to store the gesture information. The first co-ordinate in the array will obviously be the starting point, and the last co-ordinate will be the end point.

As stated above, gestures can be made with more than one pointer at the same time. For example, assuming an area of the image has been selected, and the move command chosen, two pointers can be placed on opposite sides of this area and this area moved by moving the two pointers in a similar manner to picking up a normal object between one's fingers and moving it. This method can also be used for copying, rotating, and other actions that may beneficially employ multiple pointers.

The invention may be made even more flexible and easy to use if a speech recognition system is integrated in with the computer system. Using this, there is the option of inputting commands into the system by means of speech as well as gesture. Another benefit of a speech recognition system is that it may be used for the generation of report information spoken by the operator. Whilst analysing the image the operator can speak his findings into a microphone coupled to the speech recognition system, and the verbal report of the operator can be transcribed into a computer file.

The use of speech recognition brings extra functionality to the system. Some functions have more than one parameter that may be adjusted, and speech can be used to select the parameter. For example, if it is required to filter an area to highlight the edges of shapes on the image, the area could first be selected using a mode operation gesture (assuming the system is in "Select" mode), and the "Filter" option then chosen using a mode change gesture while simultaneously saying the word "Edge" into the microphone. In this way, the computer system knows that the edge detection filter is the one required rather than any other filter that may be applied.

As another example, if the user wishes to pan across the image to bring another part of it into view, then, once the pan function has been selected using a mode change gesture, the operator can say the words "lat", or "latitude", followed by a reference number, and do the same for longitude, and the speech recognition software would convert this data into digital form. This would then be sent to the image manipulation software which would display the correct part of the image. As one alternative to this, the user, after selecting pan mode, could say the word "bookmark" followed by the name of a previously stored bookmark, as described above, and the image would then pan to the area assigned to that bookmark.

In this way, the number of gestures that the user is required to learn is reduced, as the required action is chosen by a combination of gesture and voice, not just gesture alone. Of course, many functions or sub functions of the software can be controlled in such a manner, not just the ones referred to above. The addition of speech adds an even more natural way of interfacing with the computer system and software than just gesture alone.

There are several suppliers of speech recognition technology for computer systems that are suitable for implementing the current invention. The ViaVoice system from IBM consists of a speech recognition program that takes it's input from the sound interface card found in most modern computer systems. Dragon Systems also produce speech recognition systems that can be used to implement the speech interface of the current invention.

As another aspect of the invention, the speech recognition system is used to generate written reports on the image displayed by the computer. An example of this is now described. After selecting an area using a mode operation gesture as described above, the user draws a mode control gesture pre-programmed to invoke the report generator. As an area has been selected, the report generator will know that the report is to cover just that selected area. It will note the co-ordinates of the area and store these alongside the report itself. Once the report generator has been invoked, it will record all speech input in the report until a gesture is given indicating that the report is complete. If an area had not been selected before the report generator had been invoked, then the software would know that the report is to apply to the whole image. As another aspect of the invention, the mode change operation can be performed by means of touching menu options that can be shown on the display. To avoid cluttering up the display with such items, it is an aim of the current invention to minimise the number of items that will interfere with the image being viewed. To achieve this, the menu system operates thus: Initially, the main options, such as move, copy, zoom etc are shown, as buttons on the display. When one of these is chosen, the buttons are removed from the display, and replaced by buttons representing any sub options for the function selected, plus an option to go up the hierarchy of the menu structure. When one of these is chosen, these menu options are in turn replaced by any further options lower down in the structure. In this manner, superfluous options are not left on the display taking up space.

According to another aspect of the present invention there is provided a method of inputting information to a computer system having a position sensitive display characterised in that a user draws a gesture on the display; the gesture is digitised, and this digital representation is compared against other gestures stored in a library of gestures, where each gesture in the library is associated with a software function; the software function is carried out that is associated with the gesture in the library that matches the gesture drawn by the user.

According to another aspect of the present invention there is provided a computer program product that can be used with a computer system having a display, a position sensitive panel, and a sound digitiser, the computer program product comprising: a gesture recogniser and processor to detect and match, with pre- stored gestures stored in a gesture library, any gestures made on the position sensitive panel; a command processor that takes in commands from the gesture recogniser; image manipulation software that displays an image on the display and has manipulation functionality; whereby the gesture processor, on detecting a gesture that matches one in the library, sends a command associated with the library gesture to the command handler, which in turn sends it to the image manipulation software.

Drawings Examples of the current invention will now be described with reference to the accompanying illustrative drawings.

Figure 1 shows, in block diagrammatic form, an example of a computer system upon which the current invention can be implemented.

Figure 2 shows three examples of mode change gestures, that may be used to control program functionality.

Figure 3 shows one of the gestures of Figure 2 that has been annotated with timestamp information.

Figure 4 shows both a single, and multiple fingers being used to move an object.

Figure 5 shows multiple fingers being used to rotate an object.

Figure 6 shows the hierarchical menu structure as incorporated in the current embodiment of the invention.

Figure 7 shows a high level block diagram indicating the connectivity of the software modules used to implement the invention.

Figure 8 shows in more detail the software components that go to make up the Command Handler as shown in Figure 7 Figure 9 shows a . user drawing on the touch panel, using his finger as the pointer.

Detailed Description of the Drawings

Figure 1 shows the hardware items upon which the invention is currently implemented. A display device (liyama Prolite) 1 is connected to a processing unit 3 which additionally contains a sound processing card 4, a microphone 5 and the software to implement speech recognition (IBM ViaVoice 98) on the words spoken into the microphone 5. The display device 1 is overlaid with a thin panel that is touch sensitive, such that when it is touched by a pointer 2 of some type, it transmits the position on the display device 1 that was contacted back to the gesture recogniser software. This is in communication with software controlling the displayed image on the display device, and hence the display software will know what portion of the displayed image was nearest to the point on the display being touched.

Figure 2 provides some examples of gesture traces that may be used as mode change gestures. They can all be drawn without removing the pointer from the touch panel. Note that in practice, there would be no need for the computer to actually display the trace of the gesture on the screen, except during the.gesture training phase. Seen as drawn in the figure, there is no information present that identifies the start and end points of the gestures. However, the computer stores the set of x-y reference points that make up the gesture in an array structure, a functionally contiguous set of memory locations that hold the information. The first location holds the first point recorded, the next holds the second, and so on. By means of this, it is possible to see where the gesture started and finished. Additional information can be obtained if each point making up the gesture is given a timestamp, recording the time at which it was made. Figure 3a shows a similar gesture to that in Figure 2b, that has been annotated with simulated timestamp information. The time stamp recordings shown on Figure 3a show the point on the gesture that is drawn at the indicated time interval. Here, fixed intervals of tmtervai = 0.0625 seconds are shown. It can be seen that the spatial distance between_. timestamps is shorter at the curved area of the gesture, indicating that the gesture was drawn more slowly at that point. As said above, this information can be used to distinguish between this gesture, and a similar looking one that has been drawn in a different manner. For example, a gesture that is similar in shape to Figure 3a, but where all points are half the distance apart as those shown in the diagram will be regarded as a different gesture, as it would take twice as long to draw.

Figure 4 shows a user operating the system using both single and multiple pointer mode operation gestures. Figure 4a shows a user moving an area 6 on the display 1 using a single pointer gesture. Once the area 6 is defined, and the system is in "move" mode, the user places the pointer 2 - his finger in this case - on the area 6, and moves it to where he wants the image in that area to be. The defined area 6 follows the position of his finger. The position of the area 6 after the move is shown as 6'. The movement is indicated by the arrow 7. Figure 4b shows the same operation taking place using multiple pointer mode operation gestures. These are beneficial in some instances. When moving or copying areas of the display, once the desired area has been selected, pointers placed substantially at opposite sides of the area, and then dragged to some other area of the display, will move or copy that selected area with them. Figure 4b shows someone using their thumb 2a and forefinger 2b to "grasp" onto a selected area 6. They have put their fingers at opposite sides of the selected area, and the larger pair of arrows 8 indicate the next motion is to move them in towards each other in a pinching action, to simulate the grasping of an object. This pinching action is detected by the gesture recogniser as a multi-pointer gesture, and it knows, when the pointers 2a, 2b are lifted from the touch panel 1 , that the action is not yet complete. The gesture recogniser waits for the two pointers 2a, 2b to be placed back down on the touch panel 1. When it detects this, it knows that the position touched is the destination of the move or copy command action. The thick black arrow 9 shows the movement of the area. The new position of the area 6 is indicated as 6'. The small pair of arrows 10 indicate the movement of the fingers away from the area 6', indicating the completion of the move.

Rotation of areas of the image also lends itself to the use of multi-pointer gestures. Grasping a selected area 6 as described above, followed by twisting the points on the touch panel as indicated by the arrows 11 will result in the selected area 6 rotating with the pointers. Figure 5a shows this function before the rotation, and Figure 5b shows it completed, with the area 6 having been rotated by some angle.

Figure 6 shows the hierarchical menu system, with the Main Menu box showing all top level menu selections available. Some of these options, the File, the Annotate, the Region and the Adjust options have their own subcommands that effectively sit at a layer down in the hierarchy. When one of these is chosen, using voice, gesture or the on-screen menu options, the Main Menu on-screen menu options will be replaced by the sub-commands from the chosen option. Note that each of the sub-menu options have a command to go back to the menu one level up the hierarchy.

Figure 7 is a representation of the major software modules that make up the system as currently implemented. The heart of the system is the Command Handler This takes the output of the gesture recogniser module, the speech recogniser,, and the menu items shown on the display, interprets this information, and issues the commands to the image display and manipulation (IDM) module. The IDM module then processes the displayed image according to the command received. The gesture recogniser is shown as part of the IDM module because certain gestures drawn onto the touch sensitive panel are interpreted directly by the IDM module and are not passed to the Command Handler. These gestures perform functions that interface directly to the operating system, such as opening, closing and repositioning windows on the display - functions that are not directly connected with the image manipulation software. Other gestures are passed to the Command Handler. They are separated out by the gesture recogniser according to the context in which they were entered. Once the IDM module is executed, gestures drawn within the image displayed by the module will be sent to the Command Handler. In normal use, the image displayed by the IDM module will be maximised to fill the entire display area, so in this case all gestures would be sent to the Command Handler.

It will be noticed in Figure 7 that the Command Handler also communicates with the speech recogniser and the menu display modules. This is because, according to the state the IDM module is in, some commands may not be relevant, as described above. The menu system will be redrawn on the display to reflect this. Also, depending on the IDM module state, certain speech commands may not be relevant, so the speech recognition process can be made more reliable by cutting down on the vocabulary active at any one point. This means that each word or phrase input to the speech recogniser needs be compared against fewer reference words, leading to an improved recognition rate.

Figure 8 shows in greater detail the elements that make up the Command Handler. Within the command handler box in the Figure are shown five local handler units and a command demultiplexer. It is irrelevant to the local handler units how a particular command is input to the system. Once a command is received by a local handler unit, that command is examined as to whether, judged from the current state of the system, that command is appropriate. For example, say the last command to be received was the File Menu. The next valid command can only be one of Up, Open or Exit, as seen from Figure 6. If any other command is received, say by voice or gesture, the Menu local handler unit will ignore it. Each local handler knows what commands are valid at any one time. If a valid command is received, it will be sent to the IDM module to be processed.

It will be noted that the gesture and speech recogniser outputs go via the command demultiplexer, whereas the menu inputs do not. The gesture and speech recognisers send all validly received commands to the demultiplexer, which then examines each one and distributes it to the appropriate local handler unit. Each local handler unit, on the other hand, is responsible for deciding which menus are to be displayed, and so only displays items that are to be handled by it

Figure 9 shows a user manipulating an image using a computer equipped to run the current invention. A typical session might go as follows. Once the computer hardware and software is fully set up and working, the user will call up an image that he wishes to analyse, which will be shown on the screen. He may then wish to zoom in on a particular area, to examine it more closely. To do this, he needs to switch the operating mode to "zoom". Without taking his eye from the area of interest, he then draws the mode change gesture on any part of the screen to select the zoom function. The operator may then place his finger on the screen and, maintaining contact, slide his finger either up or down. Moving down will decrease magnification, and moving up will increase magnification. The operator may also speak the zoom factor required, if he knows the exact zoom ratio required. Thus, whilst in zoom mode he may say "200%", and the system will know to expand the current visible image to a size of 200% of the original.

The currently implemented system is set up so that once a command has been executed, such as the zoom mode described above, the image manipulation software switched back to pan mode. This can be regarded as a "standby" mode. This is done because the application to which the invention is currently applied is most often used in pan mode. Thus it is more convenient for the user if this mode is entered after each command. Other applications of the invention may have another mode that is entered as the "standby" mode, or they may simply remain in a given mode until the user changes the mode to another.

When the image is displayed at the correct magnification, the operator may wish to pan across to another area of the image. As stated in the above paragraph, in the current implementation pan mode will be entered automatically when the zoom function is done. The image may then be panned by placing the finger on the screen and moving it. The software detects this movement and moves the image to try to keep the same image point underneath the operator's finger. To switch from pan mode to another mode, the user then touches one of the command options shown on the display. If the menu of command options is not currently displayed, then speaking the word "menu" will show them.

An alternative method of switching out of pan mode is to use a mode change gesture. Of course, there must be a way of differentiating between a mode change gesture and the act of panning itself. One way of doing this is as follows. The mode change gesture will be done quickly and decisively, and the operator's finger will be immediately released from the screen when it has been completed. A pan action, even if it coincides with the shape of a mode change gesture will still be interpreted as a pan action if the operator draws the gesture in a manner differently to how he draws the mode change gesture - if he draws it much slower, or pauses half way through drawing it, for example. Analysis of the gesture's temporal parameters provides the information needed to make this decision.

Claims

1. A multimedia human computer interface that is useable on a computer that has installed software for enabling an operator the ability to directly manipulate images, comprising at least a position sensitive panel and a display, characterised in that one or more gestures applied anywhere on the panel are interpreted by gesture processing software according to the gesture made and are used to control operating modes of the image manipulation software.

2. A multimedia human computer interface as claimed in claim 1 where the position sensitive panel is applied to the display.

3. A multimedia human computer interface as claimed in claims 1 or 2 whereby the computer has installed a speech recognition system that comprises a microphone, a digitiser, and speech recognition software that is in communication with the gesture processing software and image manipulation software.

4. A multimedia human computer interface as claimed in claim 3 whereby a phrase of one or more words recognised by the speech recogniser is interpreted according to the phrase recognised and is useable to control operating modes of the image manipulation software.

5. A multimedia human computer interface as claimed in claim 3 whereby following a given gesture the output of the speech recogniser is storable in the computer system.

6. A multimedia human computer interface as claimed in claims 1 to 5, whereby a library of reference gestures is stored in the computer, and an operating mode of the software is linked to each gesture.

7. A multimedia human computer interface as claimed in claim 6 whereby a time profile map s producible for each gesture in the library, showing the time taken to draw each part of the gesture.

8. A multimedia human computer interface as claimed in claim 6 or claim 7 whereby a gesture input to the system is compared to each reference gesture stored in the library and if a match is found to one of these, then it is regarded by the software as equivalent.

9 A multimedia human computer interface as claimed in claim 8 whereby a gesture is only regarded as a match to a reference gesture if it is drawn at a rate substantially similar to that of the reference gesture.

10. A multimedia human computer interface as claimed in claim 8 or claim 9 whereby when a gesture is input that matches a reference gesture, the operating mode linked to that reference gesture is executable.

11. A multimedia human computer interface as claimed in claims 6 to 10 whereby the operator can change the gesture, stored in the gesture library, that is linked to each operating mode.

12. A multimedia human computer interface as claimed in claims 6 to 11 whereby a training function allows the user to add a new gesture to the library and associate with this gesture an operating mode of the image manipulation software

13. A method of inputting information to a computer system having a position sensitive display characterised in that a user draws a gesture on the display; the gesture is digitised, and this digital representation is compared against other gestures stored in a library of gestures, where each gesture in the library is associated with a software function; the software function is carried out that is associated with the gesture in the library that matches the gesture drawn by the user.

14. A method as claimed in claim 13 where the computer also incorporates a speech recognition unit whereby, the speech recognition unit takes spoken words input using a microphone connected to the system; the words recognised are compared to words stored in a library, where each word in the library is associated with a software function; the software function is carried out that is associated with the word in the library that matches the word spoken by the user.

15. A computer program product that can be used with a computer system having a display, a position sensitive panel, and a sound digitiser, the computer program product comprising: a gesture recogniser and processor to detect and match, with pre- stored gestures stored in a gesture library, any gestures made on the position sensitive panel; a command processor that takes in commands from the gesture recogniser; image manipulation software that displays an image on the display and has manipulation functionality; whereby the gesture processor, on detecting a gesture that matches one in the library, sends a command associated with the library gesture to the command .handler, which in turn sends it to the image manipulation software.

16. A computer program product as described in claim 15, whereby the computer system also incorporates a speech recognition system that interprets spoken commands from a user and sends these commands to the command processor.

17. A computer program product for operating a method according to claims 13 and 14.

18. A multimedia human computer interface and method substantially as hereinbefore described, with reference to the accompanying drawings.