1. Introduction

The touchscreen technology has now become an inherent part of our day to day lives. Need for large touchscreen surfaces is increasing by the day. The gap between the real and virtual world is filled by this technology. Large touchscreen technologies today are in very high demand, to ease the interaction between the user and the computer. Projected surfaces are used in almost all fields such as education, hospitals and entertainment. Hence, the development of projected surfaces to be converted to touch screen surfaces has made it much easier to interact with the projected surfaces. The necessity to move to the PC, every time the user navigates through the computer or performs a particular task, is completely eliminated with this kind of technology.

The existing technologies like IWB (Interactive White Boards) are extremely expensive and can be used only by well-established organizations. Using IR sensors (Wii remote) application is also relatively expensive because of the use of hardware. This is where TOM(TOuch Mouse) comes into existence. TOM is an application that is used to convert any projected surface to a touchscreen surface. Using TOM the user can get the look and feel of touchscreen on any projected surface. TOM can be used on any 64 bit system which has an inbuilt camera or support USB camera. Since it is designed without the usage of any kind of hardware it is completely free. It is also designed with a very simple user interface so that even naive and inexperienced users can use this software. TOM was developed keeping in mind the common man, who cannot afford this kind of existing technology. Installing TOM on all systems in educational institutions enhances the quality of education, making classrooms more interactive and interesting, also giving a very practical approach to usage of computers, thereby instilling interests to pursue careers in the field of technology in the younger generation. Apart from this TOM can be used in offices, shopping centers, hospitals and entertainment industry to make work much easier and convey information in much easier and interactive way.

2. Background

Touchscreen technology allows users to interact directly with the screen without the use of any mouse, touchpad or any other such devices. This technology is commonly used in PCs, Tablets, gaming consoles and other such devices. Touchscreen technology is very popular because it makes the UI more effective and easy to use. It completely eliminates the usage of external hardware apart from the system that they are using. [2]

Need for large projected surfaces is increasing by the day. Therefore to satisfy this demand, existing touch screen projected surfaces like An Interactive Infrared Sensor Based Multi-Touch Pane [1], Direct control of the computer through electrodes placed around the eyes [3], Control units for operation of computers by severely physically handicapped persons [4] were created.

3. Tools for TOM

TOM application does not require any extra hardware, if the system has an inbuilt web-camera. If the system does not have an inbuilt web-camera then the user can use the USB-camera. This application is developed using many of the python libraries, the PyQt library for User Interface, Dlib library for tracking, OpenCV library for drawing and Windows API for controlling the mouse functionality.

[7] PyQt: This is the python binding of cross platform GUI toolkit Qt. It is freely available and contains multiple classes or methods for GUI, accessing SQL database, XML parser, etc.

[6] OpenCV: It is a cross platform freely available library which focuses on real time computer vision. It was originally developed by Intel.

[5] Dlib: This is an open source cross platform library written in C++. However its tools can be used in python applications.

4. Proposed System

TOM is an application used to build a bridge between real and virtual world. It is used to convert any projected screen to touchscreen. It is used to convert any projected screen to touchscreen. The architectural diagram of TOM is shown in Fig.1. The main architectural components of TOM are the camera (either system or USB), user, projected screen and finally the application itself. As shown in the figure, initially user opens TOM and calibrates screen after which he uses the projected screen as touch screen. The camera will continuously track hand movements and send them to TOM, which maps these coordinates to the desktop coordinates and move the mouse pointer there respectively.

Figure 1. TOM Architecture

5. Methodology

The application TOM continuously processes the movement of the human hand and moves the cursor accordingly. The user initially after launching the application must calibrate the window with respect to the projected surface size, and then user can use the projected surface as touch screen and can control the mouse. Fig.2 shows the sequence of interaction between different major components. Initially user reads instructions present on the app.

Figure 2. Sequence Diagram for TOM

The user then presses begin which then opens the calibration module. On clicking the endpoints of the screen to be matched, the ends are sent to the application. The ends are then sent to system for cropping and straightening the frame to match ends of projected screen. The user can now touch the projected surface and use it as touchscreen. The app continuously monitors the hand movements and maps the coordinates of the hand to the projected surface. The mouse cursor position is also moved with respect to the hand movements. If the cursor position is constant for a period of 3 seconds, the system initiates an appropriate click with respect to the foreground application. Once the click is initiated the cursor moves back to the center of the screen and continues tracking the hand.

6. Implementation

The application TOM is designed such that it works differently for specific foreground processes. TOM is able to recognize the foreground process opened and initiates the appropriate click with respect to the foreground application.

Figure 3. Using TOM

If windows explorer is running in foreground then the initiated click will be always be a double-click. TOM follows similar behavioral pattern for the desktop application. If foreground process is paint application, it works as follows. The window is divided into two parts: top part (the toolbar) and bottom part (the drawing area). Whenever the cursor is in top part it allows user to select the tools using a single click. Whenever the cursor is in bottom part it allows drawing. Keeping the hand still for 3 second initiates mouse-down operation. Now as the hand moves so does the cursor and tools can be used to draw, erase, etc on the paint application. Keeping the hand still for 3 seconds again will now execute mouse-up operation. Cursor can now be moved to the new position to draw something else.

For a power-point application the working of TOM is as follows. When the slide show is in progress, moving the hand from the center of screen to the right and placing at the end for 3 seconds will prompt it to move to next slide. On moving the hand from the center of screen to the left and placing at the end for 3 seconds will prompt to previous slide. Taking the hand to upwards and placing the hand for 3 seconds will exit the slideshow. On regular usage of application single click is initiated.

7. Conclusions

This paper proposes a new application called TOM(TOuch Mouse) to convert any projected surface to touch screen. TOM was developed keeping in mind the users need for large touchscreen surfaces at low costs. Using TOM a single person can operate the computer using just the projected surface and this provides seamless interaction between computer and projected screen. TOM application can be used in educational institutes or business environments especially when the inter-active boards are not affordable.