GLOVE MOUSE

by Arvind Sannasi

Welcome to my Final Project for class EE47 at Stanford. 

 

Short Description:

This is a glove that can sense finger presses and hand flexes to do mouse functions like moving the cursor, left-clicking, right-clicking, and scrolling. Watch this to learn more:

 

 

Long Description:

Watch this video and read below:

 

 

     The Glove Mouse is designed to be a fun way of controlling the mouse on a computer. My main purpose was to have a fun way of flipping through Powerpoint presentation without a remote or having to walk to the computer repeatedly. The Glove Mouse has three basic components: the Glove, the Code, and what I like to call the WristHub. Here I will describe some of the design choices I made with each component.

 

Glove

     The Glove houses the flex sensor and the two force sensors. The flex sensor is stitched into the last finger of the glove and allows hand flexes to emulate clicking. I put the flex sensor on the last finger because it reduces accidental clicking when you just want to move the mouse. Using one flex sensor is actually the optimal solution with soft gloves because bending just one might make your other fingers move as well. With one flex sensor, you no longer need to monitor your own hand movements. Instead of single finger movements, you can just flex your entire hand to click, which is more fun.

 

     The two force sensors are free floating, meaning that they are not attached to the glove per say, but are instead bent into a joystick-like position using the jumper wires connected to the WristHub (imagine a Wii Nunchuk). This makes the force sensors repositionable. I can make them into a joystick or place them on my fingertips. This helps because sometimes one mode will allow easier ways of pressing the force sensor. I used two force sensors because of time constraints, but I think that this system is somewhat simpler than having to memorize what direction each finger controls.

 

WristHub

     The WristHub is a Arduino Micro and Graphical LCD mounted on a perfboard with 3 buttons, which is tied to a piece of cardboard using pipe cleaners. A velcro wrist strap is used to attach this to a wrist. This is where information about the device state can be viewed.

     For example, the Arduino supports a Mouse Mode. This makes the Arduino act like a USB mouse, but renders it unprogrammable. To easily know if the Glove Mouse is in Mouse Mode, the Graphical LCD shows the variable's value on the screen. In addition to showing Mouse Mode, the WristHub also shows what click the Flex sensor will perform (left-click, right-click, scroll), the name of the device, my name, and the direction the mouse will move(Up, Down,Left,Right).

     The WristHub also has 3 buttons. The middle one turns on and off MouseMode(the flex sensor will no longer click). The bottom one changes what type of click the flex sensor does. The top one changes the direction, in case the force sensor is not responding correctly. 

 

Code

/*

 * This code will use 2 FSR's, 1 flex sensor, and 3 buttons to control a computer's mouse via USB cable.

 * This is all thanks to the code from class EE47 at Stanford, specifically their String.ino and BTArduino.ino files.

 */

 

#include <SPI.h>

#include <Adafruit_GFX.h>

#include <Adafruit_PCD8544.h> // These libraries are for the graphic LCD display that I will use for debug messages.

// pin 7 - Serial clock out (SCLK)

// pin 6 - Serial data out (DIN)

// pin 5 - Data/Command select (D/C)

// CS is not connected to Arduino, and as such is listed as pin -1.

// pin 3 - Reset (RST)

Adafruit_PCD8544 display = Adafruit_PCD8544(7,6,5,-1,3);

 

const int button1 = 2;//pin numbers for buttons

const int button2 = 9;

const int button3 = 12;

 

const int FSR1 = A1; //pin numbers for sensors

const int FSR2 = A2;

const int FLEX = A0;

 

volatile int direction = 1; // direction of mouse

volatile int clicker = 1; //which click flex sensor does

volatile int MouseMode = 1; //wheether the Arduino acts like a mouse

 

int x = 4; //set mouse speeds in these directions

int y = 4;

 

void setup() {

  display.begin(); //initialize display

  display.setContrast(55);

  display.clearDisplay();

  display.setCursor(35,20);

  display.print("ON");

  display.display();

  delay(500); 

  pinMode (button1,INPUT); //sets pin mode of all sensors and buttons to input

  pinMode (button2,INPUT);

  pinMode (button3,INPUT);

  pinMode (FSR1,INPUT);

  pinMode (FSR2, INPUT);

  pinMode (FLEX, INPUT);

}

 

void loop() {

  display.setCursor(5,0);

  display.print("GLOVE MOUSE");

  display.setCursor(0,10);

  display.print("by Arvind"); //print name and author on lcd screen

  display.display();

  if (direction == 1){  // prints all the variable values on the lcd

    display.clearDisplay();

    display.setCursor(0,20);

    display.print("Right");

  }

  else if (direction == 2){

    display.clearDisplay();

    display.setCursor(0,20);

    display.print("Left");

 }

   else if (direction == 3){

    display.clearDisplay();

    display.setCursor(0,20);

    display.print("Down");

 }

   else if (direction == 4){

    display.clearDisplay();

    display.setCursor(0,20);

    display.print("Up");

 }

  if (MouseMode == 1){

    display.setCursor(0,30);

    display.print("Mouse Mode OFF");

  }

  else if (MouseMode == 2){

    display.setCursor(0,30);

    display.print("Mouse Mode ON");

  }

  if (clicker == 1){

    display.setCursor(0,40);

    display.print("Left Click");

  }

  else if (clicker == 2){

    display.setCursor(0,40);

    display.print("Right Click");

  }

  else if (clicker == 3){

    display.setCursor(0,40);

    display.print("Scroll");

  }

 

  volatile int button1_val = digitalRead(button1);//stores sensor and button readings in variables.

  volatile int button2_val = digitalRead(button2);

  volatile int button3_val = digitalRead(button3);

  volatile int FSR1_val = analogRead(FSR1);  

  volatile int FSR2_val = analogRead(FSR2);

  volatile int FLEX_val = analogRead(FLEX);  

 

  if (FLEX_val < 320 && MouseMode == 2){

   //if flex sensor is bent and MouseMode is on, click the mouse

   switch (clicker){ //what click flex sensor does depends on this variable

    case 1:

      Mouse.click(MOUSE_LEFT);

      delay(200);

      break;

    case 2:

      Mouse.click(MOUSE_RIGHT);

      delay(500);

      break;

    case 3: //a scroll wheel goes in two directions, so left and right are mapped to up and down scrolling

      if (direction == 1|| direction == 3){

        Mouse.move(0,0,-2);

        break;

        }

      if (direction == 2|| direction == 4){

        Mouse.move(0,0,2);

        break;

        }

   }

    }

 

  if (FSR2_val > 0){

    switch (direction){

      case 1:

        Mouse.move (x,0,0); // when FSR2 is pressed, the cursor will move

        break;

      case 2:

        Mouse.move (-x, 0,0 );// the cursor direction depends on what state the mouse is in

        break;

      case 3:

        Mouse.move (0,y,0);

        break;

      case 4:

        Mouse.move (0,-y,0);

        break;

    }

    }

    if (FSR1_val > 0){ //this FSR changes the direction

    switch (direction){

      case 1:

        direction=2;

        delay(200);

        break;

      case 2:

        direction=3;

        delay(200);

        break;

      case 3:

        direction = 4;

        delay(200);

        break;

      case 4:

        direction = 1;

        delay(200);

        break;

    }

    }

  if (button2_val == HIGH){

    //makes arduino start or stop acting like a mouse

    switch (MouseMode){

      case 1: 

        MouseMode = 2;

        Mouse.begin();

        delay(500);

        break;

      case 2:

        Mouse.end();

        MouseMode = 1;

        delay(500);

        break;

    }

  }

  if (button3_val==HIGH){ //changes what type of click the flex sensor does

    switch (clicker){

      case 1:

        clicker = 2;

        delay(500);

        break;

      case 2:

        clicker = 3;

        delay(500);

        break;

      case 3:

        clicker = 1;

        delay(500);

        break;

    }

  }

  if (button1_val==HIGH){//changes direction

    switch (direction){

      case 1:

        direction=2;

        delay(200);

        break;

      case 2:

        direction=3;

        delay(200);

        break;

      case 3:

        direction = 4;

        delay(200);

        break;

      case 4:

        direction = 1;

        delay(200);

        break;

    }

  }

}

 

How To Build:

 

Materials:

Arduino Micro

1 Flex Sensor

2 Force Sensors

3 Buttons

Nokia 5110 monochrome screen with HEF4050BP level shifter chip

10 kohms resistors x4

22 kohms resistors x1

33 kohms resistors x2

27 kohms resistor x1

220 ohms resistors x3

Female Headers for force sensors, nokia 5110, level shifter, arduino micro

Jumper Wires (solid)

 

Pipe cleaners

cardboard

wrist strap

velcro

Glove

 

Steps:

1. Solder on header pins on breadboard for Arduino Micro, Graphical LCD, and level shifter
2. Connect the following (taken from lab 5 of EE47) 

 

Graphical LCD		Connection
1 - GND	-->	GND
2 - VCC	-->	3.3V
3 - CLK	-->	4050 Pin 10
4 - DIN	-->	4050 Pin 12
5 - D/C	-->	4050 Pin 15
6 - CS	-->	GND
7 -  RST	-->	Voltage Divider at 3.3V
8 - LED	-->	3.3V (no pullup required)

 

Arduino Pin 3

-->

Voltage Divider at 5V

 

I used three 220 ohm resistors for the voltage divider.

4050		Connection
Pin 1	-->	3.3V
Pin 8	-->	GND
Pin 9	-->	Arduino D7
Pin 11	-->	Arduino D6
Pin 14	-->	Arduino D5

 

 3. Solder on buttons and wire them each like the following : https://www.arduino.cc/en/tutorial/button to digital pins 2, 9 ,12, or just change the pin numbers in the code. It's easiest to use a common 5 voltage bus.

 

     4. Add the wiring for the the sensors. Make sure to use long enough wires. I used 27kohms for the flex sensor and 33kohms for the force sensors. The force sensors are connected to A1 and A2 and the flex sensor is connected to A0. I recommend taping the FSRs to a female header with electrical tape and then soldering the header to the jumper wires instead of directly soldering to avoid burning the FSR. Use electrical tape on the exposed contacts of the flex sensor as well.

 

     5. Punch 4 holes in a piece of cardboard the size of the perfboard and put them together with pipe cleaners. 

     6. Attach a wrist strap in between the perfboard and carboard to secure it. Add velcro on the strap.

     7.  Sew the flex sensor to the top of the last digit of the glove.

     8.  Upload the code and have fun!

 

Design Plan:

     The need I wanted to address primarily was to allow presenters to have both hands free for hand gestures during Powerpoints and not having to interrupt the presentation by going back to the computer. The current major device for making Powerpoints easier is a remote, but that requires a hand to hold it.

 

     Here is my initial plan: 

Deadline: Saturday August 15

Plan A: Glove that performs all the functions of a bluetooth mouse

Removable Features: bluetooth, middle click

Plan B: Wrist strapped cursor mover

Tasks:

1. On/Off Button

2. Use buttons to move cursor

3. use potentiometers to move cursor

4. use flex sensors to move cursor

5. use accelerometer to move cursor

6. FSR sensor to click (left, middle, right)

7. Paper prototype

8. double-click?

9. Get a glove - sew, use old gloves, tape?

10. analog stick to move cursor?

11. decide which input to use to move cursor

12. arduino to computer bluetooth communication

13. Write- Up

 

Here is my initial ideas:

 

 

Here is my Verplank Diagram: 

 

Here is my state diagram: 

 

 

I changed my plans later however. This is what I did and the mistakes along the way.

     Tasks:

1. Test how the arduino changes into a mouse using simple button circuit
2. Use potentiometers to find out how analog input can be translated into mouse movement 
3. Use FSR sensors to move cursor  
4. Use flex sensor to left-click
5. Make perfboard with LCD display and arduino and level shifter. To make things easier, I used different buses for power and ground instead of using just the arduino port. It took a long time to figure out why the lcd screen wasn't displaying after finishing. It turned out that the chip didn't lock onto the female headers so I used a DIP adapter and it suddenly worked.
6. Wire three buttons onto perfboard. I tried to see if I could get away with using one power line and one ground line to power all three buttons, but soon discovered that pushing one button triggered all the other buttons. I had to resolder the ground connection so that each button has its own 10K resistor.
7. I wired up the HC-05, but discovered that I had no way of telling the PC what to do with the bluetooth commands it received without writing another program. I decided to focus my attention on the flex and force sensors because I would have had to order a bluetooth 4.0 board.
8. I then wired up the flex sensor and tested it to see if it could click my mouse. I soldered directly to the flex sensor, but I should have probably electrical taped it to a female header and then soldered that to the wires.
9. I wired the two force sensors by soldering the wires to a female header and then electrical taping the force sensors. I was worried that the soldering iron might burn the contacts. I didn't measured the wire lengths before I soldered the wires on. I should have measured the glove and matched it with the length of the wire.
10.  At this point, I had this: https://www.youtube.com/watch?v=BXgKBGOnrBg
11. Then I put the perfboard onto the board with pipe cleaners (keyrings didn't work well) and put a velcro wrist strap on it.
12. I tried to electrical tape everything to my glove, but I eventually sewed the flex sensor to make it more stable. The FSR's I did not sew on.
13. Then I refined the code so that I could adapt it for 2 force sensors, one flex sensor, three buttons, and a graphical LCD. (Lots of switch statements). 

 

     Even though I reached my goals of making an Arduino powered glove emulate a USB mouse, there are numerous ways to move further than what I did. For one thing, adding more force sensors would make moving the mouse a breeze compared to the two I have now. If a force sensor is attached to each finger except the thumb, a pinching motion with each finger could move the cursor in different directions. 

 

     I had initially tried to make my mouse bluetooth using the HC-05 so that movement is no longer hindered by the usb cable, as can be seen in the empty socket on the right of the graphical lcd. However, after some testing, I figured out that to give the glove bluetooth compatability, a bluetooth board supporting bluetooth 4.0 was needed so that the computer knew the device was a mouse. Another option could have been to have the glove send data to the computer and then writing a computer program to decipher this data into mouse commands. Sadly, I did not have enough time to implement this feature.

 

     Another cool feature I initially thought of adding is an accelerometer to either the palm or the back of the glove. This could detect x and y movement and send it to the computer as mouse movements. I believe that this coupled with the aforementioned bluetooth feature will make this glove mouse feel intuitive and achieve my ultimate goal.