In our increasingly online word, amplified by a pandemic, much of our lives and communication with others has become virtual.
Companies have switched to an entirely work-from-home format. Education systems have also been forced to be online. Even hanging out with friends can be done completely online. This shift towards virtual has increased our dependency on our devices and the communication tools we have. But, these technologies also have their limitations.
Zoom, for example, has a drawing functionality that allows people to whiteboard, brainstorm, and discuss ideas on video calls. Teachers in wealthy areas will also use technologies like iPads, Surface Pros, or Smartboards to write and teach to their students. However, not all organizations can afford supplying $1000 tablets to every staff member.
Our goal is to make collaboration among groups more accessible. We do this by creating a computer attachment that allows users to project their screen content onto a wall. Users can then interact with the screen through a custom pen — which mimics mouse movement — and hand gestures — which map to keyboard shortcuts.
The following diagram describes the data flow for the system:
Magic Mice connects a projector, a pen, and the computer’s webcam input together. It takes in the user’s input in 3 different ways: hand gestures from OpenCV on the computer’s webcam, pen gyroscope and accelerometer data via an Arduino Pro Micro and sensors, and system customization and calibration through the computer webcam.
The hand gestures are customizable and map to custom keyboard shortcuts. The pen tracking algorithm allows the user to calibrate and control the mouse’s location as well as left and right click buttons. The GUI allows the user to calibrate the system such that it works at variable distances. The GUI is also the interface the user uses to customize hand gestures and their mapped keyboard shortcuts. With these system inputs into our algorithm the screen updates the computer and projection.
The following images show our hand gesture recognition algorithm in OpenCV and pen hardware system:
In summary, Magic Mice is able to display a Windows computer screen onto any wall variable distance and calibrate accordingly. The user will be able to passively and actively interact with the projection using the pen or with customized hand gestures.
Being able to dynamically change the interface of a screen gives many benefits to an online world. From educators to entertainers to professionals, the Magic Mice allows people to create and share on their own terms.