Jenny Han, Bradley Zhou, Jade Wang. Carnegie Mellon ECE Capstone, Spring 2021
Overall, we are on track. The biggest risk for our project at the moment is if we mess up some part of our hardware and need to reorder different parts and try again. We decided to mitigate this by buying a couple of parts as …
This week I did research on the different technologies we should use to implement our project. Specifically, I looked into parts for the Arduino including the Bluetooth module, accelerometer, and projector. I also looked into how these parts would fit together and if there was …
This week, I did more in depth research on specific implementation details and design specifications. I revisited some of the previous decisions we had made, such as what Arduino modules for our pen and what libraries we would use for the gesture recognition. Previously, we had decided on the MPU6050 module, but I discovered a module called BNO085 that boasts finer detail and sensors with lower latency. We plan on exploring both options in our development phase. We also had previously decided to use openpose for our gesture recognition, but the computational intensity of it made it a suboptimal option. I started looking into using OpenCV to tradeoff a small amount gesture accuracy for a much faster response time. Using OpenCV also simplifies the feature of letting users define their own gestures. In our development phase, we will most likely explore OpenCV first. I made a plan for our development process for the pen, specifically around prototyping on breadboards while we develop the software before we finalize the ergonomic design of the pen. I made a spreadsheet of the materials we need to buy. I worked a little bit on the design presentation that will be presented in two weeks. Our next step, after receiving the parts that we ordered, will be to start familiarizing ourselves with the sensors by reading the documentation and decide on how all the modules will be connected to the main arduino board. After that, I can start planning on the software that will actually track the pen movements using the sensor data. This will most likely require lots of iteration and bring in elements of control systems that we will need to do additional research on. Overall, I think we are on pace to completing this project.
This week, I researched the Arduino Pro Micro and HC-05 Bluetooth module compatibility. I looked into which pins should be connected to connect the two parts. I also created a Google Sheets to document our planned budget online. Additionally, I met with Jenny and Bradley. …
This week, the most significant risks that could jeopardize our success is ordering the correct parts for our project. Since so much of our project relies on the physical assembly and connection of parts that we have to buy, both time and money could be …
This week, I did some research in ways that we could implement our project idea, as well as met with Jenny and Bradley to discuss the details of our project design. Last week, we had already solidified that our overall system would include a pen, body expressions, projector, and some way to connect our inputs (from the user) to our outputs (laptop screen updating to projector). Thus, I looked into different ways we could make this happen.
As for our first input, the pen, I looked into different programmable circuit boards that would be on the smaller end and had bluetooth compatibility. After narrowing it down to an Arduino Pro Micro for its manageable size that would work as a part of the pen, I found that the Bluetooth HC-05 piece would work as a way to relay input from the user’s movements to the laptop. I found several posts giving evidence that this has been done in the past and are pieces that we could work with: Resource 1 Resource 2 Resource 3
Furthermore, I researched the different kinds of projectors that we could use for our purposes of projecting into an average sized room and with average resolution. I found that they vary a lot in terms of engine type (DLP, LCD, LED, laser, etc.), resolution, size, price, and more. For the nature of our project, it was important to find one that be able to work indoors, had a manageable size, and was within our budget. Although I didn’t narrow it down to a specific projector, I researched many that were a manageable size and had a good enough resolution for our purposes such as the AAXA P& Mini Projector
I also had many video conferences with Jenny and Bradley throughout the week as we discussed the implementation of our project and flushed out more details for our presentation. As we discussed, I helped to put together a draft of our proposal presentation and then finally flush out a final one that would be interesting. We wanted to put as little words a possible on our presentation so our audience wouldn’t just be reading our slides, so I helped in writing most of our presentation details in the speaker notes, and putting icons that would accurately represent what we wanted to talk about in our actual slides.
As of now, I think we are on schedule. We were able to flush out our design and I feel ready to dive into building! This week, I will finalize exactly which parts we are going to purchase. I will create a budget outline to ensure that we can afford all of them with a little buffer as well in case parts break or don’t end up working in our design. After purchasing them, I will start familiarizing myself again with the Arduino IDE and library.
This week, I did research on what kind of system we could build for our Capstone project. For the pen, I researched different ways to measure position and relay information with the idea that the pen should be a standalone handheld device. I found some …
We will be creating Magic Mice, a laptop projector that can display any screen, anywhere. This attachment will connect to the user’s laptop (MacOS) and project the given screen onto any indoor wall. The user can use a pen device or body expressions to interact with the screen projection.
The following diagram describes the data flow for the system:
Our project requires 3 types of interactions coming from the pen, body expressions, and projector/computer GUI
For our pen, we will have buttons so that the user can input a left click and a right-click that would mimic a mac laptop mouse. The user would also be able to hold the left click button and drag it across the projected wall and therefore mimic a laptop mouse drag. Additionally, the pen being within 6 inches of the wall will mimic a laptop’s mouse hover, meaning that there will be a maximum of 6 inches distance to the wall for the pen to be considered active.
For body expressions in which we’ll have 3 default mappings of body expression to use shortcuts or macros. Our default ones right now include a hand swipe left or right for the back or next page in a browser, a hand swipe up for closing a tab, and a wall slap for page refresh. The user will also be able to create and edit custom gestures mapped to different user shortcuts or macros
Finally, we have the projector/computer GUI system. The projector will be able to project within 10 ft of a wall for a 1280×1024 projection resolution. We chose these values because they are the average size of a room and average projector resolution. Since we want our attachment to being portable, the project cannot be too large but still have a good enough resolution such that the user will be able to use it for everyday purposes. The computer is how the user will interact with the system via GUI, we plan on making it such that the user can enter the projector calibration mode and also modify gestures there.
What is unique about our project is that having a dynamic screen is useful for many cases from education to professional to personal.
Looking at the main use cases, this project would be useful in online education such as when a teacher wants to draw an explanatory diagram for a student but doesn’t have access to an iPad or tablet. They would be able to draw on a wall with our Bluetooth pen, a much easier experience than using a trackpad or mouse to draw. This Mac attachment would also be useful in professional settings — during group conferences in which one party has multiple people in a room, one device could be used and projected onto a wall so that number of connected devices could be lower. And finally, for personal use cases, this would be effective in replacements of TV for entertainment or drawing.
Magic Mice would remove the need for a computer to be limited by the size of your screen and open up more possibilities for interaction (we do pen and body movement, but this can be expanded). This is a new medium for interacting with a system and especially exciting due to many activities forced to be online due to Covid
Project Proposal Presentation: Proposal Final Presentation