Posted on

Ishaan’s Status Report 03/27/2021

Our team met 3 times this week I worked on the following:

  • Brainstorming how to track the glove. We narrowed it down to the following options:
    • Using IR Leds + IR cameras
    • Using LED + OpenCV color threshold tracking
    • Using an IMU to detect acceleration and predict where the user will reach
  • We developed OpenCV code to track Blue LEDs. This had high accuracy and low latency. However, when background blue colors were introduced the tracker performed poorly
  • To tackle the above problem, I worked with my team to write OpenCV code to track the IR LED. This converted each frame to grayscale and then applied a mask to get pixels in the range of 240-255. After applying an IR film on the camera we were able to have very good performance and there was minimal interference.
  • To completely eliminate any background noise we decided to make the LEDs blink at different frequencies, and are currently trying to identify how to track blinks.

Progress Status – on schedule

Tasks for next week:

  • Explore the option to track blinking IR LEDs. Attempt to develop a method using openCV contours and blinking frequency to identify blinking IR LEDs
  • Explore any other options to eliminate background noise. This might involved adding a new IR LED or introducing a calibration step .
  • Experiment with the IMU, attempt to use it to track acceleration and predict future points
Posted on

Team Status Report 3/27/2021

Our team met 3 times this week and we focused on building and tracking the glove controller

The image below demonstrates our code capable of tracking an LED.

 

Significant risks 

  • The IR LED needs to be directly pointed towards the camera for tracking. Depending on how the users orient the glove the tracking accuracy can be impacted.
  • The accuracy of the tracking using computer vision is affected by any background light or colors.

 

Changes made:

  • We have decided to use multiple IR LEDs on the glove. This will eliminate the problem of LED orientation.
  • Since we have decided to use multiple IR LEDs we intend on making each IR LED blink at a particular frequency. This will help to uniquely identify LEDS.
  • We intend on using an IR filter on the cameras to eliminate any background colors or noise.

Schedule – No Changes

Posted on

Logan’s Status Report 3/27

This past week our team met twice in Hammerschlag to work on our design. Our goal for this week was to finalize our positional tracking design and test it. We have had decent success with tracking the infrared LED via the Raspberry Pi NoIR camera. However, there is still an issue of background light being detected as false positives. Our code greyscales the image from the NoIR camera and masks it to filter anything below a certain threshold, because the infrared light is essentially a perfect white as RGB. One unexpected issue is that the Raspberry Pi NoIR camera does allow infrared light, however it converts it to RGB and does not expose an infrared channel. Beyond working on the project, I also started the ethics assignment and read through the first of the assigned articles. I thought the examples provided were very interesting and expanded my perspective on the political aspects of technology.

We are on schedule and have made no changes to it as of now.

Next week we will need to improve our positional tracking and specifically solve the issue of false positives in the form of background light. This is partially due to the NoIR camera not providing an infrared channel. If it exposed an infrared channel we would likely have better success tracking the infrared LED. Solving this challenge will be our next goal, and we have a few ideas to accomplish it. One solution is that the LED can blink at a high frequency, and in this way we can differentiate it from background light. There are also background subtraction filters available in OpenCV, which may help alleviate the background light issue.

Posted on

Arthur’s Status Report for 03/27/21

We met several times this week on campus in 1307 HH to work on our project. We are currently testing out positional tracking with our Raspberry Pi NoIR Camera module and Infrared LEDs. We are all currently working on the python scripts that will go on our RaspberryPi to control the Camera module, to filter for Infrared LEDs using frequency/wavelengths, and to contour the LEDs that the camera detects in order to track them.

We are still on schedule

We want to finalize the positional tracking by the end of the week and also start the assembly of the glove (right after we figure out the tracking). For the Interim Demo on April 12th, we want to have the glove fully built, and we want to be able to showcase how accurate our positional tracking is and how fast the response time is.

Posted on

Team Status Report 3/13/2021

Significant Risks:

The largest risk we face currently is an underdeveloped plan for position tracking. We are considering using IMUs for this, however, IMUs drift at a quadratic rate (since they work on double integrals). We may need to use some sensor fusion with a different device (perhaps optical) provides the point of reference to account for drift.

Changes made:

The main change is the position tracking system, which is no longer going to be solely an LED + camera. This decision will be finalized this week as part of the design report.

Schedule – On Track. We switched to a new project management system (ClickUp) and modified our schedule to accommodate the needs for IMU testing.

Posted on

Logan’s Status Report 3/13/2021

This week we had our Design presentation which Arthur gave. We received a lot of useful feedback. One of the main takeaways is that we really need to put more thought into the IMU design decision, and how we will deal with drift. Our schedule needs to be adjusted to test and design the position tracking system first before any other tasks are completed for the glove. I completed another one of the game assets this week (the bomb). I still want to work more on the texture but that will be deprioritized until later.

We are on schedule, but need to prioritize testing the IMU position tracking.

Next week I will be working on the design report and working on nailing down our positional tracking design.

Posted on

Ishaan’s Status Report 03/13/2021

This week I worked on building the software block diagram below.

Additionally, I worked on designing our data structures, Classes, functions and any interactions in the game. This will enable us to easily develop the software once we have completed glove development.  As a group we decided to put most of our effort on developing the glove with high accuracy and low latency and later focus on Unity development.

I also researched how to use the IMU and found that we might need to do some type of preprocessing before using IMU data. It seems like we might need to use either a Kalman filter or Complementary filter in our preprocessing step, so I worked on developing these filters in Arduino.

 

We are on schedule

 

For next week I will be working on the Design report. I will specifically be working on writing about the software aspects of our project. I will also work on further implementing the preprocessing step for the IMU data and design the calibration step of the IMU-glove.

Posted on

Arthur’s Status Report 03/13/21

This week, I gave the Design Presentation for our project. We received lots of very helpful feedback which we will use to revise certain aspects and details of our design. All of the parts for the glove have arrived except for the IMU, (the positional tracking unit). I will also need to go to campus this week to retrieve some wires, a breadboard, and a Mux.

We are still on schedule.

For next week, we have the Design Report due on the 17th and for our schedule, we would like to have the glove assembly finished by EOW.

Posted on

Logan’s Status Report 3/6/2020

This week I put a hold on going further with the Unity and asset development as a result of our discussion on Monday to focus more on the physical glove itself, and treat the game as a very minimal demo to start. So instead of frontloading our time with the game assets and development, we met multiple times this week to nail down our exact goals for the glove, and hone in our design. One pivot that we made this week was to purchase IMUs which we plan to test as an alternative to the camera-based motion tracking. I helped work on the design presentation, including the block diagram and metrics/validation sections.
We are on schedule, although our focus has shifted somewhat.
Next week we plan to continue current work, where the development for the game will consist of only what we need to get the bare bones proof of concept working first. Depending on when parts arrive, I expect to spend time helping Arthur configure the IMU components to test extracting motion coordinates.
Posted on

Ishaan’s Status Report 03/06/21

I worked on the following:

  • Arthur, Logan and I created the block diagram for the overall project and also the hardware schematics.

  • Researched using IMUs instead of OpenCV + LED combination from our last post, we ultimately decided to focus on glove development and use the IMUs
  • Added the assets into unity and started spawning fruits in the environment
  • Worked on code for user fruit interaction and collision detection. Can do basic collision detection and spawn fruit in Unity.

Progress Status – on schedule 

Deliverables for next week: 

  • Add special combos, fruits and bombs
  • Help Arthur with developing the glove and any arduino scripts that we might need to add
  • Work on using the IMU and interpreting the x,y,z coordinates
  • Start researching how to add physics and gravity in the game