osu!Gosu

Matthew Du, Victor Hu, Kevin Kuan, Yue Li

Concept

osu!Gosu is an interactive rhythm motion game in 3D space played with your hands, based off the popular 2-D computer game osu! . At its core, osu!Gosu uses infrared cameras to track the user's hands and head for an exciting interactive experience.

Motivation

Current motion tracking technologies like the Kinect have input lags of up to 300 ms, which is simply unacceptable for playing any rhythm-based games. osu!Gosu will limit latencies to 100 ms, allowing users to play the game without putting a fist through the TV in frustration.

Competitive Analysis

There are many other interactive games that we have a clear competive advantage over.

osu!

osu! is a popular musical rhythm PC game where the user uses the mouse to click, drag, and spin onscreen circles to a song.

osu!Gosu

osu!Gosu will allow the user to be more involved in gameplay, using their arms and hands instead of just a mouse. We believe this will deliver a more engaging experience.

Kinect

Kinect is a motion sensing peripheral developed by Microsoft for their XBox product line. It allows for full video capture with associated color and depth information, resulting in potentially low sampling rates and high latencies.

osu!Gosu

osu!Gosu uses simple infrared location sensing without the need to capture full images, allowing it to capture hand position with less input latency, which is critical for playing rhythm games.

Guitar Hero

Guitar hero is a musical rhythm game where the user manipulates a prop guitar with buttons to simulate playing a guitar across various rock songs.

osu!Gosu

osu!Gosu allows the user to interact with the game without the need for potentially cumbersome props.

DanceDanceRevolution

DDR is a musical rhythm game where the user stands on a dance platform and steps on directional arrows as shown on a screen. DDR machines are large and bulky and typically expensive.

osu!Gosu

osu!Gosu is cheaper without fundamentally sacrificing the interactive body movements that make these rhythm games so fun.

Functional Requirements

  1. High fidelity motion tracking
    • Pixel accurate to 1024 * 768 resolution.
    • Accurately filters out input noise.
  2. Gesture recognition
    • Recognition of open fist and closed fist gestures for selection of active elements.
  3. Haptic feedback
    • Feedback after selection and release of active elements.
  4. Low input latency
    • Less than 100ms input latency for feedback, gesture recognition, and motion tracking.
  5. Synchronous audio/video playback
    • 60 fps framerate
    • Smooth-tracking active elements
    • Synchronized audio and video streams
    • Audio response to active elements

Non-functional Requirements

  1. Unintrusive hardware
    • Wireless, ightweight controller
    • Low profile camera and console unit
    • Ergonomic form factor
  2. Intuitive gameplay
    • Controls can be calibrated to suit individual users
    • Game mechanics don’t require memorization
    • Responsive display mechanism

Hardware

Software

Protocols

  • I2C
  • TCP/IP

Week 1

We spent the first week desigining the PCB for our project, which involved researching the specifications of the hardware we planned to use, as well as meeting with the TA in order to determine whether our design was feasible. We also began to take apart a Wii remote in order to better understand the IR Camera that we were going to work with for the remainder of our project. The responsibilities were broken down as the following:

  • Matt was responsible for leading the PCB design.
  • Victor was responsible for assisting with the PCB design.
  • Kevin was responsible for assisting with the PCB design and updating the website.
  • Yue was responsible for breaking apart the Wii remote.

Week 2

Now that our PCB design is complete, we have begun soldering components on to the PCB board. In order to get these components, more importantly the IR cameras, we needed to break apart the other Wii remotes. We have also begun devleoping the game software for osu!Gosu using Unity. We hope to have a very basic minimum viable product with respect to software within the next week for the purposes of testing our hardware. The responsibilities were broken down as the following:

  • Matt was responsible for finalizing the PCB design.
  • Victor was responsible for beginning the implemention of the interface for software development for osu!Gosu
  • Kevin was responsible for assisting with the software development for osu!Gosu and updating the website.
  • Yue was responsible for getting the other IR cameras from the rest of the Wii remotes.

Week 3

We have completed assembling the PCB, and now have moved onto focusing on the software components of our project. In particular, we have begun writing out basic gameplay for osu!Gosu, and we have also begun writing drivers for the Raspberry Pi.

  • Matt was responsible for writing the I2C drivers for the Raspberry Pi.
  • Victor was responsible for implementing menu screens and game objects for osu!Gosu
  • Kevin was responsible for writing scripts for osu!Gosu.
  • Yue was responsible for finshing the PCB soldering and beginning work on the IR LEDs.

Week 4

Unfortunately, becuase we were unable to acquire the IR LEDs in time, we ended up borrowed a single IR LED to conduct our tests for reading the data feed from the IR camera. We then shifted our focus on developing more software for our project. With the drivers being completed for the Raspberry Pi, we started working on the API to gather the data from the IR LED.

  • Matt was responsible for writing the socket API for the IR LED.
  • Victor was responsible for writing software for the 3D component of osu!Gosu
  • Kevin was responsible for writing sliders and other game objects for osu!Gosu.
  • Yue was responsible for putting together usable hardware with the singel IR LED.

Week 5

Our IR LEDs have arrived, so we can now begin making the glove and headset. We finished making a basic circuit that has an IR LED constantly on, so the software team can begin the initial testing of the IR LED tracking. The software team has also decided to focus on building 3-D Jenga as a game to demo, as we would be able to test 3-D headtracking much sooner with using simpler 3-D objects. Because the UDP socket API is completed, we can also begin focusing on creating a mouse driver for more applicative uses of the IR LED tracking.

  • Matt was responsible for tweaking the socket API for the IR LED and begin working on the mouse driver.
  • Victor was responsible for writing code for 3-D Jenga.
  • Kevin was responsible for writing parsing code and connecting the socket API to Unity.
  • Yue was responsible for putting together the glove and headset with IR LEDs.

Week 6

The IR LED glove is now complete with a very rudimentary headset. After some user testing, we quickly realized that the LED placements on the glove were not comfortable for the user, so a new glove will be made so that the user will be using a fist instead of an open hand for the glove position. The headset LEDs work reasonably well for testing purposes, but a new headset will be designed with more optimal placements of the LEDs on some new goggles.

The software for the mouse driver is complete, and it works reasonably well with the glove. The latency is much better than expected, we observed about 10 to 15 ms when we were shooting for less than 50 ms. The headtracking on the 3-D Jenga game also works well, as the camera perspective shifts with 10 to 15 ms latency. Overall, we are very pleased with the latency of the LED tracking, but we will focus now on optimizing it even more and making sure that the headset and the glove can work together in the same space.

  • Matt was responsible for tweaking the mouse driver code and rewriting more clean parsing code.
  • Victor was responsible for optimizing code with IR LED tracking on the glove.
  • Kevin was responsible for optimizing code with IR LED tracking on the headset.
  • Yue was responsible for making a new glove and headset.

Week 7

The core gameplay and functionality of our project is complete. The new glove and headset feel very comfortable for the user, and IR LED tracking is much more optimized than last week. IR LED tracking with the headset and glove in the same space is now fully integrated, so users can change camera perspective and move blocks in 3-D jenga. The glove is also performing very well with the mouse driver, and users can play Osu with minimal latency. Because the final demo is next week, we plan to keep tweaking the code with respect to 3-D Jenga, as the headset and glove tracking in the same space could use a bit more optimization. We also need to write code for when the Raspberry Pi resets along with having a back-up glove, as we want to make sure our final presentation of our project is free of potential issues.

  • Matt was responsible for writing code to reset the Raspberry Pi and drivers successfully.
  • Victor was responsible for more optimization of the 3-D Jenga code.
  • Kevin was responsible for more optimization of the IR LED tracking code.
  • Yue was responsible for making a back-up glove and getting more batteries for the private and public demo.

Pictures, Videos, and Documents

    IR Camera on the Raspberry Pi

    Camera

    Glove with IR LEDs

    Glove

    Headset with IR LEDs

    Headset

    Demo of camera movement with 3d objects

    Demo of 3-D motion tracking with IR LED

    Final video demo

    Click here for the code repository