Posted on

Jolie’s Status Report for 12/7

The past week, I’ve been working on the game state tracker and some integration tasks such as getting the camera to take pictures within our python program as opposed to just from the command line. Cody and I spent a good amount of time on this task that shouldn’t have taken so long. We were getting an error that the camera was busy and couldn’t take the picture. We ended up earning this was because we had initialized a cv2 camera instance we were thinking of using and forgot to comment it out.

I had already written the functions for most of the intricacies of the game state tracker, so the rest was just interfacing with the MQTT protocol that Denis set up. Today, we tested the whole system together and it went quite well. I found a couple bugs in the check_board function I had written in terms of calculating the start and end column for a horizontal word. But after fixing that, when we pressed the “Check” button on the individual player touch screen, we got the point total and validity back! We are still deciding the convention we put in place for submitting words, whether we should require a check beforehand or not.

I am glad at the place we are in now, and hopefully we can finish up the home stretch in a relatively smooth manner. I’ve been working on the final report for the past three weeks, editing from the design report, so I think we should get that done soon.

Posted on

Denis’s Status Report for 12/7

This week, we gave our presentation on Monday, and then I worked the rest of the week on setting up all of the screens so that they can all be running at once as well as the game state manager. At this point, besides creating the hardware, almost all of our time will be for integration and debugging. We will also create the poster and the demo materials. We are on pace to complete the project by Friday

Posted on

Team Status Report for 12/7

We have integrated most of our system (Cody’s computer vision model, Denis’s MQTT and user app, and Jolie’s game state tracker and board checker). We decided to take out the main user display board as it seemed redundant considering we can just flash the validity and score to the player after the “Check” button is pressed. With less than one week left, we are looking solid and close to completion. We still have some testing to do including our user study in addition to the video, poster, and final report to finish.

 

Tests Completed:

  • MQTT Data size test: We tested varying sizes of data packets to see their effect on the latency of communication. We decided to use packets of less that 300 Bytes as a result
  • Individual CV letter tests: We tested the accuracy of identifying individual letters in order to see which ones were the least consistent. This led to us adding additional marks to the I and O letter tiles
  • Size of Database vs accuracy and time: we tested varying database sizes and their impact on the CV accuracy, as well as how long that it took to process the images. Ran 3 tests, one with a full, a half and a minimal version of the database. The tradeoffs were as we expected, the larger the database, the more accurate but the slower it was.
  • Testing scoring and validity: We tested a variety of different word locations and valid/invalid words on static boards, this led us to being certain that our sub functions worked
Posted on

Cody’s Status Report for 12/7

Through the past two weeks or so, I have been finishing up the last touches with the CV, gathering testing data, and integrating with the rest of the system. The biggest change to the CV was to transform the image to account for images that are not perfectly in-line with the board. This added additional time to the classification, since I need to essentially run through the character recognition first to identify corners to be used in the transformation, then re-recognize characters on the new image while mapping them to their locations. While this certainly added significant time to the functionality, the CV is far more robust and character location mapping seems to be very accurate. This was shown through a 100% location accuracy through 5 testing boards after making this change.

I’ve also made a few modifications to some of the tiles that were being misclassified. We decided to put a diagonal line through ‘O’, which was being misclassified often as ‘D’. I have so far seen 100% accuracy with these characters after this change. Overall, OCR accuracy seems to be around 95% percent, but I am still in the process of gathering some final data.

Another change was the decision to require all 4 corners of the board to be identified before attempting to classify and locate them. Before, if not all 4 corners were found, I was trying to use 2-3 identified corners to estimate the locations of the characters as accurately as possible, however, this did not result in acceptable location accuracy. We have decided requiring all 4 corners identified (which seems to not be an issue so far) is better than mislocating words, which could significantly harm the game state.

Lastly, I’ve been working with Jolie and Denis to get our system integrated together. For example, we recently finished implementing the functionality of taking an image on the RPi and running it through the OCR before sending it to Jolie’s scoring code.

As we enter the final week, we will finish integrating, gather testing data, and finish our poster, video, and report.

Below is one result from testing. The console output is characters identified and their locations. In this test, all characters were correctly identified and located, despite the input image being far from ideal.

Posted on

Denis’s Status Report for 11/23-11/30

The past couple weeks I’ve focused my time into integrating the different individual elements of our project as well as some of the smaller tasks that our project needs.

A major thing over the past couple weeks has been configuring our touch screens and creating scripts that will run on our raspberry Pi 0s and the Raspberry Pi 4. This was very difficult and took a lot of debugging and troubleshooting. However, we now have a consistent setup and are able to get the touch screen working almost every time. We will replicate this setup across the other 3 raspberry pi 0s.

We also designed a casing for the screens to contain the battery packs and converters.

I believe that we are on pace for the final demo and will be in a good place for the rest of the semester.

I gained a lot of skills throughout the semester, both clearly defined technical skills as well as softer problem-solving skills. I got the chance to work with new python modules, and a new protocol in MQTT. Learning about something like a new protocol and considering the tradeoffs of different options was a good exercise in design. Another good experience was debugging the LCD touchscreens, as encountering a snag like that, without a clear solution was a good chance to practice troubleshooting. Having to look through the boot logs and looking through old forums was definitely a learning experience.

Posted on

Jolie’s Status Report for 11/23-11/30

These past couple weeks, Denis and I put a lot of work into the user interface with the individual touch displays. It took some trial and error with the driver to get the desktop of out RPi 0s to show up on the lcd display (which was quite frustrating) but we ultimately got the desktop to show. The touch portion was touch and go (no pun intended) for a while where it would randomly work or not work on boot. We think we found the problem with a race condition on boot up that would sometimes permit the touch functionality and sometimes  not. Denis and I laser cut a box for the battery pack and buck converter to fit in. The display then rests on top of the box for a clean wireless look.

I made the front end of the user facing app user tkinter graphics where I made a keyboard, tile display box, player scores, “check”, “submit”, and “hint” buttons, and then ultimately a page for hint results. This will be how players interact with our system.

For this upcoming week, we really need to finish integration, mostly of the cv with the rest of the system. I think we are in a good place to have everything done by the demo.

It’s safe to say I gained a lot of new skills throughout my Capstone experience this semester. First off, I was able to write the hint generation algorithm from help from the article https://www.cs.cmu.edu/afs/cs/academic/class/15451-s06/www/lectures/scrabble.pdf. I spent many hours rereading this to understand how the backtracking algorithm worked and researched more into the ins and out of the directed acyclic word graph. Furthermore, I had no prior experience with tkinter graphics so I read specs and watched videos on how to make a simple app and then expanded on it from there.  Finally, when debugging the issues we were having with the RPi 0s, I read many posts from Raspberry Pi Forums from people who were having similar issues to us. Often times, it would take many iterations of Forums to find a possible fix, which gave me good experience of dealing with this type of hardware.

Posted on

Team Status Report for 11/23 – 11/30

Last week we presented our interim demo in which we showed off the individual aspects of our project. While we found it slightly challenging to fit our work as well as a sufficient overview of our project into the constrained timeslot, we believe it overall went fairly well. As we enter the final weeks, we will focus mostly on integration and testing as well as the necessary components for our final presentations, like the slides, poster, and video.

Posted on

Cody’s Status Report for 11/23 – 11/30

During our demo this week, I presented the work I’ve done for the CV / OCR aspects of our project. I’ve managed to get the character location recognition accuracy very high with a reasonably good image. Letter recognition seems to be around 80-85 % but I plan to add characters to the nearest-neighbor database to hopefully increase this. Additionally, I will implement the necessary logic to capture images and process when necessary. Furthermore, I will help with the other work (integration, final presentation, poster, etc) as we enter the final weeks.

“what new tools or new knowledge did you find it necessary to learn to be able to accomplish these tasks? What learning strategies did you use to acquire this new knowledge?”

Computer vision was very new to me, and I was able to familiarize myself with key components of preprocessing, optical character recognition, and even the implementation of a simple nearest-neighbor classification model. This also introduced a somewhat new style of testing, as most of the testing necessary involved manually observing the output of preprocessing stages. Primary learning strategies included reading articles and publications on CV / OCR, looking through code examples similar to what I was trying to accomplish, and simply working hands-on with the techniques.

Posted on

Cody’s status report for 11/16

After trying another OCR library which ended up also being more inaccurate than I would like, I decided to implement my own character recognition. To do this, I added a small database of characters and I perform nearest-neighbor on each isolated character. This seems to result in high accuracy in the testing I’ve done. I also made the logic for mapping characters onto the board more robust, accounting for if fewer than all 4 corner characters are identified (we still need at least 2). The mapping appears pretty accurate but if a tile is close to the edge, it sometimes gets mapped to the adjacent tile. I will try to improve this as much as I can. For the demo, I plan to show the pipeline of image -> preprocessing -> contour recognition and filtering -> individual classification -> mapping onto board.

Posted on

Denis’s Status report for 11/16

This week I began creating our battery packs to make the Raspberry Pi 0s portable. Right now, they are very cumbersome because the wiring isn’t done well, but as a proof of concept they are fine for right now. For the final demo, they will be soldered and much more compact.

Also, I began working with the touchscreens as they were delivered this week. As of right now we are struggling to get the touchscreens to display the GUI and begin writing an application for the user input. We hope to have some sort of user input for the demo. I will be working a lot on Sunday to get this working and will provide an update of how far I get with it.

As of right now, I believe I am on pace for completing the project, and definitely on pace for having a successful demo