Category: Cody Status Reports

Through the past two weeks or so, I have been finishing up the last touches with the CV, gathering testing data, and integrating with the rest of the system. The biggest change to the CV was to transform the image to account for images that are not perfectly in-line with the board. This added additional time to the classification, since I need to essentially run through the character recognition first to identify corners to be used in the transformation, then re-recognize characters on the new image while mapping them to their locations. While this certainly added significant time to the functionality, the CV is far more robust and character location mapping seems to be very accurate. This was shown through a 100% location accuracy through 5 testing boards after making this change.

I’ve also made a few modifications to some of the tiles that were being misclassified. We decided to put a diagonal line through ‘O’, which was being misclassified often as ‘D’. I have so far seen 100% accuracy with these characters after this change. Overall, OCR accuracy seems to be around 95% percent, but I am still in the process of gathering some final data.

Another change was the decision to require all 4 corners of the board to be identified before attempting to classify and locate them. Before, if not all 4 corners were found, I was trying to use 2-3 identified corners to estimate the locations of the characters as accurately as possible, however, this did not result in acceptable location accuracy. We have decided requiring all 4 corners identified (which seems to not be an issue so far) is better than mislocating words, which could significantly harm the game state.

Lastly, I’ve been working with Jolie and Denis to get our system integrated together. For example, we recently finished implementing the functionality of taking an image on the RPi and running it through the OCR before sending it to Jolie’s scoring code.

As we enter the final week, we will finish integrating, gather testing data, and finish our poster, video, and report.

Below is one result from testing. The console output is characters identified and their locations. In this test, all characters were correctly identified and located, despite the input image being far from ideal.

During our demo this week, I presented the work I’ve done for the CV / OCR aspects of our project. I’ve managed to get the character location recognition accuracy very high with a reasonably good image. Letter recognition seems to be around 80-85 % but I plan to add characters to the nearest-neighbor database to hopefully increase this. Additionally, I will implement the necessary logic to capture images and process when necessary. Furthermore, I will help with the other work (integration, final presentation, poster, etc) as we enter the final weeks.

“what new tools or new knowledge did you find it necessary to learn to be able to accomplish these tasks? What learning strategies did you use to acquire this new knowledge?”

Computer vision was very new to me, and I was able to familiarize myself with key components of preprocessing, optical character recognition, and even the implementation of a simple nearest-neighbor classification model. This also introduced a somewhat new style of testing, as most of the testing necessary involved manually observing the output of preprocessing stages. Primary learning strategies included reading articles and publications on CV / OCR, looking through code examples similar to what I was trying to accomplish, and simply working hands-on with the techniques.

After trying another OCR library which ended up also being more inaccurate than I would like, I decided to implement my own character recognition. To do this, I added a small database of characters and I perform nearest-neighbor on each isolated character. This seems to result in high accuracy in the testing I’ve done. I also made the logic for mapping characters onto the board more robust, accounting for if fewer than all 4 corner characters are identified (we still need at least 2). The mapping appears pretty accurate but if a tile is close to the edge, it sometimes gets mapped to the adjacent tile. I will try to improve this as much as I can. For the demo, I plan to show the pipeline of image -> preprocessing -> contour recognition and filtering -> individual classification -> mapping onto board.

I was able to make a good amount of progress on the CV this week. I’m at the point where I believe the preprocessing is as good as possible, but Pytesseract still sometimes misclassifies some characters. In the next few days I’m going to try a few different OCR libraries to see if others are more accurate. The logic to map the identified characters onto the board is also almost done.

I’ve spent some time trying to incorporate edge detection into contour recognition for the OCR, but I’m growing convinced that is not the way to go, as it has not seemed to help with recognizing the contours of characters without curves. I instead am beginning to believe I need to instead get the preprocessing as good as possible, as it currently is fairly grainy and has some noise, despite some attempt to reduce it. I’m going to continue improving this next week as well as put together the logic to combine adjacent characters into words.

This week, I primarily spent some time discussing feedback for our design report as well as completing the required individual ethics assignment. Due to having two projects and an exam for my other courses, I did not have capacity to further develop the software of the project much. Next week should however be more available. I hope to improve recognition of curve-less characters by either improving preprocessing or incorporating edge detection.

This week, I primarily worked on the character recognition software. After some research, I found it was likely the best to first run some preprocessing to convert the image to black and white, then isolate and draw onto the original image the contours of each character for better performance of the OCR. Currently, most characters are able to be recognized, but some without curves (L, T, I, etc) are proving more difficult. I plan to further tune some of the preprocessing and contour recognition to improve this. Once the individual characters are identified, I will work on the logic to build adjacent characters into words and eventually map them to the board for score calculation. Additionally, I assisted my team with the design report.

This week, I gave our design presentation. I also helped Jolie setup the RPi and spent some time figuring out how to use the camera. Additionally, I began working with some of the Pytesseract functions, and I expect image_to_data to be particularly useful, as this returns (among other things) the location of the recognized letters. I’m still not completely sure how I’ll map the words onto the slots on the board, but feedback from the design presentation may provide some insight. The idea of color recognition was proposed, although I feel this could have some difficulties, since that might require onboarding an additional CV library, and the colored grids could be covered by tiles anyway.

This week I spent some time looking to wireless communication options, since we recently modified our planned design. I have mostly been researching HTTP Rest API and MQTT as the two primary options for wireless communication. Mosquitto and Paho MQTT are the two libraries I’ve identified as useful, as MQTT is currently our first option. This week, I will be giving our design presentation and will continue to set up the software infrastructure for the project.

I began setting up the codebase for the project. This included creating the github repo and infrastructure for the main file. Additionally, I found a csv of valid Scrabble words and added the code necessary to parse this csv and store them in the internal hashmap to be used in validation. SImilarly, I created the map of letters to scores. I also began looking into the Python libraries necessary to take and process images (cv2) as well as letter recognition (pytesseract). Specifically, I expect functions like image_to_string, image_to_boxes, and image_to_data to be useful. Next week, I will begin figuring out how to implement these methods to achieve basic functionality for taking images and recognizing individual letters. However, most of my focus will likely be figuring out basic modularity of the CV controller and game state.