Posted on

Jeremy’s Status Report for May 1, 2021

This week, I worked with Sid to achieve and verify our design requirements for classification accuracy and latency using the completed PCB.

I improved the convolutional neural network training by adding data augmentation to the training set. Since the camera occasionally shifts between the days, we added data augmentations that choose randomly-shifted crops of the input image to model those physical movements. I applied this data augmentation only to the training set.

Sid and I captured and labelled 2724 new captures with the LED’s from the PCB. Combined with the new data augmentation and a learning rate scheduler, we achieved the best validation accuracy of 99.1%. This model achieved 98.1% accuracy on the test set. This is the best validation accuracy we have achieved, so we are using it for the MVP.

We also measured the classification latency. The design requirements specify 2 second latency from capture to web update. We dealt 52 cards over a 52 second period and achieved the following latency statistics for each card:
min: 0.162s
mean: 0.184s
median: 0.177s
max: 0.224s
The maximum latency of 0.224s is far less than our design requirement of 2 seconds.

I have now completed the imaging and machine learning system. We have some final testing to carry out, but so far, we have achieved every design requirement that we have tested.

Posted on

Team Status Report 4/24/2021

This week was a continual refinement of all the components of our project. We received all the components for our PCB and hope to have that fully assembled early next week. We also continued to work on and train our classification system. This week we achieved 98.0% validation accuracy and 97.5% test accuracy with our CNN. (A card is classified correctly if both the rank and suit are correct.) It takes approximately 50ms to classify a single image on the Jetson Nano, so we have plenty of headroom to achieve the 2s latency requirement. We plan on increasing our training dataset by taking more images this week (hopefully with the PCB). We also made some updates to the logic of our webapp to better support other games besides poker (namely blackjack). We also began work on designing our final, laser-cut, assembly.

We believe we are still on track to complete our project based on our current schedule.

Posted on

Ethan’s Status Report 4/24/2021

We finally got all of our parts for the PCB in this week we began trying to assemble everything but ran into difficulty soldering everything to the board. Some of our components were just too small to hand solder so we likely will have to reflow it.

I also began work on designing a new enclosure for our project:

The plan is to laser cut it out of black acrylic (with the base that the cards sit on our of clear acrylic so the camera and ir sensor can “see” through). Then the plan is to glue the pieces together with epoxy. Once the PCB is completed I’ll begin work on assembling this.

 

 

Posted on

Jeremy’s Status Report for 4/24/21

For the past two weeks, I have worked with Sid collecting a dataset and training different convnets to classify the images.

For training, we have the following data:
Training images: 3481
Validation images: 497
Testing images: 996

The camera returns a 1280×720 image. We crop it to a fixed region-of-interest and downsample by 4 to obtain a 200×148 image. We normalize the training and validation dataset to mean 0 and std 1. That normalized 200×148 image is then passed to the network.

The network architecture is based off on the LeCun-5 network described in LeCun et al., “Gradient-Based Learning Applied to Document Recognition.” It contains 4 convolutional layers with 5×5 kernels, one fully connected layer to output the feature vector, and two separate fully connected layers to output the rank and suit probability distributions. Each convolutional layer is followed by batch normalization and 2×2 max-pooling.

So far, our best network achieves 98.0% validation accuracy and 97.5% test accuracy. A card is classified correctly if both the rank and suit are correct. This network takes approximately 50ms to classify a single image on the Jetson Nano, so we have plenty of headroom to achieve the 2s latency requirement.

Because the PCB is delayed, we are still delayed in testing the images and classification using the final prototype with the PCB. However, Sid and I are on track to hit the 98% accuracy requirement and complete the classification subsystem with our current prototype until the hardware is finished.

Here are details from the training process:

Validation metrics
Card accuracy (suit and rank): 0.979879
Suit accuracy: 0.993964
Rank accuracy: 0.985915

Test metrics
Card accuracy (suit and rank): 0.974900
Suit accuracy: 0.996988
Rank accuracy: 0.976908

This week, Sid and I will take more training images to increase our dataset size. We can experiment with larger networks with more data to hit 98% test accuracy.

Posted on

Sid’s Status Report for 4/24/2021

Since the last status report on 4/10, I made some progress with the web app by adding the necessary logic to Blackjack in case there are multiple winners. In addition, I implemented error handling in case of empty and faulty user input from web app users. An example of faulty input is if a user inputs a negative number of players into the web app. Finally, I implemented authentication in our web app so that only verified users can modify the state of the game through the text forms and buttons. I was able to showcase these updates during the interim demo, but here is a snapshot below for reference.

 

I have also been able to make progress with the machine learning component of our project. Jeremy and I have spent the last few weeks performing the following tasks: collecting image data, labeling the data, writing and modifying Python code to train and test a convolutional neural network (through interfacing with the PyTorch library), and experimenting with various hyperparameters (number of layers, size of kernel, etc). Below are some of my results from my experimentation, but as we get more data, these results are subject to change. K refers to the size of the kernel.

 

  1. Num convolution layers = 4 and K = 3
  • Epoch 56: 93.45% – best validation accuracy
  • Suit accuracy: 97.86%
  • Rank accuracy: 93.57%
  1. Num convolution layers = 5 and K = 3
  • Epoch 58: 86.90% – best validation accuracy
  • Suit accuracy: 98.66%
  • Rank accuracy: 87.67%
  1. Num convolution layers = 4 and K = 5
  • Epoch 48: 95.83%- best validation accuracy
  • Suit accuracy: 98.93%
  • Rank accuracy: 95.71%
  1. Num convolution layers = 4 and K = 7
  • Epoch 56: 83.63% – best validation accuracy
  • Suit accuracy: 96.78%
  • Rank accuracy: 84.99%

 

Overall, I am on schedule. My main future tasks involve gathering more training data and experimenting with our hyperparameters to reach our accuracy requirement.

Posted on

Ethan’s Status Report – 4/10/2021

This week I finished the PCB design! I have a minor change I need to make regarding the input resistance on the base of BJT controlling the LEDs but otherwise it should be good to go! Currently looking to have it fabricated early next week from OSHPark. Based on their lead times and shipping estimates, we should have it by next week for integration into our project (replacing our prototype protoboard).

We also had some issues with our camera module this week that we diagnosed as a hardware defect and have since shipped it back and received a new one. Hopefully, we don’t run into any more issues with the camera!

Posted on

Jeremy’s Status Report for 4/10/2021

This week, I mostly finalized the imaging system and prepared it for the demo. On Monday, we found that the camera was broken (likely due to transportation and connecting/reconnecting). I reordered a camera, and it arrived Friday. Because of that hicop, I am one week behind since I could not collect a dataset for machine learning without a functioning camera. I updated our Gantt chart and used one of our two weeks of slack time to account for this.

On Sunday before this hardware issue, I collected a dataset of ~200 captures to quickly prototype some classifiers. While this is not nearly enough data to train a classifier that generalizes well (~200 captures = 16 captures per rank), it let us bring up our SVM classification code. As expected, we got insufficient validation accuracies, so our next step is to acquire a sufficiently large dataset.

When moving a card over the trigger, the imaging system now returns two captures: the unprocessed 8-bit black-and-white capture and a cropped & thresholded binary image that contains the rank and suit. This represents successful integration with infrared sensor ADC and camera drivers in software. After examining the classification results, I only expect to change the rectangle that crops a fixed region-of-interest out of each capture to zoom in on the rank and suit. The cards can move horizontally, so occasionally the rank and suit are shifted in that ROI. Otherwise, I do not expect to make significant changes to the imaging system.

This week, my first priority is to obtain a sufficiently large dataset for training classifiers. I will work with Sid on SVM and neural network training.

Posted on

Team Status Report for 4/10/2021

We have finished the essential components of our imaging system and web application (although minor modifications might be made in the remaining weeks if necessary). We plan to get our PCB delivered next week. Unfortunately, we were placed a week behind due to turnaround/shipping times with our PCB and since our old camera stopped working (so we had to order a new camera). Our most significant risk is ensuring that our training image data doesn’t change significantly with the PCB design. Making sure our ML is accurate and has been trained on sufficient data will be very important in the coming weeks. To mitigate this risk, we plan to continually obtain training image data as the PCB gets delivered and prioritizing this process.

For our interim demo presentation, our plans have not changed significantly. We still hope to show a working prototype, a remote display with raw and preprocessed captures, and the playing card suit/rank on our web application.

Below is an updated look at our individual and team schedules.

Posted on

Sid’s Status Report for 4/10/2021

This week, I have been working with Jeremy on developing ML code and testing our models. Since I finished writing code to train and test a SVM model (with an RBF kernel), Jeremy was able to train and test this model on a limited dataset. We do not have enough training data, so we achieved very low validation accuracies. The fact that our training accuracy was high further proved that our training data is not comprehensive enough. Since my quarantine is ending soon and my symptoms have gotten better, I plan to go into the lab in person and obtain more card image data to use for training. This is one of my primary goals in the coming week, as we need to achieve higher classification accuracy for whichever model we choose (SVM vs neural network).

 

I have also started writing Python code on configuring a neural network. This has required interfacing with the PyTorch library. I plan to have the code finished by next week as per schedule. However, training the neural network will require even more training data than the SVM since neural networks don’t make modeling assumptions about the underlying data. As a result, finishing the neural network implementation is not as immediately pressing as obtaining training data.

 

I have also achieved considerable progress in making the web application more robust, intuitive, and accurate. To make it more accurate, I had to fix several logic bugs to ensure the card dealing order remained correct and the proper winner is determined at the end of Blackjack and War. I also added the following features: allowing the user to input player names, showcasing which player’s turn it is, highlighting the losers and winners of Blackjack and War, and implementing a sticky left column in our player table so that it remains constant when the user scrolls horizontally. This last feature is especially important in War, as there are up to 52 cards in a player’s hand, so users would need to scroll horizontally to view all the player’s cards. Lastly, I added some CSS styling to make the UI more elegant. In terms of UI styling, there is not much I need to do for the web app. With regard to next steps, I recently realized I need to implement the necessary logic for Blackjack in case there are multiple winners. In addition, I need to be able to handle empty and faulty user input from web app users (ex: inputting a negative number of players). This error handling is another one of my minor goals for the coming week. The feature I want to prioritize for next week, however, is implementing authentication in our web application to ensure only verified users can make changes to the web app. In a real-life professional poker setting, only casino officials and tournament officials should be able to input information into the web app (not audience members), so this level of security is necessary.

 

Overall, I am on schedule with my tasks. Below is an updated look at the web app (the password text field does not have any underlying logic yet, but I will work on that this week).

 

Posted on

Ethan’s Status Report – 4/3/2021

This week we’ve pretty much finalized the physical design of our project. We began work on creating the first fully-realized prototype. We found that the way we created the cutout and blacked-out the rest of the frame gave jagged edges and impacted how much of the card we could get into the image field. So we started work on modifying a new card shoe (which we ordered this week). This time we painted the underside instead of using black tape:

 

 

We will likely drill holes tomorrow for both camera cutout and the standoff locations for the pcb. Speaking of the PCB, I’ve begun work on a first draft. I hope to have something ready to order early this week so it can be shipped and return in time for our interim demo in two weeks!