Charlie: I was able to adapt a model I found online to train the CNN, resulting in parameters that had a training and test accuracy of over 95%. I also assisted with the conversion of the prediction function to Java, and worked on logistical things, such as the poster and report.
Mitchell: I completed the digit segmentation wrapper in Python after running into some issues with flow of the project. Since the code was going to be converted to Java, I was careful about doing things in Python that I knew existed in Java. This also meant that I had to develop a good logical flow of parameters as Java does not allow for passing tuples around like Python does. I also helped Charlie write a ParamParser class in Java that takes the CNN Python output (stored in a .txt file) and translates it into Java arrays.
Estevon: Most of my time involved converting Charlie’s new Prediction script to Java. Since Java is strongly typed and does not offer a native scientific computation library similar to NumPy, this involved a fair amount of work. I also spent time testing it and integrating it within the application in time for the demo.
Date: 5/3/2018
Blog number 8
Charlie: I trained the model on the full dataset a few times, but it is currently prohibitively slow and not as accurate as we would like. I plan to look into libraries to build the training model and then do the prediction separately, under Prof. Savvides’ suggestion.
Mitchell: The digit segmentation wrapper is almost completed, and just needs a little bit more work. It is currently able to grab a clip of the image (which includes a single digit). After completing the wrapper, I will convert it into Java and integrate it with our final product.
Estevon: I have been working on making last-minute touches to the front-end design. I also have been busy integrating Charlie’s and Mitchell’s parts within the application.
Date: 4/26/2018
Blog number 7
Charlie: I attempted to integrate the GPU with my program, but due to memory overhead and other things it did not prove to be a useful tool for speeding up the training process. However, I did discover some vectorization tricks to improve the speed of some functions, and am attempting to implement those right now.
Mitchell: I am still working on creating a wrapper function to separate a multi-digit number into separate images which can be passed into the CNN. Hopefully I can finish this by the end of this week so that it can be integrated into the project before the demo on Wednesday.
Estevon: I have finished converting part of the CNN to Java and part of Mitchell’s code to Java as well. I am still making improvements here and there to the user interface of the Java application. I am finishing the scoreboard which will display right after the student finishes submitting all of their answers. I will slightly modify the login and option screens as well.
Date: 4/19/2018
Blog number 6
Charlie: I was able to fix the problems I had with the CNN earlier and now it runs smoothly, so I am now working on improving its accuracy. I am also looking into possibly integrating my computer’s GPU into the CNN program in order to train the model faster to do testing more efficiently.
Mitchell: I was able to fix the issue with tracing around the boundary of an image. As opposed to last week, when I tried to find neighbor black pixels, I instead used Moore’s Contour Tracing Algorithm to search in a circle around the current pixel starting from the direction that the current pixel was entered from and ignoring pixels that were already visited. Once the tracer returns to the first pixel that I started from, then the program terminates. Furthermore, I solved a pre-termination issue that occured when a pixel was only connected to one other pixel (therefore only one entrance/exit point). An example of this can be seen in the image below. On row 7, the left-most pixel is isolated save for at one point. From the image below, we have a completed outline traced by the algorithm described. The next step will be extending this to using the outline to segment multi-digit numbers, which will be completed by next week.
Estevon: I have restyled the option screen. I am still working on the scoreboard layout design to make it more appealing to the client. I am finally in process of converting the CNN to Java code. I will continue to modify in the case that Charlie improves the CNN efficiency. I will soon be converting the tracing component to Java as well.
Date: 4/12/2018
Blog number 5
Charlie: I finished putting together the pieces of the CNN and got it to run, but ran into some problems with accuracy and some of the more technical details of the CNN. I am continuing to look into more sources to address these problems, as well as do additional testing.
Mitchell: I created some python functions that help us to pre-process the images we receive. For digit segmentation, the fall-back option is for users to space out multi-digit answers in a way so that we can detect the vertical white space in between digits. I developed an edge tracing algorithm on my own, that attempts to trace through a digit pixel-by-pixel and finds neighboring black pixels. Unfortunately, this did not work as intended. I am currently considering two other approaches. In one approach, we can trace around the outside of a digit by trying to straddle the area where black and white pixels meet (at the border). Alternatively, we can follow Moore’s algorithm for contour line tracing. Hopefully these can both be implemented and tested by next week.
Estevon: I did further work on the front end. I remade the work space so that it is much more UI-friendly for the client. I added outlining for the word problem. I also implemented an internal time which will work the time mode. The client will now see visible erase, color switching and help options right above the canvas. I redid the login screen to include some images with CSS fonts. I am now working on adding additional style such as a progress bar glyph for the main functionality page. I will also restyle the screen where clients go over options such as the time mode and topic choice.
Date: 4/5/2018
Blog number 4
Charlie: I finished doing the mathematical derivations for the CNN and began to translate the math into code. I finished writing the modules for the different stages of the CNN and am currently working on integrating them all together.
Mitchell: I created a python image resize wrapper that calls the resize function from the cv2 library. This was largely because there was no need to create our own function. Since Java also has an opencv library, there is also an image resize function within it. I am currently working on a digit tracing algorithm that will allow us to segment multi-digit numbers and this should be ready to demo on Monday.
Estevon: I have started working on trying to implement a practice mode and a timed test mode. I have created classes to represent the word problems with a field for acceptable solutions. I am beginning to improve the front-end to be more appealing to children. I am also working on getting scoring to display at the end when the user is done with the exercise.
Date: 3/29/2018
Blog number 3
Mitchell: After spending a lot of time on the MATLAB simulation, I finally got the network to train and classify some digits! Unfortunately, the digits were being recognized as nematodes (worms) because Alexnet is pre-trained on a vast variety of images of objects. Although I wish I had realized this earlier, I did learn a lot about how CNNs work and about some of the parameters that are passed into it. Starting Wednesday, I jumped ship to start working on Charlie’s python implementation of the CNN. With this implementation, I am currently working on resizing the images that we receive from the Surface 3 as well as thinking about ways to segment multi-digit numbers.
Charlie: I finished implementing the CNN digit recognition tutorial, which we can use as a test for integrating with the other parts of our application. I continued doing research and began implementing the actual CNN that we plan to use for our application.
Estevon: I have finally created a database with two separate tables. One table holds user and password information. Another table holds pre-defined math word problems as well as their answers. I managed to successfully create a login authentication mechanism; only valid credentials will permit the client to play with the application. I also have set up the part where the application fetches from the database based on the topics chosen. I coded a workspace in which the user can draw their answer in a small whitebox. I am now working on having the code take a screenshot of this section of the screen so Mitchell is able to invoke his digit recognition routines on it.
Date: 3/22/2018
Blog number 2
Mitchell: I continued working on a MATLAB simulation to understand neural networks by using MATLAB’s built-in Alexnet. Through it, I learned about a few of the different parameters that a neural network takes in, such as the batch size and epochs. I then trained alexnet on a reduced sample size of 200 images to cut down on training time. The next step will be for the network to correctly classify images as a proof of concept, and to figure out ways to speed it up.
Charlie: I began to implement a tutorial for CNN digit recognition that we plan to use as our fall-back in case implementing the actual CNN ends up being more problematic than expected. Additionally, I also did more research into the mathematical details of CNNs and began to work on building a CNN independent of external frameworks.
Esteban: I have started working on basic functionality and design for the GUI. I
set up a login screen as well as another screen which students may select an arbitrary
math topic. I have started setting up the screen in which students will look at the question and submit an answer. I also looked into creating a MySQL database for the next part of the project so that registered students may successfully login.
Date: 3/8/2018
Blog number 1
Mitchell: This week, I helped to write the design report. I also continued researching methods to recognize digits and deskew them before passing them into the neural net. I also created a more granular schedule for our team to follow for the rest of the semester.
Charlie: After our team decided to use convolutional neural networks for the digit recognition part I did more research on them, both on the technical details as well as how to actually implement them for our project. I discovered a few pre-existing machine learning and neural net libraries that we could possibly use as a backup. I also read a few papers that discussed various architectures for CNN design, especially in the area of digit recognition. Other than that, I helped write the design report.
Estevon: I have decided to start sketching out the front-end design for the application and have begun to slowly set up the framework for it. I have looked at third-party libraries to help with the Python-to-Java conversion when it will come. I helped set up the blog website. Besides that, I have contributed to the design report with respect to the software development component.
