Tag: status report

This week I began creating integration code and continued to work on improving accuracy requirements. One of the things I did was I wrote a script to turn a directory of component images into a single YML file that can be parsed. Before/for the demo every run I would re-run image preprocessing and feature detection on every image in the directory, but now I have made it so that you create the YML file once and it can be parsed for the already-calculated feature vectors. The dataset file is bigger than I had anticipated and mentioned in the design report. In the report I mentioned that we would expect the file to be 500 KB large with a 50 image dataset. Right now with a 27 image dataset, the file is 1.4 MB, which means we can expect our 50 image dataset to be 3 MB large. Although this is larger than we anticipated, this is still plenty small. With the YML file there is added overhead because of metadata that makes it easier to parse the file like a dictionary/map lookup, so we are okay with this size tradeoff.

I have also started doing testing and polishing of accuracy. I ran a test on 66 component images, and 64 of them were identified correctly (~97% accuracy)! This statement isn’t exactly true though, because 42 of the images were ones that had orientation associated with it (voltage + current sources, diodes), and only 24 of those were identified with the correct orientation. Besides the difficulty in classifying the correct orientation of those components, I also noticed that current sources and voltage sources would have very similar matching scores (sometimes they had the same score but it just so happens the correct one was put as the first choice). As a result of this, one thing I want to experiment with is trying to use SIFT instead of ORB for feature detection. Because orientation actually matters, it actually makes sense to use SIFT now, so this is definitely something I want to try next week.

Last week I said that I wanted to improve the node detection, but I realized in testing this week that it actually performs pretty well. I played around with some parameters and it worked consistently.

My next steps are to continue working on the individual component accuracy and the full circuit accuracy as well. By the next report I want to have a complete dataset that will be used, and the accuracies will hopefully be in the ballpark of 80-90%.

I made A LOT of additions/changes to the computer vision code this week. Here is an overview:

1. If a node is detected but doesn’t have at least two detected neighbors, then this node is removed. This is done to remove stray circles that might be detected as nodes. By doing this, we err on the side of missing nodes instead of identifying fake nodes/circles. To account for this, we make the Hough circles transform less strict so it is more lenient in its circle detection.
2. I changed the way that component subimages are generated. Before I created a hard-coded size box that had X by Y dimensions based on the coordinates of its nodes. Now, I do the following. I create a straight line from one node to the other. I iterate the line upwards + downwards / left + right (depending on if the component is horizontal or vertical), and I stop iterating once there are no black pixels in the line. Then I add a fixed value of 20 pixels, and now I have a box that is guaranteed to encompass the entire component.
3. I finally found a good combination of preprocessing functions to run on component sub images. I spent so much time on this trying to find a way to remove shadows and the grain of the paper from the images, which were seriously messing up the individual component classification. The combination I ended up with was median blurring -> non-local means denoising -> adaptive thresholding -> median blur. From my testing this combination does really well in removing grain from the paper (noise) and smoothing out shadows. I don’t think I will stray from this combination besides fine-tuning parameters in the future.
4. Before feature detection and matching, I have added an additional step. I run Hough circle transform on the processed subimage in order to determine if the component symbol is one with a circle (voltage source, current source, lightbulb). If it is, then feature matching is only performed with these components in the dataset. If it is not, then the component must be a wire, resistor, switch, or LED. I then perform probabilistic Hough line detection. I then look for the maximum and minimum X / Y coordinates (depending on if the component is vertical/horizontal) and see what the difference is. If the difference is small (less than a third of the subimage width/height), then the component must be a wire. Else, the component must be either a resistor, switch, or LED. I did these things because the individual component detection was quite sad. Sometimes a wire would get classified as a voltage/current source, which made no sense. I figured that because wire’s are so much different/simpler than every other component I could case specially on them and not even require feature matching for them.

The improvements I want to make next week are to make the node detection more consistent even with shadows. I think I will experiment with new preprocessing algorithms like what I did with the subimage preprocessing. The other thing I want to try to improve is the feature detection by tweaking some parameters.

Test that I have run and are planning to run look like this:

Node detection/subimage generation: I take images of drawn circuits and then feed them into my subimage generator. I validate that the detected nodes correspond to the drawn nodes by showing an image of detected circles overlaid on top of the circuit image. I also visually validate that the subimages generated properly encompass the entire component.

Component classification: In order to determine what image preprocessing algorithms I wanted to use, I tried many different combinations of algorithms and outputted/showed the image resulting of running each algorithm. This way I could intuitively tune what algorithms to use and which parameters I needed to change. In order to validate the output, I print the top three component classifications. Based on our use-case requirements, the top choice should be the correct classification 90% of the time.

Full circuit classification: I run everything together and print the output of the circuit classification, which is a list of the five best circuit classifications. Based on our use case requirements, one of the five classifications should be correct 90% of the time.

Next steps with my testing is to actually measure these results for a quantitative measure of how well my subsystems are performing. For the individual component classification, I will work with a testing dataset of 50 drawn components from our user group, and see if we hit the 90% mark. For full circuit classification, I will work with a testing dataset of 28 different circuit images from our user group and see if we hit the 90% mark as well.

I finally finished the python -> C++ code conversion, and I am happy to say that the code is ready to be demoed! The things that I converted this week were the dataset parser file (which currently reads from images in a folder), the individual component classifier file, and the main file that classifies the entire circuit.

Tomorrow I will be experimenting with running the code on either Jaden or Devan’s laptops for the demo. I have been working on a Linux machine, and the problem with that is I don’t have a GUI installed that can show images, which I want to do for the demonstration (so I am not just showing std output). Also it would be much better if we could just run all of our code off of one machine anyways instead of switching computers every time someone presents their part.

The steps  for next week will to be to start working on testing and improving the accuracy of the circuit and component detection. The other thing is to also start working on integration with the mobile application. This will require creating a bridge file such that the C++ classes I made can be used in Swift. I also need to do a little bit of redesigning of the code, such as with the dataset parser. Right now I have it so that every time you run the program, the dataset parser will take each image in the dataset directory and generate the keypoints and descriptors. What we want to do is have dataset be represented by just one file that already has the keypoints and descriptors. This will honestly be a pretty easy change and coding this function will probably only take an hour or two max. I will also probably make this one of the last things I do because the dataset is not set yet.

I am on schedule. I foresee that maybe the improvement of the accuracy can potentially overflow into next next week as well, but we have given a slack period in our schedule that can account for this.