Posted on

Team Status Report for 10/7/23

The only major changes made to the design were the addition of preprocessing algorithms to the computer vision. We knew that we were going to have these, but we weren’t sure exactly what algorithms we were going to use. Upon receiving the user’s image, we grayscale it, then do some simple thresholding to get rid of the effects of different lightings and shadows and the grain of the paper. Then, we do some aggressive median blurring that will eliminate all drawing marks except for the dark, filled in circles representing nodes. This has been the major change and success that has allowed our use of Hough circles to work (see Stephen’s report for more information). 

This was the first week since we changed to an iOS application from a web application. A lot of the early stages of the week was researching Swift and looking through examples to develop some of the UI for the app. The front page of the app has been completed and the page that uploads the photo needs just a little more time to be completely finished. The next step will be to feed the picture of the uploading hand-drawn image into the computer vision algorithm. 

The schedule is the same except for Stephen’s schedule rework (see below). The circuit simulator is running a bit behind due to Devan getting sick, but he will work over fall break to make up for the lost time. Jaden will also be working on the phone application throughout fall break.

Two engineering principles that we used to develop our design are linear algebra and image processing. For our circuit analysis tool, we need to run nodal analysis on nearly every single node in the circuit. This involves setting up a system of equations that models the voltages and currents going into and out of each node. By calculating equations for each node, we then will solve the matrix system of equations that represents the circuit. We also need to process the images that users input in order to make them easier for our computer vision algorithm to recognize. The images go through a grayscale filter, a binary threshold filter, then a blur in order for our algorithm to detect the nodes at the end of each component.

Posted on

Stephen Dai’s Status Report For 10/7/23

This week I spent much time and made good progress on the computer vision portion of our project! Originally in our schedule, I am to first do the individual component detection, but this week I opted to complete the full circuit detection in which I parse the original input image to generate the sub images for each component. I opted to do this because this will make it easier for me to test the individual circuit detection because I can upload drawings of full circuits instead of having to take individual photos of components. Also, by doing this I generate photos of the components that are more appropriately sized.

Here is the sub-image generation in action:

First the original image is greyscaled:

Then do some thresholding:

Then apply a median blur (see how just the nodes are left!):

 

Then use Hough Circle transform to find all the nodes (circles in white):

Then (skipping some steps here) generate individual component images by calculating bounding boxes for each edge (connection between nodes):

  

I also implemented the brute force feature matching to figure out the best matched component given an input image and a dataset of images. It actually worked better than I expected, successfully classifying 3/3 sub images, but obviously I need to do more testing. There is definitely a lot of stuff I want to tune still, and I want to try adding some blurring and more image preprocessing to see if the results are more definitive.

I am definitely on schedule. I purposely worked a lot this week because next week I have several midterms that will eat up my time, as well as the design report.

My next steps are to integrate the sub image generation and component detection so that I can work on the full-circuit re-creation and netlist generation. After this infrastructure is completed, I will start to fine tune parameters and whatnot in each step of the process.

Posted on

Stephen Dai’s Status Report for 9/30/23

This week I mostly worked on the design presentation and finalizing our algorithm design for the computer vision as well as the user workflow. For the presentation I made these two diagrams to illustrate this:

 

We made a lot of pretty huge changes to our overall design, like creating a mobile application instead of a web application, and changing how the user interacts with our application. I designed how the user interacts with our application in steps as seen in the diagram. I also designed how our backend computer vision module works, also as seen in the diagram.

I also tried to work some more on getting the segmentation for getting individual components from the user image which Devan was working on last week. Instead of using Hough circle algorithm, I tried using Hough line algorithm and having intersections of the lines be classified as nodes. What I realized was that it was just not good enough: hand-drawn lines are too wavy for the algorithm and it detects lines that make up an actual component, like a resistor. So we are definitely sticking with Hough circle now.

Unfortunately I did not have time to work on the individual component detection as I spent much time on the design presentation. Next week I plan to work vigorously on improving the feature matching. I would say that I am a bit behind schedule than what I would like, but I am confident I can makeup the work next week.

The classes that I have taken that correlate to the principles learned for our design are 15112, 15122, 17437, and 18330. A class I have not taken that significantly impacts our design is a computer vision and machine learning related course. What I have been doing for self-learning is reading many computer vision and hand-drawing detection research papers and utilizing openCV documentation. Luckily there is a lot online and openCV has tutorials that have proven quite useful for me.

Posted on

Jaden Dabreo’s Status Report 09/30/23

This week I spent most of my time finishing all the frontend portions of the initial web application. I finished the upload page earlier in the week, however I extended the scope of the website after discussing with my teammates to help people that are using the tool for the first time. Hence, I added a FAQ page that explained basic necessities that the simulator will need, such as darkening nodes for detection and attempting to draw the connections straight. All the work I have done has been purely software work and research. Last semester I took a class on web development, Web Application Development (17-437), to write the frontend code of the web application. However, our group came to a consensus that it would be better to use a mobile application instead of a web application due to the specifications of our use cases. Because of this, my work for this week will be discarded and I will have to put in more hours next week to get started on the development of the mobile application. I also looked through computer vision research papers in the hopes of understanding how to justify our accuracy expectations. After our presentation, one piece of feedback we received was that our accuracy expectations were not justified. We were told the best course of action would be to look through computer vision research papers and understand on what basis they quantified their accuracy expectations. Through my research I was not able to find anything of substantial help. Most papers did not mention any accuracy expectations and the papers that did were models building upon previous algorithms that already had a score, thus the expected accuracy expectations were greater than the previous algorithms accuracy score. Currently, I am behind on work. Because of the group’s decision to move to a mobile application, I will have to put back the work I had planned for this week and focus on researching mobile development. I plan to put in an extra hour a day this week, potentially more, in order to get back on track.

 

(Header of home page)

Posted on

Team Status Report for 9/30/23

One of the most significant risks in our project lies within the accuracy of our computer vision algorithm. We plan to combat this by using a suggestion box after analyzing a circuit – this box will display the five circuits that our algorithm has the highest confidence in. From these circuits, the user can choose which circuit is the one that they drew. By doing this, we are able to achieve higher levels of accuracy without needing to perfectly detect every component. We also are moving the labeling of each component value to the application instead of in the drawing. Rather than having users write the value associated with a component right next to it, we will have them enter in the relevant information once the circuit has been loaded into the applications. This way our computer vision algorithm will not have a hard time trying to figure out which value is associated with each component.

Changes were made in regards to our initial idea of creating a web application. After discussion, our group came to a conclusion that using a mobile application would make more sense given our specified use case of accessibility for middle schoolers. Due to this, all the work done on the web application will be discarded and next week Jaden will be focusing on learning and developing the mobile application instead. As around a week of work has been voided, more hours must be put in to make up the time that was lost over this past week.

We have also been spending lots of time on creating our design presentation – we made many changes since our proposal presentation that had to be added to the design presentation. While creating the presentation, we did lots of research on other computer vision algorithms that were used in similar applications to figure out how to justify our accuracy expectations. Between the team over 20 research papers were looked into in regards to this issue. Papers included sentence recognition, electronic component recognition, and other computer vision models. We ultimately decided that for computer vision we will utilize the Hough circle algorithm for image segmentation, ORB transform for feature detection, and a Brute Force matcher for feature matching.

We ended up making many changes to our project in the past couple of weeks that we believe will help mitigate risks that will occur with our computer vision algorithm and with having a complex backend. Even though we wasted a week on the web application, since we do not have to setup a complex backend we are still on track with our schedule and are making good progress with our project.

Posted on

Devan Grover’s Status Report for 9/30/23

I did research on possible computer vision algorithms, netlists, and worked on the design presentation this week. I did more research into netlists since most of our research was focused on computer vision last week. Netlists are commonly used in various programs such as SPICEand SystemVerilog to generate schematics of how various components are connected to each other. I plan on using a hierarchical, object oriented system in which all components share their properties – this way I do not need to handle extraneous cases for components that act differently from other components (eg. diode vs resistor). I wrote basic code to connect two resistors together to test this concept.

I also researched computer vision algorithms specifically for our design presentation. During our discussions with Professor Yu and Mukundh this week, we realized that we need to dig deeper into the computer vision aspect of our project and flesh out the techniques we will use. I read through many papers that tried to solve similar problems to our computer vision algorithm, and we all collectively decided on using ORB for feature detection and a brute force matching algorithm.
Since I am presenting the design presentation next week, I also spent time writing up scripts and notes to look at while practicing my presentation. In order to ensure I give the best presentation possible, I want to prepare and practice in front of real people. By creating notes ahead of time, I can review these and refine my presentation skills after presenting multiple times to my roommates.

I am a little bit behind schedule. I had three interviews this past week that took up lots of my time, and I was therefore unable to do as much as I originally planned in terms of researching netlists. I did not write very much code for the netlist, but since I only have one interview next week I will be able to devote much more time to this project and therefore will be able to catch up. I hope to complete a netlist with more than just two components and create more than one type of component.

In my work this week I used concepts from 18-220 for the circuit simulation. I have not taken a computer vision course, so I have been reading up on many guides and papers on the subject in order to better understand the concepts and educate myself. OpenCV also has many guides and tutorials online that I have been trying to do and follow along with to get a better grasp on computer vision.