Posted on

Team Status Report for 10/7/23

The only major changes made to the design were the addition of preprocessing algorithms to the computer vision. We knew that we were going to have these, but we weren’t sure exactly what algorithms we were going to use. Upon receiving the user’s image, we grayscale it, then do some simple thresholding to get rid of the effects of different lightings and shadows and the grain of the paper. Then, we do some aggressive median blurring that will eliminate all drawing marks except for the dark, filled in circles representing nodes. This has been the major change and success that has allowed our use of Hough circles to work (see Stephen’s report for more information). 

This was the first week since we changed to an iOS application from a web application. A lot of the early stages of the week was researching Swift and looking through examples to develop some of the UI for the app. The front page of the app has been completed and the page that uploads the photo needs just a little more time to be completely finished. The next step will be to feed the picture of the uploading hand-drawn image into the computer vision algorithm. 

The schedule is the same except for Stephen’s schedule rework (see below). The circuit simulator is running a bit behind due to Devan getting sick, but he will work over fall break to make up for the lost time. Jaden will also be working on the phone application throughout fall break.

Two engineering principles that we used to develop our design are linear algebra and image processing. For our circuit analysis tool, we need to run nodal analysis on nearly every single node in the circuit. This involves setting up a system of equations that models the voltages and currents going into and out of each node. By calculating equations for each node, we then will solve the matrix system of equations that represents the circuit. We also need to process the images that users input in order to make them easier for our computer vision algorithm to recognize. The images go through a grayscale filter, a binary threshold filter, then a blur in order for our algorithm to detect the nodes at the end of each component.

Posted on

Stephen Dai’s Status Report For 10/7/23

This week I spent much time and made good progress on the computer vision portion of our project! Originally in our schedule, I am to first do the individual component detection, but this week I opted to complete the full circuit detection in which I parse the original input image to generate the sub images for each component. I opted to do this because this will make it easier for me to test the individual circuit detection because I can upload drawings of full circuits instead of having to take individual photos of components. Also, by doing this I generate photos of the components that are more appropriately sized.

Here is the sub-image generation in action:

First the original image is greyscaled:

Then do some thresholding:

Then apply a median blur (see how just the nodes are left!):

 

Then use Hough Circle transform to find all the nodes (circles in white):

Then (skipping some steps here) generate individual component images by calculating bounding boxes for each edge (connection between nodes):

  

I also implemented the brute force feature matching to figure out the best matched component given an input image and a dataset of images. It actually worked better than I expected, successfully classifying 3/3 sub images, but obviously I need to do more testing. There is definitely a lot of stuff I want to tune still, and I want to try adding some blurring and more image preprocessing to see if the results are more definitive.

I am definitely on schedule. I purposely worked a lot this week because next week I have several midterms that will eat up my time, as well as the design report.

My next steps are to integrate the sub image generation and component detection so that I can work on the full-circuit re-creation and netlist generation. After this infrastructure is completed, I will start to fine tune parameters and whatnot in each step of the process.

Posted on

Stephen Dai’s Status Report for 9/30/23

This week I mostly worked on the design presentation and finalizing our algorithm design for the computer vision as well as the user workflow. For the presentation I made these two diagrams to illustrate this:

 

We made a lot of pretty huge changes to our overall design, like creating a mobile application instead of a web application, and changing how the user interacts with our application. I designed how the user interacts with our application in steps as seen in the diagram. I also designed how our backend computer vision module works, also as seen in the diagram.

I also tried to work some more on getting the segmentation for getting individual components from the user image which Devan was working on last week. Instead of using Hough circle algorithm, I tried using Hough line algorithm and having intersections of the lines be classified as nodes. What I realized was that it was just not good enough: hand-drawn lines are too wavy for the algorithm and it detects lines that make up an actual component, like a resistor. So we are definitely sticking with Hough circle now.

Unfortunately I did not have time to work on the individual component detection as I spent much time on the design presentation. Next week I plan to work vigorously on improving the feature matching. I would say that I am a bit behind schedule than what I would like, but I am confident I can makeup the work next week.

The classes that I have taken that correlate to the principles learned for our design are 15112, 15122, 17437, and 18330. A class I have not taken that significantly impacts our design is a computer vision and machine learning related course. What I have been doing for self-learning is reading many computer vision and hand-drawing detection research papers and utilizing openCV documentation. Luckily there is a lot online and openCV has tutorials that have proven quite useful for me.

Posted on

Jaden Dabreo’s Status Report 09/30/23

This week I spent most of my time finishing all the frontend portions of the initial web application. I finished the upload page earlier in the week, however I extended the scope of the website after discussing with my teammates to help people that are using the tool for the first time. Hence, I added a FAQ page that explained basic necessities that the simulator will need, such as darkening nodes for detection and attempting to draw the connections straight. All the work I have done has been purely software work and research. Last semester I took a class on web development, Web Application Development (17-437), to write the frontend code of the web application. However, our group came to a consensus that it would be better to use a mobile application instead of a web application due to the specifications of our use cases. Because of this, my work for this week will be discarded and I will have to put in more hours next week to get started on the development of the mobile application. I also looked through computer vision research papers in the hopes of understanding how to justify our accuracy expectations. After our presentation, one piece of feedback we received was that our accuracy expectations were not justified. We were told the best course of action would be to look through computer vision research papers and understand on what basis they quantified their accuracy expectations. Through my research I was not able to find anything of substantial help. Most papers did not mention any accuracy expectations and the papers that did were models building upon previous algorithms that already had a score, thus the expected accuracy expectations were greater than the previous algorithms accuracy score. Currently, I am behind on work. Because of the group’s decision to move to a mobile application, I will have to put back the work I had planned for this week and focus on researching mobile development. I plan to put in an extra hour a day this week, potentially more, in order to get back on track.

 

(Header of home page)

Posted on

Team Status Report for 9/30/23

One of the most significant risks in our project lies within the accuracy of our computer vision algorithm. We plan to combat this by using a suggestion box after analyzing a circuit – this box will display the five circuits that our algorithm has the highest confidence in. From these circuits, the user can choose which circuit is the one that they drew. By doing this, we are able to achieve higher levels of accuracy without needing to perfectly detect every component. We also are moving the labeling of each component value to the application instead of in the drawing. Rather than having users write the value associated with a component right next to it, we will have them enter in the relevant information once the circuit has been loaded into the applications. This way our computer vision algorithm will not have a hard time trying to figure out which value is associated with each component.

Changes were made in regards to our initial idea of creating a web application. After discussion, our group came to a conclusion that using a mobile application would make more sense given our specified use case of accessibility for middle schoolers. Due to this, all the work done on the web application will be discarded and next week Jaden will be focusing on learning and developing the mobile application instead. As around a week of work has been voided, more hours must be put in to make up the time that was lost over this past week.

We have also been spending lots of time on creating our design presentation – we made many changes since our proposal presentation that had to be added to the design presentation. While creating the presentation, we did lots of research on other computer vision algorithms that were used in similar applications to figure out how to justify our accuracy expectations. Between the team over 20 research papers were looked into in regards to this issue. Papers included sentence recognition, electronic component recognition, and other computer vision models. We ultimately decided that for computer vision we will utilize the Hough circle algorithm for image segmentation, ORB transform for feature detection, and a Brute Force matcher for feature matching.

We ended up making many changes to our project in the past couple of weeks that we believe will help mitigate risks that will occur with our computer vision algorithm and with having a complex backend. Even though we wasted a week on the web application, since we do not have to setup a complex backend we are still on track with our schedule and are making good progress with our project.

Posted on

Devan Grover’s Status Report for 9/30/23

I did research on possible computer vision algorithms, netlists, and worked on the design presentation this week. I did more research into netlists since most of our research was focused on computer vision last week. Netlists are commonly used in various programs such as SPICEand SystemVerilog to generate schematics of how various components are connected to each other. I plan on using a hierarchical, object oriented system in which all components share their properties – this way I do not need to handle extraneous cases for components that act differently from other components (eg. diode vs resistor). I wrote basic code to connect two resistors together to test this concept.

I also researched computer vision algorithms specifically for our design presentation. During our discussions with Professor Yu and Mukundh this week, we realized that we need to dig deeper into the computer vision aspect of our project and flesh out the techniques we will use. I read through many papers that tried to solve similar problems to our computer vision algorithm, and we all collectively decided on using ORB for feature detection and a brute force matching algorithm.
Since I am presenting the design presentation next week, I also spent time writing up scripts and notes to look at while practicing my presentation. In order to ensure I give the best presentation possible, I want to prepare and practice in front of real people. By creating notes ahead of time, I can review these and refine my presentation skills after presenting multiple times to my roommates.

I am a little bit behind schedule. I had three interviews this past week that took up lots of my time, and I was therefore unable to do as much as I originally planned in terms of researching netlists. I did not write very much code for the netlist, but since I only have one interview next week I will be able to devote much more time to this project and therefore will be able to catch up. I hope to complete a netlist with more than just two components and create more than one type of component.

In my work this week I used concepts from 18-220 for the circuit simulation. I have not taken a computer vision course, so I have been reading up on many guides and papers on the subject in order to better understand the concepts and educate myself. OpenCV also has many guides and tutorials online that I have been trying to do and follow along with to get a better grasp on computer vision.

Posted on

Jaden Dabreo’s Status Report 09/23/23

This week our group spent most of our team researching and testing CV algorithms for image, feature, and edge detection. I spent time testing different algorithms with two hand drawn resistors, a horizontal and vertical wire, and a voltage/current source. My research included implementing mean squared error, Haris Corner detection, and template matching algorithms. Mean square error seemed to have the lowest error rate when comparing two individual components, however implementing with complex circuits would be too inefficient to run with the WebApp. My testing included taking and comparing each hand drawn electrical component previously listed and comparing the error rate with one another to see which pairings produced the smallest error. The algorithm did work well, however is not scalable with the size of the project. The other two algorithms were not as effective with single component detection and I therefore did not pursue any further research. From there I created the WebApp for our website and started working on the upload page. The upload page is planned to be completed by early this week and I plan for it to be completed by Monday evening. Currently I am on schedule as I need to complete the upload page by Monday and continue to help with research. Next week I want to look at number detection to help with other CV components outside the circuit itself, reading the values written next to each electrical component.

Posted on

Devan Grover’s Status Report for 09/23

This week I explored computer vision algorithms to separate the components from the rest of the circuit. In order to properly do this, I came up with the idea to have circular nodes at the end of every component’s two terminals. Then, I will use an algorithm to detect circles such as the Hough Circle’s algorithm in order to find these nodes. Next, each node that is found will have an algorithm run upon it in which it will find the closest nodes in the same x and y coordinates. Then, a bounding box will be drawn between the sets of two dots and this box will be looked at to find a component.

I tried to implement this with a simple picture of a resistor but was unable to get it to work. The circles were not being detected in the image. I tried to run a simple filter on the image to detect the components being drawn. I made every pixel that had a black value of over a threshold value completely black, and every pixel with a lower black value completely white. This also did not work well in differing light conditions like the picture below.

Furthermore, the circles on the end of the resistor were not even detected. I believe I need to look further into the problem because we need to make sure we can detect components separately from the whole circuit. I tried a contour plot which detected edges relatively well, but the nodes on each component were not filled in because the contour plot only detects edges. This is problematic because we will have other circular components like voltage and current sources that we will need to ensure By individually identifying components and wires with this technique, we can easily build a netlist for our circuit simulator. I also briefly explored the concept of netlists and how they work so that we can implement it for our circuit simulator.

I am still on track with the schedule because I am still researching CV algorithms. I want to have a picture of a resistor with the circles on its ends being detected properly by our circle detection algorithm. This will prove that we can go on with the technique I discussed above to detect individual components.

Posted on

Team Status Report for 9/23/23

One of the features that could jeopardize the progress of our project is our CV algorithm. Majority of our group’s time spent this week was researching and testing different types of image, feature, and edge detection algorithms on hand drawn circuit components. While we have made progress honing in on the type of algorithm we want to use, there is still a lot of uncertainty that needs to be solved. Since we have just been researching this week no changes have been made to our initial plan. The block diagram needs to be modified slightly. What we did not account for was our CV algorithm needing to interact with the database in order to pull training and validation data. This isn’t going to incur any extra costs or difficult refactoring because we knew we would be using a database anyways. Besides this, there are no changes to what we discussed in our proposal presentation.

 Next week we would like to start testing algorithms that can separate electrical components given a whole circuit and keep experimenting with individual component detection. When researching this past week, we realized that trying to extract sub-images of the individual components from a picture of the entire circuit is harder than we thought, as per Devan’s work this week. We will likely require dots/circles at the end of each terminal of a component to mark each individual component. At this moment we have not made any scheduling changes. Prior to making the schedule, we understood that since none of us have any CV experience that deciding on an algorithm would be paramount to this project’s success and so we have dedicated time to research. Given our current progression and the deadlines that are approaching, we are confident that we can stay on track to our initial plan and finish everything we planned to finish for the next week. 

We considered both public health + safety and economic factors as we developed our proposal. Firstly, our project reduces the safety risk that electricity poses to children trying to construct circuits. This is very important because parents can have peace of mind that their children will not be hurting themselves when learning about electricity. We also are aiming to reduce the cost that students and their families will incur by studying electricity and circuits. People will no longer have to pay 75+ dollars for a power supply, breadboard, wires, and resistors – instead they can just use our product for free.

Here is what comparing two drawn resistors looks like to the computer:

Posted on

Stephen Dai’s Status Report for 9/23/23

This week I researched different types of algorithms to use for feature matching in order to detect and identify individual electrical components. The two primary feature detection algorithms I considered were SIFT and ORB. From testing what I noticed was that ORB significantly handled variation in size and orientation much better than SIFT, and just had more features matching in general.

In the above photo you can see the results of using SIFT and ORB when comparing two images of resistors. ORB is clearly better with 132 / 381 matches, while SIFT only has 8 / ~48. Here is the visualization for the matches with ORB:

Note that the drawings of the resistors were first pre-processed with a contour detector that Devan found, which is why the background is entirely black and the contour of the resistors are in white.

When I tried comparing a resistor that was aligned vertically and one that was aligned horizontally, this was the result:

SIFT could not even generate any matches while ORB still could.

At this point I am confident that ORB is the algorithm we will use to generate FAST keypoints and BRIEF descriptors.

The one problem I have run into is that when using a resistor and voltage source drawing, the results of the matchings aren’t as polar as I expect them to be.

This is the result of comparing a resistor and voltage source drawing. If we compare to the matches of two similar resistors (first screenshot), we can see that ORB produces slightly less matches for the one with a resistor and voltage source. But, if we compare these results with two resistors of different orientation (second screenshot), they are pretty similar (~100 matches). While this may seem worrisome, I think that this is not too problematic because I believe the solution to this is to include various orientations of components in our training and validation datasets. This way, the matching of the same components in different orientations will be stronger and should contrast more with the matching results of two different components.

I am on schedule with my progress. I have established a good foundation with the algorithms to use for individual piece detection, and I want to continue exploring and tweaking to see if I can better improve the matching contrast with different components. Another thing I want to specifically try is using a different matching algorithm, as all of the above experiments used a brute force matcher. My steps for next week will be to create a reasonable feature vector with metrics that will be used to determine classification.