Atulya: This week I went through the video we used and collected more data for our SVM. This required me to minutely study our current video output and identify parts of the video that were causing the SVM to fail (misclassifying a garbage can as a person, missing people entirely etc.). Once I had identified these objects, I had to add them to our SVM training data so that our SVM would learn to correctly classify these things. I also worked on creating the final presentation that we did in class. Lastly, I implemented our sliding-window pipeline in C++ so that we could do performance testing between our C++ CPU implementation and our FPGA implementation. This data was used in our final presentation.
Andres: This week I wrote and tuned the non-maximal suppression (NMS) algorithm to remove overlapping bounding boxes that map to the same pedestrian. I gathered metrics (recall / precision) on the video that we are using to demo using the old SVM and the new SVM that was trained with the original dataset combined with a few images from the scene that it consistently did poorly on. I've started working on the backend for the user interface of our application (gathering analytics on the stores in the demo video) and will continue to work on that.
Felix: This week I finished implementing the basic sliding window algorithm on the Zedboard. It currently still only reads 1 frame and for the weekend I will come up with a way for the system to process multiple frames. Also for the weekend I have decided to attempt to move to a bigger board such as the ZC706. Due to the nature of the HOG algorithm, there are lots of dependencies between loops that require a bit of all or nothing approach to optimization, and a ZC706 will allow us to half our current latency which I believe is worth the setup time that I will spend on changing the platform. I also collected data for the final presentation, such as performance on the board as well as on the C++ code, as well as power consumption for each platform.
Atulya: This week I worked closely with Andres to use our SVM and HOG code to see how it performs in a real-world video. I found a commonly used dataset called the TownCentre video, which depicts people walking through a street. We plan to have our system detect the people walking through this video. Additionally, after seeing the performance of our SVM, I brainstormed some ways in which we could improve our implementation. The plan is to implement those ideas until we hit our performance benchmarks.
Andres: This week I worked on using images from the video we want to run our code on and demo at the end of the semester to show how they interact with our pipeline. I created a script that runs through the video, passes each frame through our pipeline and then outputs a video that contains bounding boxes for all the people in the video. I added in background subtraction to reduce false-classification of static objects that look human-like (mannequins) or objects that weren't included in our test set (mailboxes, metallic pay phones). These will only be considered by our pipeline if the pixel intensities in the local area change substantially. In matlab, I added a script that finds and stores the true background with dynamic objects like people removed. I also found images that has people of varying sizes to test the sensitivity of the pipeline.
Felix: This week I finished implementing the hog vector calculation in HLS. The design fits on the board and passes timing currently but the clock cycles it requires is not ideal. I am currently in the process of optimizing for more speed, for example replacing more division statements with case statements.
Atulya: This week I started converting my MATLAB HOG code to C++, which will be used by Felix in the hardware implementation. I also helped Felix resolve some issues involving the arctan() function and the differences in its implementation between MATLAB and C++. I also fixed a bug in my gradient angle calculation that was identified by Felix.
Andres: While testing the SVM, I realized there was an issue in the code that resulted in invalid outputs (Nan or errors) from edge cases. This was fixed and the weights and bias were recomputed and stored. The change did not affect the accuracy of the algorithm. I am working on getting the UI set up but am running into difficulties with having the compiler find opencv.
Felix: This week I have successfully put the gradient code on hardware and have it running on the Zedboard with the correct results. Currently I am attempting to read and write files on the SD card on the Zedboard with the FatFs API.
Atulya: This week I worked on refining my MATLAB HOG feature generation code. I also searched a found a very useful dataset of pedestrian images (positive and negative) that we could use for our SVM training and testing. I was able to combine these two milestones to write code that generated HOG features for my test images and training images, save them in a .mat format that could then be read in by Andres' SVM implementation. Apart from that, I helped Andres debug his SVM code and come up with ways we could improve its performance. I also updated the website and schedule.
Andres: This week I worked on fixing some slight issues in our SVM and added the ability to load in a prior probability distribution. I've tested the SVM's ability to identify whether a pedestrian is in an image by training on HOG vectors that were pre-computed from Atulya's MATLAB script and then tested on a set of HOG vectors that corresponded to a different set of images. The accuracy of the SVM when initalized to random weights averages 76% but has reached an accuracy of 87% on the test data. These weights are stored and will be used on the FPGA to evaluate whether a pedestrian is in the provided image. Moving forward, I will create data to train the logistic regression model to predict the path of the pedestrian in the scene.
Felix: This week I have implemented the gradient computation and it is functionally correct. I am currently working to optimize the design a bit while I wait for the rest of the code to be done for me to implement. The two main optimizations were removing the radians to degrees conversion, because it requires division, and using the hls_dsp library to perform arctan instead of using the synthesized version in the hls_math library. The cordic atan function in hls_dsp.h requires a specific fixed point representation and currently I am working out a way to normalize the inputs so that the angle calculation is still correct and can achieve a degree perfect accuracy. Removing the division sped up latency by 500 clock cycles. By using the dsp atan function the latency has gone from 90933 clock cycles to 81789 cycles, but more importantly, uses ~15k less LUTs, reducing utilization from 60% to 31%.
Atulya: This week I worked further on my MATLAB prototype for the pedestrian detector. Having implemented the HOG feature computation logic last week, I came up with a framework that will allow me to take in an image, split it up into individual sub-sections and compute the HOG feature for each sub-section. After doing so, I will implement Non-Maximum Suppression in order to reduce nearby "hits" on pedestrians to all refer to the same person. The code to implement this is currently being worked on and is expected to be done by this weekend. Apart from this code, I also worked on updating this website and syncing up with Felix to get an idea of how C++ code will eventually be run on the FPGA. This was vitally important in order for me to understand how the code I write will need to be formatted to make the transition between software to hardware easier for us as we move forward.
Andres: This week I finished working on my logistic regression code from earlier in C++. It should be ready to "drop in" to our hardware. Apart from that, I implemented an SVM in MATLAB that we will be integrating with the HOG pipeline in order to test our prototype. This will let us see if our code is correctly be able to determine whether or not there is a person in a given image. Once we have verified our logic, we will be in a position to convert this code to C++, at which point it can be implemented on the FPGA.
Felix: This week I have started working on trying to get an AXI Streaming interface working, and I'm currently considering the interface for the project. Meanwhile I have started converting the image gradient computation code to C++ and hope to have it functionally complete in HLS soon. I took some time to understand how the gradient computation works in order to correctly implement it in hardware. I have also given my teammates and idea of how their C++ code will run on the FPGA so that everyone has a better appreciation for the technical aspects of this project moving forwards.
Atulya: My reponsibilities for this week included getting a HOG feature vector generator working in MATLAB as a prototype, as well as concluding our conversations with CMUPD. Regarding the latter, they have given us permission to take videos on campus, so we will proceed to do that at the appropriate time. In the meantime, I wrote some MATLAB code to generate the HOG feature for a given image segment. The next step is to integrate that with an SVM to actuall classify these image segments as either having a person in them or not. I have also updated the website and our schedule.
Andres: The main objective for this week was to get the machine learning side of the project working in C++. While implementing logistic regression, I ran into some library problems and decided to change the approach to using linear regression and finished implementing that in C++. I will continue to work on logistic regression but the main focus is going to be working an SVM module in matlab to be ported over to C++.
Felix: This week I focused on getting the basics down. I went through tutorials to build a good foundation on using HLS, the Vivado Suite, as well as the SDK. Got a simple PS/PL system with interrupts running and I am currently looking to expand and adapt the design for the capstone.
Overall Team Project Update
This week, all three of us worked on completing the design proposal. We incorporated the feedback we receive for our design presentation and made adjustments to our project accordingly. With our finalized architectural details, we are confident of our plan and are moving forward with the implementation. Apart from creating this website, we have also updated our schedule (see Gantt chart on Home page) to better match our current level of progress. We also received a Zedboard recently and have successful set up our development environment in Vivado. Additionally, we have got OpenCV in C++ up and running, which should allow us to start developing our computer vision pipeline soon. From a logistics standpoint, we are in the process of speaking with CMUPD to get permission to take videos of pedestrians on campus. This will allow us to create our statistical models for the machine learning pipeline. As a contingency, we have found a video of pedestrians online that has been used in research papers for pedestrian detection, and we will resort to using that video until we sort out any legal issues with CMUPD.
Individual Team Member Responsibilities (i.e. who did what above)
Atulya - Design proposal updates, website creation, OpenCV/C++ setup, meeting with CMUPD.
Andres - Design proposal updates, research on machine learning pipeline for path prediction, training data procurement.
Felix - Design proposal updates, Vivado development environment setup for Zedboard.