Posted on

Joshua’s Status Report for 10/23/21

Last week, a.k.a Week 7, October 11th-17th, I worked on the design review report with my team, and I contributed by writing the parts: Abstract, 5.2, 6.2, 7.2, 7.4, 8, and lastly, formatting and revising our Gantt chart + schedule. As my teammates have pointed out, this design report was clearly a poor reflection on our team as a whole, and there were many other parts of the project that we have explored in detail and should have added to the document, but we simply were not organized and could not meet the deadline, due to other classes, midterms and general lack of communication. My team and I hope this does not reflect extremely poorly on our group overall, as we will be addressing each and every concern that was raised in the design report evaluations and more, and will have a much more refined, detailed and accurate final report + project deliverable by the end of the semester.

To add onto the work I accomplished last week, I have been working on the model locally and ironing out bugs, and there was a very superficial but hard to find problem with conflicting Python library versions that took almost 3 days to discover and solve, but I also ran into another problem with running the code on AWS. Although I had initially made an instance using the free tier before we had received credits, I had eventually refined our design later on and pinpointed the exact instance I wanted to use – the P3 instance that uses the NVIDIA V100 GPUs, which should theoretically speed up our training process much more. Unfortunately, we hadn’t increased our vGPU limit, and AWS did not allow us to create an instance, so we had to apply for it, and we still hadn’t gotten back an answer after a full week.

Fortunately, we had already prepared for such things to happen, and we had deliberately added 2 weeks of slack/extra time to handle such things in our project, so although this software portion is basically a full week behind now due to the above listed complications, I am still confident that I can finish it in time to allow the hardware portion of the project to start and complete on time.

Looking forward to the coming week, I am aiming to finish the software portion by the end of the week, which James and Kunal are heavily involved with, as to not delay their progress.  To address the concerns about my lack of work after this week, I will be helping Kunal and James with the integration portion of our project as much as possible, whilst actively participating in the other remaining aspects of the project to ensure it goes smoothly.

Posted on

Kunal’s Status Update (10/23)

This past week I focused on building the core neural network architecture on the fpga. I got ramped up with the vivado hls & vitis platforms being used to implement the neural network on the fpga. This week I’m planning on making more substantial progress with James on the bicubic interpolation algorithm and its implementation directly in hardware. I’m getting acquainted with pragma based directives in hls and will be exploring these functionalities in depth with the hardware implementation this week. We have been working on perfecting the model so we can see a noticeable increase in resolution and then we can look into how specifically to implement the buffers across the various pipeline stages in the neural network design. This is highly dependent on the number of layers and the architecture of the neural network itself. Once we have this set in stone this week I will get into the hardware details involving the Ultra96 and it’s onboard fpga unit, and also will setup and benchmark the frame relay rates from the usb bus. This week will mostly be focused on setting up the hardware related infrastructure & get bytes from the usb interface to the fpga core and relay acks back.

Posted on

Team Status Report for 10/23/21

Last week we mainly focused on writing the design review report. To address the elephant in the room, we know that our submission was nowhere near as well done or well polished as it could be, and frankly was not even fully done in some parts. We know that this now means that we will have to pick up more work leading into the final report to make sure that it is done well, and we more than intend to do this. We do not want to repeat submitting something of such sub-par quality for the final report at all.

As for this week:

James focused mainly on improving CNN performance with marginal gains so far. More details are included in his status report.

Joshua focused on refining the software implementation of the project and ironing out bugs, as well as sorting out issues with training due to problems with AWS.

Kunal helped with improving CNN performance, as well as acquainting himself more with some of the content from reconfig, which James is currently taking but Kunal is not.

Overall, the project is about one whole week behind according to the Gantt chart, but this is not a concern since we left two extra weeks in order to address unexpected issues with our project’s development. A lot of it came down to our other courses ramping up in terms of time commitment and effort, and all members having to focus on other things, but in the end, we made a steady amount of progress, and we are still on track to finish the project on time.

Posted on

James’s Status for 10/23

Last week, I mainly focused on the design review report. I didn’t get it as far along / as polished as I would have hoped; this means it will be more work for us when updating it for the final document. I ended up writing sections 1, 2, 3, 3.2, 4.1, 4.2, 5.1, 6.1, 9, 9.1, acronym glossary, BoM, and references. I was unable to write more, or organise with my partners to write more in their sections, but I know that we messed this up badly, and we will aim to rectify it before the time the final report comes around.

This week, I focused more on optimising CNN operations on the FPGA. This is a little bit out of order, but I decided to do things a bit out-of-order because it works much more synergystically with where we are in reconfig right now. I have so far increased throughput (on a basic fast-CNN implementation) by 25% to 20MOp/s, but am expecting to settle at two orders of magnitude higher than where I am right now. I also helped Kunal on-ramp with some Vitis stuff, as he was slipping behind on ramping with the platform. I shared excerpts from Vitis tutorials that we were given for reconfig, as well as pointing him more directly in the direction of online resources for Vitis. I need to circle back with him and check where he is with progress on i/o, and plan to do so this coming Monday. This may effect the Gantt chart / schedule, but we have the slack to allow for it for now. I will be keeping tabs on how much slack I use in my tasks because I know that I have begun cutting things close with the amount of remaining slack which I am allotting myself.

Posted on

Joshua’s Status Report for 10/9/21

This week, I spent a lot of time incorporating our new design change of using SSIM instead of VMAF as our metric, by rewriting some of the code I had locally and benchmarking more using SSIM. The end results were very satisfactory – SSIM was a lot faster than VMAF and suited the training portion of our project much better, and I also experimented with the idea of using the previous/next frames as side inputs, but decided against it due to the complexity and, importantly, the little added value of doing that compared to a solely frame-by-frame upscaling method. I also worked on the design review with my teammates to refine the details of our implementation and our project overall.

For the next week, I will spend the majority of my time writing code on AWS, finishing up my model and starting to train it with our chosen dataset. Since I have several midterms next week, I will have to balance my time well and coordinate with my team to make sure that we are working on the project on time.

Note: This individual status report was published late due to a technical issue. The initial draft shows it was created last Saturday, before the due date, but was not published and not fully saved.

Posted on

Joshua’s Status Update for 10/2/21

This week, our team met up with Joel and Byron, and we discussed our progress on the project. We went into detail about the specific use of VMAF in the training part of the CNN, and we discussed various problems/issues that may arise with the use of VMAF, and came up with several solutions. Byron reiterated the importance of traditional DSP methods, specifically, wanting us to justify and confirm how using a CNN would be superior to those traditional methods, and I incorporated that into our design presentation.

Since we didn’t receive AWS credits until much later in the week, I downloaded our intended dataset and attempted to benchmark VMAF as well as Anime4K locally, a project on Github with similarities to our project, to see how they would perform. Since there were many different videos available on the website where I was getting my dataset (CDVL), I ran into a slight issue with the difference between 1080i and 1080p, as well as the FPS of videos in the dataset, but after discussing with James, I managed to compile a list of videos which were 1080p @30FPS, and worked with my team members to successfully benchmark VMAF and Anime4K. Our development of Python code for training was delayed, since we couldn’t start until much later on in the week, and we intend to catch up on that ASAP as soon as our design presentation is concluded.

I also met up with TAs Joel and Edward outside of class hours to further discuss our project and refine the details in preparation for our presentation. I also wrote the team status report for this week, and worked on the design presentation with my team members.

Posted on

Team Status Report for 10/2/21

This week, we met up with Byron and Joel again to discuss more about our project, specifically to address any comments from the feedback from our proposal presentation, as well as following up on our initial, first meeting. During the meeting, we addressed the concerns about the use of VMAF as a metric for our training, as well as our dataset and some other things that weren’t fully justified during our presentation. Byron commented on how we have to make sure that implementing a CNN is better compared to traditional DSP methods, and to make sure that implementing something much harder is still the best choice. To that end, we benchmarked both VMAF and Anime4K, a project on Github that does something similar with, specifically, animation, and we obtained concrete, quantitative measurements which we can elaborate on our design presentation to fully justify our design choice.

Joel also raised a good point about how upscaled, lower resolution videos compared to original, native resolutions videos would always result a lower score, and we addressed that by limiting our training to only comparing videos that have been upscaled to the native resolution, e.g. 1080p to 1080p. We also talked about the importance of benchmarking as soon as possible, which we successfully did this week.

Although throughout the week our team members were slightly overwhelmed by work from other classes, we managed to catch up sufficiently by meeting up after class hours and communicating to make sure our tasks were still completed on time. James and Kunal continued their research on I/O, and calculated specific quantitative measurements to put on our design presentation, and I continued my research into VMAF, as well as the model being used for training our upscaling. Referring back to our Gantt chart/schedule, we were slightly behind on developing the Python code for training our own CNN, as we only received AWS credits Friday morning, but we used that surplus time efficiently by benchmarking locally, as well as researching in more detail Anime4K. As per the feedback from Tamal, we are taking the risk of our CNN not working/not being developed in time on the software side more seriously, and our backup plan would be to simply use the CNN implemented in Anime4K and start implementing that on hardware if we cannot get it working on the software side after Week 7. We’ve changed our schedule/Gantt chart to reflect that accordingly.

Looking further into the peer/instructor feedback, we see that they were a lot of comments about the absence of justification for our FPGA selection during the proposal presentation. We’ve focused on elaborating on the choice much more for our design presentation, and we are similarly going into a lot more detail for our software section, as well as our quantitative requirements.

Overall, despite some things not going as planned this week, we believe our team was very successful in overcoming all the problems we encountered, and our initial planning, which allowed for slack time/small delays, proved useful. We look forward to delivering our well-prepped presentation on Monday, addressing all feedback from our previous one, and continued success towards the progress of our project.

Posted on

Joshua’s Status Update for 9/25/21

This week, we went to class and peer reviewed our classmates’ presentations, as well as receiving feedback and Q+A from our peers, TAs and professors. I also followed our schedule and completed my main tasks of familiarizing myself with the VMAF documentation, as well as sorting out the details of our model selection/training with my team during meetings. We discussed specifically the dataset that was best for our training, and also our backup plan of using an existing algorithm if our CNN doesn’t end up working within a reasonable amount of time, since the main portion of the project is hardware – specifically, the implementation of writing the algorithm onto an FPGA to hardware accelerate the upscaling process.

Looking at the VMAF documentation, I confirmed that the current version has a wrapping Python library that has good documentation. I had initially decided to use Python since it was the most intuitive option for ML, and I could also have Kunal helping with the beginning of the algorithm development, since he also has some experience with ML, and is eager to be involved. Also, the Github on VMAF has some very useful scripts that can be used during our training process, and I also briefly considered the sample datasets that they provided, but I discussed with my teammates and we decided that the dataset from the CDVL (Consumer Digital Video Library) fit better, since it had a greater variety of videos, such as animation.

For next week, I will begin to work on the model development in Python with Kunal, as well as working on the design presentation as a team. I will have to further consult the VMAF documentation, as this is my first time using it, as detailed on the schedule.

Posted on

Team Status Report 9/25

For this week, we followed our schedule/Gantt chart and attempted to do every thing on our list. Almost every single task was accomplished successfully, with details listed on our personal status reports, with the only hiccup being our AWS setup – the credits will be acquired by next week, and we have already looked into which AWS instance type would be most suitable for our group.

During class time, our entire group also performed peer review on all our classmates. We decided that Kunal would be presenting for the second presentation, and we also aim to address all of the concerns that were addressed during the Q/A session of our first presentation, as well as all feedback we received from TAs/professors through Slack.

As a team, we further discussed the model for our project, as that is a core part of the upscaling process. Reflecting on the feedback from Slack, as well as following our schedule, we also decided on a specific dataset and acquired the videos online from a database that provides it.

Looking towards next week, we are on track according to our schedule, and optimistic of continuing our positive trajectory. Next week, we begin trying to implement I/O with our acquired hardware, as well as preparing well for our design presentation. We will also begin writing code for the training part of our project in Python.

Posted on

James’s Status for 9/25

In the end of last week I practiced for the presentation after we finished the presentation. After giving the presentation, I acquired the Ultra96 and its peripherals. I’ve been toying around with it and getting it up and running in the time since.

As per our gantt chart schedule, I feel that I am personally on track so far.