We finished up benchmarking the latencies of our fsrcnn model and I wrote a profiler that would take upscaled frames and build metrics based on the quality of these images. This we will use in the final demo to compare and contrast latencies of the software model running on the hardware device. The nature of the computation involves a scan over the pixel set and a buffering mechanism for the previous pixels fed into this pipeline. We are actively tuning the model and analyzing our results, we expect to get to a significantly lower latency and be able to profile the results much more efficiently.
Kunal’s Status Update (10/23)
This past week I focused on building the core neural network architecture on the fpga. I got ramped up with the vivado hls & vitis platforms being used to implement the neural network on the fpga. This week I’m planning on making more substantial progress with James on the bicubic interpolation algorithm and its implementation directly in hardware. I’m getting acquainted with pragma based directives in hls and will be exploring these functionalities in depth with the hardware implementation this week. We have been working on perfecting the model so we can see a noticeable increase in resolution and then we can look into how specifically to implement the buffers across the various pipeline stages in the neural network design. This is highly dependent on the number of layers and the architecture of the neural network itself. Once we have this set in stone this week I will get into the hardware details involving the Ultra96 and it’s onboard fpga unit, and also will setup and benchmark the frame relay rates from the usb bus. This week will mostly be focused on setting up the hardware related infrastructure & get bytes from the usb interface to the fpga core and relay acks back.
Kunal’s Status Update
This week our team worked on pinpointing an algorithm for usage in the real-time video upscaling problem. We found that DSP algorithms approach the problem in a rather naive way, as they’re unable to scale out to different form factors for the video data. The inputs to the image upscaling problem are uniformly distributed but often vary in slight ways on each iteration, and hence a deep learning based approach is favored.
The deep learning algorithm I looked into was image super-resolution from sparsity. This algorithm covered how we can take batches of pixels from a low resolution image and build out 2 matrices representing a downsampling & blurring filter. The deep learning algorithm would be based on a classical layered neural network taking in pixel densities and locations as inputs. This algorithm will then train two dictionaries both representing a sparse coding for the image upscaling algorithm. Two dictionaries for both the low resolution and super-resolution images would then be correlated and through the iterative process of gradient descent we can figure the appropriate heuristics for the trained model.