Breyden Wood’s Status Report for 5-8-21

This week, I spent the majority of my time preparing for the final presentation I gave on Wednesday of our project. This process required me to flesh out all the slides and review what I was going to say in a couple practice runs for our group. Overall, I was very happy with how the presentation went and I feel we got good feedback from the course staff on ideas for our final video.

After working on the presentation, I transitioned to fleshing out some bugs in our background removal. By working on the angle of the LEDs on the sides and fine-tuning the parameters on the FPGA, I was able to get an excellent result with the majority of the objects that we tested where all the background would be removed with minimal clipping on the object.

Lastly, I have worked on collecting footage both explaining the FPGA’s inner workings and filming lots of B-roll of our entire setup. All that is left for me to do is to assist Jullia in editing all of this together into a demo video that we are all happy with. Our progress is on track, and we expect to be finished with all of our work in the coming days.

Grace An’s Status Report for 5-8-21

Over the past week, I worked on final logistics. As planned, I helped finish up the final presentation at the beginning of the week, fixing up slides so that they had less text and were more readable as well as more visually appealing with pictures of the pyramid. I also checked through the final presentation slides to affirm the content and helped Breyden shoot some footage for the final video and perform testing of our system. A link to the presentation slides can also now be found on our website here.

At the end of the week, I worked with Breyden on an outline for our final video, determining how long we will speak throughout different sections. I also put together some slides and recorded audio for the sections of the video discussing the background removal algorithm, display, and pyramid. Over the next few days, I plan to record more footage of our project with Breyden’s help, in order to have B-rolls so that Jullia can edit together our final video. Our goal is to get good footage of stationary and moving objects inside the studio and also demonstrate the user experience of using our HoloPyramid project. Once this is done, we will upload the final video, make a poster presentation, and finish up our final report. My progress is on schedule, and I am on track to finish our project over the next week. As this is the last status report, there are exist no further deliverables.

Team Status Report for 5-8-21

This week, our team worked on our Final Presentation, wrapped up our final adjustments to the background removal filter settings (to further enhance image quality), and began the process of editing the Final Video.

For our Final Presentation, we all worked together to create the slides and reviewed together what I was supposed to highlight and say to show off our progress and completed product. This included running some final tests for image quality and assembling the results together along with the challenges we faced throughout the project. After completing this, we turned to optimizing our filter settings and collecting footage for the video. We found that dark green felt and even lighting around the studio worked the best, so we dialed in the filter settings and re-ran a few of our benchmark tests to get improved results. From there, we have worked with our completed project to collect footage of various objects from various angles – making sure to highlight the strength of our presentation tool at showing a 3D representation of an object from all sides.

From here, all that is left is to finish editing this footage into the final video so that we can show off our project to the class! At this time our project is complete, so we have no further risks to manage aside from the remaining work to be done on the video and final report (which should be uploaded shortly this week).

Jullia Tran’s Status Report 5-8-21

This week, I spent the beginning of the week working on the final presentation demo and going over what Breyden was going to say for the final presentation. I worked on editing the Gantt chart to reflect our changes, and also edit some of the slides to make it more readable through the use of tables, cutting back texts, and adding more visual diagrams.

Later this week, I worked on recording some parts of the video for the final demo presentation. My part is to talk about the studio development, the design choices we made, and the final design that we have for the studio. I also worked on editing our final demo and putting together the final video. I need to go over the footages that we recorded from last week, along with some more footages we got from this week and find a way to best put together a video that would represent and displayed our project in the nicest way so that it’s easy to showcase our final product. Each member of the team recorded an audio recording, and an audio recording with the slides presentation to better explain the project’s work flow, and the final product. We also have B-roll shots of our working products from multiple angles, objects, and object interactions that needed to be put together.

We also work on putting together the poster presentation that is needed for Monday night.

Breyden Wood’s Status Report for 5-1-21

This week, I worked on integrating the entire pipeline together. I went over to Jullia’s place where the entire setup is currently assembled. Jullia and I worked together to set up the TV and the pyramid by moving the TV flat on the table. I also mounted the cameras into the studio after the holes in the studio were cut out. There was some adjustment needed after the cameras were mounted into the studio regarding the orientation of the video footage. I adjusted these issues through Quartus so that the video footage faced the correct direction and displayed correctly on the pyramid and TV setup. Testing also revealed some memory bugs with lines of incorrect colors showing at the bottom or top of the square of projected images. I resolved this error by changing the SystemVerilog code to remove these lines.

Some of the cameras also needed to be switched out due to differences in auto-exposure and white balance. This was previously identified as an issue last week, and we were able to fix it with the new cameras we purchased this week. As mentioned in the Team’s status report, longer wires resulted in greater electrical noise, so we had to place the FPGA and Arduino at the bottom of the pyramid (instead of above it). The wires are extremely sensitive and any slight adjustment of the studio results in the wires getting loose. This creates a lot of noise on the video feed and I have had to adjust these connectivities using tweezers from time to time.

I spent the bulk of my time adjusting lighting and the parameters for the chroma key algorithm to ensure optimal background removal (with a uniform background). As synthesized, the chroma-key algorithm takes in an RGB value and threshold setting for sensitivity. I carefully adjusted the RGB values while adjusting the lighting for the studio for the evenest lighting and best possible background removal.

I also captured video footage of our project using my personal camera and tripod. I took wide-angle shots, video footage, and close-up photographs. This footage will be used in the final presentation as well as the later final demo. I also did latency testing using the remote for the studio lighting, determining that our project has ~21 ms of latency (much better than our initial requirement of several hundred ms).

I am on track with our schedule. The project is fully integrated, testing has been started, and I have taken extensive amounts of footage for our final video. Over the next day, I will practice the final presentation, which Grace and Jullia will provide feedback on. Over the next week, I will work with Grace and Jullia to finish testing and adjustment of the final project as well as work on the final video.

Jullia Tran’s Status Report 5-01-2021

This week, our team gathers at my place to work on the project together. We work on integrating the entire pipeline together, from the output of the camera onto the display. Breyden and I first worked together to set up the TV and the pyramid because it’s a 55-inch TV so this set up task requires 2 persons.

I work on parts of adjusting the studio to make it works with 4 cameras. As mentioned in the team’s status report, we ran into wire length constrained and because of this, we decided to put the FPGA inside our studio. For this, I cut out a platform and place it on top of the FPGA. I also work on cutting up the felts to cover the walls. The dimension of these felts has to be so that it doesn’t cover up the LEDs running along the side of the studio. We also tried multiple colors felt and so all these sheets of felts need to be cut out. In each of the felt, there needs to be a hole so that the camera lens can fit through. There are 4 different colors that we tried (black, blue, light green, dark green) and each of these need 5 sheets of felt, 4 for the walls, and 1 for the bottom. Grace also helped me with cutting these felt. After cutting, we need to insert it into the studio and this tasks was also done with multiple iterations because we didn’t realized at first that we don’t want the felt to fully cover the LEDs lights because this turns out that the light becomes too diffused, hence some adjustment is needed.

I also work on cutting some of the holes in the studio so that wires can go through, as well as holes for the 4 cameras and their wires. This is done by directly cutting into the studio and measuring the height so that the placement of the cameras would be somewhat in the same leveling plane.

The lighting of the studio also needed to be adjusted. I helped with cutting strips of felt to diffuse specific light on the LEDs strip. Multiple iterations of this task need to be done since we need to figure out which configurations would best depict the video feed the best.

When we film some of the shots, I also help with some of the set up, clearing out space for the camera, and being Breyden’s assistance for whatever he needs to film. I also provided some feedbacks for some of the shots we might want to do for filming. I also help with documenting some of our progress with pictures as we work on this project.

I printed out the MTF test so that we can do testing with the setup. This test tests the distortion of the pyramid and since we have the MTF testing card printed out, we were able to just run this test.

We are on track with our tasks. We just have to finish up testings and put together the video, which is what I will be doing in the next week.

Grace An’s Status Report for 5-1-21

Over the past week, I have worked largely on the project presentation, transferring information from the previous slides to the final project presentation and adding new information. While our final presentation has a significant amount of information that was covered in our proposal presentation (the application area and solution approach) and the design review (hardware and solution description), our final presentation has the results of our preliminary testing and significant discussion of design trade studies. For example, LED strips on the side of our live studio give significantly more even lighting than point lighting.

This week, I also assisted with integration and assembly of our project. I cut out different felt squares and strips for the backdrop of our live studio and assisted with assembly. We tried out black, bright green, dark green, and dark blue felt backdrops. The different colored backdrops had different results on the background removal algorithm due to how uniformly lit they could be, and the brightness of the backdrops affected the cameras’ autoexposure settings, affecting how the object was shown. Unfortunately, brighter backgrounds were less uniformly lit, but darker backgrounds badly influenced the cameras’ autoexposure settings, causing the object to be whited out with very few details shown. We settled on dark green felt to serve as a backdrop, and we also blocked the bottom LED strip lights to better improve the even lighting of our studio. I also assisted Breyden with the adjustment process of background removal and visual quality of our product.

I am track with our schedule, as I have both worked on the final presentation and helped out with integration and preliminary testing. Tomorrow, I will listen to Breyden’s practice run of the final presentation to give feedback, and I will also assist with acquiring footage of our project to be used for the final presentation. Over the next week, depending on time constraints, I will also have the additional task of assisting with finishing tests, adjusting our project, and making the final video.

Team’s Status Report 5-1-2021

At the beginning of the week, we worked together on the final presentation slides.

Later this week, we worked on integrating the studio with 4 cameras and outputting to the TV, on which the pyramid sits. We came together as a team to work on this at Jullia’s house. We spent some time mounting the four cameras to the side of the studio. Because longer wires have more electrical noise, and our cameras are especially sensitive to electrical noise, the wires must be short as possible. To ensure this, we placed our FPGA and the Arduino inside the studio at the bottom, and added a cardboard platform on top where our object can be placed on. We also spent some time cutting out some holes in the studio so that we can have access to the switches of the FPGAs. This allows us to adjust chroma keying color, discussed later in this status report. These changes to the design improve our project and do not affect its cost.

Also, we spent the bulk of our time adjusting the white balance, auto-exposure of cameras and lighting of the studio to have each panel of the pyramid output to be similar enough to each other. This is done by switching out a couple cameras because their sensitivities have slight variation. We also experimented with the lighting between the LEDs along the side of the walls vs point lighting. We found that point lighting creates gradient of shadows, especially when interacting with the object. These shadows make the background extremely difficult to remove, so point lighting is not desirable for our project.

Furthermore, we also spent signficant time adjusting our chroma key to find the optimal background color. We experimented with multiple colors of felt along the wall and the colors of the platform. We tried dark green, light green, blue and black and experimented between them to have chroma key close to removing all of the background. We prepared multiple felt sheets with specific dimensions that will fit into the studio, and then adjusted the chroma key to match the color that shows up on the camera inside the studio, where lighting is also introduced. The lighting was adjusted by dimming the overall studio, and dimming or diffusing specific light on the LEDs strip.

We also started testing our HoloPyramid. The latency between display and real time action turns out to be around 22ms of delay which significantly surpasses our initial requirement of 250ms of delay. This latency is more than low enough to be imperceptible to the human eye, and confirms that  human eyes perceive the visual display as real time.

Our most significant risk is that the camera exposure does not have a lot of dynamic range. When we have a bright object against darker backgrounds, the camera struggles to auto-expose. This causes the object to be shown with the incorrect exposure. To mitigate this risk, we used brighter backgrounds and experimented with lighting. Another risk we have is that the wires are very sensitive and the connectivity is not the greatest between the camera and the GPIOs pins. When the wires get loose, noise becomes apparent on the output which decreases our video feed quality. To mitigate this risk, when noise occurs, we go into the bottom of the studio, where the FPGA lives and adjust the wire connections using tweezer to make them tighter.

We also started working on trying to film for the final demos, capturing footage of our current project.

Images of our project can be found in this link (we ran out of space for our website).

Our Gantt chart can be seen here. Our schedule has not changed; however, some additional tasks were added to the end to reflect our current integration and testing projects with greater granularity and details.

Team Status Report for 4-24-21

Over the past week, we have integrated our FPGA camera-to-VGA pipeline with our image processing suite, specifically the chroma-key filter. We have also hooked up all four cameras to the FPGA using an HSMC expansion card. Detailed pictures can be found in Breyden Wood’s status report here. What remains is integrating the fully integrated FPGA with the live studio and pyramid again as well as image enhancement (through specially configuring camera settings and such). Additionally, we need to test the studio and image quality metrics as described in our original proposal and later design report.

We have made some minor modifications of our design to allow the user to tune the background removal algorithm by using the FPGA’s hardware switches. The user can set both the background color that is removed as well as the sensitivity of the background removal, and they can also use a switch to turn the chroma-key filter on or off. This enhances the user experience of using our project and does not incur any monetary costs. The logic units and memory bandwidth costs are also well within the capability of the FPGA.

Our most significant risk at this time is the image quality of the cameras. Some of our cameras have faulty auto-exposure and/or white-balance settings that greatly impact the quality of recorded video. Our mitigation strategy is individually swapping and testing the cameras to make sure the cameras we are using do not have defects that impact our project. Because we bought eight cameras when our project only requires four, we should be able to easily mitigate this risk with our existing cameras. Otherwise, we do have enough remaining budget to buy additional cameras as necessary.

Here is our updated schedule. There are no significant changes. The only changes are final clarifications on our previously ambiguous task assignments at the end of the semester. Our progress is definitively on track as our project is almost complete.

Jullia Tran’s Status Report 04-24-2021

This week, I work with Breyden to integrate Grace’s filter into the pipeline, wiring up the remaining camera and mounting them onto HSMC.

One issue we ran into at first was the filter seems to have an MSB issue. We found out that it was due to the signed extended mechanism was incorrectly implemented. Instead of sign extending the MSB, the LSB was used instead. This was a quick fix and once that was fixed, the chroma keying algorithm works perfectly (some pictures are in Breyden’s blog post).

When we plug in the second camera into GPIO, we noticed there was an issue with the White Balance of the second camera seems to be much better than the first camera. However, they both were using the same settings and outputs generated from the same Arduino Uno. After mounting the third and fourth camera, we realized that the later cameras that were being mounted have a much better White Balance and auto-exposure sensitivity than the first one. We then switched this camera out for a better one and now all the cameras are generating video feeds with similar output. Since our feed back from the interim demo was focusing on the quality of the video, we realized that this might be due to the fact that the first camera seems to be defective, as shown when the video feed quality of the newly mounted cameras were much better than the first one.

To address the issue with our demo’s video quality being low because of streaming through Zoom and also partly because the demo was filmed in the dark, we are planning on using a nicer camera to film the demo and brightening up the scene of our product so that the camera can capture our product better and in a more impressive manner.

Below are images of the a comparison between the first and second camera’s White Balance issue. Note that the better camera captures the yellow hue of the room better, doesn’t make the room looks washed out. Because of this, the image quality seems more crips as compared to the other camera where the exposure and the white balance seems to be set too high.

This image shows the effect of auto-chroma keying. The details about the user-interface built for this feature can be read in Breyden’s post.