Team B1: Nature Photography Robot – Page 3 – Justin K., Sid M., Fernando P. – Carnegie Mellon ECE Capstone, Spring 2022

March 19, 2022

Justin’s Status Report for March 19

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress.

I was able to complete a couple of tasks this week. The first, and most important was the physical setup of the robot. I was able to finish all of this task, except for fixing the camera tower to the tripod. This task involved creating a 3 layered tower (camera, jetson nano, camera) where the cameras are screwed in and able to scan their full range of motion.

The next task I completed was updating the sharpening algorithm. We plan on having an algorithm automatically determine the number of effects to apply to our photos. With this in mind, it is important that the scales for applying the image processing algorithms in our library are natural and allow for enough flexibility. Our initial implementation of the ‘Sharpening’ algorithm would apply a single kernel one time for each level of sharpening. By the second or third iteration, the picture was unusable. This leaves very little flexibility in applying this effect. After research, I changed the algorithm such that the amount of sharpening is the size of the sharpening kernel. This led to a much more natural sharpening scale.

I also did the ethics reading and assignment this week.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

We are on schedule with the Gantt chart shown in the design review and still have an excess of slack. We will test our first design this upcoming week.

What deliverables do you hope to complete in the next week?

Initial testing results for the robot’s search/detection and the editing algorithm.

March 5, 2022

Fernando’s Status Report for March 3

What did you personally accomplish this week on the project?

This week I helped work on the Design Review paper.

Is your progress on schedule or behind?
I’m a little behind on schedule as I would like to make more changes to the Design Review and plan on finishing it in the days to come.

What deliverables do you hope to complete in the next week?
A completed Design Review and some interface between the KLT and Nvidia Jetson Nano.

February 26, 2022

Team’s Status Report for February 26

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

The largest risk to the project is not finishing the setup on time. We made satisfactory progress this week, but it is essential we finish in the upcoming week. To achieve these goals we are parallelizing the tasks needed. Sid is working on the CNN, Fernando is working on the Jetson/Camera software integration, and Justin is working on the physical setup. The biggest risk area is the physical setup because our expertise is not on woodworking or manufacturing and a lot can go wrong with the hardware. If worst comes to worst we can begin testing with a temporary set up the following week.

Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward

During this week we began to design the physical stand for our robot. Given this class’s focus is ECE, we overlooked the importance of this step. We realized that the process of stacking 2 cameras and a jetson vertically would not be as trivial as we thought, so we created sketches and bought materials for our design. The material costs were inexpensive and we covered them with our personal funds.

Provide an updated schedule if changes have occurred.

No schedule updates.

February 26, 2022

Justin’s Status Report for February 26

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress.

I spent Sunday working on the design review presentation, and I used the dedicated class time for reviewing other groups’ presentations. My main focus for the week was on the physical setup of the project. Our parts came in this week, so I worked with my team members to set up the Jetson Nano and get the cameras connected and working. I also went on a trip to home depot to find the parts needed for the stand: 3 wooden boards, threaded rods, nuts, and bolts. I had a tripod from a previous project.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

We are on schedule with the Gantt chart shown in the design review. We will finish the setup this week, so we can test the following week.

What deliverables do you hope to complete in the next week?

The final physical tripod setup.

February 20, 2022February 20, 2022

Fernando Paulino’s Status Report for February 19

What did you personally accomplish this week on your project?

This week I focused on familiarizing myself with our cameras’ SDK and zoom, focus, and autofocus functionality. I also worked on the tracking algorithm. Thus far, we have an LKT for simple translation transformations of our target.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

After settling into our schedule and more concrete individual tasks, I believe we are on schedule. With Sid making progress on the detection section, I shifted my focus to mostly tracking for now. We will be ready to start testing the baseline tracking algorithm by next week as planned.

What deliverables do you hope to complete in the next week?

By next week, the baseline tracking algorithm, an LKT for full affine transformations, will be ready for testing. I would also like to implement some code for synchronizing the camera with the tracker.

February 20, 2022

Team’s Status Report for February 19

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

The most significant risk to our project is not finishing the physical setup. We faced a delay ordering our parts, so we already plan to use our setup slack. Therefore, it is of the utmost importance that we work diligently this week. We made adequate progress on our software this week, so we should be ready to test the following week. If setup takes longer than expected, we can likely complete the testing in half a week, though a full week is dedicated.

We decided to change the cameras we were using as part of the system.

We originally planned to use 12MP cameras however decided to switch to 8MP cameras instead. We also decided to purchase a servo motor base for the PTZ system (pan tilt zoom) separately as opposed to an assembled system with the camera. These changes were suggested to us by our mentors as they resulted in a reduction in the expenditure on these parts of the system. This idea made sense since we did not have to actually need a great camera to photograph animals and needed one capable of photographing test subjects and since assembling the setup ourselves would be a more economical option too. Because of this, we get more wiggle room in our budget and space for any expenditure that may be needed during the later stages of the project.

We also discovered that our jetson nano had only one MIDI port, so we added an extra adapter to our plans. This part was ordered with the cameras and only cost $50, so we still have extra budget to spare.

February 20, 2022

Sid’s Status Report for February 19

I did research about algorithms to consider for the Detection phase which involves pointing out the location of the subject with bounding boxes around it. Refreshed my knowledge on the way CNNs work and then weighted out the pros and cons of different algorithms to narrow down 3 that we will be testing and evaluating on training datasets. The attached file shows the summary of the research and includes the URL’s of websites that would be a reference for when the algorithms are being implemented.

Apart from this, I was able to select and download a large dataset of images of animals that would be our training data for the models. I began setting up the environment, etc. for the YOLOv5 algorithm and worked on formatting the images for being inputted here.

My progress on the project is a little behind however it should not be difficult to make up time. To catch up I will begin testing and evaluating the models as soon as possible and do this while simultaneously working on building the physical setup during lab time (as planned for by the schedule)

I plan to pick an algorithm as well as have data to support the reason for doing so. Along with this we expect to have most of the physical setup ready and be able to relay the real-time from the camera onto the computer, using the Jetson.

February 20, 2022

Justin’s Status Report for February 19

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress?

I began the week researching photo editing from a user stand point. I watched multiple tutorials on properly editing pictures. From this study I selected which image editing algorithms would be necessary for our project.

Notes Taken on an Image Editing Tutorial

Afterward, I researched the implementation of these editing algorithms. This part was difficult, because editing softwares do not share their source code, and there is not a universal definition for these algorithms. I used my best judgement to find the approaches which were simple and best fit our use case. I then implemented these algorithms.

Our Final List of Image Editing Algorithms

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

I finished my assigned task for this week and am currently on schedule.

What deliverables do you hope to complete in the next week?

Next week I hope to have a finalized physical setup for our stand+cameras. Our parts have arrived, so we are doing setup tasks next week.

February 14, 2022

Fernando Paulino’s Status Report for February 12

What did you personally accomplish this week on the project?
This week I contributed to our project proposal and the corresponding
slides. I also reached out to some nature photographers to learn more about
their processes in photographing animals and some of the challenges they meet.

We also spent some time researching different kinds of cameras that may be
compatible with the Jetson Nano and would adequately meet our needs. Lastly, I reviewed some of my previous projects in recognition in preparation for the detection systems we will implement for this project.

Is your progress on schedule or behind?
I would have liked to have made some solid progress on the actual code, as well as some more research on OpenCV’s libraries to ascertain which, if any, may be of use to us so that we don’t have to waste time making something that’s already openly available.

What deliverables do you hope to complete in the next week?
In the next week, I hope to have some deliverable code in animal detection that utilizes an animal image database, as well as a more concrete log of responses nature photographers who have agreed to help us with their insight and experiences.

February 13, 2022

Team’s Status Report for February 12

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

Our most significant risk at the moment is falling behind on the initial project setup. Having the camera stand functional is essential to our plan of having multiple rounds of development and testing. We recently received a message from Professor Mukherjee suggesting some changes to our camera order, so our order has not been finalized.

Our parts should arrive a week after our original plans, but right now we still feel confident in our ability to finish the setup phase on schedule. We gave conservative time estimates on the setup tasks (almost 2 weeks), and have extra setup slack. With that being said, we must finalize the order update ASAP and be extremely diligent over the next few weeks.

We have decided to change the camera from 12MP to 8MP. This reduction will not significantly impact any of our software plans. We are not able to produce a professional quality camera system within our budget, and this choice will allow us to still complete a proof of concept while saving some of our budget for an emergency.

Provide an updated schedule if changes have occurred.

Our schedule will be slightly modified by this setback. We will finish the initial algorithm design while the parts are shipping, and push the setup tasks into the designated setup slack.