Weekly Status Reports

• Currently, we still have to finish polishing the web application and facial recognition part of the project. The basic functionality of the web app – showing a camera feed and running the facial recognition system – is working pretty well. We still have to finalize things such as displaying the check-in and check-out logs, along with manual checkout (in case the system cannot detect a checked-in user). The other risk is racial recognition. Throughout our testing in the last 2 days, it is highly effective at distinguishing between diverse populations, but when the testing set contains many very similar faces, the facial recognition system fails to differentiate them. The new facial recognition library we are using is definitely much better than the old one though. We will try our best to flesh out these issues before the final deliverables.
• The only change we have really made this week is switching to a new facial recognition library, which drastically improves the accuracy and performance of the facial recognition part of the project. This change was necessary because the old facial recognition code was not accurate enough for our metrics. This change did not incur and costs, except perhaps time.
• There is no change to our schedule at this time (and there can’t really be because it is the last week).

UNIT TESTING:

Item Stand Integrity Tests:

1. Hook Robustness: Placing 20 pounds on each of the 6 hooks at a time and making one full rotation
2. Rack Imbalance: Placing 60 pounds on 3 hooks on one side of the rack and making a full rotation
3. Max Weight: Gradually place more and more weight on a rotating rack until the max weight of 120 pounds is reached
4. RESULTS: From these tests, we determined that our item stand was robust enough for our use cases. We found that though some of the wood and electronic components did flex (expected), it still held up great. The hooks were able to handle repeated deposit and removal of items, the rack did not tip over from imbalance, and even with a max weight of 120 pounds, the rack rotated continuously.

Integration Tests:

1. Item Placement/Removal: Placing and removing items from load cells and measuring the time it takes for the web app to receive information about it.
2. User Position Finding: Check-in and check-out users and measure the time it takes for the rack to rotate to a certain position.
3. RESULTS: We found that the detection of placing and removal of items from the item stand was propagated throughout the system very quickly, less than our design requirements. So no design changes were needed there. On the other hand, when we tested the ability of the rack to provide users with a new position or their check-in positions, we found that the time it took went way above our design requirements of 1 second. We did not think about the significant time it takes for the motor to rotate safety to the target location, and thus adjusted our design requirements to 7 seconds in the final presentation.

Facial Recognition Tests:

1. Distance Test: Stand at various distances away from the camera, and see if facial recognition starts recognizing faces.
2. Face Recognition Time Test: Once close enough to camera, time the time it takes for the face to be seen as new or already in the system.
3. Accuracy Test: Check-in various faces and check-out, while measuring if the system accurate maps users to their stored faces.
4. RESULTS: Spoiler: we switched to a new facial recognition library which was a big improvement over the old one. However, our old algorithm with the SVM classifier was adequate at recognizing people at the correct distance of 0.5 meters and within the time of 5 seconds. Accuracy though, took a hit. On very diverse facial datasets, our old model hit 95% accuracy during pure software testing. While this is good on paper, our integration tests involving this system found in real life that in a high percentage of the time, its was just wrong, sometimes up to 20%. With this data from our testing, we decided to switch to a new model using the facial_recognition Python library, which reportedly has a 99% facial accuracy rate. We recently also conducted extensive testing and found that its accuracy rate was well above 90-95% on diverse facial data. It still has some issues when everyone checked into the system look very similar, but we believe this might be unavoidable and thus want to build in some extra safeguards such as manual checkout in our web application (still a work in progress).

• There are a couple of significant risks that could jeopardize the success of the project. One risk is the stability of our hardware components. Through our long testing, we discovered that our NRF24L01 wireless transceivers were not reliable 100% of the time. A majority of the time transmission from the web app side of the system to the hardware side of the system worked, and other times the signal don’t go through, which definitely is not ideal when the system hinges on the transceivers to be the bridge between our 2 major subsystems.
• As a contingency plan, we purchased extra NRF24L01 transceivers that can be swapped out if something is wrong with the system. As a last resort, our web app keeps track of item stand transactions, which allows the customer to easily find user items (which is just a normal coat checking system).
• The second risk is facial recognition. This is also another very important component to our system. Without very high accuracy facial recognition, the system could prove to be frustrating to users who may not be able to retrieve their items easily. It is also a very hard thing to perfect and there is not much time left.
• As a contingency plan for the facial recognition system, we can allow the user to enter a name or identifier in place of scanning their face. Any other kind of identification method could work.
• Through our end to end testing, we discovered that some of our design requirements were not realistic. For example, we initially required that once a user’s face was matched, the stand should display the user’s position on the rack with an LED within 1 second. This did not account for the time it would take for the motor to rotate from it’s current position to the user’s position after they had checked in. Therefore, we changed our requirement to 7 seconds to allow the motor to rotate the the target position.
• Overall, there is not much change to the schedule.

RISKS AND MITIGATION

After testing most of our components and the entire system this week, we determined 2 big risks that could jeopardize the success of the project. One risk is wireless communication. In our system, we leverage NRF24L01 wireless transceivers to communicate wirelessly between our facial recognition system and item stand. Sometimes the communication is inconsistent. Data could be send from one transceiver but not received by another. This may be because we are toggling both transceivers between a receive and transmit state. We are planning on testing this more next week. Currently, we put a big delay between toggling the receive and transmit state, so the component has enough time to adapt.

The second risk is the accuracy of the facial recognition system. The facial recognition system is not as accurate as we would like at this moment in time. This is a big deal in our system because inaccurate face detection could result in the wrong user receiving their items. We plan to continue testing and refining the algorithm, especially since the team is wrapping up on the hardware component of the project.

DESIGN CHANGES

From the stand integrity part of our design requirements, we stated that each load cell should be able to hold 25 pounds. However, we have realized that the load cell we have purchased can only detect up to 22 pounds in weight. Additionally, when doing end to end testing this past week with items of various weights, the heaviest item that was placed on the rack was only 10-15 pounds. As a result, we are decreasing the maximum weight for each load cell from 25 pounds to 20 pounds. This would mean the the maximum total weight that can ever be on the item stand at once is 120 pounds. This modification does not incur any costs. It is simply more realistic with the weights for the types of items we expect users to place on the rack. For the other design requirements (detect an item has been added or removed within 1 second, 95% accuracy for facial recognition, etc), we will do further testing and make necessary modifications to attempt to satisfy them.

SCHEDULE

The schedule remains the same. However, we are utilizing some time in our slack time to fix any unforeseen issues and ensure our use case and design requirements are meant. Not considering these minor modifications, all members have completed their required tasks, and what remains is further testing.

VERIFICATION/VALIDATION

As a team, we plan to combine all the components into one system before running our tests. For facial recognition, we want to mostly implement the tests we described during the design proposal. We will test with users standing at various distances from the camera, with the ideal result being that users can only interact with the system within 0.5 meters. We also want to maintain accuracy, so we are planning to test our facial recognition accuracy with at least 20 different faces, with an ideal result of at least 95% accuracy. To test the integration between our hardware and software components, we plan to run tests that model real life scenarios, in addition to the specific tests we listed in our design proposal. For example, we can have several people check in and out of the system in various orders in order to determine if the system is able to accurately keep track of user belongings. For the item stand, we will be using various weights and seeing if the item stand is damaged or can rotate. We will also be introducing weight imbalance- putting a very heavy weight one side and rotating the item stand. Lastly, we want to time how long the process of checking-in our checking-out takes. We want to ensure that the motor can rotate the the correct position and that updates between the facial recognition system and item stand do not take long. This will mean making modification to our code to lessen certain delays or increasing the acceleration/speed of the motor.