{"id":252,"date":"2024-04-27T22:54:49","date_gmt":"2024-04-27T22:54:49","guid":{"rendered":"http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s24-teame1\/?p=252"},"modified":"2024-04-28T01:58:57","modified_gmt":"2024-04-28T01:58:57","slug":"team-report-for-4-27-24","status":"publish","type":"post","link":"https:\/\/course.ece.cmu.edu\/~ece500\/projects\/s24-teame1\/2024\/04\/27\/team-report-for-4-27-24\/","title":{"rendered":"Team Report for 4\/27\/24"},"content":{"rendered":"<p style=\"text-align: center\"><b>Main Accomplishments for This Week<\/b><\/p>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">3D printing of phone attachment completed<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Fully functional web application (lap-top based)<\/span><\/li>\n<\/ul>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"size-medium wp-image-233 aligncenter\" src=\"http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s24-teame1\/wp-content\/uploads\/sites\/303\/2024\/04\/Screen-Shot-2024-04-27-at-8.42.13-PM-300x156.png\" alt=\"\" width=\"300\" height=\"156\" srcset=\"https:\/\/course.ece.cmu.edu\/~ece500\/projects\/s24-teame1\/wp-content\/uploads\/sites\/303\/2024\/04\/Screen-Shot-2024-04-27-at-8.42.13-PM-300x156.png 300w, https:\/\/course.ece.cmu.edu\/~ece500\/projects\/s24-teame1\/wp-content\/uploads\/sites\/303\/2024\/04\/Screen-Shot-2024-04-27-at-8.42.13-PM-768x400.png 768w, https:\/\/course.ece.cmu.edu\/~ece500\/projects\/s24-teame1\/wp-content\/uploads\/sites\/303\/2024\/04\/Screen-Shot-2024-04-27-at-8.42.13-PM-1024x533.png 1024w\" sizes=\"(max-width: 300px) 100vw, 300px\" \/><\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"size-medium wp-image-241 aligncenter\" src=\"http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s24-teame1\/wp-content\/uploads\/sites\/303\/2024\/04\/Screenshot-2024-04-27-at-8.52.38\u202fPM-300x134.png\" alt=\"\" width=\"300\" height=\"134\" srcset=\"https:\/\/course.ece.cmu.edu\/~ece500\/projects\/s24-teame1\/wp-content\/uploads\/sites\/303\/2024\/04\/Screenshot-2024-04-27-at-8.52.38\u202fPM-300x134.png 300w, https:\/\/course.ece.cmu.edu\/~ece500\/projects\/s24-teame1\/wp-content\/uploads\/sites\/303\/2024\/04\/Screenshot-2024-04-27-at-8.52.38\u202fPM-768x343.png 768w, https:\/\/course.ece.cmu.edu\/~ece500\/projects\/s24-teame1\/wp-content\/uploads\/sites\/303\/2024\/04\/Screenshot-2024-04-27-at-8.52.38\u202fPM-1024x457.png 1024w\" sizes=\"(max-width: 300px) 100vw, 300px\" \/><\/p>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Progress on displaying text via bluetooth<\/span><\/li>\n<\/ul>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"size-medium wp-image-248 aligncenter\" src=\"http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s24-teame1\/wp-content\/uploads\/sites\/303\/2024\/04\/IMG_0285-225x300.jpeg\" alt=\"\" width=\"225\" height=\"300\" srcset=\"https:\/\/course.ece.cmu.edu\/~ece500\/projects\/s24-teame1\/wp-content\/uploads\/sites\/303\/2024\/04\/IMG_0285-225x300.jpeg 225w, https:\/\/course.ece.cmu.edu\/~ece500\/projects\/s24-teame1\/wp-content\/uploads\/sites\/303\/2024\/04\/IMG_0285-768x1024.jpeg 768w\" sizes=\"(max-width: 225px) 100vw, 225px\" \/><\/p>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Final presentation<\/span><\/li>\n<\/ul>\n<p style=\"text-align: center\"><b>Risks &amp; Risk Management<\/b><\/p>\n<p><span style=\"font-weight: 400\">The biggest risk now is the major video input function, getUserMedia, is restricted to be used in secure contexts, including localhost and HTTPS. However, to enable phones to open the webpage, we are currently deploying the web application in AWS with HTTP. So, the webcam does not work if we open the web app on our phone. We are looking for methods to solve this issue, but if we cannot fix it, we will revert back to a laptop based implementation.<\/span><\/p>\n<p style=\"text-align: center\"><b>Design Changes<\/b><\/p>\n<p><span style=\"font-weight: 400\">We decided to switch from a mobile app to a web app due to lack of MediaPipe support for IOS. In order to receive an accurate prediction, it is necessary to incorporate pose landmarks, which there is limited support for in iOS. By using a web app, we can incorporate necessary landmarks in a way we are already familiar with.<\/span><\/p>\n<p style=\"text-align: center\"><b>Schedule Changes<\/b><\/p>\n<p style=\"text-align: left\"><span style=\"font-weight: 400\">No schedule changes<\/span><\/p>\n<p style=\"text-align: center\"><b>Additional question for this week<\/b><\/p>\n<p><span style=\"font-weight: 400\">We conducted unit tests on the CV module, and since our ML function is dependent on CV implementation, our tests on ML accuracy and latency were conducted after CV-ML integration.<\/span><\/p>\n<p><span style=\"font-weight: 400\">For <strong>CV recognition<\/strong> requirements, we tested gesture recognition accuracy under two metrics: signing distance and background distraction.\u00a0<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">For signing distance: we manually signed at different distances and verified if hand and pose landmarks were accurately displayed on the signer. We adopted 5 samples at each interval: 1ft, 1.5ft, 2.0ft, 2.5ft, 3ft, 3.5ft, 4ft. Result: proper landmarks appear 100% of the time at distances between 1-3.9ft.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">For background distractions: we manually created different background settings and verified if hand and pose landmarks were accurately displayed on the signer. We adopted 20 testing samples in total, with 2 samples at each trial: 1-10 distractors in the background. Result: landmarks are drawn on the target subject 95% of time (19 out of 20 samples).\u00a0<\/span><\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p><span style=\"font-weight: 400\">For <strong>translation latency<\/strong> requirements, we measured time delay on the web application.\u00a0<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">For translation latency: we timed the seconds elapsed between the appearance of landmarks and the display of English texts. We adopted 90 samples: Each team member signed each of the 10 phrases 3 times. Result: prediction of words appeared an average of 1100 ms after a gesture. Translation appeared an average of 2900ms after a sentence<\/span><span style=\"font-weight: 400\">\n<p><\/span><\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p><span style=\"font-weight: 400\">For the <strong>ML accuracy<\/strong> requirements,\u00a0<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">The validation and training accuracies from training the dataset were represented graphically, with the training accuracy reaching about 96% and validation accuracy reaching about 93%. This suggests that the model should have high accuracy, with the possibility of some slight overfitting.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">In order to test the accuracy while using the model, each of our team members signed each phrase 3 times along with 3 complex sentences using these phrases in a structure that was similar to typical sign language. Result: phrases translation accuracy: 82\/90 of the phrases were accurate, Sentences translation accuracy: 8\/9 sentences were accurate. Note that it is a bit difficult to determine if a sentence is accurate because there might be multiple correct interpretations of a sentence.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">While integrating additional phrases into the model, we performed some informal testing as well. After training a model consisting of 10 additional phrases, we signed each of these signs and determined that the accuracy was significantly lower than the original model of only 10 signs from the DSL-10 dataset, without around every \u2153 translations being accurate. Limiting to only 1 additional phrase posed the same low accuracy, even after ensuring the data was consistent among each other. As a result, we decided that our MVP would only consist of the original 10 phrases because this model proved high accuracy.<\/span><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Main Accomplishments for This Week 3D printing of phone attachment completed Fully functional web application (lap-top based) Progress on displaying text via bluetooth Final presentation Risks &amp; Risk Management The biggest risk now is the major video input function, getUserMedia, is restricted to be used in secure contexts, including localhost and HTTPS. However, to enable &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/course.ece.cmu.edu\/~ece500\/projects\/s24-teame1\/2024\/04\/27\/team-report-for-4-27-24\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Team Report for 4\/27\/24&#8221;<\/span><\/a><\/p>\n","protected":false},"author":812,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[6],"tags":[],"_links":{"self":[{"href":"https:\/\/course.ece.cmu.edu\/~ece500\/projects\/s24-teame1\/wp-json\/wp\/v2\/posts\/252"}],"collection":[{"href":"https:\/\/course.ece.cmu.edu\/~ece500\/projects\/s24-teame1\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/course.ece.cmu.edu\/~ece500\/projects\/s24-teame1\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/course.ece.cmu.edu\/~ece500\/projects\/s24-teame1\/wp-json\/wp\/v2\/users\/812"}],"replies":[{"embeddable":true,"href":"https:\/\/course.ece.cmu.edu\/~ece500\/projects\/s24-teame1\/wp-json\/wp\/v2\/comments?post=252"}],"version-history":[{"count":1,"href":"https:\/\/course.ece.cmu.edu\/~ece500\/projects\/s24-teame1\/wp-json\/wp\/v2\/posts\/252\/revisions"}],"predecessor-version":[{"id":253,"href":"https:\/\/course.ece.cmu.edu\/~ece500\/projects\/s24-teame1\/wp-json\/wp\/v2\/posts\/252\/revisions\/253"}],"wp:attachment":[{"href":"https:\/\/course.ece.cmu.edu\/~ece500\/projects\/s24-teame1\/wp-json\/wp\/v2\/media?parent=252"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/course.ece.cmu.edu\/~ece500\/projects\/s24-teame1\/wp-json\/wp\/v2\/categories?post=252"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/course.ece.cmu.edu\/~ece500\/projects\/s24-teame1\/wp-json\/wp\/v2\/tags?post=252"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}