This was our first Zoom quarantine week!  I think we managed quite well taking everything into account.  This week was mostly us rehashing our project plan.  We spent 10:30-12:30 on Monday and Wednesday during what would be our normal class time discussing what changes we need to make and contingency plans in case our Plan Bs don’t work out.  We decided a few key things – not to build a physical platform or use the motors (TechSpark closed yesterday so this was a good decision), to use SolidWorks (and some other software potentially) to model and animate the physical platform, continue using our RPis, cameras and mics to get audio and visual signal input and process them.  We spent quite a while on Friday researching and looking for cardioid unidirectional mics and unfortunately could not find any that fit our size and cost requirements.  Instead we decided to build baffles for the omni-directional Adafruit mics to contain the range of sound (also consulted on this with Professor Sullivan).  We also spent time writing our SOW.

We are definitely behind schedule and lost a week because of corona but will get our parts next Thursday and until then work on the software parts of our project (classifiers and SolidWorks animation specifically).  Since the course requirements have significantly changed this loss of time has been taken into account in our new project goals and while we know we won’t be able to build our original project, we are hoping to come close!  🙂

This week we spent most of our time on our design review presentation, finalizing parts and design decisions regarding interface communications, etc.  We met a couple times last weekend and once during the week to get these done and help Neeti rehearse.  Yesterday we met again to chalk out our design review report and get some of the things we could do before receiving parts done.  I setup a git repo and made sure we could all push and commit from both Pis.  Then I did some research into datasets available for gesture detection and found these: https://lttm.dei.unipd.it/downloads/gesture/ and http://www.nlpr.ia.ac.cn/iva/yfzhang/datasets/egogesture.html.  Because our gestures aren’t readily available, we might need to manufacture our own (this might not be enough data to train a neural net, but it also might be since we only basically need two gestures).  In our peer feedback, some people brought up the point that CNN might not be the most accurate model to do gesture detection and so I emailed Professor Savvides who taught us Pattern Recognition Theory last semester and would be the best person to advise us on this.  Then, I worked with Shrutika on getting a TCP connection between the Pis working and processing keystrokes using Pygame.

I think we are a little behind schedule because we had aimed to get our manual mode working by spring break, but spring break is next week.  We thought that since we’d ordered our parts last Monday, we’d have them by Friday, but there seems to have been a slight delay in ordering.  We will probably be able to get our manual mode (without gesture detection) working by this Friday if our parts arrive by Wednesday.  If not, I’m hoping we get the parts by the end of the week and then we might be able to realistically get manual mode working by the end of the first week after spring break.

I have two onsites though and will be flying out this Wednesday.  I will try to work on our project (specifically the ML models) a bit during spring break since I’m back on March 10th.  I’m not too worried about us catching up on manual mode, but I do think we have a significant amount of experimentation to do with the microphones and that we should get started on the in parallel in order to save time.