This last week I have worked on the board with Kemdi and also have looked into dockerization of the project, along with buying some new parts. Kemdi and I connected the peripherals to the board and first ran into some troubles with audio. The microphone was able to capture correctly but the speaker did not seem to work. We suspected it had something to do with the audio input and outputs, and we were able to figure it out although the speaker was very weak which led me to find a bigger one to purchase. We then tried to run the speech to text code on the Raspberry Pi 4, but there were a lot of errors. Although the code works fine on Kemdi’s computer, we believe that the raspberry Pi relies on certain native code which gives us errors when we tried to run it. There are two weird programs jackd and alsa. We are not sure how to continue on that front and have requested help from a TA. I then continued into researching how to create a docker container for our project as I was unfamiliar with it and our project requires it. We are still on schedule although we need to fix our board problem as soon as possible.
Kemdi Emegwa’s Status Report for 3/8
This week, my primary focus was on testing and debugging the microphone integration within our Raspberry Pi setup. I dedicated significant time to troubleshooting the microphone functionality issues encountered when executing our code. This involved meticulously reviewing error logs, verifying hardware connections, and running numerous diagnostic tests to pinpoint the problem. Additionally, I explored different software configurations and settings to identify compatibility challenges with the microphone.
In parallel to the debugging process, I worked extensively on refining our existing codebase. The goal was to enhance compatibility and ensure greater stability when running directly on the Raspberry Pi. This refinement process included optimizing performance, addressing potential memory usage concerns, and ensuring that our code efficiently interfaces with the hardware. The improvements made this week will set a solid foundation for the upcoming integration work.
Despite the encountered difficulties, our project remains on schedule. Through careful evaluation, we concluded that pivoting to a hardware setup utilizing a single sound card for both the speaker and microphone would be beneficial. This decision should simplify integration significantly and resolve the compatibility issues previously faced. Next week, I will specifically focus on testing and integrating this revised speaker/microphone configuration, which should maintain our progress and help us stay aligned with our overall project timeline.
Justin Ankrom’s Status Report for 3/8
This week, I setup the website that will live on the physical hardware as well the website we host. Originally, we had planned to have only one website which we hosted, but decided to go a different route. The reason for this was because if we hosted everything, we would’ve needed some form of authentication so we would have had to stored user information which goes against our security policies. So I had to come up with a new approach. I came up with using 2 websites instead: 1 that lives on the hardware and another that we host. The one that lives on the hardware will be where people can set their VM configuration (VM ip address) and also look at the music they have saved, and the website we host will be strictly for setup instructions and for data privacy terms of service. This means that what we host is purely static and applies to every single user, while user configuration lies on the client side. This change means that I had to scrap almost all of the existing website code and restart. I spent a lot of time coming up with this 2 website approach and thinking about how I wanted to do it. For our hosted website, I am still using React and Next.js and hosting it on Vercel. For the client website, I decided to serve a simple HTML page on a Flask app. This is because on the client side, we have very limited resources, so I decided to go with a very lightweight approach. I have initial websites that work for both client side and our side. A lot of it is filled with filler content but this is fine as it will be quick to update the actual content since the overall layout is established.
Based on this, I think my progress is on schedule given the recent pivots we discussed in the team status report. Next week I want to work on filling out the actual content of our website, meaning setting up VM setup instructions, and want to setup a docker container for at least one open source model so we can get ahead with that. I also want to pick the exact open source models we want to use so we have a finalized list of those models.
Here are some pictures of the websites.
Design Report
Here is a link to our design report: https://docs.google.com/document/d/1AmPFfLT10oq7eKXW-1Z7OjMkhDm0K1LW/edit?usp=sharing&ouid=113756577354127239169&rtpof=true&sd=true
Team status report for 3/8
The most significant risk that could jeopardize the success of our project is getting everything to run on the board. As will be explained below, we are adding 2 new features and moved some part of website to being hosted on the board, which means we will need more resources on the board itself. To mitigate this risk, we will have an MVP up and running ASAP to test on the board. This way we can see if we will need to upgrade to a different board or if our raspberry pi 4 with 4gb ram is enough.
This week we decided to make some pretty substantial changes based on the design presentation feedback:
- Add 2 new features to increase scope of our project: (1) an timer feature and (2) an mp3 like feature where users can upload songs to their device and can play them from their device.
- We decided to move part of the website from being something we hosted to being hosted on the device. The website had 2 main purposes: (1) configure your device by passing in your VM endpoint and (2) everything else such as setup instructions, data privacy policies, and docker containers. Everything in (2) is strictly static, meaning that the same content can be used by every single user, so we are still going to be hosting this part. However, we decided to move (1) to being hosted on the actual device for each user. By doing it this way, we eliminate all 3rd parties, including ourselves. If we had kept (1) on our end, it would require us or a 3rd party to store user info (we chose Clerk which is a 3rd party but if we had implemented auth ourselves we would have had to store user info) which is against what we want to do. By doing the configuration on device, we eliminate the use of any 3rd party system and then the user owns all parts of the system which is our intended goal.
- We held a meeting with David Brumley and he gave us some advice regarding how to define privacy. Based on his feedback, we will be adding a terms of service agreement to our website to give users visibility into how their data is being used. We want to make it explicit to them that we nor any other 3rd party has access to their data or information at any point, including that their data is not being stored, their data is not being reused to train any models, etc. We need to make it very explicit that they own all aspects of the user experience.
- We also made the choice to have the users setup SSL encryption on their VMs (with instructions on our end). By doing so, we are not prone to man in the middle attacks between the devices and the VMs when making endpoint requests, which was a point of concern. After this is done, we can ensure that all privacy is being maintained with regards to the data transfers.Based on these changes, this is a rough estimate of what we want our schedule to look like in the upcoming weeks:
- Week 1-2: finalize design report, get rid of auth and move configuration to a separate device hosted website, start working on dockerization containers, start testing speech to text and text to speech.
- Week 3-4: Polish out UI and have it fully working, finish ToS, have dockers ready and on website with instructions, test individual components
- Week 5-6: test everything e2e, setup instruction guidebook
- Week 7: slack time to finish up anything we didn’t finish before. This accounts for unforeseen circumstances and pivotsWeek specific report:
- Part A(David):Our product has one main global factor that it affects. It is that our product ensures privacy to the user. While in the US this is mainly a data privacy issue as our government is most likely not actively using our data to control us, in other parts of the world this could be helpful against more controlling governments and therefore offer physical protection. In areas where there is no freedom of speech or less of it, a voice assistant in the house can be a danger if the data is sent out to an unknown location.
This product could also be of assistance in protecting government officials or anyone with confidential information. Because most voice assistant companies are in the US, there might be some mistrust for foreign people, especially those with confidential information being spoken in their houses. Our product ensures that all the data is kept within the user’s control, so that people from across the world can feel free to say anything they want and have it not be sent to a US server.
- Part B: (Kemdi Emegwa)Voice Vault aims to preserve the right to privacy and the right to consent while still allowing users to leverage state of the art artificial intelligence. By allowing the user to host their own model whether that be on the cloud or on their own local server, they gain the ability to dictate how their data is stored/used. At a time when AI/ML advancements have come perpendicular to user concerns about data privacy, our lightweight solution can bridge the gapPart C (Justin Ankrom): Voice Vault minimizes environmental impact by reducing energy consumption and electronic waste. It runs on a low-power 4GB RAM Arduino 4 board and has local storage via microSD which reduces dependence on external servers for storage, lowering the system’s carbon footprint. Its customizable design extends hardware lifespan by allowing upgrades instead of full replacements, reducing electronic waste. By supporting self-hosted LLMs, Voice Vault eliminates reliance on large-scale data centers, further decreasing energy consumption.
Team status report for 2/22
This week we spent a lot of time developing with the device, now that we have the Raspberry Pi on hand. David did research into how we are going to begin integrating our other hardware components like the speaker/microphone into our solution. Kemdi worked on allowing the device to begin communicating with first party hosted models. Meanwhile, Justin continued to lead the effort on developing the website alongside out strategic pivot we made a couple weeks ago. After the pivot we made, we don’t foresee anymore changes to our plan. The main risk right now is integrating our hardware with our software properly. This will likely be the most time consuming task.
Justin Ankrom’s Status Report for 2/22
This week I worked on setting up user board configuration through our software, which we store with Clerk metadata. Progress is on schedule. In the next week, I hope to refine website so it is fully functional and done and reflects the changes we made with our pivot, and also hope to do some preliminary testing of website and board configurations.
David Herman’s Status Report for 2/22
This week I did some more work into the Raspberry Pi and setting it up. Also have done some research into how we will integrate all the peripherals to the board as the parts have just come in. We hope to have the physical hardware (board, microphone, speaker) setup by next week.
Kemdi Emegwa’s Status report for 2/22
This past week, I spent some time testing the code I wrote last week, which allowed the device to send queries to a model hosted by a third party. Additionally, I started writing code which allowed the device to target a first party model hosted on a docker container. We are currently ahead of schedule and this next week, I will spend finalizing the abovementioned code.
Justin Ankrom’s Status Report for 2/15
This week I added authentication to the our website. We chose to use Clerk as our authentication provider, since we get up to 10k active users for free and it makes it very easy to setup auth without us having to worry about setting it up ourselves. Now a user will have to be authenticated to enter our website. My progress is on schedule. Next week I will add a functionality for users to link their account with their physical board so we can ensure only they will be able to access and modify their board through our software.
