Author: dherman

This week I worked on two main things: server side streaming and 3d printing. For server side streaming, we were worried about end to end latency as current testing seemed to be around 8 seconds. To eliminate this, we decided to implement streaming for both the server/model and the on board speech/text models. I did the server side and modified the code to allow the server to return words as they are generated which massively increases the speed. This makes the speech a little choppy due to it receiving one word at a time, so next week I might try to change the granularity to maybe 5-10 words. I also have done some work on improving the 3d model. The first iteration came out pretty well, but I want to add some more components to make it complete like a sliding bottom and a sliding top to prevent the speaker from falling out if moved upside down. I am hoping this is the final iteration, but if it doesn’t come out according to plan then I might have to do one more print. I am on schedule, and along with the aforementioned things I will also try to do some testing/verification for next week.

This week I worked on Docker and some VM stuff. Throughout the week I tried to fix the docker container to run Ollama inside, but to no avail. Me and Justin also tried working on it together, but we weren’t able to fully finish it. Justin was able to fix it later, and I was also able to make a working version as well, although we are going to use Justin’s version as the final. The main issue for my version was that my gpu wasn’t running with the model in the container. I fixed this by not scripting ollama serve in the Dockerfile initially, and just downloading Ollama first. Then I would be able to Docker run with all gpus to start my container. After that I could pull the models and also run Ollama serve to have a fully functional local docker container working. If we were to have used this version I could script the pulling and running of Ollama serve to occur after running Docker. Earlier in the week I also tried to get a vm with a T4 GPU on gcp. However, after multiple tries across servers, I was not able to successfully acquire one. Me, Kemdi, and Justin also met together at the end of the week to flesh out the demo which is basically fully working. I am on schedule, and my main goal for next week is to get a 3d printed container for the board and speaker/mic through FBS.

This week I worked on the model/server code. From last week, I was initially trying to get the downloaded model from hugging face running locally. However there were many issues, dependencies, and OS problems. After trying many things, I did some more research into hosting models. I came across a software named Ollama, which allowed for easy use of a local model. I coded some python to create a server which took in requests to run through the model and then return to the endpoint.

As seen here, we can simply curl a request in which will then be parsed into the model and returned. I then tried to look into dockerization of this code. I was able to build a container and curl into its exposed port, yet the trouble I come across is that the Ollama code inside does not seem to be running. I think it stems from the fact that to run Ollama in python, you need two things (more like three things): Ollama package, Ollama App and related files, and an Ollama model. Currently, the Ollama model and App are on my pc somewhere, so when I initially tried to containerize the code the model and app were not included, only the Ollama package (which is useless by itself). I then tried pasting those folders into the folder before building the Docker image, and they were still not running. I have played around with mounting images, and other suggested solutions online, but they do not work. I am still researching into fixing it, but there are few resources that pertain to my exact situation as well as my OS(windows). We are currently on schedule.

This week I have worked on setting up the docker container and the software that deals with the model. I currently have working code that can take in a curl request with username and password and if correct the request will then be sent to a model. The model currently is a dummy model that just spits out a number, but once the actual model is put in place it should be working. For getting the actual model, I have requested and downloaded a Llama model from HuggingFace. I am currently working through setting up the model as it needs certain requirements. I have also done some research into new parts for speaker/microphone and have settled on one that should fix our issues. Our project is a little behind due to the hardware not working, and we hope to fix that next week with the new part. I personally hope to accomplish getting the model set up so that the established data flow of a command coming in, authentication checking, running prompt through model, and spitting out result works as intended.

 

As can be seen in this picture, I send a curl command from terminal and the server receives it and gives back a prediction (from dummy model).

This last week I have worked on the board with Kemdi and also have looked into dockerization of the project, along with buying some new parts. Kemdi and I connected the peripherals to the board and first ran into some troubles with audio. The microphone was able to capture correctly but the speaker did not seem to work. We suspected it had something to do with the audio input and outputs, and we were able to figure it out although the speaker was very weak which led me to find a bigger one to purchase. We then tried to run the speech to text code on the Raspberry Pi 4, but there were a lot of errors. Although the code works fine on Kemdi’s computer, we believe that the raspberry Pi relies on certain native code which gives us errors when we tried to run it. There are two weird programs jackd and alsa. We are not sure how to continue on that front and have requested help from a TA. I then continued into researching how to create a docker container for our project as I was unfamiliar with it and our project requires it. We are still on schedule although we need to fix our board problem as soon as possible.

This week I did some more work into the Raspberry Pi and setting it up. Also have done some research into how we will integrate all the peripherals to the board as the parts have just come in. We hope to have the physical hardware (board, microphone, speaker) setup by next week.

This week I ordered a replacement microphone as the one previously ordered from 18-500 inventory was too expensive and needed to be reserved for other potential groups. I then configured our raspberry Pi by downloading the OS on my laptop and plugging in the SD card to the board. I also configured the board to connect to CMU-SECURE.

I looked through the inventory to get the three main components we need: speaker, microphone, and raspberry Pi 4. The inventory had both the raspberry Pi and the microphone which I ordered, but no speaker. I did some research on a speaker that would be able to connect to a raspberry pi 4 and ordered that as well. Our project is on schedule. I hope to be able to connect the speaker, board, and microphone next week to have basic IO.