Category: Alena’s Status Reports

This week I was able to figure out how to use Bluetooth classic with the pico w.  I was able to configure the pico as an A2DP source and connect it to the Raspberry Pi which acted as the sink, and was able to transmit audio from one to the other. However, I am having trouble integrating the microphone code with the bluetooth code, since the PCM to SBC encoding is messing up somewhere along the line. The output on the Raspberry Pi is very noisy, but you can vaguely make out what the voice is saying, it just sounds very robotic. I think there’s some more filtering that I’m going to have to do.

I also researched a GPT integration with the voice command algorithm and got it to work in conjunction with my existing algorithm, so I am looking in to making the voice command output more “human” instead of just hardcoding responses based on the recipe steps. I also was able to integrate the pyttsx3 with Raspberry Pi so that the outputs can be heard on a bluetooth speaker/headset. Overall, I made a lot of progress this week, and there are multiple options we can consider for the final project. Unfortunately the wireless mic needs to be tuned a lot more, so depending on how much time we have for that, the final product may need to have a wired mic.

This week I finalized the voice command algorithm including adding support for getting and posting requests and interfacing with Kayla’s web app code to send ingredients lists and receive recipes, refactored my original design to take better care of extraneous voice inputs by using a switch statement, and added a lot of extra functionality for the recipe process. I gave the user a lot more leeway into different choices they can make throughout the cooking process. In terms of hardware, I set up the Raspberry Pi 5 and installed all of the required libraries for the voice commands and the microphone. Then I was able to successfully run the voice command algorithm with the Pico microphone that I configured, and the entire system works as expected. Integration with Kayla is essentially finished.

I spent a lot of the first half of the week trying to figure out how to make the mic wireless, but I ran into countless issues figuring out how the lwip (lightweight tcp/ip protocol) on the pico W would work for sockets/sending and receiving information. Eventually I decided that I would spend time this coming week + over break to figure that out, and if not then take an approach with BLE (bluetooth). Also, Michelle and I are running into issues with our integration because we used different platforms for our picos (micropython and C/C++), so we are going to have to work on that as well. But otherwise, progress is pretty on track.

My next steps are to integrate the pyttsx3 responses into speaker output, integrate my code with Michelle’s, and figure out how to make the mic wireless.

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

• This week, I worked on the microphone integration with the voice commands. I was able to get the microphone to initialize and act as a USB microphone to my laptop, as well as have my voice commands algorithm recognize my input mic.

Is your progress on schedule or behind. If you are behind, what actions will be
taken to catch up to the project schedule?

• I am behind schedule in terms of the mic because it took a while to initialize, but now I need to look into the Pi to Pi communication and somehow reconfigure the mic so that instead of being a USB/UART connection, it acts as a wireless mic.

What deliverables do you hope to complete in the next week?

• In the next week I hope to have the TCP/Wi-Fi integration for the mic completed, as well as added enhancements to the voice command algorithm.

This week I continued to work on the microphone hardware initialization. I am running into some issues with exporting audio data, and am exploring options for how to send the data from the Pi to the voice command algorithm that I implemented. The integration of both is a bit challenging, and I’m still working on assessing the quality of the microphone. We may potentially need to switch out a part for a new microphone, but that will be finalized by mid-next week.

Progress is a bit behind because I’m running into more obstacles for initialization than I thought, but I will have the updated information relayed to Michelle by the end of the weekend. By next week, I hope to have this completed and fully integrated with voice commands. Then we can do some more integration testing with the whole system.

What did you personally accomplish this week on the project?

Continued to refine voice command algorithm and drafted some new ideas for selecting recipes with voice commands. Most of my work was done on the hardware bringup for the microphone. I was able to get that working with the pico, but there are some latency issues with capturing audio that I am running into. I am trying to debug it by checking the waveform and potentially saving the audio into some output file to test.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

So far pretty on schedule, but a bit behind on voice command/mic integration. I will focus more on debugging the microphone, and if needed, we may need to order a separate microphone.

What deliverables do you hope to complete in the next week?

In the next week, I will try to finalize integration between the mic and the voice commands/determine if we need a new mic, and continue to iron out any edge cases that show up in the voice command algorithm through our discussions.

What did you personally accomplish this week on the project?

I refined some of the voice commands so that ingredients listing is easier and filters out some “random” phrases when parsing the ingredients.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

The progress is slightly behind for embedded integration because I was hoping to have some more mic/voice command testing done but was not able to get that.

What deliverables do you hope to complete in the next week?

In the next week, I will try to finalize integration between the mic and the voice commands, and continue to iron out any edge cases that show up in the voice command algorithm.

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This week I focused on refining the voice command algorithm, which included fixing bugs for user speech duration and also optimizing some of the background noise filtering. I did some comparison between the Google Web Speech API and the Microsoft Azure API, and decided to stay with the Google speech API for accuracy. I also refined some of the commands themselves, and made a tentative list of the current commands that the bot will support. I also used the pyttsx3 library to successfully generated prompts to respond to the user for each of these commands.

In terms of the embedded side, I did some tinkering with the speaker and the amplifier, and was also able to start some of the initialization code for connecting to the speaker.

Is your progress on schedule or behind. If you are behind, what actions will be
taken to catch up to the project schedule?

In terms of voice commands, we are actually ahead of schedule. However, we are still slightly behind on hardware integration since the PCB layout still needs to be done. We’ve decided to push the PCB back a bit so that we can confirm everything works with the pico on the breadboard first. I will be shifting my focus from voice commands to hardware integration and firmware coding for the next week.

What deliverables do you hope to complete in the next week?

I hope to have the initial voice commands implementation finished, and also the microphone and speaker initialization with the pico done by this week.

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

Personally, I implemented the initial code for voice commands and have a working prompt/response system using the SpeechRecognition library. The code is able to recognize when the user says “Hey Chef,” and can parse some specific commands that the user inputs through voice. I also used the pyttsx3 library to successfully generated prompts to respond to the user.

I was also able to start the code for the microphone integration onto the Raspberry Pi pico since our parts came in. I soldered headers onto both of the picos and the microphone board.

I also did some research with Kayla to finalize our web app structure and the frameworks we will be using for front end and back end development.

Is your progress on schedule or behind. If you are behind, what actions will be
taken to catch up to the project schedule?

In terms of voice commands, we are actually ahead of schedule because they took less time than imagined. However, we are slightly behind on hardware integration because we ordered our parts later, and the PCB layout still needs to be done. This will be the main focus for the coming week.

What deliverables do you hope to complete in the next week?

I hope to have a finalized PCB layout with Michelle so we can send it to manufacturing, and also finish the microphone and speaker initialization using the pico. A stretch goal will be to incorporate my previous work on voice commands with the microphone and speaker.

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This week, I worked on the initial research and testing of the SpeechRecognition library. I was able to implement some code for speech-to-text conversion in Python using my computer mic. I also researched components for the PCB, as well as a suitable microphone to use. I also did some initial prototyping/code for word splicing, but that is subject to change as we progress with the design. We also finalized the plan for our project, creating a Gantt chart for deadlines.

Is your progress on schedule or behind. If you are behind, what actions will be taken to catch up to the project schedule?

My progress is mostly on schedule, but the PCB design is slightly behind schedule. Michelle and I will work on that in the mean time while parts are being ordered.

What deliverables do you hope to complete in the next week?

By next week, I hope to have a PCB designed and ready to send to manufacturing, as well as all necessary parts ordered for initial prototyping. I also hope to finalize which speech library we will use in Python, and iron out some details regarding word splicing.