Carnegie Mellon ECE Capstone, Spring 2020: Ashika K, Abha A, Jade T
This week, for the storytelling component alone, I finally finished grammar correction for nouns and I worked on creating a few more templates. I changed the format slightly to improve sentence tokenization, which makes it easier to send the story sentence by sentence to the …
On Monday I worked on cleaning up the socket interface code. I wrote read and write functions for both the server and the client. On Wednesday I worked on integrating socket code into our program, this consisted of creating a new main file for the …
On Sunday, I tried to debug the issues with showing an image on the eyes display through an SD card. Initially, my plan was to put an image of a cat eye on the SD card, and the eye display would read it. Most of the time was spent on trying to get sample code online working with the files that they had given. After working through a few issues, I was able to get an image of flowers (supplied by Adafruit) displayed on the screen. My next goal was to get my own image displaying on the screen. However, even after converting my image to the appropriate file type, the display couldn’t read it. After some internet digging, the issue was that the header of my image was not in the correct format. To fix it, I would need to convert the image to hexcode, and convert the first 40 bytes of hexcode to string to read the header (and I did not want to do this lol).
On Monday, I took a step back and tried other ways to convert my image to the correct file type. Eventually, one of the websites that I used was able to convert it with the appropriate header. I (finally) got the image onto the display. However, there was two issues with the image displayed (shocking). The first one was that the colors were messed up. The second was that the image would scroll down the display every time the image was refreshed (the origin wasn’t constant for some reason). This meant that if I changed the eye image to show a different sentiment, the eye wouldn’t be in the same location.
On Wednesday, I tried to debug the scrolling image situation from Monday. However, I was not able to figure out the issue. I also played with drawing lines and shapes instead to draw eyes instead of using issues. For example, I was able to do a few of these eyes that also show emotion well.
However, we realized that just the eyes is not enough to show sentiments. The shape of the mouth is also important. Therefore, we decided to buy one screen for the entire face, which would display eyes, nose, and mouth so that we could alter the shape of the eyes and mouth to show sentiment. I ordered the display and it should come in next week.
On Friday, I switched gears to writing code for moving the motors on the Raspberry Pi. Unfortunately, it took some time to setup an environment where I could work with a Raspi since I didn’t have a mouse at home. I wrote some basic code to move the motors.
Next week, I will test the motors code, integrate Ashika’s code with the text display, and work on the head display when that comes in. I am waiting to get the head display before I cut out the cardboard pieces for the head to ensure that I have the correct dimensions for the display cutoff first.
This week I started integration with Abha for the text display. On my end, I created a display window (shown above) in Python that runs on a separate thread from the main program. This window is the same size as the pi and displays the …
What we did: Abha worked on the face display on the hardware side, updated the CAD files to reflect the design changes for the head, and wrote code to move motors via a Raspi. Ashika wrote code to display the sentence output on a graphic …
On Sunday I decided to tackle the choppy audio quality that I was getting from the audio playback. I was using pydub’s AudioSegment to play the sound back, but it was choppy, so, I investigated a few other audio processing packages including pyglet, pygame, tkSpeak, as well as just running mpg123 on command line. After doing testing with each, I found that I couldn’t get pyglet, pygame or tkSpeak to work either install or work on the pi. So, I went with playing audio with mpg123. This worked surprisingly well and provided clean audio.
On Monday I found that the audio input wasn’t working very well on the raspberry pi. Originally I was using speech_recognition’s microphone class to record audio, however it would hang or never stop recording, even when I supplied a timeout. So, I decided that instead I would record 3 seconds of audio after the user has been prompted from the microphone and pass it into speech recognition on an audio file. This worked really well and allowed for the pi to take audio input.
On Wednesday I worked on integrating Ashika’s new ML code changes with the audio code on my laptop. This involved a lot of back and forth figuring out errors between the both of us.
On Friday I got the audio input/output processing code working on both the pi and my laptop. The ML + audio system works fully on my laptop. We realized that if we could install all the ML packages onto the pi, then we wouldn’t need to run the ML code on the laptop. So, I worked on installing ML packages on the pi. Unfortunately I ran into a lot of issues and plan to try just a bit more before giving up and working on writing code to let the laptop and pi communicate.
So far I am a little behind because I should’ve started working on phase vocoding, however, I’ve decided to table that until we have a completely working ML and pi communication system.
Next week I want to try and work on getting the ML installs working on the pi. If I can’t then I will be working on cleaning up the sockets code between the laptop and pi.