Carnegie Mellon ECE Capstone, Spring 2020: Ashika K, Abha A, Jade T
The events this week did not affect the software side too much. The whole system latency requirement has been scrapped, but that is mostly due to integration problems. For the software component, I actually managed to get the program running faster after switching to using …
The major change to our design is that we will now be constructing our robot with common household materials, such as cardboard and wood. This is because Techspark has shut down and we no longer have access to laser cutters, which we needed to construct …
I spent the first part of this week editing and formatting the design report. Once that was finished, I started creating a user program for the storytelling algorithm. The user interface is command line at the moment, and this is how the program works:
Here is an example output for the user program (with the Nightingale template):
The program processes templates sentence by sentence, which is important for FitBERT input and keeps the output of the program as (sentence, POS) to pass on to the text to speech module. To work with this user program, I finished writing the FitBERT fill in the blank program, but I still have to implement grammar correction on both the user and algorithm side. On the user side, I have started implementing (but not yet finished) error detection on the user input, and I have integrated the user input processing with synonym generation to pass on to FitBERT.
Next week is spring break, which is not a part of our project gantt chart.
For the week after, I am hoping to finish error detection on the user input. I also have to implement a translation system between the parts of speech of NLTK and the parts of speech of our program because NLTK uses more specific parts of speech (like helping verbs and participles). I would also like to begin integrating the algorithm with the speech recognition module. Jade and I will work on getting the audio input captured by a laptop to the algorithm, and then getting the output sentence to the text to speech tool and saying the sentence.
If I have time, I am planning to finish the grammar correction part of the program, which should not take long because FitBERT is already set up and running, but I may have to push this till the week after.
I am still on schedule because I finished the fill in the blank portion quite early and will use the slack time to finish integrating all the programs I’ve written, which is taking a bit longer than anticipated. Once integration is done, I will circle back and work on improving synonym generation. Also, my portion is ready to begin integration, just as expected.
We have no significant changes to our schedule and the risks of our project mostly have remained the same. This week, we worked on the design presentation and report, as well as our individual tasks. For the design submissions, we created a new system architecture …
This week, I focused on two aspects of the project. First, I finished up the storytelling portion of the design report. I already finished ironing out the details and conducting research on the metrics and validation prior to the design presentation, but I edited the …
We have no changes to the existing design, and are currently on schedule. We have been deep diving into figuring out how specifically we will be implementing our requirements as well as working on our design report and presentation. We finalized the communication protocols we will be using to communicate between the peripherals, Raspberrry Pis, and laptop. This led to finalizing the system architecture diagram as well (image above). In addition, this week, Abha worked on the robot arm design and overall robot design and began CADing, Jade worked on determining speech processing/TTS packages and setting up the Raspberry Pis, and Ashika worked on story template generation and parts of speech detection.
Currently the most significant risks remain the same as what they were last week. The only change is with the speech recognition and TTS system. We chose to use packages for speech recognition and TTS that rely on an internet connection. This poses a risk as our robot should be able to work even when not connected to the internet. To mitigate these risks we will be relying on internet based speech recognition and TTS and should the Raspberry Pi disconnect we will have backup speech recognition and TTS packages that can operate without an internet connection. The reason we aren’t using packages that operate without an internet connection in the first place is that they often have worse speech recognition and a very robotic/choppy synthesized voice.
This week, I started writing the code for story creation. I wrote two python programs: the first checks that the part of speech of a word matches the expected part of speech given a sentence for context. This will be used for error detection on …
This week, I accomplished three main tasks. First, I found a story dataset for the templates: a collection of 177 of Aesop’s Fables. All these stories are about 5-7 sentences long, so they fit our desired template length and will be the only story dataset …
The dimensions of KATbot are as shown above.
Risks and Risk Management:
The most significant risk is a poor fill in the blank algorithm for the machine learning based stories. Without this feature, the stories will not be cohesive or grammatically correct, defeating the purpose of our storytelling, educational tool. To ensure this feature works, we are planning to use sentences with only one or two inputs and vocabulary at predetermined educational levels. We know, based on documentation, that the tool we are using for this algorithm works with this format. If we are still unable to get the fill in the blanks working, we can always minimize the extent to which we customize the story, or change the story templates to require even less cohesion.
Another big risk is speech recognition. Since our target audience includes children that cannot read, this feature is vital. We need to be able to identify simple phrases in the context of user interaction. Our contingency plan is to have speech recognition running on the same laptop as the ML algorithm. We also have a touch screen display, so we can ask for user input to be typed rather than spoken.
Design Changes:
The text display that best suits our project design has the added feature of a touch screen. This is beneficial since we can now use the touch screen as a backup plan for speech recognition and we can also add buttons on the screen for more control (ex. reset or help buttons).
In addition, our final goal is to have the machine learning algorithm on a Nvidia jetson nano, as opposed to just on the laptop, because we want the entire product to be in one unit. To accomodate this change, we are leaving room in out budget to purchase one.
The schedule for the project remains the same.