Posted on

Team Status Report for 04/06/2024

Risk Mitigation

The main task this week is the intriem demo. We have received many valuable advices from professor and TA. We have working word2vec on the computer, but when we installed it to Raspberry Pi, there would be some compatibility problem. Therefore, we would switch to GloVe. We have run simple scripts on the Raspberry Pi with GloVe and it works well. It could give us similar messages as word2vec. We now have all the parts needed to add entries (webapp, speech recognition, parsing, and classification model). All we need to do next is combine them together.

Design Changes

Change word2vec to GloVe due to Raspberry PI compatibility.

Updated Schedule

Our schedule does not change since last week.

Specific tasks we need to finish by next week:

  • Finish spacy
  • Further refine GloVe classfication
  • Integrate spacy with webapp

Week’s Special

  • User Experience Test

We would invite 5 volunteers to use our product without our interference. They would explore the app by themselves like what they would do when downloading a new app on their phone. We would give them half of the day to play with the app and would invite them to offer feedback after that. We would want them to list what they like about our product and what they dislike about our product.

  • Latency Test

We will test different commands and record the time it requires for our whole system to process. For example, we would measure the time it takes for our system to display the added entries after the user request it.

Posted on

Yixin’s Status Report for 04/06/2024

  • What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

I am working on the interaction between user and our app. I have prepared scripts that our app would read out when needed.

I have sent my code of spacy, which would identify DATE and PRICE to my team and they would connect that part with our web application. Additionally, I am working on the ITEM label this week. Since many things (apple, bed, laptop, etc.) would be assigned an ITEM label, it is not easy for spacy to learn which word would be recognized as ITEM. Therefore, our current trained spacy model does not perform ideally. I would add more training data to see what happen next week.

  • Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

on schedule

  • What deliverables do you hope to complete in the next week?
  1. work with spacy
  2. prepare for performance test.
  • This Week’s Special (Verification)
  •  Test parsing accuracy for ACTIONS (done)

I test this part with about 50 different text commands including different action verbs that maybe included in our system. The accuracy for this part is about 98% where “view” is considered as a noun instead of a verb. This will be fixed when I trained the model again later. The action verbs are pretty important because they will decided which webpage our app would show.

  • Test parsing accuracy for DATE (done)

Since we will always require a time range (from xx month xx year to xx month xx year) in this part. I generated random pairs of month (12 different months) and year (from 2020 to 2040). And our app could ensure 100% accuracy. The DATE information is very important when the user want to view entries or generate report.

  • Test parsing accuracy for PRICE (done)

I used the speech recognition to help generate text commands and find out that it would give price in two formats ($xx, or xx dollars). I tests both cases with randomly generated money value, and the spacy model could recognize it correctly with 100% accuracy. Price is very important when adding entries.

  • Test parsing accuracy for ITEM

Like before, I would use about 50 commands to test if our trained model could accurately parse the item out of the sentence.

 

 

Posted on

Yuxuan’s Status Report for 3/30

Progress

This week I spent most of the time installing and integrating the necessary modules and components to RPi with my team. Besides that I customized the Google word2vec model for classifying items names to our five pre-set categories. The code calculates scores that evaluate a new item name’s similarity with some words (in dataset) in each category, and it assigns the category with highest score to the new item name. The script will be finalized with a larger dataset of words for all categories and be integrated into our web app.

Schedule

I am on schedule.

Next Steps

Next week I will integrate the word2vec classification into the NLP pipeline on the RPi, which includes downloading the Google word2vec model, setting up the initial word dataset in a migration file, and integrating the classification feature in view.py. I will also work with my team to implement the audio input features (linking parsed commands to different actions) for our web app.

Posted on

Team Status Report for 3/30

Risk Mitigation

This week our team integrated all components and scripts into Raspberry Pi to make sure that they are compatible. There was an issue when downloading SpaCy, which was solved by upgrading the Raspbian OS from 32 bits to 64 bits. We successfully installed SpaCy, PyAudio, and all related modules to the RPi, and we also set up Django and were able to run our web app on the RPi. We incorporated the microphone and made sure SpeechRecognition worked fine on the RPi.

Design Changes

There is no design change made.

Updated Schedule

The main functions to implement include text-to-speech conversion, audio input feature integrated into the web app, and report generation in web app. Below is an updated schedule for the rest of the project.

Posted on

Yixin’s Status Report for 03/30/2024

  • What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

To prepare for the demo next week, our group tested the script for spacy, speech recognition, and web app on Monday. It turns out that there are some problems with running spacy. Therefore, I reinstalled the OS (64 bit) and all packages needed to run our application (spacy, speech recognition, python 3.8, etc.) Since I was sick on Wednesday, I used one excused absence for the Wednesday Lab, but I also communicated my groupmates and knew there were some problems with the onscreen keyboards. We tried different keyboards but none of them worked. We would tried to solve this problem next week.

In addition to this, our spacy model is then able to identify many labels.

It could identify DATE (xx month xxxx year to xx month xxxx year) in our commands. I have tried all 12 months and all the years from 2020 to 2030. The accuracy is 100%.

In addition, it could identify price in different formats (both $n and n dollars would work).

  • Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

on schedule

  • What deliverables do you hope to complete in the next week?
  1. solve the onscreen keyboard thing
  2. prepare the datasets for labeling items
Posted on

Lynn’s Status Report for 03/30

Progress

I first tested the speech recognition pipeline on the RPI and it worked as desired. Then I spent most of my time this week working on the implementation of the web application. I wrote the CSS styling based on the UI design diagrams and constructed the financial report charts. Now the users could view financial report charts based on the time range they want to view. The current page layouts are attached below.

As for the VUI side, the user could now press the button to give voice commands, and the web app could redirect to certain pages based on the input keywords. For example, if the user gives the command “generate report”, the report page will be rendered and the charts will be displayed successfully. 

I also wrote primary scripts for generating audio outputs. The script could convert the input txt file to audio and play the file directly, and the audio file will be deleted after playing to save memory.

Schedule

I am on schedule now.

Next Steps

I am planning to design the audio output template txt files and generate corresponding content based on the user data within the web app. The users should be able to receive audio outputs when interacting with the application.

Posted on

Lynn’s Status Report for 03/23

Progress

This week our team worked on setting up and downloading the Python libraries on the RPi. I tested the speech recognition libraries and guaranteed that the libraries worked as expected. I also concatenated the speech recognition pipeline to the primary NLP model. After conducting some unit tests on the combined system, I would conclude that the verb commands could be successfully distinguished with a relatively high accuracy. 

After that, I worked with Yuxuan to make some adjustments to the web app design. We figured out how to filter the expected data efficiently when the user requests to view an entry or financial report. I started to work on the styling of the web application, and this will also be my primary task for the next week.

Schedule

I am on schedule.

Next Steps

I will focus on the css styling of the web app first, and then I need to construct the financial report page with a pie chart and a histogram. 

I would also implement the “press to speak” button and allow it to run the speech recognition pipeline.

 

Posted on

Team Status Report for 03/23/2024

Risk Mitigation

Setting up and downloading all required libraries on Raspberry Pi required more time than we expected. It took time for us to figure out how to deal with some issues when setting up the environment. In order to run pip on RPi, we modified some built-in documents and ended up downloading all the required libraries.

A potential risk would be running the trained spacy model on RPI. Currently, our group still works on training the model, so we failed to test how the model would work on RPi. However, other signal processing features should be able to run successfully. 

Design Changes

A major change in our design is the date information input for the View Entry page. We decided to align this input with the “Year-Month” selection of the “Financial Report” page. This would improve user experience by standardizing the input information. Also, according to the primary tests of speech recognition, higher performance is achieved when entering the year number and month while the date would be harder to translate correctly.

Updated Schedule

Since we are generally on time, we would keep the current schedule for the week. 

The RPI with all environment setting up is attached.

Posted on

Yuxuan’s Status Report for 3/23

Progress

This week I worked with my team to install python libraries and modules on Raspberry Pi, which took us a lot of time because they are supported by different Python versions.

I also spent much time on the web app. For now, I have added some dummy data in the migration files, made the “submit” button functional by saving the entry form into database, and displayed all entries within the selected time range in the record page. Below are snapshots of the code and the website.

Schedule

I am on schedule.

Next Steps

I will customize the Google News word2vec model for our app and integrate the classification script with the rest of the speech recognition and NLP pipeline.

Posted on

Yixin’s Status Report for 03/23/2024

  • What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

The main work for this week is the ethics assignment and preparing for the demo for next week. Therefore, our team has extra meeting time (Friday, in the lab, for about 4 hours) to set up the hardware together. During the meeting time, we ensured the wifi connection was working, had the on-screen keyboard working, and set up some basic tools (like pip). However, it took us longer than expected to download Spacy. Therefore, I take charge to setup the spacy and speech recognition part later.  I have finished downloading all the things needed for speech recognition and the Spacy.

In addition to the hardware setup, Lynn and I also connected the speech recognition and spacy parts together.

  • Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

on schedule

  • What deliverables do you hope to complete in the next week?
  1. run the training for spacy (items. have finished the action verbs)