Posted on

Yuxuan’s Status Report for 3/30

Progress

This week I spent most of the time installing and integrating the necessary modules and components to RPi with my team. Besides that I customized the Google word2vec model for classifying items names to our five pre-set categories. The code calculates scores that evaluate a new item name’s similarity with some words (in dataset) in each category, and it assigns the category with highest score to the new item name. The script will be finalized with a larger dataset of words for all categories and be integrated into our web app.

Schedule

I am on schedule.

Next Steps

Next week I will integrate the word2vec classification into the NLP pipeline on the RPi, which includes downloading the Google word2vec model, setting up the initial word dataset in a migration file, and integrating the classification feature in view.py. I will also work with my team to implement the audio input features (linking parsed commands to different actions) for our web app.

Posted on

Team Status Report for 3/30

Risk Mitigation

This week our team integrated all components and scripts into Raspberry Pi to make sure that they are compatible. There was an issue when downloading SpaCy, which was solved by upgrading the Raspbian OS from 32 bits to 64 bits. We successfully installed SpaCy, PyAudio, and all related modules to the RPi, and we also set up Django and were able to run our web app on the RPi. We incorporated the microphone and made sure SpeechRecognition worked fine on the RPi.

Design Changes

There is no design change made.

Updated Schedule

The main functions to implement include text-to-speech conversion, audio input feature integrated into the web app, and report generation in web app. Below is an updated schedule for the rest of the project.

Posted on

Yixin’s Status Report for 03/30/2024

  • What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

To prepare for the demo next week, our group tested the script for spacy, speech recognition, and web app on Monday. It turns out that there are some problems with running spacy. Therefore, I reinstalled the OS (64 bit) and all packages needed to run our application (spacy, speech recognition, python 3.8, etc.) Since I was sick on Wednesday, I used one excused absence for the Wednesday Lab, but I also communicated my groupmates and knew there were some problems with the onscreen keyboards. We tried different keyboards but none of them worked. We would tried to solve this problem next week.

In addition to this, our spacy model is then able to identify many labels.

It could identify DATE (xx month xxxx year to xx month xxxx year) in our commands. I have tried all 12 months and all the years from 2020 to 2030. The accuracy is 100%.

In addition, it could identify price in different formats (both $n and n dollars would work).

  • Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

on schedule

  • What deliverables do you hope to complete in the next week?
  1. solve the onscreen keyboard thing
  2. prepare the datasets for labeling items
Posted on

Lynn’s Status Report for 03/30

Progress

I first tested the speech recognition pipeline on the RPI and it worked as desired. Then I spent most of my time this week working on the implementation of the web application. I wrote the CSS styling based on the UI design diagrams and constructed the financial report charts. Now the users could view financial report charts based on the time range they want to view. The current page layouts are attached below.

As for the VUI side, the user could now press the button to give voice commands, and the web app could redirect to certain pages based on the input keywords. For example, if the user gives the command “generate report”, the report page will be rendered and the charts will be displayed successfully. 

I also wrote primary scripts for generating audio outputs. The script could convert the input txt file to audio and play the file directly, and the audio file will be deleted after playing to save memory.

Schedule

I am on schedule now.

Next Steps

I am planning to design the audio output template txt files and generate corresponding content based on the user data within the web app. The users should be able to receive audio outputs when interacting with the application.

Posted on

Lynn’s Status Report for 03/23

Progress

This week our team worked on setting up and downloading the Python libraries on the RPi. I tested the speech recognition libraries and guaranteed that the libraries worked as expected. I also concatenated the speech recognition pipeline to the primary NLP model. After conducting some unit tests on the combined system, I would conclude that the verb commands could be successfully distinguished with a relatively high accuracy. 

After that, I worked with Yuxuan to make some adjustments to the web app design. We figured out how to filter the expected data efficiently when the user requests to view an entry or financial report. I started to work on the styling of the web application, and this will also be my primary task for the next week.

Schedule

I am on schedule.

Next Steps

I will focus on the css styling of the web app first, and then I need to construct the financial report page with a pie chart and a histogram. 

I would also implement the “press to speak” button and allow it to run the speech recognition pipeline.

 

Posted on

Team Status Report for 03/23/2024

Risk Mitigation

Setting up and downloading all required libraries on Raspberry Pi required more time than we expected. It took time for us to figure out how to deal with some issues when setting up the environment. In order to run pip on RPi, we modified some built-in documents and ended up downloading all the required libraries.

A potential risk would be running the trained spacy model on RPI. Currently, our group still works on training the model, so we failed to test how the model would work on RPi. However, other signal processing features should be able to run successfully. 

Design Changes

A major change in our design is the date information input for the View Entry page. We decided to align this input with the “Year-Month” selection of the “Financial Report” page. This would improve user experience by standardizing the input information. Also, according to the primary tests of speech recognition, higher performance is achieved when entering the year number and month while the date would be harder to translate correctly.

Updated Schedule

Since we are generally on time, we would keep the current schedule for the week. 

The RPI with all environment setting up is attached.

Posted on

Yuxuan’s Status Report for 3/23

Progress

This week I worked with my team to install python libraries and modules on Raspberry Pi, which took us a lot of time because they are supported by different Python versions.

I also spent much time on the web app. For now, I have added some dummy data in the migration files, made the “submit” button functional by saving the entry form into database, and displayed all entries within the selected time range in the record page. Below are snapshots of the code and the website.

Schedule

I am on schedule.

Next Steps

I will customize the Google News word2vec model for our app and integrate the classification script with the rest of the speech recognition and NLP pipeline.

Posted on

Yixin’s Status Report for 03/23/2024

  • What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

The main work for this week is the ethics assignment and preparing for the demo for next week. Therefore, our team has extra meeting time (Friday, in the lab, for about 4 hours) to set up the hardware together. During the meeting time, we ensured the wifi connection was working, had the on-screen keyboard working, and set up some basic tools (like pip). However, it took us longer than expected to download Spacy. Therefore, I take charge to setup the spacy and speech recognition part later.  I have finished downloading all the things needed for speech recognition and the Spacy.

In addition to the hardware setup, Lynn and I also connected the speech recognition and spacy parts together.

  • Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

on schedule

  • What deliverables do you hope to complete in the next week?
  1. run the training for spacy (items. have finished the action verbs)
Posted on

Team Status Report for 03/16/2024

Risk Mitigation

The WebApp part of our system is larger and more effort-consuming than we expected. Therefore, Lynn may help Yuxuan to make sure that part would be finished in time.

In addition, one risk for our project is that we have not tested any code on the Raspberry Pi. We will download the code to Raspberry Pi next Monday (our ethical lecture is on Wednesday). Since we still have a week to prepare for the first demo, we will have enough time to figure it out.

Design Changes

We did not make significant design changes this week. What we do are mainly specifying things we mentioned before. For example, in the past, we decided to use imperative sentences for commands. In this week, we specified what specific words (generate, enter, …) we would use.

Updated Schedule

The schedule will stay the same as we mentioned last week.

Things we need to finish by last week:

  1. Let things run on Raspberry Pi monitor (webapp would show up, and add entry should work)
  2. Finish the ethical assigment (attend lec + slack response)
Posted on

Yixin’s Status Report for 03/16/2024

  • What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

Work with Team:

  1.  Further finalize what commands we would use. The notes from this discussion helped me to rewrite the test case for the spaCy model.
  2. Work with the team on the ethical problem
  3. Discuss with Lynn about what the prices would look like when generating scripts (will the script give “five dollars” or “5 dollars”). This would help me to train the spaCy model.

Individual work:

The main task for me is still the spaCy model.

The function to identify the action verbs works well. As we can see from the example below. The actions.txt contains some commands as input, and after we run the get_action.py, it will give us the action verbs we needed.

As I have mentioned in previous status report, I would want to train the spaCy model for larger datasets. The base_config.cfg file would stay the same. And I have changed the preprocessor file. In the past, I would manually input commands, which will take a lot of time. With the new file, I can input some txt files and let the code read the file, making the training process easier.

I have also prepared the txt file for training (contains about 100 commands). But I have not started the new training yet.

  • Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

on schedule

  • What deliverables do you hope to complete in the next week?
  1. train the spaCy model & test accuracy
  2. run the code on raspberry pi monitor
  3. prepare for the demo