Category: Yuxuan’s Status Reports

Progress

I finalized out web application by adding the audio output for “modify”, “delete”, and “help” actions, so that all voice requests will be responded with voice output. The “help” command will trigger a voice instruction about how to use the app, giving the user some sample commands such as “enter apple for 4.99 dollars” to play around with the app.

I also conducted the user experience test with a volunteer. After the participant explored the system, I recorded the overall accuracy and latency of their command requests and responses.

 

Schedule

I am on schedule.

 

Next Steps

Next week our team will work on the poster, video, and final report, and prepare for the demo.

Progress

This week I implemented the whole VUI part of our web app with Lynn. Specifically, we parsed the voice command processed by SpaCy models and handled edge cases that SpaCy failed to catch. Based on the command and the current page, we rendered a new page with necessary parameters and generated audio output as assistance to the visually impaired people. The supported voice commands on each starting page are listed as follows.

Home page: enter an entry, view entry list (in a time range), generate report (in a time range)

Entry list page: enter an entry, modify an entry, delete an entry, view entry list, generate report

Modify page: confirm modification, enter an entry, view entry list, generate report

Report page: enter an entry, view entry list, generate report

I also ran the test for item classification accuracy. I prepared 20 item names and labeled their correct categories, and then fed those item names to my classification model based on word2vec method for the actual results.

 

Schedule

I am on schedule.

 

Next steps

We will continue with our user experience test and prepared for the final demo next week.

 

This week’s special

I learned word2vec embeddings and was responsible for customizing a word2vec model for our item classification function. I learned how to use the pretrained model by looking at examples of using the word2vec model and selecting the methods useful in my customization.

Through this capstone project I was able to learn by doing. Sometimes I won’t figure out the best way to implement a feature until I start coding it up. For example, we searched so long for a method to get data from the previous web page but could not find an effective way online. Later when we tried to keep error message in session, we realized we could use sessions to store those data and retrieve them later. There were many similar design choices we had to make during actual implementation of the project, and I learned to consider tradeoffs and actual requirements when choosing the methods.

Progress

This week I continued downloading necessary modules on RPi. I tried to run a small script to load the Google news word2vec model on RPi but the Pi always stuck probably due to the gigantic size of the model. I decided to switch to another word embedding method GloVe (Global Vectors for Word Representation), which has various models with different sizes and dimensions to choose from. I downloaded the GloVe model with 400,000 words and 100 dimensions, and I successfully loaded the model on the RPi.

I also implemented the modify button for each entry in the entry list page. This includes displaying the id for each entry and creating a new html page and a new function in views.py to handle modification action.

Schedule

I am on schedule.

Next Steps

Next week I will continue to implement the delete function for the entries and help with the audio input functions of our web app. I will also rewrite the word2vec script using the new GloVe model and incorporate the item classification into the NLP process.

Verification method

Test for manual input feature: To test that the basic manual input functionalities are implemented as expected, we will run our web app following the flow chart in our design report, covering all branches and corner cases of the flow chart.

Test for item classification accuracy: We will randomly select 20 item names outside our dataset, label them with their correct categories, and feed them into my item classification script built upon the GloVe model. We will then determine the accuracy of its prediction, which is expected to be 90% or above.

Test for latency: if the whole speech processing pipeline takes more than 3 seconds on average, I should consider reducing the number of dimensions of the GloVe model for less loading time.

 

Progress

This week I spent most of the time installing and integrating the necessary modules and components to RPi with my team. Besides that I customized the Google word2vec model for classifying items names to our five pre-set categories. The code calculates scores that evaluate a new item name’s similarity with some words (in dataset) in each category, and it assigns the category with highest score to the new item name. The script will be finalized with a larger dataset of words for all categories and be integrated into our web app.

Schedule

I am on schedule.

Next Steps

Next week I will integrate the word2vec classification into the NLP pipeline on the RPi, which includes downloading the Google word2vec model, setting up the initial word dataset in a migration file, and integrating the classification feature in view.py. I will also work with my team to implement the audio input features (linking parsed commands to different actions) for our web app.

Progress

This week I worked with my team to install python libraries and modules on Raspberry Pi, which took us a lot of time because they are supported by different Python versions.

I also spent much time on the web app. For now, I have added some dummy data in the migration files, made the “submit” button functional by saving the entry form into database, and displayed all entries within the selected time range in the record page. Below are snapshots of the code and the website.

Schedule

I am on schedule.

Next Steps

I will customize the Google News word2vec model for our app and integrate the classification script with the rest of the speech recognition and NLP pipeline.

Progress

My main focus of this week is the web application. I wrote html files for the 3 main pages: new entry, spending list, and financial report. I also implemented a navigation bar to jump between pages. I initialized the 6 categories in migration file so that user can select from the preset categories from a dropdown when creating an entry. Some buttons are dummy for now and the css style also requires further perfection.

Schedule

I am on schedule.

Next Steps

Next week I will continue work on the web app, improving the styles of the components and complete the functionalities. I aim to make the app functional with manual input by the interim demo.

Progress

This week I reviewed and discussed with my team all the feedback on our design presentation and made note of what we need to address in our design report. I also came up with an initial outline for our design report and divided up the work with my teammates.

I also started working on our web app development by defining our models and forms. I also built the basic pages in html. The next thing is to put the pages together in views.py.

 

Schedule

I am a little bit behind schedule due to midterms, but I will catch up next week.

 

Next steps

Next week I will focus on writing the design report with my team. I will be responsible for design requirements, part of design trade studies, test, and project management sections. I will also continue with linking our web app pages.

Progress

This week I discussed detailed control flow of our app with Tianyi and came up with the model-view-control (MVC) framework for our Django app.

Models:

Expense:

• Time: DateField
• Amount: FloatField (two decimal places)
• Description: CharField

Category:

• Name: CharField
• Color: CharField
• Expense: ManyToManyField

Main pages (html files):

• html (with tabs at bottom and audio input button to be extended by other pages)
• Home (contains a form to enter a new spending entry)
• Spendings (list of entries)
• Financial report

Some main controls in view.py:

• Home_action (render home page)
• Create_entry
• Generate_report
• Modify_entry
• Add_category
• Read_report

To prepare for the design presentation next week, I specified the testing process with input, expected output, and  expected result. I also updated the schedule based on changes in implementation design and created a product picture showing the hardware components we use.

Schedule

I am on schedule.

Next Steps

Next week I will start building our web app by setting up the three main pages and work towards a complete money tracking app without audio input first. I will also work on the design report and set up the Raspberry Pi with the team if we can receive the necessary components next week.

 

Progress

The week I did proposal presentation for my team on Monday. In preparation of that, I was responsible for testing, division of labor, and schedule parts of the presentation, and I discussed all other parts with my teammates.

Later this week, I researched and made a list of all hardware components we need to order or borrow from the ECE inventory. I chose Raspberry Pi 4 Model B 4GB RAM as our single board computer along with all the accessories compatible with RPi 4. After discussion with team, we decided to order a touch screen with speaker for RPi display and a USB microphone for audio input. I filled out the request forms for all components.

I also looked into the model for classifying product name to categories: our original approach was using random forest, but with further investigation I found that random forest might not be best suitable for the task; instead, we now want to use the word2vec model to group similar or related words together, so that if initially given the classifications of some sample words, the model is able to map new words into correct categories based on the distance between new words and the sample words in the vector space.

 

Schedule

I am on schedule.

 

Next Steps

Next week I will focus on testing out the viability of our selected NLP models on the laptop with my team. We will also design the MVC framework for our webapp as well.