Category: Yixin’s Status Reports

• What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

I gave our products to two users this week and collected their feedback. One of them said that some commands would not have an audio response. I also updated this feedback with Yuxuan and Yuxuan has added the audio response for those commands. In addition, I am working on the final poster right now. I have prepared the new information (e.g. what we have achieved) for the poster and will do the formatting next week.

• Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

on schedule

• What deliverables do you hope to complete in the next week?

1. prepare for final deliverables

• What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

I have trained spacy that could identify ITEM. I have about 100 commands as training datasets for the final version. In addition, I also trained the model again to add two labels: CHANGE and CATEGORY. CHANGE will identify which part of the entry (item name, price, or classification) we want to modify. CATEGORY would identify what the new classification we would want to assign to the items (e.g. if the command is “change category to entertainment”, the entertainment would have a label CATEGORY). In addition, there is also another label named CARDINAL. This label is used when we are using commands like “delete entry number 5” or “modify entry number 1”. “5” and “1” will be parsed from the commands and they will have the label CARDINAL.

In addition, I have done testing for all these new labels (ITEM, CHANGE, CATEGORY, and CARDINAL). The accuracy for CHANGE is not ideal, so we would not use this label in our model. The other three labels have pretty good accuracy (>95%) and meet our expectations.

In addition, since I need to present next week for the final presentation. I also spend some time preparing for the presentations.

• Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

on schedule

• What deliverables do you hope to complete in the next week?

1. prepare for the presentation
2. integral testing

• This Week’s Special

One of the thing I have learned is how to downloading all the things we needed on a hardware. In past courses, there would always be a detailed and clear guide provided by the professors to guide us through how to setup softwares like Python and Django. However, to download all packages we needed in Raspberry Pi does take much longer than we expected. We need to figure out why Spacy would not work on our Pi. We searched online for a while and finally found a post that have the same problem as we did. This post reminded us that Spacy required a 64-bit OS while we are not sure if our Pi have it. Therefore, we finally reinstalled a 64-bit OS and Spacy work. After having this experience, I would checked the requirements and compatability carefully next time before downloading anything.

In addition, I also encountered some problems when training Spacy, the package I used for the first time. To acquire this new knowledge, I read through the Linguistic Features  and Training Models part of Spacy’s guide. These help me to choose what features our app would use and what model I would train. In addition, I also searched online and found step by step guidance that taught me how to prepare the dataset. When I encountered some errors, I would search for posts that had the same problems and tried to fix these errors with their methods.

 

• What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

I am working on the interaction between user and our app. I have prepared scripts that our app would read out when needed.

I have sent my code of spacy, which would identify DATE and PRICE to my team and they would connect that part with our web application. Additionally, I am working on the ITEM label this week. Since many things (apple, bed, laptop, etc.) would be assigned an ITEM label, it is not easy for spacy to learn which word would be recognized as ITEM. Therefore, our current trained spacy model does not perform ideally. I would add more training data to see what happen next week.

• Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

on schedule

• What deliverables do you hope to complete in the next week?

1. work with spacy
2. prepare for performance test.

• This Week’s Special (Verification)
•  Test parsing accuracy for ACTIONS (done)

I test this part with about 50 different text commands including different action verbs that maybe included in our system. The accuracy for this part is about 98% where “view” is considered as a noun instead of a verb. This will be fixed when I trained the model again later. The action verbs are pretty important because they will decided which webpage our app would show.

• Test parsing accuracy for DATE (done)

Since we will always require a time range (from xx month xx year to xx month xx year) in this part. I generated random pairs of month (12 different months) and year (from 2020 to 2040). And our app could ensure 100% accuracy. The DATE information is very important when the user want to view entries or generate report.

• Test parsing accuracy for PRICE (done)

I used the speech recognition to help generate text commands and find out that it would give price in two formats ($xx, or xx dollars). I tests both cases with randomly generated money value, and the spacy model could recognize it correctly with 100% accuracy. Price is very important when adding entries.

• Test parsing accuracy for ITEM

Like before, I would use about 50 commands to test if our trained model could accurately parse the item out of the sentence.

 

 

• What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

To prepare for the demo next week, our group tested the script for spacy, speech recognition, and web app on Monday. It turns out that there are some problems with running spacy. Therefore, I reinstalled the OS (64 bit) and all packages needed to run our application (spacy, speech recognition, python 3.8, etc.) Since I was sick on Wednesday, I used one excused absence for the Wednesday Lab, but I also communicated my groupmates and knew there were some problems with the onscreen keyboards. We tried different keyboards but none of them worked. We would tried to solve this problem next week.

In addition to this, our spacy model is then able to identify many labels.

It could identify DATE (xx month xxxx year to xx month xxxx year) in our commands. I have tried all 12 months and all the years from 2020 to 2030. The accuracy is 100%.

In addition, it could identify price in different formats (both $n and n dollars would work).

• Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

on schedule

• What deliverables do you hope to complete in the next week?

1. solve the onscreen keyboard thing
2. prepare the datasets for labeling items

• What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

The main work for this week is the ethics assignment and preparing for the demo for next week. Therefore, our team has extra meeting time (Friday, in the lab, for about 4 hours) to set up the hardware together. During the meeting time, we ensured the wifi connection was working, had the on-screen keyboard working, and set up some basic tools (like pip). However, it took us longer than expected to download Spacy. Therefore, I take charge to setup the spacy and speech recognition part later.  I have finished downloading all the things needed for speech recognition and the Spacy.

In addition to the hardware setup, Lynn and I also connected the speech recognition and spacy parts together.

• Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

on schedule

• What deliverables do you hope to complete in the next week?

1. run the training for spacy (items. have finished the action verbs)

• What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

Work with Team:

1.  Further finalize what commands we would use. The notes from this discussion helped me to rewrite the test case for the spaCy model.
2. Work with the team on the ethical problem
3. Discuss with Lynn about what the prices would look like when generating scripts (will the script give “five dollars” or “5 dollars”). This would help me to train the spaCy model.

Individual work:

The main task for me is still the spaCy model.

The function to identify the action verbs works well. As we can see from the example below. The actions.txt contains some commands as input, and after we run the get_action.py, it will give us the action verbs we needed.

As I have mentioned in previous status report, I would want to train the spaCy model for larger datasets. The base_config.cfg file would stay the same. And I have changed the preprocessor file. In the past, I would manually input commands, which will take a lot of time. With the new file, I can input some txt files and let the code read the file, making the training process easier.

I have also prepared the txt file for training (contains about 100 commands). But I have not started the new training yet.

• Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

on schedule

• What deliverables do you hope to complete in the next week?

1. train the spaCy model & test accuracy
2. run the code on raspberry pi monitor
3. prepare for the demo

• What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

I test the PyAudio speech recognition part with Lynn. It shows that this part works well. It can clearly identify many instructions that we may use (e.g. generate the report for last week, buy a book with five dollars, etc.). We also receive the hardware. My groupmate has tested if the microphone works. We would test the speech recognition

I have also prepared a small dataset for the classification model. I grabbed item names from the shopping website and assigned them different tags (the 6 different tags). Waiting for the word2vec model to be set up and test the model.

Last week, I learned the attributes spaCy has. I learned how to train the spaCy model this week, and may begin to train the spaCy model next week. I may first train the spaCy model to identify the money value in the sentence.

• Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

on schedule

• What deliverables do you hope to complete in the next week?

1. setup the hardware with the group (especially Raspberry Pi)
2. revise the design report based on comments received
3. train the spaCy model

• What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

One of the main tasks for this week is the design presentation. I will not be the presenter this time, but I have also worked on preparation for this presentation. As for the proposal presentation, I also revised the use case and use case requirements for this presentation.  I simplified the use case to leave more time for later parts (which are more important to the design review) and I revised the use case requirements based on our new plan (a Raspberry Pi monitor).

In addition, I spent time studying PyAudio, and I think I now know how to translate Audio to text but need to test it when our device arrives. I also learned how to use spaCy, downloaded it on my computer, and ran some simple tests to understand specific attributes of this library.

• Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

on schedule

• What deliverables do you hope to complete in the next week?

1. finish design review report
2. choose the database use to train our model

• What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This week is about the proposal presentation. I write the script and construct the slides for user case and user case requirements. Create the GitHub repo for the web app. Meet with the group, decide which equipment to order, and submit the forms. In addition, we also decide which ML models we are going to use.

• Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

on schedule

• What deliverables do you hope to complete in the next week?

1. finish homepage + css of the Django project
2. help to run simulations of our current ML model with teammates and decide if the current model is optimal or if more research is needed
3. May set up the hardware (depending on whether we receive the ordered equipment)
4. prepare for the design presentation