Author: jianjunx

Personal Accomplishments

This week I have completed our final presentation and started working on the final poster and report. I have also done more unit tests on the NLP system and the results are attached below. All the simple commands only involving addition or deletions resulted in 100% accuracy; the only test cases which went wrong are the ones where addition and deletion commands appear in the same sentence. Fortunately, based on our requirement for the users to order item-by-item, this complicated sentence should not occur in the use case of our system. The reason why the inaccurate test cases are hard to fix is that different menu items can generate different dependency trees even in the exact same sentence structure. Accounting for one edge case often means giving up on another.

Trial	Test string	Addition Accuracy	Removal Accuracy	Inaccurate cases
1	can I have a [menu item (sg)] please	100%	N/A
2	two [menu item (pl)]	100%	N/A
3	two [menu item (sg)]	100%	N/A
4	five beautifully packaged [menu item (pl)]	100%	N/A
5	a box of [menu item (sg)] please	100%	N/A
6	twenty [menu item1 (pl)] and three [menu item3 (pl)]	100%	N/A
7	remove [menu item (pl)] please	N/A	100%
8	i will get rid of [menu item (pl)]	N/A	100%
9	one [menu item (sg)] oh wait can i get rid of a terrible [menu item (sg)] please	80%	100%	Hot dog, fries, onion rings
10	i want three [menu item (pl)] can i delete two [menu item (pl)]	80%	100%	Cheeseburger, chicken burger, chicken sandwich

Schedule

I am on track with our schedule.

Plans for Next Week

I will work with my teammates for the final report, poster and video submission. We will find volunteers to test our system as well.

Personal Accomplishments

I have completed all the staff UI features by adding buttons which use Nina’s order item removal function, allowing the staff to delete order items once they are done. The details of the staff UI will be shown in the final presentation as a quick demo.

The unit tests for the NLP system have been completed. I made test files with basic sentence structures, such as “can I have a [menu item] please”, “2 [menu items]” and “five beautifully packaged [menu items]”. And replaced the [menu item] with actual names of the items. According to our unit tests, I have made modifications to our NLP program to handle the parsing of “2 ice creams”. Usually the quantifier (2 in this case) will be directly linked to the last word of the menu item, but for “ice cream” it is modifying “ice”, which makes my existing NLP rules not applicable. The NLP system passed all basic sentence structure tests with 100% accuracy.

Schedule

I am on track with our schedule; the staff UI is finished according to plan.

Plans for Next Week

I will work with my teammates for the final presentation and final video submission. We will find volunteers to test our system as well.

Personal Accomplishments

There are two changes I added to the NLP algorithm this week. The first one is when deleting all of a certain item, for example, when the customer says “no hamburgers”, instead of using the hard-coded 1000 (a quantity large enough than reasonable numbers), I now use Nina’s getItemQuantity function to retrieve the menu item’s actual quantity and delete that exact number from the order. The second one is automatically changing the word “hamburg” to “hamburger” when parsing, since the speech recognition system often mishears “hamburger” as “hamburg”, and the word “hamburg” does not commonly appear in fast food orders.

I also have started designing the staff UI. My plan is to make each order appear in a small box, and make the entire page resemble a Kanban board.

Testing Algorithms

I have run tests with basic sentence structures for all the menu items, giving the NLP system user input as text and asking it to print out the final order. Two examples of basic sentences will be “I want [quantity] [menu item]” and “Can I get [quantity] [menu item]?”. Some tests with sentences including the removal of menu items also have been conducted, such as “Can I get rid of a [menu item]?” or “No [menu item]”. My initial plan was to use a timer to test the speed of NLP processing, but I soon realized that the processing speed is far below 1 second so the speed test was no longer necessary. There are no design or use case requirements specifically set for my NLP system or staff UI; the order accuracy metric involving the customer UI, speech recognition, NLP and database will be tested in a later stage with us and volunteers imitating the actual ordering situation.

Schedule

I am on track with our old (and updated) schedule (see team status report section for details). I have started working on the staff UI, and I will keep optimizing the NLP system when we test during integration.

Plans for Next Week

I will work on the staff UI design next week and aim to get it done before when the next status report is due. If more edge cases of ordering commands arise during the testing process, I will also update the NLP system to accommodate them.

Personal Accomplishments

I have changed one of our menu items “veggie burger” to “salad” then to “cheesecake” due to NLP parsing difficulties. Usually the structure of menu items as compounded words will be the one shown in the following screenshot, where the quantity modifies the anchor noun.

[INSERT IMG]

However, for “veggie burger”, spaCy recognizes it as the following structure, where the quantity is a child of the adjective “veggie”. There are possible solutions to handle the alien structure, but we have decided that it would be easier to change the menu item instead.

[INSERT IMG]

The updated item “salad” also faced the same issue. Addition of salads into the order worked perfectly, yet when we were trying to remove them, the phrase structure also becomes an outlier. Usually the quantity will directly modify the anchor noun, but here it becomes a child of the verb. To make it worse, when I tried to add a special dependency rule just for the removal of salads, I realized that no existing spaCy matcher operators can describe the relationship between the quantity and “salad” in this case. Although I expected them to be siblings according to the tree, both “$++” and “$- -” for sibling relationships fail to detect the phrase. That is when we decided to change the item again into “cheesecake”.

[INSERT IMG]

Additionally, I enabled the handling of order confirmation keywords, and debugged some edge cases that occurred during testing. An interesting edge case was that “a beautifully packaged cheeseburger” could be detected while “a beautifully package cheeseburger” (a mishearing) failed to generate an order. I solved the issue by changing some operators and tags for dependency matching.

Schedule

I am on track with our updated schedule (see in the team status report section). I pushed the design of the staff UI further back since the integration of other subsystems has not been completed yet. Noise reduction is also temporarily removed from my schedule, as the current method is already able to handle classroom-level noise. We may choose to do more with noise reduction after the interim demo.

Plans for Next Week

I will work with my teammates to complete the interim demo and debug more NLP edge cases, such as finding a way to still parse the order entry when the speech recognition system recognizes our quantity “two” as “to”. It might require adding new rules to our dependency matcher to recognize the specific sentence structures where the “to” should be changed to “two”. I will also start designing the staff UI mid-week.

Personal Accomplishments

After some unit testing of the integrated speech recognition, NLP and database subsystems, I implemented more features to my NLP system based on the results.

The first new feature is the checkout process. When a keyword (“check out”) is detected, the checkout function written by Nina will be called so that the order is uploaded to the database and sent to the kitchen side. This is achieved by adding a phrase matcher to the NLP system.

The second new feature is the detection of menu items with quantifiers. Previously, the NLP system was unable to recognize user input with quantifiers, such as “a cup of coffee”. I added another set of patterns to the dependency parsers so that the phrase structure with quantifiers can be processed as normal input.

Another feature I implemented is the removal of all mentioned items if the user does not specify the quantity in an item-deleting situation. When the user input is “no hamburgers”, all hamburgers in the existing order will be removed.

Schedule

I am on track with our updated schedule (described in detail in the team status report).

Plans for Next Week

I will keep revising the NLP system based on testing results, and work on improving the noise reduction abilities of our microphone. Some new NLP features I plan to add are the handling of order confirmation keywords, and the situation when the quantity is in a different part of the sentence than the menu item. (eg. “I want some coffee, um, 2 please.”)

Personal Accomplishments

I collaborated with Nina to finish our NLP and database integration this week. The output of my NLP algorithm can now feed directly into the database to be an Order object. I started writing the speech recognition code with Nina and the system is able to convert user input into text in near real-time. The speech recognition system is still weak on noise reduction (as described in Nina’s status report) and speech conversion speed, which we plan to improve soon. I also fixed some bugs in my NLP algorithm while making it account for more use cases, such as when a user orders and then deletes an item in the same sentence. Another change I made in the NLP is that when we cannot detect the quantity of a menu item in the sentence, we will default it to 1.

Schedule

I am on track with our schedule this week, for the NLP and database integration is almost finished and the speech recognition work has started.

Plans for Next Week

I will continue to work with Nina to improve our NLP and database integration, and tune the speech recognition to our microphone so that it can convert user speech faster and more accurately. If I have enough time left, the integration between speech recognition and NLP systems will be the next task I am going to work on.

Personal Accomplishments

The problem in my NLP system I tried to solve since the last status report is the parsing of menu items with multiple words, as I realized that the dependency parser does not support parsing multiple words as a single token identity. The first solution I came up with was to add all menu items into a list of named entities, so that the NER pipeline can recognize them. However, the menu items could not be considered as named entities in spaCy even when I tried to capitalize the initial letters. Token matching could work, but a lot of flexibility will be compromised when handling varied sentence structures, since the token matcher needs more rigid rules than the dependency matcher. I solved the problem by defining a new set of dependency matching rules for menu items with multiple words. For example, for the menu items “veggie burger” and “chicken burger”, this pattern will use “burger” as the anchor token and find its immediate adjective dependent (either “veggie” or “chicken”) and the quantifier dependent (number or determiner).

The following command line output shows the natural language processing system’s input and output.

For changing item entries, the current solution is to find a set of keywords indicating the change (“remove”, “delete”, “add”, “change”), which should be the immediate head of the menu item token, and change the order information accordingly. The actual ordering situations will be more complex than this scenario, and we plan to fix it after having a functioning system that reaches MVP.

Schedule

I have caught up with the schedule for NLP system programming: a MVP version of the algorithm can be completed by Monday March 13 the latest. Nina and I have not started integrating the database and the NLP system yet; the schedule change is mentioned in the team status report for this week. We did coordinate on what type of data structure the NLP system should return so that the information can feed directly into the database, and I believe the integration should not take long since we already have compatible data structures.

Plans for Next Week

I plan to work with Nina to integrate my NLP system with the database next week. I will also start working on programming the speech recognition system.

Personal Accomplishments

I have started programming the natural language processing system for the project this week using Python’s spaCy library. I used the built in en_core_web_sm pipeline for the basic file parsing with tokenizer, tagger, parser and NER. There are two ideas for grammar rule matching that I experimented with. The Dependency Matcher is able to get the menu item and the quantity even with multiple words in between, such as “a splendidly delicious hamburger”, but it is complicated to set rules for vocabulary without directly dependent relationships. If the matcher fails to detect that the item quantity word is related to the menu item word, there is no way they can be identified using the Dependency Matcher. The token modifier works better in this situation, since the rules can be set based on part-of-speech tags or other token properties instead. However, the token modifier is not very proficient in identifying the relationship between different parts of the sentence and might require more edge-case accommodations. Based on the findings, I will attempt to utilize both matchers to create a more comprehensive algorithm.

Schedule

I’m slightly behind our original schedule, since I haven’t made significant progress on the natural language processing algorithm due to the large amount of time spent in familiarizing myself with spaCy and the matchers. Our Gantt chart plan has changed to accommodate this. Since the database also has not been established yet, the integration between database and natural language processing system can be pushed back until after spring break to allocate more time for the development of both tools. I will make sure to work on the algorithm more in the next week and get an MVP version by the end of spring break at the latest.

Plans for Next Week

I plan to refine my natural language processing algorithm while working on the design report with my teammates. By the end of next week, my algorithm should be able to parse the user input in the following situations:

1. When there are words between the quantity and item name (eg. “a beautifully packaged cheeseburger”)
2. When the user makes an attempt to change the order using some easily detectable keywords (eg. “remove the diet coke” / “I wanna add another cheeseburger”)

Personal Accomplishments

This week, I completed research for the microphone and sound shield we need to purchase. We will proceed with Neat Bumblebee II and Moukey Microphone Isolation Shield. Both received a lot of positive customer feedback on Amazon and can arrive within 5 business days once the order is placed. Neat Bumblebee II is one of the only USB-compatible desktop directional microphones that can rotate vertically, allowing us to accommodate customers of different heights and with accessibility needs. According to past customers, it is also proficient in filtering out background noises. The Moukey Microphone Isolation Shield has 5 foldable panels, which can turn it into a semi-parabolic sound shield if necessary. I also added backup options for both microphones (Blue Yeti) and sound shields (IDoon or Aokeo) in case these two cannot satisfy our requirements after testing.

To prepare for the upcoming design review, I collaborated with my teammates to design the block diagram. I was also in charge of writing user requirements and solution approaches for the speech recognition system, and modifying our Gantt Chart schedule. Moreover, I have started doing research for implementing the natural language processing system next week.

Relevant Courses

Natural Language Processing (11-411) helped me to come up with the natural language processing pipeline for our project. Web Applications Development (17-437) which I’m taking this semester allowed me to brainstorm ideas for the kitchen-side user interface.

Schedule

According to our plan, preliminary research is to be completed by 2/19/2023, and I have done the required research for hardware components and libraries. One modification we made on the plan was that we will wait for the feedback from our design presentation to purchase the parts instead of doing it this week. Overall, I am on track with our schedule.

Plans for Next Week

I will start programming the natural language processing system that converts text to menu items next week. By next Saturday, my program should be able to identify order items in text input with around 70% accuracy.

Personal Accomplishment

I mainly focused on doing research on the sound shield and speech processing system this week to prepare for our upcoming design presentation.

Since the height of our microphone will accommodate customers both standing and in wheelchairs, our system will require a wide sound shield with an adjustable mic stand. I have found some suitable options on Amazon and plan to discuss with my teammates before purchasing one.

We plan to use Python as the programming language for our speech recognition system, since it is both compatible with natural language processing libraries — such as spaCy and Stanza — and has an open-source speech recognition library. For speech recognition, I found the python SpeechRecognition library, which works with external microphones. For natural language processing, although the tokenization accuracy of Spacy and Stanza are similar after some experiment, we decided to proceed with spaCy since it is faster and has more documentation for us to learn the functions.

About Schedule

Since we have scheduled two weeks for preliminary research (i.e. to be completed by 2/19/2023, the due date for our design presentation), I am on track with our schedule.

Plans for Next Week

I will purchase the sound shield, mic stand and microphone by the end of next week to make sure they arrive in time for our hardware assembly. As the deadline for design review is approaching, I will also work with my teammates to finalize our system design, and make slides for the presentation.