Month: March 2023

Lisa Xiong’s Status Report For 3/11/2023

Personal Accomplishments

The problem in my NLP system I tried to solve since the last status report is the parsing of menu items with multiple words, as I realized that the dependency parser does not support parsing multiple words as a single token identity. The first solution I came up with was to add all menu items into a list of named entities, so that the NER pipeline can recognize them. However, the menu items could not be considered as named entities in spaCy even when I tried to capitalize the initial letters. Token matching could work, but a lot of flexibility will be compromised when handling varied sentence structures, since the token matcher needs more rigid rules than the dependency matcher. I solved the problem by defining a new set of dependency matching rules for menu items with multiple words. For example, for the menu items “veggie burger” and “chicken burger”, this pattern will use “burger” as the anchor token and find its immediate adjective dependent (either “veggie” or “chicken”) and the quantifier dependent (number or determiner).

The following command line output shows the natural language processing system’s input and output.

For changing item entries, the current solution is to find a set of keywords indicating the change (“remove”, “delete”, “add”, “change”), which should be the immediate head of the menu item token, and change the order information accordingly. The actual ordering situations will be more complex than this scenario, and we plan to fix it after having a functioning system that reaches MVP.

Schedule

I have caught up with the schedule for NLP system programming: a MVP version of the algorithm can be completed by Monday March 13 the latest. Nina and I have not started integrating the database and the NLP system yet; the schedule change is mentioned in the team status report for this week. We did coordinate on what type of data structure the NLP system should return so that the information can feed directly into the database, and I believe the integration should not take long since we already have compatible data structures.

Plans for Next Week

I plan to work with Nina to integrate my NLP system with the database next week. I will also start working on programming the speech recognition system.

Team Status Report For 3/11/2023

New Tools

To properly access and control our cloud database, Redis, we need to use RedisLabs, an online platform for viewing and manipulating database settings, and RedisInsight, a desktop application that visualizes and allows manually changing database data. These tools will also allow us to view statistics about the database, such as latency and number of accesses, which may help with testing speed of service in the future.

In addition, we are planning to use some jQuery libraries to write JavaScript faster and easier. jQuery also works with multiple browsers so our code is compatible regardless of which features does the browser contain.

Risks

As the natural language processing algorithm is developed further, we realized that it is heavily relying on the grammar structure of input sentences to capture the necessary information. The most significant risk is that if our speech recognition system fails to generate grammatically coherent sentences, it will be difficult for the speech recognition and natural language processing subsystems to integrate. To mitigate the risk, we are ready to use the token matcher on top of the dependency matcher to capture key words in the sentences instead of grammar structures.

The risk with regards to the UIs is that some Bootstrap templates we are currently using are unstable. Depending on how well they are maintained remotely, some always work when the pages are loaded while some may not due to the fact that the servers they live on are poorly maintained. Therefore, we are considering using static styling (CSS, SCSS, and JavaScript) only, but the decision is not finalized yet.

Design Changes

Our design has not changed from our design review report, but we solidified a few design details.

First, we finalized our menu and constructed an immutable dictionary for future use:

cheeseburger              $7.99
hamburger                  $6.99
veggie burger             $7.49
chicken burger           $7.49
beef sandwich            $8.99
chicken sandwich     $8.99
hot dog                        $4.99
corn dog                      $5.99
taco                              $6.99
donut                           $3.99
fries                              $2.99
onion rings                $4.99
fountain drink          $1.29
coffee                          $3.29
ice cream                   $2.99

For the MVP, we do not plan on allowing customizations or size selections.

Second, the cloud database and the staff-side module will maintain a server-client-like relationship. When the staff-side module’s subscriber thread receives notification of a new order (sent by the customer-side module when a customer checks out), it requests the order’s information from the database by spawning a child thread. This eliminates the need to constantly poll the cloud database for new data.

Third, we may change how we conduct speech interactions (e.g. near real-time parsing vs. letting customers speak one sentence and then parse) based on how well the noise-reduction and the speech recognition libraries work together.

Schedule

We move the integration between database and NLP to the week after spring break since the MVP version of the two subsystems have just been completed. As a result, the tasks following database and NLP integration have been pushed back as well.

Nina Duan’s Status Report For 3/11/2023

Personal Accomplishment

Other than completing the design review report with my teammates, I also worked on a couple of tasks.

1. Voice Synthesizer Script

To assist Shiyi with developing an accessible UI, I created a voice synthesizer script using the open source library Google Text-to-Speech (gTTS). The script allows the user to synthesize any English text from both an input prompt and the command line:

2. Database and customer-side model for orders, items, and the menu

I finalized the representations of orders, items, and the menu both on the cloud database and in local storage:

The design review report goes into detail about the model and how they interact with each other, so I won’t repeat them here. The important thing to note is that, by design, the local copy won’t be uploaded to the cloud until the customer finishes ordering by calling checkout().

I have tested the flow and successfully added sample orders into the cloud database:

3. Staff-side model for orders

I designed a model to represent orders for the staff-side as well:

This object will automatically be generated when a subscriber to the Redis pub/sub channel receives a new orderNum. It allows the staff to view order items, cross out prepared items (using finishItem()), and remove completed orders from the cloud database (using removeOrder()).

4. Redis pub/sub and fetching orders from the database

The Redis pub/sub channel is shared by the customer-side modules (publishers) and the staff-side modules (subscribers). Once the customer-side order publishes its orderNum, the staff-side subscriber thread will receive a message containing the orderNum and spawn a child thread to fetch that orderNum’s information from the database.

I have implemented this functionality as well, but it still requires more testing.

About Schedule

Since all of us are slightly behind, the database and NLP integration hasn’t been able to happen, yet. I am fairly confident that the database component is complete functionality-wise, and unit-testing has been conducted. Therefore, once we meet again next week, Lisa and I will be able to start utilizing the database with data from the NLP module.

Plans for Next Week

Our microphone and infrared sensor are set to arrive next week. Therefore, I will shift gears and start programming the microphone against our RPi 4.

Lisa and I will also try to integrate our NLP modules and database modules during the mandatory lab meetings.