Category: Team Weekly Reports

 

Currently the most significant risk for the audio localization is noise introduced through the USB ports connection using the RaspberryPi. When tested on a local computer, the signals are in the expected range with numerically and visually different outputs for different audio input intensity. However on the RPi, every signal is already amplified and louder audio is indistinguishable from regular audio input due to the resolution range of the Arduino Uno. Lowering the voltage supplied to the microphones to the minimum viable option also doesn’t solve this issue. The contingency plan for this is to perform the audio processing on the computer that hosts the web app. This ties into the system change if we can’t find any other alternative solution for it. No financial cost is incurred due to this change.

 

Another aspect of the schedule currently worked on is hardware integration of the keypad to the RPi5. I (Jeffrey) am behind on this currently, but am trying to incorporate this with the existing software code that controls the RPS game/logic. We have already adapted the contingency plan, as the current RPS game and logic works with touchscreen display, so we have that prepared in the case that integration with hardware proves more difficult. We haven’t made significant changes to the schedule/block diagram and have built in slack time on tonight and Sunday to ensure that we can continue working on system integration while getting our poster/video demo in by Tuesday/Wednesday night. 

 

Schedule Changes: 

Our final 2 weeks schedule is included in the following spreadsheet. 

https://docs.google.com/spreadsheets/d/1LDyzVmJhB0gaUmDfdhDsG_PiFwMn-f3uwn-MvJZU7K4/edit?usp=sharing

 

TTS unit testing:

For the text-to-speech feature,  we tested various word lengths to find the appropriate default string to prevent high latency when using the feature. This included one sentence (less than 10 words, ~0.5s), 25 words (~2s), 50 words (~2.5-3s), 100 words (~6-8s) and a whole page of text (~30s). We also tried jotting down notes as the text was read and 50 words was determined to be the ideal length in terms of latency and how much one could note down as text is being read continuously. During this testing, we also noted the accuracy of the text being read to us and it was accurate almost all the time,  with the exception of certain words that reads slightly weirdly (e.g. “spacetime” is read as “spa-cet-ime”).  

 

Study Session unit testing:
We tested this by creating multiple study sessions, and then testing out the various actions that can occur. We tried:

• Creating a Standard Study Session on the WebApp and seeing if the robot display timer starts counting up
• Pausing on the robot display and seeing if the WebApp page switches to Study Session in Progress
• Resuming on the robot display and seeing if the WebApp page switches to Study Session on Break
• Ending the Study Session (before the user-set goal duration) on the WebApp and seeing if the robot display timer stops and reverts back to the default display
• Letting the Study Session run until the user-set goal duration is reached and seeing if the pop-up asking the user if they would like to continue or to end the session appears
• If the user clicks OK to continue, the robot display continues to count up
• If the user clicks Cancel to stop, the robot display reverts back to default display, the WebApp shows an End Session screen displaying session duration information.
• Created a Pomodoro Study Session and set break and study intervals on the WebApp, seeing if the robot display starts counting up
• Waiting until the study interval is reached, and seeing if the break reminder audio “It’s time to take a break!” is played
• Waiting until the break interval is reached, and seeing if the study reminder audio “It’s time to continue studying!” is played

 

TTS + Study Session (Studying features) system testing:

• Create a Standard Study Session, test to make sure that while a study session is in progress (not paused), that the user can use the text-to-speech feature with no issues → user can use the text-to-speech feature, return back to the study session, and also if the goal duration is reached while the user is using the TTS feature, when they return back the pop-up still occurs
• Test that using text-to-speech feature while no Study Session is ongoing is not allowed (if no study session created → redirects to create new study session page, if study session created but on pause → redirects back to current study session)
• Test that using text-to-speech feature during a Pomodoro Study Session is not allowed 
  • This was a design change as the break reminder sent during a Pomodoro Study Session would interfere with the audio being played while the user is using the text-to-speech feature

 

Audio unit testing: 

For audio localization simulation, an array of true source values from 0 to 180 have been fed into the program, and have received an output as follows. The general trend it follows is consistent with the expectation, and goal of 5 degrees margin of error. The mean error calculation based on the output turned out to be 2.00 degrees. It is done by calculating the mean absolute error between the true angles and the estimated angles. This mean error provides a measure of the accuracy of the angle estimation. The lower the mean error, the higher the accuracy of the results. 

The total time it takes to compute one audio cue is 0.0137 seconds. 

This is done by using the time library in python and keeping track of the start and end times of computation.

 

Audio + Neck rotation testing: 

Audio testing on the microphones, signal outputs show a clap lands on the threshold of above 600 with the Arduino Uno’s ADC pin resolution. This value is used to detect a clap cue. This input is being sent to the RPi, and there have been results of inconsistent serial communication latency. This issue is still undergoing more testing. The latency is approximately 20 to 35 seconds to send angle computation from the RPi to the Arduino to change the stepper motor’s position. The most frequently occurring latency of communication is 0.2 seconds or less. 

The time it takes for the stepper motor to go from 0 to 180 degrees(our full range of motion) is 0.95 seconds.This value is inline with our expectation of less than 3 seconds for response assuming latency of 0.2 seconds. 

The accuracy testing for audio localization accuracy using a microphone is still under progress due to the latency issue, enough data hasn’t been collected to finalize this. This will be thoroughly addressed in the final report.

 

RPS logic unit testing:

Tested individual functions such as determine winner, and ensure register_user_choice() printed out that the right input was processed. 

I also tested the play_rps_sequence to display the correct Tkinter graphics and verified that the sequence continued regardless of user input timing. For the number of rounds, I haven’t been able to test the rounds being sent from the Web App to the RPi5, but with a hardcoded number of rounds, I’ve been able to verify that the game follows expected behavior and terminates once the number of rounds is reached, which includes rounds where no input from the user is registered. Furthermore, I had to verify Frame Transitions,

Where we would call methods to transition from state to state depending on the inputs received. For example. If we are in the RPS_confirm, we would want to transition to a display that would showcase the RPS sequence, once OK is pressed (in this case, OK would be an UP arrow key press on the keypad)

 

Finally, I had to handle websocket communication to verify that information sent from the Web App could be received on the RPi5 (in the case of handle_set_duration). The next step would be ensuring that I can properly parse the message (in the case of handle_set_rounds) to have an accurate retrieval of information sent from the Web App.

 

For overall system tests, I’ve been working on the RPS game flow, testing that 3 rounds can be played and that the game produces expected behaviors. I found some cases of premature termination of displaying the sequence or missing inputs, but was able to fix that through iterative testing until the game worked fully as expected utilizing touchscreen buttons. The next step from here would be eliminating the touchscreen aspect and transitioning the code to utilize key pad inputs with evdev (a linux based system that the RPi5 can incorporate). Web App integration also needs to be worked on, for the RPS game. For the study session, Shannon and I have worked on those aspects and ensured that the study session behavior works fully as expected. I would also have to start doing overall system tests on the hardware, which would be the keypad presses in this case. I want to verify that the evdev library can properly detect hardware input on RPi5, and translate that into left -> “rock”, down -> “paper”, right -> “scissors”. In the case of the up key, we would expect different behavior depending on self.state. For instance, in the rps_confirm state, an up press would act as a call to start_rps_game. If we are in the results screen, an up arrow press would act as proceeding to the next round. If we are in the last round of a set of rps games, the up arrow would act as a return home button, with the statistics of those rounds being sent to the Web App.

Risk:
One risk that our team is currently addressing is the RPi5 acting as a server to the Web App. Our contingency plan is using the DSI display touch screen to ensure that features such as pause/resume/end study session is working, and button presses on the DSI display would be synced with the Web App. Right now, we are testing that with socket.IO, we are able to communicate with the RPi5. In particular, we want the duration to be inputted on the Web App, to initiate a study session, and for the timer display to start ticking on the DSI display. In the future, we plan to integrate this with GPIO pins, so the user can directly input key presses on the robot body, and the RPi will emit an event that has occurred, such as a pause/resume. After the emit, we want the web socket to listen for these events and update the Web App accordingly and sync itself with the DSI display (controlled by RPi5).

Validation:
We will use this process for all four of our features (Study Session, Text-to-Speech, RPS Game, Audio Response). Using Study Session feature as an example, this is what our validation process looks like:

• Have users go through and use our Study Session feature from start to end. Ideally we would want a good mix of 5-10 people who have used online study timers and people who have not.
  • We will have the different users try out starting a Study Session on the WebApp (setting the name, duration, type) and then pausing/resuming the session on the robot, and then finally ending the session on the WebApp.
  • We will have them try out the different types of study sessions (Standard vs. Pomodoro – comes with built-in break time) and also try ending a study session before the set duration is up vs. ending it after (when the study session has reached goal duration it should send an alert to the user to ask them if they wish to continue and be able to do so)
  • At the end, we will have them fill out a feedback form focusing on
    • Any confusion/unintuitive use of the product (e.g. buttons too small, want certain information to be displayed elsewhere) – General question asked for all 4 features
    • Satisfaction with the WebApp and robot interactions (e.g. latency issues) – Specific question for Study Session, RPS Game, Text-to-Speech
    • If the user felt this feature would help them develop better time management for tasks and foster better studying habits – Specific question for Study Session
    • Overall suggestions for improvement of this feature – General question
• For the Text-to-Speech feature,
  • Feedback form will also include a question asking:
    • How the feature was useful for audio learners
• For our audio response feature, it would be similar except
  • During a study session, the robot will reset to the default position of 90 degrees, giving full attention towards the student.
  • During a study session break, the user can get the attention of the robot at a different location, and proceed with any task. The user can play the rock-paper-scissors game, or do any break time activity as desired.
  • Feedback form will also additionally include:
    • How the feature facilitates their study break (was it entertaining or distracting)
• For the RPS Game feature,
  • Feedback form will also include:
    • How the feature facilitates their study break (was it entertaining, a good stress relief or not really useful/boring)

This will be a Google form with 4 sections to provide feedback on each feature.

Design changes:
There have been no design changes to our project
Schedule changes:
There have been no changes to the schedule.

The most significant risk is currently the arrival of parts being delayed. Despite the submission of parts prior to fall break, we still haven’t received them, pushing our parts integration and testing timeline backward. We have the hardboard wood, purchased from TechSpark. Once the parts are in our possession, we can start assembling the robot base. Timeline wise, the assembly should only take a couple of hours. Our contingency plan is to make good use of our slack time assigned to make sure we can catch up to our schedule and finish building the robot base. 

Another aspect of the design that poses some risk is the overall integration. We would want our Study Session and TTS feature to be able to occur at the same time (i.e. while studying, a student should be able to use the TTS feature and have the robot read to them without it affecting the session in terms of pausing or ending it). We want the WebSocket communications and the TTS processes to seamlessly integrate but there may be some threading issues that we have to account for which may cost us additional time to integrate those parts together. Our contingency plan for this is to use multithreading on the RPi. There is also a chance that the TTS feature might take up too much processing resources on the RPi, affecting the study session and so in that case, our contingency plan would be to have the TTS work directly on the user’s computer through the WebApp. 

No further changes have been made to the overall design of the system, and we are keeping a close eye on our project to make sure we adjust if necessary when any further issues arise but as for now, we foresee no issue with our current plan.

The most significant risk for our team right now would be the integration of the Robot with the WebApp. We are thinking of using WebSockets because of its low-latency nature with full-duplex communication which would allow better real-time communication between the robot and the WebApp.  However, a key challenge is that none of us have prior experience with using WebSockets in this specific context, creating uncertainty around implementation and potential delays. To manage this risk, we plan on scheduling dedicated time for learning WebSocket integration and seeking advice from mentors or who have used WebSockets.  As for our contingency plans, we plan to possibly switch to a standard HTTP-based communication using REST APIs over WiFi, (though this might introduce higher latency), or using a physical Ethernet connection to reduce the risk of network disruptions, (though this would reduce flexibility in robot placement and mobility).

Another possible challenge is integrating the DCI display with the Raspberrypi. Ensuring a reasonable frame per second value along with a smooth facial transition, such as blinking and smiling,  to ensure human-like interaction with the bot. To implement this, we will use certain python graphics libraries like Pygame for simple 2D rendering, or Kivy for a more advanced interface.  To maximize the lifespan of the screen, we will be using screensaver and sleep mode  during idle moments or periodically change the content displayed on the screen by making slight changes. This can be done during timer countdowns and is generally not a concern if the user is not using the bot. 

We also got together as a group to update our system specification diagram, which we included in our design proposal.

We decided to allocate specific time to the WebSockets integration of the robot next week.

Part A was written by Mahlet Mesfin

Our StudyBuddyBot is a study companion meant to motivate students to study, track their study habits and provide relaxation when they take breaks in between their study sessions. This allows students to have a good experience while being productive inducing psychological satisfaction. The bot can guide students to follow optimal study schedules(such as the Pomodoro technique), which ensures a well balanced approach to work and rest. This will help prevent overworking, overall leading to a better mental and emotional health.  In addition, we will be incorporating reminders for the user to take breaks in reasonable intervals reducing fatigue and eye strain. 

The game feature of this StudyBuddyBot allows for a short but fun experience during these study sessions, timed well so that they don’t cause prolonged distraction. This will also help in fostering the sense of companionship and reducing the feeling of isolation for those who can’t focus well in the presence of other individuals. This can boost the well-being of someone through emotional support. 

Part B was written by Shannon Yang

The StudyBuddyBot will improve productivity and the well-being of students in academic environments. It will serve as a structured study companion that can help students. In situations where students have limited access to in-person interaction due to cultural factors, the robot is able to simulate a studying environment with a friend. The features for interaction that the robot has can also help to bridge gaps in the social and emotional support systems that students may lack from their surroundings. Some of the robot’s features could also be used to cater to specific cultural or social preferences (for example, setting prayer time reminders for those who observe religious practices). By incorporating both study assistance and social engagement, the robot aligns with the growing trend of technology being used to support mental health and productivity, acknowledging the cultural and social importance of companionship in learning and promoting greater work-life balance. 

Part C was written by Jeffrey Jehng

With the StudyBuddyBot, we want to use cost-effective components to balance affordability with quality. By implementing a modular design, we can have a scalable distribution in the future and ensure durability of the final product. 

An example of our final product use-case could be in a school setting, where administration/students may have a limited budget for these educational tools. By designing the StudyBuddyBot with affordable components and integration with a companion web-app to decrease the need for high-performance hardware, we can focus on developing key functionalities such as robot interaction and features to motivate student studying. The emphasis on affordable components under our $600 budget can make our design a cost-effective solution to assist schools in integrating advanced technology into the classroom.   

The most significant risk right now would be the usage of the MEMS microphone. We have some concerns about how it may be difficult to triangulate well with our robot’s small form factor. We are planning on mitigating this risk by adding a directional microphone. Performing audio analysis on the MEMS requires careful and precise array design to have good accuracy. Using a directional microphone will help in identifying the general source of the audio and pairing that with an array of MEMS microphones will allow for better audio localization and recognition. 

An array of three MEMS microphones are sufficient to perform a 2D plane triangulation along the X and Z axis of the robot head. Microphones will be placed on the back of the head, on the left and on the right. Performing signal analysis on the input from each microphone will allow us to identify the source of the audio with a margin of error of about 5 degrees.

We believe that this combination of changes to the microphone system would improve the accuracy of the system. As an additional layer of risk mitigation, we plan to reduce the activation distance of the audible cues. This will ensure that the microphones would be able to accurately pinpoint the location. To that end, our desired goal would be for the audible cues to activate once the student is at the desk, within 3 feet of the robot.

Based on the feedback, slight changes to the Gantt chart have been made to account for time to identify specific parts that we need to purchase for our robot. We looked at the purchasing pdf to prepare a purchase request form from the parts inventory and we are working on narrowing down the list of parts that we will need. This will help us have a clearer picture of exactly what components our robot will have and for what purpose.