Posted on

Jubahed Qayum’s Status Report for 11/30/24

PERSONAL ACCOMPLISHMENTS

  1. Lab meetings (6h): My team and I presented our interim demos. We got great feedback and thought it went well!
  2. App backend + frontend merging (6hr): Last week, Alex’s frontend elements were isolated and did not utilize bluetooth connectivity or commit any data to device storage. Therefore, I worked with Alex to merge together my backend APIs (Bluetooth + Database commitment) with his frontend elements. This made sure that the user experience for the device was significantly more ergonomic and seamless.
  3. ESP32 programming (3hr): I showed Alex my ESP32 toolchain setup and explained my FSM backend process on the ESP such that Alex would be able to start working on the calibration sequence as easily as possible.
  4. Device Verification & Validation (6hr):  Alex and I conducted a variety of software unit + integration tests to ensure the robustness of our app. This included bluetooth distance testing (mocking the physical glove with dummy data and ensuring the app stayed alive when the device moved 15m+ away), making sure that buttons could not be spammed and that the FSM of the app could not enter any unknown state (if a user presses a button twice within 5-20 ms, before an animation sequence occurs, for example, then the app should not start any code sequence twice). Since the migration to the frontend elements removed aspects of debugging I had prior, like displaying database elements properly, we learned how to debug the app using console logging via USB and displaying the database through Android Studio’s logcat and Database Inspector tools. These were used to make sure data was properly being committed to the database.
  5. Final Presentation prepare (6hr): I helped my team put together the final presentation slides, specifically information regarding the HW/SW interface, Bluetooth, and data storage.
PROGRESS
  1. Continuing testing into a more-than-MVP state
  2. Everything is on schedule!
NEXT WEEK TASKS & GOALS
  1. Alex and I need to sync up again to make sure that the app frontends can properly **retrieve and display** data from the database with specific queries. I’ll need to work closely with him to make sure my backend APIs are clear and further more robust for his use.
  2. Integrate second glove into backend APIs. This will include a lot of redundancy but will still take a sizable chunk of time.
  3. Continue testing, get more involved with end-to-end testing
  4. Work on final report
GUIDANCE

I think I learned a lot of new skills working on this project, but more specifically I learned about both app development and the process necessary to build an app effectively for Android, as well as the general ideation + design process for a project of this caliber, especially when it comes to deciding what components to use in the first place.

With respect to Android development, prior to this project I had actually never created an Android app before. Therefore, it was definitely foreign territory and a significant amount of time was spent on trying to make sure all elements of the app were working, plus just figuring out how to navigate Android Studio in general took a long time, even before assembly of our physical artifacts began. I learned the Kotlin programming languages, how Android breaks app pages into ‘Activities’ and how those activities correspond to frontend and backend elements, and how to utilize device permissions to do awesome things (like geolocation or Bluetooth access).

With respect to strategies to tackle this sort of project again, I really think I learned how to effectively pick and place components. By this, I mean that at the start of the project, my team really started from a blank canvas, where we had to decide entirely on our own what microcontroller to use, what sensors to use, whether we should develop for Android or iOS, etc. Before Capstone, these sorts of problems were already solved for me, where someone else had done that work already, but now being in the shoes of that person, I can see why this is a difficult and daunting task. We initially stumbled around, trying out different hardware platforms like Arduino, STM32, etc., until we landed on ESP32 (and even then, the first devboard we decided to trial on didn’t support power inputs the way we wanted). This process of iteratively building our design and our components over time + over trial and error taught me that

  1. Your design doesn’t need to be perfect off the jump, but in fact that this is a growing item and can change as the parameters, scope, and level of understanding around what we want to project to obtain or achieve changes
  2. Research really is necessary before any type of assembly or work. I know our team was itching to put together physical + software artifacts ASAP, but I distinctly remember that during our design report work sessions, we genuinely had points where we realized that we had NO CLUE how to pursue a subaspect of the project and had to spend time understanding it. Sure, the funnest part of the project was the assembly, which we only really did toward the end of our project, but the prior research and acquisition  of knowledge from our past reports really was necessary for us.
Posted on

Alex Nguyen’s Status Report for 11/30/24

Personal Accomplishments
  1. Programming ESP32 (5h) – I programmed the calibration routine for the ESP32, such that upon session startup (and immediately following a confirmed bluetooth connection) the calibration routine will launch and take the average of the next 10 seconds of force readings for each sensor, storing that value as the normalization value for the rest of the readings taken during the session. This can be seen below, as the normalization values for each sensor are displayed upon completion of the CR at 10 seconds, and then the device proceeds to take readings for normal data collection. 
  2. App & Bluetooth Integration (12h) – I worked with Jubi to integrate the front end with bluetooth and database. Once Jubi established a working bluetooth connection between the device and the app, I realized that our current code was overly reliant on disconnected Kotlin coroutines, and was able to simplify them into two larger Kotlin coroutines and suspending sequential coroutines until prior ones had finished which prevented many of the bugs that were occurring when two coroutines overlapped or attempted to access data at incorrect timings.  When I completed the calibration routine on the ESP32, I synchronized the bluetooth connecting screen and calibration timer screen to match the events occurring on the ESP32 (with the bluetooth connecting screen staying until either it was connected or timed out and the calibration timer matching the timer of the ESP32).  This can be seen below where the light to indicate bluetooth connection is blinking before the session starts, then turns solid when a connection is established (matching the changing of screens in the app). 
  3. Android App Development (frontend) (10h) –  Since the last update, I implemented the rest of the frontend that was remaining. This included the threshold settings page, questionnaire, and session history selection page. I also attempted to query Jubi’s database for the max sensor reading values recorded for each pulley during a session, however it is currently inconsistent and returns either the same values or random values, which is something I will work on further. Assuming this is not an issue with the database, I believe this is most likely an issue with data sharing between the MainActivity and the HomePage, potentially caused by uninitialized key-value pairs in sharedPrefs.
  4. Integration Testing (5h) – Went to the climbing gym with Josh and conducted a series of tests on different climbing apparatuses (hangboard, pullup bar, campus wall, bouldering wall). We were able to verify several key factors, including bluetooth connection between the app and device, as well as test the reliability and consistency of the sensor when doing various climbing movements. Additionally we were able to test the usability of the device, both by climbing on routes with the glove on as well as surveying several climbers who were also at the gym.
  5. Final Presentation (8h) – I am presenting this week, so I worked on the presentation slides and prepared to present during this week’s lab meetings.
  6. Mandatory Lab Meetings (6h) – Presented intermediate demos to professors and TAs and received valuable feedback, especially regarding the data visualization and analytics components to our mobile app.

As you’ve designed, implemented and debugged your project, what new tools or new knowledge did you find it necessary to learn to be able to accomplish these tasks? What learning strategies did you use to acquire this new knowledge?

Before this project, I had never used Android Studio or Kotlin before, and I was pretty unfamiliar with app development model. As a result, I had to spend a lot of time at the beginning of the semester experimenting while building the app, even before our team had a concrete plan as to what features we wanted to have as part of the mobile app. I found tutorials online very helpful for learning the building blocks of app development, and learned a lot through tinkering with different interactions between frontend and backend elements in dummy applications that I created just for the purpose of testing out potential interactions I could want in the final app.  This learning process was a lot of trial and error, so git version control was incredibly important as there were several instances where I had to rebase after creating an error whose origin I was unable to identify.

Additionally, I learned a lot about the design process for an embedded system mobile app. Prior to this project I had never had to consider both factors when designing a product, so there were many new challenges that I had to consider (along with my teammates). While we were writing our design report, we realized how much of our design we had not considered regarding the interactions between each of our individual parts. Sitting down and talking through the overall interactions between the hardware, frontend software, and backend software, was immensely helpful, and while it took a long time, it proved to be invaluable to the success of our process later on when we began integrating. This skill was developed further throughout the semester, as we made sure to revisit (and modify if necessary) our overall integration design plans each time we made major changes to any one individual part.

Finally, designing thorough unit and especially integration testing was a skill that I found very important to learn throughout the project. While I have conducted unit testing for software projects before, the addition of a connected embedded system communicating via Bluetooth added an extra dimension of testing needed between all three systems (embedded device, BT communication, backend database, frontend UI/UX). This skill was learned on the go, as we each performed our own tests on our individual parts but had to figure out how and what to test during integration. Figuring out the many aspects that needed to be verified and validated was a process that often happened accidentally, as one of us discovered a factor that had not yet been considered (things like finger placement affecting sensor readings, BT connection failure during a preset sequence, race conditions occurring when a coroutine terminated early). Finding new factors that could adversely affect the optimal performance of our product taught me both how important it was to design tests for as many of these potential issues as possible as well as how to design appropriate tests to ensure our goals were met.

Progress
  1. Progress is on schedule.
Next Week Tasks & Goals
  1. Data Visualization and Analysis: Work with Jubi to retrieve data from database and display based on session number.
  2. Dual Glove Additions: Make any additions necessary for Bluetooth and Mobile app with the introduction of a second glove (likely frontend additions, some communication changes)
  3. Final Report: work on final report with teammates.
  4. Testing: continue testing as the final product comes closer to completion, especially with regards to new additions or aspects that were modified. Additional reliability tests and real-world application tests (in climbing gym).
Posted on

Team Update for 11/30/2024

General Update

This week Josh, Alex, and Jubi worked on the final presentation, performed verification & validation tests on the hardware & software solutions, and continued work on the device integration / began some stretch goals. Josh worked on tuning the amplification gain/range to provide more robust and consistence sensor readings. He also created a second glove using the new Olimex ESP-32 and tested/confirmed battery-powered functionality. He also worked with Alex to perform some verification and validation tests at the Iron City Bouldering Gym, and he began working on a PCB design and device capsule CAD. Alex primarily worked on programming the calibration routine on the ESP32, as well as rewriting front end code for the startup sequence (Calibration Routine, Bluetooth Connection) to avoid coroutine-related errors and sync the states on the ESP32 to match the timing of the mobile App frontend. Alex also conducted testing focusing on the interactions between the app and the wearable device during a full session from beginning to end. Jubahed assisted Alex with software end-to-end testing, as well as explaining his APIs to make sure that Alex could properly build the calibration system upon them. He also continued work on the back-end database system, as well as integrating bluetooth connectivity between the device and app. This mostly included integrating the prior frontend elements with the backend APIs, and ensuring that they worked together as intended as smoothly as possible.

Risks and risk mitigation

None

overall design changes

We are now using the Olimex ESP-32 LiPo DevKit board with an on-board power-jack because it allows us to provide battery-powered functionality to the CLIMB device without needing to design an LDO circuit on the PCB, though this solution is a bit more expensive than the previous ESP-32 we were using from AdaFruit.

schedule

The schedule remains the same as we are on track.

GANTT Chart

 

Posted on

Joshua Ramos’ Status Report for 11/30/24

Personal Accomplishments
  1. Lab meetings (6h): During our schedule lab meeting hours, we did our interm demo and recieved great feedback. I also continued work on the hardware solution, which is described below.
  2. Second glove implementation (4hr): Worked on creating a second glove. This consisted of copying the first glove in implementation, but instead using our new Olimex ESP32 with an on-board power jack so we can test a battery-powered implementation. I spent a deal of time soldering up 4 more sensors with wiring such that they can be attached to the second glove. The battery powered ESP works!
  3. Built pull-up bar for final demo (1hr):  I spent time building the pull-up bar we ordered for our final demo.
  4. Sensor tuning (3hr):  I spent time deciding what gain/voltage range we wanted to use for our sensor amplifier, after testing whether a 0.5-3.3 vs. 0.25-3.3 vs 0.15-3.3 voltage range, using a gain of (837k/[3M->100k]), I found that the larger forces were better measured, since the the gain was more linear, using the 0.15-3.3 range.
  5. Device Verification & Validation (5hr):  I spent time working with Alex to run verification & validation test at the Iron City Bouldering gym where we performed durability, safety, ergonomic, and BT speed tests using the open climbing environment with various climbing levels and hangbars for more static weight tests.
  6. Final Presentation prepare (8hr): I spent time working on the final presentation, specifically updating the solution body diagram, the hardware solution, the testing requirements, the hardware/software test results,  and design tradeoffs.
Progress
  1. Continuing work on verification and validation tests
  2. Everything is on track! Working on stretch goals (PCB, CAD, Second Glove, etc.)
Next Week tasks & goals
  1. Working on PCB and CAD for device capsule
Guidance

I learned how to find an appropriate microcontroller for a given application and how to use a new one: ESP-32. Previously I had worked with an STM32 and Teensy 4.1, but those were in projects where the MCU was chosen for me. So for this project, I had to do some research into viable MCU’s, where I learned how to perform proper tradeoff analyses for deciding which hardware to use for my application. I spent a great deal of time deciding which factors I should use to determine viability, and in doing so, found myself learning how to scan through reviews and documentation to perform comparisons of each device. The same occured in my process for finding which sensors we should use for our CLIMB device. I even had to call sensor distributors to acquire more information about the sensors I was considering for my application. This process forced and taught me to become more comfortable in information gathering and tradeoff analyses for the application design process. In regard to learning how to work with the ESP-32, I learned how to work with the Expressif toolchain to flash and debug my CLIMB code. I also worked with the Arduino libraries to utilize the on-board ADCs and PWM modules, which required me to look into the MCU’s peripheral functionalities, and decide which ports I would use for the various tasks my application required.

Posted on

Team Update for 11/16/2024

General Update

This week, Josh, Alex, and Jubi spent most of their time working on preparing the product for demo. This consisted of Josh modularizing the prototype with plug-and-play sensor functionality, soldering the sensor wiring and heat-gunning insulation covers instead of using jumpers for a clean look, running 4 sensors at once that connect to dc motors that vibrate upon threshold excession. Alex worked on the CLIMB user experience by designing the UI in Figma, and programming it in Android Studio. Jubi worked on implementing bluetooth communication between the ESP32 and the Android smartphone, achieving communication from the board to the phone and implementing a state machine that transitions the phone between start and stop modes. He also worked on implementing the database that will store the CLIMB metrics on the local device. Next steps are to achieve communcation from the phone to ESP, design a PCB to house the ESP and amplifier circuit, and data visualization and analyses to display on the UI. Also we will begin verification and validation.

Risks and risk mitigation

None

overall design changes

Using PWM to control the haptic motors which will have 3 vibration levels (1V, 2V, 3V).

schedule

The schedule remains the same as we are on track.

GANTT Chart

Guidance Question

In all, we are to perform ergonomic validation tests, durability verification and validation tests, safety verification tests, social/feedback verification tests, and finally end-to-end varification tests.

So far, we have performed basic hardware utilization tests such as working out with the glove on while monitoring the force readings, making sure that the readings are consistent all-throughout. The results of this test have been favorable, resulting in consistent readings.

(HW) Testing Durability

To verify the system’s durability requirements, the device will undergo 50 uses, each for 5 minutes with a constant 10kg applied at a 90 degree angle. This will be repeated for 50kg as well. After each, the sensitivity loss of the device will be verified to be within 2.5% and the wearable fabric will be examined for any tears or disfigurations. To validate durability, the device will undergo dynamic weight applied by a user at various angles for 1-hour, 10 times. After each, the sensitivity loss of the device will be validated to be within 2.5% and the wearable fabric will be examined for any tears or disfigurations. This meets the system’s durability use-case requirements by tackling even the most extreme cases (54kg/sensor) and verifying that the system remains robust.

(HW) Testing Ergonomics

To validate the system’s ergonomic requirements, a survey will be conducted on 50 randomly selected participants. Each participant will provide the shape characteristics of their fingers. With this information, the system sensor size requirements will be tested to only impose up to 30% surface area coverage for at least 90% of participants’ fingers. This meets the system’s ergonomic use-case requirements by providing a comfortable and unobtrusive wearable device that covers no more than 30% of a user’s finger.

(HW) Testing Safety
To verify that the system initializes properly, the system will be booted up and calibrated 50 times. The system shall successfully calibrate 95% of the time.
To verify sensor precision, incremental weights will be placed atop the sensor (1kg, 2kg, 3kg, etc.) and a voltmeter will be used to monitor the voltage changes per kg. The system shall reflect a 5mV/kg voltage change up to 54 kg. To verify that an alarm signal is triggered within 100ms when the force reading is within 20% of the safety threshold, an instant force exceeding the threshold will be applied after initialization, and an external timer will record the delay. The system shall fire its alarm within 100ms at least 98% of the time. To verify the second haptic’s functionality, the first haptic will be disconnected. Immediate force will be applied until the threshold is reached. The system shall fire its second alarm within 200ms of threshold excession with a 98% success rate. This meets the system’s safety use-case requirements by demonstrating that the system will actively work to prevent pulley injuries by remaining with a maximum error of ±2.5 and firing alarms within 200ms.

(SW) Testing Social / Feedback

To verify the feedback requirements of the system, tests will be performed using mocked force readings to verify the minimum data transfer. To do this, a mocked buffer input of 96 Kb will be generated and communicated to the mobile device. This test will be timed, recorded, and repeated 50 times to verify that the shipment completes in 10s with a 95% success rate.
To verify the user-experience related requirements, 96Kb of data will be mocked on the mobile device and passed to the database handler to ensure that information is stored properly and can be pulled to display on the application charts. This test will be repeated 50 times to verify that the flow of data on the mobile device can reach the user’s screen with a 95% success rate. These tests meet the use-case requirements by verifying that the system is providing users with a seamless experience in accessing analytical feedback that is suggestive towards improvement and can be used to compare progress with others.

(HW/SW) Testing End-to-end

To verify end-to-end integration, the system will undergo 100 simulations where mocked force readings will be generated and communicated to the mobile device, testing to determine if the data is successfully displayed analytically 95% of the time. To validate the system, a test using a hang-board with force-sensors mounted will be used to perform dynamic tests. The force on the hand-sensors will be cross-referenced with the force-sensors on the board. This test will be conducted 20 times for 10 minutes each. The system will be tested for the average error to be below 2.5%

Posted on

Jubahed Qayum’s Status Report for 11/16/24

PERSONAL ACCOMPLISHMENTS
  1. Lab meetings (4h): During our schedule lab meeting hours, I chatted with Alex about finalizing design decisions regarding the software framework of our project. I also learned how to make and flash code to the ESP32, as this would be necessary for building the Bluetooth APIs for both ends of the project to communicate with each other.
  2. Interim Demo Prep (19hr): I spent a large chunk of my week figuring out the ins and outs of ESP32 bluetooth connectivity. I created a simplified API for the ESP32 to do a variety of operations, including appending data to on-device queue, scanning and pairing with Bluetooth devices, and reading / writing from serial Bluetooth buffer. I also created a FSM for the device to follow a variety of strict states with respect to Bluetooth operations, (idle, collect data from sensors, send buffer to serial Bluetooth) where transitions are initiated by the smartphone app. This API was passed on to Josh to integrate his ADC interface onto it.
    Speaking of the smartphone app, I spent most of the week developing the backend for the app. This included utilizing the Bluetooth connectivity permissions on my Android smartphone to sync with the ESP32, to initiate an ACK system with the ESP32 API to start / stop data collection and control the ESP32’s FSM, and a simple frontend for displaying this functionality for the demo. This also includes committing data collected from the ESP32 to permanent storage on the phone via a working SQLite instance.
PROGRESS
  1. The ESP32 code is fully fleshed out and complete. The Android app backend is complete.
NEXT WEEK TASKS & GOALS
  1. Begin verification and validation tests. Sync my backend implementation with Alex’s frontend elements to make the process more user-friendly. Help Alex set up AnyChart API queries for grabbing data from the database.
Posted on

Joshua Ramos’ Status Report for 11/16/24

Personal Accomplishments
  1. Lab meetings (4h): During our schedule lab meeting hours, I continued work on prototype development and looking into potential battery sources for our device.
  2. Intermin Demo Prototype Prepare (20hr): This week I worked on preparing the hardware prototype for the interim demo. This consisted of soldering the sensors and haptics, hooking up the 4 sensors to the ESP, integrating with Jubi’s bluetooth implementation, converting the ADC readings to its corresponding force values, having the haptics fire with 3 different levels depending on how much force it placed on a particular finger, have each sensor connect to a specific motor, and add tape/velcro to make the product easier to put on and take off.
Progress
  1. The hardware prototype is complete.
Next Week tasks & goals
  1. Begin verification and validation tests. Begin designing the PCB to house the circuit and MCU. Order batteries for the MCU.
Posted on

Alex Nguyen’s Status Update for 11/16/2024

Personal Accomplishments
  1. Mobile App Development (20h):
    1. Figma Design Docs & UI/UX:  finalized app design by creating Figma design docs displaying actual UI/UX layouts/design throughout each intended functionality (home page interactions, threshold settings updating, start/stop/calibration sequence) so that I could build/develop the final version of the android app according to the Figma design docs (especially regarding interactions between frontend elements and processing data from backend databases, but also with regards to aesthetic and quality of use elements) Android Studio Development: Modified mobile app codebase to conform to the Figma design docs I created, made significant progress with regards to the “Threshold Settings” updating functionality within the app (can now confirm updates and throw exceptions if the update was not processed) and reflect threshold changes on the home page. Additionally, the session start and stop timer was implemented successfully and calibration sequence frontend screens have been implemented and their functionalities will be connected to the wearable device soon.
  2. Mandatory Lab Meetings (4h):  Met with team, TA, and Prof. and discussed interim demo preparation and expectations.
Progress
  1. Progress is on schedule.
Next Week Tasks & Goals
  1. Mobile App/Device Integration:  Major goal for this week is merging my android app repo with Jubahed’s bluetooth repo so that we can begin connecting the two parts of the android app and begin testing with all three components (glove, app, database/communication).
Posted on

Alex Nguyen’s Status Report for 11/09/2024

Personal Accomplishments
  1. Mobile App Development (6h): Tested dummy data with AnyChart API, and worked on integrating separate aspects of the app with Jubahed, specifically overall app functionalities between the session history and data visualization pages/functionalities of each respective page. Continued working on the initial startup sequence of the app and began work on bluetooth pairing/implementation within the app.
  2. Mandatory Lab Meetings (4h):  Met with team, TA, and Prof. and discussed details regarding aspects of the device, such as the possibility of custom calibration sequences and angle sensors.
  3. Overall Mobile App Testing (2h):  Continued user tests in the latest iteration of the app using AS’s android OS emulator to ensure that the app UX works as intended (scrolling between pages, interactions between buttons on each page).
Progress
  1. Progress is on schedule.
Next Week Tasks & Goals
  1. Mobile App/Device Integration: Begin integration testing between the app and the device.
  2. Testing: Begin testing the intended use case using prototype consisting of sensors attached to crack climbing glove (ordered this week, arrived on Friday). This testing will also aim to determine the consistency of readings while on the selected finger locations as well as durability and comfort of the prototype device,
Posted on

Jubahed Qayum’s Status Report for 11/09/2024

Personal Accomplishments
  1. Mobile App Development (6h): Spent a large amount of time fixing bugs and fault-testing app backend. Learned how to navigate Android Studio emulator to test app deployment without needing an actual phone. Hoping to combine the mobile app / software prototype enough to interact with the glove for initial demos coming up soon.
  2. Mandatory Lab Meetings (4h):  During our lab meetings, I caught up with Alex on changes or suggestions for the app. We worked toward getting Bluetooth read capabilities on our app.
Progress
  1. Progress is on schedule.
Next Week Tasks & Goals
  1. HW/SW interfacing: Combine both components of our project together and make sure they work as intended before demo day.