Author: yuehanh

Team’s Status Report For 10/10

Progress this week

Most of our work this week went into the design report. We brought the three workflows into one description and worked through the use cases, design tradeoffs, requirements, and tests we will need to show that the system works. We also checked that the report separates laptop results from plans for the Jetson. The three workflows still run separately on laptops. We have not run them together on the Jetson or completed a live camera-to-tablet test.

Alan compared simple image methods with the treadmill pose model and examined where its update delay comes from. Harry tested image subtraction for rack belongings. It gave more definite answers on the scored observations but also mistook a person for belongings four times. Jerry continued testing the bench workflow, documented its errors, and measured its perception check on a laptop CPU. The bench check still has the most work to do on accuracy.

Risks and responses

Running all three workflows on one Jetson is still our largest technical risk. The laptop measurements help us plan, but they do not tell us whether the combined workload will meet our update targets. We will run each check on the board, then measure them together before deciding whether we need faster models or changes to how often stations are checked.

The bench workflow still struggles to tell which person belongs to which bench and whether plates are on the working bar. Jerry is testing those failures on separate footage. The rack workflow needs longer recordings and more kinds of belongings before we can test a full five-minute wait. We also need more footage set aside for final evaluation across all three checks.

Recording permission at the apartment gym is resolved. Confirmation for the UC gym is still pending.

Design and schedule

We did not make a major change to the system design this week. We made the requirements more specific, including when a reading is unknown, what starts or stops an unattended timer, and how we will score delays and incorrect alerts. Those details give us clearer tests, but they have not been verified on the full system yet.

The design report took priority over hardware work and longer recordings this week. First Jetson checks are now planned for the week of October 19, followed by individual live workflows and then the combined tablet path.

Part A Global factors

Written by Harry

This system is not just limited to CMU gyms. People in other gyms could also benefit from knowing whether equipment is free before walking to it. Their gyms may have different layouts, lighting, machines, and internet access, though. BenchGuard processes video locally and lets someone adjust the areas assigned to each machine, which could help it work in different settings. We would still need to check camera placement and test accuracy at each new site. Our current clips do not show how well it would work everywhere.

The display should also be easy to understand without knowing how the system works. If a camera cannot see a machine clearly, showing unknown is more useful than guessing that it is free.

Part B Cultural factors

Written by Jerry

People have different expectations about resting between sets, sharing equipment, and leaving belongings nearby. Those expectations can vary between gyms as well as between communities. Our five-minute wait is a setting to test, not a rule that says someone has given up a rack or bench. A staff member would check a flagged station and decide what to do under that gym’s rules.

The message should describe what the system saw without blaming a member. We also need to test people working out with partners, taking longer rests, and walking past nearby machines. A gym would need to review the waiting time, wording, and camera notices before using the system.

Part C Environmental factors

Written by Alan

Cameras and computers use electricity and materials, so adding more equipment has a cost beyond its purchase price. We are reusing course inventory where we can and plan to test whether one clear camera view can cover more than one machine of the same type. We cannot assume that wider coverage saves resources if it makes the readings unreliable.

We will measure the Jetson’s power use with the workflows running before claiming any energy savings from a different model or setup. We also want mounts and other prototype parts that can be moved and reused as the camera positions change.

Harry’s Status Report For 10/10

Accomplished

Most of my time this week went into the design report. I worked on the squat-rack sections, the design tradeoffs and helped formalize the use case / design requirements, as well as the testing and verification. I also helped make sure the design presentation feedback was fully incorporated into the report.

I also tested whether image subtraction could replace the belongings detector. On 29 scored observations, it got 25 right but made four false calls after someone collected their items. The person was still in the belongings area, and image subtraction detected that visual change without knowing it was a person rather than an item. The detector got 19 right, made no wrong calls, and reported ten as unknown. Image subtraction gave more definite answers, but these false calls mean I would not replace the detector based on this test. I added the results to the report. The subtraction threshold was chosen using the same clips, so it still needs testing on separate footage.

Progress

The rack workflow still runs on recorded video on my laptop. I did not run it on the Jetson or collect the longer rack scenarios this week because the report took priority. Those are the main gaps now. My planned first rack run on the Jetson is October 21.

Deliverables

  • Run the rack workflow on the Jetson and record check time, memory use, and failures.
  • Record longer rack cases with different belongings, passersby, and a full unattended period, once recording permission is settled.
  • Keep some footage separate from development and ask a teammate to review a sample of the labels.
  • Work with Jerry and Alan on the shared station output and the first combined test.

Jerry’s Status Report For 10/10

Accomplishments this week

Much of my week went into combining and revising the design report. I worked through the architecture, state and timing rules, diagrams, schedule, and testing plan so that the report describes one consistent system. I also checked claims against the evidence we actually have, especially where laptop measurements or short clips could sound like completed Jetson or gym results.

I continued diagnosing the bench workflow on recorded and public footage. Its current 172 observation development bank has 96 correct full-state results, 47 wrong, and 29 unknown. The errors include trouble associating a person with the right bench and deciding whether plates are attached to the working bar. I tried several bounded changes and kept the failures documented instead of treating a better result on one view as a general fix.

For the report, I checked whether image subtraction could provide a simpler loaded-bar test. The existing bank lacks the sleeve regions and bare-bar references needed for a matched comparison. A separate, previously calibrated session gave promising load-only results, but its thresholds were chosen on that same session. I also timed the actual bench perception models on my laptop CPU: the median check took 325 milliseconds across 48 timed calls. That does not include the full camera-to-tablet path or tell us how fast the Jetson will run.

Schedule status

The report and bench diagnostics progressed, but bench accuracy is still below our target. We have not shown a reliable full five-minute bench event or run the workflow on the Jetson. I need to focus on the failures that affect the complete station state, then check changes on separate footage.

Deliverables for next week

  • Review the remaining person-to-bench and bar-attachment errors against the recorded frames.
  • Test a focused change on separate footage and keep the current baseline for comparison.
  • Collect or prepare approved bare, loaded, and return scenarios, including a full unattended period.
  • Agree on the shared output format and help connect the bench workflow to the tablet service.
  • Run the bench workflow on the Jetson when the board setup is ready.

Posted by Harry since Jerry doesn’t have his laptop right now

Harry’s Status Report For 10/3

Accomplished

Jerry and I filmed the videos for all three use cases together. We planned the recordings to cover the main states and as many edge cases as we could. Our four squat-rack clips show an empty rack, someone exercising with belongings nearby, someone leaving those belongings, and someone returning to collect them.

I built the squat-rack workflow in two phases. During setup, models find the rack and propose areas for exercise, nearby attendance, and belongings beside or in front of it. I built a region editor so a gym staff member setting things up for the first time can adjust those areas or exclude a neighboring machine.

During detection, the system checks video twice per second for people and belongings. It reports whether the rack is available, in use, waiting after someone leaves items, or unknown because the evidence is insufficient. The five-minute review rule is implemented, but our recordings are too short to demonstrate a real alert.

On the four development clips, 54 of 63 scored rack-state observations were correct and nine were unknown. Belongings presence was correct in 52 of 64 observations, with 12 unknown. I compared two belongings models on my laptop. The default took a median of 376 milliseconds per check. A faster alternative took 92 milliseconds but missed more items. I also separated the models needed only during setup from those needed on every detection check. This helps us plan what may run at the same time on the Jetson, although these are laptop measurements rather than Jetson results. The accuracy results come from one camera view and labels prepared during development.

Progress

The squat-rack workflow runs on my laptop, and the team now has separate laptop workflows for all three use cases. The Jetson baseline planned for the week of September 28 is behind schedule. I have not run my workflow on the board or tested three feeds together. We will first agree on a shared output format and measure each workflow on the Jetson, then try them together before deciding whether scheduling changes are needed.

Deliverables

  • Contribute the squat-rack implementation, results, and limitations to the October 9 design report.
  • Record longer rack scenarios with bags, headphones, passersby, people behind the rack, and a full unattended period. Keep some footage for a separate evaluation and have a teammate review a sample of the labels.
  • Work toward a three-feed Jetson test, measuring processing time and memory for each workflow before measuring them together.

Harry’s Status Report For 9/26

Accomplished

Most of my week went into working through the professor and TA feedback from our proposal presentation. To summarize, everyone agreed that the use case was valid, but questioned whether our implementation really fit it. A gym changes slowly, so we needed a better explanation of why we need models and a scheduler, how many cameras we actually need, and where our requirement numbers come from. These questions affect the cameras, parts, and software we would build, so I wanted to work through them before we went further with the design review.

I compared two proposed implementations: one where each camera covers a group of equipment, and another where we have one camera per machine. I estimated the UC gym to have around 10 treadmills, 6 squat racks, and 6 benches, and used that scale to compare setup, power, wiring, bandwidth, and processing needs. I calculated the dedicated-camera workload and put together a small script to see how sampling rate and processing time affect queue delays. These are estimates, and still need to be checked on the Jetson. I currently favor shared cameras covering small groups. A camera per machine would require 22 mounts, power connections, and feeds to maintain. Shared cameras reduce cost and wiring, but require confirming machine locations during setup. Camera count would depend on clear coverage, with separate machine groups to avoid cross-camera matching and double counting.

I also helped check the design presentation against the assignment guidance and worked on explaining the quantitative requirements. One thing the feedback pointed out was the difference between what the user needs and what each part of our system has to do to meet that need. For example, checking once per second and waiting for three consecutive observations takes up to three seconds after a change. Allowing another second for processing and updating the display gives a four-second budget within our proposed five-second target. That assumes the observations are fresh and correct, so it still needs testing. I also worked through what missing our accuracy targets would mean, like showing a treadmill as free when it is occupied or sending staff to check a station unnecessarily. We are still refining those justifications.

To get a better idea of whether shared-camera setup could work, I tried some models (Grounding DINO and SAM 2) on three treadmill photos. The straight-on photo gave five detections matching the five visible machines, including the one with a person on it. The diagonal view was messier, especially for distant machines and reflections, and some masks picked up parts of neighboring treadmills. This helped us think through camera placement and how much setup would be needed. It does not tell us how accurately we can detect whether a machine is in use. See below:

Progress

Behind on the parts order and the capture prototype. Our original schedule had parts ordering and the laptop prototype finished by September 25, with Jetson setup planned for September 28–October 9. I held off on ordering while we reconsidered the implementation, since one camera per machine would change a lot of what we needed. We had to sort out the general direction because it affected the whole team, but I recognize the order is now late. I will check the selected setup with the team, confirm what we can borrow, and submit the missing parts request as soon as possible. The laptop can now run models on still images, but I have not finished the recorded-video pipeline I planned last week. I will work on that while we wait for hardware.

Deliverables

  • Finish helping with the requirement explanations and design slides due September 27.
  • Update the parts list for the setup we plan to build, check inventory, and submit the order and Jetson loan request.
  • Get one recorded feed running through capture and inference, keeping timestamps and measuring processing time, memory, and how old each processed frame is. Repeat on the Jetson once we have it.
  • Agree with Jerry on the observation format and timestamps so my capture pipeline can connect to his application. Keep the results and remaining issues documented for the October 9 design report.

Harry’s Status Report for 9/19

Accomplished

Most of my week went into the parts list. I put together a bill of materials covering the Jetson Orin Nano Super, three USB cameras, storage, mounts, and the cables and adapters we need to actually get it running, and checked how the pieces fit together. A few things came out of that. The dev kit comes with a power supply and a WiFi module already installed, but no storage at all, so a card or drive has to be ordered with it. Its video output is DisplayPort rather than HDMI, so we need an adapter for whatever display we end up using. We also planned for a powered hub so that three cameras plus a keyboard and mouse are not all drawing from the board, and for active USB extensions, since the cameras need to sit well away from it and plain USB cables stop working past a few meters. I marked which items are worth checking the course inventory for before we order anything. I also started getting a development environment going on my laptop so the pipeline work is not blocked on hardware.

Progress

On schedule. The parts list was this week’s task and it is done. The laptop environment is started rather than finished, and I will wrap that up Sunday.

Deliverables

Submit the parts order and ask course staff about borrowing a Jetson in the meantime. Get the laptop prototype running a detector on recorded video with timing. Help write the design presentation slides due 9/27.

Team’s Status Report for 9/19

The most significant risk right now is parts lead time. Purchasing opened after the proposal presentations, so this week we finalized our parts list and we plan to submit the order early next week. Nearly all of the hardware work depends on the Jetson arriving, so a late delivery would push back setup, camera capture, and integration together. Fall break is already marked as slack on our schedule, so a delay of a week or two can be absorbed.

If the Jetson does take a while, we can keep moving by prototyping on a laptop. One of us has a Windows laptop with an NVIDIA GPU, so the detection pipeline and the application logic can be built and tested there in the meantime, and it also gives us a machine to fine tune the plate model on later. The code carries over, but the environment does not entirely. The Jetson runs a different processor architecture, so the builds and installed libraries are different, and models have to be converted on the board itself. Speed and memory also behave differently, since the laptop is much faster and its graphics memory is separate from system memory, while the Jetson shares one pool between the two. So anything we measure about timing, memory, or scheduling still has to be measured on the Jetson.

The second risk is getting permission to record at a campus gym, since we need real footage to check that detection works from our camera angles and to build a test set later. We started that request this week and will send it early next week. If it does not come through, we can record at another gym with permission, and we can still test most of the logic on a mock setup.

The third risk is that the pretrained detector does not know what a weight plate is. The model we plan to start from covers people, backpacks, handbags, and bottles, but plates are not one of its classes, so the bench check will need training on our own labeled images. We are treating that as likely rather than optional, and the time to collect and label those images is already in our schedule.

No changes were made to the design this week. The three checks, the timing targets, and the hardware plan are the same as what we presented. No changes to the schedule either.

Outside of the technical work, we set up how we work together. We created a shared GitHub repository holding our proposal, slides, course guidance, and design notes, and turned on branch protection so nothing goes onto the main branch directly. We review each other’s work through pull requests instead, which already helped catch a few things in the proposal before we submitted it. We also presented the proposal on Monday and completed peer reviews on Monday and Wednesday.