Category: Harry’s Status Reports

Accomplished

Most of my time this week went into the design report. I worked on the squat-rack sections, the design tradeoffs and helped formalize the use case / design requirements, as well as the testing and verification. I also helped make sure the design presentation feedback was fully incorporated into the report.

I also tested whether image subtraction could replace the belongings detector. On 29 scored observations, it got 25 right but made four false calls after someone collected their items. The person was still in the belongings area, and image subtraction detected that visual change without knowing it was a person rather than an item. The detector got 19 right, made no wrong calls, and reported ten as unknown. Image subtraction gave more definite answers, but these false calls mean I would not replace the detector based on this test. I added the results to the report. The subtraction threshold was chosen using the same clips, so it still needs testing on separate footage.

Progress

The rack workflow still runs on recorded video on my laptop. I did not run it on the Jetson or collect the longer rack scenarios this week because the report took priority. Those are the main gaps now. My planned first rack run on the Jetson is October 21.

Deliverables

• Run the rack workflow on the Jetson and record check time, memory use, and failures.
• Record longer rack cases with different belongings, passersby, and a full unattended period, once recording permission is settled.
• Keep some footage separate from development and ask a teammate to review a sample of the labels.
• Work with Jerry and Alan on the shared station output and the first combined test.

Accomplished

Jerry and I filmed the videos for all three use cases together. We planned the recordings to cover the main states and as many edge cases as we could. Our four squat-rack clips show an empty rack, someone exercising with belongings nearby, someone leaving those belongings, and someone returning to collect them.

I built the squat-rack workflow in two phases. During setup, models find the rack and propose areas for exercise, nearby attendance, and belongings beside or in front of it. I built a region editor so a gym staff member setting things up for the first time can adjust those areas or exclude a neighboring machine.

During detection, the system checks video twice per second for people and belongings. It reports whether the rack is available, in use, waiting after someone leaves items, or unknown because the evidence is insufficient. The five-minute review rule is implemented, but our recordings are too short to demonstrate a real alert.

On the four development clips, 54 of 63 scored rack-state observations were correct and nine were unknown. Belongings presence was correct in 52 of 64 observations, with 12 unknown. I compared two belongings models on my laptop. The default took a median of 376 milliseconds per check. A faster alternative took 92 milliseconds but missed more items. I also separated the models needed only during setup from those needed on every detection check. This helps us plan what may run at the same time on the Jetson, although these are laptop measurements rather than Jetson results. The accuracy results come from one camera view and labels prepared during development.

Progress

The squat-rack workflow runs on my laptop, and the team now has separate laptop workflows for all three use cases. The Jetson baseline planned for the week of September 28 is behind schedule. I have not run my workflow on the board or tested three feeds together. We will first agree on a shared output format and measure each workflow on the Jetson, then try them together before deciding whether scheduling changes are needed.

Deliverables

• Contribute the squat-rack implementation, results, and limitations to the October 9 design report.
• Record longer rack scenarios with bags, headphones, passersby, people behind the rack, and a full unattended period. Keep some footage for a separate evaluation and have a teammate review a sample of the labels.
• Work toward a three-feed Jetson test, measuring processing time and memory for each workflow before measuring them together.

Accomplished

Most of my week went into working through the professor and TA feedback from our proposal presentation. To summarize, everyone agreed that the use case was valid, but questioned whether our implementation really fit it. A gym changes slowly, so we needed a better explanation of why we need models and a scheduler, how many cameras we actually need, and where our requirement numbers come from. These questions affect the cameras, parts, and software we would build, so I wanted to work through them before we went further with the design review.

I compared two proposed implementations: one where each camera covers a group of equipment, and another where we have one camera per machine. I estimated the UC gym to have around 10 treadmills, 6 squat racks, and 6 benches, and used that scale to compare setup, power, wiring, bandwidth, and processing needs. I calculated the dedicated-camera workload and put together a small script to see how sampling rate and processing time affect queue delays. These are estimates, and still need to be checked on the Jetson. I currently favor shared cameras covering small groups. A camera per machine would require 22 mounts, power connections, and feeds to maintain. Shared cameras reduce cost and wiring, but require confirming machine locations during setup. Camera count would depend on clear coverage, with separate machine groups to avoid cross-camera matching and double counting.

I also helped check the design presentation against the assignment guidance and worked on explaining the quantitative requirements. One thing the feedback pointed out was the difference between what the user needs and what each part of our system has to do to meet that need. For example, checking once per second and waiting for three consecutive observations takes up to three seconds after a change. Allowing another second for processing and updating the display gives a four-second budget within our proposed five-second target. That assumes the observations are fresh and correct, so it still needs testing. I also worked through what missing our accuracy targets would mean, like showing a treadmill as free when it is occupied or sending staff to check a station unnecessarily. We are still refining those justifications.

To get a better idea of whether shared-camera setup could work, I tried some models (Grounding DINO and SAM 2) on three treadmill photos. The straight-on photo gave five detections matching the five visible machines, including the one with a person on it. The diagonal view was messier, especially for distant machines and reflections, and some masks picked up parts of neighboring treadmills. This helped us think through camera placement and how much setup would be needed. It does not tell us how accurately we can detect whether a machine is in use. See below:

Progress

Behind on the parts order and the capture prototype. Our original schedule had parts ordering and the laptop prototype finished by September 25, with Jetson setup planned for September 28–October 9. I held off on ordering while we reconsidered the implementation, since one camera per machine would change a lot of what we needed. We had to sort out the general direction because it affected the whole team, but I recognize the order is now late. I will check the selected setup with the team, confirm what we can borrow, and submit the missing parts request as soon as possible. The laptop can now run models on still images, but I have not finished the recorded-video pipeline I planned last week. I will work on that while we wait for hardware.

Deliverables

• Finish helping with the requirement explanations and design slides due September 27.
• Update the parts list for the setup we plan to build, check inventory, and submit the order and Jetson loan request.
• Get one recorded feed running through capture and inference, keeping timestamps and measuring processing time, memory, and how old each processed frame is. Repeat on the Jetson once we have it.
• Agree with Jerry on the observation format and timestamps so my capture pipeline can connect to his application. Keep the results and remaining issues documented for the October 9 design report.

Accomplished

Most of my week went into the parts list. I put together a bill of materials covering the Jetson Orin Nano Super, three USB cameras, storage, mounts, and the cables and adapters we need to actually get it running, and checked how the pieces fit together. A few things came out of that. The dev kit comes with a power supply and a WiFi module already installed, but no storage at all, so a card or drive has to be ordered with it. Its video output is DisplayPort rather than HDMI, so we need an adapter for whatever display we end up using. We also planned for a powered hub so that three cameras plus a keyboard and mouse are not all drawing from the board, and for active USB extensions, since the cameras need to sit well away from it and plain USB cables stop working past a few meters. I marked which items are worth checking the course inventory for before we order anything. I also started getting a development environment going on my laptop so the pipeline work is not blocked on hardware.

Progress

On schedule. The parts list was this week’s task and it is done. The laptop environment is started rather than finished, and I will wrap that up Sunday.

Deliverables

Submit the parts order and ask course staff about borrowing a Jetson in the meantime. Get the laptop prototype running a detector on recorded video with timing. Help write the design presentation slides due 9/27.