Posted on

Paul Wright Status Report for 2/21/26

What I have personally accomplished this week/schedule changes:

To be completely honest, I have done little this week outside of KR260 setup and working with Sean and Justin for localization alternatives. You can see his post for more about those alternatives. Over the past week, I have had to deal with an emergency in my personal life that has taken me away from anything CMU. It is now fully resolved, and I expect to get back on track. I communicated this with my teammates, and I am thankful for their recognition of and willingness to adjust to this disruption. I plan to spend significant time over spring break to make up for this week of lost work.

What deliverables to complete in the next week:

Complete KR260 setup and necessary installations.

Complete design report alongside teammates.

Continue with software setup and coding for odometry.

Posted on

Sean Johnson Status Report for 2/21/26

What did you personally accomplish this week on the project?

This week I focused on investigating alternative localization approaches after our professor had skepticism about relying solely on the optical flow sensor (PAA5160) for accurate navigation. We received feedback that odometry localization may not be reliable enough for our accuracy requirements. The main alternative I explored was marker systems, specifically ArUco markers, AprilTags, and QR codes that could be placed at known locations throughout the building hallway. The idea is that our existing camera could detect these markers and use them to reset or correct position estimates during navigation, compensating for any drift accumulated by the optical flow sensor.

I evaluated several sensor fusion architectures that could combine multiple localization inputs. The most promising approach uses the odometry sensor for continuous tracking between landmarks and visual fiducial markers for periodic position correction. This hybrid system would leverage the strengths of each component while mitigating their individual weaknesses. For implementation I researched how the OpenCV ArUco detection library could integrate with our existing camera hardware and CV pipeline. A hybrid approach using optical odometry for continuous tracking with periodic visual landmark corrections could achieve our position error requirements while being significantly more robust than odometry alone. ArUco markers can be detected in real time with our YOLOv3-tiny human detection without creating a significant performance bottleneck.

Is your progress on schedule or behind?

According to our Gantt chart, Task 3.1 should be complete by the end of Week 2. While I’ve made progress researching the optical sensor and understanding its capabilities, the professor’s feedback revealed that we need a different solution. This puts me a few days behind on finalizing our localization approach since we now need to integrate additional components we hadn’t initially budgeted time for. Our team will make a decision on our approach very soon, whether that’s pure odometry or odometry with visual landmarks.

What deliverables do you hope to complete in the next week?

My primary deliverable is getting the optical flow sensor operational with working code. I need to write the interface code to communicate with the PAA5160 sensor and pull X/Y displacement readings. This will be implemented in Python interfacing through the Arduino. Once I have raw sensor data coming through, I’ll implement a basic position tracking algorithm that integrates the displacement values over time to estimate the robot’s current position.

Posted on

Paul Wright Status Report

What I personally accomplished this week:

 

This week, I continued work with Justin around the robotics kit and chassis, where we decided on the Hiwonder Tank Chassis Kit. After we settled on the kit, I  began looking at first and 3rd party projects that used the kit. My math for the battery size needed lines up with what they needed for 1 hour of operation: 2200mah. 

Last week’s outstanding deliverable was finding a battery for the vehicle. I have finished that. We will be using 2 separate batteries: a 7.4V one for the Motor and a 12V one for the KR260 and Arduino. I also added an H-bridge to the design, which is a rather embarrassing oversight.

With the completed parts list, I began on the CAD model of the physical design. I hoped to be complete with it to remain on schedule, but the chassis has a surprisingly sparse amount of 3-dimensional information. A lot more additional work has had to be done to be able to properly represent it.

I also spent time learning about the KR260’s software stack and how we are going to integrate everything. I have no experience with ROS2, and am trying to get myself up to speed. I found these to be very useful for that.

https://youtu.be/0aPbWsyENA8?si=MZzz1h8PobvCQmXA

https://youtu.be/Se5pvRlTX8s?si=UA6VJxq1sKDpnBYE

Finally, I spent time learning about setup and configuration of our odometry sensor and how to use the data it gives so that we can properly integrate it into our whole system.

 

Progress relative to schedule:

Slightly behind schedule in terms of completion of CAD model of system, but just by a small amount. 

 

Deliverables for next week:

  1. Begin setup of KR260 to match Justin’s once we get it from ECE inventory.
  2. Complete CAD design.
  3. Order parts
  4. Continue with software setup and coding for odometry.
Posted on

Team Status Report for 2/14/26

 

Our project has received a change in use case upon further discussion with faculty and evaluation of parts within our budget. We are switching to delivery from last-mile delivery to delivery of small objects within a building. This removes the need to read and identify signs, and reduces the number of potential obstacles. In addition, the more consistent lighting and surfaces will help our camera and odometry system to function better. 

Last week, we brought up the depth-sensing cameras. We were notified that they were mistakenly marked as available on the inventory sheet, so we will not be going farther down that route and will instead be continuing with bounding boxes.

Risk 1:  With the RTL pipeline this week a risk that can happen is that right after the last chunk is said to be done on a convolutional layer the next chunk can get processed. This could lead to the Multipliers and adder tree not being done and starting another layer messing up the actual accumulated value.

Risk 2: Proper balance of vehicle with multiple batteries: We identified the need for multiple batteries for the KR260, motor systems and arduino. With limited space on the vehicle chassis, we need to align them so that we do not have an unbalanced vehicle that, while functional, will make accurate movement and turning more difficult to implement.

 

Part A: Written by Paul

When considering public health and safety in our use case and design, our primary focus is not to be a hazard to people who go through hallways. This is emphasized in two of our primary design requirements of being able to identify and change navigation when a person is within 2 meters under 50ms and to be able to stop within 0.5m. The robot needs to react to and navigate around people quickly enough that the person will not have to alter their path. This is especially important for those with movement impairments or sensory disabilities. For those without them, a robot getting in the way just means stepping out of the way or slightly changing tour path, an inconvenience that rarely would cause a real safety hazard. However, for those with impairments, quickly changing paths or stopping is a much taller order that can risk their safety, and for those with sensory impairments, they may only detect the vehicle as it is crashing into them.  

 

Aside from our design goals, our focus on public health and safety is exemplified in our curation of training data. We are placing special focus on using data that includes those who are holding objects that obscure parts of them and who are in wheelchairs. Focusing on those with impairments and disabilities also helps those who do not have them, as they can temporarily have them. Consider a maintenance worker who is holding a large box in front of them. They are partially vision impaired at that time, and should not have to worry about a small robotic car that is out of their field of vision.

 

Part B: Written by Sean

Our autonomous indoor delivery robot is designed with careful consideration of social factors, particularly its operation in shared indoor pedestrian spaces where diverse social groups are present. By prioritizing people detection and obstacle avoidance, our system respects the fundamental social principle that indoor corridors, lobbies, and pathways are primarily human spaces. Users including children, elderly individuals, people with disabilities, and those carrying packages or using mobility aids should feel safe and comfortable around our product. Our focus is on maintaining appropriate distance from pedestrians and stopping when people are detected which is critical for settings like university campuses, hospitals, office buildings, and residential complexes. Predictable behavior addresses social concerns about robots operating in confined spaces where surprise encounters could cause discomfort particularly among vulnerable populations like elderly residents or young children. Our design acknowledges the importance of prioritizing human presence over delivery efficiency. Our solution aims to gain community acceptance by operating as a considerate “neighbor” rather than prioritizing speed. 

 

Part C:  Written by Justin

When considering economic factors, our product is designed to be a practical and efficient solution for real time video processing on low power hardware. The KR260 based computer vision system delivers fast inference while consuming significantly less power than a traditional GPU and providing far more capability than a typical microcontroller. This balance between performance and energy efficiency reduces operational costs through lower power consumption, longer battery life, and reduced cooling requirements. For delivery robots that must react quickly to obstacles, this efficiency translates directly into both safety improvements and long term cost savings.

From a production and distribution standpoint, the system relies primarily on commercially available, off the shelf components rather than custom manufactured hardware. This reduces upfront development costs, minimizes supply chain risk, and allows for easier scaling. The main bottleneck would likely be shipping and lead times rather than specialized fabrication. Because the design is modular, components can be replaced individually, lowering maintenance costs and extending product lifespan.

Posted on

Sean Johnson Status Report for 2/14/26

What I personally accomplished this week:

This week I focused on refining our position tracking approach with the PAA5160E1 optical sensor. I discovered that the sensor uses I2C communication, not SPI as initially thought. This simplifies our integration, the sensor will connect to the Arduino rather than directly to the FPGA, since it has an onboard STM32 microcontroller that outputs processed position data over I2C. I reviewed the SparkFun documentation and Arduino library for the PAA5160E1 to understand the integration workflow. The sensor provides X/Y displacement values that we can accumulate into absolute position. I studied how to read position data via I2C on Arduino and forward it to the KR260 over serial communication for use in navigation decisions.

I also researched how to calibrate the sensor and handle edge cases like sensor drift and surface compatibility. The sensor requires mounting 10-27mm above the tracking surface and works best on hard floors, which matches our indoor demo environment.

I continued learning ROS2, focusing on how position data flows through navigation nodes and how to structure communication between the Arduino, handling sensors/motors, and KR260.

I coordinated with Paul on mounting requirements for the optical sensor on our chassis design.

Progress relative to schedule:

On schedule. This week’s work on optical sensor integration planning aligns with Gantt chart task 3.1 (optical sensor) and 3.4 (position tracking).

Deliverables for next week:

  • Complete integration plan for PAA5160E1. Arduino I2C interface and serial communication to KR260
  • Begin testing basic I2C communication with Arduino if sensor arrives
  • Define data format for position updates sent from Arduino to KR260
  • Continue ROS2 learning focused on position/odometry message types
Posted on

Justin Bell Status Report for 2/14/26

This week I revisited the convolutional layer in my YOLOv3-tiny hardware implementation and realized that the original conv_2d module was not properly parametrizable for the full network. It was too tightly coupled to a specific layer configuration, particularly around fixed kernel size, channel count, and accumulation structure. That meant it would not scale cleanly across the different convolutional layers in YOLOv3-tiny.

I restructured the conv_2d module to be fully parameter driven. I introduced parameters for kernel size, number of input channels, number of output channels, and maximum supported input channels. The data path is now dimensioned based on these parameters rather than hard-coded constants. Internally, I redesigned the MAC array so that instead of supporting only 27 multipliers tied to a specific 3x3x3 style structure, it now instantiates 64 parallel multipliers. This gives more parallelism and aligns better with a scalable channel processing strategy.

I also added a dedicated accumulator inside the MAC unit. Previously, partial sums were not handled in a way that would scale across large input channel counts. Now, each output feature map element accumulates products across all input channels in a structured way. The accumulator supports up to 1024 input channels. For layers that have fewer than 1024 input channels, I zero-pad the unused inputs so that the accumulation loop and hardware structure remain consistent. This allows the same hardware structure to be reused across layers with different channel depths without redesigning the control logic.

Additionally, I updated the control logic so the same conv_2d module can handle both 1×1 and 3×3 convolutions. The kernel size parameter determines how the sliding window and multiply scheduling behave. The internal indexing and loop bounds now depend on the parameterized kernel size and channel counts, so the same module can be reused across all convolutional layers in YOLOv3-tiny.

Right now I am validating all 1×1 and 3×3 MAC operations in simulation without quantization or post processing. The goal is to ensure that the core multiply accumulate behavior matches a reference implementation before introducing fixed point quantization and activation functions.

On the project direction side, based on feedback, we decided to pivot away from traffic sign detection and instead focus on making the robot more careful around people. This allows us to use the standard COCO dataset with YOLOv3-tiny instead of building and training a custom dataset. That reduces training overhead and lets us focus on hardware deployment. I will still need to properly quantize the pretrained model for the FPGA board, but this avoids retraining from scratch.

Goals for next week:

  • Have most of the model working functionally in simulation

  • Implement a Python reference version so I can verify outputs and prepare for weight loading

  • Purchase the remaining hardware components

  • Prepare slides and demos for the presentation

Posted on

Paul Wright Status Report for 2/7/26

What I personally accomplished this week:

This week, I coordinated with justin about the robotics kit/chassis, optical sensor solution, and delved farther into potential depth implementations. As Justin’s report said, we have looked at multiple different chassis.  Our current frontrunner is linked below, with our only potential reason for using it either being power consumption or mounting other components, which require more work on my end. I have yet to find any evidence that treads would cause enough vibrations to disrupt proper odometry.

Hiwonder Tank Car Chassis Kit Shock Absorbing Robot with DC Geared Motor for Arduino/ Raspberry Pi/ Jetson Nano DIY Robotic Car Learning Kit (Black)

We also decided on our optical sensor solution, the Sparkfun PAA5160E1. Below is a brief showcase of it. After reading datasheets and looking at projects that have used it, I believe that it will work for our positioning needs, given that we are using flat surfaces.

As the ECE parts list opened up, I identified potential alternate solutions for distance estimation. The most desirable one is the Intel Realsense Depth Camera D455. It would provide much more accurate depth than our currently planned bounding box estimation. However, we are worried about CPU overhead, as it would mean that both a camera feed and depth map are being input, so this would only be implemented if we have the overhead.  Below is a video that I found helpful for understanding how we would use it.

The second potential depth-sensing camera is the eYs3D Stereo Camera – EX8036. It appears to be a slightly worse option than the Realsense in terms of actual hardware and ease of integration.

Finally, a couple of hours were dedicated to rehearsing the Proposal Presentation.

Progress relative to schedule:

I feel that I am on schedule for everything except for deciding on our battery to meet our design requirement of the system lasting for more than 30 minutes. I am behind on this and expect to address this by finishing it on 2/8/26.

Deliverables for next week:

1.Finish finding battery for vehicle.

2. Investigate any potential issues with part selections to confirm choices.

3. Create a basic CAD model of entire vehicle

4. Begin preliminary work of software construction for path creation.

Posted on

Justin Bell Status Report 2/7/2026

I’ve got the CONV 3×3 implementation working with INT8, which is a solid milestone for the hardware pipeline and gets us closer to the final architecture we need for real-time inference. Right now I’m focusing on making the whole thing parametizable so we can easily adjust configurations without having to rewrite a bunch of code every time we want to change kernel sizes, data widths, or number of channels. It’s coming along pretty well and the basic structure is there, but there’s still some refactoring to do to make it clean and flexible enough to handle different use cases down the line. The goal is to have something that’s modular enough that we can swap in different layer configurations without breaking everything, which will be really helpful once we start optimizing for different parts of the YOLO network.

On the model side, I’ve been spending most of my time training with traffic signs datasets to get the detector tuned properly for our specific application and use case. The goal is to get good accuracy on sign detection specifically since that’s what we’re targeting for the real-time demo on the Kria board, and we want to make sure it actually works well in real-world conditions.

I’m also deep in the research phase for quantization strategies, which is pretty critical for getting this running efficiently on the FPGA without sacrificing too much accuracy. Trying to figure out the best approach to get from floating point weights and activations down to INT8 without totally tanking the accuracy or introducing too much quantization error that would mess up our detection performance. There are a few different techniques I’m looking into – post-training quantization which is simpler but might lose more accuracy, quantization-aware training which should preserve accuracy better but takes more time, and some hybrid approaches that try to get the best of both worlds. I’m still working through the tradeoffs and running experiments to figure out what makes the most sense for our FPGA implementation and performance requirements, especially considering we need to hit that sub-30ms inference latency target while maintaining reasonable detection accuracy on traffic signs.

Posted on

Team Status Report for 2/7/26

Our project has received a significant shakeup over the last week. Last Sunday, we put a significant amount of additional work into our proposal presentation compared to when we had finished before we wrote the last report. This resulted in a shakeup of both our schedule and roles for the 3 of us. We consider the proposal presentation to be of great value, as it forced us to think a lot more about what we are doing. Our new schedule for the individual portions of the project is in the project proposal gantt chart.

In addition, the inventory being shared on Wednesday changed some of our part and depth-sensing plans. Two depth-sensing cameras are available, which may reduce our need for additional distance sensing hardware.

Risk 1: Hardware component availability and integration
 We’re currently evaluating robotics chassis options to ensure adequate payload capacity, 2-3kg and platform size for the KR260. We are mitigating this by researching multiple vendors and chassis types.

Risk 2: Model training timeline and FPGA resource constraints
 Training YOLOv3-tiny and fitting quantized weights on FPGA could take longer than anticipated. Mitigate this risk by starting model training this week ahead of schedule. The back up plan is to use pretrained weights if custom training fails. fall back to Vitis AI if custom RTL integration proves too difficult.

Risk 3: CPU processing constraints

We are requiring the CPU on the KR260 to convert frames into 1D arrays, handle robotics control logic, and perform postprocessing of frames while evaluating depth. We will mitigate this risk by starting with simply bounding boxes for our distance measurements. If we have the processing overhead, we would then use a depth camera system for more accurate distance measurements. In the absolute worst case, we would need to offload some of the CPU responsibilities to a separate board, which would introduce difficulties in making the two boards coordinate.

No photos yet as we’re in the planning phase, but hardware selection will be finalized by next week.

 

Posted on

Sean’s Status Report for 2/7/26

What I personally accomplished this week:

This week I focused on hardware selection for our robot platform. I researched and evaluated multiple robotics chassis options including the Hiwonder MiniAuto kit, 4WD platforms, and tank chassis systems. I determined we need a platform with at least 250mm x 200mm dimensions and 2-3kg payload capacity to support the KR260, sensors, camera, battery, and Arduino.

I also began researching the PAA5160 optical flow sensor for position tracking, reviewing its datasheet and SPI communication protocol. Researching integration requirements and how to convert sensor data into position information.

Additionally, I started learning ROS2 fundamentals which we’ll use to integrate our vision, positioning, and motor control systems. I coordinated with Paul on physical design considerations.

Progress relative to schedule:

On schedule. Week 1’s proposal work is complete, and this week’s hardware research aligns with our Gantt chart tasks for Week 2.

Deliverables for next week:

  • Finalize robotics chassis selection with team
  • Contribute to CAD layout planning for component placement
  • Develop position tracking integration plan, encoder based or optical sensor
  • Continue ROS2 learning focused on navigation and motor control