Here is the final poster for the TailorBot project:
Team Status Report for 4/27/24
Risks:
The biggest risk lies in fine tuning our design in time for the final demo. The main way to fix this is to meet up over the course of the week and test all the parts of the system very rigorously.
Another issue is that a more powerful motor that we ordered has not arrived yet. Our system still works end to end, so in the worst case scenario it does not arrive, we will continue to use our current motors and accommodate accordingly.
Design Changes:
We haven’t made any major changes to the design.
Schedule Changes and Updates:
No major schedule updates.
Tests and Analysis:
Unit Tests Conducted:
Room Dimension Accuracy:
- Procedure: We scanned the room and compared the model’s width and depth with actual measurements obtained using a tape measure.
- Findings: The current model had a measurement error of approximately 150 mm, which exceeds our goal of less than 80 mm error.
Z-Axis Accuracy:
- Procedure: We measured the distance from the sensor to the floor and compared it with the height reading provided by the sensor.
- Findings: The Z-axis positioning accuracy was within 22 mm error, which is under our performance goal of less than 30 mm error.
Web App Performance:
- Procedure: We measured the response time from scan completion to the model display in the web application.
- Findings: The average response time was around 10 seconds per 6 square meters, well under the maximum allowable time of one minute.
System Test:
- Procedure: We conducted a full system integration test to evaluate how well the different components of the system work together in real-time. This test was crucial to ensure that the data flow from the LiDAR scan, through processing and up to display in the web application, was seamless and efficient.
- Findings: The system performed well during the integration test. However, during the scanning phase that involved using a stepper motor to elevate the scanning platform, we observed that the platform’s upward movement was considerably slower than expected. This slow movement could impact the overall system efficiency. There was also an unintended slight tilt forward that was occurring due to an imbalance in the weight. To accommodate this, we attached another small counterweight to the back end of the stand to level out the platform.
Findings and Design Changes:
- Room Dimension Accuracy: We are trying to improve the dimensional accuracy by adjusting parameters for generating the model. We are also trying to achieve this by varying the z-axis start and end position of the scan, as we have found that with our new tilt mechanism, the start position of the scan can cause unintended readings from the floor.
- Platform Speed Adjustment: To address the slow movement of the stepper motor-driven platform, we are testing with different models of stepper motors.
Developments:
We’re in the endgame now. We’ll be finetuning everything until demo comes.
Alana’s Status Report for 4/27/24
Personal Accomplishments:
There isn’t much left to do. I spent most of my time fixing small bugs and adding additional tweaks and upgrades to the web app. Any major refinements would involve changing how the model generation creates and interacts with the web app. Model generation is more Grace’s area and my computer has issues with the software so I can’t really run it on my own. That doesn’t stop the group from meeting up to test things but it does slow down the progress I can make on my own for really nitty gritty details. Nonetheless, I will continue to make improvements.
Progress:
Seeing as the only goal for the group is to keep testing until the final demo, I’d say that I’m on time. I’ll continue to test things out with the others.
Schedule Status: On time.
Next Week’s Deliverables:
With the final demo in sight, my goal is to make the web app the best it can be.
Team Status Report for 4/20/24
Risks:
The biggest risk is getting the stepper motors to work currently for the demo. While all the mechanisms are in place for it to work, the main issue is that the original motors we had didn’t have a high enough torque force to properly move the scanner up and down. We’ve already tried fixing this issue by getting stepper motors with a higher torque force and reorienting the design of the lift to take this into account.
Design Changes:
We plan to upgrade lift’s stepper motor to a higher torque model to improve the lift’s speed and consistency during scans. The motor with more torque will move the lift faster and more reliably, reducing the total scan time. This change resulted in the additional costs of ordering a new stepper motor, shaft coupler, and power supply.
Schedule Changes and Updates:
No major schedule updates.
Developments:
We are in the endgame for the project. Everything we’ve been doing so far has just been getting everything to work together as smoothly as possible. We’re just in the cycle of testing, fixing problems, refining, and testing again until the final demo comes.
Alana’s Status Report for 4/20/24
Personal Accomplishments:
A problem we found with the project is that while we could load in a clean room model pretty consistently, getting clean models of the individual furniture models was a lot trickier as our scanner could detect distance but lacked a depth field. To counteract this, I implemented a system that loads in a room model and replaces the messy furniture models with template furniture models (like the ones you see below) and place them in the appropriate furniture locations.
There were a couple of problems along the way. The first problem was that Javascript doesn’t have a lot of good tools to play around with its file system. What is there is good enough when the files are static, but not when the files change dynamically. Since the scanner would be sending new files to the web app as it scanned a room, this was a problem. I looked into a lot of different ways to get Javascript to do what I wanted but ultimately, I managed to get it done with the help of django. Since django already takes care of running the code that creates and sends the files to the web app, we just had it send a json file that includes the position and object type for each piece of furniture detected in the room. Javascript had a much easier time reading from a json file rather than the other methods I had tried. Getting the position of the furniture was tricky due to the mentioned problem of our scanner’s lack of depth perception but Grace was able to get position data into the file.
Aside from that, I added in a couple more tweaks like some instructions on how to use the site and fixed a few bugs.
Progress:
I managed to complete my goal from last week and we are approaching the endgame for this project. The only thing left to do on the web app side of things is more testing to make sure the models are displayed properly when scanning different types of rooms and fix any of the potential bugs that may arise.
Schedule Status: On time.
What I learned during the project:
Starting this project, I already had a good amount of experience with web app design and React. However, something I had to learn a lot about during the project was how to display 3D models in a web app. The first step was just finding a library that could do most of that for me. When it comes to web app design, there are tons of free libraries there to get specific jobs done. Low and behold, I found Three.js to load up 3D models. Three.js on its own didn’t sync up great with my React setup but then I found React Three Fiber, a free React library that uses Three.js as its base. That saved me a lot of time compared to trying to get base Three.js to work with my React setup. From there I learned how to use React Three Fiber by reading its documentation and watching Youtube tutorials on how it worked. There were instances where I couldn’t find a good Youtube tutorial for what I was trying to do so I had to use the documentation and examples of React Three Fiber projects I found online to reverse engineer the mechanics of the library to get it to do what I wanted it to do in my code. Not to mention the trips to sites like StackOverflow to see if anyone had already posted solutions to specific programming problems I ran into.
Next Week’s Deliverables:
I want to make sure that the code displays the models correctly when using different types of room scans. I’ve been testing the current code on dummy data as it’s hard to collect new data on the fly. That requires having the whole setup with scanner lift in place and that isn’t feasible to do whenever. Further testing will make sure the code works in a majority of cases.
Alana’s Status Report for 4/6/24
Personal Accomplishments:
As mentioned last week, I managed to get django to establish the connection between the lidar, rasberri pi, model generator and web application. We can now press a button in the web app and scan the room, then once that is done, enter the 3D environment to play around with the models. While the speed of the room scan relies on the speed of the physical scanner components, the ability for the web app to load the model in is less than a few seconds.
I’ve been working on getting the web app to load in room and furniture models from a models folder. I’ve been looking into JavaScript file management functions for that. Though an issue with that ties into an issue the group has a whole. Trying to figure out how to separate the furniture models from the room model. Due to our lidar’s lack of depth perception, the furniture models are baked into the wall data as large blobs that go all the way back to the wall. This makes the wall and furniture as one model instead of separate ones. For this I’ve also spent time researching ways around this. It’s difficult for me to work on the problem directly as there have been errors with installing the Open3D packages needed to make the program run on my computer. Thankfully it works on the others computers but it leaves me in a position of being unable to test anything myself. I’ve looked for a work around but none have revealed themselves to me yet.
I have also made tweaks to the code base. Fleshing out little quality of life changes like easier times moving objects around, highlights appearing around the object when you select it, labels appearing on selected objects, tweaks to the code for easier re-use, etc.
Progress:
Establishing the connection between the web app and the other systems of the project was probably the most important task for me throughout this whole thing. With my issues in getting Open3D to work on my computer, any progress I make with the furniture data issue can only be research assistance. Aside from that though, I think getting django to work has kept me on track progress -wise.
Schedule Status: On time.
Verification:
For my verification methods, I plan to clock the time it takes to for the web pages to update, the time it takes for the web app to get the room data and how long it takes for the model to appear in the 3D model viewer. I can pretty easily do this by using the web browser console and making console comments about the current time. I can just keep track of the time difference between the when I initiate the above tasks and when they are completed.
Next Week’s Deliverables:
I hope to help others find a way around the furniture problem. In the meantime, I’ll be working on getting dummy furniture files to populate the room with the room model has loaded in.
Team Status Report for 4/6/24
Risks:
So far the biggest risk to the project so far is difficulties getting the proper placement and size of furniture inside a room. This task is made difficult due to our lidar lacking a depth sensor. This results in the furniture location data being backed into the wall data as one model. We have a couple contingency plans to counter this problem. One is to make use of the room data and mathematically modify it to separate the furniture data from the wall data. Another is to make use of bounding box functions in Open3D (our 3D model generator) to get the bounds of the furniture, which could make it easier to extract the data. Another idea is to buy an additional depth sensor to work in tandem with our current setup, though this is the least appealing of the options at this stage of development.
Design Changes:
A small change was made to the lifting mechanism of the scanner. A sturdy rail guide is being built to ensure stability when the scanner is taking scans. This will allow us to implement the tilting mechanism and not interfere with the stability of the initial mechanism.
Schedule Changes and Updates:
No major schedule updates.
Plans For Verification and Validation:
Verification of Room Dimension Accuracy:
To verify the accuracy of the dimensions of the room, we plan to measure the dimensions with a tape measure. The manual measurements will be compared to the dimensions represented in the 3D mesh generated from the LiDAR scan. We will calculate the difference in measurements to assess the accuracy of the scan. We aim to establish the maximum allowable error margin based on our design requirements. To assess the reliability of our scanning process, we’ll conduct multiple scans of the same room under the same conditions to see if the dimensions reported by the system remain consistent across scans.
Verification of Furniture Classification:
To verify the model’s performance, we will use metrics such as precision, recall, and F1 score. Precision measures the model’s accuracy in identifying furniture items correctly, recall assesses the model’s ability to detect all relevant instances of furniture, and the F1 score provides a balance between precision and recall. We will run a series of accuracy tests on the furniture classification model by processing images from the validation set and comparing the model’s classification results against the true labels. This test helps quantify how often the model correctly identifies furniture items.
Developments:
Demo was this week and with it, we have established all the major connections between web app, the lidar and model generation. From here, it’s a matter of refinement and figuring out the last few bugs with our systems to make it as polished as possible for the final showcase.
Picture of the pulley for room scanning:
Team Status Report for 3/30/24
Risks:
The stability and reliability of the scissor lift are a risk because instability could result in inaccurate data collection from the lidar. A contingency plan for the scissor lift is using a pulley system to lift the lidar instead.
Another risk is furniture blocking certain areas and being unable to see behind it. This risk will be mitigated by using code to estimate the shapes and areas being blocked.
Design Changes:
We are now developing a pulley system to lift the lidar sensor. This change was necessary because we needed a contingency plan for the scissor lift. We needed to purchase a few new components, but as our budget allowed us to do so, there was not a big cost.
Schedule Changes and Updates:
No major schedule updates.
Developments:
Over the course of the week we stayed true to our word of meeting up more to discuss and work synchronous on the project. We were able to put together some backup plans for some problems (like using pulley) and discuss important steps for moving forward. Getting things in a good state for the interim demo was a big priority.
Alana’s Status Report for 3/30/24
Personal Accomplishments:
I spent this week looking at connecting the web app to the model generation code more directly. This was difficult at first because the model generation code is in Python and the web app is written in JavaScript so that it can’t interface directly super easily. I tried a couple of ways of doing this. Originally I tried using a library called react-py that works as a Python interpreter. Type in python and then it gets executed. While I did eventually get that run without any errors, the model generation code we are using was a bit more complicated than react-py could handle. The result was that it took forever for the code to actually execute and even then there were still other issues. After that, I decided to change my approach to using django.
Django is a Python based backend interface that is often used with HTML. I originally looked for other options as django requires running a server environment and is more commonly used for making forms and interacting with a database. It felt a bit too much for what I needed as I really just needed a single python function to run. Though with react-py not going to plan, django was my next best bet based on my research.
Despite the trouble getting there, I have made progress on getting the web app to react to the python function in the django files. Some adjustments still need to be made to streamline the code and make sure the files go to the right place but there seems to be promise with this method.
I have also spent time fleshing out the web app itself. A lot of small touches like getting the models to highlight when your mouse is over them, changing the tab name of the website to TailorBot Room Designer, experimenting with having text appear over objects in 3D to clarify what they are, making a proper home page for the web app and having the 3D render on another. I have included more meaningful changes like finding a way to more easily move objects in 3D space by simply dragging it and making the code more parameterized to make reusing it easier.
Example of the web app pulling data from elsewhere with django:
Showcase model highlighting:
Transform controls on models:
Progress:
The work I’ve done and my improvement in implementing the Three.js code has made the act of getting new models to appear in 3D space pretty simple. My general react knowledge makes adding to the web app UI a more manageable task. I think the web app itself and my ability to build upon it is in good order. From here it’s really just focusing on those connections with the model generator code and lidar.
Schedule Status: On time.
Next Week’s Deliverables:
Once things with django are sorted out, I want to work on getting the web app to connect to the raspberry pi to start the lidar room scan and then move on to generating the model. I plan to talk more to Zuiheb about that as he has been the one dealing with the raspberries.
Alana’s Status Report for 3/23/24
Personal Accomplishments:
I managed to find a way to get 3D models into the web app. From here it’s just getting the web app to interact with the scan data in real time. In pursuit of that I’ve also been working on building functions in the code to populate the room with furniture models that the user can move around and interact with, though that’s still in the works.
Uploaded chair and table model:
Uploaded empty room model:
Progress:
I’m at the part of the project where I need to interact with my team members more closely as a lot of the next few steps involve getting the web app to work alongside the scan data and scissor lift controls in real time. I’ve already discussed this with them and plan working with them more as we move forward.
Schedule Status: On time.
Next Week’s Deliverables:
I hope by next week to be able have the web app be able to upload a room model from the lidar scan data in real time.