This week was really focused on wrapping up the pipeline into a complete package that is “shippable”. We still have a few things to add to allow other people to use it, such as a README.md explaining how the software works and how people can perform scans of their own, and the report explaining the design, metrics, and tradeoffs that went into the development of our project. All of these things will be completed next week in time for the report and video submissions.
This last week, I have mainly been working on updating the verification module to allow us to gather sufficient data to present for our final presentation. We wanted to weigh each design tradeoff we made by using quantitative results, so we updated the driver script as well as the verification engine to produce more data that we could capture as csv files (usable files in excel).
The main new metric we came up with is the notion of scan “Accuracy”, whose formula directly follows our accuracy requirement for the project.
- 90% of vertices of our reconstructed object must be within 2% of the longest axis of the ground truth mesh to the ground truth mesh.
- 100% of vertices of our reconstructed object must be within 5% of the longest axis of the ground truth mesh to the ground truth mesh.
We noticed that while this is a good metric for accuracy of our scan, it does not paint a complete picture. We are missing the idea of “completeness”, which is how much of the ground truth mesh is covered by our reconstructed mesh. This data is captured by taking the same formulation as our requirements above, but reversing the roles of the reconstructed mesh and the ground truth mesh (that is testing how close vertices of the ground truth mesh are to our reconstructed object, if this is high then our reconstructed object “completely” fills the space the ground truth mesh did). We encode these two ideas, of accuracy and of completeness, by the notation “forward” and “backward” accuracy.
Now that we have made this distinction, it is important to note that forward accuracy is the only important metric to our requirements: we are only truly concerned about the accuracy of the points we generate. However, the notion of completeness was used to weigh some of our design tradeoffs, so it is important to include. Below is the final formulation of general accuracy (which applies to both forward, backward, and “total” accuracy).
- accuracy = 100% – 0.9 * (% points farther than 2% but closer than 5%) – (% points farther than 5%)
In a sense, this formulation of accuracy conveys our requirement while providing a numeric representation of our results. When determining total accuracy for a reconstructed mesh, we compute both forward and backward accuracy and average them weighted by number of vertices on the ground truth/reconstructed mesh.
I implemented the data processing required to capture the accuracy data in the verification engine, and we have recently utilized this data to construct graphs and figures to demonstrate the tradeoffs we have made during the project. Right now I believe we are on schedule, since we are prepared for the final presentation. There are currently no significant risks to report. As noted earlier, we plan to make a “shippable” version next week as well as work on the videos and the final report.