Category: Pat’s Status Reports

This week I worked mostly on testing and making minor adjustments to the Dawggles. One of the things I worked on was improving string comparison and confidence score in the translation feature. Beforehand, the Dawggles were accepting strings too frequently, causing some of the same translations to be repeatedly displayed. Now, they ignore text that is within a certain similarity and lower in confidence level in order to remove repeats.

This coming week I plan on continuing to fine tune the dawggles and work on preparing for the demo–including the poster.

Overall, progress is on schedule.

This week I spent a lot of time working on the translation on the iOS side. I was able to utilize VNRecognizeTextRequest in Apple’s Vision framework to who real time live translation from and to multiple languages. Furthermore I was able to show the translations and their respective bounding boxes over the live video for debugging purposes and to verify that it was working properly.

I had to do some additional research via google searches and Apple’s documentation to get this working to the desired performance level. One helpful parameter I discovered was the VNRecognizeTextRequest recognitionLevel. Changing from “fast” recognition to “accurate” significantly improved character recognition performance and hardly if at all increased latency of the model.

Overall, I’m really happy with the state of the translation, although there is still some work on handling text groupings and a large quantity of text. I think we’re definitely still on schedule to complete and survey the Dawggles this week.

This week I focused on a couple of things. First of all I worked on modifying the CAD to place the biconvex lens closer to the beamsplitter. The other change we made to the CAD was splitting the front side of the goggles at a 45 degree angle to remove the tunnel effect and give the user a more open view of the environment. We are still waiting on this print to verify the effects of these changes.

Along with this, I also worked a lot on the RPi software. I wired up our two GPIO buttons and I was able to test both of their functionalities. the first button is just a basic app cycle button. In testing this button I was able to verify seamless switching between translation and gps apps with indications on the display. Also, switching apps maintains app data from the last JSON text, which is a nice additional feature. The next button is the app-specific button which is really only used for taking pictures in the translation app–this also works as expected.

Finally, I worked on modifying the display rendering for both apps. I was able to send test JSON text to the RPi from my Mac and verify the display locations and figures. Also, sending data with a different app name automatically switches the app so that in future tests the user will be able to also switch apps from their phone.

Overall, I think we’re doing really well. We still have more iterations of CAD to go through and more work for the iOS, but we are definitely ahead on Pi software verification.

This week I worked on assembling and testing the Dawggles. We were able to run a script to display basic text and change the contrast to so that it is comfortably overlayed. One thing we noticed was the text was getting pretty warped at the edges, so we are going to work to change the mechanical design by moving the OLED closer to the lens and the lens closer to the beamsplitter.

Along with this, I also did basic testing on the server. I was able to  start a TCP connection and send and request text from the server.

Next week, I hope to test some of the other software on the Pi, including the button callback functions and camera client class. Overall, we are still doing well with our schedule, but we do need to make some changes to the CAD to perfect the display alignment.

This week I refined and added to the software framework on the goggles. I created a base implementation of the server and display classes and added all four classes (server, display, camera, and button) to the shared class. I also reorganized the translation and gps functions as two separate application files, in a shared apps folder.

This new framework allows us to start up the goggles and easily switch between applications–by triple tapping the button on the goggles. The app manager handles switching data when this action is selected by swapping the proper data-handling, button-callback, and display functions for the application being switched to. The user will also be able to switch applications on their phone, which the server will notice by receiving a different application name than the current application in the request.

Overall, our progress is slightly behind. We had a small hiccup with some of the hardware, but I feel good that we will be able to start testing software next week.

This week I aided in refining the final CAD model for the Dawggles. The goggle design took priority over software because we are more concerned with having to tweak and iterate the physical design until the print is right. I also helped to start the print at the end of the week so that we will have a first version of the physical design to test next week.

Along with this, I spent a significant amount of time completing the Ethics Assignment. I feel like the second part of this assignment renewed motivation for the goggles to be comfortable. One thing we are concerned about is external light passing through the biconvex lens. For this reason, we modified the roof part of the goggles so that the OLED wires escape from the side, allowing us to put a full roof on top to block external light.

Overall, we are still pretty much on schedule. Next week I plan on helping assemble the Dawggles and testing the hardware–primarily for testing the display and ensuring it is a comfortable experience.

This week I spent significant time designing and implementing a first draft of the RPi software. I reasoned that on the Dawggles themselves, we would want 5 threads. The main thread will initialize our Shared Class, Button GPIO, and our camera client thread. The Shared Class will essentially carry everything relevant to the Dawggles, including the phone ip address, goggle_state, gps and translation data, the current picture, the current display index, and several other things that will be used by the other threads. The second and third threads are the button and the camera threads, which both essentially wait for an action and then either signal a callback function or send the image to the phone’s IP. The fourth thread is the server thread, which runs a websocket to continuously receive the gps and translation info from the iPhone, and the fifth thread handles displaying this data.

I created the block diagrams describing this in the design document last week, and this week I was able to finish a first draft of software for all but the fourth and fifth threads. I pretty much used python classes for everything, specifically SharedClass, GoggleButton, and CameraClient classes.

Next week, I hope to finish a full draft of all five threads, and begin testing their functionality on the Pi itself. Along with this, I plan on assisting with printing and modifying the CAD design for the Dawggles.

I had a pretty busy week this week, so I wasn’t able to work on this capstone until later in the week. With that said, I made great progress on the goggle CAD. As a group we decided it would be easier to make the entire set of goggles with CAD since we would be adding so much hardware to it anyways.

I was able to get a general outline for the goggles done, including an indented holder for the lens. On top of this, I added slot holders for both the Pi Camera 3 and beamsplitter. I will likely change the beamsplitter holders, and the whole CAD certainly needs refining. With that said, I feel like I accomplished a significant amount, especially considering CAD is not my area of expertise.

Given that I have had a busy week, I did not find the time to research OCR models, but the CAD is definitely a higher priority anyways.

I still feel like we’re pretty on schedule. The monochrome OLED arrived on Friday, so we have nearly all of our hardware.

Next week, I plan on continuing to refine the CAD model, working on the Design document, and possibly researching OCR models or testing hardware.

I spent most of this week working on researching system architecture and design. I determined that the maximum number of characters we can fit within one line on the goggles is 12 characters with 12px font. We believe that we can fit both a GPS HUD with 12px font on the top of the display, and also 2 lines for translation–also 12px font–in the middle of the display.

I created diagrams for this pixel outline and determined the appropriate padding and spacing needed to make the goggle text readable and undistorted. With that said, we may change this design to have higher px font size depending on our tests with the goggles. Along with this, I also created diagrams illustrating the examples of using the goggles, as well as an overall hardware diagram, connecting all components appropriately and specifying certain communication protocols.

Lastly, I spent a lot of time determining the open source models we would use for both character detection and translation. I researched the computation time of these models, as well as the latency of RGB image transfer over wifi, to determine the best allocation of computation within our different subsystems. Overall, I feel like we are on track in terms of scheduling. I am hoping that our hardware comes soon.

I was not successful in completing a CAD model of the hardware housing, but honestly this is really difficult without having the physical goggles on hand. We may have to print our own goggles or goggle extensions depending on how deep the goggles are and how well the beamsplitter fits within the goggles. I plan on working on this CAD housing, testing and assembling any incoming hardware, and further researching and testing open source OCR models in the coming week.

 

I spent a significant amount of time this week organizing and assigning our project tasks in Jira. We are using Jira for both our work breakdown dictionary and Gantt charts, allowing us to mark tasks complete, keep up with deadlines, and update our schedule to manage risks and issues that arise. We are currently scheduled to order all of our parts and finalize our reflective display design by the end of tomorrow, which we are on track to complete.

Along with this, I continued researching other demonstrations of the Pepper’s Ghost illusion and other attempts at this project. I feel like our current design is the most simple and straightforward approach to what we are trying to achieve with this project.

Lastly, I have reviewed our parts list and confirmed hardware correspondence in terms of communication protocol and power limits.

Next week, I will work on designing the camera and overall housing for the goggles. I aim to have at least a CAD prototype for each of these by the end of the week.