Author: fcain

This week, I’ve accomplished a lot personally, which I am very proud of. I’ve been responsible for most of the software implementation for this project, and it is complete! We were able to achieve 99% OCR accuracy, which we stalled at for a bit. I did this by switching from Tesseract to EasyOCR, which has proven to be more accurate for images we take with the Raspberry Pi. We take the picture with a button press, which is then translated to text. This text is then fed to a LLM, which cleans any errors and highlights the most important text we want to print with braille. This works really, really well.

We were able to finalize and code the next steps, as well. The text string is then converted into a string of numbers, which I’ll detail here: say you would like to print the word “hi”. To print h, we need two columns. For the left column, this is a top dot, a middle dot, and no dot. We can represent this as 1, 1, 0, and can interpret this s binary, making it a decimal 6. We repeat for the right column of the h, and for any other letters (in this case, i). The ends up with the string 6224. Then we wrote the code to have the Raspberry Pi read this string and make servo moves that correspond to the corresponding numbered actuator movements.

We were also able to finalize the code to get the speaker up and running, so the device asks the user to confirm the word about to be printed before proceeding. I’m glad we got this to work, because for a long time we were wondering how to have the user interface work with the knowledge the user is blind.

We were running a little behind on the hardware still, so I’ve been helping Bella and Abby with some 3D printing and design testing. We’ve been trying some different printing wheel prototypes to get as accurate of text as possible. As we move into this week, we will complete this prior to our presentation (!!!) and video submission. As for the final report, we have internalized the feedback that was given to us for our previous paper, and are making vast improvements prior to Friday.

This week, I worked with Abby and Bella to integrate the software set up in Python with the hardware running via Raspberry Pi. On Monday, I placed the final orders needed for the project, and we received them on Wednesday. This order included mini breadboards, buttons, a speaker, and a Raspberry Pi specific camera to be used for BrailleMate. The rest of the week into the weekend, Abby and I connected the camera to the Raspberry Pi and confirmed that it works! We were then able to update the Tesseract code– how it works is that the Raspberry Pi camera takes an image called “photo.jpg” and puts it in a folder. This will always be the only image in the folder, as with every new scan, it is rewritten. Tesseract processing is then run on the image, and it produces “translation.txt” which is the English text version of the image. We will then use my python script that I wrote a few weeks ago to process translation.txt into the pins needed to have the Braille printing.

We were also able to integrate button pressing mechanisms for having the servos run, which is what will do the printing, albiet with a bit more of a lag than desired. We are integrating the button pressing for taking the picture tonight, as well as the audio output to interact with the user.

Next week is Thanksgiving break, but we intend to continue our work next week. We are taking BrailleMate home with us and will continue to work via FaceTime. Something we are a bit behind on is the housing for our product so that it is attractive for the user. We intend to complete that portion after Thanksgiving Break.

This week, I was able to essentially finalize my work on the Tesseract for use with OCR. I have utilized hundreds of public, research grade documents and images, including ~200 from FUNSD (Form Understanding in Noisy Scanned Documents) and ~1.3k from Tobacco800 (complex document image processing test set from Illinois Institute of Technology), which seemed to have about a 95% accuracy rate. I suspect this is largely because stock Tesseract is trained mainly on printed text, so its models and language priors don’t match handwritten cursive, which is what is often in FUNSD and Tobacco800. When I tested it on the Open Food Facts dataset, which is much closer to our use case (printed, commercial packaging to be scanned) and closer to what Tesseract was trained on, the accuracy rate was closer to 99%, which is our intended goal.

I’ve also written the bulk of the code for mapping Braille dot patterns to actuator sequence, but still need to more comprehensively test this, using both written test cases and with the physical BrailleMate device.

My progress is on schedule, and I am excited for how the project has come together! In my group, I have been responsible for much of the software, which has been interesting to learn about. In coming weeks, I will need to keep testing Tesseract on physical bottles, not just online datasets. Now that Abby has largely set up the Raspberry Pi, I will also need to update the Braille dot pattern code to work with our hardware, rather than just my Python test cases.

This week, as mentioned in the team report,  I continue to experiment with Tesseract, which is important as it’s the main way in which we hope to use OCR to read an image and convert it to braille. The data sets I was experimenting with last week has proved difficult to use in terms of cursive script, so I have been trying to find other, more helpful data sets to experiment with. I will also begin to venture into the real world, taking images of actual labels (medicine bottles, prescription bottles, cereal boxes, etc) to test with Tesseract. I believe my progress is where it needs to be.

In terms of software, this week I will also begin to write the logic to “convert” each English recognized character into actuator pin movement that embosses the braille lettering. For instance, here’s the letter A in Braille:

Logically, I need to tell Pin 1 to engage, Pin 2 to disengage, and Pin 3 to disengage, and emboss the first column. Then, I need to have the tape move over slightly, and tell Pin 1 disengage, Pin 2 to disengage, and Pin 3 to disengage. This is the process for each character. I will think thoughtfully about how to easily modularize this process in my code.

This week, what I personally have been working on is testing Tesseract. As our goal is to achieve 99% OCR accuracy, we need to heavily test the software first, as part of our risk mitigation strategy.  I researched data bases online to find appropriate (and a lot of) data to test the software on. So far, I’ve been working with FUNSD (Form Understanding in Noisy Scanned Documents) which should be helpful for scanned or printed text, and Tobacco800 which seems to have greater font variance and even handwritten components. So far, I’ve been working on writing the script to run these, and I would guess that I  should be mainly wrapped up with testing Tesseract by the end of the week.

This week,  I also contacted the CMU Director of the Office of Disability Resources, who has given us helpful advice on the feasibility of our design, and we have confirmed that the Western Pennsylvania School for Blind Children is interested in helping us in our testing.

I believe our progress is about on track. We had some delays in ordering due to the AWS outage, but we feel good otherwise, and are ordering ASAP.  This coming week, I will not be attending class on Monday because I will be in Washington, DC attending the American Institute for Medical and Biological Engineering Public Policy Institute, but I’m able to asynchronously work on my Tesseract testing and will coordinate with my group members to make sure we are still making progress.

This week, I finished my sections for the design report.  With the entire group, we reworked our design components, and were deciding between a one motor and three motor implementation. We decided to focus on one as our actual plan in the design report, and the other would be a backup contingency plan. We decided that it would be easy to pivot between the two implementations so long as we have the components, so we budgeted for several motors in our plan. We’ll be able to quickly reprint small 3D printed parts on campus on demand, so we are not worried about this part. We are a bit behind on ordering parts, as Abby mentioned. The original plan was to submit this to our TA over fall break, but this ended up not happening. We’ll order the parts in class this upcoming week, and we can still test software like Tesseract or CAD designs in the meantime, so I am not too worried about this. This coming week, I hope to help my teammates with the CAD and 3D print, but on my end, I would like to spend my time testing Tesseract so we know that our OCR software is extremely reliable before we start physically building the device in coming weeks. I’m hopeful we can find a way to automate such testing, which was suggested by a TA during our proposal presentation.