Category: Joshna’s Weekly Status Reports

This week I spent some time making changes due to hardware problems such as making it work for 2 solenoids instead of 4 solenoids and making it work for the solenoids being 3 characters apart rather than 4 and with lines that are 12 characters long. We also realized that the braille was in reverse since we are punching down so I reflected that in the translation.

I wanted to make an option for people to upload an image of a box and use the barcode to get the directions from that using cv2 and pyzbar and while I was actually able to make it work and even created html webpages and functions using Flask to integrate the entire platform, it turns out that the raspberry pi has restrictions when it comes to using the opencv libraries so it doesn’t seem like I will get around that. Since it wasn’t one of our original goals, it’s not a big deal but I am pretty disappointed because I got the entire platform to work and it was also super accessible with the iphone and Apple voiceover.

I spent the most significant portion of my time working on official testing. Throughout this semester, I have been doing basic, integration, and edge case testing and the occasional stress testing, but nothing officially put in a file. I now have a file that tested both the uncontracted and the contracted braille functions by trying 100 randomly generated strings and comparing it with the output of braille blaster. This helped me hone my algorithm further to get 100% accuracy. I also tried testing with 100 most popular products according to https://progressivegrocer.com/100-iconic-brands-changed-grocery and checked to see how many responses were in the database (97%) and also how many responses were relevant(100%). I tested the time it took for creating the database by converting it from a csv to the database multiple times and averaging that time as well as the average time it took for each query in the above test. Based on those times, the one time database cost seemed to be. more worth it. I tested the solenoid instruction generation by creating a sample input with unique numbers instead of just 0s and 1s and compared it to the expected output.

I also spend a significant amount of time working with Zeynep integrating the rpi with all of my software code with the physical hardware and while the individual components are working, we do have a bit of honing things down to go.

This week I focused on honing in on the hardware and electrical components.

I decided to redesign the side panel so that they have holes for the barrel jack connectors. I cut this in clear to allow light to come into the LDR sensor, which helped us get rid of a LED.

I spent countless hours tuning where the styluses go, and supported the team as the last software changes were made.

There was an issue with our y motor, which I  was  able to resolve when I replaced it. I don’t think the original was made very well.

I used much more loctite on the solenoid styluses which helped with the loose screw problem

This week was mostly just adding final touches and doing last debugging. I successfully soldered a latch switch to the rpi so that depending on if it is pressed or not, the printer will print either uncontracted or contracted braille. I also implemented a query system to test against the database. There is an obvious time tradeoff and we are leaning towards the original database but in order to use queries, we just need to change one global variable on the main page so it’s really convenient.

We also visited LAMP on Friday and got to talk to 4 visually impaired people specifically about our product and others in general. We learned a lot more about what they use and how they feel about our product and I’m pretty happy with the ease of use! I was also able to learn some things that helped me with testing.

One of the biggest things in this final week is proving through testing that we achieved the results we wanted. I was struggling with finding a way to prove that the braille translations are accurate using just technology but one of the people at LAMP told me of an app called braille blaster so I have been generating random strings, printing it like dots on my screen, copying it into braille blaster, and comparing the two.

I have also been running speed tests mostly to compare the query and database, but since the database has no speed issues after the initial loading of the website, it is definitely our best option. The last testing I did was webscrape 100 popular products from websites that compile this data and testing to see what our database returns whether it’s accurate recipes, inaccurate recipes, accurate websites from google, or inaccurate websites from google.

This week I worked on the entire pipeline from the frontend notes or websearch to the arduino controlling the solenoids which consisted of a lot of time spent on figuring out how to communicate between the rpi and arduino over serial. I also changed the actual code of how it was translating signals from the lists of combinations to make it more efficient and I debugged some translation issues since our goal is to get it to 100% accuracy. I started debugging and integrating the backup google search feature as well as working on a button that talks directly to the rpi in order to decide things like uncontracted vs. contracted braille and jumbo vs. normal text size based on user feedback.

Zeynep and I visited LAMP on Tuesday to get participants to meet with us 2 weeks from now for a more in depth user testing session and also to get basic feedback which helped us improve usability and also realize the parts that were working well like how easily apple’s existing voiceover function allows users to figure out our website.

In the next week or so, I plan to webscrape backofthebox.com and integrate it with the currently functioning database so we have a larger database. I also plan to finish work on the google search function so that it gets the important information from the searches and not just the entire html. I plan to implement a cache with the 100 most recently searched products. Lastly, I plan to create a file that stress tests the translations. Right now, I have a random generator for me to test one at a time and compare but I would like to have it automatically compare against existing websites that translates characters to a braille image. I am currently on schedule and wrapping up everything I need in order to meet our goals for this project.

This week I spent a lot of time preparing for the demo and making sure that everything worked with every change. With the interim demo this week and our demo with the LAMP advisory group next week, I spent this week adding contractions to the contracted translation. I also worked on making the translation from the braille translation to what the solenoids do and I have it working for the solenoids being 12×16 lines apart and I have it tentatively working for 4×4 lines apart since that is a more reasonable expectation of the mechanical elements and it would allow this to be print much more efficiently as it won’t have to print the entire page if less than half the page is needed. Zeynep and I also worked together on integrating and improving the frontend and backend. I debugged the translation and database extraction functions with basic and edge testing. I have started working on the rpi to arduino connection and the plan is to be able to do a full pipeline control of the solenoids by EOD Monday. I have also started working on opencv because I would love to be able to make our website as accessible as possible.

Each function in each of the backend files requires testing. So far, I have done basic and edge case testing on the translation functions, database creation and extraction functions, and the main backend file. I have also worked with Zeynep to do this testing for different searches and printing of notes. The bulk of what I plan to work on in the next 2 weeks is a comprehensive stress testing file that will test each of these functions for 50 random inputs and 50 popular inputs and compare it to the expectation and I will do the same for the full pipeline.

Along with working on stress testing, I also plan to spend the next week webscraping backofthebox.com and creating a cache. I am on schedule and all of our schedule goals appear to be attainable as of yet.

In order to get our project ready for demo, I have finished database extraction and integrated that process with the frontend. Now, when the person clicks print on either the notes or the product search, they get the relevant response back.

I have also finished contracted translation and plan to demonstrate the difference between that and the uncontracted translation by printing to a separate file as we demonstrate the frontend -> backend pipeline. I have done edge case and basic testing for both translations because those were my biggest worries and plan to work on stress testing this week.

I have run multiple database structures but a couple sites ran into errors being scraped so I currently have a database around 1800 elements big and a database around 150k elements big but the 150k one won’t upload to git. I have tested the database extraction with these and they work great for general pipeline testing but I will need to start creating unit stress tests.

Meanwhile, I have finished creating solenoid “instructions” that will eventually be inserted into arduino code based off of the braille translation made. This is because each of our solenoids will not be embossing in the order of each dot in each character but in separate places. We plan to also print this to show how the solenoids are currently able to achieve each of the necessary states. This week, I want to start working on integrating the backend with the arduino for the solenoid system.

I am currently on track with the schedule and have everything ready for the demo. In the next week, I plan to work on black box unit testing and stress testing. I also plan to make visible progress with the rpi so we have an actual internet host and not just running files. Finally, I want to start working on the rpi to arduino communication and at least get this working for the solenoid system. My reach goal is to work on the cache after working on the rpi server hosting and web scrape other sites if the rpi is large enough to handle the current database given that git refuses to.

This week, I spent a lot of my time working on the ethics report. I think that the main concern it presented for my part of work is the web security. Since I took 18-330 last semester, I am confident that I can implement some measures against things like cross site scripting. As we are not dealing with private data such as things like passwords or payments though, we don’t need to implement security for the majority of popular attacks like pretending to be someone else using cookies.

We are planning to demo the user interface to an advisory group at the Library for Accessible Media Pittsburgh (LAMP) the week after the demos in this class so our team decided that it would be good to have frontend-backend integration done, hosting on the rpi, and an example of a user going from notes to translation to rpi instructions to solenoid/motor movement. So, this week one of my main focuses was finishing the bulk of the translation function and integrating it with the rest of the backend design. Now, most things typed in notes can be represented in braille and in the instructions for the solenoids and stepper motors that will be sent to the rpi. I also have started writing the actual sending of these instructions to the gpio pins for the solenoids by testing with a simple LED circuit.

In order to get the rest of my part ready for the demo, I need to work with Becky and the gantry system to send instructions to the rest of the gpio pins so I can code that before we assemble our MVP and I also need to actually succeed in hosting this website on the RPI or find and do a method that works. As for the demo with the library advisory group, I will need to work with Zeynep to make sure that we know exactly what we want the frontend and backend to be doing when we present and ask for advice. Otherwise, this week I want to accomplish the basic flow for one product search and start web scraping the other sites for our database. I am not ahead of the schedule like I was before but I am in the right position to achieve our goals for the next coming weeks and get back ahead.

What did you personally accomplish this week?

• sent off PCB
• made a circuit for joshna to begin testing
• received stuff in the mail
• sent off gantry piece for 3D printer

Is your progress on schedule or behind? 

• a bit behind because some packages were delayed in ordering

What deliverables do you hope to complete in the next week?

• pick up print
• iterate
• recieve stepper w the leadscrew
• cut out x gantry parts for ze demo

We spent most of this week working on the design report given how lengthy it is. I wrote the related work section, integration tests, software trade-offs, system implementation for software, abstract, software index terms, software use case requirements, team member responsibilities, software risk mitigation plans, software testing and verification, and system implementation diagrams including the webscraping diagram, cache diagram, database diagram, and translation diagram. This put into words and explained everything behind what I have been working on up till now and what I plan to work on in the next couple weeks.

I also reached out to someone to show us how the braille embosser in the library LAMP works and to someone else to gather people to test our user interface.

For the software, I finished actually creating the database for directionsforme.org. It basically loops through the keywords (not words such as and, or, the) and adds the recipe to a list that is the value in a dictionary with the key being the keyword. I then use the library csv and csvwriter to write this into the database. For extraction, in webapp.py, I convert this csv file to a dictionary when the website is run so it only needs to be run once for each time we host the website. I want to explore if there is a way to only do this once in general because this will take a lot of time. Then, when someone searches the product, it goes through through each keyword in the search and records the recipes each of these keywords map to in order to figure out the most likely recipes and send that back to the user. This is done in websearch.py.

I have also written out some high level “code” for the translation. It is essentially a for loop that maps each character to the character(s) in the dictionary I have set of 6 signals. There are multiple conditions which regex looks for such as closed parentheses or a word of only capital letters or numbers or contractions which is also in this loop. I use the library re to do these evaluations.

As for communicating with the backend, I have started learning how the Raspberry Pi works and running basic things now that I have the SD card. This ties into the question of the week –

As you’ve now established a set of sub-systems necessary to implement your project, what new tools are you looking into learning so you are able to accomplish your planned tasks?

I have been spending a lot of time learning to use the Raspberry Pi and I expect to spend more time with how it keeps memory and runs programs and receives inputs/outputs for our actual project. I will also need to learn more about internet hosting because that is our current plan based on our research. I am familiar with most of the Python libraries but in order to make the webscraping more efficient, i need to learn more about the specific functions that Beautiful Soup has. I also don’t have experience with the Flask library and as I will be developing the frontend-backend communication further, I might have to learn more about it. Also, I plan to look into basic computer vision and existing open source algorithms so that users have an additional feature to print text that their camera sees whether it’s the back of boxes or flyers.

While I spent more time on the design report and therefore less time than I wanted on the software, I am still on track if not a little ahead compared to the overall schedule especially because I have been doing basic testing as we go (not stress or edge testing yet though). In the next week, I plan to look into the computer vision possibility I talked about earlier, learn how to host the website on the rpi, write the function that takes info from the other sites with product directions and puts it in the database, finish the translation, and start working on the cache. This will put me in a good place to work on integrating and testing this all in the week after so that we have a basic main input/output flow working for the demo.

This week, I visited LAMP, finished developing and testing some of the backend algorithms, and started working on the design report with my teammates. The software work I did this week is all on the github: https://github.com/joshna-ii/digibraille

I finished the webscraping algorithm for directionsforme.org and most of the database creating algorithm. The webscraping algorithm uses beautiful soup to identify the different components on the website such as ingredients and nutritional facts as well as makes tables look nicer. The database creation algorithm take these different parts and puts it in a csv that maps keywords to the title of these recipes.

My next step will be to run the database creation algorithm for all of the webpages and not just my test cases which will take about a day.

In the meantime, I am working on an algorithm that extracts information based on the keywords in the product searched. It will go through any word in the product search that is not generic such as if, and, or, not, yet, the etc. and keep track of a list of recipe titles returned. It will present these recipes in order of highest count (number of keywords that showed the recipe) back to the user so they can click on it and print it (please see Zeynep’s wireframe to see the user’s perspective). It will then take back the user’s query and use another dictionary that maps recipe titles to recipes to confirm before sending it to translation.py which will make the translations. The main_backend.py and the webapp.py files currently handle user interactions and calling functions from the different files.

In order to make this more efficient, I am converting this csv file output to a dict at the beginning of webapp.py using csv reader. Additionally, I will implement a function that stores the most recent searches in an array structure similarly to a cache. The most recent searches will be kept in a dict mapping 100 product searches (20 for testing) and the function will go through this dict before the much larger one for efficiency.

This week, I also went to LAMP which is a library for disabled people to learn more about braille and how people interact with it. I was able to talk to a visually impaired librarian and she showed me the ways she interacts with her phone and the only other technology which sees seen people use which she referred to as an ereader (very similar to our initial idea for this capstone project). She confirmed that a website would be just as easy to access as an app if people can use something like siri to open the website so we need to make sure that the domain is something clear such as digitalbraille.com because if I ask siri to open digitaltracking.com then she has no problem but if I ask her to open directionsforme.org then she gets confused. She also taught me about the difference between contracted and uncontracted braille and gave me pamphlets to learn. This lead to us deciding that we should have an option for contracted and uncontracted braille and they will be two separate functions implemented in my translation.py file. She also told me that they have an advisory group and a newsletter to reach people who are blind so our plan is to finish the bulk of the UI in the next week or so to see if they have any comments on how to make it as accessible as possible and also figure out things like if the default should be uncontracted or contracted braille.

Based on what I learned, I started working on the translation functions in translation.py. I finished creating dictionaries for the alphabet, numbers, and the majority of special characters since the majority of the uncontracted braille is 1:1 mapping. However, there are still some things to consider like capital letters and numbers and what comes before and after them so I also started those conditions in the for loop. I also started designing my plan for the contracted braille algorithm since it will be a lot more complicated to make it efficient. I am thinking of implementing an algorithm that goes through each character in the string and finds matching points in a word that are keys in a dictionary mapping things like “sh” to its braille character. It will look something like this:

I am currently on track with the goals I have set. I was able to spend the appropriate time on all the backend algorithms I have in place but haven’t been able to spend as much time as I wished on hosting our website on the raspberry pi. My plan for this week is to get the raspberry pi running enough for us to test with the LAMP advisory group, run the database creation algorithm for directionsforme.org to completion, finish the database extraction algorithm, finish the cache for recent searches algorithm, finish the uncontracted braille translation algorithm, start working on a webscraping and database algorithm for another product directions website, and start the contracted braille algorithm to the point of it working for a few contractions.