Author: jiyengar

This week I spent some time making changes due to hardware problems such as making it work for 2 solenoids instead of 4 solenoids and making it work for the solenoids being 3 characters apart rather than 4 and with lines that are 12 characters long. We also realized that the braille was in reverse since we are punching down so I reflected that in the translation.

I wanted to make an option for people to upload an image of a box and use the barcode to get the directions from that using cv2 and pyzbar and while I was actually able to make it work and even created html webpages and functions using Flask to integrate the entire platform, it turns out that the raspberry pi has restrictions when it comes to using the opencv libraries so it doesn’t seem like I will get around that. Since it wasn’t one of our original goals, it’s not a big deal but I am pretty disappointed because I got the entire platform to work and it was also super accessible with the iphone and Apple voiceover.

I spent the most significant portion of my time working on official testing. Throughout this semester, I have been doing basic, integration, and edge case testing and the occasional stress testing, but nothing officially put in a file. I now have a file that tested both the uncontracted and the contracted braille functions by trying 100 randomly generated strings and comparing it with the output of braille blaster. This helped me hone my algorithm further to get 100% accuracy. I also tried testing with 100 most popular products according to https://progressivegrocer.com/100-iconic-brands-changed-grocery and checked to see how many responses were in the database (97%) and also how many responses were relevant(100%). I tested the time it took for creating the database by converting it from a csv to the database multiple times and averaging that time as well as the average time it took for each query in the above test. Based on those times, the one time database cost seemed to be. more worth it. I tested the solenoid instruction generation by creating a sample input with unique numbers instead of just 0s and 1s and compared it to the expected output.

I also spend a significant amount of time working with Zeynep integrating the rpi with all of my software code with the physical hardware and while the individual components are working, we do have a bit of honing things down to go.

This week was mostly just adding final touches and doing last debugging. I successfully soldered a latch switch to the rpi so that depending on if it is pressed or not, the printer will print either uncontracted or contracted braille. I also implemented a query system to test against the database. There is an obvious time tradeoff and we are leaning towards the original database but in order to use queries, we just need to change one global variable on the main page so it’s really convenient.

We also visited LAMP on Friday and got to talk to 4 visually impaired people specifically about our product and others in general. We learned a lot more about what they use and how they feel about our product and I’m pretty happy with the ease of use! I was also able to learn some things that helped me with testing.

One of the biggest things in this final week is proving through testing that we achieved the results we wanted. I was struggling with finding a way to prove that the braille translations are accurate using just technology but one of the people at LAMP told me of an app called braille blaster so I have been generating random strings, printing it like dots on my screen, copying it into braille blaster, and comparing the two.

I have also been running speed tests mostly to compare the query and database, but since the database has no speed issues after the initial loading of the website, it is definitely our best option. The last testing I did was webscrape 100 popular products from websites that compile this data and testing to see what our database returns whether it’s accurate recipes, inaccurate recipes, accurate websites from google, or inaccurate websites from google.

This week I worked on the entire pipeline from the frontend notes or websearch to the arduino controlling the solenoids which consisted of a lot of time spent on figuring out how to communicate between the rpi and arduino over serial. I also changed the actual code of how it was translating signals from the lists of combinations to make it more efficient and I debugged some translation issues since our goal is to get it to 100% accuracy. I started debugging and integrating the backup google search feature as well as working on a button that talks directly to the rpi in order to decide things like uncontracted vs. contracted braille and jumbo vs. normal text size based on user feedback.

Zeynep and I visited LAMP on Tuesday to get participants to meet with us 2 weeks from now for a more in depth user testing session and also to get basic feedback which helped us improve usability and also realize the parts that were working well like how easily apple’s existing voiceover function allows users to figure out our website.

In the next week or so, I plan to webscrape backofthebox.com and integrate it with the currently functioning database so we have a larger database. I also plan to finish work on the google search function so that it gets the important information from the searches and not just the entire html. I plan to implement a cache with the 100 most recently searched products. Lastly, I plan to create a file that stress tests the translations. Right now, I have a random generator for me to test one at a time and compare but I would like to have it automatically compare against existing websites that translates characters to a braille image. I am currently on schedule and wrapping up everything I need in order to meet our goals for this project.

This week I spent a lot of time preparing for the demo and making sure that everything worked with every change. With the interim demo this week and our demo with the LAMP advisory group next week, I spent this week adding contractions to the contracted translation. I also worked on making the translation from the braille translation to what the solenoids do and I have it working for the solenoids being 12×16 lines apart and I have it tentatively working for 4×4 lines apart since that is a more reasonable expectation of the mechanical elements and it would allow this to be print much more efficiently as it won’t have to print the entire page if less than half the page is needed. Zeynep and I also worked together on integrating and improving the frontend and backend. I debugged the translation and database extraction functions with basic and edge testing. I have started working on the rpi to arduino connection and the plan is to be able to do a full pipeline control of the solenoids by EOD Monday. I have also started working on opencv because I would love to be able to make our website as accessible as possible.

Each function in each of the backend files requires testing. So far, I have done basic and edge case testing on the translation functions, database creation and extraction functions, and the main backend file. I have also worked with Zeynep to do this testing for different searches and printing of notes. The bulk of what I plan to work on in the next 2 weeks is a comprehensive stress testing file that will test each of these functions for 50 random inputs and 50 popular inputs and compare it to the expectation and I will do the same for the full pipeline.

Along with working on stress testing, I also plan to spend the next week webscraping backofthebox.com and creating a cache. I am on schedule and all of our schedule goals appear to be attainable as of yet.

In order to get our project ready for demo, I have finished database extraction and integrated that process with the frontend. Now, when the person clicks print on either the notes or the product search, they get the relevant response back.

I have also finished contracted translation and plan to demonstrate the difference between that and the uncontracted translation by printing to a separate file as we demonstrate the frontend -> backend pipeline. I have done edge case and basic testing for both translations because those were my biggest worries and plan to work on stress testing this week.

I have run multiple database structures but a couple sites ran into errors being scraped so I currently have a database around 1800 elements big and a database around 150k elements big but the 150k one won’t upload to git. I have tested the database extraction with these and they work great for general pipeline testing but I will need to start creating unit stress tests.

Meanwhile, I have finished creating solenoid “instructions” that will eventually be inserted into arduino code based off of the braille translation made. This is because each of our solenoids will not be embossing in the order of each dot in each character but in separate places. We plan to also print this to show how the solenoids are currently able to achieve each of the necessary states. This week, I want to start working on integrating the backend with the arduino for the solenoid system.

I am currently on track with the schedule and have everything ready for the demo. In the next week, I plan to work on black box unit testing and stress testing. I also plan to make visible progress with the rpi so we have an actual internet host and not just running files. Finally, I want to start working on the rpi to arduino communication and at least get this working for the solenoid system. My reach goal is to work on the cache after working on the rpi server hosting and web scrape other sites if the rpi is large enough to handle the current database given that git refuses to.

This week, I spent a lot of my time working on the ethics report. I think that the main concern it presented for my part of work is the web security. Since I took 18-330 last semester, I am confident that I can implement some measures against things like cross site scripting. As we are not dealing with private data such as things like passwords or payments though, we don’t need to implement security for the majority of popular attacks like pretending to be someone else using cookies.

We are planning to demo the user interface to an advisory group at the Library for Accessible Media Pittsburgh (LAMP) the week after the demos in this class so our team decided that it would be good to have frontend-backend integration done, hosting on the rpi, and an example of a user going from notes to translation to rpi instructions to solenoid/motor movement. So, this week one of my main focuses was finishing the bulk of the translation function and integrating it with the rest of the backend design. Now, most things typed in notes can be represented in braille and in the instructions for the solenoids and stepper motors that will be sent to the rpi. I also have started writing the actual sending of these instructions to the gpio pins for the solenoids by testing with a simple LED circuit.

In order to get the rest of my part ready for the demo, I need to work with Becky and the gantry system to send instructions to the rest of the gpio pins so I can code that before we assemble our MVP and I also need to actually succeed in hosting this website on the RPI or find and do a method that works. As for the demo with the library advisory group, I will need to work with Zeynep to make sure that we know exactly what we want the frontend and backend to be doing when we present and ask for advice. Otherwise, this week I want to accomplish the basic flow for one product search and start web scraping the other sites for our database. I am not ahead of the schedule like I was before but I am in the right position to achieve our goals for the next coming weeks and get back ahead.

We spent most of this week working on the design report given how lengthy it is. I wrote the related work section, integration tests, software trade-offs, system implementation for software, abstract, software index terms, software use case requirements, team member responsibilities, software risk mitigation plans, software testing and verification, and system implementation diagrams including the webscraping diagram, cache diagram, database diagram, and translation diagram. This put into words and explained everything behind what I have been working on up till now and what I plan to work on in the next couple weeks.

I also reached out to someone to show us how the braille embosser in the library LAMP works and to someone else to gather people to test our user interface.

For the software, I finished actually creating the database for directionsforme.org. It basically loops through the keywords (not words such as and, or, the) and adds the recipe to a list that is the value in a dictionary with the key being the keyword. I then use the library csv and csvwriter to write this into the database. For extraction, in webapp.py, I convert this csv file to a dictionary when the website is run so it only needs to be run once for each time we host the website. I want to explore if there is a way to only do this once in general because this will take a lot of time. Then, when someone searches the product, it goes through through each keyword in the search and records the recipes each of these keywords map to in order to figure out the most likely recipes and send that back to the user. This is done in websearch.py.

I have also written out some high level “code” for the translation. It is essentially a for loop that maps each character to the character(s) in the dictionary I have set of 6 signals. There are multiple conditions which regex looks for such as closed parentheses or a word of only capital letters or numbers or contractions which is also in this loop. I use the library re to do these evaluations.

As for communicating with the backend, I have started learning how the Raspberry Pi works and running basic things now that I have the SD card. This ties into the question of the week –

As you’ve now established a set of sub-systems necessary to implement your project, what new tools are you looking into learning so you are able to accomplish your planned tasks?

I have been spending a lot of time learning to use the Raspberry Pi and I expect to spend more time with how it keeps memory and runs programs and receives inputs/outputs for our actual project. I will also need to learn more about internet hosting because that is our current plan based on our research. I am familiar with most of the Python libraries but in order to make the webscraping more efficient, i need to learn more about the specific functions that Beautiful Soup has. I also don’t have experience with the Flask library and as I will be developing the frontend-backend communication further, I might have to learn more about it. Also, I plan to look into basic computer vision and existing open source algorithms so that users have an additional feature to print text that their camera sees whether it’s the back of boxes or flyers.

While I spent more time on the design report and therefore less time than I wanted on the software, I am still on track if not a little ahead compared to the overall schedule especially because I have been doing basic testing as we go (not stress or edge testing yet though). In the next week, I plan to look into the computer vision possibility I talked about earlier, learn how to host the website on the rpi, write the function that takes info from the other sites with product directions and puts it in the database, finish the translation, and start working on the cache. This will put me in a good place to work on integrating and testing this all in the week after so that we have a basic main input/output flow working for the demo.

This week, I visited LAMP, finished developing and testing some of the backend algorithms, and started working on the design report with my teammates. The software work I did this week is all on the github: https://github.com/joshna-ii/digibraille

I finished the webscraping algorithm for directionsforme.org and most of the database creating algorithm. The webscraping algorithm uses beautiful soup to identify the different components on the website such as ingredients and nutritional facts as well as makes tables look nicer. The database creation algorithm take these different parts and puts it in a csv that maps keywords to the title of these recipes.

My next step will be to run the database creation algorithm for all of the webpages and not just my test cases which will take about a day.

In the meantime, I am working on an algorithm that extracts information based on the keywords in the product searched. It will go through any word in the product search that is not generic such as if, and, or, not, yet, the etc. and keep track of a list of recipe titles returned. It will present these recipes in order of highest count (number of keywords that showed the recipe) back to the user so they can click on it and print it (please see Zeynep’s wireframe to see the user’s perspective). It will then take back the user’s query and use another dictionary that maps recipe titles to recipes to confirm before sending it to translation.py which will make the translations. The main_backend.py and the webapp.py files currently handle user interactions and calling functions from the different files.

In order to make this more efficient, I am converting this csv file output to a dict at the beginning of webapp.py using csv reader. Additionally, I will implement a function that stores the most recent searches in an array structure similarly to a cache. The most recent searches will be kept in a dict mapping 100 product searches (20 for testing) and the function will go through this dict before the much larger one for efficiency.

This week, I also went to LAMP which is a library for disabled people to learn more about braille and how people interact with it. I was able to talk to a visually impaired librarian and she showed me the ways she interacts with her phone and the only other technology which sees seen people use which she referred to as an ereader (very similar to our initial idea for this capstone project). She confirmed that a website would be just as easy to access as an app if people can use something like siri to open the website so we need to make sure that the domain is something clear such as digitalbraille.com because if I ask siri to open digitaltracking.com then she has no problem but if I ask her to open directionsforme.org then she gets confused. She also taught me about the difference between contracted and uncontracted braille and gave me pamphlets to learn. This lead to us deciding that we should have an option for contracted and uncontracted braille and they will be two separate functions implemented in my translation.py file. She also told me that they have an advisory group and a newsletter to reach people who are blind so our plan is to finish the bulk of the UI in the next week or so to see if they have any comments on how to make it as accessible as possible and also figure out things like if the default should be uncontracted or contracted braille.

Based on what I learned, I started working on the translation functions in translation.py. I finished creating dictionaries for the alphabet, numbers, and the majority of special characters since the majority of the uncontracted braille is 1:1 mapping. However, there are still some things to consider like capital letters and numbers and what comes before and after them so I also started those conditions in the for loop. I also started designing my plan for the contracted braille algorithm since it will be a lot more complicated to make it efficient. I am thinking of implementing an algorithm that goes through each character in the string and finds matching points in a word that are keys in a dictionary mapping things like “sh” to its braille character. It will look something like this:

I am currently on track with the goals I have set. I was able to spend the appropriate time on all the backend algorithms I have in place but haven’t been able to spend as much time as I wished on hosting our website on the raspberry pi. My plan for this week is to get the raspberry pi running enough for us to test with the LAMP advisory group, run the database creation algorithm for directionsforme.org to completion, finish the database extraction algorithm, finish the cache for recent searches algorithm, finish the uncontracted braille translation algorithm, start working on a webscraping and database algorithm for another product directions website, and start the contracted braille algorithm to the point of it working for a few contractions.

Our team overall have been using classes such as 18220 as well as more advanced megatronics and device science classes my teammates have taken to develop the embossing and communication design. The software design is inspired by webhacking learned in 18330, wrappers and database/cache organization learned in 18213, and testing/verification skills learned in 18240/18341. While the actual algorithms aren’t molded around an existing algorithm, the analytical skills and practice of coming up with our own stems from all the classes we have taken at Carnegie Mellon but for me, especially 15122 and 18213.

At the beginning of this week, I created a github (https://github.com/joshna-ii/digibraille) to help keep track of my progress and allow my teammates to work with my code, especially for when Zeynep develops the frontend further. I spent quite a bit of time later in the week working on the ReadMe and making it as easy to understand and use even in these early stages in order to set a precedent.

The main frontend file is called webapp.py, and I spent the first half of the week using the Flask library in order to create a website which is currently hosted locally (127.0.0.1:5000) when you run the python program. This is an example output and handler:

In general, I have been able to handle the different types of non edge-case inputs and a couple edge case inputs (non-input. poor web link formation) from the website post stage to the string that needs to be translated.

Based on the html user input, I had the webapp.py frontend handling file call the run_backend function in main_backend.py. This next file’s brings together the different parts of the backend: rpi_handler.py, translation.py, websearch.py, and soon a database file.

Websearch.py does the actual webscraping and queries depending on if main_backend.py received a link or a product name. I have a function called google which returns the first link for a google search given the keywords.

I have also created a function that handles direct links that are inputted as well as a function that finds matching keywords in all the pages of websites such as directionsforme and backoftheboxrecipes using the requests library and beautiful library.

The bulk of the execution code (outlined functions are still connected clearly everywhere) for the third function is currently in my temp_code.py file because I have been doing some unit tests, edge cases, and one-off tests. While I have been able to improve efficiency by tenfold by only accessing certain attributes of the webpages, it is still way too slow so my plan is to spend the beginning of next week transferring data from these websites to an internal database by modifying the function I created this week in order to make this process faster. If sufficient results have not been found with the internal database, main_backend.py will use the google function I worked on earlier and mentioned above and webscrape directions off of that.

I also spent a couple hours learning how to work with the raspberry pi yesterday after receiving it.

I am having trouble communicating with it as my laptop does not seem to recognize the device when I ping localhost so I am getting an ethernet cable today to try it that way and if that doesn’t work, I will go to campus and try to connect it to one of those computers. I am spending today working on rpi_handler.py in order to host the basic website through that and get the basic framework down for how the python files will communicate signals on location and movement of solenoids+motors to the raspberry pi post translation through the gpio pins.

I have also spent time this week working on the design presentation for next week including creating block diagrams, scheduling, designing algorithms, figuring out interactions between components, and calculating quantitative expectations.

While I did not achieve the reach goal I set last week of getting 50% of the webscraping done due to this method turning out to be innefficient, I am still on track with our expected schedule with the webscraping, website, and integration files so I do not have to reorganize the schedule.  My goal for this week is to get the basic rpi – python script communication down, get 50% of the webscraping done with databases and creating said database, and start the braille translations while making a more detailed schedule based on the different rules braille follows. I plan to translate the current string output I create to a list of a set of 6 signals with some of the more general braille rules after learning Braille even better myself as a person since there isn’t a 1:1 mapping between the english alphabet and braille characters.

We spent most of this week developing our design and practicing for the presentations. We honed down our use case to specifically help visually impaired people by allowing them to either enter notes and print that on our website or enter a product and print the instructions for using that product. We also figured out what tools we want to use and how to divide everything up. Our current plan is to use Python and HTML to develop the frontend, use Python for webscraping and databases and communicating to hardware, host our webpage on a CMU server, and use gcode instructions to control a RAMP board which controls the solenoids and the motors that move the solenoid system across the page.

I also started working on the webscraping portion since we were told to work from the most difficult part outwards. I have tried working with many Python libraries including googlesearch, requests, urllib, mechanize, and more in order to figure out how to make google queries in the easiest way. After a lot of debugging, I am only able to get the first google result. There isn’t much information about the generator object type for webpages and it appears like I am getting 0-1 results when I shouldn’t be. I also received a lot of errors with actually accessing a website to begin with but I believe that is because I am accessing the browser directly through code and so google treats me like a bot. I am looking into using other websites/browsers for searching.

I am able to get the first website from google now but I began to wonder what I would do if I were to get more than one result because there isn’t a standard to figure out what would be the best website for each product. I have started implementing using a website like directionsforme that already has lots of product compiled like a database while also using the first google search for any product that is not listed on this website. The webscraping should be easier for products on directionsforme because it is relatively standard so I am focusing right now on the best method for finding the information I want to scrape from each of the websites google returns. I am also considering whether or not I should have a feature that reads back the result and confirms with the user if that is what they want to be printed.

Another thing I have started to think about is how we will send instructions to the embosser. Since we are trying to make this as accessible as possible, we have to make sure that the embosser is an easy set up and that the website can actually send the instructions to the embosser. Wireless printers work by connecting to the same WiFi network as the device that will be printing. There is a RAMPS WiFi adapter so we might think about using that so we can continue with this design. I don’t have experience using gcode but Becky does so I did more research in how it works and it is common to send gcode wirelessly to 3D printers by connecting both to WiFi by using a WiFi SD card. I still have a lot more research to do on the interfacing between the webpage and the gcode output across WiFi.

Since we are still at the beginning of our project, we are on track. My goal is to make the webscraping algorithm work for 50% of the top product searches by the end of this next week. I also plan to get some of the basic html framewok up and running because neither Zeynep or I have much experience with actually building a website so it would be better to start earlier so we have more time to resolve issues. Lastly, our team also needs to make sure we understand the communication between our parts both because that will help define each of our “inputs” and “outputs” but also because it might require us to buy additional components.