Category: Leia’s Status Reports

Progress

Because of our team’s shift in solution approach from mobile app to web app, I have been adapting and rewriting code for the Arduino to implement bluetooth functionality with a webpage. With a general HTML, CSS, and Typescript setup, the demo website has been able to connect and control over BLE the Arduino unit. Textbox input has been added and tested to write sentences from the web app to the OLED screen via the Arduino. I’ve also been trying to transplant the necessary and appropriate demo web app components into our actual project web app. However, I’m facing similar issues and difficulties with trying to run our python scripts because my virtual environments and modules I download continue to be incompatible. Despite downgrading versions and installing appropriate elements, it does not work and so, I will be developing the bluetooth features so that they can be easily transferred into the web app with my team members.

The 3D case and attachment to the phone is currently underway. The design was completed and printing is in progress.

Next Steps

I will be retrieving the completed 3D print and incorporating the hardware parts into it to create the complete physical product of our project. I will also be trying to further accomplish and refine the text transmission over bluetooth between the Arduino and our web app, and assist with frontend if necessary.

With team members, we will be pushing into testing stages to ensure our product meets our use case and design requirements. We plan to have surveys to measure user satisfaction and other methods that verify our project.

Additional Prompt

To operate CAD software to import and configure models, I had to watch guidance videos provided by the program’s company to develop the final attachment. I also had to read documentation for the Web Bluetooth API to understand how a web app can implement BLE functions and pair with external devices such as an Arduino in addition to transmitting and receiving data from the Arduino. Forums such as Stack Overflow and Chrome For Developer’s articles about communicating with bluetooth devices over Javascript helped solidify my understandings.

Progress

I have got the OLED to display text from the Arduino unit as well as a variety of images and animations for testing purposes. The demo app that connects with the Arduino via bluetooth has an added text feature that the user can type on. Currently, transmitting text from the app to the Arduino to show on the OLED is a work in progress.

Next Steps

I will be trying to get the app to OLED transmission to succeed by the end of this week so that next week I can focus on designing and 3D printing the case that will hold the hardware components and latch onto the phone. After that, I will be working with my team members to integrate their CV and ML app with my demo app to develop a singular mobile application that handles all of our project’s goals.

Once that app is settled, I will be building on the frontend to make it more user friendly and simple to operate.

Verification

I’ve run example code provided by Arduino to test the fluidity and latency of the OLED screen. I’ve also already examined the speed at which the app is able to connect with the Arduino over bluetooth, which has demonstrated immediate results. Other verification tests I will be running are:

• additional latency tests of app to OLED text communication
• phone attachment assessments that ensure it is easy to latch and remove
• speed and formatting of ASL translations on OLED

Essentially, I will be handling all the inspections in regards to hardware that ensures the physical product meets the use case and design requirements.

Progress

I have successfully soldered all the components together: battery, Arduino, and OLED screen. The entire hardware side is currently self-sufficient and functional. The battery essentially powers the Arduino, and the OLED draws power from the Arduino to work. It can be charged with inductive wireless charging but for safety precautions, it will continue to be charged through plugging in the Arduino via the Micro USB cable since the Adafruit backpack enables the battery to be charged and maintained in this method. The demo app for the circuit can connect and control to the Arduino via bluetooth, and the Arduino sketches uploaded to the unit can affect the OLED screen properly with low latency and clear imagery.

Next Steps

I will now be integrating features into the demo app that allow text to be transmitted over bluetooth to the Arduino so the OLED can exhibit them. The app will be further refined in its appearance and overall frontend to eventually prepare for its integration with machine learning and computer vision codes in Xcode. I will also be combining Arduino sketches I have for bluetooth and OLED text display together into one program that handles all necessary functionalities.

Farther down the line, I will be planning on the 3D printing case for the hardware components to be fitted into a neat package and eventually attached to the phone.

Progress

I continued to practice American sign language, particularly basic greetings and the alphabet. I’ve also been tweaking the mobile app used for testing bluetooth capabilities to work on frontend development and to implement a text box typing function. Because of Adafruit issues mentioned from the last status report, another voltage regulator backpack was purchased in addition to a 3.7V 150mAh Adafruit battery, and they were obtained this week. I tried connecting the OLED screen to the Arduino, but the loose jumper wires and unbending pins that came with both items require soldering to be affixed together. The same can be said for the Adafruit backpack and battery.

Next Steps

Now that all components for the hardware side have been attained, I will be going into the 18220 lab to solder the wires. The interim demo is approaching so my goal is to get the entire hardware side joined together by then. I hope to at least make the Arduino module self-sufficient by connecting the battery and then hooking it to the OLED. The ideal is to also get the OLED working by then using the mobile app.

Progress

I was able to implement the bluetooth function between a mobile app and the Arduino unit. The app currently can switch on and off the Arduino’s LED and show temperature data taken from the Arduino all through bluetooth.

I tried to connect the battery with the Adafruit battery backpack, but some smoke came out. Turns out the battery’s wire port is not compatible with Adafruit’s receiving port because the polarities are reversed – Adafruit sets their positive on the right side while the negative is on the left, the battery is the opposite. Although the Adafruit circuit appears fine, there is a possibility that the chip did blow. I considered taking out the wires form the batterie’s port to switch the contacts, but then I was informed that I can accidentally short-circuit the battery with the contacts out if they somehow touch each other.

Next Steps

I will be trying to connect the OLED screen to the Arduino. After, I will try to display the temperature on it to ensure the connection across app, Arduino, and screen is seamless. The next step would be to try and display any text on the OLED, first directly from the Arduino with its uploaded sketch, and then from the app with the Arduino as the medium. I want to work on how a user can type in a textbox on the app and immediately have that shown real-time on the screen.

I will also have to be purchasing a new Adafruit backpack and their lips battery because none of the third-party batteries I could find on Amazon are oriented in Adafruit’s polarity.

Progress

I tried to connect all the components together but encountered significant difficulties. When connecting the Arduino to the LiPo battery, unfortunately the pins of the Arduino unit do not fit and insert into the breadboards I purchased. I tried to perform direct coupling with jumper wires but encountered there are risks to this method. Connecting an external power supply to an Arduino without proper voltage regulation may damage the unit, even when using a battery surplus that matches within the voltage range at which the Arduino uses. I also do not possess a micro usb cord to even test the Arduino itself to my computer.

I wrote the Arduino IDE sketch to connect the Arduino unit to a mobile app over bluetooth and to communicate with the OLED screen. The mobile app is a simple, testing environment solely for exercising the BLE capabilities and to try transmitting text between each other. It is not reflective of the actual mobile app we will be employing. I have downloaded all necessary CAD models and begun creating a basic casing as well.

Next Steps

I found I did not acquire the appropriate tools and materials for this project. I will be purchasing an Adafruit Battery Backpack, a voltage regulating shield to protect the Arduino unit. I will also be getting an Arduino Nano 33 BLE Sense without headers. This model has additional sensors that will aid with testing in addition to capabilities of the Nano 33 BLE. Without headers, I will be soldering the components together. I find this process is better as its headers probably won’t fit my breadboard either. Finally, I will purchase a micro usb cable to be able to test without needing a battery source hooked to the Arduino. Everything else is still fine to use, especially the jumper wires, battery, and screens. I want to try and get at least the battery and Arduino connection as well as the bluetooth capabilities done as my highest priorities. 

Progress

The components for our product have been ordered through the purchasing form: an Arduino Nano 33 BLE, an OLED 2.42” screen display module, an E-Ink 2.7” display, a Lithium Polymer battery 3.7V 2000mAh, and a breadboard + jumper wires kit. Currently, the two different screens and the battery have been received and hopefully the rest of the parts will arrive this coming week. I’ve been continuing to prepare how I’ll connect everything and learning sign language on Youtube. I’ve also been practicing the Swift language and Xcode environment via Apple Developer Tutorials. Specifically, there are three features I’m trying to learn to integrate into the mobile app for our MVP and also for backup in case our attempts at integration in the future across the app, Arduino, machine learning, and computer vision go awry: 1. Retrieving data from the internet such as URLs so we can port a web app into a mobile app, 2. Recognizing multi-touch screen gestures like taps, drags, and touch and hold, and 3. Recognizing hand gestures from the phone camera with machine learning. With Ran, I am also trying to figure out how to distribute our app into our phone for testing purposes. She raised an issue that the Xcode-simulated iPhone does not have a camera implementation so we are working to try and get the app into our phones.

Next Steps

The third feature mentioned in the Progress section needs further analysis and communication with team members. It’s performance is still uncertain and how it could amalgamate with our ASL-adapted computer vision and machine learning is questionable. For now, its primary use is to try and get our app to use the phone’s camera.

My plan is to get a working mobile app with functional buttons that lead to the settings page and the  ASL page where at its corner, a small window shows what the phone’s front-facing camera sees. This will be broken down into further steps. Additionally, once I obtain the rest of the purchased components, I will connect the Arduino to the app, using the BLE feature. I’ll attach an LED to the Arduino and see if the mobile app can control it. After, I’ll hook a screen to Arduino, control the screen via Arduino, then control the screen via app. I realized that it’s still too early to try and utilize CAD, so my priorities have shifted into working on the mobile app and operating the hardware.

Progress

I have done more research and trade-off analyses on the items we will need for the hardware side. I intend to purchase the Arduino Nano 33 BLE for its bluetooth capability and compact size. For the product to be portable and chargeable, I will attach a Lithium Ion battery to the Nano to power it. The reason for this battery in particular is that most handheld devices use these batteries. Hence, when a person uses a wired or inductive/wireless charger for their phone, they can also charge the product’s battery as well. Currently, I am considering three different displays to be used for the dual-screen aspect: LCD, OLED, and E-INK. They are all relatively priced and each have balanced pros and cons. I plan to try all screen types after I find the most suitable one of each. They all must be about 2.5” diagonally so the screen is large enough for the other person to see, but not overly big that it makes he phone difficult to handle. Everything will be wire-connected, and I’ve prepared how I will connect the components together beforehand.

I did a minimalist design of the mobile app wireframes. After receiving feedback that I should check if mobile app development for Apple operating systems requires a $99 subscription for licensing, I ensured that I do not need that annual purchase to create an app for local environment use. The subscription is for distributing an app on Apple’s App Store, but we do not have such intentions for our project.

Next Steps

After confirming with team members, I will submit purchase forms for the above items. I will be planning in-depth as to how I’ll connect them all together. I will also be practicing how to utilize CAD software so eventually I can create a phone attachment that will hold all the components for 3D printing. It must not be too bulky, and be adjustable in tilt, which will necessitate studying current phone stands in the market for comparison and development. Since I established the wireframes for the mobile app, I will begin developing the front-end on Xcode. When I have time, I’ll also be delving into the back-end aspects to identify how I can get the app to connect to a cloud database. Further in the timeline, after I get the hardware parts, I will try to connect the Arduino with the mobile app via bluetooth and test the app to manipulate the display monitor.

Progress

I did a little more comparative research between Arduino and Raspberry Pi to ensure that using an Arduino module is right for our project. The articles support using the former, particularly the Arduino UNO, because of user-friendliness, energy efficiency, simplicity, and diversity. Moreover, it can achieve bluetooth as well as seamless connection to LCD displays. However, the specified unit is too bulky for our purposes so I looked into the other numerous Arduino hardware available. I have honed in on the Nano 33 BLE. It is much smaller, capable of machine learning in case we need it, and has bluetooth features. It also has a Flash memory of 1 MB, which is enough for storing simple text translations, which would take at most a couple KB. I believe it can be coupled with 2.8-3.5 inch LCD displays; I examined its data sheet and since it can connect to 16×2 displays, I’m expecting it can do the same with wider screens. Backup Arduino I’m considering is the Arduino GIGA display bundle which consists of an Arduino GIGA R1 wifi and a GIGA Display Shield. It packages the board and display together and is relatively flat, but its very complex and powerful so not necessarily compatible with our project.

I have setup the Xcode platform to prepare for Swift programming and the Arduino application in my computer. I also studied on 1. how to develop an app that can control the Arduino via bluetooth, and 2. How to connect to cloud storage through the app. Further plans for both are addressed in the “Next Steps” section.

Next Steps

A concern is how to attach a chargeable battery to the Arduino so it doesn’t need to be constantly plugged for electric power. Further investigation needs to be done to find a charger that won’t fry the board and is small/flat/sleek enough. The integration between battery and Arduino needs to be added as a task. Moreover, I need to decide on what LCD display to get so I can determine whether a Nano can be wired to it. 

For Arduino control with mobile app, the first step will be to design a plain UI. Then once Arduino is acquired, I will test bluetooth capabilities probably with LEDs or temperature sensing, which will then lead to testing text transmissions.

For cloud data retrieval from app, I identified Firebase, an application dev platform from Google that performs backend cloud computing services. I found a guide on how to install and use Firebase SDK (software dev kit) in Xcode, but cloud storage implementation needs to be discussed more with team members as this concerns retrieving ML and CV data for app usage.