Category: Matthew Kuczynski’s Status Reports

Since our MVP was achieved last week, I spent the beginning of the week testing our use case requirements and preparing for the final presentation where I was the speaker. Later in the week, I worked with Darwin on creating a threshold for distance for drags and averaging the position for consecutive data collections where the finger is down so that less update commands are sent and actions like drawing are smoother. Overall, I feel that I am on schedule since our MVP is working and we have completed testing. Next week, I plan to work on the algorithms for two finger gestures such as zoom.

This week we worked on integrating our subsystems, so much of my work involved testing the Arduino and Python serial code that I had finished last week and debugging it. Previously, I had not been able to test my code because the PCB wasn’t finished, so this week I was able to get it working properly. Then, I worked on getting the testing metrics for our requirements so that they could be reported in the final presentation. Since we have our MVP working, I am definitely on track. Next week, I will now work on adding additional features to our project like multi-finger functionality and improving the speed of the single touch function.

This week was difficult for me because I had COVID, so I was unable to meet with my group and did not feel good Monday – Wednesday. However, I was able to make significant progress on the software at the end of the week. My primary goal was to finish the Arduino script, which I accomplished, however it has not yet been tested due to some issues with PCB that existed until about an hour ago. Additionally, I wrote the new Python serial script, so the arrays of photodiode values can be communicated to the Python side. Previously, we had decided that the photodiode values should be communicated as a set of bit-encoded integers rather than a string to improve speed. Since the maximum integer size is a 32-bit long, three were needed since we have a 56 by 31 array (two with 28 for the top and one with 31 for the right). I also developed a protocol to know which values is being sent, which is to send either a -1, -2, or -3 before each value based on which of the three longs it represents. Since we never use all 32 bits for our longs, these negative values should never appear as the encoded photodiode values. Overall, I believe that I will be able to finish the testing in time, so I think that I am on schedule as long as I put enough work in this week and there are no more major hardware issues. Next week, I will debug my Arduino script, the Python serial code, and then work on multi-finger functionality.

This week I worked on preparing for our demo, so I worked with Darwin and finished writing some test scripts could should that our two subsystems could work together. Additionally, I worked with both Matt and Darwin to write a test that went through all three subsystems, which was demonstrated at our demo. These tests revealed flaws in my subsystem, so I also spent a significant amount of time debugging the code that I had already written. Once we had our demo working, I worked on the Arduino script for LED control and photodiode readings. Overall, I feel like I am on schedule since we made significant progress with the integration this week, however, we need the decoders to arrive or I will quickly fall behind. The biggest thing that I have left involves physical testing, which requires the hardware subsystem to be fully functioning. Next week, I plan to finish the Arduino script for LED control and photodiode readings. If the decoders come in, I will also begin physical testing, otherwise I will begin to work on multi-finger functionality.

With the PCBs finally coming in, I spent the beginning of the week with my team soldering the components onto the boards in TechSpark. This involved placing the stencil onto each board, spreading solder paste onto each stencil/board, manually placing each component, and placing the boards in the oven. Later, Darwin and I tested the communication from the Arduino, through my subsystem, to the VM that he is testing on, which worked successfully. Then, we began designing a series of tests that will allow us to test the detection algorithms and touch functionality. The idea of theses tests is that rather than getting physical data (since this is still not ready), we will randomly generate realistic data that can be detected, categorized, and sent to the touch functionality. I feel that I am on track, and although the delay on the decoders shipping has shifted the schedule back a bit, our slack time should allow me to still finish on time. Next week, I plan to finish these tests and debug the software with Darwin, and to work on the soldering once the decoders come in.

This week, my main focus was to continue the software integration. As I continued to combine my software with Darwin’s, I realized that some major decisions about the overall software design needed to be made. Therefore, much of the work that I did this week involved overall planning and design of the project’s software system. The biggest question was how exactly a touch event would be communicated to the screen control software. In the end, I chose to design a system where after subsystem B detects the touch type and position, both an event type and screen position are communicated to subsystem C. Most of the time, the event type will be “no touch”, in which case the position can be ignored. Overall, I think that I am on track to finish my parts with some slack time built in, and the fact that the PCB + components came in means that we can move forward as a team. Next week, my focus will shift to the soldering that needs to be done with the PCBs, and there may be a chance to do some physical testing near the end of the week depending on how the soldering goes.

This past week, I began working on the finger detection algorithms, which were explained in some depth in the design review report. This is somewhat difficult without being able to physically test, so this is mostly a framework for now. Additionally, I began integrating my portion of the software with Darwin’s, which is combining subsystems B & C from the report. Next week, I plan to work on the frame design and continue the integration of the two software systems. Additionally, I will work with the rest of the team on the PCB when it arrives, which should be soon. The majority of the work that I still need to contribute individually on this project involves tuning the detection algorithms through physical testing. So far, I feel that I am on schedule, but I will need the PCB to complete most of the work that still needs to be done.

This week I continued my progress with the Python-Arduino interface and worked on routing the PCBs. Last week, I learned that the pyFirmata interface that we intended to use would not be fast enough to loop over all of the LEDs, so I spent a large portion of this work exploring alternative options. Eventually, I discovered that uploading the code that loops over the select line values to the Arduino is unsurprisingly much faster than using pyFirmata. In order to read the values from the Arduino, I developed a script using the Python serial library that did string parsing to get the digital values and the select value. The PCB routing task proved to be tough due to width of the PCB, and the Auto-router was unable to complete the task on its own. While some of the PCBs have been completed, the task remains ongoing. Overall, I think that I am on schedule since I am keeping up with the tasks in the Gantt chart. Next week, I plan to continue working on PCB layout and the breadboard-Arduino integration.

This week, my main task was to develop and debug the Arduino software that could recognize the basic touch functionality. As explained last week, I chose to use the pyFirmata library to interface the Arduino with Python, so I started the week by preparing the software for actual testing later in the week. Once the parts order came in, Matt S. was able to build a circuit that we connected to the Arduino Mega, so that the software could be tested. As expected, there were many issues with the software that I had developed at the start of the week, so the later part of the week was mostly spent debugging. One struggle that I faced was learning how to use a pyFirmata iterator, which is built in to the pyFirmata library. At least to me, it was unclear that the iterator was necessary for the functionality that we needed, but once I properly added it to the code many problems were solved. The other large issue was that the time between writing the select line and reading the output was not large enough, so seemingly random values were being sent back to the Python code. For now, about 10 ms are needed in order for the values to be accurate, however we will need this to be faster in order to reach our MVP goal since we will need to loop over all of the LEDs. I feel that I am on schedule since I kept up with my Gantt chart goal this week. Next week, I will aim to help Matt S. with the PCB layout, work on frame sketches, and work on the Breadboard/Arduino integration.

This week I have been working on researching, testing, and prototyping the Python-Arduino interface for our project. Since we plan on controlling the screen functionality with Python, this interface is necessary in order to communicate the digital readings from the Arduino so that finger position can be determined. Additionally, we want to use Python to set the digital pins going into the decoder (see Matt S.’s diagram).

The most important part of my work this week was determining which platform/library was most suitable for our needs. Through my research, I determined the best interfaces were PySerial using Anaconda Prompt or similar, and pyFirmata. In the end, I found pyFirmata to be the best option because it is easy to use and felt more specific to Arduino in my opinion. The most important factor that I considered was if the interface would be fast enough to toggle through all of the combinations that we need. To do this, I tested how fast I could perform the standard Blink function on the Arduino by toggling digital pin 13 on and off as quickly as possible in a loop and recording the time that it took. A subset of the values that were recorded is included in the screenshot below. As you can see, the task usually took under 0.1 ms, although sometimes it took between 0.1-0.2 ms. Regardless, this should easily be fast enough since we are targeting a reading every 150 ms, and there will likely be less than 20 combinations per reading required in the end. This should provide sufficient time for the other calculations to be performed in each cycle. Finally, I was able to develop a small script that cycled through the combinations of D2-D4 pins that we plan to use (for now), as well as read from the D5 pin.

Overall, I am somewhat ahead of schedule on my tasks since we no longer deemed the simulation task to be necessary due to our switch from an analog approach to a digital one. We plan to adjust the Gantt chart to account for this, which will hopefully make our reach goal of multi-finger functionality more realistic.

Next week, I hope to combine my work with Darwin’s work in order to begin creating a Python code base for our project. Additionally, we hope to set up a basic physical circuit once our parts come in, so I will try to incorporate my work from this week into that.