Posted on

Fiona’s Status Report for 11/16/2024

This Week

User Interface Testing [1][2][7]

This week, I began by coding a testing program for the user interface. Since the eye-tracking is not ready to be integrated with the application yet, I programmed the backend to respond to commands from the keyboard. Via this code, I was able to test the functionality of the progress bars and the backend that tests consistent command requests.

In order for the user to pick a command with this program, they have to press the corresponding key five times without pressing any other key intermediately, which will reset the progress bar.

It will hopefully be a quick task to integrate the eye-tracking with the application, based on the foundation of this code.

Here is a short video demonstrating some basic functionality of the program, including canceling commands: https://drive.google.com/file/d/16NyUo_lRSzgLYIwVOHCEuoKJ6CYjiKip/view?usp=sharing.

Secondary UI [3][4][5][6][7]

After that, I also made the integration between the primary and secondary UI smoother by writing a secondary window with the sheet music that updates as necessary.

In the interim, I also included a note about where the cursor is in the message box, since I haven’t yet figured out how to implement that on the image of the sheet music.

Pre-Interim Demo

On Wednesday, I demonstrated my current working code to Professor Bain and Joshna, and they had some suggestions on how to improve my implementation:

  • Highlighting the note length/pitch on the UI after it’s been selected.
  • Electronically playing the note while the user is looking at it (for real-time feedback, makes the program more accessible for those without perfect pitch).
  • Set up a file-getting window within the program for more accessible use.

I implemented the first functionality into my code [3][4][6], and attempted to implement the second, but ran into some issues with the python sound libraries. I will continue working on that issue.

I also decided to leave the third idea for later, because that is a more dense block of programming to do, but it will improve the accessibility of the system so it is important.

(With the new updates to the code, the system currently looks slightly different from the video above, but is mostly the same.)

Eye-Tracking

On Thursday, Peter and I also met with Magesh Kannan to discuss the use of eye-tracking in our project. He suggested we use the MediaPipe library to implement eye-tracking, so Peter is working on that. Magesh offered his help if we need to optimize our eye-tracking later, so we will reach out to him if necessary.

Testing

There are no quantitative requirements that involve only my subsystem; the latency requirements, for example, involve both my application and the eye-tracking, so I will have to wait until the subsystems are all integrated to start those tests.

However, I have been doing more qualitative testing as I’ve been writing the program. I’ve tested various sequences of key presses to view the output of the system and these tests have revealed several gaps in my program’s design. For example, I realized after running a larger MIDI file from the internet through my program that I had not created the logic to handle more than one page of sheet music. My testing has also revealed some bugs having to do with rests and chords that I am still working on.

Another thing I have been considering in my testing is accessibility. Although our official testing won’t happen until after integrating with the eye-tracking, I have been attempting to make my application as accessible as possible during design so we don’t reveal any major problems during testing. Right now, the accessibility issue I need to work on next is opening files from within the program, because using an exterior file pop-up necessitates a mouse press.

Next Week

The main task for next week is the interim demo. On Sunday, I will continue working to prepare my subsystem (the application) for the demo, and then on Monday and Wednesday during class, we will present our project together.

The main tasks after that on my end will be continuing to work on integrating my application with Shravya’s and Peter’s work, and also to start working on testing the system, which will be a large undertaking.

There are also some bug fixes and further implementation I need to keep working on, such as the issues with rests and chords in the MIDI file and displaying error messages.

References 

[1] tkinter.tkk – Tk themed widgets. Python documentation. (n.d.). https://docs.python.org/3.13/library/tkinter.ttk.html

[2] Keyboard Events. Tkinter Examples. (n.d.). https://tkinterexamples.com/events/keyboard/

[3] Overview. Pillow Documentation. (n.d. ). https://pillow.readthedocs.io/en/stable/

[4] Python PIL | Image.thumnail() Method. Geeks for Geeks. (2019, July 19). https://www.geeksforgeeks.org/python-pil-image-thumbnail-method/

[5] Tkinter Toplevel. Python Tutorial. https://www.pythontutorial.net/tkinter/tkinter-toplevel/

[6] Python tkinter GUI dynamically changes images. php. (2024, Feb 9). https://www.php.cn/faq/671894.html

[7] Shipman, J.W. (2013, Dec 31). Tkinter 8.5 reference: a GUI for Python. tkdocs. https://tkdocs.com/shipman/tkinter.pdf

Posted on

Shravya’s status report for 11/16/2024

My newest update is that I am now able to use GPIO commands to drive 13 solenoids! Here is a video demonstrating a dummy sequence.

https://drive.google.com/file/d/15j3L2sPWW3enOPX8U6tXd_phpCFsdEaZ/view?usp=sharing 

And here is the full firmware script I used to do so: https://docs.google.com/document/d/1EplQO1Y9GYS4SYbjHKwGDTp9yScVESmYp59CTWjzwfk/edit?usp=sharing

  1. Hardware Challenges:
    • I spent many, many hours this week getting the solenoid circuitry to work consistently and reliably. Last Friday, it was mostly functional (except for an issue where as solenoids actuating for a while after disconnecting from power). The next time we tried to use our circuit, it was suddenly not working (even though we didn’t change anything on our breadboard). This is because some wires got bent while I placed the breadboard in my bag. Also, I noticed that the tip of the positive terminal wire of one of the solenoids I am using is frayed thus doesn’t adhere to breadboard holes very well- it can slightly shift if I don’t take the routine effort to make sure it is plugged in COMPLETELY.
    • The DC power supply I usually use in HHA101 was being used by someone else, so I tried to use an analog DC power supply to input 3.3V into my MOSFET gate. Nothing happened at all and I spent a lot of time debugging the circuit, component-by-component. with an oscilloscope etc. before I thought of testing if this power supply is reliably outputting at all. Using an oscilloscope, I found this power supply was simply broken (it had fluctuating voltages from the scale of microvolts to 1 volt). Ultimately I was able to get back the bench supply I use and know to be functional.
    • I tried to use a function generator to apply a square wave gate input but saw that all the BNC connectors required for this were broken
    • I realised that if I am not inputting 3.3V into the gate, it still has to be connected to something (ideally ground through a 10K ohm pull-down resistor) to avoid a floating voltage at the gate. This eliminates unreliable operation. Since I am not able to find those resistors today, my workaround was keeping the 3.3V gate input on, while using the 12V power supply to dictate whether the circuit was on or not.
    • I moved everything onto the other half of our breadboard in case we caused some damage underneath the breadboard
    • Earlier this week, when we realised that our circuit wasn’t working, we tried some troubleshooting. I used a different MOSFET than the one I ordered (one of my partners kept some components after 18100).  I realise now this isn’t suitable. This ZVN3306A MOSFET doesn’t fully turn on at 3.3V (it seems it operates in triode mode) and is not built to drive large currents which are critical for activating solenoids. The ZVN3306A MOSFET overheated and there were delays before I realised I should switch back to the IRLZ44NPBF. The IRLZ44NPBF is a logic-level MOSFET which completely allows for saturation operation at a Vgs of 3.3V, whereas the ZVN3306A can require over 4 V.
  2. Software Challenges:
    • The Python MIDI parsing code still needs some debugging to handle chords properly, which has taken longer than expected. I am waiting for Fiona to compose a new midi file and send that to me.

Goals for Next Week

  1. Hardware Improvements:
    • Add pull-down resistors from gate to ground to stabilize the gate input permanently.
    • Measure the solenoid current during operation and make sure it is as expected.
  2. Finish MIDI Parsing:
    • Fix any remaining issues with the Python code, especially with the use of chords.
    • Make sure the output is formatted correctly so the STM32 can process it. I am thinking of changing my MIDI-parsed output from a string format to a 2d-array to better handle transmitting chord data to the microcontroller (a string would only show every event serially, but for chords we need to portray that some notes play in parallel)
  3. Work on Firmware:
    • Test the STM32 firmware I wrote to handle UART communication and actuate solenoids based on received MIDI data.
    • Arrange solenoids on a piano/keyboard and verify that their force when being driven by GPIO signals is adequate to push keys.
  4. Prepare for Integration:
    • After the demo, I must make sure that Fiona’s MIDI-file generation and my MIDI-parsing work together seamlessly and handle all edge cases.
Posted on

Team Status Report for 11/09/2024

This week our team worked on preparing for the interim demo, which is the week after the next. We all have individual tasks to finish, but have also begun work on integrating those tasks. 

Shravya began this week with MIDI-parsing code that was capable of accurately parsing simpler compositions (by integrating and cross-verifying with Fiona’s MIDI compositions), and since then has identified some edge cases. These include handling rest-notes (which she has successfully been able to resolve) as well as overlapping notes (which she is still working on). She worked with Peter to ensure that all components of the solenoid control circuitry are functioning properly. 

Fiona worked more on debugging the code to make MIDI files. She also worked on some other miscellaneous tasks (see report).

We are conversing with Marios Savvides and Magesh Kannan who are CV + eye-tracking biometrics experts referred by Professor Bain for guidance on our eye-tracking system.

Right now, our biggest risk as the interim demo approaches is that we discover issues while integrating the eye-tracking, the application and the physical hardware. We are hopefully well-prepared for this, because we have been working to coordinate along the way.

Posted on

Fiona’s Status Report from 11/09/2024

This Week

MIDI Backend and User Interface [1][2]

I realized that my MIDI code did not cover every edge case. I updated it to make it possible to add chords to and delete them from the composition. I also added the functionality to handle cases in which the sheet music extends beyond one page and I had to update the code for editing pre-existing MIDI files so that they would not be overwritten every time they were opened and edited.

After that, I had to update my final UI to allow for chords (a new button was necessary), see below.

I also identified some errors with the logic having to do with the rest-notes in the composition, but I wasn’t able to identify what the issue was after some debugging, so I will have to continue working on that next week.

Secondary Device

I am currently using Sidecar [3] to make my iPad the secondary device to display the sheet music of the composition during testing, but it is not very accessible because it requires moving the computer mouse to update the image and also requires a recent version of an iPad and a Mac, so I am considering other options. 

Miscellaneous Tasks

There were a few smaller miscellaneous tasks I worked on this week:

I started researching some different eye-tracking options on GitHub, because our previous option hasn’t worked out yet. I’ve identified some options, but haven’t started testing with or integrating with any of them.

I attempted to figure out how to highlight notes in the sheet music (to indicate to the user where their cursor is) and also attempted to fix some alignment issues with the sheet music, but am still working on both of those tasks [4].

Next Week

Next week we will continue preparing for the interim demo, so my main goal will be integrating all of the systems together and continuing to debug so that we can demonstrate a working system.

There is also a lot of remaining functionality to wrap up with the backend code (coordinates to commands and UI updates, mostly), so I will work on that as well. 

References

[1] Overview. Mido – MIDI Objects for Python. (n.d.). https://mido.readthedocs.io/en/stable/index.html

[2] Standard MIDI-File Format Spec. 1.1, updated. McGill. http://www.music.mcgill.ca/~ich/classes/mumt306/StandardMIDIfileformat.html

[3] Apple. (n.d.). Use an iPad as a second display for a mac. Apple Support. https://support.apple.com/en-us/102597 

[4] BYVoid. (2013, May 9) MidiToSheetMusic. GitHub. https://github.com/BYVoid/MidiToSheetMusic

Posted on

Shravya’s Status Report for 11/09/2024

  • MIDI parsing code:
    • Significant progress has been made in parsing MIDI files using Python and the mido library.
    • As mentioned in my report for LAST week, I had already implemented parsing logic to capture note-on events with their respective durations. I had ran some very small-scale tests on my own Logic Pro X compositions to verify that a bare-bones version of this code was functional
    • This week, as part of integration testing as well as addressing edge cases, I made sure to test on some MIDI files specifically created by Fiona’s MIDI generation code that our eyetracking system will use. Since we do not have eyetracking implemented yet, she has a temporary function that allows her to click notes on her UI (eventually, eye commands will take the place of this). She put the generated MIDI file into a sheet music converter and verified that the produced sheet music looks accurate. I loaded her MIDI file into my parsing code and was successfully able to match with what she intended to compose. Below is my terminal output. Bear in mind, this is a simple and short song with notes that have all equal lengths.
    • Next, I moved onto testing larger and more complicated songs. I decided to use sample MIDI files from a Github page:  https://github.com/urswilke/miditapyr/ 
    • This is where I’ve encountered issues. First,  I realised my code was outputting some zero-duration notes (which doesn’t make sense), but the list of pitches was accurate. I attempted to fix it and was able to be rid of zero-duration notes but now, every time a rest-note was encountered, 2-3 new phantom notes appeared as well.
      • I investigated potential causes related to the mido library’s use of relative timing (instead of each note possessing an absolute time value, it possesses the time since the previous note), which probably is still where the bug lies.
  • Research and troubleshooting on MIDI parsing code:
    • I explored additional resources to understand mido library limitations and potential solutions, including analyzing third-party code (Reddit, Github) and documentation from the mido developers on handling absolute vs. relative MIDI timing.
    • This may be an issue for Fiona and I to solve together. We realised that we both are still confused on mido’s nature of handling chords. It may be good to consolidate our understanding together.
    • I think there’s a possibility that even if my parsing code fails on sample third-party MIDI files, it may be still be perfectly suitable for our project. What’s important is that it’s compatible with the way Fiona generates MIDI files, and she might find a way to do it in a more intuitive manner than the person who generated the Github files.

For reference, I am attaching my code. The first version (produces accurate pitches but some zero-duration notes) is currently commented. The second version (produces unwanted phantom notes) is not commented. WordPress is not allowing me to attach the .py file for security reasons, so I’m attaching it as a PDF: current python midi parsing code

  • Hardware Preparation:
    • Set up initial hardware for solenoid testing, including connecting a single solenoid to a breadboard and verifying functionality.
    • Tested solenoid activation by inputting a square wave from function generator. We did observe expected behaviour, but encountered 1-2 unexpected actuations after disconnecting the power supply.
    • Verified that the MOSFET and flyback diode was installed correctly, ensuring protection against voltage spikes. Further investigation is underway to identify and prevent unintended post-power disconnection actuations. One idea I have is that a Zener diode may be needed additionally, to connect in series with the Schottky flyback.
    • Video of solenoid circuit successfully actuating: https://drive.google.com/file/d/1E0uiThGWiQmukMVjhNL5qg8HktBq021e/view?usp=sharing

Going forwards, I would really like to wrap up the MIDI parsing code by Monday or Tuesday, such that all edge cases (rest-notes and chords) are handled correctly and I am able to integrate/cross-verify with Fiona’s end. I will order an octave’s worth of solenoids asap. Once I wrap up MIDI parsing, I need to focus heavily on STM32 firmware code to have a bare-bones-functioning (hopefully, even more) hardware-end of the project by the interim demo on Nov. 18.

Posted on

Peter’s Status Report from 11/2/2024

This Week

This week was supposed to be the week that basic eye-tracking was implemented; basic eye-tracking has not yet been implemented. Much of this week was spent fiddling around with the eye-tracking software to try to get it to properly detect eyes. However, the current method is not very reliable at detecting eyes in the setting of a regularly lit environment with lights overhead; perhaps forward lighting would help, but this would be an odd addition given our desire to have an easy set-up with few parts. However, it may be beneficial to consider pivoting to using an eye-tracking camera to support the eye-tracking system. This would allow our eye-tracking software to be more accurate.

 

Next Week

I will do more research on eye-tracking cameras, and discuss this pivot point with our faculty and TA advisors. If we are in agreement, then I will pivot to using the eye-tracking camera to aid in the accuracy of our system; currently out system cannot consistently track both eyes.

Posted on

Shravya’s Status Report for 11/02/2024

This week:

  1. MIDI Parsing Code Development:
    • Built upon last week’s partial implementation of the MIDI parsing code. This week, I focused on completing the MIDI parsing functionality and structuring the code for readability and maintainability. In my last status report I said I want to have a bare-bones functioning MIDI-parsing feature this week and I think I have achieved that.
    • Utilized the mido Python library to process the MIDI file, extracting note-on events and their corresponding durations.
  2. Testing Framework Setup for Local Simulation:
    • Since the hardware components only got delivered yesterday, I set up a simulation environment to test the MIDI parsing logic without needing physical hardware.
    • Modified the Python code to print parsed note and duration data to the console, instead of transmitting over UART. This modification allows for verifying the output sequence and timing without hardware dependencies.
  3. Local Testing Execution:
    • Conducted initial local tests using some very simple sample MIDI files. Observed console output to confirm correct parsing of notes and durations.
    • Implemented timing simulation with time.sleep() to replicate the actual note durations, which helped in verifying the accuracy of timing based on MIDI note lengths.
    • Next week I need to work more on multiple test cases with more complex MIDI files to strengthen parsing capability under various conditions.
  4. STM 32 code which I have begun working on is designed to process MIDI commands received over UART and control solenoids accordingly. It involves use of STM32’s HAL library functions.
  5. Code Documentation and Clean-up:
    • Documented the code structure and added comments to each function, making it easier to understand and maintain.
    • Structured the parsing logic to handle only the necessary MIDI events (e.g., note-on events with non-zero velocity), improving code efficiency and readability.

Overall, this week had some significant progress in building the MIDI-controlled solenoid system. The next phase will focus on integrating the hardware components and refining the STM32’s control logic to create a robust, responsive system that accurately translates MIDI data into solenoid actuation. The hardware components got delivered a little later than expected (Friday, Nov 1st) but at least my efforts with testing MIDI-parsing functionality will allow us to proceed with more confidence when integrating the hardware.

Posted on

Team Status Report for 11/02/2024

On Sunday, we met as a group and discussed our individual ethics assignments. We identified some ways that our project could cause harm, such as malfunctioning after the user has become dependent on it for income or another need, or demonstrating racial bias, which we know is a grave issue in other facial recognition technologies. Then, we discussed our project with our peers during class on Monday, and they pointed out some more ethical considerations, such as the potential of the solenoids to break the piano and eye-strain from looking at the UI for long periods of time. This week, the biggest risk we have been considering are all of these ethical considerations.

We also worked on some tasks individually. Fiona continued to work on the MIDI backend, including integrating existing MIDI to sheet music code with our project. She also worked on the application frontend. Shravya worked to finish MIDI file parsing and set up a framework for testing this parsing feature on sample MIDI files, and began working on firmware (UART communication and actuating solenoids). Peter continues work on the eye tracking software and is considering using eye tracking cameras instead of computer vision to help increase the accuracy of the eye tracking program.

Posted on

Fiona’s Status Report for 11/02/2024

This Week

Ethics

On Sunday, I met with my group to discuss the ethical considerations of our project, and discussed again with our classmates on Monday for an outside perspective.

Secondary UI (Sheet Music Updates)

I downloaded the code I identified last week as being a candidate for MIDI to sheet music conversion [1][2], and also Mono, the framework the author used [3]. I had to make one simple edit to the makefile in order for the program to run, but otherwise the code was compatible with the most current version of Mono, despite being 11 years old.

From there, it was fairly straightforward to implement the functionality to convert the MIDI file to sheet music on button press. To finish this task, I added some code to make the image update in the system on each edit by the user. For file saving and running the executable, is used the os module in Python [4].

Below is an example of what the sheet music output might look like for a simple sequence of notes.

Seeing the sheet music in front of me made me realize that my program had been saving the MIDI notes in the wrong way. The note pitches appeared to be correct, but the lengths were note, and rests appeared that I had not placed.

MIDI File Updates [9]

Because of this, I had to go back into my code from last week and identify the issue. I examined the Mido messages from one of the example MIDI files in BYVoid’s repository [1] against the sheet music it generated, and discovered I had misunderstood the Mido “time” parameter; I thought it was relative to the entire piece, but rather it is relative to the note itself (so, notes start at 0 unless preceded by a rest). After fixing that error in my code, it appears that time and pitch are both correct.

I also added the functionality to create a new MIDI file from within the UI [4], which will allow the user to create multiple compositions without opening and closing the application. Additionally, I coded a function that allows the user to open an existing MIDI file from the UI, using a Tkinter module, filedialog [5].

Finally, I added the code to move the cursor backwards and forwards in the MIDI file, inserting and deleting notes at those locations. This marks the completion of the MIDI file responses task on the Gantt chart.

Frontend

Next, I started working on finalizing the frontend code for the project. Previously, I had been using a UI for testing responses with buttons, but we will also need a final UI for the eye-tracking functionality, so I started writing that, see below [6][7].

Among other changes, I added progress bars, which is a widget that the Tkinter library offers [8], above each command in order to make it easier to add the functionality to show the user how long they have to look at a command.

Right now the UI is pretty simple; I will ask my group if they think we should incorporate any colors or other design facets into it. I also would like to test the UI on other machines to ensure that the size-factor is not off on different screens.

UI Responses

In the UI, I set up some preliminary functionality for UI responses, like a variable string input to the message box, and each of the progress bars. I did not make other progress on the UI responses.

Next Week

I am still a little behind in the Gantt chart, but making good progress. From the previous tasks, I still need to complete:

  • the integration between the MIDI to sheet music code and our system, such that the current cursor location is marked in the sheet music. This will require me to update the MIDI to sheet music code [1].
  • the identification of the coordinate range of the UI commands, which can be done now that I have a final idea of the UI.
  • the coordinates to commands mapping program, and loading bars. I’ve outlined some basic code already, but I will need to start integrating with the eye-tracking first to make sure I’ve got the right idea of it.
  • error messages and alerts to the user.

In the Gantt chart, my task next week is to integrate the primary and secondary UI, so I will work on that.

Another thing I want to do next week which is not on the Gantt chart is organize my current code for readability and style, and upload it to a GitHub repository, since there are more and longer files now.

References

[1] BYVoid. (2013, May 9) MidiToSheetMusic. GitHub. https://github.com/BYVoid/MidiToSheetMusic

[2] Vaidyanathan, M. Convert MIDI Files to Sheet Music. Midi Sheet Music. (n.d.). http://midisheetmusic.com/

[3] Download. Mono. (2024). https://www.mono-project.com/download/stable/

[4] os – Miscellaneous operating system interfaces. python. (n.d.). https://docs.python.org/3/library/os.html

[5] Tkinter dialogs. Python documentation. (n.d.). https://docs.python.org/3.13/library/dialog.html

[6] Shipman, J.W. (2013, Dec 31). Tkinter 8.5 reference: a GUI for Python. tkdocs. https://tkdocs.com/shipman/tkinter.pdf

[7] Graphical User Interfaces with Tk. python. (n.d.). https://docs.python.org/3.13/library/tk.html

[8] tkinter.tkk – Tk themed widgets. Python documentation. (n.d.). https://docs.python.org/3.13/library/tkinter.ttk.html

[9] Overview. Mido – MIDI Objects for Python. (n.d.). https://mido.readthedocs.io/en/stable/index.html

Posted on

Team Status Report for 10/26/24

Team Status Report for 10/26/2024

This week was mostly spent on individual work. Fiona worked on the UI responses to update the MIDI file in the system, locating a MIDI to sheet music conversion software and mapping coordinates to responses in the code. Shravya worked on the MIDI-to-firmware conversion code, and devised small-scale testing plans to ensure functionality of components that will arrive this week. Peter worked on the 3D model for the box which will hold the components of our product.

More specifics are in the individual progress reports.

Our current major risk is that we are behind schedule, but we allocated slack time at the beginning of the semester for this scenario.

Next week the group will work on the ethics assignment together by meeting to discuss our responses.