Posted on

Fiona’s Status Report from 11/09/2024

This Week

MIDI Backend and User Interface [1][2]

I realized that my MIDI code did not cover every edge case. I updated it to make it possible to add chords to and delete them from the composition. I also added the functionality to handle cases in which the sheet music extends beyond one page and I had to update the code for editing pre-existing MIDI files so that they would not be overwritten every time they were opened and edited.

After that, I had to update my final UI to allow for chords (a new button was necessary), see below.

I also identified some errors with the logic having to do with the rest-notes in the composition, but I wasn’t able to identify what the issue was after some debugging, so I will have to continue working on that next week.

Secondary Device

I am currently using Sidecar [3] to make my iPad the secondary device to display the sheet music of the composition during testing, but it is not very accessible because it requires moving the computer mouse to update the image and also requires a recent version of an iPad and a Mac, so I am considering other options. 

Miscellaneous Tasks

There were a few smaller miscellaneous tasks I worked on this week:

I started researching some different eye-tracking options on GitHub, because our previous option hasn’t worked out yet. I’ve identified some options, but haven’t started testing with or integrating with any of them.

I attempted to figure out how to highlight notes in the sheet music (to indicate to the user where their cursor is) and also attempted to fix some alignment issues with the sheet music, but am still working on both of those tasks [4].

Next Week

Next week we will continue preparing for the interim demo, so my main goal will be integrating all of the systems together and continuing to debug so that we can demonstrate a working system.

There is also a lot of remaining functionality to wrap up with the backend code (coordinates to commands and UI updates, mostly), so I will work on that as well. 

References

[1] Overview. Mido – MIDI Objects for Python. (n.d.). https://mido.readthedocs.io/en/stable/index.html

[2] Standard MIDI-File Format Spec. 1.1, updated. McGill. http://www.music.mcgill.ca/~ich/classes/mumt306/StandardMIDIfileformat.html

[3] Apple. (n.d.). Use an iPad as a second display for a mac. Apple Support. https://support.apple.com/en-us/102597 

[4] BYVoid. (2013, May 9) MidiToSheetMusic. GitHub. https://github.com/BYVoid/MidiToSheetMusic

Posted on

Shravya’s Status Report for 11/09/2024

  • MIDI parsing code:
    • Significant progress has been made in parsing MIDI files using Python and the mido library.
    • As mentioned in my report for LAST week, I had already implemented parsing logic to capture note-on events with their respective durations. I had ran some very small-scale tests on my own Logic Pro X compositions to verify that a bare-bones version of this code was functional
    • This week, as part of integration testing as well as addressing edge cases, I made sure to test on some MIDI files specifically created by Fiona’s MIDI generation code that our eyetracking system will use. Since we do not have eyetracking implemented yet, she has a temporary function that allows her to click notes on her UI (eventually, eye commands will take the place of this). She put the generated MIDI file into a sheet music converter and verified that the produced sheet music looks accurate. I loaded her MIDI file into my parsing code and was successfully able to match with what she intended to compose. Below is my terminal output. Bear in mind, this is a simple and short song with notes that have all equal lengths.
    • Next, I moved onto testing larger and more complicated songs. I decided to use sample MIDI files from a Github page:  https://github.com/urswilke/miditapyr/ 
    • This is where I’ve encountered issues. First,  I realised my code was outputting some zero-duration notes (which doesn’t make sense), but the list of pitches was accurate. I attempted to fix it and was able to be rid of zero-duration notes but now, every time a rest-note was encountered, 2-3 new phantom notes appeared as well.
      • I investigated potential causes related to the mido library’s use of relative timing (instead of each note possessing an absolute time value, it possesses the time since the previous note), which probably is still where the bug lies.
  • Research and troubleshooting on MIDI parsing code:
    • I explored additional resources to understand mido library limitations and potential solutions, including analyzing third-party code (Reddit, Github) and documentation from the mido developers on handling absolute vs. relative MIDI timing.
    • This may be an issue for Fiona and I to solve together. We realised that we both are still confused on mido’s nature of handling chords. It may be good to consolidate our understanding together.
    • I think there’s a possibility that even if my parsing code fails on sample third-party MIDI files, it may be still be perfectly suitable for our project. What’s important is that it’s compatible with the way Fiona generates MIDI files, and she might find a way to do it in a more intuitive manner than the person who generated the Github files.

For reference, I am attaching my code. The first version (produces accurate pitches but some zero-duration notes) is currently commented. The second version (produces unwanted phantom notes) is not commented. WordPress is not allowing me to attach the .py file for security reasons, so I’m attaching it as a PDF: current python midi parsing code

  • Hardware Preparation:
    • Set up initial hardware for solenoid testing, including connecting a single solenoid to a breadboard and verifying functionality.
    • Tested solenoid activation by inputting a square wave from function generator. We did observe expected behaviour, but encountered 1-2 unexpected actuations after disconnecting the power supply.
    • Verified that the MOSFET and flyback diode was installed correctly, ensuring protection against voltage spikes. Further investigation is underway to identify and prevent unintended post-power disconnection actuations. One idea I have is that a Zener diode may be needed additionally, to connect in series with the Schottky flyback.
    • Video of solenoid circuit successfully actuating: https://drive.google.com/file/d/1E0uiThGWiQmukMVjhNL5qg8HktBq021e/view?usp=sharing

Going forwards, I would really like to wrap up the MIDI parsing code by Monday or Tuesday, such that all edge cases (rest-notes and chords) are handled correctly and I am able to integrate/cross-verify with Fiona’s end. I will order an octave’s worth of solenoids asap. Once I wrap up MIDI parsing, I need to focus heavily on STM32 firmware code to have a bare-bones-functioning (hopefully, even more) hardware-end of the project by the interim demo on Nov. 18.

Posted on

Peter’s Status Report from 11/2/2024

This Week

This week was supposed to be the week that basic eye-tracking was implemented; basic eye-tracking has not yet been implemented. Much of this week was spent fiddling around with the eye-tracking software to try to get it to properly detect eyes. However, the current method is not very reliable at detecting eyes in the setting of a regularly lit environment with lights overhead; perhaps forward lighting would help, but this would be an odd addition given our desire to have an easy set-up with few parts. However, it may be beneficial to consider pivoting to using an eye-tracking camera to support the eye-tracking system. This would allow our eye-tracking software to be more accurate.

 

Next Week

I will do more research on eye-tracking cameras, and discuss this pivot point with our faculty and TA advisors. If we are in agreement, then I will pivot to using the eye-tracking camera to aid in the accuracy of our system; currently out system cannot consistently track both eyes.

Posted on

Shravya’s Status Report for 11/02/2024

This week:

  1. MIDI Parsing Code Development:
    • Built upon last week’s partial implementation of the MIDI parsing code. This week, I focused on completing the MIDI parsing functionality and structuring the code for readability and maintainability. In my last status report I said I want to have a bare-bones functioning MIDI-parsing feature this week and I think I have achieved that.
    • Utilized the mido Python library to process the MIDI file, extracting note-on events and their corresponding durations.
  2. Testing Framework Setup for Local Simulation:
    • Since the hardware components only got delivered yesterday, I set up a simulation environment to test the MIDI parsing logic without needing physical hardware.
    • Modified the Python code to print parsed note and duration data to the console, instead of transmitting over UART. This modification allows for verifying the output sequence and timing without hardware dependencies.
  3. Local Testing Execution:
    • Conducted initial local tests using some very simple sample MIDI files. Observed console output to confirm correct parsing of notes and durations.
    • Implemented timing simulation with time.sleep() to replicate the actual note durations, which helped in verifying the accuracy of timing based on MIDI note lengths.
    • Next week I need to work more on multiple test cases with more complex MIDI files to strengthen parsing capability under various conditions.
  4. STM 32 code which I have begun working on is designed to process MIDI commands received over UART and control solenoids accordingly. It involves use of STM32’s HAL library functions.
  5. Code Documentation and Clean-up:
    • Documented the code structure and added comments to each function, making it easier to understand and maintain.
    • Structured the parsing logic to handle only the necessary MIDI events (e.g., note-on events with non-zero velocity), improving code efficiency and readability.

Overall, this week had some significant progress in building the MIDI-controlled solenoid system. The next phase will focus on integrating the hardware components and refining the STM32’s control logic to create a robust, responsive system that accurately translates MIDI data into solenoid actuation. The hardware components got delivered a little later than expected (Friday, Nov 1st) but at least my efforts with testing MIDI-parsing functionality will allow us to proceed with more confidence when integrating the hardware.

Posted on

Team Status Report for 11/02/2024

On Sunday, we met as a group and discussed our individual ethics assignments. We identified some ways that our project could cause harm, such as malfunctioning after the user has become dependent on it for income or another need, or demonstrating racial bias, which we know is a grave issue in other facial recognition technologies. Then, we discussed our project with our peers during class on Monday, and they pointed out some more ethical considerations, such as the potential of the solenoids to break the piano and eye-strain from looking at the UI for long periods of time. This week, the biggest risk we have been considering are all of these ethical considerations.

We also worked on some tasks individually. Fiona continued to work on the MIDI backend, including integrating existing MIDI to sheet music code with our project. She also worked on the application frontend. Shravya worked to finish MIDI file parsing and set up a framework for testing this parsing feature on sample MIDI files, and began working on firmware (UART communication and actuating solenoids). Peter continues work on the eye tracking software and is considering using eye tracking cameras instead of computer vision to help increase the accuracy of the eye tracking program.

Posted on

Fiona’s Status Report for 11/02/2024

This Week

Ethics

On Sunday, I met with my group to discuss the ethical considerations of our project, and discussed again with our classmates on Monday for an outside perspective.

Secondary UI (Sheet Music Updates)

I downloaded the code I identified last week as being a candidate for MIDI to sheet music conversion [1][2], and also Mono, the framework the author used [3]. I had to make one simple edit to the makefile in order for the program to run, but otherwise the code was compatible with the most current version of Mono, despite being 11 years old.

From there, it was fairly straightforward to implement the functionality to convert the MIDI file to sheet music on button press. To finish this task, I added some code to make the image update in the system on each edit by the user. For file saving and running the executable, is used the os module in Python [4].

Below is an example of what the sheet music output might look like for a simple sequence of notes.

Seeing the sheet music in front of me made me realize that my program had been saving the MIDI notes in the wrong way. The note pitches appeared to be correct, but the lengths were note, and rests appeared that I had not placed.

MIDI File Updates [9]

Because of this, I had to go back into my code from last week and identify the issue. I examined the Mido messages from one of the example MIDI files in BYVoid’s repository [1] against the sheet music it generated, and discovered I had misunderstood the Mido “time” parameter; I thought it was relative to the entire piece, but rather it is relative to the note itself (so, notes start at 0 unless preceded by a rest). After fixing that error in my code, it appears that time and pitch are both correct.

I also added the functionality to create a new MIDI file from within the UI [4], which will allow the user to create multiple compositions without opening and closing the application. Additionally, I coded a function that allows the user to open an existing MIDI file from the UI, using a Tkinter module, filedialog [5].

Finally, I added the code to move the cursor backwards and forwards in the MIDI file, inserting and deleting notes at those locations. This marks the completion of the MIDI file responses task on the Gantt chart.

Frontend

Next, I started working on finalizing the frontend code for the project. Previously, I had been using a UI for testing responses with buttons, but we will also need a final UI for the eye-tracking functionality, so I started writing that, see below [6][7].

Among other changes, I added progress bars, which is a widget that the Tkinter library offers [8], above each command in order to make it easier to add the functionality to show the user how long they have to look at a command.

Right now the UI is pretty simple; I will ask my group if they think we should incorporate any colors or other design facets into it. I also would like to test the UI on other machines to ensure that the size-factor is not off on different screens.

UI Responses

In the UI, I set up some preliminary functionality for UI responses, like a variable string input to the message box, and each of the progress bars. I did not make other progress on the UI responses.

Next Week

I am still a little behind in the Gantt chart, but making good progress. From the previous tasks, I still need to complete:

  • the integration between the MIDI to sheet music code and our system, such that the current cursor location is marked in the sheet music. This will require me to update the MIDI to sheet music code [1].
  • the identification of the coordinate range of the UI commands, which can be done now that I have a final idea of the UI.
  • the coordinates to commands mapping program, and loading bars. I’ve outlined some basic code already, but I will need to start integrating with the eye-tracking first to make sure I’ve got the right idea of it.
  • error messages and alerts to the user.

In the Gantt chart, my task next week is to integrate the primary and secondary UI, so I will work on that.

Another thing I want to do next week which is not on the Gantt chart is organize my current code for readability and style, and upload it to a GitHub repository, since there are more and longer files now.

References

[1] BYVoid. (2013, May 9) MidiToSheetMusic. GitHub. https://github.com/BYVoid/MidiToSheetMusic

[2] Vaidyanathan, M. Convert MIDI Files to Sheet Music. Midi Sheet Music. (n.d.). http://midisheetmusic.com/

[3] Download. Mono. (2024). https://www.mono-project.com/download/stable/

[4] os – Miscellaneous operating system interfaces. python. (n.d.). https://docs.python.org/3/library/os.html

[5] Tkinter dialogs. Python documentation. (n.d.). https://docs.python.org/3.13/library/dialog.html

[6] Shipman, J.W. (2013, Dec 31). Tkinter 8.5 reference: a GUI for Python. tkdocs. https://tkdocs.com/shipman/tkinter.pdf

[7] Graphical User Interfaces with Tk. python. (n.d.). https://docs.python.org/3.13/library/tk.html

[8] tkinter.tkk – Tk themed widgets. Python documentation. (n.d.). https://docs.python.org/3.13/library/tkinter.ttk.html

[9] Overview. Mido – MIDI Objects for Python. (n.d.). https://mido.readthedocs.io/en/stable/index.html