Category: Jake’s Status Reports

This week, I helped Daniel bring up a desktop-environment kernel, as opposed to the console version we had been using prior,  which didn’t support Advanced Linux Sound Architecture needed to interface the MIDI controller. When this initially didn’t work, we resorted to streaming over USB through a serial monitor, on a laptop that could accept MIDI inputs.

So the main motivation behind my action this week was eliminating a huge source of end-to-end latency, and to get us to a point where user-testing is feasible. I now have people waiting to test our device, and this point of contention here is what’s stopping us from moving forward.

Now that the kernel is upgraded with the working drivers, we’ve begun working a program to parse MIDI CCs from the keyboard, and serve control signals to the FPGA. Once we’ve actually got our whole keyboard support final, we’ll need to a bit of frequency correction with the hardware-synthesis of oscillator tuning words, and make sure that accepting our controls doesn’t take DSP blocks.

We’ve found in synthesis that all DSP blocks are in use, consumed by multiplies, but we still have a lot of Logic Elements to make use of.

This week things move towards conclusion as we find ourselves thoroughly in testing phase. I’ve prepared a questionnaire and a queue of test-users to play with the device. We initially planned to conduct user-testing this week, but experienced challenges with the MIDI-keyboard connectivity.

The design uses the HPS ARM core on the dev-board to parse MIDI input over USB. At present, the linux kernel advertises support for ALSA, which the core was to parse, and pass as input to the FPGA in order to save LE’s for spending on the synthesis data-path instead of on decoding USB communication. While this worked for demonstration, streaming to the FPGA from a .mid file, the kernel we were running didn’t support the technology to accept the MIDI keyboard input.

Daniel and I are in the midst of trouble-shooting this block. I tried using the linux-socfpga github repository to rebuild the kernel with the functionality we need, making  boot-images and device tree structures that would work for the device, then sending them to the device over UART.

Daniel managed to get the keyboard working with a relaying with a terminal that does have ASLA for now to begin. I took FFTs of the each oscillator in time using VISA to stream SCPI commands to an oscilloscope over USB and verified correctness by looking at the FFT. In this way, we also verified that the filters were removing the appropriate spectral content.

As I’ve designed this semester, I’ve learned a great deal about Linux systems, and bringing up verilog. I’ve learned about VCS, and DVE as well as how musical synthesis is achieved. I’ve learned mostly through reading data-sheets for the DE10 standard online, watching other people’s projects online through youtube or instructables, and asking AI to explain mathematics, or source me things to read.

This week was the WaveShaper interim demo. In trying to bring up the envelope onto the FPGA, I encountered unexpected issues with noise. The envelope successfully modulated the volume of the oscillator, moving through each respective phase, but something about the multiplication to scale the output is off. I tried things like synchronizing the inputs, or changing which clock modules made use of, I tried synchronizing the oscillator and envelope to only advance when the I2S ip-block moved to the left-sample-ready, sending zeros on the right. I tried explicitly setting a clock frequency on the FPGA, or driving the modules at 50Hz, with an added enable line to update the output samples in-respect to the audio-clock rate. Shayaan figured out that samples sent to the IP-blocks should actually be offset-K, which I intend to synthesize ASAP. The oscillators I was testing with were unsigned, and initially sounded better than did signed implementations. I also tried various things with the multiplication, such as “pipelining” it, in case it were somehow that  samples were  being dropped. What was working best was simply casting values to signed, but explicitly converting to offset-K is the correct behavior.

This week we also ordered the MIDI controller for use in the final demo. Users will be able to send at-once four voices. Arbitration will hold 4 voices in play, but drop a key in release if a new one is pressed. I chose the NEKTAR IMPACT LX49 for the controller explicitly to make Daniel’s life easier with the MIDI-CC features. It has 8 dials, and 9 sliders with a bank. The bank is desirable because the keyboard will manage a “page-toggle” control feature without us having to handle arbitration on the limited FPGA fabric. If we have more user-tunable parameters than are dials on the keyboard, this internal bank will allow the user to shift through the set of parameters the hardware buttons control, and the keyboard will do this for us by sending different MIDI identifiers.

Verification here is more than technical, as a lot of the tuning of parameters need to make sense from a user’s perspective. The first step in verification is through inspection through VCS. A more rigorous verification will involve comparing this sample-by-sample to the MATLAB pipeline equivalent, but should be functionally the same. The MATLAB comparison will become especially important as we test the effects, and summed waveforms. The MATLAB comparisons will also be important as we combine elements, as we should get the same stream out, bit-by-bit if the same elements are staged the same way. The last piece is the subjective, focusing on my own parts, the part to be verified is the playability. The last piece isn’t rigorous so much as: “are the options presented meaningful?”. When the WaveShapers various parts are all finally assembled, “Can I forget the instrument is there and just play?”.

This week I added a counter-based clock-divider to produce an audio clock from the onboard 50MHz main. This approach won’t be final, as we’ll need to generate the audio clock higher in the system-hierarchy when the entire project goes onto the FPGA, but it was an effective workaround this week. 

Testbenching the envelope generator is producing coherent shapes, but adjusting ADSR values while the “key-press” signal is active affects the way the current voice moves through the stages. I am not confident on what “correct behavior” looks like here, and will need to do some exploring to decide what the design should do when we do things like shorten the attack in the middle of a really long attack-phase gradually rising in.

The majority of time this week was spent with Daniel working on synthesis. We developed a top-module to wrap The Waveshaper System-Verilog thus far, and connected Intel IP blocks to initialize the Wolfson Codec and drive the DAC line-out.  However, somewhere in the path we’ve got a bug, which I believe pertains to the I2C used to set up the DAC, as we’re using IP blocks to do this now. 

Additional time was spent resolving Quartus environment issues, which delayed synthesis and debugging. The next steps are to address the I2C configuration issues I suspect are present, validate with audio out, and to resolve the ADSR tests.

This week I focused again on implementing the ADSR envelope generator in verilog. I fleshed out a parent bank module, which generates an envelope per-voice with a generate loop. Every voice receives a key-press signal, and its own set of input parameters. While conventional synthesizers do not usually employ a per-voice envelope, it may be interesting to see what this could allow from a sound-design perspective, and supporting it in verilog is straightforward. In our final design, it’s likely I’ll either bundle all the voice signals together and run them in parallel through one-instance, or have only one instance of ADSR  controls for all the generators. Either way, this parent-level bank will support sweeping over the number of voices in our final design.

I revised the internal logic with the output level, and scaled the user-option input level in respect to the output level, as well as compiled a testbench. I still need to vary the state-update to be a parameterizable division of the clock, but this should be a quick-fix I’ll do tomorrow.

In addition to RTL, I helped Daniel get started flashing the FPGA through Quartus. There were a few points of confusion mounting JTAG, and flashing the image, but we sorted it out after some reading, and flashed test programs ensuring our line-out, audio-codec, and other parts are working. We started together on the top-level module to flash the verilog written so far.

This week I started implementation of the ADSR envelope generator which will control amplitude/volume of each synthesizer voice. On the final design, when voices remain to be filled, each key-press on the input MIDI-controller will correspond to the triggering of its own envelope generator. At present, the generator takes in attack, decay, sustain, and release parameters along with a key-press parameter. It generates a waveform which will be used to scale oscillator amplitude.

What I’ve written so far will detect key-press and release events, and transition through the different phases of the envelope. The envelope output-level is updated with an increment each clock-edge so that the slope of each stage in terms of level-change/unit-time can be tuned to user-levels without using floating-point operations.

The timing is still under-development, at present, the envelope updates each clock cycle, instead of having a separated control-timing signal. So for the coming week as I continue this design, I’ll separate this piece out so that the parameter tuning corresponds to the scale of audible-noise, and not the internal clock. After this is done, the envelope should be connected to the output of our wave-adder Shayaan was working on this week to see if we can then modulate amplitude upon a key-press.

I’ve opened a pull-request for what I’ve committed this week visible here: github link.

This week, we also finalized our design report.

In consideration of global factors, this synthesizer is designed as a standalone hardware instrument, as opposed to the most readily-available alternatives right now: software synthesizers requiring a PC. Our FPGA design can be operated without requiring access to a personal computer. Further, if the design were later implemented on a dedicated PCB, we could extend the availability as a low-cost device. Taking this approach would improve access for any musician or hobbyist without the means to invest in more powerful synthesizers.

This week I spent some time working through the configuration of the onboard CODEC to line-out. The I2S frame should be 32 bits per frame, 28 bits for left and right stereo. The frequency = sample_rate * bit-depth * 2. So we’ll  need to drive a minimum of 3MHz with the FPGA output as master, and the codec as slave. Any line out powered PA system will work.

This week I tentatively decided to move forward with the DE10’s onboard codec. It’s at least clear now that the codec will deliver 48KHz out at 16 bit-depth, using pulse-code-modulation. What remains to be done is confirm that the codec can access its next outputs from the Bram under real timing-constraints using I2C to talk to the board. I wrote a square wave in verilog which remains to be tested.

This week I prepared for and presented our Project Proposal!

Whilst Shayaan spent time determining which FPGA we should use, and Daniel focused on how we should implement MIDI, I spent my time figuring out what additional hardware may be necessary for the synthesizer. This coming week, I am to decide how the output after the FX channel should convert into L/R stereo ¼” outputs, as well as a driven 3.5mm headphone jack.

The precursory control-path we’ve devised has some details which will dictate the external hardware required. Take for instance, the implementation of our oscillators. We could have a control-path to generate the desired output oscillation, for instance, a sawtooth generated by an accumulator register, and adder. In contrast, synthesizers have also been designed which read pre-computed samples of a waveform from a read-only memory. Each choice enables different advantages. For instance, an “oscillator” reading out samples from memory would allow for the implementation of a “user-bank” where users could send their own arbitrary waveforms to be read-out in playback. However, basic saw, sine, square, triangle function oscillators would suffer from the regularity of the access. Some argue that the use of precomputed values changes the character of the synthesis because you’re getting the exact same value every single time. Yet a wavetable implementation would require such a memory. We will likely move forward with oscillators dedicated to each style: preserving the character of basic function oscillations in a subtractive synthesis, and using memory-indexing “oscillators” to support user-defined waveform readout.

The cost of supporting memory-indexed “oscillation”  is offset by the fact we may use the memory for other things too. Among the effects we’ve elected to implement are chorus and delay, which will likely require discrete memory to push parameters like max delay time, without compromising upon our generating sample rate or sample bit-depth. Certain filtration techniques use sliding-window algorithms, which may also benefit from expanded memory capacity. If we somehow managed to skate by without discrete memory for implementing any one of these oscillators or effects, using them in parallel would likely create issues anyway. To this end, one of us will decide which specific memory shall be used when we decide upon an FPGA dev-board. 

Now a little about taking the end of the synthesizer’s control path, and forming it into an audio output. Once we have a digital signal ready for output, we need to get the signal analog so that an amp or speaker or transducer-of-your-choice can take it as input. There are a lot of dedicated ICs available from manufacturers to assist in this task: my chief consideration at this time is an audio-codec from Texas Instruments. We’d ensure our output is a well-formed I2S data-stream (a different protocol than I2C, despite the similar name), drive power to chip, and have ready-to-go output for a speaker or pair of headphones. I am unsure whether this will be suitable for a ¼”  line out however, any circuitry required for our audio output I’ll have figured out by the end of the coming week.

The TI slide-deck I’m looking at as I consider audio-codecs is available  here.