Team E0: Sonicam

This week I mainly worked on the network driver and microphone board hardware.

Last week, there was a problem that emerged with the network driver dropping up to 30% of the packets being transmitted from the FPGA, I spent most of this week working on resolving that.  The library being used previously was the “hypermedia.net” java library, which works well for low-speed data, but does not buffer packets well, and this was causing most of the drops.  By switching to linux and using the regular C network library, this problem was eliminated, though it required rewriting the packet processing and logging code in C.

The next problem was moving this data to a higher-level language like java, python, or matlab to handle the graphics processing.  Initially, started looking into ways to give both programs access to the same memory, but this was complicated, not very portable, and difficult to get working.  Instead, I ended up deciding on using linux pipes/fifos, as they use regular file I/O, which c and java of course support very well.  One small problem that emerged with this had to do with the size of the fifo, which is only 64kB.  The java program had some problems with latency relative to the C program, so the FIFO was getting filled, and it was dropping readings.  To get around this, I modified the C program to queue up 50,000 readings at once, and put them into a single call to fprintf, and the java program reads in a similar way before processing any of the readings.  In this way, the overall throughput is improved, by having just 20 large, unbroken transfers per second, rather than several thousand smaller ones.  This does introduce some latency, though only 1/20th of a second which is easily tolerable, and takes more memory, but only a few megabytes.

Progress on the hardware has mainly been in figuring out the process for manufacturing all the microphone boards.  There were initially some problems with the reflow oven blowing microphones off the board while the solder was still molten.  The fan used to circulate air to cool the chamber after it has reached its peak temperature has no speed control, and is strong enough in some areas to blow around the relatively large and light microphones.  So far, I have gotten the first few test microphones to work by reflowing them by hand with hot air, which worked but took a significant amount of work per microphone, so it may not be a viable solution for the whole array.  I have started working on adding speed control and/or flow straighteners to the reflow oven fan as well, though I suspect I’ll be in for a long day or two of soldering.

With the working test board, I was able to use the real-time visualizer to do some basic direction finding for a couple of signals, which were extremely promising:

5KHz, high resolution, centered in front of the array (plot is amplitude vs direction)

5KHz, high resolution, off-center (about 45 degrees)

Low resolution, 10KHz centered in front of the array

10KHz, low resolution, about 15 degrees off center.

Next week I mainly plan to focus on the hardware, mainly populating all of the microphones and making the wiring to the FPGA.

This week I mainly worked on updating the FPGA firmware and computer network driver.  Boards arrived yesterday, but I haven’t had time to begin populating them.

Last week, the final components of the network driver for the FPGA were completed, this week I was able to get microphone data from a pair of microphones back from it, do very basic processing, and read it into a logfile.  This seemed to work relatively well:

source aligned at 90 degrees to the pair of elements (white line peaks near 90 degrees, as it should)

source aligned at 0 degrees to the pair of elements (white line has a minimum near 90 degrees, again as it should)

However, these early tests did not reveal a problem in the network driver.  Initially, the only data transmitted was the PDM signal, which varies essentially randomly over time, so as long as some data is getting through, so it is very difficult to see any problems in the data without processing it first.  Several days later when testing some of the processing algorithms (see team and Sarahs updates), it quickly became apparent that something in the pipeline was not working.  After checking that the code for reading logfiles worked, I tried graphing an FFT of the audio signal from one microphone.  It should have had a single,  very strong peak at 5KHz, but instead, had peaks and noise all over the spectrum:

I eventually replaced some of the unused microphone data channels with timestamps, and tracked the problem down to the network driver dropping almost 40% of the packets.  While Wireshark verified that the computer was able to receive them just fine, there was some problem in the java network libraries I had been using.  I’ve started working on a driver written in C using the standard network libraries, but haven’t had time to complete it yet.

Part of the solution may be to decrease the frequency of packets by increasing their size.  While the ethernet standard essentially arbitrarily limits the size of packets to be under 1500 bytes, most network hardware also supports “jumbo frames” of up to 9000 bytes.  According  to this : https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=6&ved=2ahUKEwjrx8zAvM_oAhUvhXIEHcBHDmYQFjAFegQIBxAB&url=https%3A%2F%2Farxiv.org%2Fpdf%2F1706.00333&usg=AOvVaw1OaEu0ozlfTlaN1ZTfb-IW     paper, increasing the packet size above about 2000 bytes should substantially lower the error rate.  So far I’ve been able to get the packet size up to 2400 bytes using jumbo frames, but I have not finished the network driver in order to test it.

Next week I mainly plan to focus on hardware, and possibly finish the network driver.  As a stop-gap, I’ve been able to capture data using wireshark and write a short program to translate those files to the logfile format we’ve been using.

This week I mainly worked on the microphone boards and FPGA drivers.  The microphone boards were finished early this week, and ordered on Wednesday.  They were fabricated and shipped yesterday, and expected to arrive by next Friday (4/3).

As mentioned previously, this design uses a small number of large boards with 16 microphones each, connected directly to the FPGA board with ribbon cables.  This should significantly reduce the amount of work required to fabricate the boards and assemble the final device, though at the expense of some configurability, if we have to change some parameter of the array later.

As the schematic shows, most of the parts on the board will not be populated, but were included as mitigations to possible issues.  The microphone footprints were included in case we need to change microphones, and there are several different options for improving clock and data signal integrity (such as differential signaling and termination), if needed.  Most parts, particularly the regulator, are relatively generic, and so can be acquired from multiple vendors, in case there is a problem with our digikey order (which was also placed this week).

While working on the FPGA ethernet driver, one problem that came up was with the UDP checksum.  Unlike the ethernet frame CRC, which is in the footer of the packet, the UDP checksum is held in the packet header:

This means that the header depends on all of the data to be sent, which, means that the entire packet must be held in memory, then the checksum computed, then either the checksum modified in memory before transmission, or, during transmission the “source” of data has to be changed from memory to the register holding the checksum.  I didn’t particularly like either of these solutions, and so, came up with another.  I made the checksum an arbitrary constant, and added two bytes to the end of the UDP payload.  Those two bytes, which I termed the “cross”, are computed based on all of the data, and the header, so that the checksum works out to that constant.  The equation below isn’t exactly right, but gives the basic idea:

In this way, the packet can be sent out without knowing it’s entire contents ahead of time.  In fact, if configured to do so, could actually take in new microphone data in the middle of the transmission of a packet, and include that in the packet.  This greatly simplifies the rest of the ethernet interface controller, at the expense of a small amount of data overhead in every packet.  Given the size of the packets though, this tradeoff is easily worth it.

This coming week, I mainly plan to work on getting the information flow, all the way from the FPGA PDM inputs to a logfile on a computer working.  This was expected to be completed earlier, but the complexity of ethernet frames ended up being significantly greater than expected, and took several long days to get working.  At this point all the components of the flow are working to some degree, but do not work together yet.