sopranino recorder

Baltimore Recorders.org


Networked Music Performance


It has been more than a year now since the tragic fire at the Cathedral of Notre Dame. It has seemed like a mere instant, and an eternity. So few good things have happened, it seems like an instant, as there is no history to account for the lapse of time. However, so few good things have happened that the passing of the time has seemed to take forever.

The damage to the cathedral is significant on so many levels. One of the smaller ones is music. The loss of the building represents a loss of a place of music. This loss ties into the topic of this page, the greater loss of a place of music in the age of the novel coronavirus.

With the stay at home orders on account of the COVID-19 outbreak many of you are missing playing in groups. Some of us have experimented with networked music performance, playing music with others over a network such as the internet. The most well known services are Skype and Zoom. Since the pandemic began, Google and Facebook have overhauled and promoted their video conferencing services. Another alternative is Jitsi. Anyone who has tried to use these services will know that they are not suitable for use in real-time musical collaboration. They suffer from significant amounts of audio latency, the delay in the audio signal reaching its destination. You may have experienced this in trying to have a conversation on one of these platforms. It makes it difficult to interact naturally with other users. For example, when trying to comment on what a person is saying, it takes time for the signal to reach the original speaker. Thus, they may continue speaking, or go on to another topic before they hear the comment. In addition to latency, these services also have other issues with the audio signal that make them less than desirable. Some of the services intermittently cut out the audio of some people in a group if multiple people are speaking. One reason for this is to save on bandwidth. (This will be discussed later.) As you can imagine, being tacet randomly is not compatible with music making. The quality of the audio signal on these services tends to be poor.

Some other folks may have even used an old fashioned approach and tried to play music collaboratively over the telephone. They have found that the phone system has many of the same problems of the online video conferencing software.

There have been several services that have been created in the new millennium that cater to real-time musical collaboration. These include:

Some of these are paid services, such as eJamming. Others are free, and some are open source. Of them, and one of the most prominent, is JackTrip which was developed at CCRMA of Stanford University. So if you want to try setting up a system yourself, you can.

While being a bit better than Zoom and the like, they still suffer from major latency issues. So, what is going on, where is the latency coming from, and how much latency is acceptable?

Well let us begin with the last part of the question, how much latency is acceptable. In psychophysics there is a concept known as the just noticeable difference (JND). This is the lower limit at which something can be differentiated 50% of the time. There has been some research in music regarding the perception of differences in note onset. A study by Friberg and Sundberg found that the JND of note onset when presented in the context of a regular rhythm was 6 ms. Thus, we do not need the latency to be less than 6 ms. However, anything greater than that may be noticeable. There have also been some studies on the effect of latency on musical performance. One such study is "Effect of temporal separation on synchronization in rhythmic performance" by C. Chafe, J-P. Caceres, M. Gurevich published in 2010. It investigates the effect of latency on the synchronous production rhythms. It describes one way latency of 8 ms to 25 ms as being 'natural' and allows for the stable production of a tempo. One way delays of 25-60 ms are characterized as 'challenging'. These delays cause the tempo to slow down. This can be counteracted by intentionally pushing the beat, and or ignoring the sound of the other performers. Latencies of approximately 60 ms or more result in playing accuracy rapidly falling off. The aforementioned strategies are no longer sufficient to maintain a mutually coupled tempo. Thus, this paper indicates that the upper limit is approximately 120 ms (two times the 60 ms one way latency). Further, it provides a limit of 50 ms in which more musical interaction is possible. There is of course much more to music than simply playing together at a given tempo. Thus, studies such as this do not fully address the question of how much latency is acceptable, and neither can this page. This is not surprising, as people have different use cases and individual differences, and thus the latency that is acceptable to them can vary. Amateurs usually want to enjoy some aspect of performance, whether it be hearing the music being played, the physical act of playing an instrument, social interaction with others, or the response from an audience. I suspect that for many, hearing the music while you are performing it is central to their enjoyment. As the latency increases the music sounds more an more distorted, and thus can detract from the listening pleasure. Initially latency sounds like a small amount of resonance. Then it can make a single performer sound like several performers, as multiple onesets become apparent. To me this happens by 10 ms. As the latency increases it sounds more like a small amount of reverberation. I find that this happens between 20 and 25 ms. Eventually it sounds like separate events, as opposed to an ensemble. Certain styles of music are more amenable to having latency than others. Fast tempi, staccato markings, instruments with clear attacks and little sustain, and large numbers of independent parts are two aspects of music that do not allow for much latency.

I am including several sound files to illustrate what different amounts of latency sound like. The first files consist of eight beats at 60 BPM. Each beat consists of two sounds. There is a given amount of latency between the sounds.

The second set of files illustrates what it sounds like when one musical part is delayed by a certain amount of latency.

The third set of files illustrates what it sounds like when an entire ensemble is playing and a second ensemble that is playing the same music is delayed by a certain amount of latency.

Now, where is the latency coming from. Well, it takes time for the audio signal to reach its destination. This is not something that we ordinarily think about, but it does take time. We all have experience with this. For example, there is a delay between the time we see a flash of lightning and hear the thunder. The good news is that the audio signal is not carried by sound waves, which are slow. Rather, it is transmitted electronically. This can in theory happen at close to the speed of light.

If you recall from school, the speed of light is 186,282 miles per second. This sounds promising. We then need to to multiply by 0.01 to see how far the light can travel in 10 ms. Move the decimal point and we get 1,862.82 miles. However, it is not good enough that the signal reaches another performer in time. That performers signal must also come back to us, so that there can be real-time interaction. Thus the distance must be halved. The result is 931.41 miles. So the other people in the group must be within 931 miles to collaborate in real-time without noticeable delay if we are to obey the laws of physics.

This still sounds pretty good. However, there is bad news. There are many other sources of latency. Electrical signals, or even light traveling through fiber optic cables do not travel at the speed of light. There is refraction and loss of signal. This can slow the signal to approximately 2/3 of its original speed, and thus is very significant. There is latency in the digital to audio interface (The device that connects the microphone to the computer or other audio equipment.) Modern systems the round trip latency is in the realm of 1.75 ms. This may not seem like a lot, but when you want the rountrip latency to be 10 ms or even 20 ms, 1.75 ms is a significant fraction. There is latency in the computer sending the signal. This can vary widely based on the computer architecture and software. There is latency in the telecommunication infrastructure, the switches in the telephone company's system. Many, many sources of latency. I will not go more into specifics here.

Even with all of this, it is still possible. However, there is another problem, and that is bandwidth. Although it is possible to send a signal across fiber optic cable in the requisite amount of time, that cable can only carry a limited amount of data per unit time. Audio, or even worse audio and video signals are very data intensive. Even the modern fiber optic cabling in the USA is not designed to carry that much data for a particular user. So unless you own the cabling, you are likely to be out of luck. Thus, as they say in the telecom industry, its the network. How ironic.

There is still some hope, and that is radio communications. Radio waves travel at the speed of light. It may even be possible to use CB radio for this purpose. It should be noted that in the USA, the FCC has prohibited the use of CB radio to broadcast music for the use of entertainment. However, if playing the music had other uses, such as healthcare, or education, then it is possible it may be permitted. You would need to consult a lawyer. Another possibility would be to apply to the government to become a small radio station, or other sort of telecommunication company such as a wireless isp, WISP or fixed microwave transmission. This would be an expensive and long term endeavor.

All of this to play some music.

UPDATE: July 2, 2020

I have been investigating setting up networked musical performance systems. As I mentioned before, there is software, such as JackTrip. There is also hardware. The missing piece is the transmission of the data, whether it be via a wired connection or a wireless one. It is not practical to set up a private network for most musical groups using your own wired connection, as deploying miles of wires may as well be impossible. Baltimore County has a dark fiber network. However, it does not extend to the residences of our members, and thus is not practical. The two most available networks are the internet and cell phones. I contacted several local data centers and have been conducting latency tests. Despite having a fiber optic connection from my location, and a short distance to one of the ISP's centers, and another short distance from there to the data center, the results were not great. Some local data centers performed better than others. The root of the problem is that the ISP sends the data to far flung places, dozens or even hundreds of miles away, and also the data takes many hops between routers along the way. Each hop usually adds at least 10 ms of latency, and typically adds 20 ms or more. So the issue is the routing of the signal. The ISP does not want to send the signal in an inefficient manner. The intervening hops are used to manage network congestion. If a local network is congested, a signal may have to wait for a long period of time to be sent on its way. Thus, the signal is sent further afield to avoid the wait. Thus, paradoxically it is sent further away with more hops to reduce the latency. The distribution of the signals is also used to maximize the usage of the entire network.

There seems to be a push by the telecom industry to reduce latency. There are more and more edge data centers, and companies are touting services that make use of 5G wireless. Amazon is creating AWS Wavelength and has partnered with Verizon and other telcos. There is talk of single digit ms latency, which would be wonderful. T-Mobile is also working on its own such product. 5G of course does not miraculously solve the problem. It is all a matter of whether the ISPs will decide to provide the low latency service. I will continue to update this page.

UPDATE: July 9, 2020

Amazon has rolled out its AWS Wavelength service. However, at this point it is only available in San Francisco and Boston. I will continue to update this page.

Web design copyright 2015 Michael Berger