Your browser is out-of-date!

Update your browser to view this website correctly. Update my browser now

×

Virtual Jam Session

As busy as some musicians are, it's a wonder that any of them find time to get together to jam with other artists, let alone get into the studio to record those sessions. That's part of the reason why researchers at Rensselaer Polytechnic Institute in Troy, N.Y., and McGill University in Montreal are developing a system that would allow jam sessions to span a country or the world.

Virtual Jam Session

As busy as some musicians are, it’s a wonder that any of them find time to get together to jam with other artists, let alone get into the studio to record those sessions. That’s part of the reason why researchers at Rensselaer Polytechnic Institute in Troy, N.Y., and McGill University in Montreal are developing a system that would allow jam sessions to span a country or the world.

Although virtual jam and recording sessions have been going on for years using other technologies, the researchers believe that ViMiC provides a better experience in terms of acoustics and ease of use.

As busy as some musicians are, it’s a wonder that any of them find time to get together to jam with other artists, let alone get into the studio to record those sessions. That’s part of the reason why researchers at Rensselaer Polytechnic Institute in Troy, N.Y., and McGill University in Montreal are developing a system that would allow jam sessions to span a country or the world.

The Virtual Microphone Control (ViMiC) system creates a shared virtual acoustic space across two or more rooms. Spot mics capture the sound in a room and feed it to software, which then sends it to loudspeaker arrays in one or more remote rooms, where the same process is going on in the other direction.

Researchers at Rensselaer Polytechnic Institute in Troy, N.Y., and McGill University in Montreal have developed a system that allows jam sessions to span a country or the world. The Virtual Microphone Control (ViMiC) system creates a shared virtual acoustic space across two or more rooms. Spot mics capture instruments, with software feeding that audio to loudspeakers in one or more remote rooms, where the same process is going on in the other direction. (JackTrip is a Linux-based system used for multi-machine jam sessions over Internet 2.) Although virtual sessions have been going on for years, the researchers believe that ViMiC provides a better experience in terms of acoustics and ease of use.

“Musicians already started to collaborate seriously over the Internet through services like eSession,” explains Jonas Braasch, an assistant professor at Rensselaer. “For the layer-oriented, multi-track approach of most pop music productions, these services work very well, but a live session is not possible with these systems. With a low-latency system like ours, musicians can play live together.”

RIGHT KIND OF FEEDBACK

Braasch and his research colleagues — Rensselaer doctoral student Daniel Valente and McGill doctoral student Nils Peters — got the idea for ViMiC from their experiences as musicians, including as part of an ensemble that plays together via the Internet. In those sessions, each site has about eight loudspeakers in a ring around the musicians, with a videocamera and an LCD projector to allow them to see one another. (The projector sits behind a window to reduce background noise.)

One obvious drawback to that setup is feedback loops, where the mics pick up the loudspeakers and send that audio back to the source. If an audio- or videoconferencing system is used to link the sites, its echo-cancellation mechanisms can create problems.

“In simultaneous music communication, spectral alterations are a common side effect if the echo-cancellation system operates with individual frequency channels,” Braasch says. Although many conferencing systems are full-duplex, they still can cause problems.

“The echo-cancellation system [still] frequently [closes] a channel in one direction,” Braasch says. “Our system would work with any full-duplex conferencing system as long as it has a low latency and an echo-cancellation system that can be switched off. I am sure that it is possible to build better echo-cancellation systems for music transmissions.”

But that might be easier said than done.

“One problem is that current algorithms focus on speech communication,” Braasch says. “One has to consider though, that many advance signal-processing techniques introduce extra latency, which we need to avoid.”

Until that ideal is available, Braasch and his fellow musicians don’t use echo cancellation. Instead, they avoid feedback loops by using lavalier mics and mics on stands. “It also helps if at least one of the two sites is an acoustically treated location with low reverberation times,” Braasch says. “Because we do not use room microphones, the ambience of the room disappears during the recording.”

All of the mics are unidirectional and placed as close as possible to the instruments. They’re also placed nowhere near loudspeakers. All of that minimizes feedback loops.

“For instruments with low sound pressure levels, an audible echo can occur, which is typically at a very low level and not annoying,” Braasch says. “Personally, I find the artifacts created by the current generation of echo-cancelers to be much worse. We also observed that rooms with more reverberation are more problematic with regards to echoes.”

ViMiC also works with amplified instruments, although Braasch and his fellow musicians perform contemporary music at moderate levels. “Electro-acoustic instruments work even better because the pickups do not capture the loudspeaker signals,” Braasch says. “For music styles [with] higher sound pressure levels, like pop or funk music, the issue of transmission echoes might be more problematic.”

ARTIFICIAL AMBIENCE

ViMiC was created to compensate for that loss of ambience. The technology generates synthetic room microphone signals, and it includes room-simulation software to create a multichannel audio signal from a dry recording. “The multichannel signal sounds as if it had been recorded in a particular room,” Braasch says. “Both transmission sites share a location virtually, if the room parameters of the ViMiC systems are set to be identical at both ends.”

The ViMiC system creates an array of virtual mics with simulated directivity patterns. “The axial orientation of these patterns can be freely adjusted in 3D space, and the directivity patterns can be varied between the classic patterns that are found in real microphones: omni-directional, cardioid, hyper-cardioid, sub-cardioid, or figure-eight characteristics,” the researchers explained in a paper presented at the Audio Engineering Society (AES) convention in October 2007. “For example, the array could consist of five cardioid microphones that are arranged … with axis orientations of ±110 degrees, ±30 degrees, and zero degrees at a distance of 0.5 meter from a common center position.”

The ViMiC’s tasks include calculating the gains and delays between all first-order reflections and the mics. “The gains and delays of the direct sound and early reflections are calculated according to physical principles and the simulated geometric setup,” Braasch says. “Delays are determined [by calculating the amount of] time a sound needs to travel the distance between a sound source and a virtual microphone at the speed of sound.”

The system uses a “mirror-image technique” to calculate the path of a sound source after it reflects off of one or more walls. (Both first- and second-order reflections can be calculated.) “We can also include the loss in gain of a sound source on its path to the microphone, and we also include the gain reduction if the sound reaches the microphone off-axis,” Braasch says.

NO GLOBAL JAMS

To mimic a single-room session as closely as possible, all of the signals have to travel between venues quicker than a quarter note. Braasch and his fellow musicians play from various places around North America, including Montreal, San Diego, and Troy.

“For most types of music, a live session works out very well if the transmission delay does not exceed 25 milliseconds,” Braasch says. “This gives us a maximum distance of 7,500 kilometers between two transmission sites, assuming that the transmitted signal moves with the speed of light without further delay through switches and analog/digital converters. A connection between New York and San Francisco is feasible, while a connection between New York and Australia will always remain problematic.”

A big, broadband connection is key to providing a good experience for the musicians. Braasch and his fellow musicians currently use eight audio channels, sampled at 44.1 kHz, with 16-bit resolution.

Clearly, the audio portion of the signal is not the bandwidth hog. “This requires a constant bit rate of 11 Mbps for a two-way audio transmission,” Braasch says. “We [also] need about 50 Mbps for a two-way video signal.”

But just because a connection is broadband doesn’t mean that some packets carrying the audio won’t get delayed on their cross-country journey. Braasch and his associates get around that issue because their research is conducted at universities, so they have access to Internet 2.

“It’s very reliable in terms of low latency and packet loss,” Braasch says. “Jeremy Cooperstock at McGill University has successfully [used] uncompressed HD video signals without packet loss.”

If ViMiC one day becomes a commercial product, one issue for users will be access to a network with the speed, low latency, and quality-of-service mechanisms necessary to deliver a good experience for musicians. Other factors also can affect latency.

“The latency of the computer gear does not add that much if low-latency audio and video drivers are used,” Braasch says. “While most musicians do not have access to Internet 2, Fraunhofer’s new Ultra Low Delay Coder has demonstrated a new low-latency audio compressor. The compressor reduces the bit rate of a single audio channel (44.1 kHz, 16 bit) below 400 kbps, and will enable individuals with ISDN [Integrated Services Digital Network] connections to participate.”

WHERE’S THE MARKET?

From the perspective of a musician, Braasch says ViMiC provides a playing experience comparable to having everyone in the same venue, at least when it comes to his ears.

“Acoustically, it is often difficult to distinguish whether an instrument is virtually or physically present,” Braasch says.“ On the video side, one is always aware that the opposite site is presented on a screen, something we will not be able to resolve with conventional two-dimensional projection techniques.”

The researchers currently are refining ViMiC by adding a tracking mechanism that automatically updates the location of each instrument in real time. That feature is supposed to heighten the sense of being there.

“This will allow the automatic alignment in space between the auditory and visual component for each instrument,” Braasch says. “The auditory/visual match of each instrument is important for the musical transparency of an ensemble. We are also working on the integration of haptic devices,” which use vibration and other electromechanical forces to create a physical sense of being there.

Braasch expects ViMiC to become a commercial product at some point, simply because there are musicians who yearn for a realistic collaborative experience. “There appears to be a general interest in such applications, and many people would love to jam with their friends abroad,” Braasch says. “Only a few key elements appear to be missing today to make such an application a commercial success. The bandwidth issue will be resolved soon, either through increased bandwidth for private access or new coders to reduce the bandwidth.”

Even with the work left to be done, Braasch believes that ViMiC already provides a realistic experience.

“I find our system to be extremely functional, and it allows great social interaction. We have been rehearsing weekly for over a year now. I have never met some of our partners in person, but I got to know them really well both on a musical and personal level.”

Tim Kridel is a freelance telecom and technology writer and analyst based in Columbia, Mo. He can be reached at [email protected].

Featured Articles

Close