In the wake of the coronavirus pandemic there is new enthusiasm for hybrid live events, in which presenters and a live audience are joined by remote participants, and usually a streaming audience as well. While meetings in conference rooms might have some of the same aspects, in order to run smoothly a hybrid event may require some of the complexity of live TV production.
The reality of hybrid live events is so new, with so much constant change, that every setup is likely to be different. So this is not a simple “how-to” article. The aim is to provide food for thought about what functionality complex events may require, and what kinds of technologies might be used. These concepts can apply to fixed or portable systems, and to other situations, such as classrooms.
Building up the System
Imagine, if you will, a large conference space or small auditorium. There’s a stage area in front with a presenter, and audience seating in rows or tables. An A/V system handles sound reinforcement for the presenter in the room, and projection or video screens to show the presenter’s computer content. There might also be mics placed for audience members to ask questions via the room PA. Pretty standard stuff.
Even before the pandemic, live streaming of events was becoming common, so there might be one or more cameras, and other production equipment, that feed a streaming encoder for viewers to watch online via a CDN, Youtube, etc. That’s a basic one-way stream–viewing only–not a video conference.
This scenario also applies to concerts, house of worship, etc. Before internet streaming this would have been considered the “broadcast” output of the event and, depending on the client, might still feed other delivery paths, such as fiber or satellite transport.
Now we’ll add a remote guest, connecting from home using a UCC video conferencing app such as Zoom or Teams. We’ll call this video teleconference (VTC) generically. At the venue, the app runs on a computer and connects to the rest of the system via USB devices using UVC and UAC protocols (generally built into the operating system). The remote guest must be seen and heard by the live audience and presenter in the venue and must be able to see and hear what’s going on in the venue. The presenter and guest might engage in conversation. If the live audience is going to ask questions the remote guest may need to hear them. And if there is more than one remote guest all of this multiplies.
In addition, depending on the “production values” desired by the client doing the event, there might be a need to play video clips or graphic elements, such as show opens, intro and break slides, music, or background videos. These may be seen by the streaming audience and/or the in-room audience.
Let’s also add teleprompter for the live presenter, or perhaps an event host. The prompter output is intended only for the person reading (using a stage-facing display of some sort) but it may also be useful to see, or possibly control, the prompter in the central control room for the production.
Finally, if the event is to run smoothly it’s likely that a crew will be needed for stage management, wrangling mics, keeping an eye on the audience, perhaps running cameras, etc. And all of these people need to communicate. This is typical for any live event or show, but adding remote guests is a new twist.
Production Control
At this point let’s assume that there’s a central location where all the incoming and outgoing signals meet, where there are operators running equipment, and someone is directing the event. This could be the back of the hall, a control room, or a remote truck. If the venue is built for live events there might already be a control area and typical operator positions such as house audio and lighting. For the hybrid scenario described here, you might also need: • Producer who oversees the action and process of the live event.
• Possibly an Assistant Producer to handle communications with inside and outside personnel.
• Director for the “broadcast” stream, to call camera shots and video sources, call up graphics, etc.
• Technical Director to run the stream production equipment (or this might be the Director).
• Camera Control operator for shading or robotic control (or this might be the Director or TD).
• Playback operator to roll in videos and graphics (could also run lower-third titles if used).
• One or more VTC operators to manage the computers and remote guests.
• Someone to control what’s on the screens in the venue (or could be shared with another job).
• A Production Audio operator to handle levels, mix-minuses, playback sound, etc.
• Prompter operator (could be shared if the demand is minimal)
• Someone to monitor the live stream.
Some of the job titles are kind of traditional from broadcast TV, but the bottom line is that all of these functions need to happen, whether assigned to individuals or shared. Some choices may depend on what equipment is being used for certain functions, and how particular devices interact. For example, if video clip or graphics playback is built into the production switcher the TD might end up running playback– even for what goes to screens in the venue.
One role that should not be shared is managing remote guests (VTC Op). This is because the remote guests need to be kept “in the loop” as if they were at the venue. Nobody wants to be hanging on a call, wondering when they’ll be live. Plus VTC platforms tend to require some attention, such as muting audio, watching chat boxes, and simply making sure the connection is solid.
How much equipment is required, and what types, can vary greatly. If the outgoing stream is supposed to have the polish of a professional TV show there should be a production switcher, character generator (for titles) and playback device, or a production platform (such as the Newtek Tricaster) that does a combination of these functions. If the event is being recorded on site that might require another device or be done in the platform. Robotic cameras should have a physical control panel, and of course there’s an audio mixer.
There will be computers for bringing in the remote guests–generally one for each (although new options are appearing that allow separate video of each guest from a single UCC computer). There might be a computer or wall processor feeding in-room displays, depending on what content they will show. Prompting will have a computer, whether it’s run in the control room or in the venue.
Something to consider with all-in-one production products (such as Tricaster, VMix, etc.) is that just because the platform CAN do everything doesn’t mean it’s a good idea! For one, the operator will have to divide attention between running the outgoing stream and other functions. Secondly, there are cases where a particular signal needs to go to several places. This can be a challenge if the production platform has limited input and output capacity, and also creates complicated delay scenarios (coming up). For example, some platforms include the ability to connect with Zoom, Skype or other VTC options internally. But using this feature may not be practical in a hybrid production when the remote guest feeds are also sent to in-room displays (and makes it difficult to offer good guest management).
Speaking of Timing
By now everyone understands that the live stream of an event is always delayed. The processing required to encode at the venue, decode at the CDN, re-encode for different viewing streams, etc. takes time (far more than sending through the internet). In my experience Youtube Live is generally 20-30 seconds behind. But that doesn’t usually matter because the viewers have no knowledge of the real time event–they only know what they get at their time.
When two-way communication is added, however, there are different timing issues. As we’ve all found doing virtual meetings, the latency of VTC platforms such as Zoom is surprisingly low, so not usually a big problem–even with real-time conversations between presenters and remote guests. This can vary unpredictably, and with changes in video quality, but still is usually acceptable. (Ain’t nothin’ you can do about it anyway. For best control over remote guest participation, and highest quality, you must move away from popular UCC apps and into more proprietary signal transport. That’s another story.)
The issue now is keeping people’s picture in sync with their sound (known as “lip-sync” in TV), because of processing delays in the production equipment. We are talking here about delay in video frames, which is 33mS/frame for 30fps video. I would suggest that a 2-frame lip-sync error is on the edge of acceptable; beyond that things start to look “off” to the average viewer. A self-contained production platform might have 6-12 frames of latency–input to final output–which must be accounted for in some cases. The most obvious is if audio is being mixed outside the platform. If the audio is sent into the platform, and combined with video, the delay will be handled internally. But if audio and video are combined downstream the audio will need to be delayed to match. This is also true for video through a conventional production switcher, but the latency is often a frame or less.
Lip-sync issues also arise because remote guests are being seen in several places at the same time. When guest audio and video leave the UCC computer, via a USB converter of some sort, they should be (nominally) in sync. That a/v goes into the production system and becomes part of the outgoing stream.
But the remote guest also has to appear on in-room screens, with matching audio coming from the house PA. The guest audio coming through the production mixer will be real time. But if the video gets to the screens by way of an output from the production system, it will be late and the audience will notice a lip-sync problem. With a digital mixer it may be possible to add delay to particular inputs or outputs to line the a/v back up in the room.
Another scenario is if the in-room screens are fed from their own computer, let’s say to give the audience a finished-looking image, or show several guests together, TV-news style. Again, the remote guest audio must be delayed to match the video that finally gets to the screens.
Fortunately, when it comes to lip-sync, there’s only one variable: Latency in video paths. Nothing in any audio path–not preamps, mixers, converters, cable or any processing–will delay audio significantly. If your audio is behind your video there’s a delay dialed into some piece of equipment, or something is broken. (Yes, wireless mics can sometimes take 10-20 mS, but that is still much less than a video frame.)
Also keep in mind that evaluating lip-sync can be tricky with all the types of displays and signal paths. Assume any flat-screen display might be a frame behind all the time (an issue CRT monitors did not have). Also assume a basic video production switcher (not an all-in-one platform) will take a frame for processing non-genlocked sources. And the “multiview” output of any switching device is at least a frame behind, probably more. So even under the best conditions monitoring lip-sync in the control room may be sloppy by a few frames–generally not a big deal. As with any TV production, the ideal is to have a “reference” display that shows the final output signal–unadulterated by any additional processing–to use for judging image quality and lip sync.
Comms
Communication within the control room, and to venue crew, is critically important and can be handled fairly easily with any number of wired or wireless intercom systems. A single “party-line” (where everyone is on one channel) may be fine. Sometimes it’s handy to have two channels, so that tech and producer conversations can be separate. Communicating with presenters on stage and remote guests is where it gets complicated.
A live presenter may wear an earpiece so that IFB (interrupted foldback) can be used. In this case they will get a feed from the production mixer (often known as “IFB Program”) that includes whatever they need to hear–typically the remote guests. The producer or director interrupts (or talks over) that audio when necessary. IFB with true interrupt requires a more sophisticated intercom, while simply talking over the Program audio can be done via the mixer. (This does not address how the producer or director audio gets into the mixer or intercom.)
The ability to talk to a remote guest from the control room (not through a mic out in the venue) is tricky, but quite important as part of guest management. They need to know what’s going on, and high-profile guests, who are used to doing TV news interviews, will certainly expect communication from the producer or director. In general, as of this writing, UCC platforms using USB converters assume one audio input and one audio output (“stereo” notwithstanding). The remote guest computer gets a mix that contains the on-stage presenter, possibly audience mics, and playback audio. This should be a mix-minus feed, meaning the remote guests get a full mix minus themselves, to avoid echoes. Here again, if the producer needs to speak to the guest, that audio must be added to their mix-minus, or must interrupt the mix-minus somewhere between the mixer and the remote guest computer.
Audio and comms is one of the hardest areas to implement with UCC conferencing because these platforms were not designed for individual communication. Everyone who joins the conference hears all the audio, and what comes out of the UCC computer is a mix of all participants (although that is slowly changing). So even when done well, the backchannel communication between a producer and guest is actually heard by anyone else on the same call. In a case where remote guests should not interact with each other it’s necessary to connect each guest to a separate call and combine them in the control room to keep their audio isolated. Another comms option comes in the form of internet-based services, such as Unity Intercom, that allow cell phones to be part of an intercom network. The process is administered from a central application running on a computer, perhaps in the control room, or can be cloud-based. It can also be interfaced to conventional wired intercoms. An app like Unity can add functionality when personnel need to participate in the production from outside the venue but can’t take the place of a dedicated intercom system for handling specific individual communication. A fully functional intercom that covers all the needs may require more equipment and expertise than one might expect.
Important Points to Note
• A full-bore live-streamed event, with live audience and remote guests, is more like a television broadcast production than a teleconference.
• There are essentially three audiences being served simultaneously: The live audience in the venue, the live stream or broadcast viewers, and the remote guests. All three require somewhat different content and communication.
• Don’t underestimate the need for managing remote guest(s), including the ability for people in the control room to talk to them.
• The production mixer will need to create several different mixes for different purposes: Final show output to stream, IFB for live presenter, mix-minus(es) for remote guests, possibly room PA. Some inputs or outputs may require delays added to fix lip-sync issues.
• As convenient and ubiquitous as they are, UCC platforms create specific limitations for including remote guests in a production. Other approaches (such as using a platform like VMix as a remote guest gateway) may offer more flexibility and control, but will require guests to use different applications or equipment.
