Anyone who designs or builds media technology systems has to troubleshoot from time to time. Whether it’s during installation or tracking down a problem later, troubleshooting is a diagnostic process that should be carried out logically and methodically. This article explores troubleshooting principles and how they may be applied (or not) as technology changes. Oldtimers may find much of this familiar, but we will address new issues as well. For starters, let’s look at some basic principles for any kind of troubleshooting effort (whether you’re an AV tech or a doctor).
Be Methodical: If you’re under a critical deadline and the system just has to work, do whatever is needed. But if you want to know what caused or fixed a problem, make changes one at a time, take notes, and save presets/scenes or versions of setups. This is particularly important if you need to return something to a previous state, or if you stop and come back days or weeks later.
Learn How Things are Supposed to Work: This seems obvious but it’s easy to make assumptions that might be wrong. That applies to whole systems, individual devices, software apps, and underlying programming. Get documentation, talk to regular users, and talk to manufacturers. Sometimes what looks like a malfunction is actually normal, or odd behavior is just a setting in configurations or preferences. Remember to ask: What should happen when I push this button?
Learn the Underlying Concepts: A corollary to knowing how things should work is knowing about the principles that make them work. This can include circuit design, programming, communications protocols, and even bits of architecture, HVAC and electrical practices in buildings.
More broadly, there’s physics. Knowing about the behavior of sound is necessary to address acoustical issues. Fixing problems with wireless technology–from RF mics to Wifi to satellite dishes- -requires knowledge of electromagnetic behavior. At times troubleshooting may lead in an apparent direction, but knowing the underlying principles might suggest it’s a red herring: That can’t be right, what is really going on here? https://www.svconline.com/industry/mysteries-of-rf
Get Familiar with Symptoms and Be Suspicious: With time and experience we learn various tell-tale symptoms and what they likely mean. For example, in audio there’s a big difference between a hum, which might be a “ground current” kind of issue; a buzz, which might be RFI; a crackle from dirty contacts; or dropouts in a wireless mic.
But this knowledge can lead you down the wrong road if you get a poor description of the problem. Be suspicious of user reports and check for yourself. With audio, it can be more useful to have someone actually imitate the sound than describe it!
Return To (or Start From) a Known Good State: Imagine sending HDMI from a computer to a distant display, using HDBaseT extenders and various CAT6 and HDMI cables. The full path doesn’t work, and there are multiple potential failure points. So find out what does work. Go directly from the computer to the display locally: Yes. Then get the extender pair and connect them in the room: Yes. Now you know that the display, computer, and extenders are functional- -as well as the cables you just used. Use those tested components as you move toward the final configuration and try not to add a new unknown variable (like changing the display resolution in the middle of the process).
A different example: Troubleshooting a problem with reported bad recordings on a server. Find out how those “bad recordings” are being checked, then locate and play a known good file first, to confirm that the problem isn’t actually the playback side. Users routinely forget that a symptom could represent more than one possible problem.
This principle becomes really important when troubleshooting network-based systems because there are many ways to get mixed up or lose track of what’s happening. The configuration worked two minutes ago, now it doesn’t, what’s going on? Go back to the previous configuration and make sure it still works. If not, something else changed. Stepping back to a known good configuration can be a “sanity check” as much as a tool.
Use Known Good Signals for Testing: Test signals like colorbars and audio tone don’t just help set levels in production, they also provide a reliable source for testing. In some cases what happens to these signals will be enough to reveal a problem. For example, if a display looks purple-ish, using colorbars might quickly show that the green channel is missing. It’s extremely important to have signals that are consistent and stable (and also beats saying “testing” over and over).
Use Test Equipment When Necessary: Along with known good test signals, it’s useful to have equipment that can display or analyze a signal under test objectively–to tell you what is actually there. With simple signals this is usually straightforward and quickly reveals problems. It gets harder when the signal is really a data stream, or when there’s protocol “handshaking” between devices.
Good Old Analog
I’ll be honest, when it comes to troubleshooting, I miss analog. There is usually one signal per wire, and signals “flow” in one direction: output to input. The effects of cable characteristics and length are moderate and predictable, the behavior of output and input circuits (interfaces) is well-defined, and signals can be directly measured. Generally, what you see is what you’ve got. Plus, analog equipment and processes tend to operate in a standalone fashion. That is, they don’t rely on something happening somewhere else to define how they function. Some classic troubleshooting strategies work well for analog and provide a starting point for troubleshooting more complex technologies.
Cut the Problem in Half: When many devices are in a signal path, one simple way to narrow down the location of a problem is to test half the signal path and see if the problem is still present. If it isn’t, the problem should be in the other half. Then test half of the suspect half, and so on. Devices that generate their own test signals are extremely useful as a “starting point” for a segment under test.
While this technique applies mainly to simple, linear systems, it’s also useful for other problems, such as finding a corrupted file that’s crashing an application. Take half the files out of the directory and see if is still crashes. If it does, take out half of those and test again. Keep track of the last batch of files moved so when the crashes stop you can put them back one by one and find the culprit!
Substitution: Replacing a suspect device with a known good one needs little explanation. But there are more sophisticated variants, such as subbing modules within a larger device, subbing sections of software code, or purposely subbing in a known bad device to test a hypothesis. Substituting equivalent analog devices in a system is likely to be useful because they operate standalone. It can also be useful to substitute different signals in the test path and see how that affects the problem. For example, test tone, speech and music may produce different results that illuminate an audio issue.
Swapping: One of the most basic and natural strategies: Swap the positions of two cables going into a device. If the problem moves from one input to the other it’s upstream of the device (in other words, it “stayed with” the cable that moved). If the problem stays on the original input, then it’s in the device itself or further downstream of the device.
If you apply the strategy of swapping in an analog system, you will usually get a pretty clear result. Things get a lot less clear as interfaces become more complex and have aspects which are interactive, discussed coming up.
Measurement: Measuring and analyzing analog signals requires only basic test equipment, such as voltmeters and oscilloscopes. This is also true for low-bandwidth control signals, such as RS-232, which can be viewed on a ‘scope (you can’t tell what the data means, but you can see if it’s there). Analog video can be measured the same way but the frequencies involved are much higher than for audio. Standard colorbars, coupled with a waveform monitor and vectorscope, can reveal the nature of many problems seen on a video monitor.
VGA video (analog RGB, often with H and V sync) is also testable with the same techniques, but things got more complicated with the addition of EDID signaling. The video still “flows” from source to monitor, but now there’s data in the other direction, where the monitor describes its capabilities to the source device. This creates an opportunity for errors that aren’t really about the condition of the signal.
Baseband Digital and Digital with Negotiation
Digital recording, digital signal formats and digital signal transmission brought two key changes from analog that greatly affect troubleshooting. One is the change from interfaces that directly impact sound or picture quality, to interfaces that are concerned with data recovery. The second is the further use of handshaking between source and destination devices that began with EDID.
Like analog, serial digital video (SDI) and AES/ SPDIF digital audio are considered baseband signals, meaning the video or audio is directly represented by the signal on the wire (in this case a data bitstream). And, like analog, there is usually one signal, going in one direction. The circuit at the receiving interface is trying to retrieve a clean bitstream from whatever arrives. Since the signal is just data (voltage levels representing ones and zeros) as long as you recover that data intact the ultimate end product–sound or picture–will be good. If you don’t recover the data accurately you may lose the signal entirely or have artifacts (dropouts, sputtering, green sparklies, blockiness) as the receiving device tries to make sense of a corrupted bitstream.
The implications of data recovery as the goal are wide-ranging and affect all digital interfaces. Resistance and capacitance in the signal path lower the voltage level, distort the sharp edges of voltage transitions, and introduce jitter, making it difficult to tell the ones from the zeroes. The receiver circuit of a device must overcome these effects–from the preceding output stage through cable, connectors, and interference. And we cannot easily observe how well the data is being recovered; that is, how close we may be to the “digital cliff” of not working at all.
The practical realities of recovering the data are why digital cables have length limitations, and why those specs are not the whole truth. For starters, higher bitrates imply better cables and shorter lengths. But then, one input stage might “tolerate” longer lengths than another. And different cables may have different effects on signal recovery. Or the same signal might work going into only one of two devices. A cable that doesn’t seem to work may be poor for the given conditions, but not actually broken. This ambiguity can make troubleshooting by swapping or substitution misleading.
Here, again, test equipment can be a help. The same kinds of tools exist for baseband digital signals as for analog, meaning you can observe audio or video activity in real time, see the status of different parameters, and look at raw data. Furthermore, test gear with an eye pattern display can show the condition of the actual bitstream. This is useful for a format like 12G-SDI (meaning UHDp60 on coax) which is highly susceptible to degradation. (If you can’t observe the eye pattern you could stick a reclocking device somewhere in the path and see if that fixes things.)
Digital video formats such as HDMI, DVI, and DisplayPort also require robust data recovery at the receiver and have the same limitations for swapping and substitution. Plus, some include what we might call bi-directional metadata exchange (handshaking) to define parameters, which adds to the difficulty of troubleshooting. A given problem might not be a malfunction, just a “disagreement” between equipment. A good example is HDCP copy-protection, where the handshaking exchange asks, “is this a legitimate receiver of this program?” If it’s not implemented correctly, you might get a blank screen when you shouldn’t.
Furthermore, these formats are subject to ever-changing definitions of what is supposed to work from one generation to another. And if that weren’t enough, the implementation of interface specifications is not always done carefully, which means that one manufacturer’s device may work differently than another in the same installation. Fortunately, these types of compatibility issues are less common than when the formats were new.
Speaking of swapping cables, remember that when you unplug and re-plug a cable such as HDMI certain things get reset. The “hot-plug” pin in the connector alerts the devices to handshake again. So maybe it’s not the new cable that fixed the problem, just the act of re-plugging. Of course, re-plugging is a standard “fix-it” technique for this very reason!
With HDMI, it’s useful to have test generators that produce signals with a range of adjustable parameters. You need a variety of SD and HD formats and images, and you also might want different bit depths, audio sampling rates, and HDCP status. Likewise with scopes or analyzers; you want to know what’s actually there–to the extent that’s possible. Due to handshaking, the analyzer should have the ability to provide parameters to the source device (ie: please make your output 1080p30 at 8-bits).
Quite frankly, it’s hard to find absolute strategies for troubleshooting these formats because there is so much going on inside the interface that is essentially invisible. The tricks that are supposed to make the formats “plug and play” can actually make troubleshooting difficult. Old techniques can be used to some degree, the factors above can muck things up. It’s not unusual to make a problem go away and never know exactly why.
Networked AV Physical Considerations
Networked AV (“something over IP”) is unlike conventional AV in almost every way and takes troubleshooting to a new dimension. With conventional analog and digital it’s possible to monitor actual signals on the wire, or within equipment, in real time. Turning AV into network data makes the very idea of direct measurement almost meaningless. Not only are the data rates extraordinarily high, but the signals on the wire have no direct relationship to video or audio. That’s somewhat true for other digital formats, but network data is even more abstracted from the content it carries. The ultimate data payload (the sound and picture) is carried as tiny data words, in successive encapsulations, each of which uses specific protocols to allow the next higher layer to retrieve its part of the message. You only get AV when the pieces are reassembled!
If that last paragraph reads like gibberish, here’s the first piece of advice for troubleshooting networked AV: Learn about networking. After a few years muddling around with basic office networks I realized that I simply did not understand essential concepts. Filling in the gaps with classes has been indispensable. Despite the efforts of manufacturers to make their products easier to deploy, it’s just not enough. Even the ability to talk intelligently with tech support can be hampered by lack of networking knowledge. It is, truly, a different discipline, and the terminology is easily misunderstood.
But networked AV is similar to conventional AV with regard to the behavior of signals moving through cables and interfaces. Like balanced audio, Ethernet travels on twisted-pairs of wires in order to reject interference–both from outside the cable and between pairs. And the receiver must reconstruct the bitstream. Getting 10Gb/s or higher through simple copper wires has pushed the evolution of cable and connector design, with more stringent restrictions on length and greater demands on receiver circuits.
So the conditions that might degrade SDI or HDMI also apply to Ethernet on “category” cables (as well as other high-bandwidth signals, such as HDBaseT). Thankfully, Ethernet physical interfaces are well-defined by standards, so random differences in how equipment behaves are minimized. In my experience, the physical network connections are not a big source of trouble if done competently.
Network switches are typically the heart of a network, linking together devices and passing the appropriate data between them (these are entirely different from an AV “switch” or switcher). A lot goes on in the network switch, and while simple ones may be sufficient for basic uses, a managed switch includes capabilities for troubleshooting and monitoring network conditions. Complex systems using Dante, NDI, AVB and other transport protocols may require managed switches. Plus, newer switches tailored for AV may also include AV-oriented tech support.
Power-over-Ethernet is another potential source of troubleshooting confusion. With POE, reseating or changing a cable also cycles the power to the device, so the actual fix to a problem might be ambiguous. On top of that, proper operation of POE is another handshaking process, wherein the device and the switch (or other POE source) negotiate the necessary power specs. If that process goes awry it might look like a hardware failure at either end. You may need a way to monitor POE directly–more test equipment!
Networked AV Beyond the Physical Interface
When troubleshooting within the user interface of AV hardware or software, such as an audio processor or stream encoder, the same principles apply as any other device (review configurations, check for conflicts with other processes, revert to a working state, etc.). But many issues in networked AV are more about devices not communicating, switch and router settings, or other aspects that don’t directly relate to the audio or video payload. The idea of a signal moving through a system has expanded beyond wires, connections, and devices. Now we have to consider protocols and practices that are entirely concerned with getting data bits from an IP layer in one place to an IP layer in another.
And there are a lot of protocols involved. You can have a functioning system that fails sporadically because some configuration in a switch isn’t quite right. An example of this is the growing use (or requirement) of multicast for many types of AV-over-IP transport, which means switches must support the right protocols, the right versions of the protocols, and be configured correctly.
To make matters even trickier, Ethernet networks were not designed to carry high bandwidth, synchronous data–in other words real-time audio and video. They were designed to do things like file transfers, or writing to a database, which don’t ask much of the network itself. A lot of complicated tweaks have been added to the basic ethernet protocols in order to make AV work reliably.
Configuring and troubleshooting network systems involves a lot of work with IP addresses and their associated subnet mask, gateway, etc. Much equipment comes set for DHCP addressing (or defaults to link-local addresses) and determining which device has which address can be a pain. Plus they can change! I find that setting static IP addresses in equipment and keeping a list of addresses and logins for everything on the network, allows for quick and unambiguous connections to devices, and it’s easier to tell when a device goes missing.
When it comes to observing a system in operation, an AV problem occurs in real time, but you can’t interpret the data activity that might explain the cause in real time. To see what’s happening in the network you must capture data and analyze it after the fact. A starting point for this is system logs. Equipment and software designers need to troubleshoot their products, too, so all kinds of logging may be available internally. If you don’t already know, or don’t find it in a manual, it’s worth asking tech support (they may ask you to download and send logs as well). Managed switches have logs, as do network routers, firewalls, and many AV devices. Do yourself a favor and set the internal clocks of devices so the timestamps in event logs are accurate! When problems get really mysterious it may be necessary to use a packet analyzing program, such as Wireshark, to capture the raw network data and sort through potential errors. Doing this requires a strong understanding of networks and a lot of time.
The bottom line: Troubleshooting techniques that work for baseband connected systems may be useless in networked environments because your AV “signal” is: a) no longer a signal, b) handled as a lightning-fast series of transactions between devices, that c) don’t occur in a predictable order, and d) are overseen by a suite of protocols that move data up and down the layers of the network stack. It’s just not your grandpa’s AV system anymore.
Assorted Random Tips
When all else fails, it might be time for a factory reset. Lots of equipment has a way to do this, and sometimes it’s the only way to get a clean start. But take notes and screenshots of anything you’ll need to restore. Saving presets/scenes is also good, unless the problem is caused by something saved in the preset!
Low-cost external power supplies (inline, “wall wart,” etc.) are a frequent failure point. They can even fail in a way that might allow equipment to appear okay (indicator lights on) but not actually work. Look at these before replacing or sending out the actual equipment.
Dante, AVB, and SMPTE 2110 protocols are known for their clock requirements, and clocking issues are also common with AES/EBU digital audio. Standalone AES devices can run on “internal” clock, but as soon as you connect two of them somebody has to be in charge or you’ll have dropped audio, clicks and pops. The sampling rate also has to match.
If you are troubleshooting an issue via remote access such as VNC or Teamviewer, be careful not to inadvertently change something that will cut the connection. Sometimes working quickly leads to something dumb, like changing the IP address of the remote computer you’re connected to. It’s a sinking feeling when the connection drops and there’s no way to go back.
When someone says the network or internet service “is down” because they can’t reach websites on a computer, the computer’s IP settings may be missing a DNS source. Without DNS, a browser cannot find webpages using a URL. This happens frequently, especially if the computers are using static IP settings. Likewise, without a gateway address the internet will not be reachable.
Being able to use the Windows command line (or Terminal utility in MacOS) for simple functions is extremely useful, as is some familiarity with Unix/Linux environments. Even if you are being directed by a tech support agent it’s helpful to be comfortable using the command line.
