In “Just Tell Me What to Buy!” we discussed ways in which clients can underestimate the needs of a project. This article is about long-term concerns that we, or our clients, may overlook in project planning and deployment (see “Just Tell Me What to Buy!“)
Almost every project has needs that go beyond physical design, installation and training. Let’s call those needs “soft requirements.” They might have associated costs (so-called OpEx or operating expenses) or might be more about how the system is used. Leaving soft requirements to chance can lead to trouble!
Some Things Seem Obvious
In the excitement and hubbub of conceiving and building a project it’s not uncommon for people to forget about repeating and ongoing expenses. The most obvious include things like licenses, support contracts, and upgrades. Everyone knows that much software has moved from a “buy it/own it” model to recurring licensing. Those must be accounted for in ongoing budgets, and in some cases (such as government) that may be difficult to arrange. Plus, someone must be responsible for handling renewals. Licenses for internal equipment options, like a particular signal format, tend to be one-off purchases, but maybe not. In the case of support contracts, I’m referring to manufacturer support, such as for critical pieces of equipment. One area where I’m usually inclined to recommend this is for video servers and storage systems. Servers can be complicated and hard to troubleshoot and replacement parts may be unique. Similarly, ongoing support can be wise for complex control systems and DSP processors. Even if you or your company are providing direct support, having the manufacturer behind you can be critical.
Upgrades sound simple enough but are actually hard to characterize because there are many variables involved. Is an upgrade truly needed for a technical, operational or security reason? Or is it just nice to have? Will there be a “domino effect” that involves other system components? Will it take the system out of operation? Will user training be needed?
Personally, I’m not a fan of making software or hardware changes just because there’s a new version. There will always be a new version (especially if it brings in revenue) and we all know how updates can lead to… unintended consequences. If a system is working well why mess with it? But eventually there will be a good reason and clients need to be prepared for potential costs and downtime.
One touchy area is with computer operating system changes. Some IT departments want updates to happen regularly, which may cause AV software to break or need its own update. In extreme cases the OS update may add a security layer, or simply drop support for a function that AV operations depend upon. These issues can be hard to solve so I usually argue for minimal changes to computers until everyone is ready for whatever happens. Don’t do it the night before a big event, right?
Then there’s the simple matter of troubleshooting, repairs and replacement parts. A well-designed system, with quality equipment, in an appropriate environment, should not need a ton of maintenance or repair, but things do happen. Someone’s time will be needed to fix it.
Whether there should be a recurring budget item to cover any of the above will depend on the client and their organization. Some prefer to allocate money in advance, whether it’s spent or not, others can handle ad hoc costs as needed.
The Less Obvious – Media Files
Several soft requirements that frequently need attention are connected with recorded media files. Clients doing live streaming to an online platform may be content with automatic online recordings, but some need systems designed to record locally so that productions can be edited or used in other ways.
It’s easy to say “we’re going to record three cameras” without really understanding the implications. Local recording raises questions about what is being recorded (number of simultaneous sources, file formats), where it’s being recorded, how those files are moved around, and how files will be named, tracked and backed up. I have seen too many teetering stacks of hard drives, and folders filled with thousands of randomly named files, because there was no plan–or the plan came too late.
If there was one advantage to the days of magnetic tape it was that physical media could be labeled and put on a shelf! Today, important recordings require RAID storage with fault tolerance, possibly local or cloud backup, and a defined file organization and/or asset management approach. Over time even file migration may be an issue as formats and storage modes change.
Clients need to reckon with the sheer size of video files (even highly compressed) which translates to storage size and bandwidth, how quickly files accumulate, and how to maintain user compliance with the file management scheme. To be honest, video file capture, storage, and management is an area that may call for input from a specialist who knows the methods and products.
But answering these questions and putting methods in place before a lot of recordings are made is critical. Once systems go into regular use it is very hard to dislodge habits and workarounds, or to retroactively establish file management. It is not wise to let this fall to chance.
The Less Obvious – Continuity
Another oft-overlooked issue is what I’ll call continuity of operations. I don’t mean disaster recovery, just the simple problem of tracking IP addresses, online accounts, logins, software registrations, and other information. In the midst of working people tend to find what they need and get it going without thinking about the long term.
More than once, I have seen situations where someone downloaded and registered software using their personal or business email, then left the organization. When others try to access the software or the account, they don’t have the credentials. Nor will they receive email notifications about updates, renewals, etc. The same thing can happen with accounts for, say, YouTube streaming or Google Drive. Does everyone know how to access that Dropbox account?
This issue also applies to any type of registration or account that supports particular equipment, which is becoming increasingly common. The issue here is not warranties, which should be honored whether you “register” a product or not, but accounts that the product uses to validate a license, for example.
One approach is to create some generic email addresses that are not tied to any one person but tied to the organization. Unfortunately, this sometimes runs afoul of security rules if the IT department wants every email to be associated with an individual. Perhaps some negotiation is needed.
The same issue applies to generic computer logins, which IT security may frown upon (or simply not allow). A typical example from my work is a bunch of computers used for video editing. Users move around and don’t “own” any specific computer, so it’s convenient when they all have the same or similar logins (Edit 1, Edit 2, etc.).
People change positions, come and go, and sometimes passwords are changed for security reasons. At minimum somebody needs to keep track of this information over time! Does that actually happen? If you’re lucky, somebody put a sticky note on the monitor, which makes security people frown a lot.
That’s assuming the computers use local logins, not network credentials. In the latter case it’s nearly impossible to use something generic since the network authentication is likely connected to an individual. On the good side there might be less risk of one person leaving the organization and taking critical knowledge with them.
Needless to say, these same issues apply to passwords for control panels, locks for storage cabinets, and secure rooms. Ever been locked out of the AV closet because someone changed the passcode? Or something is amiss in a server room, but nobody knows how to get in?
The Less Obvious – Equipment and Stuff
I work with many clients that have accumulated countless cables, adapters and converters of all types, plus many pieces of equipment that have fallen out of use or were never used at all. Sometimes the newer (and usually younger) people on hand may not even know what these things are or what they do.
I encourage clients to organize this stuff in ways that minimize damage and make the items useful. I’m happy to teach people how to properly coil cables (over/over or over/ under), determine what type they are, and secure them with twist ties or Velcro straps– not tape! Organizing cables is great work for interns, as long as someone teaches them about different cable and connector types. Same for power cords, power supplies, power strips, extension cords, etc. Extra power supplies can be quite handy when something fails (see “Practical Power“).
A lot of equipment comes with extra bits and pieces that might be needed later, so as a project winds down, I usually try to organize spare and extra parts into a few labeled boxes so I (or someone) can find them. It would be nice if a client could do this, but they are usually not aware of all the components and don’t want to take the time.
Apart from quickly finding what you want, this kind of organization may reduce the tendency to buy more stuff every time a need arises. Online purchasing is so easy that people may not even look first, especially if the cache is in disarray.
Technical Management
So how does all of that get done? This is a big concern that I return to regularly. I suggest that there should be a role called Technical Manager. This person does not need to be an engineer, just someone who is well organized and has a head for tech. In my ideal world the designated Tech Manager would handle some or all of the following:
•Organize and store equipment manuals, cables, adapters and spare parts
• Keep track of current and old equipment
•Note software revisions, updates and backups
• Keep track of logins, passwords, online accounts
•Develop and enforce media management strategies
•Orient/train operators on facility systems
•Advise operators of updates and changes, retrain if necessary
• Investigate and troubleshoot problems
• Facilitate support from vendors, manufacturers, etc.
• Primary point of contact for outside tech support
• Primary or starting point of contact for upgrades, rebuilds, etc.
The idea is to have a central person who is aware of what’s going on, can keep others informed, and interact with outside entities. This need not be a full-time role in itself, just part of someone’s job. Sounds sensible, right? If you’re working with clients that are highly technical, or part of a technical organization, they might already have some kind of tech management in place.
In many other cases you’ll hear about why it’s not necessary or can’t be done: Things aren’t that complicated; we don’t have anyone capable; we don’t have enough people; everyone’s a volunteer; everyone’s a freelancer; there’s no place to store the data/parts/manuals.
What that says is that the client doesn’t entirely understand what they’re getting into. Granted, some simple systems can run without much oversight, but being too casual about technical management is a recipe for trouble down the road. Especially with non-technical volunteers, or freelancers, having someone around who knows the system is critical. I often find that “upper management” (whoever that is) thinks that hiring a “pro” eliminates the need for orientation or training. This is nonsense since every system is unique.
For my part, when I build a system, I include in the documentation a record of IP addresses and logins, plus other information, that was true when I set it up. If something changes and the doc isn’t updated there’s nothing I can do, but it provides a starting point and a place to make updates. The same goes for signal flow drawings, cable run lists, etc.
I also warn, cajole, threaten, and otherwise try to make technical management a priority. As much as I would like to hand off a finished project to an individual or staff that is fully engaged and ready to take over, that’s pretty rare. But as designers and integrators we can at least try to prepare them for what’s ahead.
