Installation Profile: Need-to-know ControlDesigning a control system for top-secret NORAD/USNORTHCOM facility 11/01/2006 7:00 AM Eastern
Nov 1, 2006 12:00 PM, By Jim Pope with Charles Conte
Designing a control system for top-secret NORAD/USNORTHCOM facility
Professional Videographics of Denver has had occasional opportunities to work in several military command-and-control centers in Colorado. In most cases, we work for the prime government contractor who performs the role of systems integrator on such jobs. We perform a range of services for this contractor, including engineering consultation, control system design review, control system programs, and installation and user support.
The consolidation of the North American Aerospace Defense Command (NORAD) and the U.S. Northern Command (USNORTHCOM) into one building in 2003 eventually led to a unique opportunity for Professional Videographics in this specialized world. The structure of the new command emerged in the months following the official merger. As it did, the flow of information among the various groups in the command also emerged. Our job was to work with users to clarify these flows of information and establish how to control and protect them. Over an 18-month period, we used a Crestron architecture and control system software development environment to meet the challenge, which allowed us to build the information infrastructure in a straightforward and efficient manner.
WATCHING INFORMATION FLOW
The NORAD and USNORTHCOM organizations comprise a large number of intertwining groups, each with its own structure and information network. The various command-and-control centers at NORAD/USNORTHCOM (the actual rooms and the people who work in them) exchange audio, video, and graphic information on a constant basis. If an analyst in an intelligence planning group sees something and determines it should also be seen by an operational planning group, then the information needs to be “published” from one group to another.
The information that needs to be published might be a display or map or satellite image that is coming in on one group's network, yet needs to be seen by someone who is not attached to that network. (This is by design, not by accident.) In a business setting, information might be sent as an email memo with a PowerPoint attachment. But the military establishment must move in realtime. And since the information is primarily visual, the problem we faced was, how does the display get in front of the right group? If everyone is in the same room, the answer is simple. When they are not, the answer is more complicated.
Command-and-control centers communicate with each other based on their security levels. Each center has an operator who must be able to monitor conferences and communications and know the security level of each at all times. If one group drops to an unclassified state, then an operator must know this so that inappropriate information is not displayed.
Since the level of activity in these facilities can be intense at times, Professional Videographics determined that the control system should support these operators by taking on some of the burden of monitoring security levels.
In summary, NORAD/USNORTHCOM is all about intergroup communication, the realtime flow of information, and confirmation of security status. To address these issues in overall system planning, a matrix was designed using Crestron 2-Series processors.
MATRIX AND SECURITY
A central facility, known as the NORAD/USNORTHCOM Command Center, houses a large number of analysts, a 6×2 videowall featuring Christie Digital rear-projection display cubes, four additional projectors, multiple LCD displays, and three separate audio zones with breakout areas.
Professional Videographics did the majority of the control systems design and implementation within this facility.
From the command center, information is published to other planning and operation groups, including the Battle Cab, where the commander of NORAD/USNORTHCOM has his staff. Seven remote groups receive information published by the command center.
A meeting/conference room or facility has a security level at all times. That level may range from unclassified to secret to top secret and higher. The security level of a room is dependent on a number of things, including the type of information present in the room, the lowest security level of the people in the room, and other criteria that is classified itself.
At any moment, each of the seven remote facilities — and indeed, the command center itself — might be at any security level. The system cannot always rely on the presence of the operator to kill the feeds or demand that a phone call be made to the command center to report the security status change. In such situations, the control systems themselves need to communicate with each other and perform certain predetermined tasks, then relay the new status to the operators of the affected rooms without requiring operator involvement.
It was in the design of a control system that would meet the requirements of this specific matrix that the Crestron technology really showed its stuff.
At least as far back as 1996, when I first opened Workshop (Crestron's early software development system), Crestron has had a system module called XSIG-Intersystem Communications. This module has always had one purpose — to transport data between different control systems. The module started out being attached to serial ports, and later became available for Ethernet.
From day one, Crestron Ethernet has worked without needing anything but a hub or a switch — no other computer, no router, nothing. Plug the processors together, set the IPs, and they talk to each other. Equally important, the individual control systems retain their own identity and individual programs. The XSIG module can pass status information, level indicators, and serial (text) strings. In the NORAD/USNORTHCOM matrix, this is the type of information we communicate — the level of each room, the names of the feeds being sent. We also give each room operator the ability to ping the other operators and ask them to call back when time permits, thereby avoiding an initial phone call at what might be an inconvenient time.
Each room constantly sends its status to the command center. If a room drops to unclassified or below, the control system automatically clears the routed sources off the command center switch (AutoPatch 128×128). Should the command center itself drop to unclassified, or if the Sanitize button is pressed, then the Crestron master control system programming will clear its displays without disturbing meetings in other remote rooms.
All of this status is available in realtime to the operator of each room and to the command center. Any time a facility changes status or security level, all connected rooms are notified in realtime. If a room disconnects, then the other rooms receive notification when the connection resumes. In addition, the type and name of each information feed is sent between connected rooms. Not all rooms communicate via the matrix, but all have multiple-level audio- and videoconferencing capabilities and the multiple-security-level backbone to ensure displayed information is appropriate.
At press time, Professional Videographics had provided software for Crestron Control systems located at 28 government facilities in Colorado and at least 15 at Los Angeles Air Force Base. (And the list is growing.)
The equipment we control in the NORAD/USNORTHCOM system is relatively straightforward: Sharp projectors, Pioneer plasmas, Samsung LCDs, Christie Digital rear-projection cubes, Jupiter wall controllers, Extron video switchers (below 32×32), and AutoPatch switchers (above 32×32). The programming module we used is rather straightforward, as well. However, it allows us to communicate certain pieces of information in the background among the control systems so that an operator is not caught unaware.And, as the facility proves, a straightforward programming module can accomplish big things when applied correctly to a complex challenge.
In 1998, I was contacted by what was then EIS (which later became Intellisys, which went bankrupt and then was bought by MCSI, which later went bankrupt also) to write a program to control several Gentner AP-800 mixers in a unique location — the main Battle Cab at Cheyenne Mountain Air Station (CMAS).
This was not my first experience working for a government installation, but it was my first working at the facilities of NORAD and Peterson Air Force Base. It was there that I first learned about security, clearances, need-to-know, and need-to-shut-up-and-work-and-get-your-behind-out-of-the-room-before-the-missile-test-at-1600-hours. It was also there that I learned that you always need to have a paperback with you in case there is a lockdown from a bomb threat and you're stuck for a while.
The technical requirements of that first job were not stringent — it mostly involved turning mics on and off, controlling the mixer matrix, and some basic volume controls. More importantly, that small job gave me the opportunity to develop some programming and program management techniques that I still incorporate into each government facility we build:
- No flowers on the GUI
Forget all the flowery backgrounds and squiggly lines on the GUI. They keep the user from instantly finding the control. You have to remember that military personnel, government workers, and civilian contractors operate these military facilities, and there's a lot of turnover. Instant familiarity with a user interface is important.
- No surprises for the General
If the indicator says the mic is muted, it had better be. You make sure by doing it in triplicate: on input, on output, in the codec. Never embarrass the user.
- No expiration on your warranty. Ever
You may design a system and program it, only to find it was installed eight months later. The user you trained may not have a chance to use it before he or she is reassigned, and your “bug” may not turn up for another year. How can you have a warranty cutoff in a situation like that? If you write good code, it will hold up. If you make a mistake, fix it. Even if it turns up a year later.
- If you think it is a good idea, put it in
Even if they did not ask for something, put it in anyway. Then show them how to use what you put in, but be prepared to take it out. Bring up the control aspects you feel the user needs, but do not think you have the only answer, or your ego will only get bruised. Keep an open mind to the operational aspects of the project, and model, model, model. Run on your own system. Test, test, test.
- Find simple answers for complex problems
This is extremely important. Do not get too fancy. Look for ways to handle things that keep the project scalable and maintainable. Always build in feedback, even if you hide it (like on hidden pages of a touchpanel). This way, you can have an operator check for status while you are on the phone with him or her 200 miles away.
— Jim Pope with Charles Conte
Not Business as Usual
Working on a high-security government project involves procedural differences that you do not encounter in typical commercial projects. Among them:
- Access to the job site
“No, you cannot go in there” is a common response. “You only have 20 minutes before the General gets back,” is another. Or “You cannot leave now. Get in the closet.” (The closet is a very small room with a desk, a chair, and a light.)
- Access to users
Typically, in military projects, you build the system for the contractor, but the user will not be taking over the facility for a few months. Then, just to make your life interesting, another firm will provide support for the user with people you will not meet until they want you to.
- Access to the system in full operation
Much of the time, even if you do have the security level necessary to get into a facility, you do not have “the need to know” about what goes on there. So, while you may get a chance to test pieces of your system, you will never see it in all of its full glory. For me, this is the hardest part of the job. I love working in these facilities and building these systems, and once in a while it would be nice to see them in full operation.
— Jim Pope with Charles Conte
For More Information
Since the early 1960s, North American Aerospace Defense Command (NORAD) crews have scanned North American skies for incoming warplanes and nuclear missiles from an operations center southwest of Colorado Springs, Colo., 8,000ft. above sea level and more than 2,000ft. inside Cheyenne Mountain.
The Department of Defense established the U.S. Northern Command (USNORTHCOM) on April 17, 2002, to consolidate under a single unified command existing homeland defense and civil support missions that were previously executed by other military organizations. USNORTHCOM assumed its responsibilities on Oct. 1, 2002, and was co-located at Peterson Air Force Base, Colo., with the headquarters of NORAD. USNORTHCOM announced full operational capability on Sept. 11, 2003, the second anniversary of the New York, Pentagon, and Shanksville, Pa. tragedies.
On July 28, 2006, Admiral Tim Keating—commander of both NORAD and USNORTHCOM—announced that day-to-day operations of NORAD would be moved from the mountain complex to a much newer headquarters building at Peterson Air Force base, directly east of Colorado Springs.
Long a symbol of the Cold War, Cheyenne Mountain is four-and-a-half acres of connected chambers and tunnels dug out of granite and 15 free-standing buildings, each two or three stories tall, mounted on springs to withstand an electromagnetic pulse from a nuclear ignition, or an earthquake. The 25-ton steel vault door guarding the complex entrance has become an icon in the popular imagination for all of the anxieties of the nuclear age.
About 1,100 people now work in the mountain, including 700 support staff and 230 surveillance crew members. An undetermined number of the support staff will make the move from the mountain within two years, according to Michael Kucharek, chief of media relations for NORAD and U.S. Northern Command Public Affairs. He adds that the mountain complex will be kept as a backup, fully operational, and staffed with minimal support personnel.
The announced move completes the physical consolidation of NORAD and USNORTHCOM, initiated with the populating of the Peterson Headquarters building in September of 2003.
Since its inception, USNORTHCOM has consolidated its defense efforts with other federal and civil authorities. USNORTHCOM’s mission includes domestic disaster relief operations that occur during fires, hurricanes, floods, and earthquakes. Its civil support mission also includes counter-drug operations and managing the consequences of a terrorist event employing a weapon of mass destruction.
Jim Pope is president of Professional Videographics in Denver. Pope wrote the article with Charles Conte, a marketing communications consultant, writer, and president of Big Media Circus. He can be reached at CharlesConte@bigmediacircus.com.