The only time moving to a contingency plan in a crisis is easy is when another contingency has already put the plan in place. That's exactly what happened to HBO when the blackout struck much of the Northeast.
The network has an agreement with the Long Island Power Authority that, on days when demand for power is high, HBO pulls all or some of its power demands off the grid and relies on its diesel generator (which has enough fuel to run for seven days). So, when LIPA saw the forecast calling for temperatures in the mid 90s on Aug. 14, the request went out to HBO: Please use your backup generator.
"Our administrative area of our Hauppauge facility was off the grid [at the time of the blackout]," says Bob Zitter, HBO senior vice president, technical operations. "But our technical facilities were on the grid, and our UPS made the move from the grid to the generator seamless."
It's safe to say that HBO had the most seamless transition to backup power during the blackout. For other networks and stations affected by the blackout, a mix of uninterruptible power supplies (UPSs), transfer switches and generators got them back on the air.
Not that many stations appear to have backup power, though. For large-market stations, the investment makes sense. But, according to Martyn Horspool, Harris Global TV products manager, only about 10% of TV stations have backup generators and UPS systems. He says a UPS can range from $25,000 to a couple hundred thousand; with generator costs running $300-$400 per kilowatt, the costs climb higher.
Given the infrequency of blackouts, renting may seem like an attractive option, but Tony Satterthwaite, vice president, commercial genset business, Cummins, points out that a station that rents has to wait for the unit to arrive and be installed.
"A supplier can't size the fleet for a peak emergency, so a station either needs a contract or risks the supplier's running out of stock," he explains, adding. "We ran out of generators by the morning after the blackout."
If the decision is made to install a backup unit, the first line of defense is the UPS. Rich Tessen, principal consultant for systems integrator National Teleconsultants (NTC), believes a UPS is mandatory in any facility today, especially given the reliance on video servers and other equipment that has computer components.
"A small glitch from lightning means a quick flash of the lights, but, to a computer system and equipment with computers at their heart, it means a reboot cycle," he says. "That 10-second glitch can knock a station off air for 5-10 minutes."
If the generator facility is large enough, it's possible to do what HBO does: use the capacity to be self-reliant and relieve stress on the power grid.
Zitter says that HBO's agreement gives the Long Island Power Authority the right to ask HBO to drop off the grid for a certain number of days, usually during the summer. And the network gets a lower average utility rate.
Beyond the gear is the issue of testing and running drills. Codes call for testing every month, but it's also important to ensure that people understand how to manually start up the system. Tessen says drills also give the equipment more workouts.
Besides purchasing or renting a generator, there are other types of contingency plans.
Take CNN's New York bureau, for example. The facility is located at 1 Penn Plaza and does not have a backup generator on site. The bureau does, however, have a contingency plan for an emergency like the blackout: It heads down to its sister network Time Warner Cable's NY1 News.
NY1 News has a diesel generator with a 1,000-gallon tank that can keep the station on air for 24-36 hours. It's a good thing because the 30,000-square-foot Chelsea-based facility was without power for 30 hours. That was long enough to outlast the phone terminal's backup systems, which gave out after 12 hours.
"The UPS is important because we live and die by servers and they don't like power spikes," says Harlan Neugeboren, vice president, engineering and technology, Time Warner Cable.
Joe Truncale, NY1 News director of operations and engineering, was on hand at NY1 when the decision was made to have CNN's Wolf Blitzer, Paula Zahn and Aaron Brown broadcast from the facility. NY1's uplink truck got the signals down to Atlanta as CNN New York broadcast from 8 p.m. to midnight from the roof of NY1.
One reason NY1 was able to accommodate CNN's needs so easily was that its own signal had been knocked off when Time Warner Cable's headends went dark.
When power was restored uptown and CNN left the NY1 facility, the real drama began. Truncale spent much of Aug. 15 trying to track down diesel fuel to keep the generators going. "We were down to our last three or four hours of generator power when the electricity came back."
