What: Service interruption to microwave and WiMAX services through Red Peak due to UPS battery exhaustion.
Where: Transmission tower in Sun Valley, NV
Duration of impact: August 8, 2013 20:02:47 to 20:27:09 local time.
Details
At 19:25:11 local time on August 8, 2013 a lightning strike at or near an NV Energy substation on Valley Road caused a widespread utility power outage in the north valleys, including Sun Valley and the transmitter towers on Red Peak where Roller Network leases space to operate licensed microwave transmitters and our WiMAX base station. At 19:25:27 we were notified of an emergency “on battery” condition.
Roller Network maintains a UPS to provide power until the site’s permanent automatic generator starts to provide emergency power, but our equipment remained on battery. Our procedure for this site is to deploy a portable generator of our own if we do not see a power restoration from the site generator within 30 seconds, and so the portable generator was deployed immediately. We were already proactively monitoring our extended network due to weather.
Unfortunately our UPS batteries were exhausted at 20:02:47 as we were heading up Clear Acre to the access road in Sun Valley (intersection of 1st and Sun Valley). We arrived to find the site dark; we were told the site generator had a dead starting battery. We deployed our portable generator with extension cords into the shelter building and restored power to our UPS at 20:27:09.
Analysis
We relied on our UPS for power up to 30 minutes, however in the event of a site generator fail-to-start (which is not ours to maintain) this proved to be insufficient time to deploy a portable generator. Travel to the site was delayed due to traffic signal outages and potentially life-threatening lightning conditions on the mountain top.
In response Roller Network will be deploying additional battery packs to provide more time to allow for dangerous site conditions were automatic generators not maintained by Roller Network exist to provide more buffer time to respond on our own.
Sites we operate without automatic generators already have sufficient battery packs for at least 4-5 hours of UPS runtime. Our procedure for these sites is to deploy portable generators after three hours on battery. Sites with automatic generators not maintained by Roller Network will now be treated the same as sites without generators.
No equipment was damaged nor was any manual intervention required beyond connecting the portable generator.
Customers with a service level agreement should submit requests for a service credit, if desired, within 30 days.
Timeline of Major Events
August 8, 2013
19:25:11 – Loss of utility power from NV Energy.
19:25:27 – Initial emergency notification.
19:26:00 – Portable generator deployment following emergency state.
20:02:47 – UPS battery exhaustion.
20:27:09 – Restored power with portable generator.
21:53:31 – Utility power restored.