There was a power outage today at 11:57 UTC-8 caused by this:
We’re actually about a mile away from it, but as you can see it hit the end of the line, popped the top middle phase, and tripped the whole line we happen to be on. Everything was fine until 14:33 UTC-8 when our new Eaton 9355 installation decided to cut power to the protected load. The alarms it gave were simultaneous “Input AC Over Voltage”, “Utility Fail” followed by “DC Link Over Voltage”. We’re guessing the generator momentarily went outside of tolerance. We’re not clear why that would cause the UPS to shut off (isn’t this the whole point of the UPS?), but it refused to turn back on and kept giving “DC link over voltage”. Because we were offline at this point, we activated the external bypass on the paralleling cabinet to tie the generator directly to the server room distribution and get things back online ASAP. We called the Eaton support hotline and the tech advised us to reset the logic by cycling all the breakers open then closed, however the unit still failed to start citing the “DC link over voltage” alarm. We do have a 24 hour service contract for the unit, so someone is on the way to take a look at it.
[Nov. 23 00:49] Eaton took a look at the unit and was unable to duplicate the problem. The only difference is that it was supplied by utility power and not by generator as we do not wish to experiment without a dummy load and risk an outage. At this time the bypasses have been removed, the generator stopped, and the UPS is back online supplying power to the protected load. We will also have the generator looked at as we suspect the most likely cause of the trouble could be the voltage regulator losing lock or the electronic governor. The units we have are transformerless and the “DC link over voltage” error can be caused by a problem with the neutral reference point going out of bounds.
[Nov. 23 13:00] Our diesel mechanic will be replacing all of the injectors on our generator. One of them had a blown port due to water.
[Nov. 24 10:28] According to the news, the driver of the van dropped something. (Source: KOLO) West Huffaker Lane is 5 lanes wide, four travel and one center turn lane.
View Outage Notice Nov. 22, 2009 in a larger map
3 replies on “Outage Notice Nov. 22, 2009”
Perhaps you should have bought the optional Star/Delta input isolation transformer. This helps protect against adverse neutral currents caused by odd current harmonics. It does reduce efficiency slightly but provides somewhere for the harmonic energy to be dissipated other than in the electronics. Unfortunately, I don’t think it’s a retro-fit option within the cabinet of the 9355 but an external transformer should be possible depending on your maintenance bypass arrangements. Take advice from Powerware engineers.
As a data center infrastructure engineer, I know that neutral currents can cause severe problems. Modern UPS systems like the Powerware 9355 cope fairly well with high crest factor loads just as computer manufacturers are improving load characteristics with (misnamed) power-factor corrected switch mode supplies.
The 9355 is optimised for a power factor of 0.9 lagging but if you have ALL power-factor corrected loads, you will have to de-rate from 15kVA to 13.5kVA. Make sure you have enough “headroom” and pay careful attention to the neutral and earthing schematic between utility, generator, UPS and load.
Finally, now that most computer and communications equipment is fitted with dual power supplies for redundancy, consider a second UPS installation so that each unit power supply can be fed from separate UPS systems and will continue operation upon the failure of one. Do NOT fitted load transfer switches or any other cross-coupling whose failure could bring down both UPS systems including the Powerware HotSync redundancy. Maintain separate pathways as far back the power chain as possible – even separate generators if financially viable. It will have to be a commercial decision that defines the amount of resilience that can be built in to your data center infrastructure.
The 9355 series is quite small, even the largest one is smaller than a standard 4-post cabinet. If you order it with the input transformer you’re really just ordering an external transformer since there’s no space inside, but we have plenty of utility space outside the UPS room to add the input transformer if needed. We have two bypasses; the standard UPS wrap around plus the external bypass on the parallel tie cabinet. We closed the tie cabinet bypass in this instance.
Of course they couldn’t duplicate the problem after the fact, and they said it should have just switched to battery, but the only reason it would report DC link over/under voltages other than a hardware fault is if the neutral reference was lost. They looked at our generator transfer switch to see if it’s switched neutral (it’s not), so there isn’t an explanation yet. The EPROM values and event logs were pulled and will be sent to engineering for review since it dumped the load.
To follow up on this: the problem was ultimately discovered to be harmonic distortion from the UPS causing the regulator on the generator to lose sync. It was corrected by adding a 6kW resistive load from a load bank on the feed side. We exercise twice a month on load (there aren’t any generator time regulations in Nevada like our neighbors in California have) and all runs with the added resistive load have been flawless.
6kW isn’t much, but during testing we found it was all that was needed to bring stability to the mixed capacitive load the UPS creates.