Author: Roller Network

Verizon Soon?

It’s been a while since our last Verizon update, but today we received word that they have given us a Firm Order Commitment (FOC) of 12/4/2009 for the corrections that needed to be made. Since that’s a Friday, it means they probably will get started next week at the earliest, not that it will be ready to put into production. We’re not entirely sure what exactly they’re delivering since someone was already out on November 13 to test and qualify a new port on the existing install.

Unfortunately this snafu has caused massive delays in being able to accept additional colocation and hosting customers due to a lack of redundancy and bandwidth. We’re trying to push the resolution forward as best we can, and we appreciate your patience.

Announcements Status

Outage Notice Nov. 22, 2009

Post author By Roller Network
Post date November 22, 2009
3 Comments on Outage Notice Nov. 22, 2009

There was a power outage today at 11:57 UTC-8 caused by this:

We’re actually about a mile away from it, but as you can see it hit the end of the line, popped the top middle phase, and tripped the whole line we happen to be on. Everything was fine until 14:33 UTC-8 when our new Eaton 9355 installation decided to cut power to the protected load. The alarms it gave were simultaneous “Input AC Over Voltage”, “Utility Fail” followed by “DC Link Over Voltage”. We’re guessing the generator momentarily went outside of tolerance. We’re not clear why that would cause the UPS to shut off (isn’t this the whole point of the UPS?), but it refused to turn back on and kept giving “DC link over voltage”. Because we were offline at this point, we activated the external bypass on the paralleling cabinet to tie the generator directly to the server room distribution and get things back online ASAP. We called the Eaton support hotline and the tech advised us to reset the logic by cycling all the breakers open then closed, however the unit still failed to start citing the “DC link over voltage” alarm. We do have a 24 hour service contract for the unit, so someone is on the way to take a look at it.

[Nov. 23 00:49] Eaton took a look at the unit and was unable to duplicate the problem. The only difference is that it was supplied by utility power and not by generator as we do not wish to experiment without a dummy load and risk an outage. At this time the bypasses have been removed, the generator stopped, and the UPS is back online supplying power to the protected load. We will also have the generator looked at as we suspect the most likely cause of the trouble could be the voltage regulator losing lock or the electronic governor. The units we have are transformerless and the “DC link over voltage” error can be caused by a problem with the neutral reference point going out of bounds.

[Nov. 23 13:00] Our diesel mechanic will be replacing all of the injectors on our generator. One of them had a blown port due to water.

[Nov. 24 10:28] According to the news, the driver of the van dropped something. (Source: KOLO) West Huffaker Lane is 5 lanes wide, four travel and one center turn lane.

View Outage Notice Nov. 22, 2009 in a larger map

Changes

Changes to “Client Validity Checks” Filter

Post author By Roller Network
Post date November 18, 2009

We made two changes to the Client Validity Checks filter:

1) All test options now have both reject and defer actions available.

2) A “strict” option was added for the HELO name test. The existing test would include subdomains, while the strict option will only perform an exact match.

Announcements

Perl API Interface

Post author By Roller Network
Post date November 11, 2009

Keith Wessel has graciously provided a Perl module interface to our control center API. We’ll be posting a download link on the API help page soon, but for now you can get it here: RollernetClient.pm Version 1.0

Uncategorized

New Facility Progress #11

A generator awaiting installation.

The “rollers”, our unofficial mascots that take residence in the server room on top of a router.

Detail view of the electrical outlet installation. The smallest circuit we offer is 20A at 208VAC in single phase or three phase, but we can easily provide multiple 20 and 30 amp circuits per rack in single phase or three phase for high density racks. (We do not offer 120VAC circuits.)

Detail view of the patchrack. By keeping patch panels out of the racks and using the overhead ladder to route them where they need to be we increase server density and eliminate wasted ports. If high port density per rack is required, top-of-rack switching would be used.

Detailed view of the overhead area. Even though you can’t see this without a ladder, we like to keep everything in order.

Power cord cable management for the router and switch racks.

A view of the server room. What was spread out over five racks in the old facility was consolidated into two racks (not counting the two-post racks to the right). The extra space will be offered for colocations and dedicated servers and we will continue installing racks to fill the room. The yellow cables hanging out of the wire management on the right indicate temporary cabling.