Roller NetworkWe recently enabled IPv6 in Postfix on our outgoing mail (SMTP AUTH, smarthost) service. This means that if a destination you’re sending mail to is accessible via IPv6 it will automatically connect and deliver using IPv6. If not, it will transparently fall back to IPv4 and continue to deliver as before.
We finally have a Verizon account manager for the first time since we last heard from our original account manager on August 3. While the IPv6 issues are not resolved, we’re trying to put a rush on the fixes we needed to be made back in August so we can at least have some basic connectivity while Verizon performs an engineering assessment with regards to IPv6. To replace the circuit would take another 3 to 6 months to get new fiber brought in, so we’re going to stick with it and see what the results of the engineering assessment are.
The fiber we have is Verizon-provided but our ILEC is AT&T which means that nobody but Verizon can use it. Back in the 90’s a company called Brooks Fiber installed a whole bunch of fiber in Reno. They were bought by MCI and MCI/UUNet eventually became Verizon Business. While we are technically in a lit building because it already had a fiber on the property, it’s a light industrial park and there was no equipment that could carry our order and a new OC-12 was installed to our telco room.
We successfully moved the entire facility over the weekend with a few minor hiccups with hosted mail service that didn’t show up during testing. In any case, everything is back to normal as of Monday morning. There’s still a lot of detail work to do and everything else (like office furniture) still needs to make its way over to the new place, but at this point there is nothing left to move that could impact our normal operations.
There, however, is one negative aspect to report: because Verizon is still unable to provide service to us we are running single-homed on our Sprint service. This should not be a problem as long as Sprint doesn’t have any outages, however, this is not something we’re comfortable with. We’ve tried to make it clear to Verizon that no matter what their policy is doesn’t change the fact that we need the service and someone will provide it.
We’re on the eve of the move and this is probably the last new facility update before actually moving in. Verizon is still missing in action, but we’re going ahead without them since it’s clear that it isn’t getting resolved anytime soon. We would like to thank Sprint for being patient with us since they were ready to hot cut back in August and we kept delaying to try to resolve things with Verizon.
Final layout of a UPS and its battery cabinet before wiring. Three of these complete units will fit into our UPS electrical room (up to 4 supported) to act as one parallel redundant unit.
A close up of the UPS wiring. The cable is 1/0 AWG.
Some of the overhead piping in the UPS electrical room.
One of the two distribution panels for the server room (right) and the UPS paralleling panelboard (left).
Overhead piping between the UPS electrical room to the panelboard that supports it. The panelboard is connected to our generator automatic transfer switch allowing it to be fed by utility power or generator power. As you can see, this is in the warehouse portion of the facility, which will house the generator.
The original load center (left), the new panelboard for our stuff (middle), and our automatic generator transfer switch (right).
The panelboard with its dead front installed, wiring cleaned up, and the transfer switch after turning the power on for the first time. The generator will be installed within the week and piped to the connectors on top of the transfer switch. Wiring the generator is as simple as connecting it to the “emergency” contacts inside the transfer switch after the not so simple process of moving a very heavy diesel engine.
The next update will be after the major move of circuits and equipment this weekend. There’s still plenty for us to do before we can consider it done and take a break, so stay tuned!
The time for us to move our facility has come. We originally planned to move the last week of August in order to open in October, but if you’re been following our story with Verizon you already know why that didn’t work out. The Verizon circuit was to be used to allow a minimal disruption outage by tunneling both sites. However, they are still not following through, and we can’t wait forever.
As such, we will be performing a hot-cut of our Sprint service on Friday, October 16, 2009 at 14:00 US/Pacific time (UTC-8). All traffic will be rerouted over our lower capacity SAVVIS circuit beginning at 13:30 UTC-8. We will begin announcing our address space via Sprint at the new location once at least 50% of the equipment has been moved and stabilized. We have timed the hot-cut to correspond with the beginning of the weekend where our average utilization falls within what the SAVVIS circuit can handle, although the slowdown may still be noticeable. Unfortunately this unavoidable due to the Verizon trouble.
THERE WILL NOT BE A COMPLETE OUTAGE. Groups of equipment are staged so that their backup/secondary remains online while the primary is moved. Once the primary is moved the secondary will follow. You may notice some brief hiccups as traffic is rerouted between locations, but you should not expect any substantial outages. Non-critical services (such as our company website and the account control center) do not have redundant counterparts and will be offline for 15 to 30 minutes while the servers are moved. Critical services (mail services, hosted mail, DNS services) will not be interrupted.
Because the move will affect our network in a substantial way we have moved our status page to an offsite location served by third-party DNS. You can follow the move progress online at:
This new status page will be permanently replacing the old status page at status.rollernet.us. If you have bookmarked the old status page, please update your bookmarks.
Our goal is to ensure that the move is as painless and transparent as possible. If you have any questions or concerns please feel free to contact us. Further updates will be posted on the newspipe (www.rollernet.us/wordpress) and status page (rollernetstatus.com) as we progress.
