Category: Status

A year ago on Nov. 22, 2009 shortly after we moved into our new facility with our then new UPS system, we experienced a generator fault that took our UPS offline. We’re fortunate that this incident took place early and afforded us the opportunity to fine tune our electrical system so that we can now mark a full year of uninterrupted service at Roller Network’s new facility in Reno.

Today, November 22, 2010 at 14:33 local time (UTC-8) we are pleased to officially announce 365 days of uninterrupted facility availability at Roller Network. Over the last year our facility has successfully insulated its customers from many external events caused by weather or human error as it was intended to.

• We’ve tested our generator on load 26 times, or every two weeks, for a period of one hour each. These tests were performed in a fully automatic manner without any human intervention.
• There have been at least two utility outages that initiated an automatic generator start and transfer outside of the scheduled testing. In conjunction with the UPS, the generator successfully powered our facility for several hours until utility power was restored.
• We performed at least one “lights out” test. In this test we open the building service disconnect to simulate a utility outage, whereas the bi-weekly generator tests are performed by the exerciser clock with an option to fall back to utility power. This also lets us practice running the business office without full power available and refine procedures.

Thank you to all of our customers who continue to support us: it is because of you that we are able to continue to maintain our facility to such standards.

Our card processor deployed an update yesterday that has subsequently broken our ability to submit transactions (all attempts are returning an authentication error). We are attempting to resolve the situation as quickly as possible.

UPDATE: After dumping the raw response we’re getting from the processor, it turns out to be a “500 Internal Server Error” message, so the problem is definitely not with us. We have contacted them for resolution.

UPDATE 2: There is still no resolution at this time. We have changed our PayPal account to allow credit card payments without requiring a PayPal account.

UPDATE 3: It looks like the problem is an extra field in the response string toward the end where the MD5 checksum field is; the checksum was shifted up one offset, resulting in our side looking at a blank field. This does not match the authorize.net guide, so we are still working with the processor to find out why there is an extra field in the response. The critical fields (trans id, invoice number, authorization, etc.) are in the correct location, so as a workaround we are disabling the checksum test.

Due to this issue, auto-pay did not run, and some other transactions were voided. We are reviewing everything since November 2 and will manually run auto-pay for accounts that were affected or contact those who need to re-submit a payment.

Last week we reported on a potential spam issue. Several customers sent us additional information in response to this post, and we have come to the conclusion that there was not any form of security breach with our account control center based on the following:

• None of the accounts we maintain received a copy of the spam. If the email addresses had come directly from us then these accounts would have been included, but they were not.
• One customer received a copy of the spam to an old email address that hasn’t been in our database for over two years; it could not have been obtained from us since we don’t keep a history of email changes.
• All of the addresses that received a copy of the spam were sent to our merchant account. We immediately notified them and altered our system to send a generic email address under our domain instead of the customer’s email.
• There is no indication in the logs or netflow records of access from outside of our network to the systems that have access to this data. Furthermore, the master database server is not internet accessible, and the admin interface to look up an account first requires an individual account name or a domain.

For those that may be concerned about their financial information, we have no reason to believe this was anything more than some kind of email scrape at the processor side. The processor does not store credit card numbers.

Again, we apologize for this issue. Security and privacy with our services are extremely important to us. We do, in fact, use our own services (take a look at the MX records for our domain) right along with our customers, including managing our domains with the same account control center.