Category: Status

UPDATE (Fri Oct 12 13:01:51 PDT 2012): AOL appears to have resolved their problem (at least for us); everything waiting for them was suddenly delivered.

UPDATE (Fri Oct 12 08:20:38 PDT 2012): Several other providers, organizations, and individuals have conformed this issue in the comments here and privately to us, so it’s definitely an AOL problem. There’s nothing anyone can really do except wait it out and recommend non-email contact methods for anyone using an AOL hosted email address. Our original report continues below.

During the morning of August 28, 2012 starting at approximately 02:34 PDT, an excessive amount of high PPS UDP traffic was directed at our Secondary DNS service for several hours. In response we activated remote triggered blackholes with our transit providers for the impacted IPv4 addresses. During this time we monitored our netflow data in an attempt to minimize disruptions to Secondary DNS. Only IPv4 was impacted, queries over IPv6 were uninterrupted. We apologize for the inconvenience.

Our credit card processor is having some unexplained trouble; we are seeing a “500 internal server error” from their gateway since we were made aware of it around 10:00 UTC-7 this morning. At this time we do not have any explanation or acknowledgement from them as to the nature of the problem.

We are working to resolve this issue as quickly as possible including the possibility of changing processors.

UPDATE (April 9 @ 08:49): Our credit card processor gateway is returning “500 internal server error” without any explanation. We are now working to change processors since tech support from our current merchant account provider has been unresponsive.

UPDATE (April 9 @ 10:35): We have set up a new merchant account with Wells Fargo which should be active tomorrow afternoon, April 10.

UPDATE (April 9 @ 10:35): We have placed service holds on billing for all current Roller Network accounts.

UPDATE (April 9 @ 16:07): We are still seeing “500 internal server error”. We are working under the assumption that the current gateway won’t be fixed before the one we’re switching to is ready.

UPDATE (April 10 @ 08:37): The error is still being returned; we are temporarily allowing PayPal payments without a PayPal account as an alternative. We are in the process of switching to a new card processor this afternoon at the earliest.

UPDATE (April 11): Our current merchant account provider has responded requesting further details; unfortunately there is still no solution to the “500 internal server error” responses. Some requests are being accepted successfully, however we are still seeing “500 internal server error” intermittently across all card types.

UPDATE (April 13 @ 11:00): We are now waiting for our new merchant account to finish activation.

UPDATE (April 13 @ 11:53): We are now configuring and testing the new gateway.

RESOLUTION: We have changed to a new credit card processing gateway as of April 13 @ 13:00UTC-7. Contact technical support if you experience any ongoing errors with payments. Additionally, we can now accept Discover cards.

Eaton will be on site on Friday, March 2 (weather permitting) Thursday, March 8 at 10:30 local time to perform the annual inspection on our UPS and battery system. No potentially intrusive actions will be taken. If any corrective maintenance is deemed necessary during the inspection, it will be deferred until an appropriate time.

UPDATE: The annual inspection has been deferred a week.

UPDATE 2 (March 8): Eaton is currently on site performing the annual maintenance and inspection.

UPDATE 3: This year’s annual preventative maintenance has been successfully completed.