Roller NetworkWe had a promising conference call with Verizon on Tuesday and a couple emails from Verizon representatives (who deferred to our assigned account team, whoever that may be, we don’t know), but unfortunately nobody who can officially help has yet to contact us as promised and no further progress has been made as of this post.
For us this is merely a continuation of the same problem we had in August/September when our original account manager disappeared.
To Verizon’s credit, we received a response back from Verizon that they’re going to escalate our issue to network engineering. We aren’t sure where it will go from there. In the meantime, there were several questions in the discussion that we should post as a followup.
That used to be true. ARIN (2620:0::/23), AFRINIC (2001:43F8::/29), and APNIC (2001:0DF0::/29) all allow for PI IPv6 to fill the gap that IPv6 left by excluding (sane, timely, managable) multihoming in the design of IPv6. RIPE decided to open the door to PI IPv6 at RIPE58. Multihoming was (is?) is a big point of contention in the community. The future of BGP may end up being a lookup-based system to reduce the overhead of carrying a full table.
Roller Network is not new to IPv6. We already offer IPv6 enabled services and have customers that use it, so yes, it’s important for us to keep the same level of service at the very least. No, it’s not a huge amount of traffic, but someday it might matter, and we want to be ahead of the game rather than be short sighted and scramble at the last minute. Adoption is also a classic “chicken and the egg” problem so we’re trying to do our part by providing IPv6 enabled services.
This is true. However, we did fully disclose our routing intentions and service requirements in May when we started all of this, and they confirmed it with the order so we were led to believe it was not a problem. However, the lost the order somewhere between May and August and turned it up with static IPv4 only. The implementations manager that processed the original order also went missing during that time and as of September our account manager was no longer with the company. Of course we forwarded copies of everything we had and it was re-engineered to a different city that supported IPv6. During all of this was bi-weekly calls for status updates, trying to find our new account manager, etc. That brings us to now.
We’ve put a lot of effort into this (it’s fiber) and we want to try to work through the myriad of bureaucracy to change things for the better as long as someone from Verizon is willing to work with us. It’s not just affecting us and our customers, but IPv6 adoption as a whole. A little publicity to an issue that has generally been kept quiet in the past might help. It could also hurt, but others have come before us quietly, so we’ll have to take a chance.
No. They’re not carrying routes in BGP most likely because it’s not allowed in their prefix lists. This is on an OC-12 delivered as Ethernet, not a residential connection. We wouldn’t have made such a big deal over this otherwise.
As mentioned before, we’re already getting the same service from other providers like Sprint. Even Hurricane Electric’s free tunnel broker supports IPv6 PI and BGP peering. We’re already visible in looking glasses worldwide. Verizon may have been right years ago prior to at least October 2006 when ARIN started issuing PI IPv6, but they’re not keeping up with where the industry is going. Again, the lack of a usable multihoming mechanism in IPv6 forced the issue.
Yeah, sorry about that. (And you actually tried to RTFA?) We weren’t expecting front page coverage. Our web server is very low traffic and mostly serves static pages. It’s a 700MHz PIII. However, it’s moving along just fine after adding the WP Super Cache plugin which provides static caching. Default wordpress really sucks performance-wise.
We’ll keep posting updates as they happen. If you’d like to keep following our Verizon saga, have an interest in IPv6, or both, please subscribe to our RSS feed. We’ve added an IPv6 category as well. If by some miracle Verizon moves in our favor we’ll try to post a followup story to Slashdot.
We were slashdotted today after our woeful story of Verizon made the front page. A basic WordPress install is very bad on server performance, and the fact that our public facing pages are all static means our web server wasn’t quite up to the task. But after some tuning and static caching, we’re in better shape.
Technology: Verizon Refuses To Provide Complete IPv6
Slashdot is welcome here; it’s a rare honor to be taken down by a front page link. 😉
On Saturday evening (September 3, 2009) we experienced a short general outage caused by our DNS and SQL cache pool servers rebooting. Normally this wouldn’t be a problem, but the PowerDNS Recursor package didn’t start at boot time, and a lack of DNS meant a general lack of anything useful taking place.
It was odd that some equipment rebooted itself while others didn’t, so after a lot of thinking, we decided that one of the APC 30A redundant switches we use was probably faulted because this isn’t the first time we’ve seen this. We pulled the switch from service (causing another reboot – although if we were right we didn’t want it risking the system anymore) and opened the cover. Inside we found relay contacts with pitting and arching:
The other relays we opened up weren’t nearly as bad, but they still exhibited discoloration and pitting on the contacts. The one in the pictures was loaded between 10 and 15 amps and it’s supposed to be rated for 30 (or 24 derated). Because this is the second failure we’ve had with this device we’ve decided to remove the remaining ones from service as they are likely to suffer the same fate in the future.
We apologize for the recent bumps in the normally smooth operation you’ve come to expect from us. We understand that your mail and DNS service is important to you, and to us, since we use the same services for our mail. As such, a discount/credit will be forthcoming.
UPDATE: Verizon Post-Slashdot Followup
The final word on Friday from Verizon is that they refuse to carry 29% of the IPv6 internet that is visible from their competitors. We calculated this percentage by taking the total number of paths Verizon provides and dividing it by the number of visible endpoints through Sprint and Hurricane Electric.
Specifically, they are completely blocking all of ARIN’s 2620:0::/23, so even by following their policies they’re still providing an incomplete view of the internet. It is their position that this is “correct”:
“If you wish your /48 to be visible globally, you’ll need to return your direct /48 allocation to ARIN and obtain a Verizon /48 from our network pool. Since our /48 assignment would be part of a /32 that we are announcing, your network would be globally routable. Otherwise, you are limited to AS701.”
Our response was:
Basically it seems to come down to 701 is a black hole for ARIN’s 2620:0::/23 (/40-/48) assignments. Let’s ignore my initial problem for a minute. Even if I were to switch to a /48 contained in your /32 or get my own /32, the following routes are still not available via 701:
Network Next Hop
*>i2620:0:30::/48 2620:0:950::242:129
*>i2620:0:70::/48 2620:0:950::242:129
*>i2620:0:80::/48 2620:0:950::242:129
*>i2620:0:90::/48 2620:0:950::242:129
*>i2620:0:B0::/48 2620:0:950::242:129
*>i2620:0:C0::/48 2620:0:950::242:129
*>i2620:0:110::/48 2620:0:950::242:129
*>i2620:0:140::/48 2620:0:950::242:129
*>i2620:0:150::/48 2620:0:950::242:129
*>i2620:0:230::/48 2620:0:950::242:129
*>i2620:0:240::/48 2620:0:950::242:129
*>i2620:0:280::/48 2620:0:950::242:129
*>i2620:0:2D0::/48 2620:0:950::242:129
*>i2620:0:310::/48 2620:0:950::242:129
*>i2620:0:320::/48 2620:0:950::242:129
*>i2620:0:330::/48 2620:0:950::242:129
*>i2620:0:380::/48 2620:0:950::242:129
*>i2620:0:380:2::/64
2620:0:950::242:128
*>i2620:0:3F0::/48 2620:0:950::242:129
*>i2620:0:630::/48 2620:0:950::242:129
*>i2620:0:6B0::/48 2620:0:950::242:129
*>i2620:0:6C0::/48 2620:0:950::242:129
*>i2620:0:6C1::/48 2620:0:950::242:129
*>i2620:0:6C2::/48 2620:0:950::242:129
*>i2620:0:800::/47 2620:0:950::242:129
*>i2620:0:802::/48 2620:0:950::242:129
*>i2620:0:830::/48 2620:0:950::242:129
*>i2620:0:860::/46 2620:0:950::242:129
*>i2620:0:862::/48 2620:0:950::242:129
*>i2620:0:870::/48 2620:0:950::242:129
*>i2620:0:880::/48 2620:0:950::242:129
*>i2620:0:8F0::/48 2620:0:950::242:129
*>i2620:0:930::/48 2620:0:950::242:129
*>i2620:0:950::/48 2620:0:950::242:130
*>i2620:0:960::/48 2620:0:950::242:129
*>i2620:0:9B0::/48 2620:0:950::242:129
*>i2620:0:9C0::/48 2620:0:950::242:129
*>i2620:0:A00::/48 2620:0:950::242:129
*>i2620:0:A00::/43 2620:0:950::242:129
*>i2620:0:A01::/48 2620:0:950::242:129
*>i2620:0:A02::/48 2620:0:950::242:129
*>i2620:0:A03::/48 2620:0:950::242:129
*>i2620:0:A04::/48 2620:0:950::242:129
*>i2620:0:A05::/48 2620:0:950::242:129
*>i2620:0:A06::/48 2620:0:950::242:129
*>i2620:0:A07::/48 2620:0:950::242:129
*>i2620:0:A09::/48 2620:0:950::242:129
*>i2620:0:A0D::/48 2620:0:950::242:129
*>i2620:0:A10::/48 2620:0:950::242:129
*>i2620:0:A16::/48 2620:0:950::242:129
*>i2620:0:A17::/48 2620:0:950::242:129
*>i2620:0:A1A::/48 2620:0:950::242:129
*>i2620:0:A1C::/48 2620:0:950::242:129
*>i2620:0:A1D::/48 2620:0:950::242:129
*>i2620:0:A1E::/48 2620:0:950::242:129
*>i2620:0:A1F::/48 2620:0:950::242:129
*>i2620:0:B10::/46 2620:0:950::242:129
*>i2620:0:B60::/48 2620:0:950::242:129
*>i2620:0:C30::/48 2620:0:950::242:129
*>i2620:0:C80::/48 2620:0:950::242:129
*>i2620:0:CA0::/48 2620:0:950::242:129
*>i2620:0:CC0::/47 2620:0:950::242:129
*>i2620:0:CC0::/44 2620:0:950::242:129
*>i2620:0:CCA::/48 2620:0:950::242:129
*>i2620:0:CCB::/48 2620:0:950::242:129
*>i2620:0:CCC::/48 2620:0:950::242:129
*>i2620:0:CCD::/48 2620:0:950::242:129
*>i2620:0:CCF::/48 2620:0:950::242:129
*>i2620:0:CF0::/48 2620:0:950::242:129
*>i2620:0:D20::/48 2620:0:950::242:129
*>i2620:0:D60::/46 2620:0:950::242:129
*>i2620:0:DA0::/48 2620:0:950::242:129
*>i2620:0:DC0::/48 2620:0:950::242:129
*>i2620:0:DD0::/48 2620:0:950::242:129
*>i2620:0:DF0::/48 2620:0:950::242:129
*>i2620:0:E50::/48 2620:0:950::242:129
*>i2620:0:EF0::/48 2620:0:950::242:129
*>i2620:0:1000::/48 2620:0:950::242:129
*>i2620:0:1002::/48 2620:0:950::242:129
*>i2620:0:101E::/48 2620:0:950::242:129
*>i2620:0:1040::/48 2620:0:950::242:129
*>i2620:0:105D::/48 2620:0:950::242:129
*>i2620:0:105E::/48 2620:0:950::242:129
*>i2620:0:105F::/48 2620:0:950::242:129
*>i2620:0:1080::/48 2620:0:950::242:129
*>i2620:0:10A0::/48 2620:0:950::242:129
*>i2620:0:10A1::/48 2620:0:950::242:129
*>i2620:0:1700::/45 2620:0:950::242:129
*>i2620:0:1A10::/48 2620:0:950::242:129
*>i2620:0:1A50::/48 2620:0:950::242:129
*>i2620:0:1B00::/48 2620:0:950::242:129
*>i2620:0:2220::/48 2620:0:950::242:129
*>i2620:0:22F0::/48 2620:0:950::242:129
*>i2620:0:2830::/48 2620:0:950::242:129
*>i2620:0:2860::/48 2620:0:950::242:129
*>i2620:0:2890::/48 2620:0:950::242:129
*>i2620:0:2B10::/48 2620:0:950::242:129That’s not exactly something to brush off (ignoring the random /64); that’s a big enough chunk missing to make 701 look undesirable to someone like me who wants to heavily promote IPv6. Compared to my existing IPv6 table, 701 is missing 29% of the IPv6 internet that I can already reach.
And based on their position, they’re probably (although we have not confirmed, but based on the 29% figure we came up with it is extremely likely) missing similar ranges from the other regional registries.