On October 25, 2012 beginning at 07:07:26 local time our Level 3 transit circuit was impacted for 21 minutes. During this time we continued to receive a full BGP route table, our route announcements were accepted, and bidirectional forwarding detection (BFD) remained in an “up” state, but traffic was not routing/forwarding properly. In a typical circuit or router fault condition BFD would cause the BGP neighbor to go idle and our autonomous system would recalculate new transit paths. In this case, however, a manual administrative shutdown of the interface (and BGP neighbors to bring the interface up for testing) was required. After ceasing to accept/announce routes via Level 3’s AS, transit through and to Roller Network’s AS was fully restored. At no time was traffic transiting via Sprint impacted.
Official explanation from Level 3: “Initial Blue Major (Cisco 7609) Due to routing issues slot 3 needed an emergency reload. It is believed that since Cisco messed with that card and ran debug commands last night and this morning, that this caused some corruption or instability in the card.”
We initially reported this issue via our Twitter account @rollernetnv.