dslreports logo
 
    All Forums Hot Topics Gallery
spc
Search similar:


uniqs
723

jasmo34
join:2008-03-20
~ London ~

1 edit

jasmo34

Member

Start Cable unaffected in London Tuesday Evening?

So, there was a major cable internet outage in London Tuesday Evening (8 p.m. to 10:30ish).

Acanac, Teksavvy, Distributel were all affected.

But, apparently, START Cable Internet customers were not affected.

Is this Start's "Aggregated POI" that saved them?

Or do they just happen to NOT use, or rent capacity on, the particular Rogers component that failed?
bt
join:2009-02-26
canada

bt

Member

said by jasmo34:

Is this Start's "Aggregated POI" that saved them?

Sounds very much like that is the case. If Rogers retail customers weren't affected, it points to an issue with the POI hand-off (or something after that point in the network). As Start doesn't get their hand-off at the more local level, they wouldn't be affected.

rocca
Start.ca
Premium Member
join:2008-11-16
London, ON

rocca to jasmo34

Premium Member

to jasmo34
I'm not sure of the exact details regarding what failed, but if I had to guess I would say either the London POI router failed or that a backhaul component that was tied to that router failed. Again, only assumptions but I'm sure you can get the specifics from the ISP's that were impacted.

With the aggregated model, we don't connect to the London POI router or use any of those components.

Teddy Boom
k kudos Received
Premium Member
join:2007-01-29
Toronto, ON

Teddy Boom

Premium Member

It probably wasn't an equipment failure at all. I should probably be posting this over in the Teksavvy forum, and if I find the right place later today I'll probably move it and just leave a link here, but meanwhile..

When Rogers does node and/or CMTS upgrades they shuffle customers around to other nodes/CMTSes while the new equipment is installed. When they are done they rebalance the customers on the new equipment. During this process the Teksavvy (and Distributel/Acanac/3web/cia) IP addresses are first shifted over to the temporary equipment, and then shifted again to the new equipment. If Rogers messes up those transfers--it is speculated that when a node is split they often dump all of the IP addresses on only one of the two new nodes--then there aren't enough IP addresses in the DHCP pool, and customers are knocked offline.

This was a major issue in Toronto over Christmas last year. All Fall groups of customers were being effected sporadically.
»blogs.teksavvy.com/2011/ ··· ntinued/

Rogers messed up some areas just before their holiday new work embargo came into place (I think that might have been December 13th?!). For whatever BS Rogers reason they considered fixing the problem to be new work, so they refused to help some of those Teksavvy customers until after New Years. Eventually Teksavvy made the astonishing step of offering free 16mbit DSL service to effected customers. I'm sure somebody at Rogers promised ritual suicide if it ever happened again.
»blogs.teksavvy.com/2011/ ··· -issues/

I'm not sure what Rogers has in store for you rocca, but there will be something eventually Meanwhile, it probably means that faster upload speeds are closer than ever.

I do wonder about how the back end infrastructure for Aggregated POI works, so that we can better understand the trade off between the two models. However, I guess it is hard to identify which of the thousand little things is likely to be of interest until something specific happens. I guess it is safe to assume that Start has one big DHCP pool for all of Rogers Ontario?

rocca
Start.ca
Premium Member
join:2008-11-16
London, ON

rocca

Premium Member

said by Teddy Boom:

I do wonder about how the back end infrastructure for Aggregated POI works, so that we can better understand the trade off between the two models. However, I guess it is hard to identify which of the thousand little things is likely to be of interest until something specific happens. I guess it is safe to assume that Start has one big DHCP pool for all of Rogers Ontario?

From a technical standpoint, we're all setup the same way at the CMTS, ie our IP space is also carved up at the node level, across the entire network with a thousand or so different DHCP pools. The difference between aggregated vs disaggregated is simply that we don't hand-off at each regional POI router, instead taking the same paths back to York Mills as the retail network and connecting to the APOI there.

jasmo34
join:2008-03-20
~ London ~

jasmo34 to Teddy Boom

Member

to Teddy Boom
said by Teddy Boom:

It probably wasn't an equipment failure at all. ...

Hmmm...

During the TSI outage, TSI Gabe posted "The latest feedback I have from Rogers it that they have a 10Gbps backhaul from London to Toronto that's completely down. I'm still following up with them." at 2012-10-09 21:35:43.
»Re: [Cable] outage in London?

Teddy Boom
k kudos Received
Premium Member
join:2007-01-29
Toronto, ON

Teddy Boom

Premium Member

said by jasmo34:

During the TSI outage, TSI Gabe posted "The latest feedback I have from Rogers it that they have a 10Gbps backhaul from London to Toronto that's completely down. I'm still following up with them." at 2012-10-09 21:35:43.
»Re: [Cable] outage in London?

Best to discuss this over there, but.. You can see all over the Teksavvy forum that there were DHCP issues as I describe that were resolved yesterday (or overnight?). It looks like this is a new issue. Probably all Rogers business (fibre-line?) customers would be effected by it.

rocca
Start.ca
Premium Member
join:2008-11-16
London, ON

rocca

Premium Member

said by Teddy Boom:

Probably all Rogers business (fibre-line?) customers would be effected by it.

We weren't. We have other transport services from them (not related to TPIA), including a link from London to Toronto which was unaffected.

Teddy Boom
k kudos Received
Premium Member
join:2007-01-29
Toronto, ON

Teddy Boom to rocca

Premium Member

to rocca
said by rocca:

From a technical standpoint, we're all setup the same way at the CMTS, ie our IP space is also carved up at the node level, across the entire network with a thousand or so different DHCP pools. The difference between aggregated vs disaggregated is simply that we don't hand-off at each regional POI router, instead taking the same paths back to York Mills as the retail network and connecting to the APOI there.

Can you do anything to correlate IP address to geography then? To anticipate node congestion issues say? I mean... I guess you can by reverse engineering it, even if Red doesn't tell you anything! I've long wondered about deploying linux routers with custom scripts on a volunteer basis so that TPIAs can keep track of local conditions regardless of whether Red cooperates or not.

And for another pipe dream, is Red ever going to provide a lantern like tool so that TPIAs can query customer modems?
Teddy Boom

1 edit

Teddy Boom to rocca

Premium Member

to rocca
said by rocca:

We weren't. We have other transport services from them (not related to TPIA), including a link from London to Toronto which was unaffected.

Haha, I forgot that you guys are in London

How many 10Gig links would London require, not including residential?! No usenet hosts in London, are there?

rocca
Start.ca
Premium Member
join:2008-11-16
London, ON

rocca to Teddy Boom

Premium Member

to Teddy Boom
said by Teddy Boom:

Can you do anything to correlate IP address to geography then? To anticipate node congestion issues say? I mean... I guess you can by reverse engineering it, even if Red doesn't tell you anything! I've long wondered about deploying linux routers with custom scripts on a volunteer basis so that TPIAs can keep track of local conditions regardless of whether Red cooperates or not.

It's an interesting idea, however I don't think you could get enough of the traffic monitored to be able to safely predict anything. Even if it was possible to get all the TPIA competitors to be able to correlate nodes and real-time traffic amongst themselves, at best you'd still only see 5% of the customers. Overall we've found that Red has been very good at updating the nodes with more channels and splits to avoid most local node issues.
said by Teddy Boom:

And for another pipe dream, is Red ever going to provide a lantern like tool so that TPIAs can query customer modems?

Not willingly.
rocca

rocca to Teddy Boom

Premium Member

to Teddy Boom
said by Teddy Boom:

Haha, I forgot that you guys are in London

How many 10Gig links would London require, not including residential?!

You'd be surprised.