smcallah
join:2004-08-05
Home
| reply to bmn
Re: That's right
Level(3) can't use the 75 day notice, if it does indeed exist, as an excuse. Because they didn't notify their customers at all before they turned it off. Not 75 days before, not 30 days before, not 1 week before, not 24 hours before. Not even 1 hour before.
The Nov. 9th date is in an official press release. |
|
jnxrox
join:2005-10-07
Las Vegas, NV
| Take a step back
Level3's general customer profile is not a small/med, single-homed business. They cater to large multi-homed carriers and businesses. Cogent's customer base is much more vunerable here. It's the single-homed level3 customer to the Cogent single-homed customer that is impacted and as low cost vendor to the masses, that fits the cogent profile more. Level3 isn't going to connect to transit to satisfy such a small % of customers just to get to Cogent's single-homed customers. Cogent has more skin in the game and you saw that desperation in their initial responses.
Disconnecting peering happens. Cogent was notified and did not make arrangements for their customers. Level3 disconnected and that was their plan. If you check the reality on the connectivity affected, it's smaller than Cogent would lead you to believe. Most companies are multi-homed and in fine shape for re-routing around these issues. |
|
bmn
? ? ?
Premium,ExMod 2003-06
join:2001-03-15
hiatus
| reply to smcallah
Re: That's right
said by smcallah  :Level(3) can't use the 75 day notice, if it does indeed exist, as an excuse. Actually they can. By giving Cogent at least 75 days notice, it gave Cogent the oppurtunity to prepare for the peering sessions to be ended.
Because they didn't notify their customers at all before they turned it off. Not 75 days before, not 30 days before, not 1 week before, not 24 hours before. Not even 1 hour before. They don't have to nor should they have to, otherwise Level3 would be required to notify customers of every tiny change. All they are required to do is tell Cogent. Once Cogent is notified, the ball is 100% in Cogent's court - they either fix their traffic profiles or they take measures to ensure routes to Level3.
As it stands, from the information available, Cogent totally dropped the ball here and this problem, considering they were given notice, appears to be their fault for the most part. They knew they would be cut off and failed to ensure they had alternate routes to Level3.
--
Tor server operator... Helping the free flow of information daily. |
|
smcallah
join:2004-08-05
Home
| said by bmn :They don't have to nor should they have to, otherwise Level3 would be required to notify customers of every tiny change. Tiny change? You think THIS was a tiny change?
Level(3) DOES notify its customers of major changes in the network. Most large providers I know do this, it's in their contracts with their customers. Especially ones with Service Level Agreements that require daily or monthly credits to be given when that SLA is not met.
I can't believe you consider this a tiny change that Level(3) made. |
|
smcallah
join:2004-08-05
Home
| reply to jnxrox
Re: Take a step back
Well, since Level(3) turned this back on because of customer complaints, obviously there was a lot of need for their customers to get to Cogent.
If it was as "small" an impact as you say, Level(3) wouldn't have been bothered by a "few" customer complaints.
Also, sometimes multihomed customers don't want to use their backup connection, because they may be paying more per megabit for it. Say you're paying Level(3) for a 155 mbps connection, and it's $50/mbit. And you have another connection to Provider X, who charges you $150/mbit, but ONLY if you use them. So that peer is running at 0, you're paying them nothing until you use it. Then suddenly 20% of your 155 mbps goes to your backup connection. All of a sudden you're looking at paying an extra $4600 a month that you didn't have to pay, because now one of your providers doesn't have any routes to Cogent.
Many multihomed customers are setup like that. They have a main link, and a backup. Sure, it was their choice to get such a link. But it is people like this that can pressure Level(3) to do what they had to do. I worked for a year for a small hosting company that had a Level(3) connection and a connection to another ISP that worked like this. |
|
vferrari3
join:2005-10-07
Boston, MA
| said by smcallah :Well, since Level(3) turned this back on because of customer complaints, obviously there was a lot of need for their customers to get to Cogent. If it was as "small" an impact as you say, Level(3) wouldn't have been bothered by a "few" customer complaints. It wasn't turned back on because of customer complaints. It was turned on to help Cogent which was the more desperate party here. Notice the lack of information from Level3 as opposed to the screaming and press releases coming out of Cogent. It will be the same in Nov., Level3 will not pay to reach a small % of single homed cogent customers.
Mostly you'll still use the Level3 link to get to the many Cogent multi-homed customers, you'll just take a different path. But, yes in your scenario you would start using the unused link to get to the single-homed Cogent customers. Tell me why this is Level3's problem? They run their business, you run yours. With all the people up in arms there's one thing that seems to always be forgotten...this is not an unusual thing to happen, the Cogent response is what is the unusual piece. They did the same thing with others in the past and had to purchase routes to fill in their 'gaps'. You never hear about the other times carriers de-peer with each other because it's taken care of during the period of notice. Cogent uses the period to call the other carrier's bluff, and when connectivity is lost they smear. It's happened more than once.
This was more a PR war then anything and Cogent won the battle.
_VF |
|
bmn
? ? ?
Premium,ExMod 2003-06
join:2001-03-15
hiatus
| reply to smcallah
Re: That's right
said by smcallah :said by bmn :They don't have to nor should they have to, otherwise Level3 would be required to notify customers of every tiny change. Tiny change? You think THIS was a tiny change? No, but you misunderstood, so I'll clarify...
Level(3) shouldn't need to be required to notify its customers of a peering change, otherwise then they would have to start to notifying their customers to tiny changes to the network. Is that more clear ?
Level(3) DOES notify its customers of major changes in the network. Most large providers I know do this, it's in their contracts with their customers. Especially ones with Service Level Agreements that require daily or monthly credits to be given when that SLA is not met.
I can't believe you consider this a tiny change that Level(3) made. Actually, you are right, this isn't a tiny change in the network, but I never considered it one. The point here is that it was Cogent's responsibility to make sure that it had routes to Level3 once Level3 notified them about the peering sessions being dropped. Level3 incorrectly assumed that Cogent would do ITS job and find alternate paths to their network via other transits providers.
--
Tor server operator... Helping the free flow of information daily. |
|
smcallah
join:2004-08-05
Home
| I still think you misunderstood.
Obviously Level(3) customers were pulling a lot of traffic from Cogent, or there wouldn't have been an issue here.
So this peering change would have been a change worthy of notifying their customers. They didn't notify their customers of it. And suddenly their customers' customers were complaining, and Level(3)'s customers could only say, "Not our fault." But that doesn't make customers understand the problem, even if it was explained to them, a non-technical customer isn't going to understand, neither are the ones that think they know what BGP is.
So Level(3) had RoadRunner and several large dialup ISP's fielding calls, giving credits, and losing customers during the almost 3 days that this was down.
I don't see how you think that notifying their customers of a major peering change would suddenly open them up to having to notify of every tiny change.
And what makes it ONLY Cogent's responsibility to make sure they could get Level(3) routes? Why wouldn't Level(3) have a resposibility to give their own customers routes to Cogent? And now obviously they have taken up that responsibility, by turning this back on because of customer complaints. |
|
smcallah
join:2004-08-05
Home
| reply to vferrari3
Re: Take a step back
said by vferrari3 : It wasn't turned back on because of customer complaints. Sorry, but it was because of Level(3) customer complaints that this was turned back on.
Level(3) didn't care about helping Cogent, they cared about helping themselves. If they cared about helping Cogent, they would have extended this without turning it off.
But since Level(3) also had losing customers and losing revenue on its mind, they had to turn this back on. They were the only ones with the ability to turn it on.
It doesn't matter if it's Level(3)'s fault or not. The only reason they exist is for customers. If they lose those, what do they have? |
|
bmn
? ? ?
Premium,ExMod 2003-06
join:2001-03-15
hiatus
| reply to smcallah
Re: That's right
said by smcallah :I still think you misunderstood. Obviously Level(3) customers were pulling a lot of traffic from Cogent, or there wouldn't have been an issue here. No, the issue is that Cogent doesn't have enough users on its network to balance out the amount of traffic being sent to other networks. I recommend checking out when AOL dropped its SFI with Cogent and when C&W dropped its SFI with Cogent
said by smcallah :And what makes it ONLY Cogent's responsibility to make sure they could get Level(3) routes? Why wouldn't Level(3) have a resposibility to give their own customers routes to Cogent? And now obviously they have taken up that responsibility, by turning this back on because of customer complaints. Because, as an upstream Tier 1, Level3 receives the routes TO Congent's network FROM Cogent or via Cogent's transit providers. Without receiving those routes from Cogent, Level3 has no way of knowing how to get there... Level3 can't magically guess how to get to Congent without getting BGP information from Cogent. Level3 is not the one responsible to make sure that it can even see Cogent's networks since they don't have any control over Cogent's BGP information.
And the reason that Level3 took up the responsibility of getting routes to Cogent is because Cogent did not.
--
Tor server operator... Helping the free flow of information daily. |
|
vferrari3
join:2005-10-07
Boston, MA
| reply to smcallah
Re: Take a step back
said by smcallah :Sorry, but it was because of Level(3) customer complaints that this was turned back on. Level(3) didn't care about helping Cogent, they cared about helping themselves. If they cared about helping Cogent, they would have extended this without turning it off. You don't get it. They have no obligation to extend without turning it off. Peering is for mutual benefit. When one party no longer perceives benefit they back out. The party that relies on that arrangement too much is the weaker of the two. Level3 turned it back on because clearly there were PR implications where the public was not understanding the issues and they gave Cogent a reprieve. They knew what they were doing, they've done it before and come november, they will still not have those Cogent routes unless cogent utilizes their transit providers. Level3 did a risk analysis and/or cost analysis. The arrangement doesn't work for them. There's nothing wrong with that. They believe that the cost to keep the peering up is too much and is not balanced by the revenue they take in from the smaller single homed customer that were affected. If those customers go away, it may even be that they perceive that as a good thing. Smaller customers in the wholesale carrier business are usually higher cost to operate and higher cost to provision. It's a business decision and if you don't like it, take your business elsewhere. Again, I applaud that decision and there's nothing wrong with that. Forcing Level3 to keep up links that are not beneficial to their business is what would be wrong.
How would you like it if you were forced to keep a business relationship going that you perceived as bad for your business? You'd want out if an arrangement with a supplier was no longer benefitting you.
_VF |
|
vferrari3
join:2005-10-07
Boston, MA
| reply to smcallah
Re: That's right
said by smcallah :I still think you misunderstood. So Level(3) had RoadRunner and several large dialup ISP's fielding calls, giving credits, and losing customers during the almost 3 days that this was down. I don't see how you think that notifying their customers of a major peering change would suddenly open them up to having to notify of every tiny change. And what makes it ONLY Cogent's responsibility to make sure they could get Level(3) routes? Why wouldn't Level(3) have a resposibility to give their own customers routes to Cogent? And now obviously they have taken up that responsibility, by turning this back on because of customer complaints. That's what the notice was for. Level3 is in the position of strength. They gave Cogent the notice to make arrangements to get to Level3 after the peering goes down. Level3 has the eyeballs and that's the position of strength. I know you don't like it, but this is the way it is and has been. Cogent is the one with the history of not being able to play nice and has been through this before. Most carriers respect the model and when peers de-peer them, they make the arrangements.
BTW, what does RR have to do with this. RR should be multi-homed and should not be caught up in this. If they were, shame on them. I think you're over estimating the number and type of customers that Level3 would be losing. Again, for their business model they may even expect to lose the lower end of the base in order to continue mining the higher end of their base. They have multiple 10Gb links to some customers, the single homed DS3 customer leaving is not affecting them.
_VF |
|
DSLTech
join:2000-12-30
San Jose, CA
| reply to vferrari3
Re: Take a step back
I know for sure Level 3's customers were complaining. Can you imagine what a handfull of unhappy ISPs complaining might do to convince Level 3 to make a change?
Who was complaining? Oh, perhaps ISPs like Earthlink, Speakeasy, Megapath, etc. who all have connections via Level 3 to the internet, as well as other forms of peering. Dont you think that their complaints weigh heavily on Level 3?
Earthlink could, for example, choose to use Wiltel instead of Level 3 as a primary internet access connection, so their displeasure is very, very important to Level 3.
Fortunately our ISP is dual-homed to the internet, Level 3 being only one of the 2 options. |
|
smcallah
join:2004-08-05
Home
| reply to bmn
Re: That's right
said by bmn : Level3 is not the one responsible to make sure that it can even see Cogent's networks since they don't have any control over Cogent's BGP information. Obviously Level(3) has TOTAL CONTROL over the BGP routes it sees from Cogent on its network.
They can peer with Cogent, if they drop this peer, they lose the routes. They can also peer with a transit provider of Cogent, if Cogent has such a transit provider, and receive those routes. But Level(3) can also CHOOSE to drop those BGP routes.
Why do you think Level(3) has no control over Cogent's BGP routes on its network? |
|
smcallah
join:2004-08-05
Home
| reply to vferrari3
Re: Take a step back
I don't get it? My whole point was theoretical. Obviously it wasn't the true case. I was saying IF Level(3) cared about Cogent, they would have tried to work something out with them or extend the deadline without turning it off. They did not. So they spent 3 days getting customer complaints, which they DO care about. And so had to end up extending the deadline anyway.
But they obviously didn't care enough about their customers to let them know beforehand, so maybe my point is moot anyway.
And if the links aren't beneficial to Level(3)'s business, why did they turn them back up in the face of customer complaints?
It was customer complaints that brought this back up, whether you know that or not. What else would have brought it back up? Level(3) obviously wasn't willing to work with Cogent, for whatever reason, and dropped the connection.
They're more willing to work with them now, in the face of customer complaints, I hope anyway that it means they're more willing. It may simply mean that they are telling their customers that in 30 days you won't reach Cogent through them, and they'll end it at that with no negotiation. |
|
smcallah
join:2004-08-05
Home
| reply to vferrari3
Re: That's right
said by vferrari3 :BTW, what does RR have to do with this. RR should be multi-homed and should not be caught up in this. If they were, shame on them. I think you're over estimating the number and type of customers that Level3 would be losing. Again, for their business model they may even expect to lose the lower end of the base in order to continue mining the higher end of their base. They have multiple 10Gb links to some customers, the single homed DS3 customer leaving is not affecting them. _VF Why are you even asking what does RR have to do with this? RR is a huge cable ISP. They use Level(3), they might be Level(3)'s largest customer. You don't think RR was complaining? And RR is multihomed, they appear to use Level(3) and ATDN in most markets. But the fact remains, that they complained to Level(3) as well. So THAT'S what they have to do with this.
You're definitely underestimating the customer complaints that would have been coming into Level(3). Especially since they were the reason that Level(3) had to turn this back up. Now that they've given their customers a 30 day notice, perhaps it won't be so bad for them on November 9th.
If Level(3) didn't stand to lose revenue or customers, they wouldn't have had to make a move. How do you even doubt this? Have you seen their financials? They're going to take a beating this month alone on service credits they had to give to keep customers from wanting to leave.
I for one wouldn't want to be fielding support calls for something I couldn't support. Which is what RR, some DSL, and some dialup ISP's hanging off of Level(3) had to do. So they passed that pain on to Level(3). At that point, Level(3) is responsible to its customers. Sure Cogent is responsible to its own customers on the other end. But Level(3) had the ability to immediately get their own customers fixed, and they did this, 2 1/2 days later, without a notice before the cut off.
If you can't see that from the Level(3) customer and the customers' customers standpoint, you need to drop out of this discussion, because you clearly do not get it. |
|
vferrari3
join:2005-10-07
Boston, MA
| reply to smcallah
said by smcallah :said by bmn : Level3 is not the one responsible to make sure that it can even see Cogent's networks since they don't have any control over Cogent's BGP information. Obviously Level(3) has TOTAL CONTROL over the BGP routes it sees from Cogent on its network. They can peer with Cogent, if they drop this peer, they lose the routes. They can also peer with a transit provider of Cogent, if Cogent has such a transit provider, and receive those routes. But Level(3) can also CHOOSE to drop those BGP routes. Why do you think Level(3) has no control over Cogent's BGP routes on its network? Because the BGP tables were showing that they weren't even getting them. On the other side, Cogent fills in gaps in it's routing table from it's transit providers. It never asked them to send them Level3's routes. They did nothing. I'm impressing upon you that this is not a unique situation and what made it unique was Cogent. On Nov. 9 it will go down again and unless Cogent realizes it's position the same thing will happen.
_VF |
|
vferrari3
join:2005-10-07
Boston, MA | reply to smcallah
My point is that RR should not be complaining to Level3. They should have had full visibility to all routes through multiple providers. So I'm trying to understand from RR's point of view, what was the outage?
_VF |
|
bmn
? ? ?
Premium,ExMod 2003-06
join:2001-03-15
hiatus
| reply to smcallah
said by smcallah :Obviously Level(3) has TOTAL CONTROL over the BGP routes it sees from Cogent on its network. Incorrect. Level(3) does NOT have total contol over the BGP sessions from Cogent. If Cogent doesn't send them, Level(3) won't get them. If one of Congent's upstream providers filters them from reaching Level(3), Level(3) will not receive them. Its much more complex than that.
They can peer with Cogent, if they drop this peer, they lose the routes. Correct.
They can also peer with a transit provider of Cogent, if Cogent has such a transit provider, and receive those routes. But Level(3) can also CHOOSE to drop those BGP routes. Correct, however, there's no indication that Level(3) filtered Cogent's routes at the peering routers, but its possible.
Why do you think Level(3) has no control over Cogent's BGP routes on its network? They do have SOME control, but that control ONLY exists if Cogent is sending its routes to Level(3). Period.
--
Tor server operator... Helping the free flow of information daily. |
|
vferrari3
join:2005-10-07
Boston, MA
| reply to smcallah
said by smcallah :said by vferrari3 :If you can't see that from the Level(3) customer and the customers' customers standpoint, you need to drop out of this discussion, because you clearly do not get it. I sympathize with the customers. BUT...it's a policy decision. Do you know that some providers don't have access to every route on the internet because they filter inbound on allocation boundaries? Probably not. Because Cogent hasn't jumped up and down screaming about it. There are routinely 1-5% of the net inaccessible by certain ISPs due to filtering policies. Level3 shuts down peering, loses 1000 out of 150k routes and everyone flips out because Cogent stirs up a hornet's nest. Again, I sympathize with the customer, but I understand the bigger picture. The Internet is NOT a public utility and carriers are going to protect their resources and make decisions for their shareholders. This is not a recent phenomenon. _VF |
|
