jlivingood
Premium,VIP
join:2007-10-28
Philadelphia, PA
| Response Now Filed
A short time ago, we filed our response electronically with the FCC. Those documents will be made available to the public shortly (I'm guessing within 2 hours) on our Network Management website at »www.comcast.net/networkmanagement/. News outlets are starting to report on the story now, including ArsTechnica here at »arstechnica.com/news.ars/post/20···ing.html.
--
JL
Comcast |
|
funchords
Hello
Premium,MVM
join:2001-03-11
Washington, DC | »gigaom.com/2008/09/19/comcast-in···th-hogs/
GigaOm (link above) has the story out, with links to the documents. |
|
funchords
Hello
Premium,MVM
join:2001-03-11
Washington, DC
 ·Verizon Online DSL
 ·Skype
| So I just did the quick readthrough of »gigaom.files.wordpress.com/2008/···ices.pdf
It looks like there is no floor, am I right? So the "bandwidth starvation" scenario is theoretically possible (but not experienced under testing) as talked about on page 13 (along with the couple of paragraphs leading up to this).
So was there any basis to the comparison to "A Really Good DSL Experience," which I took to mean that there would be a floor? Or did I miss a detail on my quick read through?
--
Robb Topolski -= funchords.com =- Hillsboro, Oregon
More features, more fun, Join BroadbandReports.com, it's free...
|
|
jlivingood
Premium,VIP
join:2007-10-28
Philadelphia, PA
| said by funchords  :So I just did the quick readthrough of » gigaom.files.wordpress.com/2008/···ices.pdfIt looks like there is no floor, am I right? So the "bandwidth starvation" scenario is theoretically possible (but not experienced under testing) as talked about on page 13 (along with the couple of paragraphs leading up to this). I guess we have two risk scenarios depicted that we've tried to address. I figured you may ask this so I already wrote up something that I'll paste below. I just didn't think you'd be so fast. 
1 - What is oscillation, and how is it prevented or minimized in the new method?
Oscillation refers to the situation where a cable modem is assigned to a lower Quality of Service (QoS) priority (Best Effort, or BE) after one measurement interval, then promoted back to the higher QoS priority (Priority Best Effort, or PBE) in the next interval, just to be assigned to the lower priority again in the following interval. We refer to this up and down assignment as oscillation.
Part of the new congestion management system is the use of thresholds to determine when a cable-modem is in a high consumption state and when it is not. At the start of the trial the congestion management algorithm used a single threshold to determine if a cable-modem was in a high consumption state or not. If a cable-modem’s utilization was just over the threshold in one time interval and it was then put in the lower priority, it is very possible that because of the lower priority its use will now be just below the threshold and so it will be promoted back to the higher priority in the next interval.
To resolve this issue, we instead implemented two thresholds. The higher threshold determines when a cable-modem is put into the lower priority class (BE). The lower threshold determines when the cable-modem can be taken out of the lower priority class and restored to normal priority (PBE). By having the two separate thresholds, the chance of oscillation occurring is greatly reduced.
This kind of oscillation is often found in control systems that use a single threshold for both on and off states. A standard approach to reduce rapid cycling through the states is to add a second threshold which gives the system a range of control. The range between the two thresholds is called hysteresis. Most thermostats that control residential heating systems operate this way, for example, and many other control systems have oscillation-prevention features as the Comcast system does.
2 - What is user starvation and how is it prevented or minimized in the new method?
User starvation was an extreme case identified with theoretical or simulation models of the new congestion management system. That model identified a possible situation that could occur when the connection is fully (100%) utilized, which is an extreme congestion state. If a group of high consumption state cable-modems are operating at low priority (BE), and a new heavy cable-modem appears at high priority (PBE), it is theoretically possible that the high priority cable-modem with PBE QoS can take enough bandwidth away from the low priority BE QoS cable-modems that they appear to no longer be demanding heavy use. At the next measurement interval the low priority cable-modems are promoted to high priority (BE to PBE), and the just recently “new” high priority cable-modem is put at low priority (PBE to BE). The heavy cable-modems at high priority (PBE) now can consume all of the available bandwidth and the low priority cable-modem (BE) is starved, or prevented from getting any bandwidth, until the end of the next measurement period (approximately 15 minutes).
This problem is closely related to the oscillation problem described above. Since the large group of heavy use cable-modems operating at low priority will need to meet a lower threshold before being allowed to be promoted to high priority, this starvation scenario is much less likely to occur. Furthermore, the described scenario only occurs when the network is consistently used at or near 100% utilization, which based upon trials conducted by Comcast, is extremely, extremely rare.
During the trials we worked hard to find locations where utilization was very high, and still did not find a situation that would cause this starvation. We also tested this possible use case in our lab and were unable to produce starvation, possibly due to the exact manner in which CMTS packet schedulers actually work in practice, which could not be adequately simulated in a model. The likelihood of this starvation problem occurring in a real network is very low.
All of that being said, we expect to continue to fine-tune the system. We also had help in performing this analysis -- from an independent consulting firm. They'll be continuing on as we implement and into 2009, to periodically audit system behavior and make independent recommendations.
So was there any basis to the comparison to "A Really Good DSL Experience," which I took to mean that there would be a floor? Or did I miss a detail on my quick read through?
I have a feeling that's one of those things that sometimes gets lost in translation. You should consider the information released today the definitive documents on that question.
--
JL
Comcast |
|
TKJunkMail
Enjoy the sun
Premium
join:2002-03-03
Avalon, NJ
 ·Sprint Mobile Broa..
 ·Comcast
3 edits | reply to jlivingood
News has also reached mainstream online press thru Yahoo posting:
»news.yahoo.com/s/ap/20080920/ap_···internet
Comcast's old way of managing congestion(19 pages):
»downloads.comcast.net/docs/Attac···ices.pdf
Comcast's new way of managing congestion(20 pages):
»downloads.comcast.net/docs/Attac···ices.pdf
Schedule of cutting over to new congestion management system(4 pages):
»downloads.comcast.net/docs/Attac···Plan.pdf
--
My BLOG .. .. Internet News .. .. My Web Page
Ask yourself one question: 'Do I feel lucky?' Well, do ya punk? |
|
jlivingood
Premium,VIP
join:2007-10-28
Philadelphia, PA | Yup, and looks like our Network Management site is now updated at »www.comcast.net/networkmanagement/
--
JL
Comcast |
|
TKJunkMail
Enjoy the sun
Premium
join:2002-03-03
Avalon, NJ
·Sprint Mobile Broa..
·Comcast
| reply to TKJunkMail
I found this map of what a typical Comcast HSI path to the backbone internet looks like interesting:
It may help people understand what a hybrid coax/fiber cable network looks like. |
|
jlivingood
Premium,VIP
join:2007-10-28
Philadelphia, PA
| said by TKJunkMail :I found this map of what a typical Comcast HSI path to the backbone internet looks like interesting: Indeed. We wanted to include this graphic so people were really clear on exactly where in the network the new servers will be installed. I know there was some debate about where the old equipment was installed, which we wanted to avoid. (And in the FCC document regarding the current method, we have tried to be as clear as possible about where those devices are installed.)
--
JL
Comcast |
|
k1ll3rdr4g0n
join:2005-03-19
Homer Glen, IL
| reply to TKJunkMail
said by TKJunkMail :I found this map of what a typical Comcast HSI path to the backbone internet looks like interesting: [att=1] It may help people understand what a hybrid coax/fiber cable network looks like. Now, I'm not N+ certified but judging from that picture there seems to be "statistics collection servers"...ummm I definitely don't already like the sound of that.
And comcast is DEFIANTLY over thinking this. All they have to do is make the "Local Market Router" throttle based on usage per connection. These connections each have a unique MAC addresses on the cable modem. Once the user starts to request a lot of bandwidth in a short period of time, throttle him down to what hes paying for for a time. After an hour of not requesting massive amounts of bandwidth then restore power boost. And once they hit 250GB then throttle them down to the package of what they are paying for the rest of the billing cycle. And if customers wanted powerboost after 250GB then they have to call up (or go on the site) and explicitly authorize another 250GB for a fee (what $50 sounds good). Bandwidth problem solved and we can all go home, and the CEO's are lining their pockets again. It's a win-win.
I don't understand the need to purchase ALL this new equipment, unless the network engineers don't know anything about networking (which could be the case because I was told by a tech at one time that in order to send email to comcast customers from a source that is not on the comcast network, I need to upgrade to comcast business in order to send email to comcast customers from outside the network (sorry for the runon but I just wanted to emphasis that he told me that I need to pay comcast more in order to send to comcast email addresses).
*sigh* We will never learn, human history always repeats itself. AOL started as a pay-per-use service...but now look at where it is (granted dial-up is so last year, but do you actually see pay-per-use dial-up anymore? It seems like cable broadband is about to suffer the same fate, go FiOS!) |
|
jlivingood
Premium,VIP
join:2007-10-28
Philadelphia, PA
| said by k1ll3rdr4g0n :Now, I'm not N+ certified but judging from that picture there seems to be "statistics collection servers"...ummm I definitely don't already like the sound of that. I think we were trying to use a generic term that was easy to understand. To be technically precise, those are IP Detail Record (IPDR) collection servers. Their sole job is to collect usage data, on a per device basis, such as bytes per upstream and bytes per downstream by device. It is very straightforward and simple. You can search the DOCSIS specs at CableLabs for more info about what IPDR is, if you want to learn more.
And comcast is DEFIANTLY over thinking this. All they have to do is make the "Local Market Router" throttle based on usage per connection. These connections each have a unique MAC addresses on the cable modem.
Great question, and that is certainly one design path that can be considered. However, we felt it was better for overall performance to functionally separate / externalize this analysis and action function from the CMTS. That way the CMTS can just focus on handling packets as fast as possible, and does not need to store lots of device usage data - you let this external function handle that.
Jason
--
JL
Comcast |
|
funchords
Hello
Premium,MVM
join:2001-03-11
Washington, DC
·Verizon Online DSL
·Skype
| reply to jlivingood
I think that the fact that there is no floor is one of the biggest risks of this plan. And while the scheduler may be effectively be creating a floor (which sure sounds possible), designing in a floor is a better strategy -- especially if some ISP comes along and wants to copy the method and whose particular implementation doesn't recreate the scheduler-caused effect.
Thoughts?
Part of the reason your testing has been been successful (or, at least, your testing has not failed) is that the thresholds are high and the sampling periods are long and that the areas tested were prepped for upgrade.
But what happens when demand increases, or upgrade money is delayed in the budget and you need to lower the thresholds to get by? "Scavenger service" was never intended to carry VOIP traffic.
Why isn't that risk being considered "dangerous?"
Thoughts?
--
Robb Topolski -= funchords.com =- Hillsboro, Oregon
More features, more fun, Join BroadbandReports.com, it's free...
|
|
jlivingood
Premium,VIP
join:2007-10-28
Philadelphia, PA
| said by funchords :I think that the fact that there is no floor is one of the biggest risks of this plan. And while the scheduler may be effectively be creating a floor (which sure sounds possible), designing in a floor is a better strategy -- especially if some ISP comes along and wants to copy the method and whose particular implementation doesn't recreate the scheduler-caused effect. Thoughts? Fair to expect that such a capability would be discussed by engineers on the project, but as you noted (and we discussed in the 'oscillation and starvation risk' discussion) it just didn't prove necessary. I am not sure how a network with a different kind of packet processor would implement something like this. Like everything in the system, though, we'll continue to assess all aspects of it (especially as we deploy).
But what happens when demand increases, or upgrade money is delayed in the budget and you need to lower the thresholds to get by? "Scavenger service" was never intended to carry VOIP traffic.
You are spot-on that scavenger, as implemented on Internet2 (and as being discussed at the IETF now), was never intended for real-time traffic. It is more for bulk transfers that are not delay or jitter sensitive, and where the difference of a few seconds (or even minutes) to complete a transfer of data is not problematic.
As for lowering the thresholds to stretch out upgrades, that's expressly stated as something that this congestion management is not for. Capacity upgrades are going to continue as always, which by the way are according to some strict trigger points (I think our CTO even disclosed them at some point a few months ago -- may want to Google that).
Jason
--
JL
Comcast |
|
funchords
Hello
Premium,MVM
join:2001-03-11
Washington, DC
·Verizon Online DSL
·Skype
| You guys. Sheeesh. Build in a floor. It's the sound thing to do. I wouldn't set up my family members with VOIP service on Comcast under this situation. No way.
Throwing the users traffic into "scavenger class" without their notification or involvement borders on carelessness if not recklessness.
And even with a floor, you need more input and you need more test results (no complaints is not data). I've been reading these comments for 3 days and I'm not seeing appropriate scrutiny.
I like the detail, and I like the heads up, and I even like the scavenger class idea coupled with a way to direct some of my traffic into it (as you suggested in another message about user tools that might later be made available). But implementing this plan as written is premature.
--
Robb Topolski -= funchords.com =- Hillsboro, Oregon
More features, more fun, Join BroadbandReports.com, it's free...
|
|
