dslreports logo
 
    All Forums Hot Topics Gallery
spc
uniqs
182

SpaethCo
Digital Plumber
MVM
join:2001-04-21
Minneapolis, MN

1 recommendation

SpaethCo to pandora

MVM

to pandora

Re: [Caps] the dreaded call - 1,817G

said by pandora:

The real choke on a rich media streaming future is caps.

This statement makes it sound like an arbitrary policy decision, and if you look at the infrastructure it's simply not the case.

So, from Comcast's skirmish with the FCC in 2008 we know that the scaling numbers were roughly 275 users per downstream channel group and 100 users per upstream channel. Over the last couple years they have been aggressive with upstream channel splits, so let's assume the numbers now are somewhere in the ballpark of 250 downstream and 50 upstream.

Most cable segments now have 4x38mbps for DOCSIS 3, and are still at a single 27mbps upstream channel.

1,000,000 bits / (8bits/byte) = 125,000 bytes
125KB/sec * 60sec/min * 60min/hr * 24hr/day * 30days/month = 324GB per month per megabit.

So segment capacity is 4*38mbps*324GB/month/mbps = 49,248GB per month downstream on the shared channel group.

For upstream it's 27*324GB/month/mbps = 8,748GB per month.

If you carve that out equally across the subscribers, that's 49,248GB/250 subs + 8,748GB/50, or 196.99GB downstream + 174.96GB upstream capacity per subscriber.

The bigger issue is the realtime rates though. Netflix has rough numbers of:
1.5mbps for SD
3.0mbps for DVD quality
5.0mbps for HD quality
8.0mbps for 1080 quality(PS3)

So on a 152mbps DOCSIS 3 channel group, you can serve:

101 streams of SD quality
50 streams of DVD quality
30 streams of HD quality
19 streams of 1080 quality

That's not going to scale when applied to downstream segments of 250 subs, especially when a large number of folks are going to want to view different content simultaneously in the house.

This, of course, is "false artificial scarcity" because the network can always be upgraded. So let's look at the costs to bump up the network so every home can get at least a single 1080 quality Netflix stream. Since I work for a Cisco partner business I have access to their global pricing, so I'm going to swag some numbers based on their uBR 7246VXR modular CMTS.

On page 27 of their most recent quarterly financial statement, Comcast identified they had 16.7 million HSI subs. At 250 subs per downstream channel group, that's (16,700,000 / 250 * 4) about 267,200 downstream DOCSIS channels across Comcast's current network.

In order to get every sub the capacity to stream a single 1080 video feed at the same time, they need to get that down to 19 subs per channel group or less. In order for that to happen, they have to keep doubling the scale of the network until the subscriber per channel group ratio comes down. So the upgrade iterations look something like this:

1 125subs/group 534,400 downstream channels
2 62subs/group 1,068,800 downstream channels
3 31subs/group 2,137,600 downstream channels
4 16subs/group 4,275,200 downstream channels

The 7246VXR modular CMTS has 4 slots for cards, the most likely card being the uBR-MC88V which has 8 downstream & 8 upstream channels. A 7246VXR fitted with 4 of those cards (for a total of 32 downstream ports) comes in around $220k list price.

4,275,200 downstream channels - 267,200 (existing today) = 4,008,000 new channels of capacity required.

4,008,000 / 32 DS ports per 7246VXR = 125,250 new CMTSes required.

125,250 * $220,000 = $27,555,000,000

Let's assume they get a rockstar 60% discount for ordering that quantity of gear, that's still $11,022,000,000

That's also only factoring in the CMTS cost, not the cabling, engineering, RF plant splits, upstream network capacity augmentation to support all those GigE interfaces coming off the 7246s, etc.

So the real question is, after your cable Internet bill goes up to support the billions of dollars in network upgrades required to make any kind of meaningful video scale happen, can Internet video itself become cheap enough that the endeavor is still a financial win for the average subscriber?
pandora
Premium Member
join:2001-06-01
Outland

pandora

Premium Member

said by SpaethCo:

said by pandora:

The real choke on a rich media streaming future is caps.

This statement makes it sound like an arbitrary policy decision, and if you look at the infrastructure it's simply not the case.

So, from Comcast's skirmish with the FCC in 2008 we know that the scaling numbers were roughly 275 users per downstream channel group and 100 users per upstream channel. Over the last couple years they have been aggressive with upstream channel splits, so let's assume the numbers now are somewhere in the ballpark of 250 downstream and 50 upstream.

Most cable segments now have 4x38mbps for DOCSIS 3, and are still at a single 27mbps upstream channel.

1,000,000 bits / (8bits/byte) = 125,000 bytes
125KB/sec * 60sec/min * 60min/hr * 24hr/day * 30days/month = 324GB per month per megabit.

So segment capacity is 4*38mbps*324GB/month/mbps = 49,248GB per month downstream on the shared channel group.

For upstream it's 27*324GB/month/mbps = 8,748GB per month.

If you carve that out equally across the subscribers, that's 49,248GB/250 subs + 8,748GB/50, or 196.99GB downstream + 174.96GB upstream capacity per subscriber.

The bigger issue is the realtime rates though. Netflix has rough numbers of:
1.5mbps for SD
3.0mbps for DVD quality
5.0mbps for HD quality
8.0mbps for 1080 quality(PS3)

So on a 152mbps DOCSIS 3 channel group, you can serve:

101 streams of SD quality
50 streams of DVD quality
30 streams of HD quality
19 streams of 1080 quality

That's not going to scale when applied to downstream segments of 250 subs, especially when a large number of folks are going to want to view different content simultaneously in the house.

This, of course, is "false artificial scarcity" because the network can always be upgraded. So let's look at the costs to bump up the network so every home can get at least a single 1080 quality Netflix stream. Since I work for a Cisco partner business I have access to their global pricing, so I'm going to swag some numbers based on their uBR 7246VXR modular CMTS.

On page 27 of their most recent quarterly financial statement, Comcast identified they had 16.7 million HSI subs. At 250 subs per downstream channel group, that's (16,700,000 / 250 * 4) about 267,200 downstream DOCSIS channels across Comcast's current network.

In order to get every sub the capacity to stream a single 1080 video feed at the same time, they need to get that down to 19 subs per channel group or less. In order for that to happen, they have to keep doubling the scale of the network until the subscriber per channel group ratio comes down. So the upgrade iterations look something like this:

1 125subs/group 534,400 downstream channels
2 62subs/group 1,068,800 downstream channels
3 31subs/group 2,137,600 downstream channels
4 16subs/group 4,275,200 downstream channels

The 7246VXR modular CMTS has 4 slots for cards, the most likely card being the uBR-MC88V which has 8 downstream & 8 upstream channels. A 7246VXR fitted with 4 of those cards (for a total of 32 downstream ports) comes in around $220k list price.

4,275,200 downstream channels - 267,200 (existing today) = 4,008,000 new channels of capacity required.

4,008,000 / 32 DS ports per 7246VXR = 125,250 new CMTSes required.

125,250 * $220,000 = $27,555,000,000

Let's assume they get a rockstar 60% discount for ordering that quantity of gear, that's still $11,022,000,000

That's also only factoring in the CMTS cost, not the cabling, engineering, RF plant splits, upstream network capacity augmentation to support all those GigE interfaces coming off the 7246s, etc.

So the real question is, after your cable Internet bill goes up to support the billions of dollars in network upgrades required to make any kind of meaningful video scale happen, can Internet video itself become cheap enough that the endeavor is still a financial win for the average subscriber?

I'd like to make 3 comments about your wonderful work above.

1) If the Comcast cap were based on network capacity it would be limited to periods of congestion (peak hours) it is not. As such it is difficult for me to accept that the cap is predicated on network issues.

2) Internet use and growth has and will always continue to grow, it is the nature of internet. An ISP must anticipate growth of use over time. Browsers, MMO games, even youtube and flash player all added immensely to user bandwidth requirements when they were released. ISP's adapted.

3) In a thread here - »Netflix Ranks Top ISPs For Streaming Video [184] comments is a link to a Netflix graph which is here - »2.bp.blogspot.com/_gC6nM ··· _usa.png it shows average Netflix bandwidth to Comcast users is about 2,600 kbps or 2.6 mbps, which is closer to your DVD estimated rate than the SD rate.

I'd assume based on your estimate that on average up to 50 Netflix users (maybe 55-58? if we use the Netflix charts) could share a Comcast node prior to saturation solely based on Netflix activity.

On the belief that 20% of users streaming Netflix will saturate a Comcast node on average, I believe Comcast already is in a position to need infrastructure improvement without any additional growth of Netflix programming. Google and Apple are entering this market, others will follow, everyone will want to have an applet on a smart device. If 20% watching streaming content will bring Comcast down, then it's already under water imo.

Fortunately Comcast is amazingly profitable.

SpaethCo
Digital Plumber
MVM
join:2001-04-21
Minneapolis, MN

2 recommendations

SpaethCo

MVM

said by pandora:

1) If the Comcast cap were based on network capacity it would be limited to periods of congestion (peak hours) it is not. As such it is difficult for me to accept that the cap is predicated on network issues.

It's not that simple -- they have a congestion management system as defined in RFC 6057 to handle peak period congestion. The problem is that congestion management systems are just prioritization schemes -- ultimately you still need to have ample capacity to move the requested amount of data which means you can only have your congestion management kick in for brief durations or you'll never be able to catch up. You can, for example, make a large purchase on a credit card that exceeds your monthly income, but unless you have available funds in subsequent months you'll never pay it off. The cap is essentially that long-term budget mechanism.

Still, the big picture is that no matter how you look at it, you're only going to move 49TB on a 4 channel DOCSIS channel group. If everyone is going to present demand you're still going to hit limits just in the infrastructure, and performance will be abysmal as queuing delay skyrockets.
said by pandora:

2) Internet use and growth has and will always continue to grow, it is the nature of internet. An ISP must anticipate growth of use over time. Browsers, MMO games, even youtube and flash player all added immensely to user bandwidth requirements when they were released. ISP's adapted.

This isn't a direct cause and effect relationship. You don't spend more money every year and change your income to match your level of spending, but rather I suspect you adjust your level of spending to your level of income. Any kind of compute / network technology essentially has planned obsolescence (often driven around equipment manufacturers setting end of life / end of support dates), so equipment refresh cycles are an integral part of the network ecosystem. This means that you are augmenting capacity by replacing hardware at the end of its service life with new hardware that is likely the same cost or cheaper as the previous generation of hardware. As the capacity become available, applications naturally crop up to leverage that new added capacity.

Leading by capacity (and backfilling demand) is the most cost effective option because you're simply deploying equipment that you had to deploy anyway. Leading by demand, by contrast, tends to be more expensive to scale because you're not getting the advantages of technology advancement. The example above highlights this perfectly, because if you need to expand today you're limited to hardware that can service 32 downstream channels. In another 3-5 years they will likely have another device with the same power and space footprint that will service 128-192 downstream channels at the same cost -- but that doesn't help you until the product exists and is orderable.
said by pandora:

3) In a thread here - »Netflix Ranks Top ISPs For Streaming Video [184] comments is a link to a Netflix graph which is here - »2.bp.blogspot.com/_gC6nM ··· _usa.png it shows average Netflix bandwidth to Comcast users is about 2,600 kbps or 2.6 mbps, which is closer to your DVD estimated rate than the SD rate.

I'd assume based on your estimate that on average up to 50 Netflix users (maybe 55-58? if we use the Netflix charts) could share a Comcast node prior to saturation solely based on Netflix activity.

I think that's a reflection of the limits of the Netflix library more than anything. Most of their titles are still DVD-ish quality, most don't have 5.1 surround sound. Also, to display movies on things like phones and tablets you're not going to need the same pixel data as if you're planning on blowing up the image to fit on your 55" HD TV.

I think the Netflix model still has limits. How many, for example, would be willing to give up their cable and satellite feeds of ESPN to watch Monday Night Football in Netflix DVD streaming quality? For me there is absolutely no way I'm giving up my satellite service until they can deliver NHL games in the same quality. I have satellite service through DirecTV and Shaw Direct (Canadian) and also get a subscription to GameCenter Live. I've watched games on both my PC and PS3 through GameCenter Live in a pinch, but there is no way that replaces the quality you get off the broadcast RSN feeds.

Higher prices and usage limits aren't necessarily barriers to a technology if it solves enough problems. Look at cell phones: people abandoned having an "unlimited" use home phone with generally high quality in favor of mobile phones which have more expensive monthly plans, minute limits, and worse overall voice quality (including audio dropouts / dropped calls / lower bandwidth codecs / etc).

Folks who are bullish on the technology mostly tend to be so because they feel it will bring costs down. I have yet to hear a cogent argument for why that would occur though, especially not the way the current alignment of content producers is structured.
said by pandora:

Fortunately Comcast is amazingly profitable.

They made 10 cents of profit for every dollar of revenue they took in last year. I'm not sure I'd classify that as "amazing."

tshirt
Premium Member
join:2004-07-11
Snohomish, WA

tshirt

Premium Member

said by SpaethCo:

I'm not sure I'd classify that as "amazing."

Maybe you should ... they were slightly under 10 % in the recent quarters BUT (very) slightly better than the general industry.

This is a high cash flow business so today's profit shows a trailing effect.
if you wish to attack industries by profit margin start with banking, then insurance, energy, housing, retailing, grocery, ETC.ETC.ETC>.
the point being it took 50+ years to build the cable plant, amazing investment, modest returns, and finally they provide higher then DSL speeds to a majority of US homes, only to be called 'fly by night, take the money and run, carpetbaggers'

MyDogHsFleas
Premium Member
join:2007-08-15
Austin, TX

MyDogHsFleas to SpaethCo

Premium Member

to SpaethCo
Nice analysis, putting the lie to those who want to equate marginal costs with overall costs.

To take it to a higher level, there's a reason why a huge majority of consumer-viewed video is broadcast from a very small number of sources to a very large number of sinks -- whatever the transmission media and protocols. The reason is that it's efficient and cost-effective and massively scalable -- by orders of magnitude over single-stream-per-user consumer-viewed video.

To equate broadcast with single-stream-per-user on-demand video, and say that the latter should be as cheap, and if it's not, it's greedy corporations conspiring to kill it, is just mindless (to put it gently).
BiggA
Premium Member
join:2005-11-23
Central CT
·Frontier FiberOp..
Asus RT-AC68

BiggA to SpaethCo

Premium Member

to SpaethCo
I think you're really overestimating the number of people streaming at a given time. No network is going to handle everyone using it at the same time, Comcast's own VOD system included. People will still have a lot of DVR'ed, disc'ed, and otherwise imported content.
sphinx11
join:2011-01-06
Saratoga, CA

sphinx11

Member

said by BiggA:

I think you're really overestimating the number of people streaming at a given time. No network is going to handle everyone using it at the same time, Comcast's own VOD system included. People will still have a lot of DVR'ed, disc'ed, and otherwise imported content.

I concur. If all users watched Netflix for only two hours per day they would each stay within the monthly cap. But if every user on a segment chose the same two hours in the day for watching a movie on Netflix, then it is easy to see even from this analysis that the monthly cap is not solving a congestion problem that will exist for those two hours every day.

Many networks are oversubscribed by design because user behavior is not synchronized across the entire user population.
jaymz668
join:2002-03-21
Bloomington, IN

jaymz668

Member

said by sphinx11:

I concur. If all users watched Netflix for only two hours per day they would each stay within the monthly cap.

How are you defining user? Person in the house? Household total? I can easily see more than one streaming program at once.

Tordek7
Make the outages go away
Premium Member
join:2009-09-07
Great White

Tordek7

Premium Member

netflix only allows 2 concurrent streams
sphinx11
join:2011-01-06
Saratoga, CA

sphinx11 to jaymz668

Member

to jaymz668
said by jaymz668:

said by sphinx11:

I concur. If all users watched Netflix for only two hours per day they would each stay within the monthly cap.

How are you defining user? Person in the house? Household total? I can easily see more than one streaming program at once.

For the sake of simplicity, lets assume user = cable_modem.
BiggA
Premium Member
join:2005-11-23
Central CT

BiggA

Premium Member

It's pretty likely that a secondary stream would be SD, and relatively few users would have a second stream at all, so count them out.
jaymz668
join:2002-03-21
Bloomington, IN

jaymz668

Member

Why is it likely the second stream would be SD?
Amazon is now also making noises of offering this stuff as well...

EG
The wings of love
Premium Member
join:2006-11-18
Union, NJ

EG

Premium Member

said by jaymz668:

Why is it likely the second stream would be SD?

Less of a capacity/congestion burden, so potentially better overall performance ?
jaymz668
join:2002-03-21
Bloomington, IN

jaymz668

Member

said by EG:

said by jaymz668:

Why is it likely the second stream would be SD?

Less of a capacity/congestion burden, so potentially better overall performance ?

It's not like you can tell Netflix to only send the SD version of a program. So it is entirely dependent on what you are watching.

EG
The wings of love
Premium Member
join:2006-11-18
Union, NJ

EG

Premium Member

O/k so you already know the answer then !
BiggA
Premium Member
join:2005-11-23
Central CT

BiggA to jaymz668

Premium Member

to jaymz668
Watching on a small device that wouldn't support HD.
jaymz668
join:2002-03-21
Bloomington, IN

jaymz668

Member

Not sure I follow. Why would it be to a device that doesn't support HD. One TV and one PC stream and you are streaming two HD streams.
BiggA
Premium Member
join:2005-11-23
Central CT

BiggA

Premium Member

It might be, but the number of those situations would be low, hence overall use of two HD streams over many customers on a node would be very low.

SpaethCo
Digital Plumber
MVM
join:2001-04-21
Minneapolis, MN

SpaethCo

MVM

Netflix is only part of the equation -- it's just being used as a point of discussion because it's an IP video platform that exists today in reasonably wide deployment.

The original statement was that ISP caps are the sole barrier to more widespread entry into the IP video market, including providers who will replace your cable TV lineup with video sourced over the Internet.

My point was that if people really believe Internet video is going to take over, you have to plan for this kind of concurrency, and the costs are going to be massive.
BiggA
Premium Member
join:2005-11-23
Central CT

BiggA

Premium Member

Why would the costs be so massive? IP video isn't just going to appear overnight, it's slowly changing, and while it changes bandwidth gets cheaper and cheaper. We already consume a lot of internet bandwidth in aggregate.

SpaethCo
Digital Plumber
MVM
join:2001-04-21
Minneapolis, MN

SpaethCo

MVM

As it stands right now, just a 12% adoption of IP video customers pushing a single HD feed (30 out of 250 subs in a channel group) drives the network to saturation. Video is a huge element that everyone is grappling with, from equipment manufacturers down to service providers.

The network just went through a refresh for DOCSIS 3, and the scale to support IP video as a replacement (not augmentation) for current broadcast video is huge.

Netflix is a perfect example of how big - with 10 million subs streaming some quantity of video they are pushing 20% of all Internet bandwidth in the US.

Tordek7
Make the outages go away
Premium Member
join:2009-09-07
Great White

Tordek7

Premium Member

said by SpaethCo See Profile

The network just went through a refresh for DOCSIS 3, and the scale to support IP video as a replacement (not augmentation) for current broadcast video is huge.

Very agreed. Unless your HE supports double-down, then you get to this question.. Do we do rolling outages while we upgrade (pissing customers off) or do we build out to support having both equipment running side by side until the cut over?

Option 2, if the HE is full up, or real estate is slim is very frickin expensive (Mine is in the process now of being rebuilt.. beautiful now though)

You need double the fiber, double the CMTS', double the DNCS/DAC etc.

SpaethCo
Digital Plumber
MVM
join:2001-04-21
Minneapolis, MN

SpaethCo to BiggA

MVM

to BiggA
said by BiggA:

Why would the costs be so massive? IP video isn't just going to appear overnight, it's slowly changing, and while it changes bandwidth gets cheaper and cheaper. We already consume a lot of internet bandwidth in aggregate.

Just to re-address this point:
Netflix instant accounts for 20 percent of all non-mobile internet use during prime time in the United States, according to a new study.

Streaming media — real-time entertainment — accounts for 43% of peak period traffic in the U.S., according to Sandvine, which helps ISPs manage their networks and thus has access to buckets of information about usage patterns.

But Netflix alone accounts for nearly half of that between 8 and 10 p.m., and that usage comes from only 1.8 percent of the service’s subscribers.
Source: »www.wired.com/epicenter/ ··· ith-use/

Netflix has 16 million total subscribers, so that means that about 288,000 folks streaming from Netflix are driving 20% of Internet bandwidth consumption on a nightly basis in the US.

Even at slow adoption rates, the impact here is absolutely massive.

tshirt
Premium Member
join:2004-07-11
Snohomish, WA

tshirt to BiggA

Premium Member

to BiggA
said by BiggA:

......it's slowly changing, and while it changes bandwidth gets cheaper and cheaper.

TRANSIT bandwidth is getting cheaper, last mile bandwidth is about the same as always (wage and equipment pricing is actually up, but new more efficent equipment, keeps the last mile cost similar per unit)
said by BiggA:

We already consume a lot of internet bandwidth in aggregate.

YES we do! and last years "a lot" is this years "a lot PLUS XX%" and next years "a lot PLUS XX%, PLUSXXX% more"
It's not just the amount used today, but the idea that it can continue to grow at the current exponential rate, without any addtional cost to the end user is ignorant of the facts.