dslreports logo
site
 
    All Forums Hot Topics Gallery
spc

spacer




how-to block ads


Search Topic:
uniqs
1817
share rss forum feed

givemesam

join:2011-05-03
Seattle, WA

1 IP, Lots of devices (200+) - lets brainstorm

Heyo,

Got a question for you. What are some of the issues users may face when on a lan with a lot of devices?

I will break this down in detail:

250mbps fiber line with 40-50% utilization w/ 1 Static IP - high end router doing Nat w/ lots of processor free (@20% load) - 50+ wireless APs, each with a different subnet, also Natting with isolation ON (keeps broadcasts low) Probably have 200 devices on at any time, all wireless.

(APs have maybe 10-15 wireless devices on at a time, many idol, doing NAT, Main Router only sees the 50 or so APs as its clients)

The question is what issues could arise? Can we share some of the categories and what it would look like to the end users? Popular services on the lan are web traffic, browsing, https traffic/banking, Netflix, Hulu, Xbox, PS4, AppleTV. Lets say there are 20 of each of these happening at the same time on that IP.

Lets stay away from, oh, your double natting, or other empty comments about the elegance of this network layout please and support me with identifying the reasons behind issues so we/I can come up with some ways to program the router/s to better support this large lan/number of devices.

I am trying to understand if/what the benefit would be to break the network down to have 5-6 static ips, and shrink the number of devices using the same ip, and if this shrinking will make a difference or not. (or if the core issues are still there, but less prevalent to show themselves, ie a band aid, but not a fix)

Thank you for sharing your collective wisdom with me!



Anav
Sarcastic Llama? Naw, Just Acerbic
Premium
join:2001-07-16
Dartmouth, NS
kudos:5

SOunds like a homework question, suggest do your own research and if you have pointed specific questions that may be more useful.


voxframe

join:2010-08-02

2 recommendations

reply to givemesam

said by givemesam:

Lets stay away from, oh, your double natting, or other empty comments about the elegance of this network layout please and support me with identifying the reasons behind issues so we/I can come up with some ways to program the router/s to better support this large lan/number of devices.

What you just said/asked completely contradicts itself.

From the sounds of it, you've been given the keys to a high powered Ferrari, a racetrack, and you're asking how fast can you go in the rain, but you don't want to hear anything about tires or fuel.

You've got a serious machine in your hands, running at full speed, with many little issues that could completely crash it. EVERYTHING matters. This isn't a simple affair, and it involves serious planning from start to finish, including thinking about things like NAT, "elegance".

This is the scope of a project that usually takes a dedicated engineer to build. I'm assuming this is a school, or some crazy large hotspot.

Your high end router will self destruct if you load this network up on it with the QoS structure that you will NEED to keep it from locking up solid.

You will need a complete QoS plan no matter what. Your biggest challenges will also be frequency overlap and re-use.

You will also need to have either your core router see ALL the clients, or have a high end router at each AP to process QoS properly. You can't get away with having the core router process QoS if it only sees the APs.

BTW- WiFi protocols/APs can only handle maybe 30 clients at the absolute MAX per AP before the protocol itself starts to break down and become horribly inefficient. Just something to think about.

You will need serious AP hardware that is made to do exactly this job. Not a shitpile of Apple or Linksys routers. But something made for large deployments. Managing this grid will be an absolute nightmare.

What you're asking and what you're saying, tells me right away you don't understand the scope of this project, and I'm sorry, but this is a lot more complicated and tougher to manage than you may think (I've been doing this for 10 years now, trust me).

If money is a problem, look at Ubiquiti UniFi. Router look at Mikrotik.
If money is not a problem, look at a large deployment Cisco solution.

LLigetfa

join:2006-05-15
Fort Frances, ON
kudos:1
reply to Anav

said by Anav:

SOunds like a homework question...

Actually it sounds like a fixed WISP question. How much of it is real versus imagined, I'm not sure.

Now I am of the opinion that each and every sub should have their own public IP so whether it is 1 IP or 5-6 IPs is neither here nor there.

If it is a hotspot and not fixed, then that is a different matter.
--
Strange as it seems, no amount of learning can cure stupidity, and formal education positively fortifies it. -- Stephen Vizinczey

AsherN
Premium
join:2010-08-23
Thornhill, ON
reply to givemesam

200 devices NATing through an enterprise firewall is not even worth worrying about.


givemesam

join:2011-05-03
Seattle, WA

Ok, well in some cases it could be as much at 6-700 until we city the network down in size a bit. Let's go extreme and say that half of them are watching Netflix on an Xbox at the sand size utilizing Xbox live and Netflix at the same time.

What issues could arise? Is tcpip this robust?


voxframe

join:2010-08-02
reply to givemesam

When you have full control over the policies of those devices, MAYBE.

The OP is talking about a complete mashup of everything, with streaming/IPTV to boot. You can't tell me for a second that it's not something to worry about. Perhaps the NAT itself, no. But the actual QoS needed to transport that crap, especially once it hits a very limited, and inefficient, medium (WiFi) you're gonna need it.

Again, the NAT isn't a huge issue. It's the management of the 99.9% of the rest of the mess. Plain old NAT doesn't kill a router CPU, but the QoS needed to keep everyone happy, will. This still doesn't touch the actual wireless setup and the complexities involved with that. Again, corporate controlled environments, this is considerably easier. But given the list the OP gave, it's not. (And it if is, I really question an enterprise allowing in that much junk)


givemesam

join:2011-05-03
Seattle, WA

Ok, so we have no QoS at the head end, just Nat, then the 50 or so APs are doing "per user throttling" to pass around the data. We see our throughput at about 100mbps at peak times.

So the QoS/throttling load is distributed over 50 devices that seem to handle it.

The network would be similar to a dorm network setup. All resident access is over the Wi-Fi only.


voxframe

join:2010-08-02
reply to givemesam

What are these magic AP devices? I've NEVER seen one that can reliably do its own user QoS (To any acceptable level of control) asides from serious enterprise gear.

The bandwidth you mention, just scares me that much more.



Semaphore
Premium
join:2003-11-18
101010
kudos:1
reply to givemesam

Google will start dropping your search requests as DOS or spam/scraping when multiple hits come from the same IP. I don't know what their threshold is but I do know it's happened to others.
"DNS tunelling" Systems like UnblockUS will not operate properly if you have multiple users of that service sharing a single IP.
If multiple users try to use a streaming system that limits number of connections by Source IP, like NetFlix is rumored to start doing soon, you're going to have a very bad time.
Tracing back digital media copyright or abuse complaints in this situation would be impossible.


givemesam

join:2011-05-03
Seattle, WA
reply to voxframe

Ok
So I own an ISP. We have an 1100ah at the head end. And 50+ high end APs serving data, each running 'per user throttling' to the devices.

They are very beefy and can handle the throttling fine. Think of it as distributed computing.

We have routers set up with as low Tx power as we can, and of course are utilizing the channels as best we can too. We are aware of the complexity of the network to make this work.

BUT my question is in regards to the capacity of the L4-7 protocols being able to handle say 100 xboxs streaming from Netflix To one IP. This is the question. Would breaking it down to 4 IPS make this situation perform better? Does it matter? Etc.

Let's try and bring wisdom into the question itself and not discuss the competence of the rest of the network please. Although I appreciate you checking.


givemesam

join:2011-05-03
Seattle, WA
reply to LLigetfa

Yes. I own an ISP/giant hotspot. All of it is real. No imagination.

Not sure why the mods moved it here from the wisp forum

I agree that all sub's should have their own IP, but sadly this site is 3 years old and the gear used prevents us from doing that at this site.

Question is really can L4-L7 protocols handle hundreds of devices on the same IP doing something like Xbox live or Netflix.


givemesam

join:2011-05-03
Seattle, WA
reply to voxframe

In many places we use openmesh gear. Low cost. Quite capable of keeping users in check. We are not giving out crazy fast speeds either and have lots of capacity at the head end. Nds and chili depending on the site seem to be handled well on the routers.


givemesam

join:2011-05-03
Seattle, WA
reply to Semaphore

Yes. U are right. The DMCA stuff is a pain. And almost impossible. We are hopefully going to make our own reporting system to track something useful enough to track Mac addresses. I'll leave that to a capable engineer to sort out.

The Netflix and Google limiting connections per IP is interesting. Any links you can share?
Does it give message or just go to a dead page? Stop working altogether?


raytaylor

join:2009-07-28
kudos:1
reply to givemesam

If your 250mbit fiber upstream isp places a connection limit on it in the routing table on their end, then you will have issues.

Eg. I limit my natted customers to 200 connections each because my upstream limits my public ip addresses to 2000 connections each.
I use 5 public ip addresses for my 200 natted customers and it seems to be okay.

Opening bittorrent will easily chew up 1000 connections.

Once the upstream isp sees more than 2000 connections coming from one ip address, any new ones are dropped, until old ones time out.


givemesam

join:2011-05-03
Seattle, WA

very interesting. I will send a line to my isp and ask them this exact thing. we do our best to stop BT but there is only so much we can do.

What method do you use to share 5 ips among 200 customers?

I was thinking to assign an IP to each port on our head router, and segment each switch coming off that router onto its own static ip. Somewhat heterogeneous, but should do the job.

appreciate your thoughts.

thoughts?


raytaylor

join:2009-07-28
kudos:1

I have customers in subnet groups, so throughout our network there are subnets with public ip addresses in blocks of 8, and then a 172.16.x.x subnet for private ip addresses in blocks of 256

So no matter where you are on our network, i can put your mac address into a public ip group or a private ip group. Each Radio high site has a small subnet for public and a big private subnet for natting

At the router behind our border router, it will NAT the private ip addresses into one of the 16 public ip addresses I have on its WAN port (only use 5)

Basically if you come from private block 172.16.1.x,2,3,4,5,6 you get publicip1
If you come from private block 172.16.7.x,8,9,10,11 you get publicip2
And so fourth.

If you want to block bittorrent, set the maximum connection count to 100 per host.
Thats enough to surf quite happily (10-20 connections at most) with skype and a few other apps running in the background.
As soon as they open bittorrent, their surfing ability will stop - they may still be able to slowly download a torrent, but they wont be able to do anything else.



Semaphore
Premium
join:2003-11-18
101010
kudos:1
reply to givemesam

Google: »Google blocking automated requests... For you NATers out there.

Netflix has been in the news since Oct. re: limiting regular accounts to 1 or 2 connections per IP and you'd need to upgrade to a Premium account for more than that... in your case that would be a problem.


givemesam

join:2011-05-03
Seattle, WA

Are you sure this is per IP? Or per account allowing so many Mac addresses to use the same account.


gunther_01
Premium
join:2004-03-29
Saybrook, IL
reply to givemesam

We have used NAT for hundreds of customers for years. If Netflix did that, we WOULD know, VERY fast. I think they are smarter than that to block per IP.
--
»www.wirelessdatanet.net


wirelessdog

join:2008-07-15
Queen Anne, MD
kudos:1
reply to givemesam

Ditch the 1100AH


givemesam

join:2011-05-03
Seattle, WA

Go on. What would you recommend?


wirelessdog

join:2008-07-15
Queen Anne, MD
kudos:1
reply to givemesam

I stopped using Mikrotik for my edge router and switched to Cisco. I had a NAT limitation issue where I hit a brick wall on TCP traffic at around 30 megs. Nobody could answer why, I switched from a 1100AH to a custom Baltic box (dual core 2 gigs ram, etc) that made no difference. I switched to Cisco and now I see the 100+ megs I should be all the time.

Mikrotik support was useless. I contacted Dennis Burgess and told them I would be happy to pay them if they knew they could fix it without issue and they told me there are no guarantees with them which I took to mean they didn't have a clue either.



tubbynet
reminds me of the danse russe
Premium,MVM
join:2008-01-16
Chandler, AZ
kudos:1

1 edit
reply to givemesam

large numbers of nat translations are a non-issue with appropriate sized gear. any of the larger cisco asa devices or the asr1k can handle nat load with ease. if you need larger nat tables than that, an asr9k with an ism and the cgn license will bear the load and not sweat.
however, this is isnt your x86 mtk boxen -- its serious hardware to solve a serious problem.

q.
--
"...if I in my north room dance naked, grotesquely before my mirror waving my shirt round my head and singing softly to myself..."


givemesam

join:2011-05-03
Seattle, WA

Can you re-write this for me?

"however, this is isnt your x86 mtk boxen -- its serious hardware to solve a serious problem."


gunther_01
Premium
join:2004-03-29
Saybrook, IL
reply to givemesam

That is an interesting problem Wirelessdog. But, trying to use NAT and other QOS on a MT box is very hard on it. I would have to take a guess you were doing more than NAT on it. The MT "core" is just Linux. We use an Imagestream at our head end and have never seen that issue as we push over 100meg using NAT and other QOS, bandwidth shaping, and the like on the same box.

I'm just not sure I would blame MT for your issue is all. Albeit, I can't prove it at this point. But I'm sure you aren't the only one doing NAT and has less than a 30meg circuit using MT. Not arguing, just not convinced that was the problem is all.

For the record we have close to 500 siting behind one IP. Cisco actually has a write up about it, and they even said a couple class C's is OK. You have to remember there are some 65,000 ports a NAT machine can utilize to pass traffic under one IP. As long as it has the horsepower, and you don't run out of ports, it WILL work just fine.

We are moving away from it though. Just saying it certainly does work.
--
»www.wirelessdatanet.net


lutful
... of ideas
Premium
join:2005-06-16
Ottawa, ON
kudos:1

said by gunther_01:

You have to remember there are some 65,000 ports a NAT machine can utilize to pass traffic under one IP. As long as it has the horsepower, and you don't run out of ports, it WILL work just fine.

max NAT clients = approx 60,000 available ports / per client maximum set in a particular implementation

So we are talking about 1000+ NAT clients per IP with a good implementation on robust hardware.


tubbynet
reminds me of the danse russe
Premium,MVM
join:2008-01-16
Chandler, AZ
kudos:1
reply to givemesam

said by givemesam:

Can you re-write this for me?

not sure whats difficult about this.
microtik is linux on x86 doing routing. everything is passed through the cpu for forwarding decisions. this is perfectly acceptable for lower end routers that aren't expected to pass more than a few hundred megs of traffic with services (nat, application profiling, qos, etc). additionally, you have a "shared fate" scenario -- as your control-plane processor is also the data-plane processor -- pass too many packets or put too much load on the box and you lock yourself out.

for the "higher-end" platforms offered by cisco -- this isn't a problem. the data-plane is isolated from the control-plane (in terms of access) and there is dedicated hardware asics that are designed to pass packets without "software" intervention -- the asics are made to forward the packets.

q.
--
"...if I in my north room dance naked, grotesquely before my mirror waving my shirt round my head and singing softly to myself..."

voxframe

join:2010-08-02
reply to givemesam

I run RB-1100AHx2 on a very large head end, using NAT, with ZERO issues.

Asides from the Google/Netflix/Apple bullshit. Which, by the way, you will get zero support from them, or anyone else. The best thing you can do is set up NAT groups of 50 users behind a single address etc. It tends to cut it down.

We used to use an RB-1100 and it would choke bad. The AHx2 solved that.
CCR would also be a great option here.

I would be VERY worried about your APs. I'm not a fan of anything with the name MESH in it lol. Never ever seen one that holds true to its name.


LLigetfa

join:2006-05-15
Fort Frances, ON
kudos:1

said by voxframe:

The best thing you can do is set up NAT groups of 50 users behind a single address etc...

I suggest giving each AP its own NAT.
--
Strange as it seems, no amount of learning can cure stupidity, and formal education positively fortifies it. -- Stephen Vizinczey