dslreports logo
 
    All Forums Hot Topics Gallery
spc
Search similar:


uniqs
11866
Turtlesoft
Premium Member
join:2014-08-25
Keller, TX

Turtlesoft

Premium Member

[Networking] Fios Compatible Router than can handle 200+ VMs without crashing

Hello, long time lurker, first time poster.

We have a small home business that entails us running a lot of virtual machines in VMware. Since it started as more of a hobby, until recently we had done this on our residential fios using the standard Actiontec MI424WR Rev 1. Recently we expanded to a new physical machine that increased our VM count to ~ 120 and the net started crashing (losing all connectivity) every 24 hrs. After resetting the router, everything was fine for another 24 hrs or so.

I assumed that it was too much traffic on the network so we signed up for fios small business in addition to our residential. In the meantime, I talked to verizon tech and they said it was because the router couldn't handle all the IP's needing to be handed out to the VM's. To my dismay, our small business service came with the same router so the only way I'm able to run at this point is by splitting the VM's over the two networks.

So I hired a network tech I found on craigslist (probably a bad idea I know) to direct me to find a better router, as everyone at fios tells me the limitation is the router, not the actual fios service. The guy tells me I need a $4k Sonic Wall 3000 and a $600 support contract, otherwise any other router I run is going to be limited to ~ 100 VM's.

So my question is, is there really no middle ground between my $150 Actiontec that can handle around ~ 100 IP's and the $4k router that is supposedly capable of handling 255?
themagicone
join:2003-08-13
Osseo, MN

themagicone

Member

Get a Edge router from Ubiquiti »www.ubnt.com/edgemax/edg ··· er-lite/. $80-$100 depending on where you get it from. It could easily handle that load. As for your network, just set up a Class B subnet/dhcp - 255.255.252.0.
Expand your moderator at work
Springbok
join:2002-09-13
Leander, TX

1 recommendation

Springbok to Turtlesoft

Member

to Turtlesoft

Re: [Networking] Fios Compatible Router than can handle 200+ VMs without crashing

It looks like you are computer savvy. Hop over to the pfSense forum, do some reading and then build your own router on a i386 platform.
Dissonance
join:2007-03-26
Floral Park, NY

Dissonance to Turtlesoft

Member

to Turtlesoft
I'm using DD-WRT x86 and I'm very satisfied with it. It can be installed on any x86 hardware so you can have as much power as you want.
Turtlesoft
Premium Member
join:2014-08-25
Keller, TX

Turtlesoft

Premium Member

Someone on another forum mentioned PFsense as well. While I am slightly computer savy, I'm not very network savy My basic understanding I guess is that I build a cheap physical computer, run PFsense or DD-WRT on it and it somehow handles the handing out of IP's? Does this mean my fios router goes away completely and this machine serves as my router or does the PFsense machine attach to the router and then the other computer attache to the PFsense machine?

If that's my only option over the $4k router I guess I'll attempt it, I was really just hoping for less than a grand I could just buy a switch or upgraded router that would handle the issue, lol.

More Fiber
MVM
join:2005-09-26
Cape Coral, FL

More Fiber

MVM

said by Turtlesoft:

run PFsense or DD-WRT on it and it somehow handles the handing out of IP's?

You're talking about a function in the router called DHCP Server.
The DHCP Server in a router is responsible for handing out DHCP addresses on your local LAN.
said by Turtlesoft:

Does this mean my fios router goes away completely and this machine serves as my router

That depends on whether you have FIOS-TV or not and how your existing Actiontec in connected to the ONT.

If your existing Actiontec is connected to the ONT via coax, then you need to switch your WAN connection from coax to cat5. Once you've done that, you can correct the router of your choice directly to the ONT.

If you have FIOS-TV you can place the Actiontec behind the router of your choice so that your STBs will continue to get guide data and VOD.
Turtlesoft
Premium Member
join:2014-08-25
Keller, TX

Turtlesoft

Premium Member

On my residential fios circuit it's coax and I have Fios TV, but my 2nd circuit (fios business) is just net and it's connected via Cat5, so that shouldn't be a problem. That said, you say I can use the router of my choice, which is what I assumed. You also say it's the DHCP server portion of the router responsible for handing out IP's, so I guess then I need to ask, what router has a dhcp server capable of handing out 250 IPs?

guppy_fish
Premium Member
join:2003-12-09
Palm Harbor, FL

1 edit

guppy_fish

Premium Member

I doubt you issue is DHCP related, unless your exceeding the class C subnet limit of 254 IP's. What is your current DHCP pool size?

What likely is the issue is your router's running out of memory for NAT.

This router is spec'ed for 80K connections ~$700.00

»www.newegg.com/Product/P ··· 33339191
flashcore
join:2007-01-23
united state

flashcore to Turtlesoft

Member

to Turtlesoft
said by Turtlesoft:

My basic understanding I guess is that I build a cheap physical computer, run PFsense or DD-WRT on it and it somehow handles the handing out of IP's?

If you go the pfsense route you do not need a dedicated machine to do it, if you have an extra NIC in your VMWare server that is not being used you can run the whole thing in a virtual machine. I am currently running my 150/150 connection through a pfsense VM on an i3 with 16GB of ram along with a couple of small VMs and it uses hardly any CPU power to do so.
Expand your moderator at work
Turtlesoft
Premium Member
join:2014-08-25
Keller, TX

Turtlesoft to guppy_fish

Premium Member

to guppy_fish

Re: [Networking] Fios Compatible Router than can handle 200+ VMs without crashing

said by guppy_fish:

I doubt you issue is DHCP related, unless your exceeding the class C subnet limit of 254 IP's. What is your current DHCP pool size?

What likely is the issue is your router's running out of memory for NAT.

This router is spec'ed for 80K connections ~$700.00

»www.newegg.com/Product/P ··· 33339191

How do I find out my DHCP pool size? I had read about the NAT issues with previous versions of the Actiontec, but I thought that was solved in the latest revision I had. The router you pointed out is compatible with fios cat 5 connection? Thanks for all the replies btw everyone.
Moffetts
join:2005-05-09
San Mateo, CA

Moffetts to Turtlesoft

Member

to Turtlesoft
Sounds like you're in way over your head, OP.

rebus9
join:2002-03-26
Tampa Bay

rebus9 to Turtlesoft

Member

to Turtlesoft
said by Turtlesoft:

The guy tells me I need a $4k Sonic Wall 3000 and a $600 support contract, otherwise any other router I run is going to be limited to ~ 100 VM's.

What a load of crap!!! Let me guess... the tech guy offered to provide the Sonicwall and install for you. (?)

Unless these VMs are handling a huge number of simultaneous sessions, you'll be just fine with something like a Juniper SRX210 or SRX220. I'd look at the specs for the SRX210 first. The SRX210 is under $1000 and the SRX220 us well below $1500.

»www.juniper.net/us/en/pr ··· /srx210/

Note, Juniper gear is enterprise-class hardware, with the SRX 200-series considered "branch" routers.

At $DAYJOB we have a truckload of SRX210, SRX220, and SRX240 installed in a many-location healthcare environment with 24x7 in-patient units, community counseling offices, and corporate offices. These things are bulletproof, and Juniper TAC (support) is top-notch.

My $0.02.
serge87
join:2009-11-29
New York

serge87 to Turtlesoft

Member

to Turtlesoft
I ran a similar setup as OP and had to ditch the Actiontec router for something more powerful. I ran about 50 VM's which combined created 200-500,000 NAT table states, which caused the Actiontec to crash and hang. The Actiontec can handle 180-200K, but not completely stable for long term usage. I built a pfSense router which easily handles the load for less than $150.

Some shots:
NAT states
NAT states 2
Packets per second
Mbps

You should not have to spend hundreds or pay licensing costs.
Turtlesoft
Premium Member
join:2014-08-25
Keller, TX

Turtlesoft

Premium Member

When you say you "built" a pfSense router, this is basically a cheap computer with something like a Quad NIC and running the free PFsense software?

Thinkdiff
MVM,
join:2001-08-07
Bronx, NY

Thinkdiff

MVM,

You don't need a multi-port NIC on the box. You can get by with two NICs (or one if you're adventurous). FiOS connection would go in one NIC and you'd plug a switch into the second NIC. Then all your other equipment into the switch.

PFSense is somewhat easy to setup if your needs are basic. Configuring the firewall or more advanced routing requires quite a steep learning curve compared to SOHO routers, though. Once you get the hang of it, you won't want to go back.

As for the other poster suggesting running pfSense as a VM - that's exactly what I do and it works reasonably well. There are a few issues that pop up if you have the management interface configured for DHCP and your pfSense VM doesn't start for some reason, but setting it to static solves that.

BSAIII
join:2007-06-16
Las Vegas, NV

BSAIII to Turtlesoft

Member

to Turtlesoft
If I remember right a VM has 2 IP address options. One is to get an address from the network which could overwhelm the router. With the other option the VM uses the host IP address so the network router only 'sees' the host machine.

guppy_fish
Premium Member
join:2003-12-09
Palm Harbor, FL

guppy_fish to Turtlesoft

Premium Member

to Turtlesoft
Sound like the OP is making Seedboxes with his VM's and crashing the router with massive NAT table entrees. ( large web hosting could also possible be an problematic application using large NAT entrees )

pfSense router is a good choice, 30 bucks for 4GB of memory takes care of the NAT table needs but trying to be a network admin with some foundational knowledge is going to be challenging
Turtlesoft
Premium Member
join:2014-08-25
Keller, TX

Turtlesoft

Premium Member

lol, not seedboxes for sure. They are all running proprietary software I wrote, basically similar to twitch.tv bots, so nothing illegal like torrents.

And yes, VMware workstation has two options, Nat or Bridged (connected directly to the network) but I can't seem to get the vm's to have internet connectivity when I try to run them in Nat mode. However, even in bridged mode, if I look up the ip in a browser on any given VM, they still show the same IP as the host, which is part of what confused me when the verizon tech said I was running out of IP's.

Thinkdiff
MVM,
join:2001-08-07
Bronx, NY

Thinkdiff

MVM,

If you look at the IP address in a browser, I assume you're using a site like whatsmyip.com. That would be showing you your public IP address, not your VM's local IP. You only have one public IP.

guppy_fish
Premium Member
join:2003-12-09
Palm Harbor, FL

1 recommendation

guppy_fish to Turtlesoft

Premium Member

to Turtlesoft
Off topic, but if your doing streaming, you have a soft cap of 2TB month before Verizon come knocking
serge87
join:2009-11-29
New York

1 recommendation

serge87

Member

said by guppy_fish:

Off topic, but if your doing streaming, you have a soft cap of 2TB month before Verizon come knocking

Residential AND Business(idiotically, per usual Verizon practice)
Springbok
join:2002-09-13
Leander, TX

Springbok to Turtlesoft

Member

to Turtlesoft
said by Turtlesoft:

I can't seem to get the vm's to have internet connectivity when I try to run them in Nat mode. However, even in bridged mode, if I look up the ip in a browser on any given VM, they still show the same IP as the host, which is part of what confused me when the verizon tech said I was running out of IP's.

If you are using NAT on VM you will be NAT'ing twice; first all the VMs on that host will pull a IP address from the host's DHCP server and all those VMs will share the host's IP address externally to the host; secondly, the host(s) will pull a IP address from the router and that address will be an shared address externally. This could work fine unless you use UDP or have inbound connections. If you use UDP you should adjust the timeout value in the VM host settings and if you have inbound connections you must set port forwarding in both the router and the VM host. In short, just be aware that the host must have a IP address and have connectivity before the VMs will work.
Turtlesoft
Premium Member
join:2014-08-25
Keller, TX

Turtlesoft

Premium Member

Yeah, I'm not anywhere close to 2 TB a month, the traffic they produce is minimal.

So, to recap, there are some less expensive, sub $1k options? I see the sonicwall and the juniper networks mentioned. Alternately I could use an older machine to run pfsense? I don't think anyone answered me however on the question of "if the problem is NAT overflow rather than IP address shortage" will the new router or the PFsense setup remedy that?

I guess my whole point is that in order to expand, I either have to A) find a way to overcome this ~120 vm per circuit limit or B) add a 3rd circuit at the cost of ~ $200 per month.

Smith6612
MVM
join:2008-02-01
North Tonawanda, NY
·Charter
Ubee EU2251
Ubiquiti UAP-IW-HD
Ubiquiti UniFi AP-AC-HD

Smith6612

MVM

Honestly try the EdgeRouter or pfSense solution. The EdgeRouter has become a bit more user friendly in recent times with firmware upgrades. It's basically 5 click setup. Getting INTO the router is where some challenge is, but if you know how to set up static IPs that's really it. There's also some sample configs out on the Internet you can copy which should do the job for you.

guppy_fish
Premium Member
join:2003-12-09
Palm Harbor, FL

1 recommendation

guppy_fish to Turtlesoft

Premium Member

to Turtlesoft
Click for full size
Login to router Select My Network
Click for full size
Click on Network Connections, Then Action for Network Home Office
Click for full size
Click on Settings
Click for full size
Make changes to make the yellow highlighted items. This will expand the DHCP pool to ~509 IP's
You can easily expand your DHCP pool, follow in image/instructions. If things still fail after this, its not a DHCP address pool issue
elefante72
join:2010-12-03
East Amherst, NY

elefante72 to Turtlesoft

Member

to Turtlesoft
What is your ESX config? vSwitch. Are you using teaming? Is there an intermediate switch between the AT and your server(s). Are your VM's running VMXNET3 for adapters? Memory overcommitment? How is your storage configged (local, vsan, iscsi, NAS). What version of vSphere (free, essentials, etc), and have you patched to prevent against Heartbleed? Also what pNICs are you using in the server?

The most likely issue is a memory one, but it certainly could be a vSphere config or something going in your distribution layer (switch).

I have used this router successfully at a number of small businesses and it's only $350. This would completely replace your existing switch and actiontec. They wont charge you for updates:

»routerboard.com/RB1100AHx2

houkouonchi
join:2002-07-22
Ontario, CA

1 recommendation

houkouonchi to Turtlesoft

Member

to Turtlesoft
If u don't need crazy bandwidth throughput buy a $300 atom based machine and use that as a router with clearos, zeroshell or pfsense.

The lab I manage has 300 + baremetal machines + 200+ vms and an old quad core 2.4 Ghz xeon handles all the layer 3 routing for the lab and the only thing that uses any CPU at all is VPN stuff. Even a lowly atom (which is good for routers for its low power, low heat) can easily handle regular NAT @ line speed (1 gbit/sec).

buzz_4_20
join:2003-09-20
Dover, NH

buzz_4_20 to Turtlesoft

Member

to Turtlesoft
Most any Linux based router can handle this.

You could even put the Router inside a VM (depending on the router)