dslreports logo
site
 
    All Forums Hot Topics Gallery
spc

spacer




how-to block ads


Search Topic:
uniqs
1883
share rss forum feed

wirelessdog

join:2008-07-15
Queen Anne, MD
kudos:1

Tis the Season, HELP

Well lightning season is fully upon us and we had our first major hit today totaling everything on the tower including the nice shiny Routerboard 1100AH at the bottom which I hated anyway. Warning this post is extremely long but I believe it has the needed background and I am in need of help.

As I move in to this season I am faced with the inevitable fact I will replacing much equipment this year as every year. The only difference is I would like to focus on upgrading equipment as it is replaced as opposed to maintaining the status quo.

So here goes...

My upstream provider has a multi-homed fiber connection. They only limit by bandwidth by the port switch speed (100megs) I can plug my laptop into their switch and see 100/100 at any time day or night. From that location I have a set of RocketM5's making an 8 mile backhaul. Their limitation is as well the 100meg port speed. At that remote tower I have my edge router (Routerboard 1100AH). The RocketM5 backhauls actually are used to extend my upstream provider's Metro network so they are trunking across the M5 link and I plug into their Cisco Metro switch at the remote tower aka my headend.

Connecting directly to their Metro switch at my Headend I can pull 70-80megs down on an Internet speed test consistently.

Now, for my network. My network is comprised of what used to be 5 different WISP's now all interconnected. The network spans over two states with around 20 tower locations. Due to the historical nature of the infrastructure I literally have dozens of different subnets on my LAN ports. Everything is flat now even though it wasn't originally. I would like to maintain the flat network. It is made of a mix of Motorola Canopy, Alvarion, and Ubiquiti equipment. Most of the switching is unmanaged.

So... 70-80meg Internet speed directly to the Upstream Provider. Connecting through my 1100AH with no other traffic I get the same speed. Connecting my customers even with 10megs of traffic drops what I can pull to around 8-10 megs. I never see the router exceed 18megs total.

So, if you have made it past my lengthy introduction here are the actual questions:

1. Why can I only move 18-20megs through the Routerboard 1100AH? Could it have anything to do with my NAT rule being applied to all ports not just my WAN port? If I change the rule so the out port is set to WAN I lose connectivity between the different subnets on the network...

2. Should I move over to PPPoE. This would obviously be an extremely labor intensive task but I'm ready to do what is needed.

3. Should I get rid of my many subnets and work on say a single /20 subnet for the network?

4. Would I be better off using my Cisco 1842 router I have laying around to do NAT and set the 1100AH to do transparent traffic shaping?

Right now traffic shaping happens at the CPE only. The 1100AH does NAT Only.

Are there any other suggestions with a multi-vendor network on improving the overall network speed?


Inssomniak
The Glitch
Premium
join:2005-04-06
Cayuga, ON
kudos:2
I can't comment on the flat network as I have never run or worked on one that large. But the part about when you switch the NAT to the wan port and then you lose connectivity to everything else. It sounds like you are natting everything including traffic internally? Something doesnt sound right there. Sounds like none of your subnet devices have a default gateway maybe?

You are saying if you plug into your LAN side of your 1100ah with your laptop you can't pull but 20 Meg's, but plugging to wan side you do get 80 Meg's?
I'm not a big fan of running routerboard products at my core when I need power. I've always used x86 servers, and have no issue pulling full 100 meg speed out of it while its doing a thousand other things. My current router is a core i3 with 4 gigs and is loaded with queue trees, hundreds of pppoe users and firewall and barely breaks a sweat. Dell r210 1u 5.24 routerOS

If the network is flat switching to pppoe shouldn't be that hard as you can run pppoe and your current dhcp/static ip at the same time. But still would be time consuming.
--
OptionsDSL Wireless Internet
»www.optionsdsl.ca

wirelessdog

join:2008-07-15
Queen Anne, MD
kudos:1
reply to wirelessdog
So... if I stop NAT on the LAN side of the router and I have different devices on different subnets they need to have gateways set up? I'd have to look but I'm not sure the Canopy stuff gives me the option for a gateway. I haven't tested to confirm my theory completely. I assume I lose connectivity because the Dude turns these devices red. The Dude server is external of the network so I guess it is possible it is something wacky with the VPN connectivity as well. If I switch the NAT rule to the WAN interface being the out interface do I need to do anything special with the VPN server at that point? Perhaps a secondary NAT rule?

That being said, yes, I believe with a generic NAT masquerade rule without specifying the "out" interface it is doing NAT for everything.

What do you suggest? Using a PC with a SSD for the OS? I used to use PC's for the routing but the problem I would run into were fans stopping and hard drives failing.

On my UBNT devices I am routing/NAT at all CPE locations.

Will I see a benefit of running PPPoE?


Inssomniak
The Glitch
Premium
join:2005-04-06
Cayuga, ON
kudos:2

1 edit
Ya you need a gateway on your devices. Just assign the gateway as the ip that is assigned to your 1100ah for that subnet.

The dude goes red also because once you turn off NAT the devices don't know how to communicate back to the dude server. The way you have it now "works" but honestly isn't the way it should be done, and probably isn't the reason your speeds are low.

Your VPN shouldn't have to be changed though. Even though your network is flat, you have multiple subnets so some routing has to be involved. Natting between them internally is unnecessary and just adds confusion and trouble with diagnostics.
Technically you have a hybrid flat network I will call it but it's really a segmented network you are flattening out with internal NAT . Heh

As for pppoe benefits? This usually starts off arguments, I'm a pppoe guy all the way, it has built in authentication, accounting. All modern CPE support it. No worries of broadcast storms, easy to troubleshoot and I'm sure other reasons.

And don't buy an off the shelf pc for an x86 router! Buy a machine designed for the job, a server! Like a dell or an hp DL series with data Center grade parts and fans. Ue an SSD for sure! Awesome idea. 32 or 64 gig is lots and cheap. Also USB sticks work good, or sd cards. I went with the dell r210 because it came recommended on the mikrotik forum, it's cheap enough, has a pci-e slot to add in more network ports, and has been perfect for me since I installed it, oh and the OOB stuff is awesome too if you have a model with idrac in it!!
--
OptionsDSL Wireless Internet
»www.optionsdsl.ca

gunther_01
Premium
join:2004-03-29
Saybrook, IL
reply to wirelessdog
I'm pretty sure you are supposed to do your NAT on the outbound interface.

What you would want to do is have your one WAN port as the NAT. Then assign each subnet a masq rule to that port.Then your other ports in bridge mode with your subnets assigned to the bridge, not ports or interfaces..

I wouldn't say hybrid myself. You are a bridged network, with many subnets. The fact you have different subnets doesn't change the bridged aspect if they are all assigned to your head end router... If you are in fact assigning IP addresses to interfaces, that is the only place you are routing and may cause issues if you don't have any types of dynamic routing in place.

With MT, you need to have the proxy ARP turned on also for VPN use I believe. That may be your hang up, if and when you switch the NAT to the correct port. If you are using the MT for your VPN concentration point.
--
»www.wirelessdatanet.net

jcremin

join:2009-12-22
Siren, WI
kudos:3
reply to wirelessdog
said by wirelessdog:

Will I see a benefit of running PPPoE?

Personally, I've been running PPPoE since day 1, and I don't think I'd want to go any other way. There are a few drawbacks (all traffic has to go all the way to the PPPoE server, even if it is destined for another customer on your network) but there are so many benefits keeping all the extra junk off your network.

I too started out running my stuff on a PC, and the issue I ran into was that the power consumption drained my battery backups way too fast. About 1 year in, I swapped out the PC for an RB450G for my PPPoE concentrator running usermananger for radius, and have been running my whole network through it since (a little over 5 years now). It's terminating 300 sessions and peaking at a little over 40 megs.

It is a little underpowered for growing too much beyond that, as the CPU is peaking out around 75% during my highest bandwidth portions of the day, but it seems to still keep chugging along without flinching, and something with a bit more CPU power (like the 1100hx2) would go a long way to more than double my active sessions and handle at least 100 megs. I'm sure the CCR's would do a ton more.

Otherwise a rackmount server of some sort would probably work just fine as long as you have the power to run it in an extended power outage. About how many customers do you currently have, and what kinds of speeds do you offer? That certainly factors in to what kind of hardware would be appropriate.

wirelessdog

join:2008-07-15
Queen Anne, MD
kudos:1
reply to gunther_01
said by gunther_01:

What you would want to do is have your one WAN port as the NAT.

Setting the "out interface" as the WAN port correct?

said by gunther_01:

Then assign each subnet a masq rule to that port.

Can you elaborate a bit on this point?

said by gunther_01:

Then your other ports in bridge mode with your subnets assigned to the bridge, not ports or interfaces..

Can I accomplish this with the Mikrotik switch function and set the subnets on the Master Port?

I have a challenge with that - either Bridge or Master Port. How does QoS work? Lets say Port 1 has a Rocket 5 backhaul that can do 100megs than Port 2 has an old Alvarion VL Rev A backhaul that can only do around 10 megs... See the issue?

said by gunther_01:

may cause issues if you don't have any types of dynamic routing in place.

What kind of dynamic routing and how would I implment that?

said by gunther_01:

With MT, you need to have the proxy ARP turned on also for VPN use I believe.

Do you know where to look to see if Proxy ARP is turned on for the VPN?

gunther_01
Premium
join:2004-03-29
Saybrook, IL
reply to wirelessdog
In order.
Yes.
Each subnet has it's own Masq rule. How are you doing this now for NAT with multiple subnets...?
I've never used the "switch chip" portion that I know of. Just the bridge modes.
I also don't limit based on radio and or port speeds. I'm not sure there, but I'm sure it can be done
If you are routing at multiple places, you would want to use something like RIP, or OSPF. I'm not sure that you are. And in this case may just be because of another misconfiguration on your head end router..
The proxy deal is in the manual I'm pretty sure. That's how I figured it out I believe.
--
»www.wirelessdatanet.net

wirelessdog

join:2008-07-15
Queen Anne, MD
kudos:1
reply to wirelessdog
This is the current masq rule:

chain=srcnat action=masquerade


Inssomniak
The Glitch
Premium
join:2005-04-06
Cayuga, ON
kudos:2
said by wirelessdog:

This is the current masq rule:

chain=srcnat action=masquerade

I think for your setup you shouldnt need to do a masq rule for each subnet, but masq if it going out WAN. But its probably best you have one for each subnet as its sorta more future proof.

So you could have

chain=src-nat src-address=192.168.1.100 out-interface=WAN action=masquerade

One for each different subnet you want to masq for that clients need for internet (I do not masq out internal maintenance IPs unless necessary)

I dont use proxy-arp for any of my VPN configs. I have a standard PPTP VPN for logging in remotely or even locally to gain access to internal device IPs that customer IPs aren't allowed to get to.

You dont NEED any dynamic routing for your network, its flat, so I wouldnt worry about it. Again I wouldnt worry about different switch chip or bridge configurations right now either.

If I was to overhaul your config? I would route the whole thing, then move to OSPF for dynamic routing. Then Move to MPLS for VPLS tunnels to each PoP/AP back to your core router, and then run DHCP or PPPoE or both on those tunnels.
But Im biased, thats how I do it And assumes you have a mikrotik router or other MPLS/VPLS capable router at each PoP.
--
OptionsDSL Wireless Internet
»www.optionsdsl.ca


warwick

join:2009-06-05
Hollywood, FL
reply to wirelessdog
Hmm, wirelessdog can you provide a mock diagram of your current network (as best you can) ... that way I can draw another up for you explaining (as best I can) all points you mentioned. Should help you get started in the right direction.

The concepts are relatively easy... although it begins to get a bit tricky when you throw dynamic routing protocols in the mix.

Heres how we try to do things within our network (very basic) ... I will give you a one AP scenario.

----------------------------jibber jabber begin------------------------------

Core Router Switch AP1 CPE

Core Router Address Specifics.
Wan = 10.x.x.x./24
Lan = 192.168.2.1/24

Core Router Route Specifics.
dest=0.0.0.0/0 gateway=Wan
dest=192.168.3.0/24 gateway=192.168.2.3

Core Router Firewall/Nat Rules
chain=src-nat src-address=192.168.3.0 action=masq
chain=src-nat src-address=192.168.2.0 action=masq out-interface=Wan

Core Router Ip Pool
name=Sector 1 Pool address-space=192.168.3.2-192.168.3.254

Core Router Dhcp Server

(Here we simply add a dhcp server with AP1 as a relay serving out 192.168.3.x addresses)

AP1 Address Specifics
Lan = 192.168.2.3/24
Wlan1=192.168.3.1/24

AP1 Route Specifics
dest 0.0.0.0/0 gateway=192.168.2.1

AP1 Firewall/Nat Rules
None

Heres a sample traceroute from the router to the cpe.

# ADDRESS RT1 RT2 RT3 STATUS
1 192.168.2.3 1ms 1ms 1ms
2 192.168.3.190 10ms 8ms 3ms

...thats pretty much it. (You can scale the above to your liking) - Hopefully its relevant to what your trying to achieve.

What is your primary purpose of using VPN?
Perhaps it would be more efficient to traffic shape at the core router.
Was never a fan of pppoe so I cant really chime in on this one. (guilty

gunther_01
Premium
join:2004-03-29
Saybrook, IL
reply to wirelessdog
Just this tid bit any way...

PPTP client from the laptop should connect to routers public IP which in our example is 192.168.80.1.
Please, consult the respective manual on how to set up a PPTP client with the software You are using.

At this point (when pptp client is successfully connected) if you will try to ping any workstation form the laptop, ping will time out, because Laptop is unable to get ARPs from workstations. Solution is to set up proxy-arp on local interface

[admin@RemoteOffice] /interface ethernet> set Office arp=proxy-arp
[admin@RemoteOffice] /interface ethernet> print
Flags: X - disabled, R - running
# NAME MTU MAC-ADDRESS ARP
0 R ether1 1500 00:30:4F:0B:7B:C1 enabled
1 R ether2 1500 00:30:4F:06:62:12 proxy-arp
[admin@RemoteOffice] interface ethernet>

After proxy-arp is enabled client can successfully reach all workstations in local network behind the router.
--
»www.wirelessdatanet.net


warwick

join:2009-06-05
Hollywood, FL
said by gunther_01:

Just this tid bit any way...

PPTP client from the laptop should connect to routers public IP which in our example is 192.168.80.1.
Please, consult the respective manual on how to set up a PPTP client with the software You are using.

At this point (when pptp client is successfully connected) if you will try to ping any workstation form the laptop, ping will time out, because Laptop is unable to get ARPs from workstations. Solution is to set up proxy-arp on local interface

[admin@RemoteOffice] /interface ethernet> set Office arp=proxy-arp
[admin@RemoteOffice] /interface ethernet> print
Flags: X - disabled, R - running
# NAME MTU MAC-ADDRESS ARP
0 R ether1 1500 00:30:4F:0B:7B:C1 enabled
1 R ether2 1500 00:30:4F:06:62:12 proxy-arp
[admin@RemoteOffice] interface ethernet>

After proxy-arp is enabled client can successfully reach all workstations in local network behind the router.

The above is indeed true, hence the term proxy or intermediary - although Im curious as to wirelessdog's purpose of using pptp? I.e. for a means of external access or as a means of tunneling data back to his core router.

gunther_01
Premium
join:2004-03-29
Saybrook, IL
reply to wirelessdog
I took it to be an external access. Could be wrong though... I use a RDP server on our inside to access things myself, with VPN access via that same machine if needed.
--
»www.wirelessdatanet.net


Inssomniak
The Glitch
Premium
join:2005-04-06
Cayuga, ON
kudos:2
reply to warwick
said by warwick:

said by gunther_01:

Just this tid bit any way...

PPTP client from the laptop should connect to routers public IP which in our example is 192.168.80.1.
Please, consult the respective manual on how to set up a PPTP client with the software You are using.

At this point (when pptp client is successfully connected) if you will try to ping any workstation form the laptop, ping will time out, because Laptop is unable to get ARPs from workstations. Solution is to set up proxy-arp on local interface

[admin@RemoteOffice] /interface ethernet> set Office arp=proxy-arp
[admin@RemoteOffice] /interface ethernet> print
Flags: X - disabled, R - running
# NAME MTU MAC-ADDRESS ARP
0 R ether1 1500 00:30:4F:0B:7B:C1 enabled
1 R ether2 1500 00:30:4F:06:62:12 proxy-arp
[admin@RemoteOffice] interface ethernet>

After proxy-arp is enabled client can successfully reach all workstations in local network behind the router.

The above is indeed true, hence the term proxy or intermediary - although Im curious as to wirelessdog's purpose of using pptp? I.e. for a means of external access or as a means of tunneling data back to his core router.

I never had to do any proxy-arp? I can reach all my internal network just fine. Hmm
--
OptionsDSL Wireless Internet
»www.optionsdsl.ca

gunther_01
Premium
join:2004-03-29
Saybrook, IL
reply to wirelessdog
If your concentrator is the same as your VPN box it may be a different scenario. PPPoE does funny things internally to make stuff work like it does on the concentrator box. By all rights you wouldn't even have IP addresses assigned to your interfaces or the bridge group on the concentrator. They would be assigned to the concentrator interfaces which is different than the others LOL.

IDK, could be why it's not a problem for you.
--
»www.wirelessdatanet.net

wirelessdog

join:2008-07-15
Queen Anne, MD
kudos:1
reply to wirelessdog
I only use PPTP for management purposes. My dude serve sits off network.

wirelessdog

join:2008-07-15
Queen Anne, MD
kudos:1
reply to wirelessdog
So right now I am working to eliminate all the different subnets and migrate everything to a single /20 subnet. Once I do that, with everything set to the correct gateway I should be able to modify the Masq rule if I understand correctly.

I am working towards the CPE doing NAT. If my CPE's are doing NAT is there a way to completely eliminate NAT off the Edge router or will I need it regardless. Unfortunately it is not an option to assign public IP addresses to each customer...

jcremin

join:2009-12-22
Siren, WI
kudos:3
said by wirelessdog:

I am working towards the CPE doing NAT. If my CPE's are doing NAT is there a way to completely eliminate NAT off the Edge router or will I need it regardless. Unfortunately it is not an option to assign public IP addresses to each customer...

If your CPE's are assigned private IP addresses and doing NAT, you'll still need to do NAT at the edge where the public IP resides. I'm doing the same thing with PPPoE and private IP's for any customer who doesn't need a public. CPE NAT's the customer's network traffic through the tunnel back to the PPPoE server. Then the PPPoE server NAT's the private IP range through the WAN of the router.

wirelessdog

join:2008-07-15
Queen Anne, MD
kudos:1
reply to wirelessdog
Do you have time to give me the reader's digest version of setting up the PPPoE server practically for the customers?


Inssomniak
The Glitch
Premium
join:2005-04-06
Cayuga, ON
kudos:2
I could help with this but not time right now to go onto detail but the good thing is you can test without disturbing your existing config, but just adding a pppoe server in mikrotik to your LAN interface. Also I do recommend a radius server in the end you can use mikrotiks built in radius server (user manager) to get started.
--
OptionsDSL Wireless Internet
»www.optionsdsl.ca


Jerm

join:2000-04-10
Richland, WA
kudos:2
reply to wirelessdog
said by wirelessdog:

So right now I am working to eliminate all the different subnets and migrate everything to a single /20 subnet.

wirelessdog you've been here a long time, just curious, how does this seem like a good idea?

When in wired land, if you have a large Layer 2/3 broadcast domain and speed issues the first thing you do is section stuff up!
(ie if nothing else it will limit/contain damage of a failing network port or segment so it doesn't drop the entire network.)

I would think in wireless world where airtime is precious having huge broadcast domains would be exceptionally painful and quite a problem. Do you really want 4000 IPs all ARP'ing constantly on the same broadcast domain? I know on 1/10Gbit wired networks I have strange problems with some subnets half that size that generate a lot of broadcast traffic.

As I have no experience with MT, I just see the NAT config as a huge red flag and most likely contributes to your issues. I would hazard a guess with the RB1100 doing NAT you really *don't* have an actual flat network, and your NAT is somewhat is limiting the broadcast domains today.

What really should happen here is a proper routing and design refresh. Perhaps I am wrong and all this is easily done with some NAT magic in MT.

wirelessdog

join:2008-07-15
Queen Anne, MD
kudos:1
Perhaps I should clarify or maybe I do need to be smacked on the head.

My thought was to place all devices on the same subnet but have the storms controlled by PPPoE...

Even with the customers that do not utilize PPPoE if their CPE's are routing they would not contribute to a storm, no?


Inssomniak
The Glitch
Premium
join:2005-04-06
Cayuga, ON
kudos:2
said by wirelessdog:

Perhaps I should clarify or maybe I do need to be smacked on the head.

My thought was to place all devices on the same subnet but have the storms controlled by PPPoE...

Even with the customers that do not utilize PPPoE if their CPE's are routing they would not contribute to a storm, no?

What does your network look like now?

Any chance of a diagram, where routers and subnets are now?
Right now your CPEs are all bridged straight to the customer computer/router?
--
OptionsDSL Wireless Internet
»www.optionsdsl.ca

wirelessdog

join:2008-07-15
Queen Anne, MD
kudos:1
reply to wirelessdog
The Network is a mismash of 7 years worth of separate networks that became interconnected. Flat network but with many different subnets traversing over the same interfaces. Some clients route some bridge.


Jerm

join:2000-04-10
Richland, WA
kudos:2
reply to wirelessdog
Well all PPPoE does is tunnel the traffic. So it would prevent a user's Layer2 issue from spilling over into your Layer2. (some radios already do this or can do this easily, ie UBNT) But it still doesn't prevent issues within your own network. The client still has a single IP talking back to your PPPoE concentrator. Do you really want a single ARP from a far tower hitting every device on the network? Ouch!

You can split these things up independently of each other. Swap clients to PPPoE but have each tower be its own network/subnet and have a router at each? Or perhaps if you have such a large geographic spread that you split everything up into five subnets and group things together... These are all questions Network Engineers noodle over every day!

I don't know, there are others on here that probably love their own personal designs and feel their way is *the best*. Personally I like the idea of OSPF and route every tower, but I also like VLANS too.

wirelessdog

join:2008-07-15
Queen Anne, MD
kudos:1
reply to wirelessdog
From an expense standpoint vlan's and managed switches at each tower will probably be the way I will need to go to segment things.


Inssomniak
The Glitch
Premium
join:2005-04-06
Cayuga, ON
kudos:2
said by wirelessdog:

From an expense standpoint vlan's and managed switches at each tower will probably be the way I will need to go to segment things.

I dont know how many customers you have in all these mashed networks, but this might be the easiest way too.

I havent done it but I know damn well if I took my 16 sites, and had to convert them to routed/ospf now, I would just about lose my mind.
--
OptionsDSL Wireless Internet
»www.optionsdsl.ca

wirelessdog

join:2008-07-15
Queen Anne, MD
kudos:1
reply to wirelessdog
Yeah, I've got closer to 30 sites spread over, gosh I don't want to think how many square miles.

gunther_01
Premium
join:2004-03-29
Saybrook, IL

1 recommendation

reply to wirelessdog
That many sites, and that many miles I would in no way shape or form bridge that whole thing. I have two networks now. Our original that was fully routed, and a second one that was purchased. It's all bridged. The bridged network continuously gives us grief. And that's with the CPE doing NAT.

Unless you have some kind of client isolation at the head end. If you concentrate PPPoE at your head end. All of those broadcasts actually traverse your network twice. Once to the head end, then back out to each client. You could do a hybrid of having the concentrators at each tower, then route your BH's. Or even bridge your BH's and concentrate at each tower. There are tons of different methods. But one big flat network is pretty well a no no. It's easy, but a flippen mess when it acts up. Everyone is different, but I would say most who don't run in to "strange" issues, route. If routing works for the WWW. Why not use it on your own professional network? That's my take on it anyway
--
»www.wirelessdatanet.net