 yaplej
Premium Member
join:2001-02-10
White City, OR |
yaplej
Premium Member
2011-Mar-15 4:48 pm
Re: Route distribution for dual WAN
BGP Overview |
We peer to our MPLS network using ASN 65001 our neighbor AS is the providers public ASN ### in the diagram. We use the same private ASN for all our sites. At DC2 we also add a community string that lowers the priority of the routes sourced from that site in the MPLS network. This is because we advertise a default route from both DC1 and DC2 with DC1 being the primary. Only the providers network uses the community our other equipment does nothing with it. Here is our BGP config from DC2. I know its bad to redistribute an entire routing protocol but I just have not gotten around to cleaning it up. I will remove that and add network statements for each network I need to advertise eventually. router bgp 65001
no synchronization
bgp log-neighbor-changes
network 172.31.9.1 mask 255.255.255.255
redistribute ospf 1 match internal external 1 external 2
neighbor ab.abc.bcd.161 remote-as ###
neighbor ab.abc.bcd.161 send-community
neighbor ab.abc.bcd.161 route-map bgpcomm out
default-information originate
distance bgp 115 115 115
no auto-summary
route-map bgpcomm out
route-map bgpcomm permit 10
match ip address bgpcomm10
set community ###:90
ip access-list extended bgpcomm10
permit ip any any
I think there are 1024 private ASN so we could go around and assign each site one of those private ASNs. It would take a little time but our MPLS provider wont have any issues with that. Ill just need to get them on the phone to reconfigure their end of the BGP association between our CPE and their edge router for each site. From what I have read about route reflectors it would turn our full meshed Metro-E network into a Hub-and-Spoke network because all the routes would be sourced from either DC1 or DC2. I would like to keep the full mesh capability if possible on the Metro-E network but it does not look like that's possible with BGP unless you explicitly configure fully meshed BGP peering between all 40+ of the nodes again Bleh. Maybe I misunderstood how the route reflector server and clients work. |
|
| |
meta
Member
2011-Mar-15 8:10 pm
We use the same private ASN for all our sites.
^--so that is the first major issue. That is going to create years of routing headaches as a network grows. From tracking down where a prefix is originating from, to multipath loop prevention, that decision can and should be changed.
Every site, every datacenter should have a unique BGP ASN assigned. Furthermore, aAll the routing devices within the site (wan, core, distribution, etc) should participate within iBGP within the site.
With that topology, redistribution is unnecessary. All endpoint prefixes will be carried in BGP, and OSPF is just used for loopbacks and transit interfaces within a given BGP AS.
Route-reflectors can be configured within sites on edge routers (in this case wan routers) to reduce the number of configured neighbors. This permits very large scalability both within a site and across the network.
A routing topology like this with a single protocol carrying all of the prefixes will allow for granular traffic management, uniform routing policy, and ease of support & troubleshooting.
There are some bad habbits in the bgp configuration posted as well. Manually setting the distance should be avoided, default-info orig should be off as the default route should be injected by your upstream internet gateway, and redistribution should be avoided. The community sent looks like it is changing the local preference of everything advertised to 90, which is odd. Workarounds like setting admin distance, local preference, and redistribution make this look like a network in need of redesign.
There are alot of topology and configuration options available but to make the right choices you need to understand where the traffic is coming from and going to. For example, what percentage of the traffic is between remote sites, as opposed to the datacenters? |
|
yaplej
Premium Member
join:2001-02-10
White City, OR |
yaplej
Premium Member
2011-Mar-15 9:06 pm
said by meta:We use the same private ASN for all our sites.
^--so that is the first major issue. Ok so that will be my first step to assign a unique private ASN to each site. said by meta:All the routing devices within the site (wan, core, distribution, etc) should participate within iBGP within the site.
With that topology, redistribution is unnecessary. All endpoint prefixes will be carried in BGP, and OSPF is just used for loopbacks and transit interfaces within a given BGP AS. Our "CORE" switches cannot run BGP only OSPF. I can run BGP between the WAN routers but thats about it. I would prefer not to redistribute also but just have not gotten around to fixing that. said by meta:Route-reflectors can be configured within sites on edge routers (in this case wan routers) to reduce the number of configured neighbors. This permits very large scalability both within a site and across the network. Correct me if I am wrong but this does turn a full meshed network into a hub-and-spoke network yes? If SITE 4 neeeds to route to SITE 5 will it route back to DC1 or DC2 first then to its destination? When I read about a route reflector thats how it sounded to me. said by meta:A routing topology like this with a single protocol carrying all of the prefixes will allow for granular traffic management, uniform routing policy, and ease of support & troubleshooting. Sounds good.  said by meta:There are some bad habbits in the bgp configuration posted as well. Manually setting the distance should be avoided, default-info orig should be off as the default route should be injected by your upstream internet gateway, and redistribution should be avoided. Our Internet edge is just a static route right now. The firewall does not run BGP and is running OSPF with the CORE switches. Default-info-orig gives a default route to the MPLS and the rest of our sites. That might need some work I can peer with our Internet provider with BGP and they will advertise a default route to us. Still have the problem with lack of BGP support at our core. said by meta:The community sent looks like it is changing the local preference of everything advertised to 90, which is odd. This is to lower the preference of the default route coming from DC2 into the MPLS network. So outbound internet traffic will default to DC1. Its how our MPLS provider said to configure it. They have a policy that community 90 is a lower preference. The reason for this might be gone now. said by meta:Workarounds like setting admin distance, local preference, and redistribution make this look like a network in need of redesign. The admin distance setting needs to be removed. Thats something residual from before we had a separate CORE and WAN layer. said by meta:There are alot of topology and configuration options available but to make the right choices you need to understand where the traffic is coming from and going to. For example, what percentage of the traffic is between remote sites, as opposed to the datacenters? Most traffic is from remote sites to DC1 or DC2 there is a considerable volume of traffic between DC1 and DC2. Most traffic between sites is VoIP traffic. Phew giving my fingers a workout. |
|
| |
meta
Member
2011-Mar-15 11:49 pm
I think we need a whiteboard session ;P
Regarding the any-to-any vs hub-and-spoke topology, both can be constructed with caveats in an L2 wan scenario. While the BGP peering relationships form a hub/spoke topology with route reflectors, traffic can still flow from spoke to spoke directly with recursive next-hop lookup. Data doesnt necessarily have to flow along the same lines as the neighbor relationships.
Regarding local preference, usually if you want to make prefixes secondary, its easier to use a transitive BGP attribute like as-path. In your case, it would be simpler by performing a simple prepend rather than a throw-catch scenario with community values trusting the PE to manipulate local preference accordingly.
Is the bgp-free-core an issue with licensing or platform?
I think it would be beneficial to draw a detailed visio including the internet, l3 wan, l2 wan, both "datacenters", and a sample remote site. Let me know when you have time on skype and ill start scribbling. |
|
yaplej
Premium Member
join:2001-02-10
White City, OR |
yaplej
Premium Member
2011-Mar-17 1:08 am
If using a route reflectors still allows any-to-any traffic flows then that sounds like a great solution. Its so obvious once you know about it.  If prepending the as-path would be easier Im all for it. I think maybe that's something to address after reassigning new private ASNs to each of our sites. The bgp-free-core is a platform issue the core switches are Dell PC6248's and Dell has not found it in their interest to implement BGP on them yet. Its a nice cheap, fast platform but lacking some of the more advanced features. Today has been pretty busy trying to implement GETVPN on our MPLS network. Very odd issue... for another post. I will work on the diagrams and trying to indicate all the stupid stuff I done. |
|
|
| |
meta
Member
2011-Mar-17 8:11 am
What hardware are you running? Are all the WAN routers fairly recent? If this isnt a mostly legacy network there are alot of options for secure multipoint communication over both WAN clouds. |
|
yaplej
Premium Member
join:2001-02-10
White City, OR |
yaplej
Premium Member
2011-Mar-17 8:33 pm
Overview | 
Office | |
The routers are all 2811's and 7206VXRs at each DC. I am about done getting GETVPN setup though and it should work over both WAN clouds with very little changes. Just add the crypto map on the second WAN facing interface and be done with it. I created this network overview for you. Keep in mind that Metro-E does not exist yet. The EDGE routers have a static default route to the Internet and are the source for default-info-orig to the rest of the OSPF area. I have talked with our provider and we can receive a default route from them using BGP. ROUTER1 is also a source for default-info-orig in OSPF and BGP. The BGP admin distance was increased to prevent a default route loop so the default route from OSPF will be preferred. The CORE switches do not support BGP maybe in a future update or if I can get new switches. The remote sites are super simple just a 2811 and a few switches. Yes, its sad but thats one big OSPF Area 0.  |
|
1 edit |
aryoba
MVM
2011-Mar-18 9:04 am
said by yaplej:ROUTER1 is also a source for default-info-orig in OSPF and BGP. The BGP admin distance was increased to prevent a default route loop so the default route from OSPF will be preferred. Are you referring to the DC1-ROUTER1 and DC2-ROUTER1 when you say ROUTER1 that has OSPF default-information origin? I wonder why you need such OSPF default-information origin command on the ROUTER1. Is the BGP MPLS providing default gateway from somewhere? Maybe a default gateway in form of static route configuration on the ROUTER1 to provide reachability to other sites? said by yaplej:The CORE switches do not support BGP maybe in a future update or if I can get new switches. Since your network is small, you could get away without CORE switches running BGP. Running BGP on the DC ROUTERs and EDGEs should be fine at this point. What are the CORE switches anyway? Non-Cisco switches? said by yaplej:The remote sites are super simple just a 2811 and a few switches. What are the switches? Catalyst 2950? said by yaplej:Yes, its sad but thats one big OSPF Area 0. The OSPF Area 0 of DC-EDGE, DC-ASA; and the OSPF Area 0 of DC-CORE, DC-ROUTER; are they the same Area 0? No redistribution between the two Area 0? |
|
|
yaplej
Premium Member
join:2001-02-10
White City, OR |
yaplej
Premium Member
2011-Mar-18 12:42 pm
said by aryoba:Are you referring to the DC1-ROUTER1 and DC2-ROUTER1 when you say ROUTER1 that has OSPF default-information origin? Yes. DC1-ROUTER1 and DC2-ROUTER1 are both running default-info-orig in both OSPF and BGP. They provide a default route to/from the local OSPF network and to/from the BGP network. So if the default route from OSPF is lost ROUTER1 will advertise the default route coming from the other site from BGP to the local OSPF network so devices can still get out to the Internet through the remote DC. said by aryoba:What are the CORE switches anyway? Non-Cisco switches? They are Dell PC6248s. said by aryoba:What are the switches? Catalyst 2950? Yes, the office switches are all 2950s. said by aryoba:The OSPF Area 0 of DC-EDGE, DC-ASA; and the OSPF Area 0 of DC-CORE, DC-ROUTER; are they the same Area 0? No redistribution between the two Area 0? Yes, they are the same area no need for redistribution between them. |
|
| |
aryoba
MVM
2011-Mar-18 4:34 pm
said by yaplej:said by aryoba:Are you referring to the DC1-ROUTER1 and DC2-ROUTER1 when you say ROUTER1 that has OSPF default-information origin? Yes. DC1-ROUTER1 and DC2-ROUTER1 are both running default-info-orig in both OSPF and BGP. They provide a default route to/from the local OSPF network and to/from the BGP network. So if the default route from OSPF is lost ROUTER1 will advertise the default route coming from the other site from BGP to the local OSPF network so devices can still get out to the Internet through the remote DC. I don't think this is a good design since the "default route" pointing through the MPLS BGP network is not really default gateway. A default gateway should be pointing to outside network (i.e. the Internet) or be pointing to non-local network. Remote sites' IP addresses are not outside network, they are still "known" networks. Are you doing BGP peering with the MPLS provider routers? If yes, then you should receive BGP route advertisement from them regarding networks of remote sites. When this is the case, you should never put static routes on the ROUTERs to reach remote sites since the ROUTERs receive the remote site IP addresses dynamically via BGP. In other words, you would only receive default gateway (the IP address of 0.0.0.0) from the ISP router (iNet). |
|
| aryoba |
to meta
said by meta:I think we need a whiteboard session ;P I would agree But no matter, I think we are moving forward to the right direction. There are a lot of network design issues to clean up, however I'm confident we can help you dealing with them and keeping up the good work. So far I think you've done a good job |
|
yaplej
Premium Member
join:2001-02-10
White City, OR |
to aryoba
said by aryoba:I don't think this is a good design since the "default route" pointing through the MPLS BGP network is not really default gateway. A default gateway should be pointing to outside network (i.e. the Internet) or be pointing to non-local network. Remote sites' IP addresses are not outside network, they are still "known" networks. Its probably not the best way to do it but if our Internet connection goes down it was how I could get the Internet traffic routed to the other connection. There is a default route out both sites. said by aryoba:Are you doing BGP peering with the MPLS provider routers? If yes, then you should receive BGP route advertisement from them regarding networks of remote sites. When this is the case, you should never put static routes on the ROUTERs to reach remote sites since the ROUTERs receive the remote site IP addresses dynamically via BGP. In other words, you would only receive default gateway (the IP address of 0.0.0.0) from the ISP router (iNet). DC1-ROUTER1, DC2-ROUTER2 and OFFICE routers all peer with the MPLS provider routers. The only place we have static routes is the EDGE routers pointing to the Internet (iNet). When distributing OSPF into BGP and BGP into OSPF at DC1-ROUTER1 and DC2-ROUTER1 the default route was not being included. That's why I had to turn on default-info-orig on each of routing protocol so when one of the Internet connections went down Internet traffic would begin flowing through the MPLS network to other site and out the other Internet connection. Tested this several times actually. |
|
| |
aryoba
MVM
2011-Mar-21 8:59 am
said by yaplej:said by aryoba:I don't think this is a good design since the "default route" pointing through the MPLS BGP network is not really default gateway. A default gateway should be pointing to outside network (i.e. the Internet) or be pointing to non-local network. Remote sites' IP addresses are not outside network, they are still "known" networks. Its probably not the best way to do it but if our Internet connection goes down it was how I could get the Internet traffic routed to the other connection. There is a default route out both sites. said by aryoba:Are you doing BGP peering with the MPLS provider routers? If yes, then you should receive BGP route advertisement from them regarding networks of remote sites. When this is the case, you should never put static routes on the ROUTERs to reach remote sites since the ROUTERs receive the remote site IP addresses dynamically via BGP. In other words, you would only receive default gateway (the IP address of 0.0.0.0) from the ISP router (iNet). DC1-ROUTER1, DC2-ROUTER2 and OFFICE routers all peer with the MPLS provider routers. The only place we have static routes is the EDGE routers pointing to the Internet (iNet). When distributing OSPF into BGP and BGP into OSPF at DC1-ROUTER1 and DC2-ROUTER1 the default route was not being included. That's why I had to turn on default-info-orig on each of routing protocol so when one of the Internet connections went down Internet traffic would begin flowing through the MPLS network to other site and out the other Internet connection. Tested this several times actually. If I'm not mistaken, it sounds like you need to do manual configuration on DC-ROUTERs when there is an outage, which include static route (default gateway) implementation and have OSPF on DC-ROUTERs announce it using the default-information originate command. Am I correct? Some clarification; what is the MPLS BGP AS number? Is it the same 65001 as your current DC1 AS number? Or else? The Metro-E fiber between DC1 and DC2; and the EoC to remote offices are not in place yet, correct? |
|
yaplej
Premium Member
join:2001-02-10
White City, OR |
yaplej
Premium Member
2011-Mar-21 1:52 pm
said by aryoba:If I'm not mistaken, it sounds like you need to do manual configuration on DC-ROUTERs when there is an outage, which include static route (default gateway) implementation and have OSPF on DC-ROUTERs announce it using the default-information originate command. Am I correct? No, the DC-ROUTERs dont need manually reconfigured in an outage. First each DC-EDGE router advertises its static default route to the Internet into OSPF via default-into-orig. This is the preferred default route for the local OSPF area 0. The DC-ROUTERs advertise the default route from the local OSPF area 0 via default-info-orig a default route into BGP with DC1-ROUTER taking preference due to the community 90 added to DC2-ROUTER. At this point the DC-ROUTERs are both receiving two default routes first one from OSPF via DC-EDGE and second from BGP via other DC-ROUTER that just a redistribution from the local OSPF default route via DC-EDGE. In both cases the default route is ultimately from one of the DC-EDGE routers. Then the DC-ROUTERs distribute the default route they receive from BGP into OSPF as an E2 route so the local OSPF route from DC-EDGE takes preference. When there are any problems with the circuit from either DC-EDGE it stops advertising the default route so the E2 route from DC-ROUTER automatically route traffic destined to the Internet through the other DC. said by aryoba:what is the MPLS BGP AS number. Our providers public ASN ###. Not a private ASN. said by aryoba:The Metro-E fiber between DC1 and DC2; and the EoC to remote offices are not in place yet, correct? Correct. They are not in place. As mentioned I agree that the best thing to do at this point is assign all our sites a unique private ASN and run BGP on both WANs. With both WANs running BGP we can route our traffic as needed. What can be done about the "CORE" routing? We are limited to an OSPF only core right now so there has to be some redistribution between BGP/OSPF at the DC-ROUTERs. Rather than redistributing OSPF to BGP and BGP to OSPF perhaps just network statements to OSPF and BGP? I am not sure if BGP/OSPF would advertise a network statement if that network were not directly connected but just a route from another routing protocol. I think we might need to keep the BGP admin distance set to 115 to prevent routing loops between DC-ROUTER1 and DC-ROUTER2. Without that it would prefer the BGP route to/from each other over delivering the traffic to OSPF where the destination network ultimately resides. |
|
| |
aryoba
MVM
2011-Mar-22 6:34 am
First of all, thanks for clarification on how your current network design works said by yaplej:As mentioned I agree that the best thing to do at this point is assign all our sites a unique private ASN and run BGP on both WANs. With both WANs running BGP we can route our traffic as needed. Good decision. If you haven't done much of BGP work, then this is gonna be a good starting project said by yaplej:Rather than redistributing OSPF to BGP and BGP to OSPF perhaps just network statements to OSPF and BGP? I would agree to use the network commands under the router bgp configuration to advertise local networks off to other sites. With the network command, you have more control of which network should be advertised, compared to OSPF-to-BGP redistribution. Further, the use of the network command states that the networks you advertise via BGP is considered local (coming from internal network) from BGP perspective by default, hence it would appear with i in BGP table. said by yaplej:I am not sure if BGP/OSPF would advertise a network statement if that network were not directly connected but just a route from another routing protocol. From BGP perspective, it does not matter if the networks you advertise via BGP is connected directly, is stated as static route in local router, or is advertised from "local" dynamic routing protocol (in your case, the local OSPF). All of these from BGP perspective is considered internal. As long as the networks are in the local routing table, BGP would advertise them. said by yaplej:What can be done about the "CORE" routing? We are limited to an OSPF only core right now so there has to be some redistribution between BGP/OSPF at the DC-ROUTERs. As a note that you still have to have local routing in place (whether as directly connected, static routes, or in your case OSPF) to maintain connectivity between your core switches and the DC-ROUTERs even when you run BGP internally. Since your core switches are only speaking OSPF, then you can keep maintaining the same OSPF domain between the DC-EDGE, DC-ASA, CORE, and DC-ROUTER. Since your DC-EDGE and DC-ASA are considered outside network compared to CORE and DC-ROUTER, I would suggest to place the DC-EDGE and DC-ASA as their own OSPF area while you can keep CORE and DC-ROUTER as Area 0. The DC-ASA would then be ABR. With this area segregation, you can do some network summarization so that the outside network don't have to see every little network you have internally. Similarly, the DC-ROUTER does not have to see every little network you have on the outside network with network summarization. said by yaplej:I think we might need to keep the BGP admin distance set to 115 to prevent routing loops between DC-ROUTER1 and DC-ROUTER2. Without that it would prefer the BGP route to/from each other over delivering the traffic to OSPF where the destination network ultimately resides. When you have different BGP AS between DC1, DC2, MPLS cloud, and remote sites; by default BGP route would prefer path with least hop (as the attribute of BGP being Path-Vector routing protocol). Therefore you don't need to change the default BGP administrative distance unless you are running BGP with non-Cisco devices that have different administrative distance by default than the Cisco devices. |
|
