Cisco → [Info] BGP Design

So it looks like we are getting a 2nd internet connection at a 2nd site, and plan on using BGP to advertise our public network out both connections. We are going to be connecting to the same ISP, but the circuits will be terminated through different "Terapops".

I am trying to figure out how we should go about setting up the addressing, access router, firewall, and DMZ configuration.

Currently we use our public IP's between our access router, and firewall then use the remaining public IP's, and NAT into a DMZ off the firewall.

If we start advertising the public IP's out both connections I am guessing we cannot use those IP's on our access router, and firewall directly anymore. Could we use private IP's, and just setup static routers or something? Both sites would use different private subnets between the access router, and firewall I would guess.

The next step would be dealing with our DMZ. We currently one have one e-mail server, and one web server. So the only way both internet connections would work is if the DMZ were inside our private network say in its own VLAN, and then restrict traffic coming out of that VLAN so any potential compromised systems would have very limited access back into our network.

By placing the DMZ inside our network traffic from both Internet connections would be able to hit the same server. Later for redundancy we do plan on putting another server at the 2nd site, but for now I just have to make it work for current, and future plans.

Our private network has/will have enough bandwidth to route the traffic from one site to another internally.

Thanks.

We need more details for further considerations such as

1. Do you currently BGP peer at the 1st site with your ISP?

2. If you do, pick up BGP looking glass from this forum's FAQ to verify how exactly your ISP announce your subnet. The key points to look are how many hop to reach your subnet and if your subnet is announced as part of bigger subnet

3. If you don't BGP peer currently at the 1st site; how big BGP-announced your subnet will be? Would the subnet be full Class C network? Smaller? Bigger?

We are not peering at our 1st site just static routes.

The network we will be advertising is /27.

I don't think that applies - he's using the same ISP at both locations, so there's no need for his route to leave the ISP.

Step one is to contact the ISP and verify this is a supported configuration. Technically it's totally feasible, but it does fall outside of cookie-cutter config.

If you currently don't BGP peer, then then you may not need BGP at all. Each site could have its own subnet. Further, each site could be served by different carriers and things still work without BGP (kind of).

You are on the right track of having DMZ within your internal network as regard of announcing your server. To have the server reachable via either site, there will be two set of Public-Private IP address pair pointing to the same server. The server itself will have dual NIC that each has its own private IP address within different subnet. NAT device (probably the firewall) at each location will be doing static NAT/PAT between one set of Public-Private IP address pair accordingly.

On the Internet side, there will be a DNS A record pointing to both Public IP addresses. This way, any incoming traffic that hit the 1st site Public IP address will go through 1st site to reach the server. Similarly, any incoming traffic that hit the 2nd site Public IP address will go through 2nd site to reach the server.

Note that such setup would be the choice if you don't BGP peer at any location, or if you don't have direct connection between each site BGP router outside the MPLS network (outside the NAT box or firewall).

If you like "the real" automatic failover and/or load sharing, then definitely you need BGP. In addition, there must be a direct connection between two sites outside the MPLS network as mentioned. This direct connection will serve the iBGP traffic between each site BGP router. The direct connection itself could be point-to-point circuit (i.e. long-haul T1), frame relay, or utilize the ISP POP.

With this "real" failover BGP setup, then you don't need to have two set of Public-Private IP address pointing to the same server since one set is enough.

I have been in contact with them, and its a supported configuration. They will not comment on anything like DMZ placement so that's one of the reasons I'm here.

I was trying to figure out my first question about IP addressing between the access routers, and firewalls, but all I could get was "yes the addresses must be unique, and NAT should help". But I still don't know if the ISP will provide us with two additional networks one for each site, and then use the third to advertise via BGP or what.

Here is an illustration which is one way of doing it

Background
* No direct circuit between two sites outside MPLS network
* You BGP peer with the same carrier (BGP with same AS domain, i.e. AS 4)
* Both sites are at the same BGP AS domain (i.e. AS 500)
* You receive one IP block (one subnet) from the carrier to announce via BGP

BGP Implementation
* Split the one subnet into two equal smaller subnet
* Announce one smaller subnet from 1st site and announce another smaller subnet from 2nd site via BGP
* 1st site AS 500 peers with AS 4 and 2nd site AS 500 peers with also AS 4
* Verify with carrier if they support such configuration

DMZ Implementation
* Setup one pair of Public-Private IP address at 1st site and setup another pair of Public-Private IP address at 2nd site
* DMZ server is within inside network
* Each pair must come from different subnet
* Install multiple NIC to the server
* Assign both IP address pairs to the same server, where one pair goes to one NIC and another pair goes to another NIC

What are your plans to get around problems if you lose your private link between the two sites? Say traffic comes into site 1 that needs to go through the internal network to site 2, but your connectivity across MPLS is down?

Get a high speed connection though a different provider later down the road between site 1 and 2.

That is pretty much what our IPS is telling us we need to do too.

I was told they would provide us with two networks so each site would announce one of the networks.

I know that if one site fails the other would begin announcing the other network so all we would need to do is setup NAT entries for both networks to point to what we want, and it should work?

So each firewall would have NAT entries for both networks.

I found this document from Cisco that helped me a bit. I still have not found much about putting the DMZs inside the network.

»www.cisco.com/applicatio ··· b957.pdf

yaplej,

Assuming you are going with my and your ISP suggestion, here is a closer look of the BGP and NAT implementation using Cisco router and PIX Firewall.

Background
* You have 5.0.0.0/27 from your ISP to announce via BGP
* Split the network into two becomes 5.0.0.0/28 and 5.0.0.16/28
* Announce 5.0.0.0/28 from Site 1 and announce 5.0.0.16/28 from Site 2
* Both sites BGP peer with your ISP AS 4
* ISP BGP peer Site 1 IP address is 1.1.1.1 and Site 2 IP address is 2.2.2.1
* Your DMZ server has dual NIC cards
* Each NIC card is within its own subnet
* Each NIC has internal IP addresses of 10.10.0.2/24 and 10.10.1.2/24 respectively
* Traffic using Site 1 circuit will use a pair of 5.0.0.1-10.10.0.2
* Traffic using Site 2 circuit will use a pair of 5.0.0.17 10.10.1.2

Site 1

Router:
router bgp 500
network 5.0.0.0 mask 255.255.255.240
neighbor 1.1.1.1 remote-as 4

Firewall:
static (inside,outside) 5.0.0.1 10.10.0.2 netmask 255.255.255.255

Site 2

Router:
router bgp 500
network 5.0.0.16 mask 255.255.255.240
neighbor 2.2.2.1 remote-as 4

Firewall:
static (inside,outside) 5.0.0.17 10.10.1.2 netmask 255.255.255.255

On the Internet, there will be a DNS A record pointing to both 5.0.0.1 and 5.0.0.17 as the same server. The DNS server might do round robin for the server reachability.

Thanks for the info. How does that provide failover if one of the sites were to fail? In that case don't both networks need to be advertised out both sites?

I'm thinking about getting some books on BGP, but I really want something that will cover BGP for multisite designs. So far I have not found a book that specifically states it covers that topic.

The failover would rely on the DNS round-robin mechanism between the two Public IP addresses. Yes, such DNS round-robin mechanism is not an optimum failover solution as mentioned previously.

There is another approach fortunately. This time you dedicate one site (i.e. Site 1) to run the public servers and another site (Site 2) to run the Internet access for internal users. Here is the detail using the above illustration.

Background
* You have 5.0.0.0/27 from your ISP to announce via BGP
* Split the network into two becomes 5.0.0.0/28 and 5.0.0.16/28
* Announce 5.0.0.0/28 from Site 1 and announce 5.0.0.0/27 from Site 2
* Both sites BGP peer with your ISP AS 4
* ISP BGP peer Site 1 IP address is 1.1.1.1 and Site 2 IP address is 2.2.2.1
* Your DMZ server has single NIC card
* The NIC has internal IP addresses of 10.10.0.2/24
* There are two identical pair of Public-Private IP address on both Site 1 and Site 2. The pair is 5.0.0.1-10.10.0.2

Site 1

Router:
router bgp 500
network 5.0.0.0 mask 255.255.255.240
neighbor 1.1.1.1 remote-as 4

Firewall:
static (inside,outside) 5.0.0.1 10.10.0.2 netmask 255.255.255.255

Site 2

Router:
router bgp 500
network 5.0.0.0 mask 255.255.255.224
neighbor 2.2.2.1 remote-as 4

Firewall:
static (inside,outside) 5.0.0.1 10.10.0.2 netmask 255.255.255.255

On the Internet, there will be a DNS A record pointing to only 5.0.0.1. There is no need to have DNS round robin for the server reachability since BGP routing will cover it.

Note:
You may need to run BGP routing internally within your MPLS cloud to cover the Internet access. If your internal MPLS network is already BGP-based, then you can use the existing AS domains to peer with the upcoming Public AS. If your internal MPLS network is not yet BGP-based, then you may need to set one up.

As for the BGP books, I couldn't think one that is using basic practical approach. Yes, there are a lot of BGP books out there. There are even some BGP case-study books. But those books are I believe for intermediate-to-advanced reader.

If you like to have a BGP book list, here are some BGP books I have on my shelf

* BGP by Iljitsch Van Beijnum; ISBN 0596002548
* Internet Routing Architectures (2nd Edition) by Sam Halabi; ISBN 157870233X
* Routing TCP/IP, Volume II by Jeff Doyle, Jennifer DeHaven Carroll; ISBN 1578700892
* BGP Design and Implementation" by Randy Zhang, Micah Bartell; ISBN 1587051095
* Cisco BGP-4 Command & Configuration Handbook by William R. Parkhurst; ISBN 158705017X

Minimal block allowed for internet BGP is /24.