dslreports logo
spacer

spacer
 
    «« DSL Hurdles Share Tool
spc

spacer




how-to block ads




50.5 Load Balance 2 ISP with Cisco

Assumption:

* Single router with multiple WAN interfaces. In this case, one WAN is Cable Internet (DHCP Client) and another WAN is ADSL PPPoE
* There are dual equal-cost gateway routes; one goes to Cable Internet and another goes to ADSL
* Public servers are using Cable Internet Public IP addresses; therefore inbound traffic from the Internet to servers and outbound traffic from servers to the Internet must always use Cable Internet
* To go out to the Internet with specific destination TCP/UDP port number, outbound traffic from Inside to the Internet must take Cable Internet
* Any other outbound traffic take the ADSL
* DMZ traffic to Inside is restricted where Inside traffic to DMZ in unrestricted

Sample Configuration

version 12.3
no service pad
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime msec localtime show-timezone
service timestamps log datetime msec localtime show-timezone
service password-encryption
!
hostname Router
!
boot-start-marker
boot-end-marker
!
no ip source-route
ip cef
!
interface FastEthernet0
 description Inside
 ip address 192.168.10.1 255.255.255.0
 ip access-group 100 in
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip nat inside
 ip virtual-reassembly
 ip route-cache flow
 no cdp enable
 ip policy route-map Inside-route
!
interface FastEthernet1
 description DMZ
 ip address 192.168.20.1 255.255.255.0
 ip access-group 103 in
 ip helper-address 192.168.10.254
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip nat inside
 ip virtual-reassembly
 ip route-cache flow
 no cdp enable
 ip policy route-map DMZ-route
!
interface FastEthernet2
 description WAN to ADSL modem
 no ip address
 pppoe-client dial-pool-number 1
 no cdp enable
!
interface FastEthernet3
 switchport mode trunk
 duplex full
 speed 100
!
interface FastEthernet4
 description Cable Internet - DHCP Client
 ip address dhcp client-id FastEthernet4
 ip access-group 101 in
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip nat outside
 ip virtual-reassembly
 ip route-cache flow
 duplex auto
 speed auto
 no cdp enable
!
interface Dialer1
 description ADSL Internet - PPPoE Client
 ip address negotiated
 no ip proxy-arp
 ip mtu 1492
 ip pim sparse-dense-mode
 ip nat outside
 ip inspect SERVICE out
 ip virtual-reassembly
 encapsulation ppp
 ip route-cache flow
 ip tcp adjust-mss 1452
 dialer pool 1
 dialer-group 2
 no cdp enable
 ppp authentication chap pap callin
 ppp chap hostname *******
 ppp chap password 7 ******
 ppp pap sent-username ****** password 7 ******
 ppp ipcp dns request
 ppp ipcp route default
 ppp ipcp address accept
!
no ip http server
no ip http secure-server
ip nat inside source list 151 interface FastEthernet4 overload
ip nat inside source list nat-ADSL interface Dialer1 overload
ip nat inside source static udp 192.168.10.140 3074 interface FastEthernet4 3074
ip nat inside source static tcp 192.168.10.140 3074 interface FastEthernet4 3074
ip nat inside source static tcp 192.168.10.141 9999 interface FastEthernet4 9999
ip nat inside source static udp 192.168.10.254 47624 interface FastEthernet4 47624
ip nat inside source static tcp 192.168.10.254 80 interface FastEthernet4 80
ip nat inside source static tcp 192.168.10.254 22 interface FastEthernet4 22
!
access-list 1 remark VTY Access-class list
access-list 1 permit 192.168.10.0 0.255.255.255
access-list 100 remark ACL Inside
access-list 100 deny   udp any any eq 3544
access-list 100 deny   udp any eq 3544 any
access-list 100 deny   ip host 255.255.255.255 any
access-list 100 deny   ip 127.0.0.0 0.255.255.255 any
access-list 100 permit ip any any
!
access-list 101 remark ACL Outside Cable
access-list 101 deny   ip 10.0.0.0 0.0.0.255 any
access-list 101 deny   ip 172.16.0.0 0.0.15.255 any
access-list 101 deny   ip 192.168.0.0 0.0.255.255 any
access-list 101 deny   ip 127.0.0.0 0.255.255.255 any
access-list 101 deny   ip host 255.255.255.255 any
access-list 101 permit udp host 192.43.244.18 eq ntp any eq ntp
access-list 101 permit tcp any any eq www
access-list 101 permit tcp any any eq 9999
access-list 101 permit udp any eq bootps any eq bootpc
access-list 101 permit icmp any any echo-reply
access-list 101 permit icmp any any time-exceeded
access-list 101 permit icmp any any unreachable
access-list 101 permit udp any any eq 47624
access-list 101 permit tcp any any eq 47624
access-list 101 permit udp any any eq 3074
access-list 101 permit tcp any any eq 3074
access-list 101 permit udp any eq domain any
access-list 101 permit tcp any any established
!
access-list 102 remark ACL Outside ADSL
access-list 102 deny   ip 10.0.0.0 0.0.0.255 any
access-list 102 deny   ip 172.16.0.0 0.0.15.255 any
access-list 102 deny   ip 192.168.0.0 0.0.255.255 any
access-list 102 deny   ip 127.0.0.0 0.255.255.255 any
access-list 102 deny   ip host 255.255.255.255 any
access-list 102 permit icmp any any echo-reply
access-list 102 permit icmp any any time-exceeded
access-list 102 permit icmp any any unreachable
access-list 102 permit udp any eq domain any
access-list 102 permit tcp any any established
!
access-list 103 remark ACL DMZ
access-list 103 permit udp any eq bootps any eq bootpc
access-list 103 permit udp any eq bootpc any eq bootps
access-list 103 permit ip 192.168.20.0 0.0.0.255 host 192.168.10.254
access-list 103 deny   ip 192.168.20.0 0.0.0.255 192.168.10.0 0.0.0.255
access-list 103 permit ip any any
!
access-list 151 deny ip 192.168.10.0 0.0.0.255 192.168.20.0 0.0.0.255
access-list 151 permit udp host 192.168.10.140 eq 3074 any
access-list 151 permit tcp host 192.168.10.140 eq 3074 any
access-list 151 permit tcp host 192.168.10.141 eq 9999 any
access-list 151 permit tcp host 192.168.10.254 eq 22 any
access-list 151 permit tcp host 192.168.10.254 eq 80 any
access-list 151 permit udp host 192.168.10.254 eq 47624 any
access-list 151 permit tcp 192.168.10.0 0.0.0.255 any eq 6667
access-list 152 permit ip 192.168.10.0 0.0.0.255 any
access-list 153 deny ip 192.168.20.0 0.0.0.255 192.168.10.0 0.0.0.255
access-list 153 permit ip 192.168.20.0 0.0.0.255 any
!
dialer-list 2 protocol ip permit
!
!
route-map Inside-route permit 10
match ip address 151
set interface Dialer1
!
route-map Inside-route permit 20
match ip address 152
set interface FastEthernet4
!
route-map Inside-route permit 30
!
!
route-map DMZ-route permit 10
match ip address 153
set interface Dialer1
!
route-map DMZ-route permit 20
!
route-map nat-ADSL permit 10
match ip address 152
set interface Dialer1
!
route-map nat-ADSL permit 20
match ip address 153
set interface Dialer1
!
route-map nat-ADSL permit 30
!
control-plane
!
banner login 
                   Unauthorized Use Is Prohibited
       All access to this device and network are logged. If
          you do not own this device or have access you
                   must disconnect immediately

!
line con 0
line aux 0
line vty 0 4
 access-class 1 in
 transport input telnet ssh
!
scheduler max-task-time 5000
scheduler allocate 4000 1000
scheduler interval 500
ntp clock-period 17175114
ntp server 192.168.10.254 prefer
!
end
 

Some Discussions

»[HELP] Cisco Router Config (2811) with two ISPs

by aryoba See Profile
last modified: 2011-11-17 13:06:49

Following thread could be a useful reference for those who may want to configure a Cisco router to do load balancing on outbound traffic across 2 Internet connections (i.e. Cable/DSL); by utilize OER (Optimized Edge Routing) with only static routes to each line (without deploying BGP at all on either line).

»Cisco router with "load balancing"

Contributed by: Angralitux See Profile

Note:

Keep in mind that some applications require consistent IP address at all times. If you use multiple public IP addresses to connect to hosts within the Internet (either outbound or inbound), then your connections might not be stable or even fail to connect.

Check out the following FAQ for more info regarding the use of multiple IP addresses for redundancy to keep stable connections.

»Cisco Forum FAQ »Redundant Link Graceful Internet Load Balance/Failover

Sample Configuration

Below is a full working sample configuration based on the thread:

version 12.4
service timestamps debug datetime localtime
service timestamps log datetime localtime
service password-encryption
!
!
resource policy
!
ip subnet-zero
!
!
!
!
ip cef
vpdn enable
!
!
clns routing
!
!
!
key chain cisco0
 key 1
  key-string 7 *******
!
!
!
!
!
!
!
!
!
!
!
!
oer master
 policy-rules OER
 max-range-utilization percent 10
 keepalive 1
 !
 border 192.168.20.1 key-chain cisco0
  interface FastEthernet0/0.21 internal
  interface FastEthernet0/0.22 internal
  interface Ethernet1/0 external
  interface Ethernet1/1 external
 !
 learn
  throughput
  delay
  periodic-interval 1
  monitor-period 2
  prefixes 200
  expire after time 1
  aggregation-type prefix-length 32
 max prefix total 2500
 backoff 180 360
 mode route control
 mode select-exit best
 periodic 180
 resolve loss priority 1 variance 1
 resolve delay priority 2 variance 1
 resolve utilization priority 3 variance 1
 resolve range priority 5
!
oer border
 local Loopback0
 active-probe address source interface Ethernet1/0
 master 192.168.20.1 key-chain cisco0
!
!
!
!
!
!
!
bba-group pppoe global
!
!
interface Loopback0
 description OER Master Controller
 ip address 192.168.20.1 255.255.255.255
!
interface FastEthernet0/0
 description To L2 Switch port 15
 no ip address
 duplex auto
 speed auto
!
interface FastEthernet0/0.1
 description Native VLAN Trunking
 encapsulation dot1Q 1 native
 ip address 192.168.205.1 255.255.255.248
!
interface FastEthernet0/0.21
 description To Switch A port 10
 encapsulation dot1Q 21
 ip address 192.168.21.1 255.255.255.252
 ip nat inside
 ip virtual-reassembly
!
interface FastEthernet0/0.22
 description To Switch B port 2
 encapsulation dot1Q 22
 ip address 192.168.22.1 255.255.255.252
 ip nat inside
 ip virtual-reassembly
!
!
interface Ethernet1/0
 description To NAT box of ISP #1
 ip address 172.16.0.2 255.255.255.252
 full-duplex
!
interface Ethernet1/1
 description To NAT box of ISP #2
 ip address 172.16.0.6 255.255.255.252
 ip virtual-reassembly
 full-duplex
!
interface Ethernet1/2
 no ip address
 shutdown
 half-duplex
!
interface Ethernet1/3
 no ip address
 shutdown
 half-duplex
!
!
!
!
ip classless
ip route 0.0.0.0 0.0.0.0 172.16.0.1
ip route 0.0.0.0 0.0.0.0 172.16.0.5
!
!
no ip http server
no ip http secure-server
!
ip prefix-list OER seq 10 permit 0.0.0.0/0
!
access-list 20 remark Approved IP addresses
access-list 20 permit 10.0.0.0 0.255.255.255
access-list 20 permit 172.16.0.0 0.15.255.255
access-list 20 permit 192.168.0.0 0.0.255.255
dialer-list 2 protocol ip permit
no cdp run
!
!
!
oer-map OER 10
 match ip address prefix-list OER
!
control-plane
!
!
!
line con 0
line aux 0
line vty 0 4
 access-class 20 in
 password 7 ******
 login
!
!
end
 

Network Setup

The above sample configuration comes from Cisco 2620XM with NM-4E module installed, running Cisco IOS 12.4.3 version. The Ethernet 1/0 goes to a NAT box #1 that connect to ISP #1 and the Ethernet 1/1 goes to another NAT box #2 of ISP #2. NAT box #1 inside interface that goes to the Ethernet 1/0 has IP address of 172.16.0.1. Likewise, the NAT box #2 inside interface that faces the Ethernet 1/1 has IP address of 172.16.0.5. Each NAT box outside interface IP address is the corresponding public IP address from their own ISP. As the LAN side, there are FastEthernet 0/0.21 and FastEthernet 0/0.22 on the 2620XM.

Please note that to make the OER works, your router setup does not necessary have to be exactly the same as the sample. The LAN side can be only one subnet for example.

Keep in mind that as prerequisite, the router needs to be able to go out to the Internet via either ISP. This sample configuration assumes basic connections to either ISP are already working. When this is not the case, please refer to different FAQ topics on how to properly setup your router (i.e. PPP/PPPoE/PPPoA, Static, DHCP).

Another essential issue is that how I setup the network. As previously mentioned, I use one dedicated NAT box for each ISP connection. Therefore the NAT and PAT process are not done at the 2620XM router. Instead they are done at each of the NAT box. The diagram below shows the network setup:
          ---- ISP #1 --- NAT box ---+
+--- 1st OER external (int e1/0)
INTERNET Router --- LAN
+--- 2nd OER external (int e1/1)
---- ISP #2 --- NAT box ---+

The reason I use dedicated NAT box for each ISP connection is the following. NAT and PAT process (according to the "industry standard" or RFC) only allows one traffic (one-one relationship); from single inside to single outside, and from single outside to single inside. In short; once the NAT process decides that single traffic from one host to use the interface Ethernet 1/0 (ISP #1), it cannot just easily change to the interface Ethernet 1/1 (ISP #2); and vice versa. To keep the NAT and PAT works as usual and integrate them with OER, I then use dedicated NAT box for each ISP. As implementation, you can use smaller router or a PIX 501 as the NAT box.

Expected Behavior

To get a better understanding of how OER works, here is the routing table of static routes from the 2620XM router:

Router>show ip route static
     68.0.0.0/32 is subnetted, 2 subnets
S 68.142.194.14 [1/0] via 172.16.0.5
S 68.142.197.57 [1/0] via 172.16.0.5
64.0.0.0/32 is subnetted, 1 subnets
S 64.65.196.6 [1/0] via 172.16.0.5
216.109.119.0/32 is subnetted, 1 subnets
S 216.109.119.252 [1/0] via 172.16.0.5
66.0.0.0/32 is subnetted, 1 subnets
S 66.163.175.128 [1/0] via 172.16.0.1
216.73.87.0/32 is subnetted, 1 subnets
S 216.73.87.187 [1/0] via 172.16.0.1
216.155.193.0/32 is subnetted, 2 subnets
S 216.155.193.186 [1/0] via 172.16.0.1
S 216.155.193.184 [1/0] via 172.16.0.1
12.0.0.0/32 is subnetted, 1 subnets
S 12.130.60.2 [1/0] via 172.16.0.5
216.109.127.0/32 is subnetted, 1 subnets
S 216.109.127.60 [1/0] via 172.16.0.1
192.149.252.0/32 is subnetted, 1 subnets
S 192.149.252.44 [1/0] via 172.16.0.5
S* 0.0.0.0/0 [1/0] via 172.16.0.1
[1/0] via 172.16.0.5

As you can verify, the above subnets are the Yahoo! website IP addresses and others. From the configuration, you can see that static routes to those subnets are not added manually; instead it is OER doing. It is the OER process that decide "the best route" for specific IP address destination, and put them as static routes.

Note that the above routing table comes up when the outbound traffic are light. When the traffic are heavier, you will see many more static routes the OER process adds.

Field notice

Note the prefix-list below if implementing active-probing (mode monitor active) from egress interfaces: ip prefix-list OER seq 10 permit 0.0.0.0/0 The 0.0.0.0/0 prefix will never timeout and will always show in the MTC table, thus any and all prefixes ever learned will always be probed every "periodic" time, regardless if the more specific learned prefix has timed out. The 0.0.0.0/0 prefix still has to probe something, so it just doesn't choose a random prefix, it probes ALL that it ever knew. You'll see that the active probes magically appear for every single prefix that has EVER been learned will begin probing them all. To get around this, don't specify the 0.0.0.0/0 as a parent route, rather use the default behavior (which is learn all routes). That way that 0.0.0.0/0 prefix is never in the MTC and thus the active probes associated with it are gone for good. Regular prefixes will timeout per the "expire after time" as well as it's associated probes. This was tested with 12.4(15)T11 on 3700 series routers. Didn't test using "monitor mode both" which uses active and passive monitoring.

OER Evolution: PfR (Performance Routing)

Cisco Performance Routing (PfR) FAQs
Performance Routing Configuration Guide
Load balancing using Performance Routing pfr/OER
Configuring Advanced Performance Routing

Some discussions

»[Config] Dual WAN with OER/PfR

by Angralitux See Profile edited by aryoba See Profile
last modified: 2014-04-23 14:13:46

Suggested prerequisite reading
»Cisco Forum FAQ »OER on Routers (without BGP; just static routes)

IP SLA is a Cisco IOS 12.3 feature available starting from 800 series router. This feature can be seen as OER alternative that is currently not available on 800 series router.

Sample Configurations
IOS NAT Load-Balancing with Optimized Edge Routing for Two Internet Connections
IOS NAT Load-Balancing and Zone-Based Policy Firewall with Optimized Edge Routing For Two Internet Connections
ASA/PIX 7.x: Redundant or Backup ISP Links Configuration Example

Discussions

»Wan failover on cisco 2620xm
»[Config] CISCO 857 Drops Internet Access when idle
»Dual WAN Saga continues
»history for ip sla icmp-echo

Command Description
»[Config] PPPoE default route

Official Cisco User Guide
Cisco IOS IP Service Level Agreements User Guide

by aryoba See Profile
last modified: 2011-07-29 15:36:50

There are a lot of questions in this forum regarding load balance of two ISP in terms of Internet-access related issue. The idea is to take connection to the alternate or second ISP in case the main or the 1st ISP connection is down, bouncing, or slow.

It might sound simple to have, but not quite simple to implement. There are several factors to consider as follow:

1. NAT (Network Address Translation) between private and public IP address
2. IP Address Reachability
3. Telco Local Loop
4. Power Outage or Mother Nature

NAT between private and public IP address

Let's say you have two independent ISP. You receive different subnet from each ISP. You decide to use the 1st ISP as your main connection to the Internet and the 2nd ISP as backup. You have a private network (using i.e. 10.x.x.x, 172.16.x.x, or 192.168.x.x) that is NAT to both ISP public IP address.

Even using OER and in occasion of simple Internet browsing, the connection might not gracefully switch from 1st ISP to the 2nd in case of down link. This situation applies when combining OER with static routes. The reason is that some applications (including simple Internet browsing) are sensitive to switching public IP address, even the 2nd ISP public IP address is NAT into the same physical internal device.

IP Address Reachability

As mentioned above, some applications (including simple Internet browsing that is using HTTP or HTTPS) are sensitive to public IP address switching from ISP #1 IP address to ISP #2's. This situation is true especially when dealing with TCP connection (i.e. HTTP, HTTPS, FTP, Mail). On TCP connection, basically you need to have the same IP address all the time.

Therefore when the main ISP connection is down, the 2nd ISP must have the knowledge on how to reach the main ISP public IP address to keep the current connection working. This IP reachable situation applies to traffic from the Internet entering the router and from inside LAN leaving the router to the Internet.

If you are a SOHO (Small Office or Home Office) user that only have broadband connection (DSL or cable Internet), then most of the time both of your ISP do not trade the knowledge on how to reach each other IP address. This situation then causes the 2nd ISP never has the knowledge on how to reach the main ISP public IP address or vice versa, in case of down link.

Telco Local Loop

Ever notice how the physical cable from your site or place goes to the ISP? If you are a SOHO user that only have broadband connection, then most likely the physical cable from your site are using the same cable bundle to the same CO (Telco Central Office). If somehow the cable bundle got disconnected (i.e. by falling tree), then connection to both ISP would be disconnected as well.

Power Outage or Mother Nature

Power Outage or Mother Nature factor is always haunting everybody, even large corporations. Keep in mind if you have power outage in your area, then connection to both ISP might as well disconnected. Mother Nature (i.e. tornado, lightning, earthquake, fire) could be causing the same effects.

Solutions

There are several network designs to accomplish load balance gracefully between two redundant links.

1. Have a multilink connection to the same ISP over different POP (Point of Presence)
2. Have a multilink connection to the same ISP using two different SLA (Service Level Agreement) or different link technology
3. Have a "virtual multilink connection" to the same 3rd ISP over two ISP
4. Have multiple links to two different ISPs

Multilink over different POP (POP Diversity)

This is basically the traditional established choice to provide the load balance. Usually the ISP requires you to have redundant T1/E1 of Frame Relay or point-to-point links (leased line or dedicated line) from your site to their nearest POP, in form of bonded T1/E1 circuits.

From physical cable connection redundancy perspective, each link should terminate at different POP. This is to ensure that you still have connection in case one of the POP fails.

In addition, you also need to discuss with your ISP as to how these POP terminate to. The ideal is to have each POP terminate to different ISP network or at least different CO. When both POP terminate to the same CO, then there is a single point of failure on the CO.

In bonded T1/E1 circuits, you will not assign two different IP addresses to each link. Instead you bond both links into one larger link, and assign just one IP address to the larger link. Physically your data might travel over the 1st or 2nd link, however logically (in IP perspective) the data travel over the same link.

Since there are actually at least two different physical circuits, a situation when one circuit is down; the 2nd circuit will automatically take over all data from the 1st circuit. Further, overload data on one circuit will activate and move into the 2nd circuit. These mechanisms are taken care of by the layer 1 and layer 2 (transparent from IP perspective). Therefore there is no need of fancy configuration on the router (no need of OER, BGP, nor any other similar stuff) since from IP perspective, the link is still up so then the router will be passing data as usual.

Usually your ISP only requires static route over the bonded link. No need to run BGP as mentioned previously (unless you ask the ISP to do so).

When one circuit is down and 2nd one is up, you might experience latency; which make sense. However your crucial applications are still able to work, which is the good news. To eliminate the latency, you can just contact your circuit provider (telco or ISP) to take a look at the circuit and repair it until both circuits are up.

Multilink using two different SLA or different link technology

When somehow you or your company can not yet afford to have bonded T1/E1 (or you simply choose not to), then you might consider having two links with different technology, i.e. Frame Relay and DSL. DSL SLA level is lower than Frame Relay, therefore having these two links is more cost-affordable than the bonded T1/E1. The usual term is that the Frame Relay would be the main connection to the ISP where the DSL would be as the backup for failover design.

To maintain redundant physical cable connection, each link should terminate to different ISP network or different CO; like the previous multilink scenario.

Having two independent links to the Internet would require the IP Address Reachability situation as mentioned previously. Therefore this design requirement is usually that you need to have both links to connect to the same telco or the same ISP.

When you have Internet connection using any link technology, your ISP would provide you with subnet. For Frame Relay or T1/E1, you might receive two subnets where the 1st is for the WAN side (assigned on the Serial interfaces) and 2nd is for the LAN side (assigned on your Ethernet LAN interface). For DSL, you probably only receive single subnet. It is technically possible however, that your ISP assigns two subnets also for the DSL link for the load balance or failover design, to match the Frame Relay or T1/E1 setup.

With that in mind, then there are two possible design using this kind of connection setup

1. The Frame Relay and DSL LAN sides are in the same subnet (in the same IP Block)
2. The Frame Relay and DSL LAN sides are in the different subnet (each LAN has its own IP Block or subnet)

Both LAN sides are in the same subnet

Your telco or ISP needs to setup their end to direct all traffic to the subnet using Frame Relay as primary link and using DSL as secondary or backup link. The router at your location needs to match such setup.

Since both links have the same subnet, usually you only need one router at your location where both links terminate to. You can have a choice to have failover router in case the main one is having problem such as lost power or hardware problem.

The downside of this design is that the secondary or backup link would never be used until the main link is down. You will be also required to have periodical connection test on this backup link (i.e. every four months) to make sure that the backup link is always ready to use whenever the main link is down.

Each LAN side has its own subnet

When it is not quite possible to have the same IP block for both Frame Relay and DSL LAN sides (or you simply choose not to have such condition), then you can have the following design. You can have the telco or ISP propagate the Frame Relay LAN subnet via the Frame Relay link as primary route and via the DSL link as secondary route. Similarly, the telco or ISP also need to propagate the DSL LAN subnet via the DSL link as primary route and via the Frame Relay link as secondary route.

This kind of design usually requires you to have two routers facing the telco or the ISP; where one is for the Frame Relay link and the another for the DSL link. To interconnect the two subnets, you would also need another router sitting behind the Frame Relay and DSL routers. This 3rd router would do the failover routing between the two LAN subnets, to match the telco or ISP routing design.

The advantage of this setup is that you can have a choice to use the DSL for less-critical applications (such as browsing to the Internet) where reserve the Frame Relay bandwidth for the most-critical applications.

You also have a choice to put failover router for all three router where each has its own; or just having a failover router for the 3rd router that does the failover routing.

Side Note

As mentioned, usually the 2nd design requires at least three routers on your location. It is technically possible however to use just one router for both links and as the failover router.

The most important issue is that either design should be on your SLA with the telco or ISP, so then you can have firm faith that the failover mechanism would go smoothly at least on the telco or ISP side.

Illustration:

You need to load balance your traffic between the Frame Relay and the DSL links. For simplicity, only necessary info is shown.

Keep in mind that this illustration serves only to show you ideas of how the network is setup. This might not the actual implementation since conditions can be varied from one ISP to another. Please discuss with your ISP on how the actual implementation is going to be.

You receive the following subnets from your ISP:

Frame Relay
Serial: 1.0.0.0/30
Ethernet: 1.0.1.0/24

DSL: 1.0.0.4/30

Following is the ISP router setup

interface Serial0
description Frame Relay
ip address 1.0.0.1 255.255.255.252

interface Ethernet0
description DSL link
ip address 1.0.0.5 255.255.255.252

ip route 1.0.1.0 255.255.255.0 1.0.0.2
ip route 1.0.1.0 255.255.255.0 1.0.0.6

Following is your router setup

interface Serial0
description Frame Relay
ip address 1.0.0.2 255.255.255.252

interface Ethernet0
description DSL link
ip address 1.0.0.6 255.255.255.252

interface FastEthernet0
description LAN
ip address 1.0.1.254 255.255.255.0

ip route 0.0.0.0 0.0.0.0 1.0.0.1
ip route 0.0.0.0 0.0.0.0 1.0.0.5

Virtual Multilink to the same 3rd ISP

This could be seen as a new approach compared to the previous choice. Basically you can keep your existing two ISP connections. However you do not go to the Internet directly over either ISP. Instead your data goes to a 3rd ISP, which then forward the data to the Internet.

The possible setup is to use VPN tunnel (IP Sec ISAKMP tunnel) over each ISP connection that goes to the 3rd ISP. This 3rd ISP provides you IP addresses that are known in their network and in the Internet. Note that

* You would have site-to-site VPN between your site and the 3rd ISP
* The 3rd ISP IP addresses would be totally different than the existing two ISP public IP addresses you have
* The two existing ISP public IP addresses would only serve as VPN peer between your site and the 3rd ISP
* The IP addresses you receive from the 3rd ISP would be the actual IP addresses you use to go out to the Internet

The VPN tunnels over the two ISP between your site and the 3rd ISP would be the virtual multilink that could be cost-effective compared to the previous traditional multilink. The implementation would be similar to the 2nd design choice, which would let you to have two different LAN subnets (one for each peer) or the same IP block for both peers.

As illustration, consider the following connection setup. Let's say that currently you have two Internet connections. One is served by DSL ISP and another is served by Cable Internet ISP. The ISP are independent to each other (not under the same group, company, nor umbrella).

You plan to have load balance or automatic failover mechanism using the existing Internet connections with minimal changes. You will then use a 3rd ISP to establish two separate IPSec VPN tunnels. One tunnel goes over the 1st ISP and another tunnel rides over the 2nd ISP.

On implementation, there would be VPN device on your side and another on the 3rd ISP side. These devices will have redundant IPSec tunnels to provide the load balance and/or automatic failover mechanism.

Note that since the IPSec VPN tunnels can ride over any circuit types, you don't need to deploy special circuits. Any circuits including broadband (DSL and Cable Internet) would work.

Keep in mind that since the VPN tunnels are across the Internet and not actual dedicated links between your site and the 3rd ISP, then there are challenging connection stability factors you need to understand. These factors become more apparent when there are one or multiple intermediate ISP (backbone ISP) between your site and the 3rd ISP.

In site-to-site VPN, the networks that come in to play are your site network, your current ISP (or your two ISP) network, the 3rd ISP network, and the intermediate or backbone ISP network. In the previous two network designs (the physical connections), there are only two networks; yours and the ISP's. In network stability perspective, less network interconnect means more stability.

Followings are situations that might affect the stability connection of site-to-site VPN.

Let's say the traffic at backbone ISP (or at either of your two first ISP) has a bottleneck situation caused either by overutilized bandwidth, maintenance, routing problem, or simply administration error. This situation will affect your connection to the 3rd ISP, which could be in form of latency or even disconnected tunnel.

In addition, site-to-site VPN could be down "with no apparent reason" even when the tunnel has been up and running stable for months. Assuming you and the 3rd ISP are using reliable VPN devices and the VPN tunnel are never blocked between two sites, this situation is quite rare even though they sometime do happen.

For reliability (and security), site-to-site VPN requires static IP; which means the IP addresses are never changed for any reason. Your ISP might provide you with static IP address, however the IP address might change once there is a power outage or lightning strikes to the ISP or your equipment. Once your IP address is changing, the VPN tunnel will be down.

Multiple links to two different ISP: Introduction to BGP Multihoming

This is considered the ideal setup for full redundancy. In case you have one failed ISP, you still have another as backup. When you have both up links to both ISP, you could do load balance or load share between the two links.

Setting up this connection to your both ISP considers the following.

* You are required to run BGP with both ISP (BGP Multihoming)
* Usually on each link, you are required to have at least full T1/E1 circuit
* Each BGP relationship with each ISP should ride over dispersed POP circuit
* You are required to have Public AS (Autonomous System) number
* You are required to have Public Subnet within the Public AS number
* You are suggested to have one dedicated router for each link or each ISP
* The router is required to meet certain hardware specification, such as having certain amount of CPU power and certain amount of available RAM
* You are required to understand BGP routing concept, which is considered advanced networking topic

Keep in mind that with multiple ISP scenario, you still need to consider the basic physical connection redundancy as with a single ISP scenario. This basic includes connection to different CO or different backbone network. When both ISP terminate to the same backbone network, then you have a single point of failure on the backbone network.

The need to run BGP with both ISP

BGP is used when one ISP needs to communicate with different ISP and to whole Internet users. When you are planning to have redundancy connection over multiple ISP, you are considered as ISP eventhough your network is not like one. This is why you need to run BGP with both ISP.

The need to have at least full T1/E1 circuit to each ISP

Redundancy involving BGP requires the "real" data network that is originally designed to carry and support Internet data. Broadband connection such as cable Internet and DSL is most likely considered the "extension" of existing non-data network. Cable Internet network is originally designed to broadcast TV programs. DSL network is originally designed for voice communication (POTS). Neither network is designed originally to carry and support Internet data.

Although some ISP might be able to support BGP over DSL, the DSL technology used is most likely SDSL instead of ADSL. Still, BGP over DSL is uncommon.

On the other hand, T1/E1 circuit is originally designed to carry and support Internet data; including the BGP support. As a note, T1/E1 circuit falls under the similar "real" data network as other "larger bandwidth" circuit technologies such as DS3, OC-x, ATM, and Gigabit Ethernet.

That is the reason why most ISP requires you to have T1/E1 circuit or larger to them to be able to do BGP peering with them.

Each BGP relationship with each ISP run over different POP termination (Dispersed POP)

This is basically following the same concept of dispersed POP for Multilink (bonded) circuit concept. Note that BGP Multihoming is just a logical separation and redundancy, and does not necessarily mean physical separation and redundancy. You can't really have a full redundancy without having both physical and logical separation.

The need to have your own Public AS number

When ISP run BGP with different ISP, then each ISP needs its own Public AS number. This AS number is used to distinguish between one ISP network and other ISP network.

Since you are considered as ISP when running BGP to multiple ISP, then you are also required to have your own BGP AS number. When you don't have one yet, then one of the ISP can provide you one.

Keep in mind that you need to inform both ISP beforehand that you will run redundancy over multiple ISP. This is to ensure that all parties involved understand what required setup to implement. The key is to make sure that your would-be Public AS number will be recognized by all ISP as valid Internet-routable Public AS number (or in other words, the Public AS number will be seen by any ISP and the rest of the Internet users).

When you don't inform the ISP of your purpose and you are requesting AS number from one of the ISP, the ISP might provide you Private AS number or AS number that is only seen by single ISP and unknown to other ISP and the rest of the Internet users.

The need to have Public Subnet within the Public AS number

Along with your own Public AS number, there must be your own Public Subnet. This Public Subnet usually in the form of followings

* At least it is full Class C (/24 CIDR), i.e. 31.45.81.0/24
* The subnet must be routeable within the Internet, therefore it can't be within 10.0.0.0/8, 172.16.0.0/12, or 192.168.0.0/16 (can't be the Private Subnet)
* The Public Subnet is statically assigned to you and only you at all times (never change)

To communicate with the Internet using this Public Subnet, there are following considerations

* Each ISP must have direct access to the Subnet using dedicated circuit you have with each of them respectively
* To communicate with the Internet, the Subnet traffic must go through either ISP and no other
* Both ISP must have direct BGP peering with each other to ensure that one ISP can reach the Subnet indirectly via the other ISP to provide redundancy (the IP Address Reachability requirement)

Router Hardware Specification

When you are running BGP to your ISP, you need to have equipment that is capable of running BGP routing. In addition, the equipment needs to have certain amount of CPU power and of RAM (memory) availability.

For Cisco router, usually it is suggested to have at least Cisco 2821 model; although the "standard" is 7206 or 7600 series model. For Cisco Layer-3 switch, it is suggested to have at least Catalyst 4500 or 6500 series model. The memory suggestion is at least 512 MB.

Since running BGP to your ISP requires a lot of CPU power and memory space availability, it is suggested to have dedicated router or Layer-3 switch on each link. When you only have a single equipment to terminate both links, make sure that the equipment is powerful enough to take the load.

You might be able to run BGP using less powerful equipment or with less memory availability. However your equipment could be severly impaired, especially when your ISP is decided (without your knowing) to propagate full BGP table instead of partial or default gateway.

Ability to Understand BGP Routing

In networking, BGP routing is considered an advanced topic. In order to understand BGP, you need to understand the IGP routing such as static, RIP, and OSPF. In addition, these IGP routing is required to support iBGP or to provide load balance (load share) between two ISP.

BGP Peering even when you only have one ISP: BGP Singlehoming

Note that it is still feasible to run BGP peering even when you only have one ISP. Several considerations to have this setup are the followings.

* You have multiple T1/E1 or larger circuits across multiple geographical locations where all circuits terminate to the same ISP
* You need to have more independent routing path decisions (instead of a mere static route of default gateway), compared to basic bonded T1/E1 circuits
* "True" BGP Multihoming is not yet an option
* You only have subnet smaller than /24 to announce via BGP to your ISP AS domain

When you don't have yet your own AS number and you plan to request one from your ISP, confirm with your ISP if the AS number you receive is Private (only seen by your ISP and unknown to the rest of Internet users) or Public (recognized by any Internet users).

More info on BGP

»BGP baby
»BGP, Hardware, etc...
»[Info] BGP Design

Following is a BGP sample configuration.

»Cisco Forum FAQ »BGP Design

Conclusion

Site-to-site VPN to 3rd ISP might sound cost-effective compared to the traditional bonded multiple dedicated circuits, the two different SLA design, or the BGP peering with ISP design. However the cost-effective factor comes with a price since then there are more challenging connection reliability factors to consider. When you have a critical application that does not tolerate down link at any time, then it is suggested to have the bonded circuit option, the two-different-SLA option, or the BGP peering option. If you can tolerate the down Internet, then you can have the option of having site-to-site VPN to 3rd ISP.

List of ISP Provides Both Physical and Virtual Multilink For Small and Medium Businesses

USA

Perimeter

Headquarter
440 Wheelers Farms Road
Suite 202
Milford, CT 06460
Phone: 800.234.2175
Website: »www.perimeterusa.com

Some Discussion

»IOS question, Two ISPs, can only ping one?
»Cisco 2811 or 2911 or something else for dual-WAN bgp

by aryoba See Profile
last modified: 2011-08-24 08:48:26