dslreports logo
 
    All Forums Hot Topics Gallery
spc
uniqs
30

rolande
Certifiable
MVM,
join:2002-05-24
Dallas, TX
ARRIS BGW210-700
Cisco Meraki MR42

rolande to SkellBasher

MVM,

to SkellBasher

Re: BGP Question

This scenario can work for you. It does take time for new routing announcements to propagate across the Internet. I think you are looking at a minimum 3-5 minute reconvergence delay.

With that said, there are tons of ways you can tune BGP to do what you want. Personally, if I were you, I would fire up BGP and leave it active all the time with your Disaster Recovery provider. Just use a route-map to prepend your AS a ton of times (many more than 7) at the beginning of your advertisements. This way, when both of your main links fail, your backup routes will start being selected much faster than if you had to wait for a new advertisement to converge. The backup BGP routes would already exist across every Internet router. They would only be selected in the event that the other 2 routes are removed from BGP if your primary BGP peers are disconnected.

There are other things you can do as people have listed here, but the most effective solution is AS prepending on the Internet. Many providers do not accept MEDs. Even if they did, that can be much more difficult to manage. MEDs typically only work if all paths have a positive MED value. The majority of providers will not enable 'always-compare-med'. Not to mention you can't control what they do with their BGP processes. They could enable the 'best-as-worst' MED option and then you'd be screwed.

I would adjust the AS prepends to a much higher number to solve the problem you saw before. You need to figure out how many AS hops your backup provider sees to your primary routes. It must be 7 or more. I would just bump your AS prepends up to like 15 at least. That should solve the problem.

If you want, post the 3640's config minus passwords and mask any identifiable config and subnets etc.

SkellBasher
Yes Sorto, I'll take my Prozac
join:2000-10-22
Niagara Falls, NY

SkellBasher

Member

Before I forget, thank you everyone for your comments and advice, much appriciated!!

The AS prepend option seems to be the best. At present it's this (And I was wrong, it was 8 prepends):

route-map Choice1-out permit 10
match ip address NOC-Prepend
set as-path prepend 7321 7321 7321 7321 7321 7321 7321 7321

So I'll bump it up to 15 and see what happens. MED seems to be out of the question, as I was told by two of the three carriers in question that they may accept the MED attribute, and the third didn't even know what I was talking about, so I left it alone. It would seem that MED, although great in theory, is one of those options that is not practical in production use for manageability reasons.

I'm working on getting access from my disaster recovery carrier to see how many AS Hops they see to my main site. I don't get the full BGP route table there, I only advertise, so I can't look there.

**Update. I'm only allowed to do 10 AS prepends, then I error out. Restriction of the hardware or IOS?

For reference:

Cisco Internetwork Operating System Software
IOS (tm) 3600 Software (C3640-I-M), Version 12.2(28), RELEASE SOFTWARE (fc5)
Technical Support: »www.cisco.com/techsupport
Copyright (c) 1986-2005 by cisco Systems, Inc.
Compiled Thu 24-Feb-05 15:52 by kellmill
Image text-base: 0x60008930, data-base: 0x60A3E000

ROM: System Bootstrap, Version 11.1(20)AA2, EARLY DEPLOYMENT RELEASE SOFTWARE (fc1)
ROM: 3600 Software (C3640-I-M), Version 12.2(28), RELEASE SOFTWARE (fc5)

XXXXXXX uptime is 8 weeks, 4 days, 22 hours, 20 minutes
System returned to ROM by reload at 17:38:43 EST Fri Mar 5 1993
System image file is "flash:c3640-i-mz.122-28.bin"

cisco 3640 (R4700) processor (revision 0x00) with 98304K/32768K bytes of memory.
Processor board ID 27581893
R4700 CPU at 100Mhz, Implementation 33, Rev 1.0
Bridging software.
X.25 software, Version 3.0.0.
2 Ethernet/IEEE 802.3 interface(s)
2 FastEthernet/IEEE 802.3 interface(s)
DRAM configuration is 64 bits wide with parity enabled.
125K bytes of non-volatile configuration memory.
8192K bytes of processor board System flash (Read/Write)

Configuration register is 0x2102
SkellBasher

SkellBasher to rolande

Member

to rolande
Click for full size
Network Diagram
Well, here's where I'm at. I've sketched a quick diagram for visual reference.

Under normal conditions, I see external BGP routes from the transit provider to AS 2 or AS 3, then AS 1. From how I understand things to work, broadcasting the extended AS Path from r1.backup site should show two BGP routes from the outside world:

1. Best path : Transit Provider -> AS 2/3 -> AS 1
2. Next Best path : Transit Provider -> AS 4 -> AS 1 x 10

However, when I enable the BGP broadcast at the backup site, the outside world only sees 1 path:

1. Best path : Transit Provider -> AS 4 -> AS 1 x 10

Would this be an indication of Carrier 3 not properly passing along my broadcasts?

I'm sanitizing configs to post for reference...

rolande
Certifiable
MVM,
join:2002-05-24
Dallas, TX
ARRIS BGW210-700
Cisco Meraki MR42

rolande

MVM,

Post a copy of the whole BGP config as well as route-maps and/or relevant ACLs or prefix lists etc. on your backup router and mask any identifiable information like networks and AS numbers etc. You must have something wrong in your prepend config for it to not properly advertise.

Also, BGP doesn't broadcast anything. It only sends and receives update messages that describe the routes in the router's BGP Routing Information Base or RIB. BGP does take time to update new or changed attributes on its routes. You may have to do a 'clear ip bgp * soft out' to kick it in the butt so it walks the RIB immediately and sends out any updates right away.

SkellBasher
Yes Sorto, I'll take my Prozac
join:2000-10-22
Niagara Falls, NY

SkellBasher

Member

Here's the BGP configs on the backup router:

router bgp AAAA
no synchronization
no bgp log-neighbor-changes
neighbor XXXX remote-as ZZZZ
neighbor XXXX ebgp-multihop 4
neighbor XXXX password YYYY
neighbor XXXX next-hop-self
neighbor XXXX route-map Choice1-out out

ip access-list standard NOC-Prepend
permit BBBB 0.0.31.255

route-map Choice1-out permit 10
match ip address NOC-Prepend
set as-path prepend [AS # 10 times]

rolande
Certifiable
MVM,
join:2002-05-24
Dallas, TX
ARRIS BGW210-700
Cisco Meraki MR42

rolande

MVM,

Where are you originating the route advertisement from? I see no network statement in this router for the route you are advertising. Are you running iBGP and advertising it on another router internally or are you redistributing the route from an IGP protocol somewhere?

SkellBasher
Yes Sorto, I'll take my Prozac
join:2000-10-22
Niagara Falls, NY

SkellBasher

Member

Sorry, I took that line out of the running config. Once I add it back, the problem of this router's advertisments becoming the only ones seen externally happens.

When I have it there, it's in the normal form:

network BBBB mask 255.255.224.0

rolande
Certifiable
MVM,
join:2002-05-24
Dallas, TX
ARRIS BGW210-700
Cisco Meraki MR42

rolande

MVM,

Have you gone to a route server and done a sh ip bgp for your prefix(es) when you advertise the backup route? I'd be curious to see if your backup provider is stripping your AS path somehow or else something really strange is going on with your advertisement to that provider.

SkellBasher
Yes Sorto, I'll take my Prozac
join:2000-10-22
Niagara Falls, NY

SkellBasher

Member

I was using a Telus route server to watch changes. Right now, with the backup router isde (as far as BGP is concerned):

route-views.on>sh ip bgp XXX
BGP routing table entry for XXX, version 6810612
Bestpath Modifiers: deterministic-med
Paths: (1 available, best #1)
Not advertised to any peer
852 3356 3549 [MY AS x1]
154.11.63.85 from 154.11.63.85 (154.11.0.148)
Origin IGP, localpref 100, valid, external, best
route-views.on>

BGP routing table entry for XXXX, version 6829081
Bestpath Modifiers: deterministic-med
Paths: (1 available, best #1)
Not advertised to any peer
852 7911 13609 [MY AS x11]
154.11.63.85 from 154.11.63.85 (154.11.0.148)
Origin IGP, localpref 100, valid, external, best
Dampinfo: penalty 482, flapped 1 times in 00:00:52
route-views.on>

So, I am presuming that the 10 prepends are being advertised out correctly, as I'm seeing 11 in the path from the route server external to everybody. But, what I can't figure out is why I only see that one route. Everything that I have read leads me to beleive that this longer route should be added as a backup route, as you said, leaving the primary, shorter AS Path untouched.

rolande
Certifiable
MVM,
join:2002-05-24
Dallas, TX
ARRIS BGW210-700
Cisco Meraki MR42

rolande

MVM,

So the prefix is showing up with the proper prepending in that particular route server's BGP table. When you say that you "only see that one route" explain that a little better. Where are you only seeing that one route? What providers are selecting the path to your backup provider as best for that route that clearly has a much longer AS path? Where have you tested from to validate this result? Can you show the route and its corresponding BGP table entry?

On your router at the backup site, you will never learn the valid prefix from your primary BGP peers, as long as you are advertising the same prefix. BGP will see the prefix announcement originating from the same AS across an eBGP peer so it believes it is a routing loop and it will not accept it inbound from your backup provider. As long as your backup router is in the same AS as your primary BGP peers it will never learn any prefixes advertised from those routers since they are in the same AS.

Technically, all routers within the same AS must have connectivity to each other via iBGP if they need to all see the same routes. In your case, this is a backup site, so you should not care about connectivity from there to your primary location using this duplicate address space.

SkellBasher
Yes Sorto, I'll take my Prozac
join:2000-10-22
Niagara Falls, NY

SkellBasher

Member

I have been assuming (most likely incorrectly) that when viewing my prefix from external route servers , I should see two paths listed, one showing the 'normal' path through the ASes connected to the NOC, and a 2nd showing the extended path to the backup site through the backup AS. Im guessing that this is not the case?

Im starting to get some differing information from the parties involved here, so I'm not going to go much further with this until I can verify everything. In additon, I think I'm a little over my head with this, and want to understand this more before I continue playing with it.

rolande
Certifiable
MVM,
join:2002-05-24
Dallas, TX
ARRIS BGW210-700
Cisco Meraki MR42

rolande

MVM,

You are correct if no attributes are being modified on your advertisements by any of your providers. However, this looks strangely like a local preference issue here. The only problem is I am assuming you are checking remote route servers that are not in the same AS as your backup provider. That would eliminate that as an option.

Default local preference for all routes in BGP is 100. If you manually tune the local pref higher, then all routers within that iBGP process will prefer that particular route. The actual symptom you will see on the router is that only the BGP path with the higher local pref will actually be injected into the BGP RIB on all iBGP routers, let alone into the routing table.

Anyway, this gets more confusing as we go further. I may need to see more detail offline to actually isolate the problem.

sporkme
drop the crantini and move it, sister
MVM
join:2000-07-01
Morristown, NJ

sporkme

MVM

One thing that may or may not apply, and this is something I was recently schooled on on a mailing list is this:

Many providers (a surprising amount, judging by my netflow stats) are buying transit from Cogent, WIlliams and other "bargain basement" providers. If they see that they can reach you via say a Level3 or UUNet, but they also see you via their (cheaper) Williams or Cogent pipe, guess which one they pick?

Even if you prepend the bejesus out of your announcements, if one of your upstreams is buying transit from one of these bottom feeders, no amount of prepending will help. Dig around a bit and you'll find a big, old-school provider that is "transit free" does some very interesting stuff to shove as much traffic as possible off to Cogent.

It would be helpful to know who transit provider #3 is. From my limited view, I only see the original poster's AS via 1239 (but via three different paths). I haven't gone route-server hunting on it though...

rolande
Certifiable
MVM,
join:2002-05-24
Dallas, TX
ARRIS BGW210-700
Cisco Meraki MR42

rolande

MVM,

This would only explain the symptom if by luck the route server he is checking happens to be using local preference to prefer the routes via the cheap transit provider. Otherwise, how would you explain the primary paths not even showing up in the BGP RIB when they advertise the backup path?

That is a very evil thing to do to customers or even transit non-customer routes. They should set MEDs instead of local pref so at least all paths continue to be advertised. Squashing the "backup" advertisement is not an Internet friendly thing to do.