dslreports logo
site
 
    All Forums Hot Topics Gallery
spc

spacer




how-to block ads


Search Topic:
uniqs
740
share rss forum feed

Bigpaddy_Irl

join:2005-12-12
Ireland

Load Balancing over BGP

Hi Lads,
This is kind of a follow on from this thread....
»New OSPF network and Public IP's

So Far so good I have managed to finally peer with 2 ISP's and set up iBGP between my two eBGP routers.

I don't seem to be able to set up the prepends after researching as much as I could.

Basically what is happening is if I put one of my /22 publics on a loopback on the my 2nd BGP router which is connected to ISP2, it traces back through my BGP1 router which is connected to ISP1.

Now if I bring down the BGP session with ISP1, the loopback traces fine through ISP2.

At this stage I am sure its prepends or some kind of traffic engineering I need to implement, but not sure where to start.

A heads up would be great,

This is purely for load balancing and Fail Over BTW.


DaSneaky1D
what's up
Premium,MVM
join:2001-03-29
The Lou
What do you want to balance against?

Bigpaddy_Irl

join:2005-12-12
Ireland
I want to send the traffic from the routers on my network closest to the ISP2 out away from ISP1

Bigpaddy_Irl

join:2005-12-12
Ireland
reply to Bigpaddy_Irl
There is loads of tutorials out there how to do it if you have 2 or more subnets, but none explaining how you would do it with only one public subnet :/


Rhaas
Premium
join:2005-12-19
Bernie, MO
reply to Bigpaddy_Irl
Are you wanting to engineer traffic Inbound to your network or traffic Outbound from your network?
--
I survived Hale-Bopp!

Bigpaddy_Irl

join:2005-12-12
Ireland
Inbound I guess as it's the most consumed by us. But would the outbound go out the same router?


Rhaas
Premium
join:2005-12-19
Bernie, MO
Have you looked to see what communities your upstreams support?

Prepends may work a little bit, communities can control inbound a bit better. Lastly you can deaggregate your network into smaller chunks and announce smaller subnets (frowned upon but sometimes the only choice you have).

I was forced to deaggregate as one of my upstreams does not support communities and no matter how many prepends I did the traffic would always prefer one upstream over the other.
--
I survived Hale-Bopp!


tubbynet
reminds me of the danse russe
Premium,MVM
join:2008-01-16
Chandler, AZ
kudos:1
said by Rhaas:

Have you looked to see what communities your upstreams support?

even without that -- it really depends on your architecture.
if you have a pair of "core" routers -- where everything flows towards -- you can enact policy on those two devices in bgp that will handle what you need.
a topology would work well in this scenario -- as there are many ways to skin the cat.

q.
--
"...if I in my north room dance naked, grotesquely before my mirror waving my shirt round my head and singing softly to myself..."

Bigpaddy_Irl

join:2005-12-12
Ireland
reply to Rhaas
Hi Rhaas, I don't understand you when you speak about communities support?
When you talk about deaggregate my network, do you mean break the /22 up into 4 individual /24 and announce 2 /24's on each core router?


Rhaas
Premium
join:2005-12-19
Bernie, MO
Communities are 'tags' you can add to your announcements. They are generally in the form of ASN:XXXX
ASN Being the ASN of your upstream peer. XXXX being the 'tag' which will control certain aspects of your announcements within your peers network.
For instance your peer is ASN 12345 and has a community of 12345:5001 which will prevent your peer from announcing any routes it learns from you to it's North American Peers. You would tag that announcement with that community and then those routes would not go to any of their north American peers.

»evilrouters.net/2009/03/18/using···routing/

A list of known provider communities:
»onesc.net/communities/

Yes, that is what I was referring to. I have three upstreams, two with one provider and a third with a separate provider. Because the one provider does not support communities and they are the preferred path - with everything equal, (reason being, they manipulate the MED to their upstreams which is why they are preferred) I've had to deaggregate my announcements. To that provider I announce two /23's (one /23 to each of the connections) and the third provider I announce the full /20. Because the /23 are more specific the traffic to those /23's will always come in the first provider unless there is a failure.

Tubby, can you perhaps expand more? I would like to know if there is a better solution.
--
I survived Hale-Bopp!

Bigpaddy_Irl

join:2005-12-12
Ireland
Man oh Man why does it have to be so complicated
Ok, something new happened today. Where we have our connection to our new ISP2 and our 2nd BGP router, we connected a Radio station into our BGP router on one of the ether ports. I took the loopback IP off the loopback and put it onto the WAN port and src-nat them to it via a private. When I plugged into their connection and done a tracert to a far away www site, it went out through the IPS2 to my delight

But, when I traced back via an online tracert site, it came back in through ISP1 :/

Hope this little bit of info helps...


Rhaas
Premium
join:2005-12-19
Bernie, MO
I guess we need to know exactly what you are expecting/wanting to happen.

Are you getting full tables, partial tables, or default route only from your upstream?

Have you checked your announcements through looking glasses from multiple providers to see what they believe is the best path?

I also suggest checking bgplay when making changes.
»bgplay.routeviews.org/
--
I survived Hale-Bopp!

Bigpaddy_Irl

join:2005-12-12
Ireland
At the minute, I am only getting full routes from ISP2 and default route from ISP1. I am waiting on ISP1 to send me full routes, but this is proving difficult from them!

ISP2 is not coming up in any LG's.

What I want to achieve is for all my routers close to ISP2 to go through ISP2, and if ISP2 fails, for all traffic to go through ISP1.


tubbynet
reminds me of the danse russe
Premium,MVM
join:2008-01-16
Chandler, AZ
kudos:1
reply to Bigpaddy_Irl
said by Bigpaddy_Irl:

But, when I traced back via an online tracert site, it came back in through ISP1 :/

Hope this little bit of info helps...

bgp is policy based prior to anything else. if the traceroute tool was located closer (a/s-wise) or has policy pushing it towards isp1 anywhere along the chain -- this is the direction in which it will go. this is beyond your control, really. some isolated cases of using communities, etc, or as-path prepending may work, but its highly dependent on the path that each "remote" website will take to your network.

q.
--
"...if I in my north room dance naked, grotesquely before my mirror waving my shirt round my head and singing softly to myself..."

Bigpaddy_Irl

join:2005-12-12
Ireland
So like what is the way everyone else does load balancing between two isp's when they have their own PI space?
I take it that this load balancing is not an exact science as such?


Rhaas
Premium
join:2005-12-19
Bernie, MO
reply to Bigpaddy_Irl
Click for full size
If you are taking full routes from a provider and default via the other - by default ALL* traffic will exit your network via provider you are getting full routes from. The reason being is that the learned routes are more specific than the default route. *Only unknown routes will exit your ISP1 connection.
Routes taken from an upstream will only influence the route traffic takes *leaving* your network.

Your IGP (OSPF) will take care of getting the traffic to the closest BGP router and handling routing around an offline router. Your BGP routers will then decide whether to send it out it's own upstream connection or to send it over to the other BGP router to us it's upstream.

You will be better served to think of inbound traffic control and outbound traffic control being two separate entities.

You can *control* where the traffic leaves your network.
You can *influence* where the traffic enters your network.

BGP can be as simplistic or as complex as you want to make it..

How I load balance traffic leaving my network (see image above):
BGP Router A:
Provider LC - Connection #1 50Mb/s AGGREGATE, Default Route Only
Provider Co - 600Mb/s Symmetric, Customer Routes (Partial Routes)

BGP Router B:
Provider LC - Connection #2 45Mb/s Symmetric, Default Route Only

I have two separate connections from LC into two different towns and a third connection from Co that comes into a third town but has a physical appearance on router A. My network is a ring so nearly every site has a path to router A or B.
Both routers A&B propagate a default route into OSPF.
Traffic leaving my network from a client will always go to the BGP router 'closest' (Based upon OSPF route decisions).
If there is a learned route to the destination from Provider Co, then the traffic will be routed to Router A (if it is not already there) and sent to Co. Otherwise the traffic will go directly out of that BGP routers LC connection (default route).

Inbound, I had to deaggregate my announcements (frowned upon - I know). I could not get the traffic to *NOT* come in the LC connections otherwise.
--
I survived Hale-Bopp!


tubbynet
reminds me of the danse russe
Premium,MVM
join:2008-01-16
Chandler, AZ
kudos:1
reply to Bigpaddy_Irl
said by Bigpaddy_Irl:

I take it that this load balancing is not an exact science as such?

you can influence the exit on your network anyway you choose. the beauty of bgp (i would personally run some sort of bgp on the 'core' of the network so that policy is universal between all bgp-speaking devices) is that you can push/pull traffic wherever you want.

on the ingress -- you're at the mercy of your upstreams and how the "best" path looks from their autonomous system (a/s). you can do things like communities, as-path prepending, etc -- but it may or may not work the way you want. as Rhaas See Profile has stated, deaggregating your networks into unique smaller subnets will work as well -- but check with your upstreams on their policy about network advertisements.

q.
--
"...if I in my north room dance naked, grotesquely before my mirror waving my shirt round my head and singing softly to myself..."

Bigpaddy_Irl

join:2005-12-12
Ireland
Thanks for the input lads, I will have to go away and take all what you just said into account and start playing with filters!

On a slightly different note, I had set up iBGP between my only 2 BGP routers to all them to exchange routes.

The only way I could get them to talk though, was to create an Eoip tunnel on both LAN interfaces of each BGP speaking router. But unfortunately this has created OSPF routing loops on me.

How do ye get around this? Both BGP routers are at different geographical locations.


tubbynet
reminds me of the danse russe
Premium,MVM
join:2008-01-16
Chandler, AZ
kudos:1

1 recommendation

said by Bigpaddy_Irl:

On a slightly different note, I had set up iBGP between my only 2 BGP routers to all them to exchange routes.

yes. it is a requirement that all bgp speaking devices in your network have an ibgp relationship.
in the early starts -- you can get away with direct peering. however -- if you plan to add in a ton of bgp routers -- you'll want to start looking at having a bgp route reflector.

said by Bigpaddy_Irl:

The only way I could get them to talk though, was to create an Eoip tunnel on both LAN interfaces of each BGP speaking router. But unfortunately this has created OSPF routing loops on me.

i'm not sure why (if this is a 'tik) configuration requirement or not. in theory -- if you're using loopbacks to peer internally with ibgp -- you should just be able to have your loopbacks dumped into ospf and peer between those. it is best practice to have that link between the two routers (i've seen many people use a dedicated l2 link or some kind of l2vpn to carry this information.

q.
--
"...if I in my north room dance naked, grotesquely before my mirror waving my shirt round my head and singing softly to myself..."

gunther_01
Premium
join:2004-03-29
Saybrook, IL
reply to Bigpaddy_Irl
This may get you back in to some of my previous suggestions Bigpaddy
--
»www.wirelessdatanet.net

ke4pym
Premium
join:2004-07-24
Charlotte, NC
Reviews:
·Northland Cable ..
·Time Warner Cable
·ooma
·VOIPO
·Verizon Broadban..
reply to Rhaas
said by Rhaas:

You can *control* where the traffic leaves your network.
You can *influence* where the traffic enters your network.

If you throw a Radware Linkproof in the mix, you can change that influence to control.

But you're going to eat up a lot of IP addresses in the process.


Rhaas
Premium
join:2005-12-19
Bernie, MO
No, that unit wont accomplish what we are talking about here. We are talking about influencing inbound traffic via BGP.
That unit is just a load balancer for multiple connections.
--
I survived Hale-Bopp!

Bigpaddy_Irl

join:2005-12-12
Ireland
reply to Bigpaddy_Irl
Another weird one for ye lads....
Tonight, ISP1, whom were only sending us the default route, are now sending us the full table.
When they implemented this tonight, the BGP came back up and I could see the full global routing table populating, but we had not www connectivity.

So I logged into ISP2's BGP router, and brought down the BGP session and walla!!! our www was back up. When I re-enabled the session, it didn't make a difference.

It seems now that if ISP1 goes down and comes back up, we will have to flick off the BGP session at ISP2. Why the heck is this?

I have also noticed that since we started peering with 2, if I drop to the command line and do a tracert to anywhere, I get no trace, just the direct hop to the domain name I was tracing.

Bigpaddy_Irl

join:2005-12-12
Ireland
reply to Bigpaddy_Irl
I am also seeing a lot of ospf packet discards since I connected both BGP routers into the network. It looks like some kind of a loop has occurred.

When I look under routing/bgp/advertisements, should I not just see what I am advertising? Cause when I go in there, it gives me a warning about too many advertisements to display.


Rhaas
Premium
join:2005-12-19
Bernie, MO
Can you post the ospf logs and a screen shot of the traceroute?
--
I survived Hale-Bopp!


tubbynet
reminds me of the danse russe
Premium,MVM
join:2008-01-16
Chandler, AZ
kudos:1
reply to Bigpaddy_Irl
said by Bigpaddy_Irl:

It looks like some kind of a loop has occurred.

as Rhaas See Profile has said -- you'll need to post traceroutes/routing table dumps to understand the loop. again -- this is where the topology diagram would come in handy so that we can visualize the network and understand where things are. a picture is always worth a thousand words. in networking -- multiply that by 10.

that being said -- you need to understand what prefixes are being advertised and where. in cisco land -- the concept of 'administrative distance' is used to determine which routing protocol will be preferred for the routes. in cisco case -- ebgp is preferred over ospf which is preferred over ibgp. i think in juniper-space (cue TomS_ See Profile), i think there isn't a difference between ebgp and ibgp -- but they are both more preferred than ospf. you need to understand how 'tik sets the a/d for each routing protocol so that you can understand where you should look to see if routes are being installed correctly.

while the multihoming is great for redundancy -- it can lead to problems (as you're seeing) if you don't have a handle on the routing within the network and understanding how each routing protocol interacts with each other, because its not always clear-cut.

q.
--
"...if I in my north room dance naked, grotesquely before my mirror waving my shirt round my head and singing softly to myself..."


tubbynet
reminds me of the danse russe
Premium,MVM
join:2008-01-16
Chandler, AZ
kudos:1
reply to Rhaas
said by Rhaas:

If you are taking full routes from a provider and default via the other - by default ALL* traffic will exit your network via provider you are getting full routes from.

i would say be careful making this statement.
it really depends on how you're passing bgp routes into your igp (if you are at all). if i take full feeds on the edge, but only advertise default routes into my core -- i have to influence traffic somehow, as my core will see equal-cost (or paths that are directly correlated to the igp metric between edge and core) paths.

if you are running bgp to the core -- then yes, this will be the case.

i also only recommend redistributing a dfz into ospf to my competitors.

q.
--
"...if I in my north room dance naked, grotesquely before my mirror waving my shirt round my head and singing softly to myself..."


Rhaas
Premium
join:2005-12-19
Bernie, MO
In a meshed Ibgp network, wont the most specific route win out by default? (assuming your aren't monkey'ing with anything) Therefore, receiving full routes via one provider and only a default via the second provider would mean nearly all traffic would be exiting towards the provider you are receiving full routes from (longest prefix > shorter prefix). I'm still learning BGP myself so I want to make sure I am understanding everything correctly as this is how the traffic exiting my network behaves - I received default route X 2, and a partial (customer) table. I only redistribute the defaults into OSPF.

AD's on mikrotik are the same as cisco.
Connected - 0
Static - 1
EBGP - 20
OSPF - 110
IBGP - 200
--
I survived Hale-Bopp!


tubbynet
reminds me of the danse russe
Premium,MVM
join:2008-01-16
Chandler, AZ
kudos:1
yes, with emphasis on meshed. because you are carrying reachability information via ibgp, which means you are no more than one bgp hop away, everything will be equal. as you scale and use bgp-rr, this changes. bgp-rr needs to be in forwarding path and will only pass best path to rrc's. in mpls, this is less significant, as bgp-rr for l3vpn does not require to be in forwarding path for computation.
my only point in clarifying this statement was that just because you receive full (or partial) tables on the edge, doesnt translate into core reachability. for edification, i'd look at the bgp path selection process for 'tik and make a mnemonic for it. in this way, you always know how to override a behaviour within bgp. if you talk to most carriers, they recommend only carrying loopbacks in $igp, with customer routes only in bgp, for scale. this isnt to say that all customers must peer via bgp, just that nlri (at some point) must be put in bgp (whether that is an igp with customer, static route redisty, or some trickery on your lac/lns for pppoe sessions). in doing so, you get the flexibility of using the igp for internal path selection (what path do i take to get between loopbacks) and the scale and flexibility of bgp to carry and manipulate path selection for customer traffic.

q.
--
"...if I in my north room dance naked, grotesquely before my mirror waving my shirt round my head and singing softly to myself..."