dslreports logo
 
    All Forums Hot Topics Gallery
spc
Search similar:


uniqs
2529

DarkLogix
Texan and Proud
Premium Member
join:2008-10-23
Baytown, TX

DarkLogix

Premium Member

[Info] BGP and 512K routes

So how much ram does it take to host the full BGP table

»www.reddit.com/r/sysadmi ··· _causing

tubbynet
reminds me of the danse russe
MVM
join:2008-01-16
Gilbert, AZ

tubbynet

MVM

said by DarkLogix:

So how much ram does it take to host the full BGP table

its not an issue of ram -- its an issue of tcam space in hardware forwarding platforms.
most older switching platforms have static allocations of how tcam is carved. ipv4, mpls, ipv6, and mcast.

if you don't take the time to allocate your tcams correctly -- you can overflow and cause routes to either be dropped or software switched.

q.

DarkLogix
Texan and Proud
Premium Member
join:2008-10-23
Baytown, TX

DarkLogix

Premium Member

So I guess someone failed to "allocate tcams correctly"
maybe a few someones

tubbynet
reminds me of the danse russe
MVM
join:2008-01-16
Gilbert, AZ

tubbynet

MVM

said by DarkLogix:

So I guess someone failed to "allocate tcams correctly"
maybe a few someones

probably only customers who aren't aware of what is going on in the world. carriers/providers have been constantly working and tweaking this for years -- and have been moving to xl chassis where appropriate.

if a customer is taking a full dfz -- they *should* be aware of how the box really works and what some of the concerns/caveats are.

q.
tubbynet

tubbynet to DarkLogix

MVM

to DarkLogix
also -- there is so much dumb in that thread -- it makes my head hurt.

q.

DarkLogix
Texan and Proud
Premium Member
join:2008-10-23
Baytown, TX

DarkLogix

Premium Member

Well DSLR was down for enough people for there to be a thread in the DSLR Comcast forum and its believed to be the 512K table issue

tubbynet
reminds me of the danse russe
MVM
join:2008-01-16
Gilbert, AZ

1 recommendation

tubbynet

MVM

said by DarkLogix:

Well DSLR was down for enough people for there to be a thread in the DSLR Comcast forum and its believed to be the 512K table issue

its possible.
the issue is that all pfc3-based platforms (sup720, rsp720) that are 'xl' (can handle 1m *ipv4* routes) in tcam have static carvings. ipv4 (by default) on the box is set to 512k, with mpls, mcast, and ipv6 taking up the rest. the deal is that to allocate more space to any of the categories -- you must reload the box for the change to take effect.

the dfc4-based xl cards/sups also have a 1m ipv4 route limit -- but they automatically adjust tcam as required by the system.

asr1k store the bgp fib in the esp -- with the rib being stored in sram. its possible to shove upwards of 16m routes in the rib when you disable fib learning to use the box purely as a control-plane device for bgp (i.e. bgp-rr).

trident-based linecards on the asr9k have a 1m route capacity -- which is shared between mac and route space on a per-linecard (not per chassis) basis.
the typhoon-based linecards on the asr9k expand that limit to 4m in the fib (also shared between mac and route space per linecard).

the issue is that the providers/hosting companies who don't keep track of the size of the dfz -- this will keep happening. its important to stay on top of your route table size in fixed-tcam platforms.

q.
HELLFIRE
MVM
join:2009-11-25

HELLFIRE to DarkLogix

MVM

to DarkLogix
said by tubbynet:

the dfc4-based xl cards/sups also have a 1m ipv4 route limit -- but they automatically adjust tcam as required by the system.

Question, what platforms / SUPs use the DFC4, remind me?

Can't point to all the dumb in the thread, but some good learning in there... man, I have to get back into BGP... one day.

Regards

tubbynet
reminds me of the danse russe
MVM
join:2008-01-16
Gilbert, AZ

tubbynet

MVM

said by HELLFIRE:

Question, what platforms / SUPs use the DFC4, remind me?

sup2t, sup1/sup2 on n7k are all pfc4/dfc4 based platforms.

q.
HELLFIRE
MVM
join:2009-11-25

HELLFIRE

MVM

...ahh, thanks for the reminder tubbynet See Profile

Regards

rolande
Certifiable
MVM,
join:2002-05-24
Dallas, TX
ARRIS BGW210-700
Cisco Meraki MR42

2 recommendations

rolande to DarkLogix

MVM,

to DarkLogix
This thread is pure awesomeness. This is like Bill Gates all over again... "No one will ever need more than 640K of main memory." The thing is that the Internet routing tables made it many years without actually running into this 512K route hardware limitation. Here is a better technical explanation of the likely root cause...

»www.bgpmon.net/what-caus ··· -hiccup/

So, it appears that Verizon is the culprit who accidentally de-aggregated 15K route announcements this morning and blew it up. "Can you hear me now?"

tubbynet
reminds me of the danse russe
MVM
join:2008-01-16
Gilbert, AZ

tubbynet

MVM

said by rolande:

So, it appears that Verizon is the culprit who accidentally de-aggregated 15K route announcements this morning and blew it up.

i'd love to shake the hand of the engineer that caused that.

q.

Wily_One
Premium Member
join:2002-11-24
San Jose, CA

Wily_One to DarkLogix

Premium Member

to DarkLogix
Just in case y'all have not seen this:
»supportforums.cisco.com/ ··· -effects

DarkLogix
Texan and Proud
Premium Member
join:2008-10-23
Baytown, TX

1 recommendation

DarkLogix to rolande

Premium Member

to rolande
said by rolande:

So, it appears that Verizon is the culprit who accidentally de-aggregated 15K route announcements this morning and blew it up. "Can you hear me now?"

If this is how VZ manages their network are you sure you still want FIOS? lol YES.

But IMO it kind a makes me thing that in the L3 v VZ dispute over Netflix maybe L3 is the one being honest.
DarkLogix

1 recommendation

DarkLogix to Wily_One

Premium Member

to Wily_One
said by Wily_One:

Just in case y'all have not seen this:
»supportforums.cisco.com/ ··· -effects

Can we blame IANA? ok ok its VZ's fault.

rolande
Certifiable
MVM,
join:2002-05-24
Dallas, TX
ARRIS BGW210-700
Cisco Meraki MR42

1 recommendation

rolande to Wily_One

MVM,

to Wily_One
said by Wily_One:

Just in case y'all have not seen this:
»supportforums.cisco.com/ ··· -effects

Although it is only a band-aid, something tells me that Cisco is about to have a whole lot of orders for memory show up all at once before the end of this week.

I bet Jan 1st they will probably see a bump in sales of specific ASR models and larger switch chassis as next year's budgets open up.

Wily_One
Premium Member
join:2002-11-24
San Jose, CA

1 recommendation

Wily_One

Premium Member

Yup. Funny how it takes a fire to motivate action. (And in big companies, that means the necessary buy-in and approvals from on high, without which those in the trenches can't be proactive.)
HELLFIRE
MVM
join:2009-11-25

HELLFIRE to DarkLogix

MVM

to DarkLogix
Sometimes I think the epithet "I never knew [insert here] was going to be so prevalent" should be written on the tombstone of humanity [/sarcasm]

...and this gives me some insight into some of the comments made on this thread , namely HOLDING a full BGP table (~500k routes) and SEARCHING through it are two different things.

The link rolande See Profile supplies makes a pretty good point, namely checking with your vendor about the limits of your gear.

Anyone aware of the other major router manufacturers out there -- Juniper, et al -- a) do they have the same 512K limit? , and b) what
workaround(s) / solution(s) they have to address this?

Regards

tubbynet
reminds me of the danse russe
MVM
join:2008-01-16
Gilbert, AZ

tubbynet

MVM

said by HELLFIRE:

Anyone aware of the other major router manufacturers out there -- Juniper, et al -- a) do they have the same 512K limit?

i can only speak for juniper -- but i believe that most (all?) of their routers actually use sram (rather than tcam) to store routes in rib/fib. because of this -- its all a factor of how much ram is in your 'routing engine (re)' of your specific device.

of course -- juniper is not without its issues -- most notably the krt download issue that richard steenbergen (ras) has spoken about at length on [j-nsp] and other venues.

q.
HELLFIRE
MVM
join:2009-11-25

HELLFIRE

MVM

said by tubbynet:

juniper is not without its issues -- most notably the krt download issue that richard steenbergen (ras) has spoken about at length on [j-nsp] and other venues.

Link or 50K foot summary, please?

Regards

tubbynet
reminds me of the danse russe
MVM
join:2008-01-16
Gilbert, AZ

tubbynet

MVM

said by HELLFIRE:

Link or 50K foot summary, please?

»www.nanog.org/meetings/n ··· wfib.pdf

q.

rolande
Certifiable
MVM,
join:2002-05-24
Dallas, TX
ARRIS BGW210-700
Cisco Meraki MR42

1 recommendation

rolande to DarkLogix

MVM,

to DarkLogix
said by DarkLogix:

Can we blame IANA?

IANA just hands out the netblocks. They have nothing to do with route announcement policy. Over time the carriers have all relaxed restrictions on route announcements as more and more customers want better announcement diversity for smaller and smaller netblocks. Aside from a few specific legacy exceptions, it used to be that most carriers would only accept /19 prefix block announcements or larger in BGP and they would filter everything else. Nowadays that has devolved into most providers will accept down to a /24 announcement. Hence, how we arrived in the predicament we are in on the brink of an Internet IPv4 routing "apocalypse". All of these issues have been discussed for years at NANOG and what should be done about it without much widespread change. Here is a presentation from 8 years ago that invoked more discussion on the topic. It seems that there still is no good resolution to the problem.

»www.nanog.org/meetings/n ··· _N43.pdf
cramer
Premium Member
join:2007-04-10
Raleigh, NC
Westell 6100
Cisco PIX 501

1 edit

cramer to rolande

Premium Member

to rolande
said by rolande:

something tells me that Cisco is about to have a whole lot of orders for memory

Nope. TCAM is built into the hardware and cannot be changed. (at least on the older platforms.) It has what it has. Of course, you can buy newer, more up-to-date hardware with bigger TCAM's.

tubbynet
reminds me of the danse russe
MVM
join:2008-01-16
Gilbert, AZ

tubbynet

MVM

said by cramer:

TCAM is built into the hardware and cannot be changed. (at least on the older platforms.)

correct.
the only platform (that i'm aware of) where the amount of ram will affect route capacity is the asr1k -- wherein the rp ram is used to hold bgp rib. i believe that the fib is still finite at 1m entries in the esp -- but if you have a box serving as bgp-rr or taking multiple full feeds -- you can stuff that into the bgp rib in sram -- and run bgp best-path algorithm to determine the route to be installed in the fib for a given prefix.

i could be off on that understanding though -- as i've not been a part of a lot of asr1k deep-dives in a few years.

q.
HELLFIRE
MVM
join:2009-11-25

HELLFIRE to DarkLogix

MVM

to DarkLogix
said by tubbynet:

said by HELLFIRE:

said by tubbynet:

juniper is not without its issues -- most notably the krt download issue that richard steenbergen (ras) has spoken about at length on [j-nsp] and other venues.

Link or 50K foot summary, please?

»www.nanog.org/meetings/n ··· wfib.pdf

Ahh, thanks for that tubbynet See Profile , saving for future reference.

Regards

rolande
Certifiable
MVM,
join:2002-05-24
Dallas, TX
ARRIS BGW210-700
Cisco Meraki MR42

rolande to cramer

MVM,

to cramer
said by cramer:

Nope. TCAM is built into the hardware and cannot be changed. (at least on the older platforms.) It has what it has. Of course, you can buy newer, more up-to-date hardware with bigger TCAM's.

I thought that Cisco had modularized or extended TCAM memory to use virtual pools on certain platforms. I guess it is all still dedicated and fixed on chip.

tubbynet
reminds me of the danse russe
MVM
join:2008-01-16
Gilbert, AZ

tubbynet

MVM

said by rolande:

I thought that Cisco had modularized or extended TCAM memory to use virtual pools on certain platforms.

not that i'm aware of.
in some instances -- like the nexus 7000 -- wherein both 'regular' and 'xl' versions were made -- they have removed the 'regular tcam' size from the gpl. everyone buys an 'xl' card -- and the tcam is unlocked with a license.
however this moves the platform from 256k ipv4 --> 1m ipv4.

q.

battleop
join:2005-09-28
00000

battleop to tubbynet

Member

to tubbynet
He may have a stack of resumes in the other hand...

tubbynet
reminds me of the danse russe
MVM
join:2008-01-16
Gilbert, AZ

tubbynet

MVM

said by battleop:

He may have a stack of resumes in the other hand...

the sad part of it is -- i'm sure things like this happen more often than not. he just happened to be the one that did it at the wrong time.

q.

battleop
join:2005-09-28
00000

battleop to Wily_One

Member

to Wily_One
I always get what I want after an outage. Some times it's impossible to get a CEO to understand why you have to have a $30k line card that may sit in its box for years.