Equipment Support > Hardware By Brand > Cisco > Small ISP core design, need input

Small ISP core design, need input

So I am pretty sure that the 3750-e does NOT support vlan translation ... so that may not work.

reply to deaddawg
How small are you talking about? Is it really worth being concerned about the 8gbit limitation for a "small" ISP?

Some large ISPs in Australia (for example) dont even handle this much traffic. Were talking top 5 or 6 large, with hundreds of thousands of users.

If you ever reach a situation where you are running out of capacity at 8gbit/sec, by that stage I'd imagine youre making tens of millions of dollars a month and could afford an upgrade.

Also, a thread I read on the cisco-nsp mailing list not too long ago suggests that the 3750's (-E and non -E), and the 3560's (-E and non -E) do not handle bursty traffic very well. For an ISP this may be of concern as it will inevitably lead to packet loss.

The 4900M switches were suggested as a much more robust switch.

My concern with the 6524's was the null routing issue that i was told would not work properly.

I need to be able to offer my customer ability to send me a BGP route to black hole traffic. Does anyone do this using 6500's / 7600's? I would assume their features are very similar with the 6524.

My question about the backplane of the 24 port switch was because I would like to use a few gigabit ports as an iSCSI SAN vlan.

It seems to me you are suggesting to go with the 6524's?

reply to deaddawg
I dont have any experience with the 6524's so I couldnt recommend them as I have nothing to base a recommendation on. If anytyhing I was leaning towards your second proposal but with 4900M's.

Are you really expecting to generate a couple of gbit/sec of iSCSI traffic?

How much data traffic would you expect to be handling at the same time?

How big is this small that youre talking about?

reply to deaddawg
This is sounding alot less like an ISP and more like a colo / hosting provider network.

If that is the case i would suggest you get a nice multilayer switch that can keep up with growth and 10g.
I suspect the 6524s are built more for a niche market (metro ethernet endpoint) and could potentially carry some baggage.
6504-Es are a good option for a small shop. Alternatively if you need more routing-centric features, 7600's can perform the same role (with basically the same sup and linecards).

The 4900 series TomS mentioned are also good for L2 switching roles, i use 4948-10G's extensively right now. They are essencially a 4500 hard wired to a linecard.
If you are doing alot of routing however, i quickly outgrew them in an L3 role.

reply to deaddawg
I think I would be best suited to have two 7201's at the edge to handling my peerings.

The decision left if what to use for my core / dist / access (all-in-one). I do not want to use a 7206vxr to handle the aggregation as I believe the software-based platform would not be best suited as we grow.

The more I read, the more I am thinking that the 3750's would not handle the bursty traffic correctly. This is more an ISP, not colo, as that main reason for this setup is providing Internet access to businesses, either over DSL or Metro ethernet.

The 4948's look decent, however I cannot find the vlan translation command on that platform either. I am beginning to that I may need to move to a small 6500 chassis to get this functionality.

The issue I have with going to two 6504-E or 7604 is the cost to go with two chassis. That is where the 6524 caught my eye, seeing as it shares a lot of the 6500's abilities. Is there another product is this area that would be suited for me?

I guess another option would be to go a single chassis with dual sup's for redundancy, but I am not sure if that would be better or not.

reply to deaddawg
I will likely be pushing over a gigabit to my customers within the next 1 - 2 years, so I am uncertain if the NPE-G2 could handle it. I have heard best case ~900Mbit.

Using the 7201's as edge, they will likely be stressed running 500 - 700 Mbit/s to each provide.

I am starting to lean towards dual 7201 edge devices + a single 6504-e with sup720-3b for now. I think this solution would leave room for expansion while suiting me needs for now. While I would not have the full redundancy I wanted, I suppose I could add a second Sup for now, then add a second chassis and line card in the future.

Thoughts of this?

Without knowing what your budget is, its difficult to speculate on the best approach. If you get two sup's now you can always buy a second chassis later and move the standby sup to it. I always go with two chassis in my design (often due to budget i only get one sup per chassis) but in the end its more reliable. My experience is that software bugs on cisco that cause a supervisor (or service module) to crash, often cause the standby to crash on switchover as well. Kinda defeats the purpose of the redundant sups.

I tend to steer away from the 7200 platform, the most powerful sup can still only handle sub gigabit speed and chokes on a high PPS load. Im using ASR's for that role right now, and loving them. They are a little price prohibitive at the moment unfortunately, but most of my infrastructure is 10gig.

If this is truly a metro-E style customer facing network, i would almost exclusively go with the 7600s. Their ES-20G linecards are vastly superior to the switch linecards in the 6500s (67xx series) with service-provider centric features. L2 vpn in hardware, interface vlan locality, significantly improved QOS offerings, truly routed interfaces. You wouldnt need a WAN aggregation device apart from the 7600s, and the RSP720 with MSFC4 has a processor basically twice as fast as the current sup720 msfc3's and offers more ram. This really comes into play when you start talking about full BGP feeds from multiple upstream providers.

edit: are all your links ethernet? Or do you take POS/ATM/etc as well?

reply to tubbynet

rofl @ "router chess" thats a great term!

reply to deaddawg
Everything is ethernet in this case.

The 7201's would not technically be core, they would be facing my upstream providers and peers.

I like the advice of a 7600, perhaps I could look to get just one of these for now, a quick config shows it is out of my budget. I could try to stretch for one but def not two.

Perhaps I could use my current 3750me and 7201 along with the 7600 for my redundancy.

reply to deaddawg
You could also try and find a decent refurbished seller, i.e. someone who stocks enough of everything that they might be able to do a next flight shipment of replacement parts.

Or, at second hand rates you might be able to spare enough parts to get your box running again yourself.

A second hand 7600 will be a lot cheaper than a brand new one.

Ive had pretty good success with second hand parts in the past. Ive also got a good supplier who can generally have a part to me next day. I also try and stock a spare whereever I can, so worst case scenario is that I have to wait a day to restock my spare.

7600's are also a heavily hardware assisted platform. The supervisor is relatively unimportant to forwarding performance as long as you have the right line cards. Each of the line cards do forwarding themselves, while the supervisor takes care of processing routing updates etc, and then distributing routing tables to the line cards. This is why its such an expensive box.

Ive heard that your supervisor can die, and the whole box will continue to forward packets.

reply to deaddawg
If I were to go the 7600 route, why would I want to go with the RSP720 over the SUP720?

Also, would I need to PFC-3C (3B on 720) verisons of the SUP and DFC-3C Line cards, or would I want to go the PFC-3CXL / DFC-3CXL route? I am guessing I could manage without and upgrade my sup/line cards later?

Would a good 7600 starter box be a SUP720 and an ES+20 20xSFP line card? Or, to my question above, should I start out with RSP720 right away.

And back to one of my first concerns, someone told me not to use the 6500 series due to some null routing problems? Any idea what this could be, possibly outdated since the sup720?

(upon reading, it seems I would want PFC-3BXL or 3CXL right away due to the 256k vs 1m routes)

Yes, if you are taking full tables you will need an "XL" version of the supervisor.

If you get distributed forwarding daughterboards you need a matching DFC (if your sup is 3bxl, you need a dfc-3bxl for the linecard. If your sup is 3cxl, you need a dfc-3cxl)

You can still do millions of packets per second in hardware without distributed linecard forwarding.

The linecard depends what role you want the ports to play, if you are primarily doing L2 switching, the 6748 is a good card (providing you get the distributed forwarding card).
If you are primarily doing L3 you should look at the ES-20g linecards.
The 7600s are very hardware specific platforms, you should research the features you need and decide what the best hardware for the job is.

The RSP720 offers performance upgrades against the SUP720. If you have a SUP720 and the CPU is running hot, you will want to upgrade to a RSP720.

I have and will continue to use refurb and used hardware. I have had great luck with several suppliers (some list their gear on ebay, I dont know what your smartnet requirements are).

Providing you have full distributed forwarding capabilities, you can use a feature called nonstop forwarding. NSF will let you literally rip one or both supervisors out of a chassis and the linecards will continue sending packets. Their forwarding table wont change but it will continue as if no interruption has occoured. We tested this with the new nexus 7k platform and it worked wonderfully.

reply to deaddawg
I'm sure if they're 6500's with Sup 720's you're fine. We use them for BGP just fine and support over 80 million customers.

NSF is fine, but you rip out those sups the cards will not pass traffic between them I think. The 720's have the fabric built into them instead of the old MSFC cards that needed seperate fabric cards.

Thats why the higher end chassis like the Nexus 7k's and GSRs have fabric modules built into the chassis rather than the S720 on-the-sup approach.
With 65/76's im wondering if it can still use the crossbar though. The distributed forwarding cards can certainly switch between ports on the same card... it sounds like an interesting test in the lab!