Cisco → [Config] Port Channels with different distance links

I was asked this question today and I can't seem to find a solid yes or no answer.

I have a customer with two data centers that are fairly close. They are trying to create a poor man's Sonet. Right now there is about 2500 feet of fiber between site A and site B and they have 4 1Gb links in a Port Channel between the sites. They now have the ability to buy some dark fiber from site A to site B but using a different path. Taking a guess that second path is between 15-18k feet. They want to split the 4 channels between the two paths.

For what it's worth they said the link maxes out around 1.2-1.4Gb.

Has anyone ever done this where the distance between the port channels are this different? I'm thinking there may be a problem with the latency between the two distances but maybe there is a way to retard the short link to compensate for the long link.

4x1Gb over the 2500' path A, and 4x1Gb over the 15-18k' path B right? Or are they planning on mixing and matching
between the 2500' and 15-18k' paths into a portchannel? ie. 2x1GB from the 2500' path + 2x1GB from the 15-18k'
path = 4x1Gb portchannel

If the former, they've pretty much got themselves a short / long (protected) path, and I don't see any issues with that.

If the latter, then my warning would be watch what kind of traffic you shovel across that link. I had an incident
not involving a portchannel, but involved a short / long layer 2 path -- seems it'd flipped over to the long path
right before this database? replication job completely went off the rails performance-wise. Seems the traffic was
VERY intolerant of the increased physical distance over the long path, even though we couldn't see the delta from
a layer 3 latency perspective.

Stuff that's very intolerant to missing or out-of-order frames / packets would also be another possible 'gotcha' I can see.

My (useless) 00000010bits

Regards

Today it's a 4x1 over a single path and they want to split it to 2x1 over the 2500' and 2x1 over the 15-18k' path.

My biggest concern is that they run a decent amount of voice over that link and that's not very tolerant with out of order packets.

I told them they may be better doing this with BGP or OSPF but they don't want any loss of service if they loose a link. In my own network we use some iBGP for fail over but you loose audio for about 10-15 seconds if the primary link fails.

Where I work there's a building that links to the core switches via long-distance single-mode fiber, probably about a good 7,000-foot run per IDF or more. They're supposed to be gigabit.

There's noticeable slowness on Windows-based file transfers depending on the location, but the Internet flies.

Although that's probably anecdotal reference at best, I suspect you would experience less sustained throughput at worst, unless something's wrong with the media in between.

Like HELLFIRE would say..... just my 00000010 bits....

Crazy question, but is the portchannel currently a layer 2 or a layer 3 interface?

If it's layer 2, then you're going to be relying on layer 2 recovery mechanisms, most likely (R)STP and that
takes upwards of 30seconds, IIRC. I'm thinking of the "sub-50ms" recovery mechanisms claimed on main / protect
path of circuits -- here's a quick paper about it , but I've never dealt with it personally.

I'm also wondering what the driving desire is of combining the 2x1GB / 2500' path + 2x1GB / 15-18k' path... did
your client get written guarantees that the two are completely SEPARATE and DIVERSE fiber runs, or something like that?

@tired_runner


Regards

First, yes, I have done this. But it was with Cabletron equipment -- it did not go well. (it required sync'd receives to reassemble frames.)

The way etherchannel works, you'll find some src-dst pairs cross a short link while others cross a long link. A hash function determines which port a flow uses. Thus there will be some slight variation in packet throughput, but not enough to be a layer-3 problem. *pause* Probably.

I would recommend making two po's and let STP kill the "slow" (long) one unless the other fails. Or get two linux boxes and set 'em for balance-rr. (i.e. "cef per-packet" for IOS L3 interfaces -- I've done that many times to avoid the BS that comes with MLPPP)

I don't know if they have any thing written but I do know that the new path that's available does not follow the current path.

I've got a bunch of links in our data centers where we have 4x1GB but they are all nearly the same lengths. If a member goes down we just see a reduction in available throughput so there is no delay when one "path" goes down.

What's different about this possible scenario is that one "path" a couple miles longer than the other.

...and this question is more directed at your client than you, but what do they think mixing and matching
those paths will buy them? Personally, I'd say do it short / long path for redundancy (like telco best practices would
do it) and be done with it....don't know if anyone else may chime in, but I think the main takeaway is Caveat Emptor / YMMV / et al

Regards

"but what do they think mixing and matching those paths will buy them?"

They are looking for some redundancy. If path A gets dug up they will drop back from 4GB to 2GB. Since the traffic never exceeds 2GB then the would be OK during that time.

"..don't know if anyone else may chime in, but I think the main takeaway is Caveat Emptor / YMMV / et al"

That's why I am asking. I've never done this where the paths had a large difference in length. Maybe someone else has.

If this was my network I would use some Adtran TA5000s to get the job done. Their problem is they don't have any real budget for the CapX on the gear until next year.

Even at the 15k distance, would latency still not be less than 1ms? Its not like you are talking something being 2500 feet and something being 25 miles. But again as was already said it may work fine and it may not, but IMHO you could always try it, then if it doesn't work do like cramer said and make it 2 separate links, and have STP keep the secondary one on standby that only comes up in the event the primary goes down, yes you would have a small blip but should be 15 seconds max depending on your stp timers. Either way they get a backup.

The question is what is the hashing mechanism used on the devices at either side? Is it flow based or per-packet?

If its flow based, then read no further and off you go, since all packets from any individual flow will always use the same bundle member, which will rule out any possibility for out-of-sequence packets.

If its per packet, then I would look at tieing up the paths from each geographical route in to the same bundle so that there are as minimal delays as possible.

But... even at the distances quoted, with some quick and dirty math it looks like the propagation latency will be about 4µs for the short path and about 27µs for the longer path. A 1000 byte packet will have a serialisation delay of 8000ns, or 8µs (since the serialisation delay per bit for gigabit ethernet is 1ns - we'll say that for evensies, but it will likely be a bit quicker when encoding and other overheads are taken in to account.) So lets say a maximum of 35µs for a packet to optically travel from one site to another (and thats assuming c=200,000,000m/s, since light only travels at its full speed in a vacuum, and optical fibre is not a vacuum.)

Most VoIP protocols are probably only sending packets at a frequency of about 20ms, so even if you are per-packet load balancing, the jitter is likely to be unrecognisable to the end points.