Cisco → Re: One app didn't work over Frame Relay; worked over ISDN

Here is the update.

We have been working with the telco to solve the problem. This time we asked telco to recheck their previous PVC rebuilt work.

The telco engineer hyphotized that the previous PVC rebuild was probably only between the frame relay switches. Then the engineer decided to rebuild the PVC between the switches plus he completely removed the logical ports before adding them back in.

Since the 2nd PVC rebuild, it has been two days that the problematic http application not as "problematic" anymore. Since the 2nd rebuild, site A kept using the Frame Relay link and never faced the http payload problem. So far, the 2nd PVC rebuild seemed to cure the problem.

Here is my question. Let us say that the improper PVC rebuild was the culprit. How could this situation only affect ONE SPECIFIC HTTP APPLICATION and no other?

Anybody has a thought?

There may have been a bit pattern in the specific packet that disappeared in the Frame cloud that caused an issue at Layer 1/2 based on the previous PVC config. I am not pointy-headed enough in that arena to even begin to take a stab at it what it could have been. All I know is the whole reason for pattern substitution like B8ZS at Layer 1 is based on the physical issues with the circuit signaling. There are other things in the Layer 1 facilities that react to specific bit patterns, as well and those must be avoided lest you have strange problems like this occur. I have no idea at Layer 2 though what is going on in that Frame cloud.

See my post on 3/23.
Looks like the telco cloud was dropping some of your data, maybe due to policing, which is not indicated on any of the counters in the routers because your physical and data link hook-up to the telco CO was/is doing fine, AND the application was receiving an incomplete response based on framerelay not retransmitting those dropped packets. It doesn't happen that often but I've seen more than my share of those.

Then again, that's all i can provide from several hundred -or thousand- miles away and based on the symptoms described on the beginning of the thread.

Take it or leave it. I usually don't answer why questions except to a very few selected group.

Keep in mind that this is problematic http payload transaction (layer 7 problem), not just layer 2 or layer 3 problem.

From my understanding; if the problem is the layer 1, then it must affect ALL application and not just one. It is like layer 1 problem directly causes layer 7 problem without affecting layer 2 to layer 6 at all.

What I don't understand is how improper PVC rebuild in the frame relay cloud might only affect one specific http application?

In addition, other http applications were not affected; just one specific http application.

Weird?

As rovernet put it, it is very hard to work out what went wrong remotely but my money is on a software issue, not a physical issue as the same rebuild but this time including the logical ports would follow the same physical path as the previous PVC with the only difference being the "logical" or "software path". Seen loads of those for ATM PVCs but rarely seen "stale" PVCs on frame relay ccts but then again, it depends on the hardware platform that is being used with not all FR switches created equally.

What I tell my team is whenever you have anything unusual going on, or something that can't be explained by logic or RFCs/IEEE standards, then look into the possibility of it being a software issue cos nothing doesn't make sense like a bug.

Yes, I quite agree Covenant.

However if it is a software issue, then we must have the problem from the very beginning. The application has been stable for years. The problem started since telco rebuilt the PVC.

How could the application become so picky when others had no complains?

I guess we will never know as happens quite often with logical paths. I do not know the platform that your FR network is built on but in the same sentiment we can ask why do we get stale dialers, or ATM PVCs which get frozen in Cisco routers and the only way round it is to reconstruct it with no documented bugs to explain it. Take into account that most of the core infrastructure would not have been reloaded in a very long time so things don't quite act as expected more often. I guess what I am saying is that with software, somethings unfortunately can't be explained. I guess that is not what you want for your inquisitive mind but it is one of those things that you use as "experience". Glad it got sorted in the end but I would have been interested to see the result of applying ip tcp adjust-mss 1300 on the routers as I had an issue similar to what you describe over the MPLS network and short of rebuilding the ATM PVC and causing an outage, I applied that command and the application started working again. Just a thought.