Topic 'Re: [HELP] Need help determining errors' in forum 'Cisco' - dslreports.com

Re: [HELP] Need help determining errors

Fri, 21 Dec 2012 03:24:52 EDT

HELLFIRE posted :

no it wasn't down or having issues at the time

Which, the router itself or the Ser0/0 interface?

If Ser0/0, then IMO that possibly rules out a physical layer issue, at least from your router's
perspective. Another dumb question, a) is Ser0/0 a xWIC card of any sort, and if so what is the
part number, and b) is there any equipment between router Ser0/0 towards the smartjack / demark
besides the cable itself?

I'm wondering if something like the gear in the first row of image results is present or not, or is
it a straight cable shot.

said by bbwithdrawls:

we looked at the one cable from the router to smart jack and replaced it also ,

Another thought is whether any sort of cable test was done on the cabling itself. How long is the cable, just
out of curiousity?

Regards

Re: [HELP] Need help determining errors

Thu, 20 Dec 2012 07:27:46 EDT

bbwithdrawls posted : ran head to head form here at the house and no it wasn't down or having issues at the time , my router was up and online , the tech rewired the stuff they were responsible for and we looked at the one cable from the router to smart jack and replaced it also ,

Re: [HELP] Need help determining errors

Thu, 20 Dec 2012 07:25:07 EDT

bbwithdrawls posted : that was exactly what i told them to do , they kept coming back to the router so I told them to provide it to discount it if they felt that strongly about it ,I already had 2 to test that theory and did , you knwo ATT nor ACC wanted to provide one

Re: [HELP] Need help determining errors

Wed, 19 Dec 2012 12:45:32 EDT

HELLFIRE posted : Where'd they run the head to head from, and was it running while the circuit was down?
Also, what was the status of the physical interface of your router during this time?

You MAY be able to request the TBIRD gear be left onsite for extended testing -- mind you, I hear these are
several dollar figures' worth of gear to be leaving in an unsecured area, so ATT may not go for it -- but given the
history and fingerpointing, may be the only way in figuring out if it's the demark onwards is the issue, or
demark back to your gear.

Did you get your inhouse wiring checked professionally as well?

Regards

Re: [HELP] Need help determining errors

Wed, 19 Dec 2012 01:49:14 EDT

cramer posted : Check the logs/counters in *all* 13 repeaters
(old college problem) stop propping wet mops against the demark

Re: [HELP] Need help determining errors

Wed, 19 Dec 2012 00:09:48 EDT

nosx posted : few options:
switch access type to a non-T1 serial TDM connection.
order a managed CPE device so ATT techs can argue with ATT techs when there are issues
order service from a different provider with different access
completely reprovision the circuit on new access to a different CO etc.

Re: [HELP] Need help determining errors

Wed, 19 Dec 2012 00:03:41 EDT

bbwithdrawls posted : well we are still at it , I have pushed and escalated and we are now chronic, they tried Friday to do intrusive testing but couldnt , then took it down later that day right in the middle of a webex , went down sunday all day and again this morning for about 5 hours , the tech came out monday and he is about as frustrated as i am , he replaced the smart jack , enclosure ran a new cable ,they replaced a repeater at the CO, they ran head to head with a tberd , ran quasi and 1's and 0's and all good and it went down again today , i hadn't received why yet , i did get that they saw an issue at the CO and they dispatched

I,m watching my counters , still seeing crc and frame errors in my logs , ACC sees it as a problem on the network , ATT i always get some guy with a harsh accent telling me its my equipment , i just don't see how , i have 2 identical routers and the config copied from one to the other , been switching them out to see if its my equipment , still happening ,

anyone got any suggestions , it is labeled chronic , have a team at ACC working on it ,monitoring but still going down , the last 2 weeks have been frustrating to say the least

thanks in advance

Re: [HELP] Need help determining errors

Thu, 13 Dec 2012 18:11:35 EDT

HELLFIRE posted : ...best of luck iwht it all bbwithdrawls, let us know how it goes.

Regards

Re: [HELP] Need help determining errors

Thu, 13 Dec 2012 07:03:19 EDT

bbwithdrawls posted : yep there's a few , circuit went down again at 900 am yesterday , they dispatched very quickly and they couldnt find anything , it was same guys from Monday and they told me they found a squirrel nest at one point and the wire was crimped and another break further down the line , couldn't find anything on yesterday drop though , of course ATT came back with its my equipment

still showing the errors in my logs , the only equipment is my router and a cable to smart jack, I have 2 router exactly the same , I can switch them out and get same errors, i did change the cable last week when this started based on logs it show ser int out , nothing to do with anything on my internal side

still pushing to get the intrusive testing done and working with a manager

Ser 0/0

Re: [HELP] Need help determining errors

Wed, 12 Dec 2012 23:09:34 EDT

battleop posted : "there's only 13 of them"

You must be way out in the boonies... :)

Re: [HELP] Need help determining errors

Wed, 12 Dec 2012 23:05:40 EDT

battleop posted : If ACC is your provider then you need to beat them up and not AT&T even though AT&T owns ACC.

Re: [HELP] Need help determining errors

Tue, 11 Dec 2012 16:42:44 EDT

cramer posted :

said by bbwithdrawls:

I'm curious as to what it was , never seen a circuit up but not pass any web pages, did pass voice ,very strange , I'm going to go ahead and have them complete the testing

*sigh* I have. Bottom line, voice is a completely different bit pattern.

Re: [HELP] Need help determining errors

Tue, 11 Dec 2012 12:28:10 EDT

bbwithdrawls posted : I would agree , as I had to pack up ad move to high speed internet 2 days in a row, unfortunaltly ATT is the only choice ,I wish i had other options options but there are none ,

take a good look at the avatar

Re: [HELP] Need help determining errors

Tue, 11 Dec 2012 08:18:51 EDT

aryoba posted :

said by bbwithdrawls:

the only good thing is because of my SLA I will not have to pay next months bill

In some businesses, simply free bill for a month is insufficient since the revenue you have lost during outage sometimes is larger loss.

I would assume in your situation/area, AT&T was the only provider so that you don't have much of a choice.

Re: [HELP] Need help determining errors

Mon, 10 Dec 2012 23:24:23 EDT

bbwithdrawls posted : well went down again at 1100 am , a tech did show , he started at the house and after talking to him was going to check the route , and about 5 pm I was notified they did find a issue at one of the repeaters ( there's only 13 of them ) and were waiting on a cable truck dispatch , about 9pm it was back up , so far knock on wood

thank you all for your advice , I had escalated last week and was working with a manager and they were pushing all the buttons the whole time , , ATT was just not in a hurry

they did schedule intrusive testing friday night and it failed ,

I'm curious as to what it was , never seen a circuit up but not pass any web pages, did pass voice ,very strange , I'm going to go ahead and have them complete the testing

the only good thing is because of my SLA I will not have to pay next months bill

again thanks all for the advice

Re: [HELP] Need help determining errors

Mon, 10 Dec 2012 20:13:14 EDT

HELLFIRE posted : Much as I hate doing this to Cisco gear, schedule a reboot of the chassis and reseat / replacement of the WIC card
to rule out hardware, as another thought.

Regards

Re: [HELP] Need help determining errors

Mon, 10 Dec 2012 20:12:00 EDT

HELLFIRE posted : I agree with what's been said by others on the provider / ticket end
- open tickets
- open chronic tickets
- escalate to the duty manager
- escalate with your account team
- DOCUMENT what has (not) been done during the histories of the previous / chronic tickets -- that's usually enough to
light a fire under the account mgr's butt if it's clear ppl are waiting for the issue to clear on their own and not doing
anything to fix it while it's down.

From a technical end
- looping the smartjack and intrusive testing (24hour minimum) are basic things the telco can do
- get them to pull historical PMs
- checking and testing your own internal wiring's always a good idea
- is this a layer 2 circuit or layer 3 circuit? Run your own extending pings with the option of specifying the ping
pattern (data pattern, IIRC) -- not quite good as a TBIRD tester, but close enough.

Based on the output, its errors on the INCOMING direction (from the telco), CRC and frame tells me its coming
from the layer 1 and 2 (physical and logical) side of things.

My 00000010bits

Regards

Re: [HELP] Need help determining errors

Mon, 10 Dec 2012 10:33:51 EDT

bbwithdrawls posted : well when it rains it pours
down again all day Sunday since 700 am , ATT played around all day long and then refused to dispatch citing safety reasons and dark ,

circuit bounced all day Sunday , then show up and be unusable , just started working on its own about 1.5 hours ago , now they will not dispatch again saying circuit is working , working with my provider , there definatly is a problem , I think I've done all on my end i can do , I have 2 routers configured the same , both have the same issue , changed out all cables to the card, took out my internal network equipment and plugged into the router with a static ip and the same thing its unusable , it pings but tracert to att dns or google dns times out , web pages time out , just spin from all computers , whats weird is my VOIP works intermittently but it only requires 16 kbps for a call

I'm at my wits end with them , my provider keeps escalating but nothing moving quick

Re: [HELP] Need help determining errors

Sat, 08 Dec 2012 23:28:17 EDT

battleop posted : These are circuits that are between an Adtran TA900 and the customer's PBX that replaced T1s that were once point to point T1s back into another carrier's 5ESS. It took some time to figure out this combination because the original phone vendor was no longer around and it took trial and error to discover this because the circuits would take errors and slips.

There is no reason why this would not work but there is no reason to do it unless there is some limitations on old equipment. Most of the PBX's that were installed with this configuration were put into service in the early 90's when PRIs and PRI cards were considerably more expensive than T1's. None of these PBX's can utilize Caller ID name even with a PRI and only a few of them will take Caller ID number using Feature Group D.
--
I do not, have not, and will not work for AT&T/Comcast/Verizon/Charter or similar sized company.

Re: [HELP] Need help determining errors

Sat, 08 Dec 2012 21:25:11 EDT

nosx posted : You would have to be going through one weird dacs to be running anything other than esf/b8zs on a data-only T1 in 2012. Please post some show controler t1 output for us to argue over =)

Re: [HELP] Need help determining errors

Sat, 08 Dec 2012 19:42:12 EDT

battleop posted : D4/B8ZS is rare but valid. I have a customer with about a dozen Voice T1s into ancient Mitel PBXs that are configured for D4/B8ZS. The circuit will work if your DF/EFS or AMI/B8ZS is in correct but you will end up with slips and strange, random trouble tickets that will make you pull your hair out.
--
I do not, have not, and will not work for AT&T/Comcast/Verizon/Charter or similar sized company.

Re: [HELP] Need help determining errors

Sat, 08 Dec 2012 19:32:27 EDT

battleop posted : First open at ticket with ACC and insist that they do some intrusive testing. If they do non intrusive testing and just monitor the circuit they won't see the errors that are headed to you, just errors headed to them. An intrusive test will loop the smart jack and they should then see errors headed back to them.

If this happens often I can't stress how important it is to keep detailed records of your trouble tickets. If you can get 3 tickets withing a 30 day period you can ask them to open a chronic ticket which will usually get some attention.

They are probably going to tell you they found a bad pair and will move you to the next pair so that your bad pair will eventually become someone else's bad pair. :)
--
I do not, have not, and will not work for AT&T/Comcast/Verizon/Charter or similar sized company.

Re: [HELP] Need help determining errors

Sat, 08 Dec 2012 15:35:42 EDT

cramer posted : The problem with techs... they're kind of lazy. They'll run all 1's and all 0's and declare a circuit "not broken". I would mark it down to "lack of training", but these people have been doing this job long enough to know how to use the other 2 dozen tests the T-Berd can run. Repeat after me children: Two to the twenty one quasi-random!

The more hops in the circuit, the more places there are for someone to have a jumper/dipswitch wrong. The only way to find it is to test from *every* junction. To both ends of the circuit.

(I had one case where jpg's wouldn't go across the circuit: one of the wires was broken... a 3 wire T1 won't do! The next was a line card set for D4(SF) in the path: I'm not sure how that ever worked as D4/B8ZS isn't a valid configuration.)

Re: [HELP] Need help determining errors

Sat, 08 Dec 2012 13:16:30 EDT

DaSneaky1D posted : Your T1 is taking errors. Have AT&T monitor it for 24 hours. If they don't find anything, open another TT. If they don't find anything, open another TT.

You're having issues that they need to help isolate. They'll be able to tell if the errors are at the smartjack or CSU/DSU.

Re: [HELP] Need help determining errors

Sat, 08 Dec 2012 10:45:41 EDT

nosx posted : First of all, be a pain. Keep complaining, opening tickets, escalating tickets, etc. Its the only way to get attention from big telco's. Companies with round the clock operational staff (especially in APAC) can afford to keep the telco techs on the line until the issue is resolved, making it costly enough they actually fix the root cause as opposed to ignoring it and hoping it goes away.

Second, if you are seeing the issue and the guys on the other side arent, check your local cabling between you and the smartjack/xjack. Get a physical T1 loop made (easy to make with wires and crimper). Use a female-to-female adapter to put it on the end of your cable facing your router at the jack and run patterns to it, 1's, 0's, alternating 10101010 etc. Make sure your house is clean before pointing fingers =P

Then, over a weekend or something, if you can tolerate the lack of connectivity, plug the physical loop plug into the jack and make the carrier run patterns to it. Get them to validate a million 1's, 0's, etc.

Check your show controller t1 blah output for more information and times. Figure out if the issue is load related, a stressed repeater somewhere down the line, etc. If all else fails and you are ready to give up, re-order the circuit and cancel the old one. See if the new one is on a better path. Reprovisioning on a new path is a lame fix but can resolve the problem when all other efforts fail.

[HELP] Need help determining errors

Sat, 08 Dec 2012 09:48:47 EDT

bbwithdrawls posted : Hello all , been back and forth with this one , first off I am in a rural area and this line has been painful for the 3 yrs I've had it , comes with the territory I guess

I don't mind calling out names at all , I have ACC business , they've been pretty good to deal with , however ATT has been a pain , getting them to respond is an issue and getting them to dispatch is an issue, they try not to , I'm not badmouthing the techs at all , they are overworked , underpaid , if this happens after hours or weekend they have to come halfway across the state because one guy services 1/2 the state

anyway last round of issues has been different , I can watch the circuit drop , bounce it on my end see it back up , and then it is unusable for a couple hours . 4-5 hours , the circuit shows up and ATT refuses to look any further , I can ping internet sites but drop tracert, web pages time out , my voip works spotty during this whole time , basically for this time period it is unusable , Att's response is check my equipment ,they see the circuit up and thats it

I have 2 routers configured exactly the same ,both exactly the same, I have changed cables on my side , still happening , its not a config issue because it works most of the time ,

Im watching my logs and I see the output above constantly , I can clear the logs and 8 hours later were right back here from what I'm interpreting this is from the serial interface out , nothing to do with anything internal to my network ?? I have ACC pulling the logs on the other side and they say look clean , I need to know where to look on this or better yet be able to say with some confidence its not my problem and force them to look at it ,

let me know your thoughts

thanks

SER 0/0