dslreports logo
site
spacer

spacer
 
    «« DSL Hurdles Share Tool
spc

spacer




how-to block ads




60.0 Troubleshooting

Suggested prerequisite reading:
»Cisco Forum FAQ »Deleted/Corrupted/Wrong Type/No IOS image and router won't boot

Have you tried to upgrade an image but were unsuccessful? The router is stuck in rommon mode? If yes to any one of those questions; it means that the router cannot find a valid image in the flash memory. The image usually gets corrupted or lost.

Try the following link to recover the problem. The URL should work on any router that has tftpdnld command in the rommon.

Routers ROM Monitor IOS Image Download Procedures

The whole idea is to utilize tftpdnld ROMMON mode command to download working IOS image from your TFTP server to the router. To recover a router using this method, there are assumptions as follow

* You have a valid working IOS image
* You have a working TFTP server
* The working IOS image is stored in the TFTP server

When you don't have TFTP server yet, check out the following FAQ for preparing the server
»Cisco Forum FAQ »How to prepare TFTP server

If you don't have a working IOS image file around, then you should be able to download one from Cisco website; assuming you have Smartnet contract. Check out the following FAQ for more info on Smartnet
»Cisco Forum FAQ »How/where do I get Cisco images (such as IOS, PIX/ASA OS)?

As an illustration on utilizing the tftpdnld command, check out this thread.

»[HELP] ROMmon corrupted

by aryoba See Profile
last modified: 2009-05-19 10:24:55

»Cisco 2960 Switch not booting all the way
»CISCO Catalyst 2948G-L3 no keystrokes in console

by aryoba See Profile
last modified: 2013-04-10 16:18:11

Get LANCE error message? The Ethernet controller might be the problem

Do you have error message concerning LANCE? If so, you might have problem with the ethernet controller.
%LANCE-1-INITFAIL: Unit [dec], initialization timeout failure, csr[dec]=[hex]
The hardware has failed to initialize correctly.

Recommended Action: Repair or replace the controller.

On modular equipment, the ethernet controller is on the module. Whenever possible, try to remove the module concerned; just to ascertain whether the device goes through the bootup process with no issues. If that is the case, you might want to replace the module.

FYI, the LANCE error message is from the Ethernet controller. The following thread provides details.
»Initialization timeout failure - fix controller?

Other Hardware Issues
»[HELP] ALIGN-3-SPURIOUS Error
»Verizon DSL and Cisco 678
»[HELP] Cisco 7604
»[Info] Cisco 877 and 887 switch port lights..

by nozero See Profile edited by aryoba See Profile
last modified: 2013-12-17 14:12:40

ASA/PIX Firewall hangs and won't fully boot up
»Cisco PIX hang and wont reload
»Cisco ASA 5520 Error Code: -5

ASA/PIX Firewall does not boot up at all
»[H/W] Cisco ASA 5505 - Issues booting up

by aryoba See Profile
last modified: 2014-02-18 21:22:00

»[CCNA] CCNA 3550 switch with a warning error: switch core BIST
»POST Failure Error-Catalyst 3500 XL

by aryoba See Profile
last modified: 2010-09-28 13:10:13

When one of the following situation occurs, your router "loses" its configuration

• power back on a router after power off for a while

• there is lightning strike around

• unprotected power outlet (i.e. no UPS)

• after configuring a router, you reload the router

• after installing new IOS image, you reload the router to let the new IOS image become active

• you implemented a wrong config register value

First step to take is to verify whether the router actually lost configuration. In order to do so, you might want to check the config register value to verify if it is 0x2102 (the default) or else. When the config register value is not set as 0x2102, then the router might act differently than the default.

Following is an example. Let's say the router config register value is set to 0x2142. By having such value, the router will never load saved configuration every time the router boots up from either power up, power cycle, or reload command. The router ignores the saved configuration and instead boots up using default (blank) configuration.

Such abnormal behavior is typically unwanted at any time. To avoid, you need to set the value back to the 0x2102.

To find out what the current router config register value is, you can simply do a "show version" from the CLI prompt and look at the last line.

To set the value back to 0x2102 (default value), one way of doing it is via the CLI with following commands
configure terminal
config-register 0x2102
end

If for some reason the router goes to ROMMON mode, you can issue the 0x2102 value with following command
confreg 0x2102

When the router has 0x2102 config register value, the router will act as factory recommended (Cisco recommended) including loading the current configuration after reload or power cycle.

Note:

When you are in luck, there is one possibility is that the router still has the configuration even though the router does not boot the configuration. This situation applies when the router configuration prior reload or prior power loss is saved (by issuing "copy running-config startup-config" or "write memory" prior).

Cisco documentation
Configuration Register Setting Descriptions

Feedback received on this FAQ entry:
  • Thank you. Saved my life

    2010-11-16 21:30:10

  • Spot On! Many Thanks.

    2009-11-03 14:48:01



by aryoba See Profile
last modified: 2013-10-18 08:32:10

Suggested prerequisite reading
»Cisco Forum FAQ »Config Register Value - router lost configuration, how to recover

The idea is to set the config-register value back to default, which is 0x2102. Note that certain config-register values not just stop the router to boot up normally but also change the baud speed of the CONSOLE port which makes the CONSOLE port output is unreadable or even does not show at all. Therefore to fix the issue, you need to do the following

1. Make sure the CONSOLE port output become readable
2. Issue the 0x2102 config-register value, either via CLI prompt or ROMMON prompt

Following is an illustration.

If you are directly consoled into the router, open a new HyperTerminal session (or any preferred terminal simulator) with these settings:

Bits per second: 1200
Data bits: 8
Parity: None
Stop Bits: 1
Flow control: None

Once this is open, power cycle the router and press the spacebar for roughly 10 seconds (press and release). If you are able to see the rommon prompt, change the register back to normal by typing:

rommon>confreg 0x2102
rommon>reset

if you are not able to see anything on the screen, close the window and open a new one with these settings:

Bits per second: 9600
Data bits: 8
Parity: None
Stop Bits: 1
Flow control: None

And you should now see the rommon> prompt. Change the register from there and the router should be back in normal mode.

Side Notes:

* If you know the current setting of your router's config register value is, you can use the Config Register calculator to find out how the router is configured when it boots. You can download the calculator from this site
* By any chance the router model is 2600 or 3600 series, check out the following FAQ
»Cisco Forum FAQ »Unreadable output from Cisco Router Console




Derived from this thread.

Cisco documentation
Configuration Register Setting Descriptions

Discussions

»[HELP] Cisco 1800 garbled output in putty

Feedback received on this FAQ entry:
  • Peeeerrrffeecctt bro!!!!!.. THaaankss!!!!

    2012-11-02 16:38:21



by Covenant See Profile edited by aryoba See Profile
last modified: 2012-04-09 09:40:15

Password Recovery Procedures for most Cisco appliances

The following link is to the index of password recovery procedures for most Cisco appliances. Go to next discussion for IronPort Password Recovery procedure.

Note: For security reasons, the password recovery procedures described there require physical access to the equipment.

Password Recovery Procedures

Properly Sending BREAK key

You need to enter ROMMON mode which may require you to send the BREAK key. Check out the following link to find the suitable BREAK key.

Standard Break Key Sequence Combinations During Password Recovery



Tips

* For newer machines running Windows XP, Vista, or Windows 7 using Hyperterminal with no BREAK key available on the keyboard, try to use PAUSE key instead. Therefore instead of issuing CTRL-BREAK combination key, try to issue CTRL-PAUSE combination key instead.




It is highly recommended to use a computer that has an actual Serial port (RS-232) when sending BREAK key. Whenever possible, don't use computer that utilize USB port to emulate Serial port since the BREAK key might not be sent properly.

If the router password recovery functionality is disabled, then you can perform tasks provided on this link to re-enable the password recovery functionality.

To Disable and Re-enable Password Recovery Functionality

NOTE:
The Cisco documentation shows copying saved configuration to running configuration as part of the recovery procedure. In some (if not most) cases, this step is a no-no.

Password recovery procedure is necessary when the password to log into the device is unknown and something in the configuration blocks administrative (enable) mode to activate. When you copy saved configuration (that has been stopping you to enter enable mode) to running configuration, then basically you are going back to the point before you make any password recovery attempt, which will nullify the password recovery work you have done.

When this is the case, then there should be no copying saved configuration to running configuration as part of the recovery procedure. Instead put enough configuration (or no configuration at all) to the device just to keep it accessible and do the things that can pass traffic.

Some discussion:
»[HELP] 1811W forgot login info

IronPort Password Recovery Procedures

Please follow the steps below in case you have lost your "admin" password on your IronPort appliance.

* Contact Customer Support for a temporary password. You will need the Serial Number of the device.
* Once you receive the temporary password, please access the IronPort via serial connection.
* Log in as user 'adminpassword'.
* Enter the temporary password you received from the Customer Support Engineer and hit return.
* Enter the new password that will be used for the 'admin' user.

Some discussion
»Password Recovery Procedure for Cisco Ironport Mail Gateway?

Cisco equipment's default password

Access Point
username: cisco
password: Cisco
(case sensitive)

by nozero See Profile edited by aryoba See Profile
last modified: 2013-01-25 09:28:05

Related FAQ
»Cisco Forum FAQ »Used wrong config-register and now the router does not boot!

Discussions
»[Config] Cisco 871w - All Baud Rates result in Gibberish

by aryoba See Profile
last modified: 2013-01-22 15:22:34

MTU Size regarding PPPoE over ATM/DSL

This FAQ provides a guaranteed working config for anyone using any ADSL PPPoE connectivity types such as Ameritech ADSL for their circuit provider and Megapath.net for ISP. It took 2 calls 2 cisco and weeks of fighting with ISP tech support, but I learned a valuable lesson about ADSL PPPoE specifically.

The MTU on the dialer interface should be 1492 as PPPoE adds an 8 byte encapsulation header. The key is setting ip tcp adjust-mss 1440 on the inside ethernet interface. You will find many different suggestions and recommendations out there. Some will say 1492 or 1460. Some will even say 1452. 1452 MSS is pretty much the standard for DSL with a PPPoE transport. Normal MSS is 1500 bytes. But you have to account for the 40 byte IP header and the 8 byte PPPoe header. That gets you to 1452.

Following is from the mouth of Cisco, "If you have ADSL running PPPoE and run into problems resolving DNS, adjust your MTU on your ethernet interface using the command ip tcp adjust-mss 1452. This is because PPPoE requires more bits in the header packet than any other type of circuit."

The last bit of optimization is a little more subtle and is a debatable topic. As the PPPoE traffic is carried over ATM cells, it has to be chopped up before it can be transmitted. ATM cells are 53 bytes long and have a 5 byte header. So a total of 48 bytes of payload. If you were to take 1452 bytes of data and split it up across 48 byte payloads. You would come up with 30.25 cells. The .25 is a 12 byte remainder that would have to be sent in a separate ATM cell. ATM cells are always 53 bytes. So the payload would have to be stuffed with an additional 36 bytes of null data for that last chunk. So to be completely optimized you would set the MSS to 1440 to eliminate those wasted 36 bytes.

Adjusting MTU size was news to me, but the minute we did it all my problems were fixed. Feel free to experiment to set MTU size to either 1452 or 1440 to see which size brings you the most suitable result.

MTU Discussion
»Best IOS for 1801W

So with that in mind, here is a 100% working config from my 827 ADSL router. Hope this lesson I learned helps someone out in the future!!!!

SANCH_INT_RTR#sh run
Building configuration...

Current configuration : 2593 bytes
!
version 12.1
no service single-slot-reload-enable
no service pad
service timestamps debug uptime
service timestamps log uptime
service password-encryption
!
hostname SANCH_INT_RTR
!
logging rate-limit console 10 except errors
no logging console
enable secret 5 $encrypted password$
!
username Cisco privilege 15 password 7
username Router password 7
ip subnet-zero
no ip finger
ip name-server 66.80.130.23
ip name-server 66.80.131.5
!
no ip dhcp-client network-discovery
vpdn enable
no vpdn logging
!
vpdn-group pppoe
request-dialin
protocol pppoe
!
!
!
!
interface Ethernet0
ip address 69.33.X.X 255.255.255.224
ip tcp adjust-mss 1452
no ip mroute-cache
!
interface ATM0
no ip address
ip access-group 101 in
ip access-group 101 out
no ip mroute-cache
no atm ilmi-keepalive
bundle-enable
dsl operating-mode auto
hold-queue 224 in
!
interface ATM0.1 point-to-point
pvc 0/35
protocol pppoe
pppoe-client dial-pool-number 1
!
!
interface Dialer0
no ip address
no cdp enable
!
interface Dialer1
mtu 1492
ip address 69.33.XX.XX 255.255.255.0
encapsulation ppp
dialer pool 1
dialer-group 1
ppp authentication pap chap callin
ppp chap hostname ppp-username@megapathdsl.net
ppp chap password 7 Encrypted password
ppp pap sent-username ppp-username@megapathdsl.net password 7 encrypted password 0A
!
ip classless
ip route 0.0.0.0 0.0.0.0 69.33.X.X
no ip http server
!
access-list 101 deny tcp any any eq 135
access-list 101 deny tcp any any eq 136
access-list 101 deny tcp any any eq 137
access-list 101 deny tcp any any eq 138
access-list 101 deny tcp any any eq 139
access-list 101 deny tcp any any eq 140
access-list 101 deny udp any any eq 135
access-list 101 deny udp any any eq 136
access-list 101 deny udp any any eq netbios-ns
access-list 101 deny udp any any eq netbios-dgm
access-list 101 deny udp any any eq netbios-ss
access-list 101 deny udp any any eq 140
access-list 101 deny tcp any any eq 445
access-list 101 deny udp any any eq tftp
access-list 101 deny tcp any any eq 4444
access-list 101 deny tcp any any eq 593
access-list 101 permit tcp any any eq www
access-list 101 permit tcp any any eq 1723
access-list 101 permit tcp any any eq 110
access-list 101 permit tcp any any eq 25
access-list 101 permit gre any any
access-list 101 permit icmp any any
dialer-list 1 protocol ip permit
banner login ^CC^C
!
line con 0
password 7 XXXXXXXXX
transport input none
stopbits 1
line vty 0 4
exec-timeout 30 0
password 7 XXXXXXXXX
login
length 0
!
scheduler max-task-time 5000
end

This FAQ created using this post by sanchito75 See Profile and the naming suggested by Covenant See Profile.

Feedback received on this FAQ entry:
  • Great write up..old but gold! :) Kumar

    2013-10-06 22:12:08

  • Don't normally post, but thank you so much for this information. It resolved an issue with SIP not working on VVX 1500 phones. Other models worked just fine without changing the mtu.

    2013-05-08 16:52:48

  • Thank you very much for this explanation ;-)

    2011-08-03 03:20:45

  • I'm so glad I found this site! I'm having issues all weekend hooking up my new Cisco 877. Bad DNS and inability to connect to HTTPS and IMAPS servers. I never set my MSS adjust. Thank you so much!!!!! John

    2011-02-13 13:19:40



by nozero See Profile edited by aryoba See Profile
last modified: 2013-01-25 16:42:36

Question - I just got a new Cisco router and the Cisco Router Web Interface (= CRWS) just hangs when I try to start it What do I do?

Introduction
CRWS is one of two GUI interfaces for the 800 and SOHO series routers (the other being SDM). It resides in a section of the routers flash memory called webflash. When you want to access CRWS, you open an Internet browser window and type "http://10.10.10.1" in the URL address window (similar to open up Yahoo! website by typing "http://www.yahoo.com").

Note:
If accessing the "http://10.10.10.1" does not show anything but bunch of error messages in the page, there is a possibility that the router LAN IP address has changed. To verify and troubleshoot further, there is no other choice but to use CLI. Check out the following FAQ to revive CRWS/SDM via CLI.

»Cisco Forum FAQ »My SDM/CRWS (web configuration mode) doen't work. How do I revive it?

IF CRWS Hangs/Does Not Launch when opening up using Internet Explorer
A common cause of CRWS not working is a known bug in CRWS in which Microsoft Virtual Machine is required for it to run, the subject of this FAQ.

Microsoft Virtual Machine Configuration/Installation
Figure 1(»/showpic/faqs?···&1=1&1=1) Open Internet Explorer and go tools\internet options
Figure 2(»/showpic/faqs?···&1=1&1=1) In the Internet options window, click the advanced tab.
Figure 3(»/showpic/faqs?···&1=1&1=1) Then scroll down and see if there is a main heading for Microsoft VM (if there is no Microsoft VM heading, skip to figure 4). If the Microsoft VM main heading is there, under this heading check the box for JIT compiler for Virtual Machine enabled. Then go to the main heading above it entitled Java (Sun) and uncheck all the checkboxes there. Then click apply, exit out of the window and reboot your PC to save the settings
Figure4 (»/showpic/faqs?···er=0&1=1) - If there is no main heading for Microsoft VM you don't have it, you need this file filename msjavx86.exe - from a trusted website One website is »java-virtual-machine.net/download.html. Download it and (after checking for viruses as you would do with any download, right?) install it (it will ask you to reboot, do so). Then configure Internet Explorer as explained previously in figure 3 above
Figure 5 (»/showpic/faqs?···&1=1&1=1) Go to Windows update to update the Microsoft VM.
Figure 6 (»/showpic/faqs?···&1=1&1=1) Select the VM update(s) and install them, reboot if requested.
Figure 7 (»/showpic/faqs?···&1=1&1=1) Open Internet Explorer, type "http://10.10.10.1", hit enter and CRWS should properly start now.

Note on CRWS usage:
1. leave the LAN IP address at 10.10.10.1, (see figure 7) as changing it can create problems for the router.
2. CRWS allows basic router functionality. It does not allow you to do everthing the router is capable of. Learning the Command Line Interface (CLI) is necessary to make use of all the routers features.

Useful Links:
CRWS demo at Cisco website »www.cisco.com/warp/public/779/sm···crws.htm
CRWS description at Cisco website
»www.cisco.com/en/US/products/sw/···dex.html
CRWS User Guide
»www.cisco.com/univercd/cc/td/doc···ws30.htm
CRWS software downloads (includes CRWS caveats document, Cisco-speak for bugs)
»www.cisco.com/pcgi-bin/tablebuild.pl/crws
Switching Between SDM and CRWS
»www.cisco.com/en/US/products/sw/···cc8.html

by Requiems See Profile edited by aryoba See Profile
last modified: 2007-07-08 08:10:56

Suggested prerequisite reading:

»Cisco Forum FAQ »The most straight-forward way to configure Cisco router: Introduction to CLI

There are various reasons why the SDM does not work. Some of them are the following.

* old Java issue
* SDM is not activated
* SDM software does not exist on the router
* SDM software is corrupted
* The LAN interface IP address is changed
* The LAN interface is shutdown

This FAQ is not meant to be the complete SDM troubleshooting guide. This FAQ however points you to the right direction to find out what the cause is and to revive your router.

When the web configuration mode doesn't work, use the CLI (Command Line Interface) as the most reliable way to configure and troubleshoot routers, including troubleshooting SDM access issue.

Here are the steps to revive inactivated SDM.

1. Do a "show running-config" from enable (privilege) mode and check if there are such commands of "no ip http server" and/or "no ip http secure-server"

2. When you find it, it means the router is currently configured to disable the SDM/CRWS. To enable it, issue "ip http server" and/or "ip http secure-server" from global configuration mode

3. When there is no IP address under the LAN interface, you need to assign one. Make sure that the LAN interface IP address and your PC IP address are within the same subnet

4. Using your web browser on your PC, open "http://[YOUR ROUTER LAN IP ADDRESS]" and see if you are able accessing the SDM/CRWS

The following thread shows some walk through using CLI step-by-step from very beginning to revive the web configuration feature.

Keep in mind that even though your product may not be an 837 router as is used in the thread example, the description on CLI introduction still applies to any Cisco router that supports web configuration.

»[Config] cisco 837 defaults

by aryoba See Profile
last modified: 2009-04-06 15:42:40

If you use a KVM switch, then there might be something wrong with the switch. Test by removing the switch from the scenario. Check out the following thread for more info.

»[CCNA] Bootup Error occuers on Cisco 2514 Router

by aryoba See Profile
last modified: 2009-04-01 11:42:40

Scenario 1: Remote Users are able to ping

The remote user is able to receive IP address off DHCP pool. Users at both sites are able to ping (ICMP echo and echo reply) each other's IP address.

You are using Microsoft network. One task is to have remote users try to map share drive and it fails. What is the problem?

1st of all, let's break down the problem. When the remote user is able to receive the IP address and are able to ping each other's IP address, then the VPN tunnel must be up. The fact that remote user is unable to map share drive is then not the VPN setup problem, but something on your Microsoft network is preventing the drive sharing.

Check out the following official Cisco link for more info.
Troubleshooting Microsoft Network Neighborhood After Establishing a VPN Tunnel

Scenario 2: Remote Users are unable to ping

The remote user is able to receive IP address off DHCP pool. Users at both sites are unable to ping (ICMP echo and echo reply) each other's IP address.

1st thing is to confirm that no firewall that blocks ICMP echo and echo reply. Once it is confirmed, then you might want to confirm IPSec VPN device configuration. If the VPN device is a PIX/ASA Firewall, then a isakmp nat-traversal command might be necessary to be in place. Check out the following FAQ for details.

»Cisco Forum FAQ »Configure router and ASA/PIX Firewall to support various VPN technologies

Some discussions
»[Config] ASA 5510 Firewall vpn not mapping drives
»[2K3] Mapping network drives After you VPN into network

by aryoba See Profile
last modified: 2010-06-16 11:05:02

Cisco GSS appliances

»[Info] Subtle GSS load balancing issue

by aryoba See Profile

Cable Internet (Coax)

»Interface errors on ASA 5505 to SMC DOCSIS modem

DSL

»[Config] Hmmm. interface ATM0 remains down, line protocol remains down
»Line Noise
»Cisco 678 what is the Alarm meaning
»[HELP] Strange interface issues with cisco 2611
»[Config] 2610 w/ ADSL WIC - Upgraded to 8Mbit - Poor Connectivit
»Cisco 678 as bridge & have d-link 514 do NAT/DHCP
»what is crc_errors(52) exceeded threshold 678
»Cisco 837 CRC / Header errors...

T1

»Errors and the Telco. How do I know who it is?
»Input Errors
»[HELP] Need help determining errors
»Multilink traffic stats seem goofy

Ethernet

»Ethernet port shows up/down with no cable attached
»[HELP] input errors question
»[H/W] Interface Transmit Errors
»[H/W] 3750 POE Model Line Noise
»[HELP] Cisco ASA5580-20 SSL VPN intermittent issues

For those who like to further testing cable run to ensure Layer-1 connectivity, check out the following discussions to find out a good cable tester.
»Cable testers
»Fluke Cable IQ Rental

by aryoba See Profile
last modified: 2014-11-17 07:56:38

Things that are worth to explore

1) check your interface between your gear and the ISP. Ensure the right
speed and duplex is present, and whether there are any errors / drops / etc.

2) check your interface between your gear to the LAN for the same thing.

3) if possible, use Fluke meter to check the cabling.

4) check the outputs of the following Cisco IOS commands:

- show process cpu history
- show proc mem
- show process cpu | exclude 0.00%__0.00%__0.00%

5) trend your router's performance with MRTG or similar to see what kind of
performance it has. Also useful is speedtest.net or similar.

6) get Cisco IOS iperf command or similar to check the performance of your current config.

7) provide the COMPLETE running config for review to the forum.

Discussions

Some examples of posts with specific pieces of equipment.

»[Info] Cable testing built in to IOS
»[HELP] Cisco Switches shows cable not connected
»[HELP] Cisco 800/837 Slow Upload Speeds Various Download Speeds
»Cisco 1841 connected to cable modem - slow performance
»[Config] 3745 Throughput **SOLVED**

by HELLFIRE See Profile edited by aryoba See Profile
last modified: 2013-01-28 14:38:42

Desktop-related Issues

»Terrible internet video streaming through ASA
»867vae slow VDSL?

Routers

»[HELP] Cisco 1811 Speed Problem on VLAN side
»Cisco 1841 connected to cable modem - slow performance
»FIOS - Slow when using 871W
»[Config] Problem with Cisco 2621xm Router and time outs

Firewalls

»[HELP] PIX 501 slow download speed
»Slow PIX 501
»PIX 501 Slow Using UNC on Outside

Switches

»Question about Cisco Switches and manual v. auto uplinks
»[Other] Cisco 6509 Possible Distance Issue
»[Config] QoS for Home Lab

Note:
Regarding slow connection that is found in small network or small businesses, typically the cause is due to poor infrastructure or poor device/software implementation. Therefore it is highly suggested to read the following FAQ for ideas to improve network connectivity in long-term solution approach.
»Cisco Forum FAQ »Improving Small Business network performance

by aryoba See Profile
last modified: 2014-04-29 13:52:05

»Viewing access list violations

by aryoba See Profile

Following is official Cisco link which requires Cisco CCO account and may require active Smartnet contract. If you are unsure what Smartnet is, check out this FAQ.
»Cisco Forum FAQ »What is Smartnet? Do I need one?

Cisco Tools & Resources

by aryoba See Profile
last modified: 2013-03-20 12:40:46

»[HELP] Can See SSIDs But Randomly Can't Connect

by aryoba See Profile

Cisco website
Troubleshooting High CPU Utilization

Discussion
»[HELP] Weird CPU Spikes on 3750/3560 Switches

by aryoba See Profile
last modified: 2014-02-06 10:06:25

Scenario 1
You have machine's IP address and need to locate the switch port the machine connects to

This is useful when you need to locate rouge machines (servers, PC, or else) that cause network havoc.

»DHCP snooping

Scenario 2
A machine experiences network slowness

»Cisco Forum FAQ »Basic Troubleshooting for Speed Issues with Cisco Equipment
»Cisco Forum FAQ »Slow connection through a router, firewall, or switch
»Cisco Forum FAQ »Getting around Layer-1/2 line error troubleshooting
»Cisco Forum FAQ »High CPU Utilizations

by aryoba See Profile
last modified: 2014-04-29 13:48:51

Scenario 1: Dropped Voice calls

For the sake of this discussion, let's consider the following network diagram.



The diagram illustrates a typical LAN/WAN setup in organizations. There are multiple offices interconnected into WAN (in a form of MPLS or IPSec VPN) where each office has IP Phones and Call Manager as IP-based PBX system. This specific diagram however implements MPLS as WAN.

Users have been raising concerns of dropped call. The objective is to find the cause and then to find possible mitigation plan.

Understanding the problem from technical perspectives

With any troubleshooting process, you need to collect as much info as possible in order to move forward. Following points are some of the basic procedure that season technologists go through.

1. The phones that are having problem

This part ought to cover the following.
* The phone's IP addresses
* The phone location (i.e. only the office with the two phones; between one phone at one office and another phone at other office)
* How the phones are connected (i.e. wireless, wired)
* What network devices sit between the two phones (something like Phone 1 -- Switch 1 -- Router 1 -- WAN -- Router 2 -- Switch 2 -- Phone 2; or Phone 1 -- Switch 1 -- Phone 2)

2. Incident Time frame

This part ought to cover the following.
* When the incident occurs (i.e. morning, lunch time, off hour)
* How often the incident occurs (i.e. everyday, twice a day, once in a while)

3. Incident detail description

This part ought to cover the following.
* What happen before, during, and after (i.e. no voice, garbled voice, delay voice, or echoing voice from other end before disconnect tone sounds)

4. Nature of the setup and incident

In this specific situation, following are essential questions to ask in order to understand the breath of the issue.

* When was this phone system setup?
* How long has the phone system been working fine before the incident took place?
* Did the issue ever happen in the past?
* Was anybody else experiencing the same issue?
* Was there any service ticket open with the phone system vendor?
* Has the phone system vendor ever certified the setup to ensure it followed best practice or vendor recommendation?

5. Review any existing network monitoring system alerts

Assuming there is an existing network monitoring system in place, you can review the alerts to see if there are outages reported during the incident takes place. For those who are unfamiliar with network monitoring system, feel free to review the following FAQ.
»Cisco Forum FAQ »Automatic Network Health Monitoring and Reporting System: An Introduction

6. Review any (recent) infrastructure changes

Here are potential situations that could cause outages
* Somebody make changes on network, server, phone system, or PC
* Power outage
* Cable cut or loose cable
* Air flows, temperatures, humidity, or simple dust clogs

Troubleshooting Process

1. Prepare network capture

Assuming the phone connects to a switch, you can capture the IP packet traversing the wire before, during, and after the incident. You can either setup a switch port span and having network tap between the phone and the switch. For those who are unfamiliar with switch port span or network tap, feel free to review the following

Switch port span (Port Mirroring)
Port Mirroring Vs Network Tap
Network tap - Wikipedia
Network Taps by Ixia
G-TAP: Network Tap by Gigamon
Network Instrument - nTap
Fluke Networks - Network Traffic Analyzer
Network TAP by VSS Monitoring

2. Prepare network analysis system

As you may be aware, network captures have to have some kind of collectors or analyzer to provide readable info. Common software to be the collectors is Wireshark. In an environment where microsecond-delay is business critical, Corvil is implemented. These network analyzer software ought to be installed on dedicated machine.

3. Review traversing network packets

At this point, let's discuss IP packet traversing from one IP device to another. Depending on the application running on the IP device, there could be some TCP and UDP packet combination going back and forth between the two devices. By tapping the network where these packet flows take place, you could review the packet differences and similarities.

Typically packet flows in certain order. For TCP packets, there are Sequence Number labels on each packet to show such order. For those who are unfamiliar with TCP packet analysis, feel free to review the following.
Transmission Control Protocol - a Wikipedia

As of UDP packet, there may be certain Sequence Number mechanism on higher layer such as the Application layer since by default, UDP packets do not carry Sequence Number. For those who are unfamiliar with UDP packet analysis, feel free to review the following.
User Datagram Protocol - a Wikipedia

In regards of general IP protocol and TCP/IP stack info, feel free to review the following.
Internet protocol suite
RFC 1180: TCP/IP Tutorial

Notice if there are any missing packets between results coming from those multiple points of taps. Note the time of those missing packets occur. If the time frame of those missing packet occurrences match or closely match the time frame of the incident, then these missing packet issues could be the culprit.

4. Review Missing Packet Issues

Various situations could lead to missing packets depending at where the missing incident take place. A simple cable breaks could cause the issue. If the packet traverses over WAN, the WAN provider could drop the packets somehow. For traversing packet over LAN, switch buffer or QoS improper configuration could cause the switch to drop packets.

Feel free to review following FAQ for additional info.
»Cisco Forum FAQ »Improving Small Business network performance
»Cisco Forum FAQ »Getting around Layer-1/2 line error troubleshooting
»Cisco Forum FAQ »Basic Troubleshooting for Speed Issues with Cisco Equipment
»Cisco Forum FAQ »Slow connection through a router, firewall, or switch
»Cisco Forum FAQ »General Wifi connectivity issues
»Cisco Forum FAQ »Basic Network Troubleshooting
»Cisco Forum FAQ »High CPU Utilizations
»Cisco Forum FAQ »Checking traffic activities against ACL

5. Review Dropped Packet Issues

Whereas Missing Packet issue is a Layer-1/2 incidental issue, Dropped Packet issue is considered Layer-2/3 by-design issue. Specifically for Voice over IP (VoIP) phone packet traverse, certain LAN and WAN design incorporates QoS (Quality of Service) methodologies. Each QoS methodology has its own advantages and disadvantages.

One QoS technique is to reserve certain bandwidth allocation to be used by voice traffic. At a time of heavy-bandwidth usage, the voice traffic is guaranteed to keep flowing uninterrupted while data traffic will be delayed. At a time of light or no bandwidth usage, either voice or data traffic flow freely with no delay or interruption.

Different QoS technique is to drop data packet instead of delaying in a time of heavy-bandwidth usage while the voice traffic flow is guaranteed to proceed. Note that either techniques do not scale up when heavy-bandwidth usage occurs often.

Long-occurrence of heavy-bandwidth usage may cause the data packet being dropped often. Too many phone call going through at the same time will overwhelm the QoS guarantee mechanism, that at one point one of those voice traffic will be dropped as well. Either voice or data packet drop causes the voice or data application to break eventually.

For those who are unfamiliar with QoS and VoIP technologies, feel free to review following FAQ.
»Cisco Forum FAQ »Deploying VoIP - An Introduction
»Cisco Forum FAQ »Introduction to Voice over IP
»Cisco Forum FAQ »Working with Voice over IP protocols
»Cisco Forum FAQ »QoS Basic and Implementation
»Cisco Forum FAQ »How do I configure QoS for VoIP?
»Cisco Forum FAQ »Improving Performance of Cable/DSL Internet using QoS
»Cisco Forum FAQ »Router runs VoIP, Bit Torrent, Online Gaming; DynDNS - QoS Sample Configuration
»Cisco Forum FAQ »Troubleshooting QoS

by aryoba See Profile
last modified: 2014-05-15 10:10:19