dslreports logo


60.0 Troubleshooting

Suggested prerequisite reading:
»Cisco Forum FAQ »Deleted/Corrupted/Wrong Type/No IOS image and router won't boot

Have you tried to upgrade an image but were unsuccessful? The router is stuck in rommon mode? If yes to any one of those questions; it means that the router cannot find a valid image in the flash memory. The image usually gets corrupted or lost.

Try the following link to recover the problem. The URL should work on any router that has tftpdnld command in the rommon.

Routers ROM Monitor IOS Image Download Procedures

The whole idea is to utilize tftpdnld ROMMON mode command to download working IOS image from your TFTP server to the router. To recover a router using this method, there are assumptions as follow

* You have a valid working IOS image
* You have a working TFTP server
* The working IOS image is stored in the TFTP server

When you don't have TFTP server yet, check out the following FAQ for preparing the server
»Cisco Forum FAQ »How to prepare TFTP/FTP server

If you don't have a working IOS image file around, then you should be able to download one from Cisco website; assuming you have Smartnet contract. Check out the following FAQ for more info on Smartnet
»Cisco Forum FAQ »How/where do I get Cisco images (such as IOS, PIX/ASA OS)?

As an illustration on utilizing the tftpdnld command, check out this thread.

»[HELP] ROMmon corrupted

by aryoba See Profile
last modified: 2009-05-19 10:24:55


»Cisco 2960 Switch not booting all the way
»CISCO Catalyst 2948G-L3 no keystrokes in console

by aryoba See Profile
last modified: 2013-04-10 16:18:11

Bootup Errors

Discussion

»problems with boot in AS5300

Get LANCE error message? The Ethernet controller might be the problem

Do you have error message concerning LANCE? If so, you might have problem with the ethernet controller.
%LANCE-1-INITFAIL: Unit [dec], initialization timeout failure, csr[dec]=[hex]
The hardware has failed to initialize correctly.

Recommended Action: Repair or replace the controller.

On modular equipment, the ethernet controller is on the module. Whenever possible, try to remove the module concerned; just to ascertain whether the device goes through the bootup process with no issues. If that is the case, you might want to replace the module.

FYI, the LANCE error message is from the Ethernet controller. The following thread provides details.
»Initialization timeout failure - fix controller?

Other Hardware Issues

»[HELP] ALIGN-3-SPURIOUS Error
»[H/W] C3560-24PS-S v05, reboots in loop, how to set tcam value
»Verizon DSL and Cisco 678
»[HELP] Cisco 7604
»[Info] Cisco 877 and 887 switch port lights..
»[H/W] 2911 SegFault

by nozero See Profile edited by aryoba See Profile
last modified: 2016-02-25 09:38:33

ASA/PIX Firewall hangs and won't fully boot up
»Cisco PIX hang and wont reload
»Cisco ASA 5520 Error Code: -5

ASA/PIX Firewall does not boot up at all
»[H/W] Cisco ASA 5505 - Issues booting up
»[HELP] ASA 5520 Not Booting

Clock-related issue
»[H/W] ASA5506 - 'sfr experienced a data channel communication failure'

by aryoba See Profile
last modified: 2017-03-06 12:20:47

»A 3750X switch unable to boot - Welcome to Stardust! and MrIncredible
»[CCNA] CCNA 3550 switch with a warning error: switch core BIST
»POST Failure Error-Catalyst 3500 XL

by aryoba See Profile
last modified: 2016-10-19 16:15:33

When one of the following situation occurs, your router "loses" its configuration

• power back on a router after power off for a while

• there is lightning strike around

• unprotected power outlet (i.e. no UPS)

• after configuring a router, you reload the router

• after installing new IOS image, you reload the router to let the new IOS image become active

• you implemented a wrong config register value

First step to take is to verify whether the router actually lost configuration. In order to do so, you might want to check the config register value to verify if it is 0x2102 (the default) or else. When the config register value is not set as 0x2102, then the router might act differently than the default.

Following is an example. Let's say the router config register value is set to 0x2142. By having such value, the router will never load saved configuration every time the router boots up from either power up, power cycle, or reload command. The router ignores the saved configuration and instead boots up using default (blank) configuration.

Such abnormal behavior is typically unwanted at any time. To avoid, you need to set the value back to the 0x2102.

To find out what the current router config register value is, you can simply do a "show version" from the CLI prompt and look at the last line.

To set the value back to 0x2102 (default value), one way of doing it is via the CLI with following commands

If for some reason the router goes to ROMMON mode, you can issue the 0x2102 value with following command

When the router has 0x2102 config register value, the router will act as factory recommended (Cisco recommended) including loading the current configuration after reload or power cycle.

Note:

When you are in luck, there is one possibility is that the router still has the configuration even though the router does not boot the configuration. This situation applies when the router configuration prior reload or prior power loss is saved (by issuing "copy running-config startup-config" or "write memory" prior).

Cisco documentation
Configuration Register Setting Descriptions


Feedback received on this FAQ entry:
  • Thank you. Saved my life

    2010-11-16 21:30:10

  • Spot On! Many Thanks.

    2009-11-03 14:48:01

by aryoba See Profile
last modified: 2013-10-18 08:32:10

Suggested pre-reading
»Cisco Forum FAQ »Config Register Value - router lost configuration, how to recover

The idea is to set the config-register value back to default, which is 0x2102. Note that certain config-register values not just stop the router to boot up normally but also change the baud speed of the CONSOLE port which makes the CONSOLE port output is unreadable or even does not show at all. Therefore to fix the issue, you need to do the following

1. Make sure the CONSOLE port output to become readable
2. Issue the 0x2102 config-register value, either via CLI prompt or ROMMON prompt

Following is an illustration.

Routers

If you are directly consoled into the router, open a new HyperTerminal session (or any preferred terminal simulator) with these settings:

Bits per second: 1200
Data bits: 8
Parity: None
Stop Bits: 1
Flow control: None

Once this is open, power cycle the router and press the spacebar for roughly 10 seconds (press and release). If you are able to see the rommon prompt, change the register back to normal by typing:
rommon>confreg 0x2102
rommon>reset

When you are not able to see anything on the screen, close the window and open a new one with these settings:

Bits per second: 9600
Data bits: 8
Parity: None
Stop Bits: 1
Flow control: None

And you should now see the rommon> prompt. Change the register from there and the router should be back in normal mode.



Derived from this thread.

Switches

If, for some reason, your console output is garbled, and no matter how many times you boot holding the MODE button, you still can't figure it out, try using a PC (Not a Mac, as this seems to make a difference) with PuTTY and connecting at 170000 baud, 8n1. It might work just enough for you to boot into RomMon, run "set BAUD 9600" and reboot. Strange, right?

I found at least 2 other posts regarding this seemingly undocumented bug. Some people say it may be caused by setting a bad/unsupported baud rate for a specific piece of hardware, or in my case a seemingly mysterious flash chip wipe from a switch out of the lab. I also saw posted that a Prolific USB-Serial adapter was required, but my trusty FTDI 232 worked fine for this fix (On a PC only).

I saved a 3560 from the trash today with this tip.



Derived from this thread.

Side Notes

* If you know the current setting of your router's config register value is, you can use the Config Register calculator to find out how the router is configured when it boots. You can download the calculator from this site
* By any chance the router model is 2600 or 3600 series, check out the following FAQ
»Cisco Forum FAQ »Unreadable output from Cisco Router/Switch Console port

Cisco documentation
Configuration Register Setting Descriptions

Discussions

»[HELP] Cisco 1800 garbled output in putty


Feedback received on this FAQ entry:
  • Peeeerrrffeecctt bro!!!!!.. THaaankss!!!!

    2012-11-02 16:38:21

by Covenant See Profile edited by aryoba See Profile
last modified: 2016-11-01 13:46:20

Password Recovery Procedures for some Cisco appliances

The following link is about the index of password recovery procedures for some Cisco appliances. Next discussion is for others such as IronPort and Nexus switch Password Recovery.

Note: For security reasons, the password recovery procedures described in the link require physical access to the equipment.

Password Recovery Procedures

Properly Sending BREAK key

You need to enter ROMMON mode which may require you to send the BREAK key. Check out the following link to find the suitable BREAK key.

Standard Break Key Sequence Combinations During Password Recovery



Tips

* For newer machines running Windows XP, Vista, Windows 7, and Windows 8 using Hyperterminal with no BREAK key available on the keyboard, try to use PAUSE key instead. In other words, issue CTRL-PAUSE combination key as a replacement CTRL-BREAK combination key.




It is highly recommended to use a computer that has an actual Serial port (RS-232) when sending
BREAK key. Whenever possible, don't use computer that utilize USB port to emulate Serial port since the BREAK key might not be sent properly.

In rare case where the router password recovery functionality is disabled, you can perform tasks provided on this link to re-enable the password recovery functionality.

To Disable and Re-enable Password Recovery Functionality

NOTE

The Cisco documentation shows copying saved configuration to running configuration as part of the recovery procedure. In some (if not most) cases, this step is a no-no.

Password recovery procedure is necessary when the password to log into the device is unknown and something in the configuration blocks administrative (enable) mode to activate, stopping you to enter enable mode. When you copy the saved configuration to running configuration, then basically you are going back to the point before you make any password recovery attempt, which will nullify the password recovery work you have done.

With that in mind, there should be no copying saved configuration to running configuration as part of the recovery procedure. Instead put sufficient configuration (or no configuration at all) to the device just to keep it accessible and do the things that can pass traffic.

Discussion:
»[HELP] 1811W forgot login info
»[HELP] AS5350: user default password

Catalyst 3750-X Switch Password Recovery Procedures

Official link
Troubleshooting

IronPort Password Recovery Procedures

Please follow the steps below in case you have lost your "admin" password on your IronPort appliance.

* Contact Customer Support for a temporary password. You will need the Serial Number of the device.
* Once you receive the temporary password, please access the IronPort via serial connection.
* Log in as user 'adminpassword'.
* Enter the temporary password you received from the Customer Support Engineer and hit return.
* Enter the new password that will be used for the 'admin' user.

Discussion
»Password Recovery Procedure for Cisco Ironport Mail Gateway?

Cisco equipment's default password

Access Point
username: cisco
password: Cisco
(case sensitive)

Nexus 3000 switch Password Recovery Procedure

Here is one method that has been field tested to work. Similar to some Cisco equipment, this method in general requires the following.

* Console access
* Physical access in order to power cycle

Depending on your network accessibility setup, physical presence might not be necessary when there is remote access to console port using some terminal server and to power supply using IP-based power strip.

Background

Cisco Nexus 3000 boots up by calling Linux-based GNU grub. Notice that there are two GNU grub calling during the boot-up process. The idea is to send a break key before the first GNU grub calling.

With Nexus 3000, the proper break key is CTRL ] (a combination of CTRL button and right square bracket button). Pressing different combination such as CTRL C results to different situation which you may not want to be in.

Note the Proper Image Version

It is essential that you are aware which image version the switch run, especially when there are multiple image stored in bootflash. If you are unaware, then simply power cycle the switch to find out. Here are the steps.

1. Make sure you have proper console access so that you can see every step the switch take during boot up
2. Power cycle the switch
3. During first GNU grub call, the switch shows which kickstart image is being used. Note the image version.
4. During second GNU grub call, the switch shows the main image being used. The image version should match the kickstart image version
5. Make a note of this main image version as identification of the proper image to use during password recovery process

Password Recovery Steps

1. Make sure you have proper console access so that you can see every step the switch take during boot up

2. Power cycle the switch

3. Immediately send CTRL ] as the break key before the first GNU grub shows up.

4. If you see the second GNU grub call, this means that you are late to send the break key. Redo steps 1 to 3

5. If you don't see the second GNU grub and you have the prompt of switch(boot)#, then issue dir bootflash: command to ensure you see the proper switch image. This image must be the one shows during normal bootup (Step 5 of the Note the Proper Image Version as mentioned above)

6. When you don't see the proper image or you see no files at all in bootflash, this means that you probably send incorrect break key. Redo steps 1 to 5.

7. When you see the proper image, then be ready the reset the password as follows

Note:
The switch requires you to implement certain restriction as acceptable password, which is a combination of uppercase, lowercase, and numbers in addition of meeting a eight-character minimum.

8. You may reissue dir bootflash: command to ensure you remember the proper image version to bootup

9. Boot up the switch using the proper image


10. The switch resumes the boot-up process by calling the second GNU grub

11. Log in to the device using the new administrator password.


Password Recovery Proceedure for 3850 (IOS-XE based) Stack Switches

PART 1

Step 1
Connect a terminal or PC to the switch.
* Connect a terminal or a PC with terminal-emulation software to the switch console port. If you are recovering the password for a switch stack, connect to the console port of the active switch or
* Connect a PC to the Ethernet management port. If you are recovering the password for a switch stack, connect to the Ethernet management port of a stack member.

Step 2
Set the line speed on the emulation software to 9600 baud.

Step 3
Power off the standalone switch or the entire switch stack.

Step 4
Reconnect the power cord to the or the active switch. Within 15 seconds, press the Mode button while the System LED is still flashing green. Continue pressing the Mode button until all the system LEDs turn on and remain solid; then release the Mode button.

proceed to the Procedure with Password Recovery Enabled section, and follow the steps. ( see below PART 2 Step 1)

Step 5
After recovering the password, reload the switch or the active switch.

On a switch:


On the active switch:


Step 6
Power on the remaining switches in the stack.

Procedure with Password Recovery Enabled

If the password-recovery operation is enabled, this message appears.


PART 2

Step 1
Initialize the flash file system.


Step 2
Ignore the startup configuration with the following command:


Step 3
Boot the switch with the packages.conf file from flash.


Step 4
Terminate the initial configuration dialog by answering No.


Step 5
At the switch prompt, enter privileged EXEC mode.


Step 6
Copy the startup configuration to running configuration.

Press Return in response to the confirmation prompts. The configuration file is now reloaded, and you can change the password.

Step 7
Enter global configuration mode and change the enable password.


Step 8
Write the running configuration to the startup configuration file.


Step 9
Confirm that manual boot mode is enabled.


Step 10
Reload the switch.


Step 11
Return the Bootloader parameters (previously changed in Steps 2 and 3) to their original values.


Step 12
Boot the switch with the packages.conf file from flash.


Step 13
After the switch boots up, disable manual boot on the switch.


Step 14
Proceed with the step 5 PART 1

Up to this point your password recovery should have been done.


Feedback received on this FAQ entry:
  • Good Document !

    2016-07-21 05:30:46

by nozero See Profile edited by aryoba See Profile
last modified: 2017-03-08 16:29:03

Related FAQ
»Cisco Forum FAQ »Used wrong config-register and now the router/switch does not boot!

Discussions
»[Config] Cisco 871w - All Baud Rates result in Gibberish

by aryoba See Profile
last modified: 2013-01-22 15:22:34

MTU Size regarding PPPoE over ATM/DSL

This FAQ provides a guaranteed working config for anyone using any ADSL PPPoE connectivity types such as Ameritech ADSL for their circuit provider and Megapath.net for ISP. It took 2 calls 2 cisco and weeks of fighting with ISP tech support, but I learned a valuable lesson about ADSL PPPoE specifically.

The MTU on the dialer interface should be 1492 as PPPoE adds an 8 byte encapsulation header. The key is setting ip tcp adjust-mss 1440 on the inside ethernet interface. You will find many different suggestions and recommendations out there. Some will say 1492 or 1460. Some will even say 1452. 1452 MSS is pretty much the standard for DSL with a PPPoE transport. Normal MSS is 1500 bytes. But you have to account for the 40 byte IP header and the 8 byte PPPoe header. That gets you to 1452.

Following is from the mouth of Cisco, "If you have ADSL running PPPoE and run into problems resolving DNS, adjust your MTU on your ethernet interface using the command ip tcp adjust-mss 1452. This is because PPPoE requires more bits in the header packet than any other type of circuit."

The last bit of optimization is a little more subtle and is a debatable topic. As the PPPoE traffic is carried over ATM cells, it has to be chopped up before it can be transmitted. ATM cells are 53 bytes long and have a 5 byte header. So a total of 48 bytes of payload. If you were to take 1452 bytes of data and split it up across 48 byte payloads. You would come up with 30.25 cells. The .25 is a 12 byte remainder that would have to be sent in a separate ATM cell. ATM cells are always 53 bytes. So the payload would have to be stuffed with an additional 36 bytes of null data for that last chunk. So to be completely optimized you would set the MSS to 1440 to eliminate those wasted 36 bytes.

Adjusting MTU size was news to me, but the minute we did it all my problems were fixed. Feel free to experiment to set MTU size to either 1452 or 1440 to see which size brings you the most suitable result.

MTU Discussion
»Best IOS for 1801W

So with that in mind, here is a 100% working config from my 827 ADSL router. Hope this lesson I learned helps someone out in the future!!!!

SANCH_INT_RTR#sh run
Building configuration...

Current configuration : 2593 bytes
!
version 12.1
no service single-slot-reload-enable
no service pad
service timestamps debug uptime
service timestamps log uptime
service password-encryption
!
hostname SANCH_INT_RTR
!
logging rate-limit console 10 except errors
no logging console
enable secret 5 $encrypted password$
!
username Cisco privilege 15 password 7
username Router password 7
ip subnet-zero
no ip finger
ip name-server 66.80.130.23
ip name-server 66.80.131.5
!
no ip dhcp-client network-discovery
vpdn enable
no vpdn logging
!
vpdn-group pppoe
request-dialin
protocol pppoe
!
!
!
!
interface Ethernet0
ip address 69.33.X.X 255.255.255.224
ip tcp adjust-mss 1452
no ip mroute-cache
!
interface ATM0
no ip address
ip access-group 101 in
ip access-group 101 out
no ip mroute-cache
no atm ilmi-keepalive
bundle-enable
dsl operating-mode auto
hold-queue 224 in
!
interface ATM0.1 point-to-point
pvc 0/35
protocol pppoe
pppoe-client dial-pool-number 1
!
!
interface Dialer0
no ip address
no cdp enable
!
interface Dialer1
mtu 1492
ip address 69.33.XX.XX 255.255.255.0
encapsulation ppp
dialer pool 1
dialer-group 1
ppp authentication pap chap callin
ppp chap hostname ppp-username@megapathdsl.net
ppp chap password 7 Encrypted password
ppp pap sent-username ppp-username@megapathdsl.net password 7 encrypted password 0A
!
ip classless
ip route 0.0.0.0 0.0.0.0 69.33.X.X
no ip http server
!
access-list 101 deny tcp any any eq 135
access-list 101 deny tcp any any eq 136
access-list 101 deny tcp any any eq 137
access-list 101 deny tcp any any eq 138
access-list 101 deny tcp any any eq 139
access-list 101 deny tcp any any eq 140
access-list 101 deny udp any any eq 135
access-list 101 deny udp any any eq 136
access-list 101 deny udp any any eq netbios-ns
access-list 101 deny udp any any eq netbios-dgm
access-list 101 deny udp any any eq netbios-ss
access-list 101 deny udp any any eq 140
access-list 101 deny tcp any any eq 445
access-list 101 deny udp any any eq tftp
access-list 101 deny tcp any any eq 4444
access-list 101 deny tcp any any eq 593
access-list 101 permit tcp any any eq www
access-list 101 permit tcp any any eq 1723
access-list 101 permit tcp any any eq 110
access-list 101 permit tcp any any eq 25
access-list 101 permit gre any any
access-list 101 permit icmp any any
dialer-list 1 protocol ip permit
banner login ^CC^C
!
line con 0
password 7 XXXXXXXXX
transport input none
stopbits 1
line vty 0 4
exec-timeout 30 0
password 7 XXXXXXXXX
login
length 0
!
scheduler max-task-time 5000
end

This FAQ created using this post by sanchito75 See Profile and the naming suggested by Covenant See Profile.


Feedback received on this FAQ entry:
  • Great write up..old but gold! :) Kumar

    2013-10-06 22:12:08

  • Don't normally post, but thank you so much for this information. It resolved an issue with SIP not working on VVX 1500 phones. Other models worked just fine without changing the mtu.

    2013-05-08 16:52:48

  • Thank you very much for this explanation ;-)

    2011-08-03 03:20:45

  • I'm so glad I found this site! I'm having issues all weekend hooking up my new Cisco 877. Bad DNS and inability to connect to HTTPS and IMAPS servers. I never set my MSS adjust. Thank you so much!!!!! John

    2011-02-13 13:19:40

by nozero See Profile edited by aryoba See Profile
last modified: 2013-01-25 16:42:36

Question - I just got a new Cisco router and the Cisco Router Web Interface (= CRWS) just hangs when I try to start it What do I do?

Introduction

CRWS is one of two GUI interfaces for the 800 and SOHO series routers (the other being SDM). It resides in a section of the router's flash memory called "webflash." When you want to access CRWS, you open an Internet browser window and type "http://10.10.10.1" in the URL address window (similar to open up Yahoo! website by typing "http://www.yahoo.com").

Note:
If accessing the "http://10.10.10.1" does not show anything but bunch of error messages in the page, there is a possibility that the router LAN IP address has changed. To verify and troubleshoot further, there is no other choice but to use CLI. Check out the following FAQ to revive CRWS/SDM via CLI.
»Cisco Forum FAQ »My SDM/CRWS (web configuration mode) doen't work. How do I revive it?

CRWS Hangs/Does Not Launch when opening up using Internet Explorer

A common cause of CRWS not working is a known bug in CRWS in which Microsoft Virtual Machine is required for it to run, the subject of this FAQ.

Microsoft Virtual Machine Configuration/Installation

Figure 1 (»/showp ··· &1=1) Open Internet Explorer and go tools\internet options
Figure 2 (»/showp ··· &1=1) In the Internet options window, click the advanced tab.
Figure 3 (»/showp ··· &1=1) Then scroll down and see if there is a main heading for Microsoft VM (if there is no Microsoft VM heading, skip to figure 4). If the Microsoft VM main heading is there, under this heading check the box for "JIT compiler for Virtual Machine enabled." Then go to the main heading above it entitled Java (Sun) and uncheck all the checkboxes there. Then click apply, exit out of the window and reboot your PC to save the settings
Figure 4 (»/showp ··· &1=1) - If there is no main heading for Microsoft VM you don't have it, you need this file filename msjavx86.exe - from a trusted website One website is »java-virtual-machine.net ··· oad.html. Download it and (after checking for viruses as you would do with any download, right?) install it (it will ask you to reboot, do so). Then configure Internet Explorer as explained previously in figure 3 above
Figure 5 (»/showp ··· &1=1) Go to Windows update to update the Microsoft VM.
Figure 6 (»/showp ··· &1=1) Select the VM update(s) and install them, reboot if requested.
Figure 7 (»/showp ··· &1=1) Open Internet Explorer, type "http://10.10.10.1", hit enter and CRWS should properly start now.

Note on CRWS usage:
1. Leave the LAN IP address at 10.10.10.1, (see figure 7) as changing it can create problems for the router.
2. CRWS allows basic router functionality. It does not allow you to do everthing the router is capable of. Learning the Command Line Interface (CLI) is necessary to make use of all the router's features.

Useful Links:
CRWS demo at Cisco website »www.cisco.com/warp/publi ··· crws.htm
CRWS description at Cisco website
»www.cisco.com/en/US/prod ··· dex.html
CRWS User Guide
»www.cisco.com/univercd/c ··· ws30.htm
CRWS software downloads (includes CRWS caveats document, Cisco-speak for bugs)
»www.cisco.com/pcgi-bin/t ··· .pl/crws
Switching Between SDM and CRWS
»www.cisco.com/en/US/prod ··· cc8.html

by Requiems See Profile edited by aryoba See Profile
last modified: 2015-08-17 15:57:24

Suggested prerequisite reading:

»Cisco Forum FAQ »Straight-forward way to configure Cisco router: Introduction to CLI

There are various reasons why the SDM does not work. Some of them are the following.

* old Java issue
* SDM is not activated
* SDM software does not exist on the router
* SDM software is corrupted
* The LAN interface IP address is changed
* The LAN interface is shutdown

This FAQ is not meant to be the complete SDM troubleshooting guide. This FAQ however points you to the right direction to find out what the cause is and to revive your router.

When the web configuration mode doesn't work, use the CLI (Command Line Interface) as the most reliable way to configure and troubleshoot routers, including troubleshooting SDM access issue.

Here are the steps to revive inactivated SDM.

1. Do a "show running-config" from enable (privilege) mode and check if there are such commands of "no ip http server" and/or "no ip http secure-server"

2. When you find it, it means the router is currently configured to disable the SDM/CRWS. To enable it, issue "ip http server" and/or "ip http secure-server" from global configuration mode

3. When there is no IP address under the LAN interface, you need to assign one. Make sure that the LAN interface IP address and your PC IP address are within the same subnet

4. Using your web browser on your PC, open "http://[YOUR ROUTER LAN IP ADDRESS]" and see if you are able accessing the SDM/CRWS

The following thread shows some walk through using CLI step-by-step from very beginning to revive the web configuration feature.

Keep in mind that even though your product may not be an 837 router as is used in the thread example, the description on CLI introduction still applies to any Cisco router that supports web configuration.

»[Config] cisco 837 defaults

by aryoba See Profile
last modified: 2009-04-06 15:42:40

If you use a KVM switch, then there might be something wrong with the switch. Test by removing the switch from the scenario. Check out the following thread for more info.

»[CCNA] Bootup Error occuers on Cisco 2514 Router

by aryoba See Profile
last modified: 2009-04-01 11:42:40

Scenario 1: Remote Users are able to ping

The remote user is able to receive IP address off DHCP pool. Users at both sites are able to ping (ICMP echo and echo reply) each other's IP address.

You are using Microsoft network. One task is to have remote users try to map share drive and it fails. What is the problem?

1st of all, let's break down the problem. When the remote user is able to receive the IP address and are able to ping each other's IP address, then the VPN tunnel must be up. The fact that remote user is unable to map share drive is then not the VPN setup problem, but something on your Microsoft network is preventing the drive sharing.

Check out the following official Cisco link for more info.
Troubleshooting Microsoft Network Neighborhood After Establishing a VPN Tunnel

Scenario 2: Remote Users are unable to ping

The remote user is able to receive IP address off DHCP pool. Users at both sites are unable to ping (ICMP echo and echo reply) each other's IP address.

1st thing is to confirm that no firewall that blocks ICMP echo and echo reply. Once it is confirmed, then you might want to confirm IPSec VPN device configuration. If the VPN device is a PIX/ASA Firewall, then a isakmp nat-traversal command might be necessary to be in place. Check out the following FAQ for details.

»Cisco Forum FAQ »Configure router and ASA/PIX Firewall to support various VPN technologies

Some discussions
»[Config] ASA 5510 Firewall vpn not mapping drives
»[2K3] Mapping network drives After you VPN into network

by aryoba See Profile
last modified: 2010-06-16 11:05:02

Cisco GSS appliances

»[Info] Subtle GSS load balancing issue

by aryoba See Profile
last modified: 2008-09-03 09:12:25

Cable Internet (Coax)

»Interface errors on ASA 5505 to SMC DOCSIS modem

DSL

»[Config] Hmmm. interface ATM0 remains down, line protocol remains down
»Line Noise
»Cisco 678 what is the Alarm meaning
»[HELP] Strange interface issues with cisco 2611
»[Config] 2610 w/ ADSL WIC - Upgraded to 8Mbit - Poor Connectivit
»Cisco 678 as bridge & have d-link 514 do NAT/DHCP
»what is crc_errors(52) exceeded threshold 678
»Cisco 837 CRC / Header errors...

T1

»Errors and the Telco. How do I know who it is?
»Input Errors
»[HELP] Need help determining errors
»Multilink traffic stats seem goofy

Ethernet

»Ethernet port shows up/down with no cable attached
»[HELP] input errors question
»[H/W] Interface Transmit Errors
»[H/W] 3750 POE Model Line Noise
»[HELP] Cisco ASA5580-20 SSL VPN intermittent issues

For those who like to further testing cable run to ensure Layer-1 connectivity, check out the following discussions to find out a good cable tester.
»Cable testers
»Fluke Cable IQ Rental

by aryoba See Profile
last modified: 2014-11-17 07:56:38

Things that are worth to explore

1) check your interface between your gear and the ISP. Ensure the right
speed and duplex is present, and whether there are any errors / drops / etc.

2) check your interface between your gear to the LAN for the same thing.

3) if possible, use Fluke meter to check the cabling.

4) check the outputs of the following Cisco IOS commands:

- show process cpu history
- show proc mem
- show process cpu | exclude 0.00%__0.00%__0.00%

5) trend your router's performance with MRTG or similar to see what kind of
performance it has. Also useful is speedtest.net or similar.

6) get Cisco IOS iperf command or similar to check the performance of your current config.

7) provide the COMPLETE running config for review to the forum.

Discussions

Some examples of posts with specific pieces of equipment.

»[Info] Cable testing built in to IOS
»[HELP] Cisco Switches shows cable not connected
»[HELP] Cisco 800/837 Slow Upload Speeds Various Download Speeds
»Cisco 1841 connected to cable modem - slow performance
»[Config] 3745 Throughput **SOLVED**

by HELLFIRE See Profile edited by aryoba See Profile
last modified: 2013-01-28 14:38:42

Desktop-related Issues

»Terrible internet video streaming through ASA
»867vae slow VDSL?

Routers

»[HELP] Cisco 1811 Speed Problem on VLAN side
»Cisco 1841 connected to cable modem - slow performance
»FIOS - Slow when using 871W
»[Config] Problem with Cisco 2621xm Router and time outs

Firewalls

»[HELP] PIX 501 slow download speed
»Slow PIX 501
»PIX 501 Slow Using UNC on Outside

Switches

»[HELP] Issues with Netflix, Hulu, YouTube and other streaming during high usa
»Question about Cisco Switches and manual v. auto uplinks
»[Other] Cisco 6509 Possible Distance Issue
»[Config] QoS for Home Lab

Note:
Regarding slow connection that is found in small network or small businesses, typically the cause is due to poor infrastructure or poor device/software implementation. Therefore it is highly suggested to read the following FAQ for ideas to improve network connectivity in long-term solution approach.
»Cisco Forum FAQ »Improving Small Business network performance

by aryoba See Profile
last modified: 2015-11-23 15:56:37

Traditional IOS

»Viewing access list violations

NX-OS

Preliminary

Unlike traditional IOS ACL logging, we do not see a log message EVERY time there is an ACL hit in NX-OS. Instead we see hit notifications sent at specific intervals.

Optimized Access-list Logging is a feature that was introduced on the 6500 platform a while back. The Nexus 7000 uses this same infrastructure to keep the CPU protected from ACL logging that users may implement.

According to Cisco documentation
Catalyst 6500 Release 12.2SX Software Configuration Guide

here is a snippet.

"OAL provides hardware support for ACL logging. In this case OAL permits or drops packets in hardware and uses an optimized routine to send information to the RP to generate the logging messages"

On the 6500/7600, OAL was optional, and you could still use CPU intensive acl logging if desired (on by default). The Nexus 7000 however has only OAL for ACL logging.

So now lets look at the n7k specific implementation of ACL Logging, or OAL.

For the sake of the illustration, let's assume the following configuration.

By default, if you have an access-list configured with the log parameter, you will not see any logs in the buffer. However, you will be able to see them in the OAL cache (this output comes after sending 5 pings with source 10.0.0.2 to 11.0.0.2):

Note that the above output shows the "Interface" value as Ethernet2/11 for the first flow. This is the ingress interface for that packet, not the interface where the ACL was applied.

The cache entry persists while the flow is active, but the hit counters are cleared at a set interval (configurable). This interval is also used to set frequency of the actual log messages that are displayed. In other words, once the flow is created, you will receive a log message every [interval] seconds with an update showing the current hit counters for that flow (while that flow is still active in the cache).

By default, the following settings are applied, which do not have to be changed.

Options of NX-OS ACL Implementation

Option 1

OAL specific parameter configuration

Default OAL values are sufficient to get the logging enabled.

Required non-default changes for logging

As shown above, the logging level for the "acllog" facility must be configured to be greater than or equal to the "acllog match-log-level" setting, and the "logging logfile" severity must be equal to or greater than that setting as well. Otherwise, the log messages do not show up in the logs. I used the value of 3 by choice, but is not a required setting.

Once this is configured, you can see the logs show up as desired:

As you can see, each active flow updates the log with the number of hits during the current interval, and will continue to do so while the flow is active.

Option 2

This time, I insert statistics per-entry command line in the ACL in questions as follows.

Inserting the command is like inserting any permit or deny command

When you issue show access-list TEST command, here is what you see.

To check the match traffic, simply issue the show log ip access-list cache command.

Closing Statement

Note that you may want to implement both options since they complement each other.

And that's it! The settings can be tweaked to match your needs, but you now have acl logging, without impacting the cpu!

by aryoba See Profile
last modified: 2016-11-15 16:29:24

Following is official Cisco link which requires Cisco CCO account and may require active Smartnet contract. If you are unsure what Smartnet is, check out this FAQ.
»Cisco Forum FAQ »What is Smartnet? Do I need one?

Cisco Tools & Resources

by aryoba See Profile
last modified: 2013-03-20 12:40:46

»[OT] Hunting a Rouge DHCP server.
»[HELP] Can See SSIDs But Randomly Can't Connect
»[Config] BVI on Autonomous AP stopped working after iOS upgrade

by aryoba See Profile
last modified: 2015-10-16 09:01:55

Cisco website
Troubleshooting High CPU Utilization

Discussion
»[HELP] Weird CPU Spikes on 3750/3560 Switches

by aryoba See Profile
last modified: 2014-02-06 10:06:25

Scenario 1
You have machine's IP address and need to locate the switch port the machine connects to

This is useful when you need to locate rouge machines (servers, PC, or else) that cause network havoc.

»DHCP snooping

Scenario 2
A machine experiences network slowness

»Cisco Forum FAQ »Basic Troubleshooting for Speed Issues with Cisco Equipment
»Cisco Forum FAQ »Slow connection through a router, firewall, or switch
»Cisco Forum FAQ »Getting around Layer-1/2 line error troubleshooting
»Cisco Forum FAQ »High CPU Utilizations

Scenario 3
A machine does not receive IP address

»Rant

by aryoba See Profile
last modified: 2016-04-07 14:23:15

Scenario 1: Dropped Voice calls

For the sake of this discussion, let's consider the following network diagram.



The diagram illustrates a typical LAN/WAN setup in organizations. There are multiple offices interconnected into WAN (in a form of MPLS or IPSec VPN) where each office has IP Phones and Call Manager as IP-based PBX system. This specific diagram however implements MPLS as WAN.

Users have been raising concerns of dropped call. The objective is to find the cause and then to find possible mitigation plan.

Understanding the problem from technical perspectives

With any troubleshooting process, you need to collect as much info as possible in order to move forward. Following points are some of the basic procedure that season technologists go through.

1. The phones that are having problem

This part ought to cover the following.
* The phone's IP addresses
* The phone location (i.e. only the office with the two phones; between one phone at one office and another phone at other office)
* How the phones are connected (i.e. wireless, wired)
* What network devices sit between the two phones (something like Phone 1 -- Switch 1 -- Router 1 -- WAN -- Router 2 -- Switch 2 -- Phone 2; or Phone 1 -- Switch 1 -- Phone 2)

2. Incident Time frame

This part ought to cover the following.
* When the incident occurs (i.e. morning, lunch time, off hour)
* How often the incident occurs (i.e. everyday, twice a day, once in a while)

3. Incident detail description

This part ought to cover the following.
* What happen before, during, and after (i.e. no voice, garbled voice, delay voice, or echoing voice from other end before disconnect tone sounds)

4. Nature of the setup and incident

In this specific situation, following are essential questions to ask in order to understand the breath of the issue.

* When was this phone system setup?
* How long has the phone system been working fine before the incident took place?
* Did the issue ever happen in the past?
* Was anybody else experiencing the same issue?
* Was there any service ticket open with the phone system vendor?
* Has the phone system vendor ever certified the setup to ensure it followed best practice or vendor recommendation?

5. Review any existing network monitoring system alerts

Assuming there is an existing network monitoring system in place, you can review the alerts to see if there are outages reported during the incident takes place. For those who are unfamiliar with network monitoring system, feel free to review the following FAQ.
»Cisco Forum FAQ »Automatic Network Health Monitoring and Reporting System: An Introduction

6. Review any (recent) infrastructure changes

Here are potential situations that could cause outages
* Somebody make changes on network, server, phone system, or PC
* Power outage
* Cable cut or loose cable
* Air flows, temperatures, humidity, or simple dust clogs

Troubleshooting Process

1. Prepare network capture

Assuming the phone connects to a switch, you can capture the IP packet traversing the wire before, during, and after the incident. You can either setup a switch port span and having network tap between the phone and the switch. For those who are unfamiliar with switch port span or network tap, feel free to review the following

Switch port span (Port Mirroring)
Port Mirroring Vs Network Tap
Network tap - Wikipedia
Network Taps by Ixia
G-TAP: Network Tap by Gigamon
Network Instrument - nTap
Fluke Networks - Network Traffic Analyzer
Network TAP by VSS Monitoring

2. Prepare network analysis system

As you may be aware, network captures have to have some kind of collectors or analyzer to provide readable info. Common software to be the collectors is Wireshark. In an environment where microsecond-delay is business critical, Corvil is implemented. These network analyzer software ought to be installed on dedicated machine.

3. Review traversing network packets

At this point, let's discuss IP packet traversing from one IP device to another. Depending on the application running on the IP device, there could be some TCP and UDP packet combination going back and forth between the two devices. By tapping the network where these packet flows take place, you could review the packet differences and similarities.

Typically packet flows in certain order. For TCP packets, there are Sequence Number labels on each packet to show such order. For those who are unfamiliar with TCP packet analysis, feel free to review the following.
Transmission Control Protocol - a Wikipedia

As of UDP packet, there may be certain Sequence Number mechanism on higher layer such as the Application layer since by default, UDP packets do not carry Sequence Number. For those who are unfamiliar with UDP packet analysis, feel free to review the following.
User Datagram Protocol - a Wikipedia

In regards of general IP protocol and TCP/IP stack info, feel free to review the following.
Internet protocol suite
RFC 1180: TCP/IP Tutorial

Notice if there are any missing packets between results coming from those multiple points of taps. Note the time of those missing packets occur. If the time frame of those missing packet occurrences match or closely match the time frame of the incident, then these missing packet issues could be the culprit.

4. Review Missing Packet Issues

Various situations could lead to missing packets depending at where the missing incident take place. A simple cable breaks could cause the issue. If the packet traverses over WAN, the WAN provider could drop the packets somehow. For traversing packet over LAN, switch buffer or QoS improper configuration could cause the switch to drop packets.

Feel free to review following FAQ for additional info.
»Cisco Forum FAQ »Improving Small Business network performance
»Cisco Forum FAQ »Getting around Layer-1/2 line error troubleshooting
»Cisco Forum FAQ »Basic Troubleshooting for Speed Issues with Cisco Equipment
»Cisco Forum FAQ »Slow connection through a router, firewall, or switch
»Cisco Forum FAQ »General Wifi connectivity issues
»Cisco Forum FAQ »Basic Network Troubleshooting
»Cisco Forum FAQ »High CPU Utilizations
»Cisco Forum FAQ »Checking traffic activities against ACL

5. Review Dropped Packet Issues

Whereas Missing Packet issue is a Layer-1/2 incidental issue, Dropped Packet issue is considered Layer-2/3 by-design issue. Specifically for Voice over IP (VoIP) phone packet traverse, certain LAN and WAN design incorporates QoS (Quality of Service) methodologies. Each QoS methodology has its own advantages and disadvantages.

One QoS technique is to reserve certain bandwidth allocation to be used by voice traffic. At a time of heavy-bandwidth usage, the voice traffic is guaranteed to keep flowing uninterrupted while data traffic will be delayed. At a time of light or no bandwidth usage, either voice or data traffic flow freely with no delay or interruption.

Different QoS technique is to drop data packet instead of delaying in a time of heavy-bandwidth usage while the voice traffic flow is guaranteed to proceed. Note that either techniques do not scale up when heavy-bandwidth usage occurs often.

Long-occurrence of heavy-bandwidth usage may cause the data packet being dropped often. Too many phone call going through at the same time will overwhelm the QoS guarantee mechanism, that at one point one of those voice traffic will be dropped as well. Either voice or data packet drop causes the voice or data application to break eventually.

For those who are unfamiliar with QoS and VoIP technologies, feel free to review following FAQ.
»Cisco Forum FAQ »Deploying VoIP - An Introduction
»Cisco Forum FAQ »Introduction to Voice over IP
»Cisco Forum FAQ »Working with Voice over IP protocols
»Cisco Forum FAQ »QoS Basic and Implementation
»Cisco Forum FAQ »How do I configure QoS for VoIP?
»Cisco Forum FAQ »Improving Performance of Cable/DSL Internet using QoS
»Cisco Forum FAQ »Router runs VoIP, Bit Torrent, Online Gaming; DynDNS - QoS Sample Configuration
»Cisco Forum FAQ »Troubleshooting QoS

by aryoba See Profile
last modified: 2014-05-15 10:10:19

ICMP Ping and Traceroute are probably common tools network engineers use to give basic network layout and understanding, though network security officers may not agree.

For those who like to test certain TCP-based application connectivity, network engineers use Telnet on specific port. As example, telnet to port 80 can be used to test whether certain web server is setup to receive incoming HTTP request. A successful HTTP service setup on a server responds with some OK message. An unsuccessful one gives out errors.

At a point of failure, you may want to find out which equipment that cause the failure. Just because there is error message, does not necessarily mean that the web server is not yet setup since some device between the host client and server could be blocking the HTTP request. A TCP-based traceroute tool would give you idea of which device it is that has been blocking.

TCP-based Traceroute tool
Tcproute.exe for Windows

by aryoba See Profile
last modified: 2017-04-11 13:02:05