| Cisco 3925 web traffic slows to a crawl I have a Cisco 3925 ISR I use to connect my 100Mbps ethernet connection. About every 12 hours, all surfing or web traffic slows to a crawl (takes minutes to load even the basic pages). All other traffic appears to be passing just fine. A simple reboot of the router and all web traffic speeds return to normal.
Basic router configuration with NATs. No firewall or other rules in place. Router averages 70Mbps / 10 Mbps at peak times.
Any ideas on what might be causing this? |
|
|
|
pearcy
join:2004-12-08
Chicago, IL | what troubleshooting steps have you taken so far? |
|
| I have not tried any configuration changes. Mostly just the basics - checked log files, checked stats and not seeing anything out of the ordinary. |
|
sk1939Premium
join:2010-10-23
Washington, DC
kudos:9 | Whatkind of internet traffic? Just plain HTTP, or HTTPS, or are you streaming, using VPN, etc? |
|
| Just plain http traffic. Streaming seems to be fine |
|
sk1939Premium
join:2010-10-23
Washington, DC
kudos:9 | Post a show run minus password and such. |
|
nosx
join:2004-12-27
00000
kudos:5 | Your default route points out the right interface but does not specify a next hop. This is one cause i have seen before of that exact problem.
Change your default route to either be learned from your upstream provider, either via DHCP or a routing protocol, OR include a next hop IP address with the static route. |
|
| reply to sk1939
Building configuration...
Current configuration : 7426 bytes
!
! Last configuration change at 18:26:08 Chicago Wed Oct 26 2011 by xxxx
!
version 15.1
no service pad
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime msec localtime show-timezone
service timestamps log datetime msec localtime show-timezone
service password-encryption
service sequence-numbers
!
hostname xxxxx
!
boot-start-marker
boot system flash0 c3900-universalk9-mz.SPA.151-3.T.bin
boot-end-marker
!
!
security authentication failure rate 3 log
security passwords min-length 6
logging buffered 51200 informational
enable secret 5 xxxxx
!
no aaa new-model
!
clock timezone Chicago -6 0
clock summer-time Chicago date Apr 6 2003 2:00 Oct 26 2003 2:00
!
crypto pki token default removal timeout 0
!
crypto pki trustpoint TP-self-signed-1224775824
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-1224775824
revocation-check none
rsakeypair TP-self-signed-1224775824
!
crypto pki trustpoint test_trustpoint_config_created_for_sdm
subject-name e=sdmtest@sdmtest.com
revocation-check crl
!
!
crypto pki certificate chain TP-self-signed-1224775824
certificate self-signed 01
xxxxx
quit
crypto pki certificate chain test_trustpoint_config_created_for_sdm
no ipv6 cef
no ip source-route
ip cef
!
!
!
!
!
no ip bootp server
no ip domain lookup
ip domain name xxxxx
ip name-server 8.8.8.8
!
multilink bundle-name authenticated
!
!
license udi pid C3900-SPE100/K9 sn FOC14300RYB
license boot module c3900 technology-package securityk9
!
!
username xxxxx privilege 15 secret 5 xxxxx/
!
redundancy
!
crypto key pubkey-chain rsa
named-key realm-cisco.pub
key-string
30820122 300D0609 2A864886 F70D0101 01050003 82010F00 3082010A 02820101
00C19E93 A8AF124A D6CC7A24 5097A975 206BE3A2 06FBA13F 6F12CB5B 4E441F16
17E630D5 C02AC252 912BE27F 37FDD9C8 11FC7AF7 DCDD81D9 43CDABC3 6007D128
B199ABCB D34ED0F9 085FADC1 359C189E F30AF10A C0EFB624 7E0764BF 3E53053E
5B2146A9 D7A5EDE3 0298AF03 DED7A5B8 9479039D 20F30663 9AC64B93 C0112A35
FE3F0C87 89BCB7BB 994AE74C FA9E481D F65875D6 85EAF974 6D9CC8E3 F0B08B85
50437722 FFBE85B9 5E4189FF CC189CB9 69C46F9C A84DFBA5 7A0AF99E AD768C36
006CF498 079F88F8 A3B3FB1F 9FB7B3CB 5539E1D1 9693CCBB 551F78D2 892356AE
2F56D826 8918EF3C 80CA4F4D 87BFCA3B BFF668E9 689782A5 CF31CB6E B4B094D3
F3020301 0001
quit
!
!
!
!
ip tcp synwait-time 10
!
!
!
!
!
!
!
interface Null0
no ip unreachables
!
interface GigabitEthernet0/0
description $ETH-SW-LAUNCH$$INTF-INFO-GE 0/0$$ETH-WAN$$FW_OUTSIDE$
ip address xxx.xxx.xxx.xxx 255.255.255.252
no ip redirects
no ip unreachables
no ip proxy-arp
ip nbar protocol-discovery
ip flow ingress
ip nat outside
ip virtual-reassembly in max-reassemblies 64
ip verify unicast reverse-path
duplex full
speed 100
no mop enabled
!
interface GigabitEthernet0/1
description $ES_LAN$$ETH-LAN$$FW_INSIDE$
ip address 172.16.1.1 255.240.0.0
no ip redirects
no ip unreachables
no ip proxy-arp
ip flow ingress
ip nat inside
ip virtual-reassembly in
duplex auto
speed auto
no mop enabled
!
interface GigabitEthernet0/2
description $FW_INSIDE$$ETH-LAN$
ip address xxx.xxx.xxx.xxx 255.255.255.0
no ip redirects
no ip unreachables
no ip proxy-arp
ip flow ingress
ip virtual-reassembly in
duplex auto
speed auto
no mop enabled
!
interface GigabitEthernet0/1/0
!
interface GigabitEthernet0/1/1
!
interface GigabitEthernet0/1/2
!
interface GigabitEthernet0/1/3
!
interface GigabitEthernet0/1/4
!
interface GigabitEthernet0/1/5
!
interface GigabitEthernet0/1/6
!
interface GigabitEthernet0/1/7
!
interface Vlan1
description $FW_INSIDE$
ip address xxx.xxx.xxx.xxx 255.255.255.0
no ip redirects
no ip unreachables
no ip proxy-arp
ip flow ingress
!
ip forward-protocol nd
!
ip http server
ip http access-class 23
ip http authentication local
ip http secure-server
ip http timeout-policy idle 60 life 86400 requests 10000
ip flow-top-talkers
top 100
sort-by bytes
!
ip nat inside source list 1 interface GigabitEthernet0/0 overload
ip nat inside source static 172.16.4.124 xxx.xxx.xxx.124
ip nat inside source static 172.16.16.199 xxx.xxx.xxx.201
ip nat inside source static 172.16.12.152 xxx.xxx.xxx.202
ip nat inside source static 172.16.16.16 xxx.xxx.xxx.203
ip nat inside source static 172.16.12.220 xxx.xxx.xxx.204
ip nat inside source static 172.16.8.53 xxx.xxx.xxx.205
ip nat inside source static 172.16.20.210 xxx.xxx.xxx.206
ip nat inside source static 172.16.4.110 xxx.xxx.xxx.207
ip nat inside source static 172.16.12.98 xxx.xxx.xxx.209
ip nat inside source static 172.16.9.224 xxx.xxx.xxx.210
ip nat inside source static 172.16.97.11 xxx.xxx.xxx.211
ip nat inside source static 172.16.37.21 xxx.xxx.xxx.212
ip nat inside source static 172.16.9.39 xxx.xxx.xxx.213
ip nat inside source static 172.16.25.166 xxx.xxx.xxx.214
ip nat inside source static 172.16.13.166 xxx.xxx.xxx.215
ip nat inside source static 172.16.57.3 xxx.xxx.xxx.220
ip route 0.0.0.0 0.0.0.0 xxx.xxx.xxx.xxx permanent
!
logging xxx.xxx.xxx.xxx
access-list 1 remark CCP_ACL Category=2
access-list 1 permit 172.16.0.0 0.15.255.255
access-list 23 permit xxx.xxx.xxx.xxx 0.0.0.255
access-list 23 permit xxx.xxx.xxx.xxx 0.15.255.255
access-list 23 permit xxx.xxx.xxx.xxx 0.0.0.255
!
no cdp run
!
!
!
!
!
control-plane
!
!
banner exec ^CAUTHORIZED USERS ONLY!^C
banner login ^C
-----------------------------------------------------------------------
AUTHORIZED USERS ONLY!
All login attempts are monitored and logged.
If you are not an authorized user, please disconnect now.
-----------------------------------------------------------------------
^C
!
line con 0
login local
transport output telnet
line aux 0
login local
transport output telnet
line vty 0 4
access-class 23 in
privilege level 15
login local
transport input telnet ssh
line vty 5 15
access-class 23 in
privilege level 15
login local
transport input telnet ssh
!
scheduler allocate 20000 1000
scheduler interval 500
ntp update-calendar
ntp server 207.46.232.182 source GigabitEthernet0/0
end |
|
| reply to nosx
Thanks nosx, I'll check that out.
Any other advice? |
|
sk1939Premium
join:2010-10-23
Washington, DC
kudos:9 Reviews:
 ·T-Mobile US
| Try changing this "no ip redirects" to "ip redirect". That will cause it to send a ICMP Redirect to the client pointing it to another next hop, rather than itself, for a given destination in hopes the client will take this new next hop to this destination. |
|
| reply to zrob_12
Looks pretty bogstandard config. Any sort of syslogs / monitoring on the device, interface loads, etc?
How's 'show proc cpu [history]' and 'show proc mem' look? What about extended pings from the 3925 to
the next hop device look?
Regards |
|
| I just turned on logging to see if there was anything abnormal.
CPU between 20-30% at all times (even when http traffic is slow).
Proc consistently under 20% as well.
Pings look good to both interfaces of the router and look good to external sites as well. |
|
| reply to zrob_12
type
sh deb
if anything shows do a
u all |
|
| reply to zrob_12
WAN interface was logging “fragment table has reached its maximum threshold 16”. I configured “ip virtual-reassembly max-reassemblies” to 64. I don’t see the errors anymore. Waiting to see if issue resolved. |
|
| reply to Da Geek Kid
When I type "sh deb" nothing appears. |
|
| reply to zrob_12
ok... now why would you do ip vfr???
VFR will cause a performance impact on the basis of functions such as packet copying, fragment validation, and fragment reorder. This performance impact will vary depending on the number of concurrent IP datagram that are being reassembled. |
|
| I understand that when NAT is enabled on an interface, VFR is automatically enabled on that interface. We were logging max reassemblies of 16 reached; hence the increase to 64. Errors are discontinued; router goes longer but still requires reboot after about 12 hours. |
|
| reply to zrob_12
What's the running software on the device? Can you provide a show ver?
May also want to involve TAC and see if they have any thoughts.
Regards |
|
nosx
join:2004-12-27
00000
kudos:5 | Paste output from:
show ip nat stat
show cef interface gig0/0
show ip cef gig0/0
show cef not-cef-switched
show ip virtual-reassembly |
|
| Added ip route-cache flow to all interfaces.
Default route changed to:
ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/0 xxx.xxx.xxx.xxx permanent
Previously, the default route pointed LAN interfaces to IP address of upstream perimeter router. The route now points to the WAN interface on my router, with next hop set to upstream router, per nosx recommendation. Will try static routes next if we still are having issues.
Here are the shows nosx requested.
orb-c3925-rtr0#sh ip virtual-rea
GigabitEthernet0/0:
Virtual Fragment Reassembly (VFR) is ENABLED [in]
Concurrent reassemblies (max-reassemblies): 64
Fragments per reassembly (max-fragments): 32
Reassembly timeout (timeout): 3 seconds
Drop fragments: OFF
Current reassembly count:0
Current fragment count:0
Total reassembly count:4673
Total reassembly timeout count:69
GigabitEthernet0/1:
Virtual Fragment Reassembly (VFR) is ENABLED [in]
Concurrent reassemblies (max-reassemblies): 16
Fragments per reassembly (max-fragments): 32
Reassembly timeout (timeout): 3 seconds
Drop fragments: OFF
Current reassembly count:0
Current fragment count:0
Total reassembly count:287
Total reassembly timeout count:2
GigabitEthernet0/2:
Virtual Fragment Reassembly (VFR) is ENABLED [in]
Concurrent reassemblies (max-reassemblies): 16
Fragments per reassembly (max-fragments): 32
Reassembly timeout (timeout): 3 seconds
Drop fragments: OFF
Current reassembly count:0
Current fragment count:0
Total reassembly count:0
Total reassembly timeout count:0
orb-c3925-rtr0#sh ip nat stat
Total active translations: 23383 (16 static, 23367 dynamic; 23367 extended)
Peak translations: 29805, occurred 00:10:33 ago
Outside interfaces:
GigabitEthernet0/0
Inside interfaces:
GigabitEthernet0/1
Hits: 29158894 Misses: 0
CEF Translated packets: 28818616, CEF Punted packets: 340628
Expired translations: 592576
Dynamic mappings:
-- Inside Source
[Id: 1] access-list 1 interface GigabitEthernet0/0 refcount 22019
Total doors: 0
Appl doors: 0
Normal doors: 0
Queued Packets: 0
orb-c3925-rtr0#show cef interface gig0/0
GigabitEthernet0/0 is up (if_number 3)
Corresponding hwidb fast_if_number 3
Corresponding hwidb firstsw->if_number 3
Internet address is xxx.xxx.xxx.xxx/30
ICMP redirects are never sent
Per packet load-sharing is disabled
IP unicast RPF check is enabled
Input features: Stateful Inspection, Ingress-NetFlow, Virtual Fragment Reassembly, Virtual Fragment Reassembly After I
PSec Decryption, uRPF, NAT Outside
Output features: Post-routing NAT Outside, Stateful Inspection, Post-Ingress-NetFlow
IP policy routing is disabled
BGP based policy accounting on input is disabled
BGP based policy accounting on output is disabled
Hardware idb is GigabitEthernet0/0
Fast switching type 1, interface type 27
IP CEF switching enabled
IP CEF switching turbo vector
IP prefix lookup IPv4 mtrie 8-8-8-8 optimized
Input fast flags 0x404040, Output fast flags 0x10100
ifindex 3(3)
Slot Slot unit 0 VC -1
IP MTU 1500
orb-c3925-rtr0#show cef not-cef-switched
% Command accepted but obsolete, see 'show (ip|ipv6) cef switching statistics [feature]'
IPv4 CEF Packets passed on to next switching layer
Slot No_adj No_encap Unsupp'ted Redirect Receive Options Access Frag
RP 0 0 365973 0 370383 0 0 0
orb-c3925-rtr0# |
|
