 d_lBarsoomPremium,MVM join:2002-12-08 Reno, NV kudos:7 | Afternoon-evening speed slowdown in Reno I have an afternoon-evening throughput slowdown on my connection. Typically throughput speeds will begin to slowdown at about 2 to 3 PM in the afternoon and then the speeds will gradually deteriorate until about 6 to 10 PM at which point they will begin a fairly rapid recovery until about 10 PM to midnight when they are completely back to normal. Note that upload speeds are never affected.
This speed slow down is NOT associated with any increase in first hop latency (always at 7-8 ms), nor with any increased latency along the routes to the test servers, nor with any packet loss. So it doesn't appear to be a pure TCP/IP layer problem. This problem is not a physical layer problem either. My sync speeds are perfect and I now have minimal CRC errors with a 4100.
I'm connected to Redback 1 at the Reno NOC: PPPoX-pool rback1 SBCIS-101917-215843 (NET-64-169-4-0-1)
I believe the problem began in early November, but at that time I was using a ZyXEL modem that had CRC problems so those masked the underlying slowdown problem. It was definitely present when I switched to the 4100 modem over Thanksgiving. When the SBC speed test was fixed, I set up some spreadsheets to collect, parse, and plot speed stats. I initially thought that this problem only occurred after school hours until the kids were in bed; however, testing over the holidays indicate that the slow down pattern was still present. Normally at its worst, the throughput speeds will deteriorate to about 50% of normal. On the 18th, they dropped to 30% of normal.
The following are daily plots of multiple speed tests taken throughout the day. For internal consistency, three sites were chosen to test against: the SBC Pleasanton Java site which is normally the most consistent and fastest of the SBC site for me; the Santa Clara or Stanford NDT Java tests which yield consistent results and have minimal number of hops off the SBC backbone; and a cross country, FTP download test of about 11 MB in size from my Prodigy personal pages. All routing for this FTP download is over SBC backbones.
 12-22-05
 12-30-05
 12-31-05
 01-17-06
 01-18-06
The slow down also happened again last night (the 19th), but I did not track it in detail. It started later than normal, slowed to at least 50% of normal, and had not fully recovered by 10 PM. |
|
 d_lBarsoomPremium,MVM join:2002-12-08 Reno, NV kudos:7 | In an effort to make more sense out of the data scatter on the above graphs, I've averaged the data points by a moving average on each of the three different speed tests and then averaged the averaged Pleasanton, NDT, and FTP results with a respective 4:3:2 weighting to give a summary speed at the time each test was made. Obviously this much averaging will greatly smooth the throughput drop extremes, but the timing of these drops should not be shifted.
 12-22-05
 12-30-05
 12-31-05
 01-17-06
 01-18-06
|
|
 tonydiPremium,MVM join:2001-05-11 San Jose, CA | Excellent stuff! Maybe they should let you work on finding the fix.  |
|
|
|
 d_lBarsoomPremium,MVM join:2002-12-08 Reno, NV kudos:7 | reply to d_l In order to understand what is happening at the packet level and eliminate the possibility of any TCP/IP layer problems, ethereal captures and throughput analyses were made on some Pleasanton speed tests during the slowdowns and during normalcy. The speed as reported by the Java test was added to the graphs in red. Note that ethereal plots the graphs in bytes/sec and the speed tests are in Mbits/sec.
 Normalcy
 Slight impairment at the end of the download
 Moderate impairment
 Strong impairment
The speeds of the individual packets can be seen to fluctuate more widely as the reported speeds become more impaired. No TCP/IP problems such as re-transmits or packet losses were recorded during the download portion of the tests. |
|
 d_lBarsoomPremium,MVM join:2002-12-08 Reno, NV kudos:7 | reply to d_l
 01-20-06 |
A definite improvement last night, but there still was evidence of an intermittent slow down. Maybe there wasn't as much of a load on the system on a Friday night. |
|
 Reviews:
·Charter
| reply to d_l I'll start doing some speed tests from my end of town (northwest, Bridgewood CO) to see if I've got a problem. I run Ping Plotter 24/7 to check from connection problems, but it wouldn't necessarily show a speed issue.
Here's my 7AM test for comparative purposes »support.sbcglobal.net/cgi-bin/sp···obal.net |
|
 d_lBarsoomPremium,MVM join:2002-12-08 Reno, NV kudos:7 | reply to d_l Weekend results. Only brief, intermittent, slow downs on Saturday (some were curiously concentrated in the morning hours between xx:10 and xx:20), but a pretty consistent afternoon-evening slow down on Sunday.
 01-21-06
 01-22-06
Zero first hop packet loss and no latency increase even during the worst of the slowing as checked by PingPlotter and DOS pings. |
|
 d_lBarsoomPremium,MVM join:2002-12-08 Reno, NV kudos:7 | reply to d_l  01-23-06
 01-25-06
Some more speed graphs. I made a change yesterday that may have restricted the length of the slow down period (changed the starting/stopping time), but certainly didn't help the intensity of the slowing. |
|
 Reviews:
·Charter
| I'm probably not much help. I keep forgetting to do an evening speed check until after peak hours. So far here's some test results I've writen down (all to the SF test site)...
1/23 21:27 2396/438 1/24 19:37 2540/439 1/24 20:37 2436/439 1/24 22:40 2526/438 1/25 21:53 2495/438
So are sitting by your computer clicking the speed test every 10 minutes, or using some automated method to test? |
|
 d_lBarsoomPremium,MVM join:2002-12-08 Reno, NV kudos:7 | It's about half automated, but I have to be at the computer. Judging from those results, I don't think you are affected. It must be due to your different rback. |
|
 Reviews:
·Charter
| I ran 6 tests between 18:44 - 20:07 that ranged from 2551/441 on the high side and 2417/438 on the low side. Looks like my area is unaffected.
Interestingly, I ran 3 tests while I was on my VOIP line, which lowered my speed to an average of 2398/349. While running the test my call quality suffered a bit. |
|
 d_lBarsoomPremium,MVM join:2002-12-08 Reno, NV kudos:7 1 edit | reply to d_l
 02-06-06 |  02-07-06 |  averaged 02-06-06 |  averaged 02-07-06 |
The latest speed test results from yesterday and the day before with averaged graphs. The test results from the day before are as bad (or possibly the worst) as I've ever seen, but there was some improvement last night. |
|
 d_lBarsoomPremium,MVM join:2002-12-08 Reno, NV kudos:7 | reply to d_l
 02-08-06 |  averaged 02-08-06 |
More improvement. |
|
 d_lBarsoomPremium,MVM join:2002-12-08 Reno, NV kudos:7 | reply to d_l
 02-09-06 |  averaged 02-09-06 |
 |
|
 d_lBarsoomPremium,MVM join:2002-12-08 Reno, NV kudos:7 | reply to d_l
 02-14-06 |  averaged 02-14-06 |
Well there is no question in my mind that the speeds are deteriorating as this problem continues. Since starting these measurements almost two months ago, I had to change the speed scale on the graphs because the slow downs are 1000 kbps slower now than in December. Last night I was contemplating having to lower the scale again.
I've also switched to plotting a percentage of maximum normal throughput on the averaged graphs. Last night's averaged speed dropped to as low as 35%. Previously no averaged speed has ever dropped below 45% and usually about 50% was normal worst case each night. Individual speed tests have been well below 40% at times the past month, but last night they were consistently so. |
|
 darthfadarOnly Gonna Tell You Once join:2001-08-25 Mcminnville, OR | Destroying our speeds as we knew them...
The SBC/ATT Deathstar... |
|
 d_lBarsoomPremium,MVM join:2002-12-08 Reno, NV kudos:7 | reply to d_l
 03-07-06 |  averaged 03-07-06 |
After an hour and a half outage on Monday night during which my modem failed the "Perform ATM OAM segment ping" test, I had these speed test results the following day. Notice that the scales on the speed axis had to be changed from those in the previous graphs. 
The first hop latency averaged 40 to 120 ms with spikes as high as 350 ms before 3:00 PM. There were sporadic packet losses of about 1%. After 3:00 PM, latency returned to the normal 7 ms. |
|