dslreports logo
site
 
    All Forums Hot Topics Gallery
spc

spacer




how-to block ads


Search Topic:
uniqs
2355
share rss forum feed

jayde

join:2012-06-10

Diagnosing packet loss in realtime?

When diagnosing packet loss I usually turn to MTR on my Linux machine, but Cox is throttling ICMP. Is there any way to get around the ICMP throttle?

Here's what you get when using MTR on Cox



Do you guys know a reliable way to diagnose packet loss in realtime on Cox? Windows and/or Linux solutions are welcome-

PS- Throttling ICMP, blocking incoming port 80.. Cox is very restrictive to residential consumerse >_>


No_Strings
Premium,MVM,Ex-Mod 2008-13
join:2001-11-22
The OC
kudos:6
Would tcptraceroute help?

»michael.toren.net/code/tcptraceroute/


CoxTech1
VIP
join:2002-04-25
Chesapeake, VA
kudos:77
reply to jayde
Loss at the 2nd hop suggests a problem very close to home. If you go to »192.168.100.1 what do the signal levels look like?


Ender3rd

join:2001-07-15
Connecticut

2 edits
reply to jayde
Click for full size
Red is bad (packet loss)
PingPlotter software makes pretty pictures that help to form a nice historical record over a period of time. Make sure you ping to a destination within the Cox network. In the picture above I am pinging www.cox.net and displaying the packet loss to my first hop at 98.190.163.112. I have found that setting the ping interval to 2.5 seconds rather than the default 1.0 second interval alleviates the ICMP de-prioritization issue. Using a setting of 5.0 seconds seems to eliminate it entirely, unless there is a lot of net traffic through a specific server, but it also seems to paint a much more optimistic picture as far as packet loss goes.

Ender
--
"The dog days are over... The dog days are done..."

Rakeesh

join:2011-10-30
Mesa, AZ
Reviews:
·Sprint Mobile Br..
·Cox HSI

1 edit
reply to jayde
I never heard of MTR before today. I googled it and it says it's a combination of ping and traceroute...uh wut?

Ping uses ICMP-ECHO-REQUEST flag to a destination address, and measures the time it takes for the ICMP-ECHO-REPLY to come back.

Traceroute uses ICMP-ECHO-REQUEST flag, same as ping, with iteratively increasing TTL values, so each successive hop returns an ICMP-TIME-EXCEEDED flag.

So traceroute, which uses ping by definition, combined with ping makes MTR? Color me confused.

I'm not dogging it, I just can't find a description that explains what it does beyond traceroute.

-

Anyways, when I am trying really hard to find dropped packets, I'll ping flood google.com. That isn't bad like it sounds. What I'm doing is sending a flood of 48 byte packets (which includes estimated overhead) at a rate of 100 per second, and seeing if ANY of them get dropped over a period of 10 minutes. 5k bytes per second isn't going to cause a denial of service any time soon, so it is safe.


CoxVegas

join:2011-07-25
Las Vegas, NV
kudos:10
said by Rakeesh:

I'm not dogging it, I just can't find a description that explains what it does beyond traceroute.

MTR is a repeating traceroute - it does the same traceroute over and over again and reports on the statistics. In the screenshot above, it's done 224 and in the middle of the 225th traceroute. It's actually a pretty nice tool - it's a Linux console version of pingplotter essentially.


CoxVegas

join:2011-07-25
Las Vegas, NV
kudos:10
reply to Rakeesh
said by Rakeesh:

Anyways, when I am trying really hard to find dropped packets, I'll ping flood google.com. That isn't bad like it sounds. What I'm doing is sending a flood of 48 byte packets (which includes estimated overhead) at a rate of 100 per second, and seeing if ANY of them get dropped over a period of 10 minutes. 5k bytes per second isn't going to cause a denial of service any time soon, so it is safe.

Just FYI, that's Windows' definition of ping flood. Other OSes (Linux and probably Mac) define ping flood much higher (some define it 'as fast as the NIC will allow'):

coxvegas@secretserver:~# time ping -c 10000 -f mylittlepony.com
PING mylittlepony.com (12.154.191.10) 56(84) bytes of data.

--- mylittlepony.com ping statistics ---
10000 packets transmitted, 10000 received, 0% packet loss, time 19287ms
rtt min/avg/max/mdev = 1.679/1.892/6.504/0.125 ms, ipg/ewma 1.928/1.892 ms

real 0m19.292s
user 0m0.052s
sys 0m0.236s


DrDrew
That others may surf
Premium
join:2009-01-28
SoCal
kudos:16
reply to jayde
Use the -u switch on MTR and it will use UDP packets instead of ICMP packets.

nickphx

join:2009-10-29
Phoenix, AZ
huh?
I do not see a -u switch.
neither does mtr.
]#mtr -u google.com
mtr: invalid option -- u

]# man mtr
MTR(8) mtr MTR(8)

NAME
mtr - a network diagnostic tool

SYNOPSIS
mtr [-hvrctglspni46] [--help] [--version] [--report] [--report-cycles COUNT] [--curses] [--split] [--raw] [--no-dns] [--gtk]
[--address IP.ADD.RE.SS] [--interval SECONDS] [--psize BYTES | -s BYTES] HOSTNAME [PACKETSIZE]

DESCRIPTION
mtr combines the functionality of the traceroute and ping programs in a single network diagnostic tool.

As mtr starts, it investigates the network connection between the host mtr runs on and HOSTNAME. by sending packets with purposly
low TTLs. It continues to send packets with low TTL, noting the response time of the intervening routers. This allows mtr to print
the response percentage and response times of the internet route to HOSTNAME. A sudden increase in packetloss or response time is
often an indication of a bad (or simply overloaded) link.

OPTIONS
-h

--help
Print the summary of command line argument options.

-v

--version
Print the installed version of mtr.

-r

--report
This option puts mtr into report mode. When in this mode, mtr will run for the number of cycles specified by the -c option,
and then print statistics and exit.

This mode is useful for generating statistics about network quality. Note that each running instance of mtr generates a sig-
nificant amount of network traffic. Using mtr to measure the quality of your network may result in decreased network perfor-
mance.

-c COUNT

--report-cycles COUNT
Use this option to set the number of pings sent to determine both the machines on the network and the reliability of those
machines. Each cycle lasts one second.

-s BYTES

--psize BYTES

PACKETSIZE
These options or a trailing PACKETSIZE on the commandline sets the packet size used for probing. It is in bytes inclusive IP
and ICMP headers

-t

--curses
Use this option to force mtr to use the curses based terminal interface (if available).

-n

--no-dns
Use this option to force mtr to display numeric IP numbers and not try to resolve the host names.

-g

--gtk
Use this option to force mtr to use the GTK+ based X11 window interface (if available). GTK+ must have been available on the
system when mtr was built for this to work. See the GTK+ web page at »www.gimp.org/gtk/ for more information about
GTK+.

-p

--split
Use this option to set mtr to spit out a format that is suitable for a split-user interface.

-l

--raw
Use this option to tell mtr to use the raw output format. This format is better suited for archival of the measurement
results. It could be parsed to be presented into any of the other display methods.

-a IP.ADD.RE.SS

--address IP.ADD.RE.SS
Use this option to bind outgoing packets’ socket to specific interface, so that any packet will be sent through this inter-
face. NOTE that this option doesn’t apply to DNS requests (which could be and could not be what you want).

-i SECONDS

--interval SECONDS
Use this option to specify the positive number of seconds between ICMP ECHO requests. The default value for this parameter is
one second.

-4
Use IPv4 only.

-6
Use IPv6 only.

BUGS
Some modern routers give a lower priority to ICMP ECHO packets than to other network traffic. Consequently, the reliability of these
routers reported by mtr will be significantly lower than the actual reliability of these routers.

CONTACT INFORMATION
For the latest version, see the mtr web page at »www.bitwizard.nl/mtr/.

Subscribe to the mtr mailing list. All mtr related announcements are posted to the mtr mailing list. To subscribe, send email to
majordomo@lists.xmission.com with subscribe mtr in the body of the message. To send a message to the mailing list, mail to
mtr@lists.xmission.com.

Bug reports and feature requests should be sent to the mtr mailing list.

SEE ALSO
traceroute(8), ping(8).


DrDrew
That others may surf
Premium
join:2009-01-28
SoCal
kudos:16

1 edit
Odd, my version has it. Online man entry shows it too: »man.he.net/man8/mtr

[edit] I'm running 0.82

Rakeesh

join:2011-10-30
Mesa, AZ
Reviews:
·Sprint Mobile Br..
·Cox HSI

3 edits
reply to CoxVegas
said by CoxVegas:

Just FYI, that's Windows' definition of ping flood. Other OSes (Linux and probably Mac) define ping flood much higher (some define it 'as fast as the NIC will allow'):

I do ping floods from either my internal server (runs ubuntu) or my asus switch, which is running tomato (a derivative of the linux that was included on the original linksys routers.)

The ping tool included on both doesn't do the -f option, however they do the -i option which lets you specify the interval in seconds. The minimum it allows for a regular user is 200ms, and it calls anything below that a flood:

quote:
jjd@files:~$ ping -i .1 google.com
PING google.com (74.125.224.168) 56(84) bytes of data.
ping: cannot flood; minimal interval, allowed for user, is 200ms
So you have to su to do anything lower. It doesn't seem to work with any value lower than .01 though, (i.e. lowering the interval doesn't seem to ping faster than 100 per second) so that is what I use. For my internal purposes, that level of granularity is good enough.

I know all of the redhat distributions and their derivatives include the -f option, is that what you are using? I could certainly imagine why somebody in your position would need much higher granularity. It's nice having these tools available isn't it? Linux and all of the coreutils provide such a great tool for networking, I love it.

By the way, I like how you guys talk networking with the community, it speaks volumes about your work environment probably being a rather friendly one, at least in the particular office you work at. I had an internship like that, once I finish my university degree I hope to have another job like that.


CoxVegas

join:2011-07-25
Las Vegas, NV
kudos:10
said by Rakeesh:

The minimum it allows for a regular user is 200ms, and it calls anything below that a flood:

Since you're on Ubuntu, try an "apt-get install hping3" - that version of ping has a LOT of fun/interesting options to tinker with.

said by Rakeesh:

I know all of the redhat distributions and their derivatives include the -f option, is that what you are using?

That particular box is a Debian base, so I'm surprised your Ubuntu image doesn't have it, being based on Debian.

said by Rakeesh:

I could certainly imagine why somebody in your position would need much higher granularity. It's nice having these tools available isn't it? Linux and all of the coreutils provide such a great tool for networking, I love it.

We use a lot of Linux boxes (actually VMs, any more) for a large variety of little one-off purposes (and some not quite one-off).

said by Rakeesh:

By the way, I like how you guys talk networking with the community, it speaks volumes about your work environment probably being a rather friendly one, at least in the particular office you work at. I had an internship like that, once I finish my university degree I hope to have another job like that.

Heh, some of us on here are dye-in-the-wool network engineers (including myself) - we've been doing this for a long time and like doing it.

And yes, the office can be pretty amusing at times. Although I'm disappointed no one commented on my choice of ping targets!

Rakeesh

join:2011-10-30
Mesa, AZ
Reviews:
·Sprint Mobile Br..
·Cox HSI

2 edits
said by CoxVegas:

Heh, some of us on here are dye-in-the-wool network engineers (including myself) - we've been doing this for a long time and like doing it.

And yes, the office can be pretty amusing at times. Although I'm disappointed no one commented on my choice of ping targets!

I thought perhaps it was fictitious. Often the little guys don't take kindly to ping floods, even if they only use a tiny bit of bandwidth and compute resources.

Rarely the big guys do it too, microsoft for example blocks incoming pings on basically every public facing server they have, citing security concerns. Not sure why - generally public facing services should have these services available for their own troubleshooting purposes. Public addresses with no public services on the other hand...yeah, probably best to eliminate any possible surveillance at all. FWKNOPD is a nice tool for that by the way, look into it if you haven't, it's really neat; I think it is one of those current day "hacks" that will one day become standard given the problems we have with bots constantly probing login credentials on SSH ports (even when you use non-standard ports.) Traceroute for example started out as just a hack somebody did.

Google on the other hand seems to be fine with being a frequent network troubleshooting target. It helps a lot, especially considering that they use anycast IP addresses.

I love networking too by the way. In my CCNA courses, we had these case studies where we configure a mock network that were supposed to take a few weeks to solve. In each of the four courses finished them in under 8 hours, usually just picking a saturday to knock the whole thing out. Everybody else was stressing over it, but I had fun doing it. I just finished going through the entire CCNP course track. CCNP route was hard as hell, OSPF and BGP theory being the major sticklers. I still need to review it in better detail before I take the route exam.

I had to have the local cox installers come out here a few times due to signaling issues (new house) and when it came to troubleshooting they didn't understand half of the things I was doing. When I showed one of them how I was doing a ping flood to ensure the packet loss issue was resolved, he sort of had this panic like "won't they get angry if you do that?" But in their defense, they at least understood DOCSIS to a degree, which I have little to no knowledge of.

EDIT: By the way, if you've ever heard of the Avnet tech games, I was on the three person team that came in first in the cisco battle in 2012, and did the same thing again in 2013. I also won the patch panel contest in 2013. First place in each contest was $1,000, so I won $3,000 overall.

nickphx

join:2009-10-29
Phoenix, AZ
reply to DrDrew
sweet.. maybe i need to update that crusty old box.. lol

jayde

join:2012-06-10
reply to jayde
Thanks for the tip about '-u', that solved my problem entirely!

Also.. is CoxVegas.. a brony?


CoxVegas

join:2011-07-25
Las Vegas, NV
kudos:10
reply to Rakeesh
said by Rakeesh:

I thought perhaps it was fictitious. Often the little guys don't take kindly to ping floods, even if they only use a tiny bit of bandwidth and compute resources.

It was... I edited the hostname and IP to be correct, but was flooding my own local switch.

said by Rakeesh:

Rarely the big guys do it too, microsoft for example blocks incoming pings on basically every public facing server they have, citing security concerns. Not sure why - generally public facing services should have these services available for their own troubleshooting purposes.

Hysteria still around from »en.wikipedia.org/wiki/Ping_of_death mostly.

said by Rakeesh:

I had to have the local cox installers come out here a few times due to signaling issues (new house) and when it came to troubleshooting they didn't understand half of the things I was doing. When I showed one of them how I was doing a ping flood to ensure the packet loss issue was resolved, he sort of had this panic like "won't they get angry if you do that?" But in their defense, they at least understood DOCSIS to a degree, which I have little to no knowledge of.

Yep, the ages-old OSI Layer 1 versus Layer 2 on up divide.


CoxVegas

join:2011-07-25
Las Vegas, NV
kudos:10
reply to jayde
said by jayde:

Also.. is CoxVegas.. a brony?

(consults Google)

Nope. It was just what was on the TV by my cube at the time I wrote the message and made me chuckle.

Rakeesh

join:2011-10-30
Mesa, AZ
Reviews:
·Sprint Mobile Br..
·Cox HSI
reply to CoxVegas
said by CoxVegas:

Yep, the ages-old OSI Layer 1 versus Layer 2 on up divide.

Speak of that, does DOCSIS use its own layer 2 protocol? Or is it just ethernet only on a different medium?