how-to block ads
|
|
Share Topic
 |
 |
|
|
|
 UHFAll static, all day, ForeverPremium,MVM
join:2002-05-24
 ·Callcentric
 ·DIRECTV
 ·surpasshosting
·Dish Network
·VOIPo
| reply to WizLayer
Re: arp info overwritten (OpenBSD) I've been getting ARP floods here for a long time. None of the Mediacom techs that post here seem to want to touch the posts about them with a 10 foot pole... I'm not sure why that is, but I suspect a router problem is causing it. 99.7% of the traffic seen by my modem is ARP requests, according to my Snort reports. | |
 BAINCHPremium,VIP,MVM
join:2003-04-02
Middletown, NY
kudos:10 | I'll touch it, not that I'm a tech. We are investigating. It doesn't appear to be a router issue. I will report more soon. | | |
|
| reply to UHF
said by UHF:
I've been getting ARP floods here for a long time. None of the Mediacom techs that post here seem to want to touch the posts about them with a 10 foot pole... I'm not sure why that is, but I suspect a router problem is causing it. 99.7% of the traffic seen by my modem is ARP requests, according to my Snort reports.
Greetz, mon...
Arp traffic is a necessity with DHCP service. ARP = "Address Resolution Protocol" = the protocol used to help maintain meaningful conversation between a gateway server and every client that connects via DHCP. In more detail, the ARP protocol is used to verify whether or not an IP address is in use (so the server knows whether or not it can lease the IP address out to someone else). As you know, Mediacom has hundreds of customers connected to the same gateway that you are connected to. Therefore, there are lots of ARP requests flying around...
When you do a tcpdump (snort, or whatever), you will notice hundreds of such packets being broadcast ("broadcast" = sent to the entire network [ff:ff:ff:ff:ff:ff], and not an individual connection). Unless you are getting hundreds of arp type packets addressed _only_ to you (ie, addressed specifically to _your_ IP address), then you are in no way being flooded. That's just the way this type of network setup works.
Check out »www.rfc-editor.org/rfc/rfc2131.txt. It's everything you wanted to know (and probably more) about how DHCP works. If you find that the link didn't make sense, then I'm sure there are other sites out there that can explain it to an easier crowd... Google is your friend.
"Arp info" is changed _only_ when the MAC addresses of the connected computers have changed. Any such change which could imply a security breach _should_ be reported, so the reports I'm getting are actually a good thing.
To update the thread, I just received a reply from the tech reps the other day, and it turns out there was a problem with one of their routers (it was reloading 3-4 times a day for some reason). They've replaced the defunct router, and the connection has gotten _much_ better.
I'm still having to reset the modem (even though I can ping the gateway and get reply). That leads me to believe that there are still issues, but it's nothing compared to what it was. I don't mind the problems so much, as long as I know that they're being worked on... (and I like to know what's going on too because I'm still learning).
Oh... And the pings I mentioned earlier in the thread... The reason I couldn't get echo requests beyond the gateway is because mediacom drops them both ways (kind of a bummer, but hey... I drop _everything_ incoming unless it's return, so I can certainly understand and even appreciate their measures).
Cheers,
WizLayer
btw... If I wanted to visit the local Mediacom station (which, for me is in Moultrie), does anyone know what the policy on that would be? Does Mediacom allow people to come in and get a tour of the place? I'd love to look around and ask lots of questions.
[text was edited by author 2003-10-24 02:24:55] | |
| I've posted about the ARP flooding before - never got any rational response.
What ARP is for is mapping an IP address to a physical (Ethernet in most cases) address. When you see an ARP REQUEST, that's a host asking:
I'm at 1.2.3.4, Ethernet 12:34:56:78:90:12 -
what is the ethernet addr for ip addresses A.B.C.D
These are always BROADCAST. Most hosts will see the broadcast and remember and cache the source information. The host that has ip address A.B.C.D sends an ARP RESPONSE (typically directly to the sender, but sometimes in broadcast as well)
hey you at 1.2.3.4, Ethernet 12:34:56:78:90:12 -
I'm ip addr A.B.C.D, ethernet addr AB:CD:EF:GH:IJ:KL
I think this really has very little to do with DHCP (which is a way for a host to discover it's IP address).
Anyway, most hosts keep caches of ARP mappings - i.e. they remember that IP address A.B.C.D mapped to Ethernet address AB:CD:EF:GH:IJ:KL - usually for 20 minutes or so. As traffic passes, these mappings are updated, so if a host is relatively active, it's ARP mapping should essentially never time out. However, this cache is limited in size, so if it fills up, entries will drop out, requiring them to be re-discovered. Routers are no different in this - they should cache ARP table mappings for a period of time.
I expect there are some tweaks to this process in a Cable Modem DOCSIS environment, but I haven't downloaded the DOCSIS documents to figure out what they might be.
I believe the ARP storms we see are probably due to the fact that many cable modem nodes are running with fairly large sub-nets - we've got a /22 here - which means there are 1024 nodes on the same sub-net. I expect this is too large for the local routers to handle - their ARP caches fill up, so they end up dropping nodes out, and having to constantly rediscover them.
This could also be due to a misconfiguration: if the router thinks it's sub-nets are larger than they really are, then if it thinks it has traffic for an IP address in one of it's sub-nets, it'll try to ARP for that host. But if the host is not there, either it's off or it's on a different sub-net, it'll get no response and will keep trying. This is the only way I can explain why the gateways are ARP'ing for the same IP address over and over again frequently.
And, with the various viruses running around, probing every possible IP address, that really pushes up the problem as now the routers are seeing traffic for *every* IP address in the sub-net, regardless of if the host is there or not. This really kills the ARP cache.
I would still like Mediacom to do something about it. The ARP traffic, while small, chews up bandwidth. Since cable modems are shared, we're all paying the price for this in more latency and less overall available bandwidth. It also takes processing time on the routers, hurting performance there as well.
Oh, one other thing: I've had @Home/Mediacom cable modem at the same location since it first become available, almost 5 years ago. The ARP storm problem only really became noticeable in the last year or so, which is why I think it's a configuration/router problem. Overall, I'm quite happy with Mediacom, although I wish they'd give us a little more upstream BW.
[text was edited by author 2003-10-28 09:50:20] | |
UHFAll static, all day, ForeverPremium,MVM
join:2002-05-24 Reviews:
·Callcentric
·DIRECTV
·surpasshosting
·Dish Network
·VOIPo
| said by TazMainiac:
I've posted about the ARP flooding before - never got any rational response.
What ARP is for is mapping an IP address to a physical (Ethernet in most cases) address.
I know what ARP is, what I want to know is, why do I see ARP requests for the same host multiple times per minute?? That indicates a problem. Since Bainch is looking into it, I assume it will eventually get better. I don't know who that guy is, but he seems to be able to get things done around there. | |
| said by UHF:
I know what ARP is, what I want to know is, why do I see ARP requests for the same host multiple times per minute??
That's the same thing I posted about before and got no response. I can frequently count a dozen or more requests for the SAME host from the gateway in under a minute. My conjecture is that the viruses running around are ping sweeping various IP address ranges and the routers don't have enough ARP cache RAM to handle all the hosts in their sub-nets. So ARP tables overflow all the time, causing lots of ARP request storms.
Hopefully BAINCH will see something... | |
BAINCHPremium,VIP,MVM
join:2003-04-02
Middletown, NY
kudos:10 | Well, the issue seems to be caused by a particular type of home gateway/router that reports a generic MacID when it has an error. Get two on the same part of the network (and thus two devices with the same MacID) and is causes an arp issue. We are trying to determine what material impact this has on customer performance (if any) and what the best course of action would be to resolve it. | |
UHFAll static, all day, ForeverPremium,MVM
join:2002-05-24 Reviews:
·Callcentric
·DIRECTV
·surpasshosting
·Dish Network
·VOIPo
| Nice. You would think manufacturers would try to stick to the 802.3 specs for ethernet devices and not do screwy things with the MACs. I sure would like to know which device this is so I can be sure I don't buy one.
Thanks for the update Bainch. | |
| reply to BAINCH
said by BAINCH:
Well, the issue seems to be caused by a particular type of home gateway/router that reports a generic MacID when it has an error. Get two on the same part of the network (and thus two devices with the same MacID) and is causes an arp issue. We are trying to determine what material impact this has on customer performance (if any) and what the best course of action would be to resolve it.
Is this something that Mediacom is seeing, or is this something that I would be seeing? I can tell you for certain that what I am seeing is a misconfig of Mediacom's router service (I've already proven that to them, and I've also verified it by asking other people who are knowledgeable enough to explain it). I say that because this _sounds_ like upper management's answer to something that they don't want to spend the time/money on hiring someone who knows what they're doing to fix it.
What do you mean, "generic MacID?" Furthermore, can you identify "a particular type of home gateway/router?" Are you talking about a hardware router, a gateway, a firewall system with NAT, etc? That sounds a little like upper management's bs as well... (I figured I would ask because perhaps you could shed some light on that).
As for the post suggesting that ARP has little to do with DHCP... Perhaps you should read a little more into it (the link provided in my last post explains this very clearly... Read the Friendly Manuals). Without ARP, DCHP simply won't work efficiently. That's just the way it was designed.
Furthermore, _if_ there is a virus problem as someone has suggested, then it would be a problem with either lazy sysadmin or mediacom trying to do a NIX job with a MS product (not trying to sound anti-MS or anything, but... well... sometimes the truth hurts).
I haven't ran a scan to find out what OSes mediacom is running locally, but I would at least _hope_ that they're not that bad off (Out of I-don't-know-how-many, I actually got to hold conversation with two people who at least knew what they were talking about, so I still give them the benefit of the doubt, there).
If there _are_ needless ARP requests for a single IP address being broadcast as you have suggested, then what you're seeing is a client that has either been mis-configured or compromised (both of which are common, btw because people in general are clueless). If Mediacom doesn't have the know-how (or at least the desire) to pinpoint this, then perhaps you would be wise to bring the specifics to their attention (such as firewall logs, explaining what each entry means)... Perhaps they'll be able to figure out who it is from that and get them straightened out).
Reiteration... (not trying to be too obvious, here) The original problem posted in this thread has been resolved. Mediacom had a defunct router, the router was replaced (I guess they couldn't figure out how to fix it), and the problem that I had (past tense) no longer exists. The rest of this thread has nothing to do with it...
Cheers,
WizLayer | |
BAINCHPremium,VIP,MVM
join:2003-04-02
Middletown, NY
kudos:10 | The router issue was a bad I/O card and was just responding, badly, to the ARP issue. It wasn't the cause. The problem was created by three Linksys BEFW11S4v4s. It has been acknowledge by Linksys and they have a fix available on their website (a firmware upgrade.)
We have been calling the customers with the offending device and helping to resolve it. | |
| said by BAINCH:
The problem was created by three Linksys BEFW11S4v4s. It has been acknowledge by Linksys and they have a fix available on their website (a firmware upgrade.)
I did a quick search for it, and can't seem to find anything. Can you post a link that explains what the issue is with this hardware?
Thx,
WizLayer | |
BAINCHPremium,VIP,MVM
join:2003-04-02
Middletown, NY
kudos:10 | I'm away for a few days but I will see if I can look it up from here. | |
|
