US Cable Support > Mediacom > arp info overwritten (OpenBSD)

arp info overwritten (OpenBSD)

After a while of examining _many_ tcpdumps, I have determined that there is no other plausible explanation for this except that Mediacom has a misconfig in one of their routers.

This _shouldn't_ cause a loss of Internet accessability though, even if the current connections are reset because the arp info is rewritten (switched back) within a second or two.

Other times (and this may be coincidence, because this also seems to happen regardless of the arp overwrites), I find that I am able to ping the gateway, access the DNS server, but nothing beyond. Of course, when I cycle power to the modem, all seems to return to normal.

That leads me to my next question (and this is for you Mediacom, tech folks)... Why is it that if I can ping the gateway and get response, ping 'www.yahoo.com,' get the IP resolution, but no echo reply, ping the IP address which your DNS server sends to me, get no echo, and then get on the phone with my mother and have her ping the same IP address and get reply, that I need to "reset the modem?"

Because I can ping the gateway, there's nothing wrong between the gateway and my box (including the modem).

Because I can ping www.yahoo.com and get resolution, there's nothing wrong between your DNS server, your gateway, and my box (including the modem... It _does_ suggest that there's a problem between your gateway and www.yahoo.com, though (because I get no return from the outgoing echo requests).

Because I get no reply when pinging the IP address of www.yahoo.com directly, it shows that there is a definite problem _somewhere_ and it has nothing to do with your DNS server.

Because my Mother can get on her box (from one of those _other_ ISPs ) and get reply from pinging that same IP address, it shows that www.yahoo.com is not dropping echo requests.

I'm not trying to sound cynical or anything, but "you need to reset the modem" isn't the solution, here. If it were a problem with the modem, then I wouldn't be able to interact with the gateway, right? It sounds more like I'm having to back out of the system alltogether and connect again, which would be a jury-rig, not a fix (kind of like a reset button on a M$ box ).

You've got to be aware of the problems, because I know for a fact that I'm not the only one who has noticed it, and I'm not the only one who has reported it either.

The last time I reported it to support, I included an excerpt from tcpdump, with system messages, dhclient configs (showing that nothing bogus was written to them by your DHCP server/s), and outputs from the previously mentioned pings, showing good connection with the gateway, and the email was forwarded to upper management for futher review. I say this because I don't want anyone to think I'm being negative here. I'm just applying good, troubleshooting techniques with a bit of common sense to help identify the problem so it can get fixed.

Also, I appreciate all the timely responses and positive attitudes when I call tech support, but for real... What's going on?

thx

WizLayer

Are you sure this is not indicative of ethernet frame collisions caused by two NICs with the same MAC ADDRESS on the same net?

This _would_ bring on the arp info overwritten notices and cause some major connection problems, yes (and I would hope that Mediacom would immediately notice such an attempt and deal with it)... however, when arp info is overwritten the first time, it wouldn't almost immediately switch back if that were the case.

If someone were to "hijack" my connection, then I would loose connection alltogether until he/she was done using it... that is, unless the hijacker _also_ allowed my box to connect through his (that was my initial concern going into this).

Unfortunately, my knowledge of cisco routers is lacking (it's not like I have one at the home to play with ). btw... If you know of a good book on cisco, let me know... I'd love to learn it.

When I first started putting the pieces together, I figured that _somehow_ I must have set my box up wrong. I posted specifics to a newbies forum for openbsd. The last message in that thread, I posted specifics (go to »mailman.theapt.org/pipermail/ope···522.html to see my last post to them regarding this).

What you'll find in the tcpdump is that during regular operation, [MAC address 1] is sent requests and [MAC address 2] replies to each request (that suggests routing). The MAC addresses themselves are Cisco's. Sometimes, [MAC address 2] (as opposed to [MAC address 1]) decides it wants to know who I am and that is when arp info is overwritten. BUT... It is immediately written back.

This is Greek to me for the most part because like I said, I know nothing of cisco. I DO know that nothing is misconfigured in my system, though. And the little I have read on this (long live google!), arp messages are either due to someone hijacking the account (which would have been obvious enough that I would have picked up on it even if Mediacom didn't), a network card being changed out (a one-time switch only), or a misconfigured cisco router. The misconfigured cisco router is the only one that fits this scenario.

As for the rest of the problem (connection problems), this may be something different altogether. I'm not so sure that these two are related. I can take my LAN to a friend's house, connect it directly to his DSL service, and the problems simply don't exist. Kinda weird (that was a 'peace of mind' test for me), but it at least indicates where the problem/s is/are at.

I was just wanting to know if any tech reps would give me more than, "we've sent your email to upper management" because surely they know of this (and if they don't, then goodness... why not?)...

thx

Mike

reply to WizLayer
I've been getting ARP floods here for a long time. None of the Mediacom techs that post here seem to want to touch the posts about them with a 10 foot pole... I'm not sure why that is, but I suspect a router problem is causing it. 99.7% of the traffic seen by my modem is ARP requests, according to my Snort reports.

I'll touch it, not that I'm a tech. We are investigating. It doesn't appear to be a router issue. I will report more soon.

reply to UHF

said by UHF:

---

I've been getting ARP floods here for a long time. None of the Mediacom techs that post here seem to want to touch the posts about them with a 10 foot pole... I'm not sure why that is, but I suspect a router problem is causing it. 99.7% of the traffic seen by my modem is ARP requests, according to my Snort reports.

---

I've posted about the ARP flooding before - never got any rational response.

What ARP is for is mapping an IP address to a physical (Ethernet in most cases) address. When you see an ARP REQUEST, that's a host asking:

I'm at 1.2.3.4, Ethernet 12:34:56:78:90:12 -
what is the ethernet addr for ip addresses A.B.C.D

These are always BROADCAST. Most hosts will see the broadcast and remember and cache the source information. The host that has ip address A.B.C.D sends an ARP RESPONSE (typically directly to the sender, but sometimes in broadcast as well)

hey you at 1.2.3.4, Ethernet 12:34:56:78:90:12 -
I'm ip addr A.B.C.D, ethernet addr AB:CD:EF:GH:IJ:KL

I think this really has very little to do with DHCP (which is a way for a host to discover it's IP address).

Anyway, most hosts keep caches of ARP mappings - i.e. they remember that IP address A.B.C.D mapped to Ethernet address AB:CD:EF:GH:IJ:KL - usually for 20 minutes or so. As traffic passes, these mappings are updated, so if a host is relatively active, it's ARP mapping should essentially never time out. However, this cache is limited in size, so if it fills up, entries will drop out, requiring them to be re-discovered. Routers are no different in this - they should cache ARP table mappings for a period of time.

I expect there are some tweaks to this process in a Cable Modem DOCSIS environment, but I haven't downloaded the DOCSIS documents to figure out what they might be.

I believe the ARP storms we see are probably due to the fact that many cable modem nodes are running with fairly large sub-nets - we've got a /22 here - which means there are 1024 nodes on the same sub-net. I expect this is too large for the local routers to handle - their ARP caches fill up, so they end up dropping nodes out, and having to constantly rediscover them.

This could also be due to a misconfiguration: if the router thinks it's sub-nets are larger than they really are, then if it thinks it has traffic for an IP address in one of it's sub-nets, it'll try to ARP for that host. But if the host is not there, either it's off or it's on a different sub-net, it'll get no response and will keep trying. This is the only way I can explain why the gateways are ARP'ing for the same IP address over and over again frequently.

And, with the various viruses running around, probing every possible IP address, that really pushes up the problem as now the routers are seeing traffic for *every* IP address in the sub-net, regardless of if the host is there or not. This really kills the ARP cache.

I would still like Mediacom to do something about it. The ARP traffic, while small, chews up bandwidth. Since cable modems are shared, we're all paying the price for this in more latency and less overall available bandwidth. It also takes processing time on the routers, hurting performance there as well.

Oh, one other thing: I've had @Home/Mediacom cable modem at the same location since it first become available, almost 5 years ago. The ARP storm problem only really became noticeable in the last year or so, which is why I think it's a configuration/router problem. Overall, I'm quite happy with Mediacom, although I wish they'd give us a little more upstream BW.
[text was edited by author 2003-10-28 09:50:20]

said by TazMainiac:

---

I've posted about the ARP flooding before - never got any rational response.

What ARP is for is mapping an IP address to a physical (Ethernet in most cases) address.

---

said by UHF:

---

I know what ARP is, what I want to know is, why do I see ARP requests for the same host multiple times per minute??

---

Well, the issue seems to be caused by a particular type of home gateway/router that reports a generic MacID when it has an error. Get two on the same part of the network (and thus two devices with the same MacID) and is causes an arp issue. We are trying to determine what material impact this has on customer performance (if any) and what the best course of action would be to resolve it.

Nice. You would think manufacturers would try to stick to the 802.3 specs for ethernet devices and not do screwy things with the MACs. I sure would like to know which device this is so I can be sure I don't buy one.

Thanks for the update Bainch.

reply to BAINCH

said by BAINCH:

---

Well, the issue seems to be caused by a particular type of home gateway/router that reports a generic MacID when it has an error. Get two on the same part of the network (and thus two devices with the same MacID) and is causes an arp issue. We are trying to determine what material impact this has on customer performance (if any) and what the best course of action would be to resolve it.

---

The router issue was a bad I/O card and was just responding, badly, to the ARP issue. It wasn't the cause. The problem was created by three Linksys BEFW11S4v4s. It has been acknowledge by Linksys and they have a fix available on their website (a firmware upgrade.)

We have been calling the customers with the offending device and helping to resolve it.

reply to WizLayer
i've been having the arp flood problem for over a month now.
sometimes 2000+ arp requests per second.
here's a snort log from a couple of weeks ago:
»www.quantumfish.com/files/snort.log.txt
i don't have any linksys hardware. my cablemodem is a motorola and it doesn't matter what device is plugged into it for the arps to be received.
so, any other ideas?

BAINCH was not saying that the people who were getting the floods had linksys hardware. What was stated is that the particular routers were the cause of the floods.

reply to sigsegv
Sig,

These logs look about normal... except... I would wonder why there are so many dhcpds at the helm... A couple, okay... but my quick look through it shows 8. That seems a little much. Mediacom would be better to split this up a little more (especially if they expect growth in their customer base).

It could also be that they are not filtering the noise between clusters, which would be an easy fix.

Of course, it could also be due to idiot client's gateways who who haven't yet learned to point dhcpd in the right direction.

If you're getting 2000+ arp requests per second at any time, then it should _definately_ get split up... or they should round up the idiot clients ahd hold a public hazing. (Whichever the case may be... )

Try to get a dump when arp traffic is like you say. Then send it to Mediacom and ask them to explain it. If they tell you that you need to reset your modem, that you must be configured incorrectly, or that they have no idea what you're talking about, then insist on talking to someone else.

Good luck,

WizLayer

reply to BAINCH

said by BAINCH:

---

The problem was created by three Linksys BEFW11S4v4s. It has been acknowledge by Linksys and they have a fix available on their website (a firmware upgrade.)

---

I'm away for a few days but I will see if I can look it up from here.

reply to WizLayer
I see that Linksys is now up to Ver 1.45.7 dated 9/16/03 firmware for the BEFW11S4 V4. That's typical for linky (I have one also). Mine was purchased in June and had Ver 1.45.1 firmware which would just die during data transfer. Tried to upgrade to 1.45.3 and locked up the unit. Sent it back to linksys for new replacement which came with 1.45.3. If Linksys would spend 2-3 months additional testing before releasing new products, they would make a killing, but their profits are going to customer support issues.