| |
to Michael mi
Re: Michigan Internet outageThank you for contacting WOW! My name is Leslie.
I'm sorry that your service was out this weekend. Unfortunately, WOW! was the target of a DoS attack, which prevented many of our Michigan customers from connecting to the Internet. That issue has been resolved and your services should be fully restored.
This is what I got in an email from WOW.
So 6 hops on a traceroute to WOW DNS and 12 hops to Google
At the time of outage, ping was like 4000ms to WOW DNS and 37ms to Google DNS |
|
 Bill_MIBill In Michigan
MVM
join:2001-01-03
Royal Oak, MI
Linksys WRT54GS
Linksys WRT54G v4
|
Hi raypsi1, and thanks for the note. I assume that was copy/paste? It said DoS and not DDoS, right? The freep article from mix clearly said "distributed denial of service attack" (DDoS) which is a very specific bandwidth-sucking attack which there was no other symptoms of over the long period of time DNS was down. But a DoS can happen in an instant and break the DNS servers so they weren't or couldn't be restored timely. The difference between the two is probably unknown to those that use the terms because they heard them with no real understanding. Reporters and support people are often very bad at exactly this. We'll probably never know for sure what actually happened.  |
|
| |
yes copy and paste
Leslie signed:
Respectfully,
Leslie - C6463
Auntie, K-Pop Fan, Trekkie
Advanced Support Specialist - Tier II Tech Support
WOW! Internet, Cable and Phone |
|
 aes128
join:2003-12-19
Warren, MI |
to Bill_MI
I was under the impression the difference between ddos and dos is that they are both denial of service attacks however the ddos is initiated by many computers at once hence the term "distributed". Usually it's some botnet doing the attack and controlled by someone. Being distributed, it's harder to stop. The ways to deny service are many but for a single server to take out the WOW DNS server would be hard unless WOW was running some old version of BIND with a known security hole in it.
The iPhone attack crashing an iPhone with a single text message was an example of a known bug that could be exploited by a single source.
I suppose if there were enough bots they could have simply overloaded the WOW DNS server with so many queries it just could not respond in a timely fashion. Still one would think the WOW server would be large with a lot of bandwidth but then who knows.
I used to manage a IPS system at Chrysler in the old days. Our IPS would have alerted on such a thing but I have no idea what WOW uses. |
|
Bill_MIBill In Michigan
MVM
join:2001-01-03
Royal Oak, MI TP-Link Archer C7
Linksys WRT54GS
Linksys WRT54G v4
1 edit |
I was doing some reasoning and measuring. It turns out from outside WOW's network, the DNS servers don't reply to queries. Nothing is heard, not even a query rejection. They do, however, reply to ICMP echo request which could certainly be involved in a DDoS flood but would take a pretty massive one. General reasoning concludes such a botnet is unlikely inside the WOW network, thus, it had to come from outside. Yet no one was seeing slow downs if they had alternate DNS. Now... I happen to be smoke ping monitoring 64.233.217.10, started from an old thread with Wow_James as a good Wow-local address and it showed no long periods of anything unusual - just my own network activity raising spikes every now and then. The dead DNS servers are at 64.233.217.2 and 64.233.217.3 which may not be on the same network but I bet they are. Tracing isn't identical by address but is in time. So I'm concluding if they didn't just fail on their own, a DoS would be likely exploiting some vulnerability to make them crash. Then... take dozens of hours to restore? That's pretty bad. Just sayin'  |
|
 chip89
Premium Member
join:2012-07-05
Columbia Station, OH |
to aes128
It even took down MS a couple times this year. |
|
aes128
join:2003-12-19
Warren, MI |
to Bill_MI
Well if the WOW servers are not public then it would seem hard for them to be taken out by traffic outside the WOW net. All they have to do is block incoming port 53 to them at the edge routers. At Chrysler we used to block a ton of ports we did not serve to the Internet at the edge, this stuff never even made it to the firewalls.
Then if they are simply using BIND config to not respond to external queries, one could still swamp them with external traffic they would need to process it to reject it. I have my BIND "allow-query" set to only respond to 192.168/16 queries however it does have a public IP so all traffic could hit it. It is not behind a separate firewall, just it's own iptables one so it does see all traffic.
I am sure we will never know. |
|
wkcole
Premium Member
join:2007-11-08
Eastpointe, MI |
wkcole
Premium Member
2015-Jul-20 2:11 pm
said by aes128:Well if the WOW servers are not public then it would seem hard for them to be taken out by traffic outside the WOW net. All they have to do is block incoming port 53 to them at the edge routers. It's actually not as simple as that because "DNS Server" is not really one function, it's 2: authoritative zone information for the world and cached recursive resolution for local users. WOW has DNS servers that provide both and I'm not sure whether they live on the same servers. By this late date most people running significant DNS servers have split those out to entirely distinct machines, but the bigger you are the harder it is to actually do that sort of change. If the servers being slammed are authoritative for any zone, they can't drop inbound port 53 traffic to them. Also worth noting: even if recursion and authority functions are split, filtering port 53 has to be done very carefully at the border to be of any use AND to not harm valid traffic. It CANNOT be done based on source IP, because the nature of most DNS mischief involves UDP queries with bogus source IPs matching ones that will be replied to by the target machines. It also MUST only protect specific target machines, because WOW has customers like me running authoritative nameservers for our own domains on WOW-provisioned IPs. said by aes128:At Chrysler we used to block a ton of ports we did not serve to the Internet at the edge, this stuff never even made it to the firewalls. I can confirm that, having served some time in the Centerline Sea of Cubes on the first Extranet Support team and before that in C/S Site Support. Whole piles of garbage that no one really needed to work just didn't work because of filtering. And yet, I am entirely sure that WOW has a more complex edge filtering challenge than any company like Chrysler because WOW has not only people who want/need nothing more complicated than the Internet access they can get at work (plus a little porn from time to time...) but also bozos like me who sit on the same physical wire as those users, demanding unfiltered, unshaped traffic for multiple static IPs as if WOW were just an open pipe. |
|
aes128
join:2003-12-19
Warren, MI |
aes128
Member
2015-Jul-20 6:42 pm
said by wkcole:It's actually not as simple as that because "DNS Server" is not really one function, it's 2: authoritative zone information for the world and cached recursive resolution for local users. WOW has DNS servers that provide both and I'm not sure whether they live on the same servers. By this late date most people running significant DNS servers have split those out to entirely distinct machines, but the bigger you are the harder it is to actually do that sort of change. If the servers being slammed are authoritative for any zone, they can't drop inbound port 53 traffic to them. I already knew this about DNS since I run my own BIND resolver and have the famous O'Reilly book "DNS & BIND" on my desk. I was just trying to keep it simple. I assumed that WOW ran local recolvers to hand out via DHCP and these had nothing to do with their authoritative servers. said by aes128:At Chrysler we used to block a ton of ports we did not serve to the Internet at the edge, this stuff never even made it to the firewalls. said by wkcole:I can confirm that, having served some time in the Centerline Sea of Cubes on the first Extranet Support team and before that in C/S Site Support. Whole piles of garbage that no one really needed to work just didn't work because of filtering. And yet, I am entirely sure that WOW has a more complex edge filtering challenge than any company like Chrysler because WOW has not only people who want/need nothing more complicated than the Internet access they can get at work (plus a little porn from time to time...) but also bozos like me who sit on the same physical wire as those users, demanding unfiltered, unshaped traffic for multiple static IPs as if WOW were just an open pipe. Well Center Line has not been a part of FCA (Chrysler) IT in a number of years now. We decommissioned that data center a few years back. I did work there for a few years but then we moved to the Auburn Hills tech center, then back to Sterling Heights and then out to the new FCA IT building off Hamlin Road in Auburn Hills. I know quite well the guy who runs DNS for FCA and he has done this for many years. He taught me all I know about DNS and also the horrid sendmail configuration file. If you worked at Chrysler you know we did NOT route non-Chrysler IPs inside our network (with a few exceptions) so Chrysler internal was very unlike WOW internal. This routing policy was hard to maintain through the Daimler years and then into the Fiat times. |
|
wkcole
Premium Member
join:2007-11-08
Eastpointe, MI |
wkcole
Premium Member
2015-Jul-20 8:41 pm
said by aes128:I already knew this about DNS since I run my own BIND resolver and have the famous O'Reilly book "DNS & BIND" on my desk. Lots of us senior folk shuffling about here doing more than meets the eye... I didn't intend to suggest that you didn't understand DNS, but merely to point out that it may not be as simple as described to block the block-worthy traffic in a harmless manner. said by aes128: I assumed that WOW ran local recolvers to hand out via DHCP and these had nothing to do with their authoritative servers. The authoritative nameservers for wowway.{com,biz.net} and wideopenwest.com all resolve to the same pair of IPs and both happily answer my recursive queries from my WOW link. The names (dns1 and dns2) make me suspect they intentionally perform dual service. I dunno what they hand out in DHCP, but it sure seems like it could be those 2. said by aes128: Center Line has not been a part of FCA (Chrysler) IT in a number of years now. We decommissioned that data center a few years back. I did work there for a few years but then we moved to the Auburn Hills tech center, then back to Sterling Heights and then out to the new FCA IT building off Hamlin Road in Auburn Hills. I know quite well the guy who runs DNS for FCA and he has done this for many years. He taught me all I know about DNS and also the horrid sendmail configuration file. Yes, I left in 2000 as they were moving a bunch of teams out. I also definitely recall K.D., at that time God of Many Arcane Things, although I doubt he'd remember me (except maybe for pestering him about the pager gateway.) said by aes128: If you worked at Chrysler you know we did NOT route non-Chrysler IPs inside our network (with a few exceptions) so Chrysler internal was very unlike WOW internal. Absolutely, which was sorta my point. What a corporate network can do to manage traffic at their border is radically different from what an ISP can do, largely because the ISP's borders usually are more complex. Or at least should be... I've got a strong impression that WOW has basic architectural problems that are non-trivial to fix even if they understand the need. I hope that's a misimpression. |
|
Chaldo
join:2008-03-18
West Bloomfield, MI |
to aes128
said by aes128:Well if the WOW servers are not public then it would seem hard for them to be taken out by traffic outside the WOW net. All they have to do is block incoming port 53 to them at the edge routers. At Chrysler we used to block a ton of ports we did not serve to the Internet at the edge, this stuff never even made it to the firewalls.
Then if they are simply using BIND config to not respond to external queries, one could still swamp them with external traffic they would need to process it to reject it. I have my BIND "allow-query" set to only respond to 192.168/16 queries however it does have a public IP so all traffic could hit it. It is not behind a separate firewall, just it's own iptables one so it does see all traffic.
I am sure we will never know. I guess they couldn't be that smart enough, especially if you can control your iptables exactly how you did it, even if you are only responding to 192.168 at the /16 sub in your internal network, all the requests to your public IP would just be dropped like you stated, they must have not setup anything like that at all which is embarrassing to say the least. |
|
Bill_MIBill In Michigan
MVM
join:2001-01-03
Royal Oak, MI TP-Link Archer C7
Linksys WRT54GS
Linksys WRT54G v4
|
to wkcole
said by wkcole:The authoritative nameservers for wowway.{com,biz.net} and wideopenwest.com all resolve to the same pair of IPs and both happily answer my recursive queries from my WOW link. The names (dns1 and dns2) make me suspect they intentionally perform dual service. I dunno what they hand out in DHCP, but it sure seems like it could be those 2. Hi guy, Here's an old list I have: 64.233.207.2 dns2.wideopenwest.com
64.233.207.8 nap11-dns1.nap.wideopenwest.com
64.233.207.9 nap11-dns2.nap.wideopenwest.com
64.233.207.16 dns1.wideopenwest.com
64.233.214.34 clv11-dns1.clv.wideopenwest.com
64.233.214.41 clv11-dns2.clv.wideopenwest.com
64.233.217.2 try11-dns1.try.wideopenwest.com
64.233.217.3 try11-dns2.try.wideopenwest.com
64.233.217.5 try11-dns3.try.wideopenwest.com
64.233.222.2 col11-dns1.col.wideopenwest.com
64.233.222.7 col11-dns2.col.wideopenwest.com
I'm sure those are the dns1 & dns2 you refer but they're over in Naperville, IL best I can tell. The other 2 in the same subnet being for customers. Troy, MI is the "try" guys and DHCP gets .2 and .3 here. I haven't seen the .5 server in a long time but I'm not looking often (also a bind9 local resolver). Cleveland and Columbus are there, too. ALL WORK NOW. Anyway... using some of Steve Gibson's tools (» grc.com/dns) GRC will report the resolving server(s) so I just gave it a try. HEY! When I use Troy's .2, .3 or .5... it doesn't matter which, all use the .2 address as the resolver. It's like the .3 & .5 are aliases. That means... THAT'S ONLY ONE EFFECTIVE DNS SERVER! No independent secondary. And I think everyone on this side of Michigan uses this server. It went down Sunday the 12th and the rest was history. I'm not giving up my local bind9 server anytime soon. |
|
1 edit |
to Chaldo
said by Chaldo:I guess they couldn't be that smart enough, especially if you can control your iptables exactly how you did it, even if you are only responding to 192.168 at the /16 sub in your internal network, all the requests to your public IP would just be dropped like you stated, they must have not setup anything like that at all which is embarrassing to say the least. WOW is not even remotely that bright. Let me show you. traceroute: Warning: www.google.com has multiple addresses; using 74.125.226.81
traceroute to www.google.com (74.125.226.81), 64 hops max, 40 byte packets
1 vrrp0.cle.ts.local (10.22.0.1) 0.259 ms 0.261 ms 0.284 ms
2 10.216.0.1 (10.216.0.1) 11.829 ms 12.757 ms 12.951 ms <--- Oh look, they're leaking the 10net again...
3 dynamic-76-73-172-9.knology.net (76.73.172.9) 12.424 ms 12.962 ms 12.708 ms
Offending packet: TCP 50.X.X.X:41973 > 10.216.0.1:39172 FPU ttl=57 id=9923 iplen=15360 seq=3122385871 win=65535 <wscale 15,nop,mss 265,timestamp 4294967295 0,sackOK>
sendto in send_ip_packet_sd: sendto(4, packet, 60, 0, 10.216.0.1, 16) => Operation not permitted
Offending packet: TCP 50.X.X.X:?? > 10.216.0.1:?? ?? ttl=45 id=47822 iplen=15360 frag offset=512 (incomplete)
sendto in send_ip_packet_sd: sendto(4, packet, 60, 0, 10.216.0.1, 16) => Operation not permitted
Offending packet: TCP 50.X.X.X:41973 > 10.216.0.1:39172 FPU ttl=56 id=41804 iplen=15360 seq=3122385871 win=65535 <wscale 15,nop,mss 265,timestamp 4294967295 0,sackOK>
sendto in send_ip_packet_sd: sendto(4, packet, 60, 0, 10.216.0.1, 16) => Operation not permitted
<output trimmed>
Too many fingerprints match this host to give specific OS details
Yeeeeeeeeeeeeeep. Oh, and they're also announcing bogons. Again. I wonder how the DoD feels about that... |
|
| RootWyrm |
to Bill_MI
said by Bill_MI:THAT'S ONLY ONE EFFECTIVE DNS SERVER! No independent secondary. And I think everyone on this side of Michigan uses this server. It went down Sunday the 12th and the rest was history. Not even that many. They're a half-assed Xerocole cluster. Xerocole being in bed with "Search Guide Inc." which we've already well established as a criminal operation. Literally criminal. They have a couple hundred typosquats, a whole ton of just plain squats, and it's all registered to a fake address and a string of shell companies. Which all leads right back to former NebuAd employees. |
|
wkcole
Premium Member
join:2007-11-08
Eastpointe, MI |
to Bill_MI
said by Bill_MI:Anyway... using some of Steve Gibson's tools (»grc.com/dns) GRC will report the resolving server(s) so I just gave it a try. HEY! When I use Troy's .2, .3 or .5... it doesn't matter which, all use the .2 address as the resolver. It's like the .3 & .5 are aliases. That means... The DNS clues at WOW MAY BE spread even thinner than I had let myself believe. Probably NOT. The other 2 may just be in 'forward first' mode such that they funnel through the .2 machine when present to maintain cache coherency on all 3, but are capable of doing their own recursion as needed. If I recall correctly, that's the way DNS Liarware (of various brands...) is usually set up. It's useful to keep the lies consistent, but it is possible for most DNS servers to automatically fall back on their own functionality when their primary resolver can't help them. said by Bill_MI:I'm not giving up my local bind9 server anytime soon. A wise choice. There's no justification for an ISP to be running DNS so shoddily that average users have to pick between providers of free public DNS with its less obnoxious and/or avoidable lying or learning how to manage an honest nameserver properly (which isn't terribly hard, it just demands more active awareness of security issues than the average modern personal computer.) WOW put all of their users into that position last week. They MAY have now done whatever was needed to make their DNS usable for >90% of users, although they also may have simply outlasted the willingness or ability of the attacker(s) to sustain an attack. On the other hand, those who use their Internet link for substantial business purposes (e.g. self-hosting various servers or even just VPN-based telecommuting) are about a decade past any sound justification for not having a local recursive resolver. Even if WOW has addressed their recent DNS reliability/vulnerability issues, WOW has a history of DNS *HONESTY* issues that I don't think have been entirely fixed and they may have no intention of fixing. They are far from alone in having that flaw and there is a rational argument for mangling some DNS answers to protect the vast bulk of ISP users, but it presents an unnecessary risk for anyone willing and able to run the simplest sort of DNS server for themselves. |
|
|
| |
said by wkcole:They MAY have now done whatever was needed to make their DNS usable for >90% of users, although they also may have simply outlasted the willingness or ability of the attacker(s) to sustain an attack. See bolded section. WOW is in bed with Xerocole and "Search Guide Inc." They will not move away from the fraud setup. Ever. |
|
| |
Inflex
Member
2015-Jul-21 6:47 pm
I'm still wondering why RootWyrm hasn't switched to Time Warner. |
|
