dslreports logo
 
    All Forums Hot Topics Gallery
spc
Search similar:


uniqs
2402

USAccess
Premium Member
join:2013-09-08
Oshawa, ON

1 edit

USAccess

Premium Member

USAccess DNS - Rogers (affecting 3rd-party Cable ISPs) blocking our DNS

Hello Everyone,
It looks like, for the second time in the past week, Rogers (and thus all TPIA's) are having issues with our DNS.

The DNS addresses (107.20.195.51/107.20.190.171) are responding to ping queries, but not DNS queries. If you check from outside the network (from your VPS/servers/friends places not on Rogers) the DNS requests pass through completely fine.

Our first attempt at a Region Switcher did not work out so well, but we have been beta testing our Region Switcher 2.0, which has access to US/UK/Canada/Brazil/Sweden/Mexico. If you are using USAccess and are having issues with the production DNS, please open a ticket and we'll send you the Region Switcher 2.0 information.

Regards,
USAccess Support Team

HiVolt
Premium Member
join:2000-12-28
Toronto, ON

HiVolt

Premium Member

Re: USAccess DNS - Rogers and 3rd-party Cable ISPs blocking our DNS

Maybe Rogers is blocking it, but as far as I know they don't control any of the TPIA's access...

You should conduct a test by setting up some test accounts for people willing to test it that are not your subscribers, if you don't get enough of a sample size.

USAccess
Premium Member
join:2013-09-08
Oshawa, ON

USAccess

Premium Member

Send us an email (support at usaccessdotca) if you are willing to test. We're seeing that ICMP traffic (ping) is getting through fine, however, DNS requests on port 53 are not heading down the pipe at all.

HiVolt
Premium Member
join:2000-12-28
Toronto, ON

HiVolt

Premium Member

Unfortunately I'm not on Cable, but on DSL.

bbbc
join:2001-10-02
NorthAmerica

bbbc to USAccess

Member

to USAccess
Seems like a good time to file a CRTC complaint.

Rogers response, "We're not throttling blocking DNS servers."

twizlar
I dont think so.
Premium Member
join:2003-12-24
Brantford, ON

twizlar to USAccess

Premium Member

to USAccess
Works fine here on tpia cable.

USAccess
Premium Member
join:2013-09-08
Oshawa, ON

USAccess

Premium Member

said by twizlar:

Works fine here on tpia cable.

And it is here now too. This is the second time in a week that for a good 6-7 hours, certain areas (Spot checked in Oshawa and Mississauga) were completely dropping DNS packets to our system.
Madwand
join:2002-12-03
Toronto, ON

Madwand to USAccess

Member

to USAccess
It was me that told you originally last weekend that Rogers was blocking the DNS when I was visiting a friends place that has Rogers however you didn't believe me.

My Teksavvy cable at home works fine.

USAccess
Premium Member
join:2013-09-08
Oshawa, ON

USAccess

Premium Member

said by Madwand:

It was me that told you originally last weekend that Rogers was blocking the DNS when I was visiting a friends place that has Rogers however you didn't believe me.

My Teksavvy cable at home works fine.

Hey Madwand,
I'm sorry that you felt that way. Its not necessarily that we don't believe people, we just need to be able to verify and recreate the issue, which we have now. It seems to be only the production DNS' that get affected during these times. For those reading this when the DNS does stop responding, I would suggest contacting your ISPs.
USAccess

USAccess

Premium Member

Looks like someone has been attacking our main website. Noticed this at about 10am and put some extra security in check.

We take nightly backups so we have all of the information we mayneed to restore. The Customer database is not tied in with the main website, so none of that information is affected. We have our security guy (OSCP certified) analyzing everything right now and will report back.

You can still access the customer portal here: »accounts.usaccess.ca/client/login/

The system is still up and running completely, it is just the main website that is having issues at the moment.

Regards,
USAccess Support Team
USAccess

USAccess

Premium Member

said by USAccess:

Looks like someone has been attacking our main website. Noticed this at about 10am and put some extra security in check.

Might have jumped the gun in assuming that this website error was in relation to the attacks early this morning. Seems like it was just a small corruption in the main website database. Still keeping an eye on everything, but the main website is back up.

Exidor
Premium Member
join:2001-05-04

Exidor

Premium Member

From July 13/14:

Start experienced a "Massive DDoS", which apparently affected "DNS servers such as those from usaccess.ca"

»Re: Problems again with Rcable

USAccess
Premium Member
join:2013-09-08
Oshawa, ON

USAccess

Premium Member

said by Exidor:

From July 13/14:

Start experienced a "Massive DDoS", which apparently affected "DNS servers such as those from usaccess.ca"

»Re: Problems again with Rcable

Good to know. They failed to mention this to me when I opened up a ticket with them yesterday.

Thanks for the info @Exidor

rocca
Start.ca
Premium Member
join:2008-11-16
London, ON

rocca

Premium Member

said by USAccess:

Good to know. They failed to mention this to me when I opened up a ticket with them yesterday.

Probably because it's not relevant.

That said, if you want to PM me the ticket number and your DNS servers I can investigate and provided they weren't used as reflectors in the attack then we can look at whitelisting them for any future incidents.

I have difficulty believing that Rogers is blocking this for TPIA. I could see them doing it for retail, but for them to do this at the wholesale level it would mean they'd have to implement this filtering down behind the POI level which would be immensely expensive/intrusive rather than simply filtering in their own IP core. Also other people seem to be saying it's fine on their TPIA but not Rogers which would jive with the above, and lastly looks like a potential system/database error on your end may be the culprit.

In any case, happy to investigate if you send me the information requested above - thanks.

sbrook
Mod
join:2001-12-14
Ottawa

sbrook

Mod

The only sites and NS servers that I've heard of Rogers actually not resolving and actually blocking are from Russia and were absolutely virus laden sites, responsible for hundreds of SpamBot infections.

USAccess
Premium Member
join:2013-09-08
Oshawa, ON

USAccess to rocca

Premium Member

to rocca
said by rocca:

said by USAccess:

Good to know. They failed to mention this to me when I opened up a ticket with them yesterday.

Probably because it's not relevant.

That said, if you want to PM me the ticket number and your DNS servers I can investigate and provided they weren't used as reflectors in the attack then we can look at whitelisting them for any future incidents.

I have difficulty believing that Rogers is blocking this for TPIA. I could see them doing it for retail, but for them to do this at the wholesale level it would mean they'd have to implement this filtering down behind the POI level which would be immensely expensive/intrusive rather than simply filtering in their own IP core. Also other people seem to be saying it's fine on their TPIA but not Rogers which would jive with the above, and lastly looks like a potential system/database error on your end may be the culprit.

In any case, happy to investigate if you send me the information requested above - thanks.

I understand Rocca. I also had difficulties believing it from our customers (Rogers/Tek/Start) until I started experiencing it myself. Perhaps this title is a little misleading in the sense that its assuming Rogers is doing the blocking, however, it was an assumption by our devs when we saw it on different TPIAs at the same time.

During both occurrences ICMP traffic flowed fine but DNS traffic was not reaching our servers.

As far as reflection attacks, we try and be proactive to prevent our servers from participating in amplification attacks. We update our blocklists daily to filter out known DNS amplification queries using iptables rules like the ones found here: »github.com/smurfmonitor/ ··· list.txt

We also rate-limit requests from IPs that are not in our system (not IPs belong to USAccess members).

I will try and find a ticket number to send you along with our server IPs. Appreciate your input on this.

rocca
Start.ca
Premium Member
join:2008-11-16
London, ON

rocca

Premium Member

No problem.

Something you might also want to do to set a baseline is run a traceroute on TCP port 53 and then next time it's blocked do it again to see where it's failing -- ie, if using a recent Linux traceroute:

traceroute -T -p 53 your.ip.address.here

USAccess
Premium Member
join:2013-09-08
Oshawa, ON

USAccess

Premium Member

said by rocca:

No problem.
traceroute -T -p 53 your.ip.address.here

Great suggestion rocca. I built a couple scripts just now to do some data collection during the next outage.

Thanks for the suggestion.

hulu
@198.168.152.x

hulu

Anon

slightly unlrelated to the op but is hulu working with USAccess?

I just switched my router and get US netflix no problem but Hulu is not working...I haven't used Hulu in at least a week so I'd rather not waste more time troubleshooting if it is a universal problem :P

USAccess
Premium Member
join:2013-09-08
Oshawa, ON

USAccess

Premium Member

said by hulu :

is hulu working with USAccess?

We had a brief outage with Hulu yesterday but I can confirm that it is working again.

TOPDAWG
Premium Member
join:2005-04-27
Calgary, AB

TOPDAWG to USAccess

Premium Member

to USAccess

Re: USAccess DNS - Rogers (affecting 3rd-party Cable ISPs) blocking our DNS

I seem to be having issues with the WWE network. It will cut in and out it's done that for the last few days. I just signed up for a free trail on a different DNS serve and everything is working fine now.

USAccess
Premium Member
join:2013-09-08
Oshawa, ON

1 edit

USAccess

Premium Member

Hey TOPDAWG,
Could you open a ticket about this so I can assign it to one of our devs to look in to tonight?

Thanks

EDIT: Should be resolved now
Laidback
join:2001-09-30
Cochrane, ON

Laidback to USAccess

Member

to USAccess

Re: USAccess DNS - Rogers and 3rd-party Cable ISPs blocking our DNS

said by USAccess:

Send us an email (support at usaccessdotca) if you are willing to test. We're seeing that ICMP traffic (ping) is getting through fine, however, DNS requests on port 53 are not heading down the pipe at all.

Is your testing done? I sent you an email a while back and never heard from anyone.
Thanks

elwoodblues
Elwood Blues
Premium Member
join:2006-08-30
Somewhere in

elwoodblues

Premium Member

said by Laidback:

said by USAccess:

Send us an email (support at usaccessdotca) if you are willing to test. We're seeing that ICMP traffic (ping) is getting through fine, however, DNS requests on port 53 are not heading down the pipe at all.

Is your testing done? I sent you an email a while back and never heard from anyone.
Thanks

Same here

USAccess
Premium Member
join:2013-09-08
Oshawa, ON

USAccess

Premium Member

said by elwoodblues:

said by Laidback:

said by USAccess:

Send us an email (support at usaccessdotca) if you are willing to test. We're seeing that ICMP traffic (ping) is getting through fine, however, DNS requests on port 53 are not heading down the pipe at all.

Is your testing done? I sent you an email a while back and never heard from anyone.
Thanks

Same here

said by Laidback:

said by USAccess:

Send us an email (support at usaccessdotca) if you are willing to test. We're seeing that ICMP traffic (ping) is getting through fine, however, DNS requests on port 53 are not heading down the pipe at all.

Is your testing done? I sent you an email a while back and never heard from anyone.
Thanks

Hey guys,
Sorry about the lack of correspondence. Our anniversary sale and region switcher 2.0 has us swamped. Send in another email quoting this post and I'll get some accounts created.

For any new inquiries, we have the the appropriate amount of people testing.

Regards,
Craig
USAccess
Laidback
join:2001-09-30
Cochrane, ON

Laidback

Member

Thanks
I just sent another email. My provider is start.ca
Regards
MaynardKrebs
We did it. We heaved Steve. Yipee.
Premium Member
join:2009-06-17

MaynardKrebs to USAccess

Premium Member

to USAccess

Re: USAccess DNS - Rogers (affecting 3rd-party Cable ISPs) blocking our DNS

It's nothing to worry about. Rogers is just prepping for the introduction of their "Showmi" Netflix **killer** and are just testing their 'kill switch' for your service when "Showmi" goes live.

Better have your CRTC petition and papers for a court injunction ready.

openvz_ca
join:2008-12-13
canada

openvz_ca to USAccess

Member

to USAccess
Your DNS servers seem to support full recursion from any source location:

$ dig @107.20.195.51 google.ca A

; > DiG 9.8.2 > @107.20.195.51 google.ca A
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER- opcode: QUERY, status: NOERROR, id: 28125
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;google.ca. IN A

;; ANSWER SECTION:
google.ca. 300 IN A 74.125.228.119
google.ca. 300 IN A 74.125.228.120
google.ca. 300 IN A 74.125.228.127
google.ca. 300 IN A 74.125.228.111

;; Query time: 32 msec
;; SERVER: 107.20.195.51#53(107.20.195.51)
;; WHEN: Tue Sep 2 16:26:48 2014
;; MSG SIZE rcvd: 91

Running a secure open DNS server is a very challenging and demanding task. Since UDP does not have the ability to source validate, sophisticated algorithms and appliances are needed to sniff the traffic and appropriately deny or throttle requests so that it doesn't turn in to an amplification attack. Especially since the source can be easily spoofed.

Most companies who offer your type of service do so by IP restricting their DNS servers. By quickly viewing the support section of your website, it seems you do require clients to login and update their IP address prior to the service working. If that is the case, your DNS servers don't need to allow any requests go through unless they are part of the IP Whitelist generated by your system, which at this point they are not doing.

This explains the difficulties you are having with connectivity to some networks. If your servers were recently used in a DNS amplification attack due to open recursion, it's likely that these networks have blacklisted your IP addresses.

USAccess
Premium Member
join:2013-09-08
Oshawa, ON

USAccess

Premium Member

Open recursion is used on all 3rd party DNS services like ours. Blocking requests from non-IP-authenticated users would be a very foolish move - what happens when a dynamic IP updates, the user loses all internet connectivity (assuming most users don't know the IP address of every website they with to visit)?

Our servers (on the production side) are rate-limited, and we also subscribe to DNS iptables lists like »github.com/smurfmonitor/ ··· es-rules

I understand you would like to contribute to dialog on this matter, but your information here is just blatantly wrong. You can openly recurse on all of our competitors.
MrToady
join:2011-08-30
Medicine Hat, AB

MrToady to USAccess

Member

to USAccess
Not all competitors =).....here is a little snippet for you from my bind config:

view "Redirect" {
match-clients { any; };
recursion no;