Tell me more x
, there is a new speed test available. Give it a try, leave feedback!
dslreports logo
 
    All Forums Hot Topics Gallery
spc

spacer

Search Topic:
uniqs
35941
share rss forum feed


TSI Marc
Premium,VIP
join:2006-06-23
Chatham, ON
kudos:28

2 edits

6 recommendations

[Cable] Delays and Outages - status

At TekSavvy we pride ourselves on delivering excellent customer service and so we are very concerned about the impact that any outages and service delays have on our customers, especially lengthy ones. We want to assure our customers that we have not been taking the situation lightly and have been working diligently to restore service. Since the 15th of August we have been working both directly and through counsel with the network operator to address the service affecting issues and we have also been employing the alternate dispute resolution services of the CRTC, all with a view to reaching the speediest resolution possible for the benefit of our customers. The details of the discussions must remain confidential in order for the various processes to be pursued in the most effective manner, but please rest assured that TekSavvy is fully and proactively engaged in seeking a resolution to the current problems.

We know that some of you are offline and we understand that this is not acceptable. As everything is functioning properly at our end, we believe the extended service outages are due to changes that have been made by the incumbent. Our challenge is that they are not giving us proper explanations or expected timeframes. We are working as quickly as we can to get our customers back online.

We are convinced that the wholesale model used by TekSavvy and the large incumbents whose networks we have to use is flawed. This model is overseen by the government through the CRTC. And, we believe it needs to be re-visited and modified. We are working with other ISPs who have experienced similar service issues and plan to address the matter more formally with the CRTC in the weeks ahead.

other related:
»[Outages] tek you made cbc ottawa with your outages

»www.cbc.ca/news/canada/o ··· ?cmp=rss

--
Marc - CEO/TekSavvy


fluffybunny

@teksavvy.com
...and I have no doubt the incumbent-run CRTC will no doubt take this situation seriously and diligently decide to do nothing about it.
the state of the telecomms market in canada -- real competition tilts at windmills, incumbents sit back on their monopolies and laugh (while randomly shafting consumers every 6 months with fee increases for teh lols).

slvrsi

join:2009-09-08
kudos:1
reply to TSI Marc
This sounds like a positive step in the right direction. I did have one question:

Back in December (or November?) of last year, I think on these forums there were reports of massive outages as well for entire POIs. DHCP or RF, doesn't matter.

At that time was anything done to mitigate these risks to downtimes, was there any lessons learnt or actions taken to prevent further issues?

I see this time around it seems that there are actions which may improve overall services to your clients. Hopefully you'll be able to report the improvements if any going forward from these actions today.


KPaul

join:2007-02-08

1 edit
reply to TSI Marc
Thank you for an honest post.

Been hoping for this for a while.

EDIT: This should be a sticky up top..... I'm super cereal


Teddy Boom
k kudos Received
Premium
join:2007-01-29
Toronto, ON
kudos:22
said by KPaul:

EDIT: This should be a sticky up top..... I'm super cereal

Ya, it should be. With a link to the cbc article thread, or the article itself:
»[Outages] tek you made cbc ottawa with your outages
--
electronicsguru.ca


Teddy Boom
k kudos Received
Premium
join:2007-01-29
Toronto, ON
kudos:22

1 edit
reply to slvrsi
said by slvrsi:

Back in December (or November?) of last year, I think on these forums there were reports of massive outages as well for entire POIs. DHCP or RF, doesn't matter.

In fact I think it was all through the fall of 2011, but the most egregious cases were in November and December. This is the best summary of the situation then that I know of:
»blogs.teksavvy.com/?p=169

To the best of my knowledge, no postmortem was ever made public, and the systems that failed were not improved fundamentally altered. Further reading here maybe:
»Re: [Outages] Outage Report 08/30/2013- morning update

From a customer's perspective, now is obviously the perfect time to know what the resolution was for those 2011 outages. From Teksavvy's perspective, addressing those outages can't help but prejudice the handling of the current situation. I'm not at all in favour of this non-disclosure nonsense that pervades our culture nowadays..
--
electronicsguru.ca


TSI Marc
Premium,VIP
join:2006-06-23
Chatham, ON
kudos:28
It improved from weeks offline to days offline. Recent events not included. The overall problem remains.
--
Marc - CEO/TekSavvy


Teddy Boom
k kudos Received
Premium
join:2007-01-29
Toronto, ON
kudos:22
said by TSI Marc:

It improved from weeks offline to days offline. Recent events not included. The overall problem remains.

Ya, I edited my post slightly Obviously what happened after that last mess was basically 'things have gotten better, so let's get on with business'. And in general, aside from the absurdly bureaucratic trouble ticketing regime, cable had been getting better and better until last month. Less slow speed issues, and dealt with quickly when they happened, fewer PIEs, and so on. So I can see how that approach had at least some merit.

Still.. Well, why is DHCP handled the way it is for TPIA. Cogeco seems to do it the same as Rogers. Does Videotron too?
--
electronicsguru.ca

vincom

join:2009-03-06
Bolton, ON
kudos:1
reply to TSI Marc
ever since the floods I noticed a lot more complaints/problems w/rcable, I wonder if it has anything todo with it


jmck
formerly 'shaded'

join:2010-10-02
Ottawa, ON
said by vincom:

ever since the floods I noticed a lot more complaints/problems w/rcable, I wonder if it has anything todo with it

that's a small part of it, the other is TSI's move to aggregated and the issues they've had with Rogers's implementation of a few things (10GE port configs, DHCP, etc).


TSI Marc
Premium,VIP
join:2006-06-23
Chatham, ON
kudos:28
Uh. No.

This is plain and simple, our install and repair tickets went unanswered for weeks.
--
Marc - CEO/TekSavvy


jmck
formerly 'shaded'

join:2010-10-02
Ottawa, ON
said by TSI Marc:

Uh. No.

This is plain and simple, our install and repair tickets went unanswered for weeks.

i agree, all those issues were Rogers issues, wasn't putting the blame on TSI. i was just saying the flood had nothing to really do with it since it's affecting other areas and the flood didn't have any affect on the DCHP/10GE ports, fibre cuts and whatever else is happening outside of Toronto.


TSI Marc
Premium,VIP
join:2006-06-23
Chatham, ON
kudos:28
Oh sorry.

slvrsi

join:2009-09-08
kudos:1
reply to TSI Marc
Hopefully current actions will help to resolve this. I don't think myself nor others around me could survive another 15 day outage....

ezebob2

join:2008-06-06
reply to TSI Marc
If you need specific customers experiences feel free to use mine. I hereby grant TSI the authority to use any and all of my personal information it deems necessary in any way it deems beneficial to it's business, and with any government, supplier, or any other person TSI deems appropriate. Is there anything we as consumers can do to help you with these incumbent bone heads feel free to request my assistance.

I was already planning a letter to my MP, Industry Minister, Competition Bureau. Anyone else you think I should write to Marc please feel free to let me know. Living in Ottawa my be an advantage I have, and having worked in the news business may also help shed some much need national press coverage.

I am willing and able to help if you wish.

I am ready to open a big can of whoop @$$ on these clowns.


lleader

join:2011-01-01
Mississauga, ON
reply to TSI Marc
I can't help but feel that the timing of the this latest incident of rCable's intransigence is intended to maximize their capture of the influx of students starting or returning to university.

tintin

join:2013-07-19
Scarborough, ON
Based on my current experience with the cable internet difficulties, I think most likely the DNS service is not reliable. I found that sometimes the IP address of some websites can not be found, this will cause troubles when we use web browsers to surf the web.

If TekSavvy is maintaining its own DNS server, then we should not blame Rogers.


TSI Marc
Premium,VIP
join:2006-06-23
Chatham, ON
kudos:28
said by tintin:

Based on my current experience with the cable internet difficulties, I think most likely the DNS service is not reliable. I found that sometimes the IP address of some websites can not be found, this will cause troubles when we use web browsers to surf the web.

If TekSavvy is maintaining its own DNS server, then we should not blame Rogers.

Our DNS server have no problems at all. Many users even from different incumbents use those systems.
--
Marc - CEO/TekSavvy

tintin

join:2013-07-19
Scarborough, ON
I just tried:

C:\Windows\system32>nslookup verizon.com
...

Non-authoritative answer:
DNS request timed out.
timeout was 2 seconds.
Name: verizon.com
Address: 192.76.85.245

C:\Windows\system32>nslookup ontario.ca
...

Non-authoritative answer:
DNS request timed out.
timeout was 2 seconds.
Name: ontario.ca
Address: 54.243.138.110

If DNS server has no issue, then why is there a 'DNS request timed out' issue?

And I did notice that in the TekSavvy Direct forum there are others reporting DNS issue, is it just an coincidence? I suggest TekSavvy to look at this DNS issue first.

The network connection may be perfect OK, however if DNS doesn't work properly, then the network will still be useless, since the majority of the usage would be web browser access.


TSI Marc
Premium,VIP
join:2006-06-23
Chatham, ON
kudos:28
If we had a DNS server problem, thousands would be flooding this place to find out what's wrong. It's not our DNS.

you're not showing your dns server there.

type ipconfig /all

post that for your main interface.
--
Marc - CEO/TekSavvy

falkon21

join:2004-02-27
reply to tintin
I use one of Teksavvy's DNS servers as my primary DNS server, and one from Radiant Communications as my secondary.

I had previously been receiving DNS issues with webpages not resolving and what not. I changed my router configuration to disable DNS relay, therefore letting NICs on my network to get the DNS server addresses directly rather than using the default gateway as a DNS server. Since making this change webpages have loaded noticeably faster and I've yet to experience a webpage not resolving.

I'd recommend checking out the program called DNS Bench for measuring which DNS servers give you the best performance. For me, Teksavvy was the quickest out of more than 2,000 around the world.

tintin

join:2013-07-19
Scarborough, ON

2 edits
reply to TSI Marc
I'm testing on my computer which is directly plugged into my cable modem. See below:

C:\WINDOWS>ipconfig/all

Windows IP Configuration

Host Name . . . . . . . . . . . . : LGX110
Primary Dns Suffix . . . . . . . :
Node Type . . . . . . . . . . . . : Unknown
IP Routing Enabled. . . . . . . . : No
WINS Proxy Enabled. . . . . . . . : No

Ethernet adapter Local Area Connection:

Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Realtek RTL8102E/RTL8103E Family PCI
-E Fast Ethernet NIC
Physical Address. . . . . . . . . : xxxxxxxxxxxxxxxx (masked)
Dhcp Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes
IP Address. . . . . . . . . . . . : 24.52.253.xxx (masked)
Subnet Mask . . . . . . . . . . . : 255.255.255.224
Default Gateway . . . . . . . . . : 24.52.253.193
DHCP Server . . . . . . . . . . . : 24.246.76.65
DNS Servers . . . . . . . . . . . : 206.248.154.22
206.248.154.170
Lease Obtained. . . . . . . . . . : Saturday, September 07, 2013 8:55:00
PM
Lease Expires . . . . . . . . . . : Saturday, September 14, 2013 8:55:00
PM

Now I try nslookup 4 times:

C:\WINDOWS>nslookup family.ca
Server: dns.pppoe.ca
Address: 206.248.154.22

Non-authoritative answer:
Name: family.ca
Address: 204.101.15.67

C:\WINDOWS>nslookup family.ca
DNS request timed out.
timeout was 2 seconds.
*** Can't find server name for address 206.248.154.22: Timed out
Server: dns2.pppoe.ca
Address: 206.248.154.170

Non-authoritative answer:
Name: family.ca
Address: 204.101.15.67

C:\WINDOWS>nslookup family.ca
DNS request timed out.
timeout was 2 seconds.
*** Can't find server name for address 206.248.154.22: Timed out
Server: dns2.pppoe.ca
Address: 206.248.154.170

DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
*** Request to dns2.pppoe.ca timed-out

C:\WINDOWS>nslookup family.ca
Server: dns.pppoe.ca
Address: 206.248.154.22

DNS request timed out.
timeout was 2 seconds.
Non-authoritative answer:
Name: family.ca
Address: 204.101.15.67

Only the first try succeeded without any issue. Then the 3 subsequent tries all have some problem. There are two DNS servers used, it seems neither of them works perfectly.

Can anyone explain why?

PS.
I'd post more findings here:

I pinged the gate way, DHCP, and DNS servers, and can see that one of the DNS servers can not be reached:

C:\WINDOWS>ping 206.248.154.170

Pinging 206.248.154.170 with 32 bytes of data:

Request timed out.

Ping statistics for 206.248.154.170:
Packets: Sent = 1, Received = 0, Lost = 1 (100% loss),
Control-C
^C

Tracert to this DNS server didn't work either:

C:\WINDOWS>tracert 206.248.154.170

Tracing route to dns2.pppoe.ca [206.248.154.170]
over a maximum of 30 hops:

1 9 ms 8 ms 17 ms 10.126.225.1
2 20 ms 12 ms 11 ms 24.156.149.85
3 12 ms 11 ms 11 ms gw02.mtnk-6-1-2.mtnk.phub.net.cable.rogers.com [
69.63.248.1]
4 * * * Request timed out.
5 * * * Request timed out.
6 * * * Request timed out.
7 ^C

ping / tracert to the other DNS server works:

C:\WINDOWS>ping 206.248.154.22

Pinging 206.248.154.22 with 32 bytes of data:

Reply from 206.248.154.22: bytes=32 time=10ms TTL=60
Reply from 206.248.154.22: bytes=32 time=8ms TTL=60

Ping statistics for 206.248.154.22:
Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 8ms, Maximum = 10ms, Average = 9ms
Control-C
^C
C:\WINDOWS>tracert 206.248.154.22

Tracing route to dns.pppoe.ca [206.248.154.22]
over a maximum of 30 hops:

1 8 ms 19 ms 9 ms 10.126.225.1
2 13 ms 11 ms 11 ms 24.156.149.85
3 10 ms 11 ms 12 ms gw02.mtnk-6-1-2.mtnk.phub.net.cable.rogers.com [
69.63.248.1]
4 10 ms 9 ms 9 ms 2110.ae0.agg01.tor.packetflow.ca [69.196.136.44]

5 10 ms 14 ms 8 ms dns.pppoe.ca [206.248.154.22]

Trace complete.

tintin

join:2013-07-19
Scarborough, ON
reply to TSI Marc
The reason why not so many people are pointing out this DNS server issue, may simply be that they may not know that much in how TCP/IP network works.

I don't think an average user should know the technical details. If everything is OK, then people don't need to come here and have such discussions.

falkon21

join:2004-02-27
Maybe try manually assigning DNS servers to your NIC? This shouldn't be necessary, of course, but maybe it will help?

I use 206.248.154.170 for primary and 66.163.0.161 for secondary.


TSI Marc
Premium,VIP
join:2006-06-23
Chatham, ON
kudos:28
reply to tintin
Authoritative name server family.ca appears to be down or unreachable.. 204.101.15.67

Try a different domain name.
--
Marc - CEO/TekSavvy

tintin

join:2013-07-19
Scarborough, ON
OK, try again: 2 out of 10 times have issue:

C:\WINDOWS>nslookup microsoft.com
DNS request timed out.
timeout was 2 seconds.
*** Can't find server name for address 206.248.154.22: Timed out
Server: dns2.pppoe.ca
Address: 206.248.154.170

Non-authoritative answer:
Name: microsoft.com
Addresses: 64.4.11.37, 65.55.58.201

C:\WINDOWS>nslookup microsoft.com
Server: dns.pppoe.ca
Address: 206.248.154.22

Non-authoritative answer:
Name: microsoft.com
Addresses: 64.4.11.37, 65.55.58.201

C:\WINDOWS>nslookup microsoft.com
Server: dns.pppoe.ca
Address: 206.248.154.22

Non-authoritative answer:
Name: microsoft.com
Addresses: 64.4.11.37, 65.55.58.201

C:\WINDOWS>nslookup microsoft.com
Server: dns.pppoe.ca
Address: 206.248.154.22

DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
*** Request to dns.pppoe.ca timed-out

C:\WINDOWS>nslookup microsoft.com
Server: dns.pppoe.ca
Address: 206.248.154.22

Non-authoritative answer:
Name: microsoft.com
Addresses: 64.4.11.37, 65.55.58.201

C:\WINDOWS>nslookup microsoft.com
Server: dns.pppoe.ca
Address: 206.248.154.22

Non-authoritative answer:
Name: microsoft.com
Addresses: 64.4.11.37, 65.55.58.201

C:\WINDOWS>nslookup microsoft.com
Server: dns.pppoe.ca
Address: 206.248.154.22

Non-authoritative answer:
Name: microsoft.com
Addresses: 64.4.11.37, 65.55.58.201

C:\WINDOWS>nslookup microsoft.com
DNS request timed out.
timeout was 2 seconds.
*** Can't find server name for address 206.248.154.22: Timed out
Server: dns2.pppoe.ca
Address: 206.248.154.170

Non-authoritative answer:
Name: microsoft.com
Addresses: 64.4.11.37, 65.55.58.201

C:\WINDOWS>nslookup microsoft.com
Server: dns.pppoe.ca
Address: 206.248.154.22

Non-authoritative answer:
Name: microsoft.com
Addresses: 64.4.11.37, 65.55.58.201


TSI Marc
Premium,VIP
join:2006-06-23
Chatham, ON
kudos:28
Looking into it.

tintin

join:2013-07-19
Scarborough, ON
Now I manually set the DNS server to the one that falkon21 suggested:

C:\WINDOWS>ipconfig/all

Windows IP Configuration

Host Name . . . . . . . . . . . . : LGX110
Primary Dns Suffix . . . . . . . :
Node Type . . . . . . . . . . . . : Unknown
IP Routing Enabled. . . . . . . . : No
WINS Proxy Enabled. . . . . . . . : No

Ethernet adapter Local Area Connection:

Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Realtek RTL8102E/RTL8103E Family PCI
-E Fast Ethernet NIC
Physical Address. . . . . . . . . : xxxxxxxxxxxxxxxxx
Dhcp Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes
IP Address. . . . . . . . . . . . : 24.52.253.xxx
Subnet Mask . . . . . . . . . . . : 255.255.255.224
Default Gateway . . . . . . . . . : 24.52.253.193
DHCP Server . . . . . . . . . . . : 24.246.76.65
DNS Servers . . . . . . . . . . . : 66.163.0.161
Lease Obtained. . . . . . . . . . : Saturday, September 07, 2013 10:20:2
4 PM
Lease Expires . . . . . . . . . . : Saturday, September 14, 2013 10:20:2
4 PM

Then I tried nslookup, 10 out of 10 times succeeded:

C:\WINDOWS>nslookup microsoft.com
Server: 66-163-0-161.ip.tor.radiant.net
Address: 66.163.0.161

Non-authoritative answer:
Name: microsoft.com
Addresses: 65.55.58.201, 64.4.11.37

C:\WINDOWS>nslookup microsoft.com
Server: 66-163-0-161.ip.tor.radiant.net
Address: 66.163.0.161

Non-authoritative answer:
Name: microsoft.com
Addresses: 64.4.11.37, 65.55.58.201

C:\WINDOWS>nslookup microsoft.com
Server: 66-163-0-161.ip.tor.radiant.net
Address: 66.163.0.161

Non-authoritative answer:
Name: microsoft.com
Addresses: 65.55.58.201, 64.4.11.37

C:\WINDOWS>nslookup microsoft.com
Server: 66-163-0-161.ip.tor.radiant.net
Address: 66.163.0.161

Non-authoritative answer:
Name: microsoft.com
Addresses: 64.4.11.37, 65.55.58.201

C:\WINDOWS>nslookup microsoft.com
Server: 66-163-0-161.ip.tor.radiant.net
Address: 66.163.0.161

Non-authoritative answer:
Name: microsoft.com
Addresses: 65.55.58.201, 64.4.11.37

C:\WINDOWS>nslookup microsoft.com
Server: 66-163-0-161.ip.tor.radiant.net
Address: 66.163.0.161

Non-authoritative answer:
Name: microsoft.com
Addresses: 64.4.11.37, 65.55.58.201

C:\WINDOWS>nslookup microsoft.com
Server: 66-163-0-161.ip.tor.radiant.net
Address: 66.163.0.161

Non-authoritative answer:
Name: microsoft.com
Addresses: 65.55.58.201, 64.4.11.37

C:\WINDOWS>nslookup microsoft.com
Server: 66-163-0-161.ip.tor.radiant.net
Address: 66.163.0.161

Non-authoritative answer:
Name: microsoft.com
Addresses: 64.4.11.37, 65.55.58.201

C:\WINDOWS>nslookup microsoft.com
Server: 66-163-0-161.ip.tor.radiant.net
Address: 66.163.0.161

Non-authoritative answer:
Name: microsoft.com
Addresses: 65.55.58.201, 64.4.11.37

C:\WINDOWS>nslookup microsoft.com
Server: 66-163-0-161.ip.tor.radiant.net
Address: 66.163.0.161

Non-authoritative answer:
Name: microsoft.com
Addresses: 64.4.11.37, 65.55.58.201

There are two possibilities:
- The TekSavvy DNS servers have some issue;
- The routing between rogers servers and the TekSavvy DNS servers have some issue;


TSI Marc
Premium,VIP
join:2006-06-23
Chatham, ON
kudos:28
Each one of those IPs is actually two different servers. 4 servers in total. It's possible one of the first ones has issues. Looking into it. Thanks for catching that. Will find out soon what the deal is.
--
Marc - CEO/TekSavvy


sbrook
Premium,Mod
join:2001-12-14
Ottawa
kudos:13
Reviews:
·TekSavvy Cable
·WIND Mobile
Marc ...

analysing tintin's tracert above

Tracing route to dns2.pppoe.ca [206.248.154.170]
over a maximum of 30 hops:

1 9 ms 8 ms 17 ms 10.126.225.1
2 20 ms 12 ms 11 ms 24.156.149.85
3 12 ms 11 ms 11 ms gw02.mtnk-6-1-2.mtnk.phub.net.cable.rogers.com [
69.63.248.1]
4 * * * Request timed out.
5 * * * Request timed out.
6 * * * Request timed out.
7 ^C


hop 1 ... the CMTS
hop 2 ... the CMTS router
hop 3 ... is the Headend router VLANed to the APOI
hop 4 ... should be teksavvy's packetflow APOI router
and that's where things go wrong

When I go to the the hop 3 I go to

s0-4-0-0.gw02.ym.phub.net.cable.rogers.com (64.185.82.125)

He's going to one of the headend routers at Newkirk Rd Richmond Hill if you're to believe the rDNS. My suspicion is that is probably just a faulty rDNS entry and it really is in YM.

BUT here's the real rub ... that router should be going reliably to teksavvy's packetflow APOI router ... but it's not. Sometimes it works, sometimes it doesn't given another tracert.

Without knowing the network topology I'll make a guess ... I think there are two paths between the MTNK router and the TekSavvy APOI router, and they're supposed to share the load, but one of those paths is broken. Sound familiar to a problem a few days ago ?

If it's only a single path, it's intermittently broken.