TekSavvy Down (EDIT: Now back online) in TekSavvy http://www.dslreports.com/forum/r22656071 en Thu, 26 Nov 2009 11:09:58 EDT Thu, 26 Nov 2009 11:09:58 EDT Re: TekSavvy Down (EDIT: Now back online) http://www.dslreports.com/forum/remark,22673614 51019512 :
said by Inssomniak See Profile :

At least it was early.
What program is that?]]> http://www.dslreports.com/forum/remark,22673614 Wed, 08 Jul 2009 13:05:55 EDT Re: TekSavvy Down (EDIT: Now back online) http://www.dslreports.com/forum/remark,22673556 LazMan :
said by Guspaz See Profile :

Consumer/business UPS are technically "sealed" lead-acid batteries that don't need to be run in any given orientation (as in they run fine upright or on their side). I'm not sure if they're gel-based or not.
"Gel-Cell" is a misnomer... VRLA's (Valve regulated lead acid) batteries do use a liquid electrolyte (aka acid) but it's absorbed in fibreglass mats between the lead plates. A large VRLA will only have about 2 cups of liquid within it, unlike "flooded" cells that may have gallons.

Headspinning - not only do I remember 220 Simcoe - I was there... ;) Bell's union wouldn't let their people in - Nortel's had no such issue - so in we went, to try and bring equipment back online.]]> http://www.dslreports.com/forum/remark,22673556 Wed, 08 Jul 2009 12:48:41 EDT Re: TekSavvy Down (EDIT: Now back online) http://www.dslreports.com/forum/remark,22671615 DKS : Peer 1's official response:

From: "Ted Smith"
Date: July 7, 2009 6:04:33 PM GMT-04:00
To:
Cc: "'Dorothy Williams'"
Subject: Toronto Data Center -- 151 Front Street Post Mortem

Dear Valued Customer,

As you are aware, over the weekend the 8th floor of PEER 1’s Toronto data center located at 151 Front Street experienced a fire with a UPS cabinet that set off a chain of events that resulted in power outages in multiple suites and network outages in others. I have detailed the events below.

PEER 1 is extremely thankful that no one was injured during the fire. The fire itself was contained within a single UPS cabinet and the Toronto fire department was able to extinguish it with the use of dry chemicals. No water was used, which significantly reduced the damage and aided in our ability to restore services to all data centers as quickly as possible.

I was personally involved in PEER 1’s response to the situation, and would like to take a moment to apologize to you for the information contained within our first few email alerts regarding the outage. When we were first organizing the team, we received an early report from our on-site staff that all power was out at 151 Front Street. That was not accurate. The fire caused a power outage in PEER 1’s suite 802. The local fire department cut power to PEER 1’s suite 709 as a preventative and safety measure to contain and extinguish the fire. The network gear providing connectivity for Suite 816 are kept in a cage in Suite 802. As a result, Suite 816 suffered a network outage. Power in the rest of the building was unaffected throughout the incident. There is a fine balance to providing information during a crisis, and we do our best to make sure updates to our customers are, quick, transparent, and accurate. In no way would we try to misrepresent the situation.

PEER 1 manages all of the equipment it provides to customers with safety and performance in mind. The UPS mentioned above has been in service for approximately four years. Its last service inspection was on July 2nd and it was only at 65% load. We do not yet have the root cause, but there was nothing in the unit’s maintenance record to indicate there was potential for a problem. Our service contractor and manufacturer are both involved in the investigation.

As part of the process to restore power quickly, we had to bypass the affected UPS. This means that suite 802 is running without UPS power until we can schedule delivery and replacement. We will expedite the process and keep customers in this suite up-to-date with the latest information on the timeline.

Outage Details:
(Timelines represent impact to “most” clients. Some specific network devices failed due to power loss and restore took longer.)
· Suite 802 – Power and network outage. Start 23:32PT End 03:10PT
· Suite 816 – Network outage. Start 23:32PT End 03:10PT
· Suite 709 – Power and network outage. Start 23:32PT End 00:50PT
· Suite 402 – Limited network outage (some latency/packet loss issues). Start 23:32PT End 00:50PT
· Firewall Clients. Network outage. Start 23:32PT End 03:10PT

Ted Smith
SVP, Operations

PEER 1
1000-555 West Hastings Street
Vancouver, BC, Canada V6B 4N5
tsmith@peer1.com
T: (678)365-2896
F: (678)365-2956
www.peer1.com
--
Need-based health care not greed-based health care.]]> http://www.dslreports.com/forum/remark,22671615 Wed, 08 Jul 2009 06:39:11 EDT Re: TekSavvy Down (EDIT: Now back online) http://www.dslreports.com/forum/remark,22664022 Inssomniak : At least it was early.
Click for full size
]]> http://www.dslreports.com/forum/remark,22664022 Mon, 06 Jul 2009 21:07:42 EDT Re: TekSavvy Down (EDIT: Now back online) http://www.dslreports.com/forum/remark,22663724 anon : The original article reference from slashdot's headline:

»www.datacenterknowledge.com/arch···-uptime/

- power outage at its Dallas data center
- Early Thursday Equinix Inc. (EQIX) data centers in Sydney, Australia and Paris each experienced power failures.
- Google App Engine, the company’s cloud computing platform, had lengthy performance problems on Thursday, experiencing high latency and data loss.
- A fire at Fisher Plaza in Seattle late Thursday night left many of the building’s data centers without power.
- Early Sunday, July 5, a fire at 151 Front Street, the major carrier hotel in Toronto, knocked out power on several floors of the facility used by Peer 1 networks. ]]> http://www.dslreports.com/forum/remark,22663724 Mon, 06 Jul 2009 20:27:30 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22661989 HeadSpinning :
said by Davesnothere See Profile :

Interesting.

I have encountered some smaller APC's where the battery had swelled to the point that it had to be Pried out.
Pregnant batteries in APC UPSs is a common problem.

I had always supsected they're running the float voltage too high. A few years ago, I was at a lunch seminar and was seated at a table with one of the founders of APC, and he basically confirmed (without actually saying it...) that some units have a problem with float voltage regulation, that might cause battery failure.]]> http://www.dslreports.com/forum/remark,22661989 Mon, 06 Jul 2009 15:21:47 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22661840 Davesnothere : Interesting.

I have encountered some smaller APC's where the battery had swelled to the point that it had to be Pried out.]]> http://www.dslreports.com/forum/remark,22661840 Mon, 06 Jul 2009 14:56:51 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22661589 Thane_Bitter : It could happen, here is an example of a battery issue in a APC smartUPS UPS (I think it’s the 2200 model, the pic looks identical to one I have...minus the damage ;) ). The right most battery has broken internally (short, overcharged, or just plain broke) and had been generating gas for some time. SLA batteries do have a vent valve that is designed to release the excess gas; however nothing is 100% reliable. It failed as well and the preasure built up over time causing the cell to swell up. Assuming the rest of the UPS is unaffected you have to disconnect everything and remove the top cover do remove swollen batteries (a rackmount UPS with batteries in it is a very heavy thing to move).
»farm3.static.flickr.com/2291/226···f4_b.jpg]]> http://www.dslreports.com/forum/remark,22661589 Mon, 06 Jul 2009 14:16:50 EDT Re: TekSavvy Down (EDIT: Now back online) http://www.dslreports.com/forum/remark,22660363 Guspaz : Consumer/business UPS are technically "sealed" lead-acid batteries that don't need to be run in any given orientation (as in they run fine upright or on their side). I'm not sure if they're gel-based or not.]]> http://www.dslreports.com/forum/remark,22660363 Mon, 06 Jul 2009 10:38:37 EDT Re: Battery Types http://www.dslreports.com/forum/remark,22659573 jfmezei : Many moons/centuries ago, I visited the Bell CO on Belmont street in Montreal. They still have crossbar equipment feeding McGill university back then. (this was one step ahead from the first generation rotary switches, and the first to support touch tone). And they still had one rack of rotary (step by step) switches, but they had just been made offline and not yet removed).

They had a large portion of a floor dedicated to batteries. They had semi transparent plastic enclosure, and you could see bubbles flowing up between the metal plates inside. Pretty neat sight.

When I worked for a bank, I did see their turbine generators on the top floor, but I don't recall their batteries. (that would have been about 10 years after visiting the Belmont one). When I visited the Whitehorse CO in 1989 they had more modern equipment there. (and got to meet some of the operators, some of whome I spoke t later on when making calling card calls :-)]]> http://www.dslreports.com/forum/remark,22659573 Mon, 06 Jul 2009 04:32:26 EDT Battery Types http://www.dslreports.com/forum/remark,22659556 Davesnothere : It's my understanding that they are a variant of Lead-Acid called a 'Gel Cell' battery and are supposed to be completely 'maintenance free' (cannot top up the electrolyte as it is not a free flowing liquid), unlike some lead-acid car batteries which were also known as 'maintenance free', though Could be topped up if you pried off some covers believing you needed to.

Those car batteries are now are called 'Low Maintenance', due to changes in / enforcement of consumer laws, I believe.]]> http://www.dslreports.com/forum/remark,22659556 Mon, 06 Jul 2009 04:15:13 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22659374 Guspaz : I believe that pretty much all UPS batteries are lead acid. In fact, my consumer grade APC unit is lead acid; all of APC's stuff is. ]]> http://www.dslreports.com/forum/remark,22659374 Mon, 06 Jul 2009 01:23:43 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22659245 TheMG : Meanwhile in Alberta not even a hiccup on my connection.

said by EG See Profile :

Why assume a lack of maintenance right away ??

Maybe there was a malfunction of the re-charger and it overcharged/overheated the battery/s ??
Is that just rumors or an actual fact from an official source that the batteries were at fault?

Anyways, I do believe the batteries for these large UPS units are lead-acid batteries (just like in your car). Those do not burst into flames when overcharged, they start spitting acid and hydrogen gas. Though with the amount of ventilation, it's unlikely hydrogen could have accumulated enough to burn/explode.

More likely there was an internal fault in the actual UPS. At the power levels we're dealing with here (several hundred kVAs) we're talking serious arcs, arc flash, and vaporized metal.

Anyways, I've seen the insides of such a UPS in person. Scary stuff, even just the sheer size of the bus bars!

Another fact many people don't know: the batteries are not normally in the same cabinet as the UPS. There's a separate cabinet for the batteries alone, which can often be in a totally separate room from the UPS itself.]]> http://www.dslreports.com/forum/remark,22659245 Mon, 06 Jul 2009 00:21:47 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22658749 DKS :
said by EG See Profile :

said by HKPolice See Profile :

Battery blew?? Why wasn't it changed regularly :(
Why assume a lack of maintenance right away ??

Maybe there was a malfunction of the re-charger and it overcharged/overheated the battery/s ??
There is anecdotal information that there has been a smell of burning electricals at peer 1 for a long time.
--
Need-based health care not greed-based health care.]]> http://www.dslreports.com/forum/remark,22658749 Sun, 05 Jul 2009 21:50:09 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22658423 EG :
said by HKPolice See Profile :

Battery blew?? Why wasn't it changed regularly :(
Why assume a lack of maintenance right away ??

Maybe there was a malfunction of the re-charger and it overcharged/overheated the battery/s ??]]> http://www.dslreports.com/forum/remark,22658423 Sun, 05 Jul 2009 20:29:15 EDT Re: TekSavvy Down (EDIT: Now back online) http://www.dslreports.com/forum/remark,22658127 Crowbar1 : Teksavvy = #1 ;)]]> http://www.dslreports.com/forum/remark,22658127 Sun, 05 Jul 2009 18:58:30 EDT Re: TekSavvy Down (EDIT: Now back online) http://www.dslreports.com/forum/remark,22657972 DKS : If you want to read all about 151 Front St. and how it came to be, see here. It's got an interesting history going back to the days of the telegraph.
--
Need-based health care not greed-based health care.]]> http://www.dslreports.com/forum/remark,22657972 Sun, 05 Jul 2009 17:59:11 EDT Re: TekSavvy Down (EDIT: Now back online) http://www.dslreports.com/forum/remark,22657922 AkFubar : Yeah
I've been checking throughout the day. Haven't had any issues either.
--
"No matter where you go, there you are." - Buckaroo Banzai]]> http://www.dslreports.com/forum/remark,22657922 Sun, 05 Jul 2009 17:42:37 EDT Re: TekSavvy Down (EDIT: Now back online) http://www.dslreports.com/forum/remark,22657875 bdevrye : No issues here

]]> http://www.dslreports.com/forum/remark,22657875 Sun, 05 Jul 2009 17:28:04 EDT Re: TekSavvy Down (EDIT: Now back online) http://www.dslreports.com/forum/remark,22657751 Horrigan : Nevamind! it's working now.]]> http://www.dslreports.com/forum/remark,22657751 Sun, 05 Jul 2009 16:51:33 EDT Re: TekSavvy Down (EDIT: Now back online) http://www.dslreports.com/forum/remark,22657712 nitzguy : Did the "make sessions die" button get pushed for ERX7?

Here's an excerpt from what I saw in the logs:
Jul 5 16:35:12 SSACRES-RTR1 local2.info pppd[4264]: Terminating on signal 15
Jul 5 16:35:12 SSACRES-RTR1 local2.info pppd[4264]: Connect time 585.7 minutes.
Jul 5 16:35:12 SSACRES-RTR1 local2.info pppd[4264]: Sent 25056034 bytes, received 141298110 bytes.
Jul 5 16:35:12 SSACRES-RTR1 local2.notice pppd[4264]: Connection terminated.
Jul 5 16:35:13 SSACRES-RTR1 local2.info pppd[4264]: Exit.

Just curious...everything was ok after things came up this morning from the outage and I noticed in this thread that you were mentioning things were unbalanced...not sure how they could be unbalanced on ERX7 but wondered maybe because some people were reporting some MLPPP on Dual-links that maybe the "Disconnect now" button was pressed :).

Just wondering so I'm not going crazy as I've been fighting some problems with it over the past few days and it was running pretty good since I fixed the MTU settings as Automatic wasn't quite cutting the mustard.

If its just a blip, I'll take as such :) Thanks for your time.]]> http://www.dslreports.com/forum/remark,22657712 Sun, 05 Jul 2009 16:40:39 EDT Re: TekSavvy Down (EDIT: Now back online) http://www.dslreports.com/forum/remark,22657449 slow joe : Uh ... never mind previous post about still slow news server speeds.

Just checked my back up server and it is at full speed on downloads. Stupid me should have checked that first. It's just very rare for my main news server server to have problems.]]> http://www.dslreports.com/forum/remark,22657449 Sun, 05 Jul 2009 15:30:39 EDT Re: TekSavvy Down (EDIT: Now back online) http://www.dslreports.com/forum/remark,22657415 slow joe : Can't get past 500Kbps on premium news server that normally runs at 4.3Mbs.

When I disconnect it takes me 15 to 20 minutes to reconnect with no difference in speed.

Teksavvy speed test shows me at 4.3Mbps though.

I'll give it a while longer I guess. ]]> http://www.dslreports.com/forum/remark,22657415 Sun, 05 Jul 2009 15:17:57 EDT Re: TekSavvy Down (EDIT: Now back online) http://www.dslreports.com/forum/remark,22657378 a1_Andy : was good while it lasted :(
]]> http://www.dslreports.com/forum/remark,22657378 Sun, 05 Jul 2009 15:02:45 EDT Re: TekSavvy Down (EDIT: Now back online) http://www.dslreports.com/forum/remark,22657326 TSI Marc : and consequently, you don't know what you're connected to. if you're slow... try reconnecting.
--
TSI Marc - TekSavvy Solutions Inc.]]> http://www.dslreports.com/forum/remark,22657326 Sun, 05 Jul 2009 14:46:58 EDT Re: TekSavvy Down (EDIT: Now back online) http://www.dslreports.com/forum/remark,22657322 smurphy30 : im on the ERX 7]]> http://www.dslreports.com/forum/remark,22657322 Sun, 05 Jul 2009 14:45:20 EDT Re: TekSavvy Down (EDIT: Now back online) http://www.dslreports.com/forum/remark,22657312 TSI Marc : Anybody who was on ERXs 1 - 5 would have been slow. I cleared some sessions, it's below peak now but still high.

The E320 has loads of capacity... if you reconnect to it.
--
TSI Marc - TekSavvy Solutions Inc.]]> http://www.dslreports.com/forum/remark,22657312 Sun, 05 Jul 2009 14:42:23 EDT Re: TekSavvy Down (EDIT: Now back online) http://www.dslreports.com/forum/remark,22657306 AkFubar : This must be only affecting the MLPPP folks my connection has been pretty good all day.
--
"No matter where you go, there you are." - Buckaroo Banzai]]> http://www.dslreports.com/forum/remark,22657306 Sun, 05 Jul 2009 14:39:51 EDT Re: TekSavvy Down (EDIT: Now back online) http://www.dslreports.com/forum/remark,22657301 smurphy30 :

i still find that mine sucks im on single link mlppp and to tell you the truth no diffenrence then before the mlppp upgrade on my account]]> http://www.dslreports.com/forum/remark,22657301 Sun, 05 Jul 2009 14:37:49 EDT Re: TekSavvy Down (EDIT: Now back online) http://www.dslreports.com/forum/remark,22657297 Jman99 : Up and down like a cheap whore today.

Seriously, wtf?]]> http://www.dslreports.com/forum/remark,22657297 Sun, 05 Jul 2009 14:36:44 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22657272 a1_Andy :
Much better Thank You :)]]> http://www.dslreports.com/forum/remark,22657272 Sun, 05 Jul 2009 14:30:39 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22657258 justen : Still not 100%, but much improved. Nice duct tape, in any event. ;)

]]> http://www.dslreports.com/forum/remark,22657258 Sun, 05 Jul 2009 14:26:29 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22657248 TSI Marc : I did a bit of duct taping. how does it look now?
--
TSI Marc - TekSavvy Solutions Inc.]]> http://www.dslreports.com/forum/remark,22657248 Sun, 05 Jul 2009 14:23:31 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22657191 a1_Andy :
Your lucky mine is just slow.... after3 reboots of router and modem :(]]> http://www.dslreports.com/forum/remark,22657191 Sun, 05 Jul 2009 14:03:45 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22657171 51019512 : Multiple. True MLPPP.

Justen just a question if you are true mlppp. Do you ever get wrong endpoint messages with the new MLPPP logins?]]> http://www.dslreports.com/forum/remark,22657171 Sun, 05 Jul 2009 13:56:40 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22657164 justen : My first response would be just to disconnect, power down, and restart everything. Given that I had such a difficult time getting back on to begin with this morning, I'm a little hesitant. Is your MLPPP single or multiple link, allegro?

justen]]> http://www.dslreports.com/forum/remark,22657164 Sun, 05 Jul 2009 13:55:15 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22657155 51019512 : My MLPPP speeds are normal as well.]]> http://www.dslreports.com/forum/remark,22657155 Sun, 05 Jul 2009 13:52:07 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22657143 justen : Trade you! :hmm:



Normally, I'm very well taken care of... :D



/me goes to turn up his speed dial knob at 151 Front Street...

justen]]> http://www.dslreports.com/forum/remark,22657143 Sun, 05 Jul 2009 13:45:54 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22657129 Mike2009 : I'm not using a multilink setup and my speeds are normal.

]]> http://www.dslreports.com/forum/remark,22657129 Sun, 05 Jul 2009 13:41:34 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22657108 justen : Well, I have connectivity, and that's good. Unfortunately, right now I'm seeing about 3.0 Mb/s to 4.0 Mb/s down, as opposed to the 8.5 Mb/s to 9.0 Mb/s down that I normally see. Up is a tad sluggish, although not nearly that bad, at about 1.0 Mb/s, compared to the usual 1.25 Mb/s.

Just after I signed up, there was some issue in which I was getting bonded speeds for upload but not for download. That sort of looks like what might be going on here, again... Any ideas?

justen]]> http://www.dslreports.com/forum/remark,22657108 Sun, 05 Jul 2009 13:34:56 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656991 51019512 :
said by CaptainObvious :

Teksavvy is down.
Your Wrong.]]> http://www.dslreports.com/forum/remark,22656991 Sun, 05 Jul 2009 12:47:43 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656936 anon : Teksavvy is down.]]> http://www.dslreports.com/forum/remark,22656936 Sun, 05 Jul 2009 12:36:28 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656909 BurnHavoc : just got up, everything seems to be back in working order here =D awesome work guys, very much appreciated.
--
"A society that gets rid of all its troublemakers goes downhill." -Heinlein]]> http://www.dslreports.com/forum/remark,22656909 Sun, 05 Jul 2009 12:19:54 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656856 a1_Andy :
said by kewlkeed See Profile :

Marc you've got my vote to hit the button :)
+1

My phone is ringing now as well... :uhh:]]> http://www.dslreports.com/forum/remark,22656856 Sun, 05 Jul 2009 12:04:46 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656765 justen :
said by TSI Marc See Profile :

actually, not yet. giving it a bit of time still to see if it starts easing up on its own.
Well, I don't know about anybody else, but I seemed to have slipped back through. Granted, it did take the better part of four hours, but better late than never...

justen]]> http://www.dslreports.com/forum/remark,22656765 Sun, 05 Jul 2009 11:39:35 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656758 TSI Marc : actually, not yet. giving it a bit of time still to see if it starts easing up on its own.
--
TSI Marc - TekSavvy Solutions Inc.]]> http://www.dslreports.com/forum/remark,22656758 Sun, 05 Jul 2009 11:37:22 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656751 justen : And... We're back... With both links. I presume you flipped the "make it work" switch, Marc? Danke. :)

justen]]> http://www.dslreports.com/forum/remark,22656751 Sun, 05 Jul 2009 11:35:01 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656720 kewlkeed : Inssomniak, lucky you. Blew me right off the grid last night, and even now I'm limping on 2 out of 6 links as the others won't re-establish and I'm getting total goobledyguk in the logs.

Got to run to the center and going to reload the entire thing from scratch and hopefully it comes up totally.

Marc you've got my vote to hit the button :)
--
Justin - DSLR resident grouch and Mr Negativity
TSI Fanboy - "Dontchya wish your 'net was hot like mine! Ohhh Dontchya!"
Have a nice day!]]> http://www.dslreports.com/forum/remark,22656720 Sun, 05 Jul 2009 11:26:37 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656692 Inssomniak :
said by TSI Marc See Profile :

We are down.

The latest news I have is that a battery blew up in our Peer1 colo space and that a fire had started. The situation appears to be under control, I'm told that they are trying to bypass the UPS' and go to direct power. Hopefully this is the case and we can get things online again. We are making our way there where we have help waiting to help if needed.

Sorry for the inconvenience.
Didnt even notice any outage.
Im sure Marc isnt pleased about being woken up in the middle of the night, but at least it was overnight, the phone rings much less during that time! ]]> http://www.dslreports.com/forum/remark,22656692 Sun, 05 Jul 2009 11:13:07 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656681 TSI Marc : It sure gives me a nice warm fuzzy feeling to know that 16 firetrucks were here. If any of those guys are reading this. thank you!
--
TSI Marc - TekSavvy Solutions Inc.]]> http://www.dslreports.com/forum/remark,22656681 Sun, 05 Jul 2009 11:10:03 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656665 inferno_gn : Hi there,

No wonder why my Internet went down in the middle of the night.

Ju Leon...
--
Otaku Anime Network
»www.otakuanime.com/]]> http://www.dslreports.com/forum/remark,22656665 Sun, 05 Jul 2009 11:06:04 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656636 TSI Marc : gonna hit the kill switch soon. should help balance things a bit.
--
TSI Marc - TekSavvy Solutions Inc.]]> http://www.dslreports.com/forum/remark,22656636 Sun, 05 Jul 2009 10:55:19 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656634 vintagewino :
said by TSI Marc See Profile :

ERX's 1 through 5 never went down, many users on those routers stayed connected even though they couldn't surf. when things came back online. and people were able to reconnect, they maxed out the capacity of ERX's 1-5, so now only 8 out of the 18 tunnel end points are accepting connections.

I have two options.. boot users and force a re-balance or wait it out and you guys have to keep trying. I think we should probably clear some sessions to at least give things a chance at getting back online.


My connection came back just before 7am according to the router. Never lost sync, but the router couldn't send the log, so can't say when things went weird here.

Just disconnected & reconnected voluntarily. Easier that than Marc hitting the big red button.

]]> http://www.dslreports.com/forum/remark,22656634 Sun, 05 Jul 2009 10:54:24 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656623 Cyborg994 : Anyone else having speed issues, I cannot even reach 2 mbit/s on any of my 2 lines... Last night each one was doing 5 mbits.

I rebooted my modems twice, and I switched each on individually, and speed is the same with both.]]> http://www.dslreports.com/forum/remark,22656623 Sun, 05 Jul 2009 10:49:52 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656621 MikeGeiger : Batteries were replaced earlier this year... I was checking the maintenance slips on those very cabinets last week.
--
»ServerNorth.net »Mike.Geiger.ca »LHNet.ca »ThePackets.net]]> http://www.dslreports.com/forum/remark,22656621 Sun, 05 Jul 2009 10:49:34 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656561 TSI Marc : ok I think we figured out what's going on.

ERX's 1 through 5 never went down, many users on those routers stayed connected even though they couldn't surf. when things came back online. and people were able to reconnect, they maxed out the capacity of ERX's 1-5, so now only 8 out of the 18 tunnel end points are accepting connections.

I have two options.. boot users and force a re-balance or wait it out and you guys have to keep trying. I think we should probably clear some sessions to at least give things a chance at getting back online.
--
TSI Marc - TekSavvy Solutions Inc.]]> http://www.dslreports.com/forum/remark,22656561 Sun, 05 Jul 2009 10:32:57 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656560 Farchord : Well I don't know about the temperature in Ontario, but if their place had an AC problem, tonight, they could have fixed it by opening a couple windows....

Man it was cold last night....]]> http://www.dslreports.com/forum/remark,22656560 Sun, 05 Jul 2009 10:32:39 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656552 nordtwilight : Great work guys !]]> http://www.dslreports.com/forum/remark,22656552 Sun, 05 Jul 2009 10:30:16 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656543 TSI Marc : I definitely used a few of those this morning.
--
TSI Marc - TekSavvy Solutions Inc.]]> http://www.dslreports.com/forum/remark,22656543 Sun, 05 Jul 2009 10:28:59 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656528 drewis : still down here. test works but it will not accept my login credentials.

says remote computer did not respond in a timely matter. ]]> http://www.dslreports.com/forum/remark,22656528 Sun, 05 Jul 2009 10:23:33 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656524 HeadSpinning :
said by TSI Marc See Profile :

said by Trisomy21 See Profile :

I bet Bell tried to Sabotage you.
LOL, not this time, I think its fair to say they had nothing to do with this. In fact I've been trying to avoid mentioning their name.
Of course, it is a four letter word.]]> http://www.dslreports.com/forum/remark,22656524 Sun, 05 Jul 2009 10:22:54 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656511 TSI Marc : yes, things are as solid as they were before. We will be looking into what can be done to move more equipment into the Switch & Data cage faster but the problem is getting the fiber cross connects up from our Bell suite. right now, we can move stuff but we don't have a way to bypass Peer1.
--
TSI Marc - TekSavvy Solutions Inc.]]> http://www.dslreports.com/forum/remark,22656511 Sun, 05 Jul 2009 10:18:02 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656502 AkFubar : Marc, is it fair to say that things are pretty solid now going forward?
--
"No matter where you go, there you are." - Buckaroo Banzai]]> http://www.dslreports.com/forum/remark,22656502 Sun, 05 Jul 2009 10:14:43 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656501 Trisomy21 : Understandable. That word/name is comparable to "cancer".]]> http://www.dslreports.com/forum/remark,22656501 Sun, 05 Jul 2009 10:14:41 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656498 TSI Marc : not seeing anything so far. no. we'll keep looking. maybe try to reload both connections?
--
TSI Marc - TekSavvy Solutions Inc.]]> http://www.dslreports.com/forum/remark,22656498 Sun, 05 Jul 2009 10:13:47 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656495 HeadSpinning :
said by fkkejrbn :

Is the redundant equipment in the same building? What happens if 151 burns down??
You're more likely to have a major power issue or other problem in the building. Although possible with all that diesel on site, a large fire isn't that likely.

What is more likely is a small fire, causing the power (including backup generators and UPSs) to be shut down for safety.

Remember 220 Simcoe's fire years ago? ]]> http://www.dslreports.com/forum/remark,22656495 Sun, 05 Jul 2009 10:12:56 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656490 TSI Marc :
said by Trisomy21 See Profile :

I bet Bell tried to Sabotage you.
LOL, not this time, I think its fair to say they had nothing to do with this. In fact I've been trying to avoid mentioning their name.
--
TSI Marc - TekSavvy Solutions Inc.]]> http://www.dslreports.com/forum/remark,22656490 Sun, 05 Jul 2009 10:11:31 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656486 Trisomy21 : more firewalls....ROFL get it?]]> http://www.dslreports.com/forum/remark,22656486 Sun, 05 Jul 2009 10:10:22 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656483 justen :
said by TSI Marc See Profile :

we're checking to see if there might be something else.
Any insight?]]> http://www.dslreports.com/forum/remark,22656483 Sun, 05 Jul 2009 10:09:25 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656481 TSI Marc : this was already a massive upgrade, had we had it completed, it would have saved us from this problem. I think we're on the right line of thinking overall.
--
TSI Marc - TekSavvy Solutions Inc.]]> http://www.dslreports.com/forum/remark,22656481 Sun, 05 Jul 2009 10:08:53 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656477 Trisomy21 : I bet Bell tried to Sabotage you.]]> http://www.dslreports.com/forum/remark,22656477 Sun, 05 Jul 2009 10:07:23 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656456 anon : how about just out of the down town core??]]> http://www.dslreports.com/forum/remark,22656456 Sun, 05 Jul 2009 09:55:35 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656455 TSI Marc : I agree that multiple locations would be even better. for now, this was the best possible plan.
--
TSI Marc - TekSavvy Solutions Inc.]]> http://www.dslreports.com/forum/remark,22656455 Sun, 05 Jul 2009 09:53:59 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656450 pnjunction : Is there where I complain that TSI should have redundant systems on the moon or at the bottom of the ocean?]]> http://www.dslreports.com/forum/remark,22656450 Sun, 05 Jul 2009 09:52:04 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656448 anon : True, but wouldn't it make more sense to have the redundant equipment in a different location??]]> http://www.dslreports.com/forum/remark,22656448 Sun, 05 Jul 2009 09:51:47 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656442 R0CKY : If 151 Front burns down, you pretty much have a national telecommunications issue.]]> http://www.dslreports.com/forum/remark,22656442 Sun, 05 Jul 2009 09:46:58 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656431 anon :
said by TSI Marc See Profile :

said by Nonteksavvyuser :

There us a thing called redundancy and disaster recovery. Teksavvy might want to investigate these. :)

Sorry bout your luck
The redundant equipment is there, just 2/3 setup. The last changes for the upgrades we've been doing were to deal with exactly these types of problems. Had this have happened a month from now we would have remained online for the most part.
Is the redundant equipment in the same building? What happens if 151 burns down??]]> http://www.dslreports.com/forum/remark,22656431 Sun, 05 Jul 2009 09:43:02 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656423 ktan :
I'm having this issue as well. I can log on to bell just fine though.
]]> http://www.dslreports.com/forum/remark,22656423 Sun, 05 Jul 2009 09:35:48 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656418 justen : After more than a twenty minute break for the second modem, I'm still stuck at single link. I'm now getting:


Eventually going back to:


I did call in and do the MLPPP switch during the downtime an hour or two ago, which is the only actual variable change here on my side... I dunno.

justen
]]> http://www.dslreports.com/forum/remark,22656418 Sun, 05 Jul 2009 09:32:45 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656388 dcorreia : My apologies, it's been a long night!]]> http://www.dslreports.com/forum/remark,22656388 Sun, 05 Jul 2009 09:14:21 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656378 AkFubar : I originally had trouble getting my connection back too. Took a couple of attempts and had to reboot my puter as well. Working well now.
--
"No matter where you go, there you are." - Buckaroo Banzai]]> http://www.dslreports.com/forum/remark,22656378 Sun, 05 Jul 2009 09:06:25 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656370 TSI Marc : we're checking to see if there might be something else.
--
TSI Marc - TekSavvy Solutions Inc.]]> http://www.dslreports.com/forum/remark,22656370 Sun, 05 Jul 2009 09:03:07 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656363 justen :
said by dcorreia See Profile :

stop trying to connect for 10 minutes. Bell has you blocked for trying too many times too quickly.
As I mentioned, I've had the throttled(?) modem offline for multiple ten+ minute breaks.

justen]]> http://www.dslreports.com/forum/remark,22656363 Sun, 05 Jul 2009 08:59:43 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656362 dcorreia : stop trying to connect for 10 minutes. Bell has you blocked for trying too many times too quickly.]]> http://www.dslreports.com/forum/remark,22656362 Sun, 05 Jul 2009 08:58:26 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656361 justen : Still having issues.

I'm currently running into this:


Or, alternatively, this:


If I swap the copper between the two modems, I can eventually get one or the other up, but so far I've not been able to re-establish multiple links. Even after multiple ten minute (or more) breaks for the errant (throttled?) modem, I'm still getting:


Any ideas?

justen
]]> http://www.dslreports.com/forum/remark,22656361 Sun, 05 Jul 2009 08:57:04 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656360 TSI Marc : We're in the suite now. We can see where the fire was. everything appears to be online.
--
TSI Marc - TekSavvy Solutions Inc.]]> http://www.dslreports.com/forum/remark,22656360 Sun, 05 Jul 2009 08:56:10 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656345 davidl : Connection is back!

Thanks to all at TekSavvy (and Peer1?) for getting on top of this so quickly on a Sunday morning :)]]> http://www.dslreports.com/forum/remark,22656345 Sun, 05 Jul 2009 08:38:10 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656341 Kareeser : For Teksavvy, a 5 hour downtime is pretty damn amazing.

Well done!]]> http://www.dslreports.com/forum/remark,22656341 Sun, 05 Jul 2009 08:34:05 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656336 twizlar : Glad all the equipment is ok, that would have sucked majorly. ]]> http://www.dslreports.com/forum/remark,22656336 Sun, 05 Jul 2009 08:29:49 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656323 51019512 :
said by justen See Profile :

said by TSI Marc See Profile :

yeah, we spent the first hour getting everything to the office to help. when we left earlier there must have been a good 20 techs that woke up in the middle of the night to come and help. we really do have a great bunch.
I hope you're providing extra coffee and other caffeinated drinks to said bunch. ;)

justen
As TSI employees come into work a onsite nurse connects them via IV to a tube system that runs in the office so that caffeine can be pumped into the techs and reps so they can keep going without the need to stop for a coffee break. They tried to push food through the tube but it didn't work.]]> http://www.dslreports.com/forum/remark,22656323 Sun, 05 Jul 2009 08:25:53 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656326 R0CKY :
said by justen See Profile :

said by TSI Marc See Profile :

yeah, we spent the first hour getting everything to the office to help. when we left earlier there must have been a good 20 techs that woke up in the middle of the night to come and help. we really do have a great bunch.
I hope you're providing extra coffee and other caffeinated drinks to said bunch. ;)

justen
It has been a long night for all.... heading to eat some breakie with a few of them right now!
--
TSI Rocky - TekSavvy Solutions Inc.

Authorized TSI employee ( »TekSavvy FAQ »Official support in the forum )
]]> http://www.dslreports.com/forum/remark,22656326 Sun, 05 Jul 2009 08:23:39 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656313 51019512 : Is there anyway this could all be blame on poor rick and his bad luck with computers?]]> http://www.dslreports.com/forum/remark,22656313 Sun, 05 Jul 2009 08:19:55 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656319 justen :
said by TSI Marc See Profile :

yeah, we spent the first hour getting everything to the office to help. when we left earlier there must have been a good 20 techs that woke up in the middle of the night to come and help. we really do have a great bunch.
I hope you're providing extra coffee and other caffeinated drinks to said bunch. ;)

justen]]> http://www.dslreports.com/forum/remark,22656319 Sun, 05 Jul 2009 08:19:49 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656314 TSI Marc : yeah, we spent the first hour getting everything to the office to help. when we left earlier there must have been a good 20 techs that woke up in the middle of the night to come and help. we really do have a great bunch.

for the auth, we did just fix a problem with one of the radius. maybe that will help.
--
TSI Marc - TekSavvy Solutions Inc.]]> http://www.dslreports.com/forum/remark,22656314 Sun, 05 Jul 2009 08:16:23 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656311 HeadSpinning :
said by TSI Marc See Profile :

you're likely being denied by the DSLAM at this point. Turn off your equipment for 5 - 10 minutes and try again after. If you try too many times it'll start thinking that you're hammering the network and block you.
The DSLAMs operate at Layer 2, so technically,won't specifically allow or deny anything. The denial would be occurring further upstream at Bell's ERX.]]> http://www.dslreports.com/forum/remark,22656311 Sun, 05 Jul 2009 08:14:33 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656303 justen : Well, I'm back. Partly. I used the downtime to call in and do the official MLPPP switch, so I'm now authenticating using copperdsl.ca instead of dslinternet.ca. Kind of impressive that, even given the disaster scenario, I got through to a rep in about one second. I guess if you're going to have your data centre catch on fire, it's best to have it happen early Sunday morning?

justen

(p.s. I, too, am only getting one link at the moment. Still getting "Timeout waiting for PADO packets" on the other link. Bummer. Going to leave the second modem off for another ten minute cycle, see if that helps.)]]> http://www.dslreports.com/forum/remark,22656303 Sun, 05 Jul 2009 08:13:13 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656302 TSI Marc :
said by AkFubar See Profile :

wow... fortunate no equipment was damaged
yeah, when i first heard about a fire. I was assuming the worst. I was freaking out pretty good. A bunch of people called and offered help if we needed it. That part was really nice to know we had options. They also kept us updated, a few were onsite and giving us updates. By the time I posted here, it seemed to be fairly obvious that no plane had crashed into the building and that we should be fairly safe. Our cage is completely at the opposite side of the room. I was also wondering with all the changes we had made, if it was somehow possible that our equipment could have started the fire. very weird. but I wrote that off quickly.. everything still shinny it's all so new.

anyway, I'm sure relieved at this point. it could easily have been worse. Even the time of night helped since the same thing during the day would have been much worse of a problem.
--
TSI Marc - TekSavvy Solutions Inc.]]> http://www.dslreports.com/forum/remark,22656302 Sun, 05 Jul 2009 08:13:00 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656284 TSI Martin : My own connection

Status 2 of 2 Connected
MLPPP Multi-link
Connection Uptime 0 days, 01:12:05]]> http://www.dslreports.com/forum/remark,22656284 Sun, 05 Jul 2009 08:04:23 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656241 51019512 : I got one line to come up on 07. 2nd line keeps getting PADO Timeouts.]]> http://www.dslreports.com/forum/remark,22656241 Sun, 05 Jul 2009 07:54:52 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656274 AkFubar : wow... fortunate no equipment was damaged]]> http://www.dslreports.com/forum/remark,22656274 Sun, 05 Jul 2009 07:54:46 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656270 TSI Marc : yes. in fact, the last maintenance we did, we removed a bunch of servers from a rack that would have been just beside the fire as I've been told. I haven't seen it for myself yet though.
--
TSI Marc - TekSavvy Solutions Inc.]]> http://www.dslreports.com/forum/remark,22656270 Sun, 05 Jul 2009 07:53:28 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656266 AkFubar : Was the fire actually in TSI's suite?]]> http://www.dslreports.com/forum/remark,22656266 Sun, 05 Jul 2009 07:50:06 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656260 TSI Marc : everything appears to back up at this point.

EDIT: yes, it sure does suck :\

--
TSI Marc - TekSavvy Solutions Inc.]]> http://www.dslreports.com/forum/remark,22656260 Sun, 05 Jul 2009 07:44:08 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656255 exseven : Wow that sucks guys. Are all your routing links back up to TorIX and Peer1?]]> http://www.dslreports.com/forum/remark,22656255 Sun, 05 Jul 2009 07:41:45 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656254 TSI Marc : you're likely being denied by the DSLAM at this point. Turn off your equipment for 5 - 10 minutes and try again after. If you try too many times it'll start thinking that you're hammering the network and block you.
--
TSI Marc - TekSavvy Solutions Inc.]]> http://www.dslreports.com/forum/remark,22656254 Sun, 05 Jul 2009 07:39:56 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656253 justen : Stuck at "Timeout waiting for PADO packets" every little bit for a while now. I wonder how long until the RADIUS server(s) can cope?

justen]]> http://www.dslreports.com/forum/remark,22656253 Sun, 05 Jul 2009 07:38:13 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656252 TSI Marc : We have an environment monitor that we setup in the Peer1 cage. Looking through the logs it appears as though temperature had been rising over the last 7 days as well as air flow sensors were tripping in the 24 hours prior to the problem. Temperature just before the power problem was a whopping 84f. Temperature jut when things came back online was 69f and it's now back to 78f.

Based on this, it would appear to me that the root of the problem is A/C related or at least that A/C was a big part of the problem. We will be looking into moving some gear out of there once we get there. We'll give Peer1 another call to see what they have to say about it all.
--
TSI Marc - TekSavvy Solutions Inc.]]> http://www.dslreports.com/forum/remark,22656252 Sun, 05 Jul 2009 07:37:53 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656240 TSI Marc : as a side note, if you were connected to ERX's 1 through 5, you would have stayed online however could not have routed out. These sessions do not need to restart their modems if they had not disconnected.
--
TSI Marc - TekSavvy Solutions Inc.]]> http://www.dslreports.com/forum/remark,22656240 Sun, 05 Jul 2009 07:26:06 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656232 TSI Marc :
said by Nonteksavvyuser :

There us a thing called redundancy and disaster recovery. Teksavvy might want to investigate these. :)

Sorry bout your luck
The redundant equipment is there, just 2/3 setup. The last changes for the upgrades we've been doing were to deal with exactly these types of problems. Had this have happened a month from now we would have remained online for the most part.
--
TSI Marc - TekSavvy Solutions Inc.]]> http://www.dslreports.com/forum/remark,22656232 Sun, 05 Jul 2009 07:19:20 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656230 TSI Marc : We're on our way to Toronto to make sure everything is back online. Gabe and I are connected via our cellphones and tweaking things while in transit. Everything appears to be back online, still checking things though.
--
TSI Marc - TekSavvy Solutions Inc.]]> http://www.dslreports.com/forum/remark,22656230 Sun, 05 Jul 2009 07:18:15 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656211 follaz : Yaaaay. I can breath again :)]]> http://www.dslreports.com/forum/remark,22656211 Sun, 05 Jul 2009 07:00:06 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656100 anon : Third link works better when:
»search.twitter.com/search?q=%23151fire]]> http://www.dslreports.com/forum/remark,22656100 Sun, 05 Jul 2009 06:55:17 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656154 anon : Well my cell rang off the hook as well, for other things.. mainly our Ontario MPLS network going up and down.. tried to logged into my VPN to check router logs and was like.. hmmmmmm

thank god for a Rogers stick... LOL.. 4

Knew something major happened when the teksavvy.com site was down.. ]]> http://www.dslreports.com/forum/remark,22656154 Sun, 05 Jul 2009 06:54:13 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656187 anon : There us a thing called redundancy and disaster recovery. Teksavvy might want to investigate these. :)

Sorry bout your luck]]> http://www.dslreports.com/forum/remark,22656187 Sun, 05 Jul 2009 06:52:49 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656206 xxgg : it's back up now!]]> http://www.dslreports.com/forum/remark,22656206 Sun, 05 Jul 2009 06:52:39 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656162 MissM : Hehe Rocky's TWitter freaked me out. I'm posting from my cell phone! I think it was B(H)ell trying to sabotage TSI! ;)]]> http://www.dslreports.com/forum/remark,22656162 Sun, 05 Jul 2009 05:31:08 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656152 follaz : So are we all stealing our neighbor's wireless connection now? :P]]> http://www.dslreports.com/forum/remark,22656152 Sun, 05 Jul 2009 05:16:24 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656150 TSI Marc : LOL, yes, the force made my cell phone freak out too
--
TSI Marc - TekSavvy Solutions Inc.]]> http://www.dslreports.com/forum/remark,22656150 Sun, 05 Jul 2009 05:14:04 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656145 dpkform : no wonder I suddenly woke in the night and immediately tried to go online. I must have sensed a disturbance in the Teksavvy "force" as if a million connections, suddenly cried out and then vanished......

the evil empire wanted to force me under their control - I have had to tether to my Bell Mobility Mobile.........

do I go back to sleep, or should I look for my tin hat?]]> http://www.dslreports.com/forum/remark,22656145 Sun, 05 Jul 2009 05:05:05 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656127 TSI Marc : *shrug*

whatever it is. It's not good.
--
TSI Marc - TekSavvy Solutions Inc.]]> http://www.dslreports.com/forum/remark,22656127 Sun, 05 Jul 2009 04:42:25 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656126 TSI Marc : New ETA, should be another hour as per Peer1's website... so roughly 6AM.
--
TSI Marc - TekSavvy Solutions Inc.]]> http://www.dslreports.com/forum/remark,22656126 Sun, 05 Jul 2009 04:41:50 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656123 HKPolice : Battery blew?? Why wasn't it changed regularly :(]]> http://www.dslreports.com/forum/remark,22656123 Sun, 05 Jul 2009 04:40:19 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656112 TSI Marc :
said by pnjunction See Profile :

So at least all of the fancy new equipment didn't go up in flames? :)
The gear is fine. Everything is fine except there's no power. I don't have an ETA, but they are working to get the power online ASAP.
--
TSI Marc - TekSavvy Solutions Inc.]]> http://www.dslreports.com/forum/remark,22656112 Sun, 05 Jul 2009 04:31:21 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656103 follaz : Well. At least we get to know what's going n through this lovely community. Get well Teksavvy :)]]> http://www.dslreports.com/forum/remark,22656103 Sun, 05 Jul 2009 04:23:14 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656102 pnjunction : So at least all of the fancy new equipment didn't go up in flames? :)]]> http://www.dslreports.com/forum/remark,22656102 Sun, 05 Jul 2009 04:17:54 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656099 TSI Marc : Just got word that we should start seeing some power shortly.
--
TSI Marc - TekSavvy Solutions Inc.]]> http://www.dslreports.com/forum/remark,22656099 Sun, 05 Jul 2009 04:13:15 EDT Re: TekSavvy Down http://www.dslreports.com/forum/remark,22656090 BurnHavoc : Peer1's forum announcement:
»forums.peer1.com/viewtopic.php?f···117#p571

City of Toronto fire services monitoring page:
»www.toronto.ca/fire/cadinfo/livecad.htm

Twitter Search #151fire:
»twitter.com/#search?q=%23151fire
--
"A society that gets rid of all its troublemakers goes downhill." -Heinlein]]> http://www.dslreports.com/forum/remark,22656090 Sun, 05 Jul 2009 04:04:44 EDT TekSavvy Down (EDIT: Now back online) http://www.dslreports.com/forum/remark,22656071 TSI Marc : EDIT: we are back online now. the fire is out, literally!

We are down.

The latest news I have is that a battery blew up in our Peer1 colo space and that a fire had started. The situation appears to be under control, I'm told that they are trying to bypass the UPS' and go to direct power. Hopefully this is the case and we can get things online again. We are making our way there where we have help waiting to help if needed.

Sorry for the inconvenience.
--
TSI Marc - TekSavvy Solutions Inc.]]> http://www.dslreports.com/forum/remark,22656071 Sun, 05 Jul 2009 03:48:10 EDT