republican-creole
site Search:


 
   
story category
CallCentric Victim of Devastating Two-Week DDoS Attack
Users Have Reported Major Problems Since October 3
by Karl Bode Wednesday 17-Oct-2012 tags: trouble · VoIP · Callcentric
Users in our VoIP Provider forum indicate that Callcentric has been suffering from severe connectivity issues since the beginning of this month. Users in the forum first started reporting issues on October 3, and the company began investigating issues on October 4. By October 5 the company acknowledged on Twitter that they had been the victim of a sustained "sophisticated DDoS attack." The company then noted that while they hadn't found any indication of theft of personal data, they were struggling to deal with the ongoing DDoS attack:

Click for full size
For the past two days we have been experiencing a sophisticated type of attack. As soon we noticed the first attempt we commenced an immediate physical upgrade to all of our servers increasing capacity and CPU power by a factor of four in addition to other precautions.

Unfortunately even though this is similar to a "typical" DDoS attack it is targeted specifically at the SIP protocol and causes server load to increase to 100% within 1 minute of initiation. As such, standard and extraordinary prevention measures were unable to prevent it. We do not know the specific methodology of the attack but are aware that it is *similar* in effect to a DNS TRASH flood attack. We are performing forensic analysis on the data we have and are capturing traffic to find an exact reason and solution.

Almost two weeks later and customers in our forums indicate they're still having trouble with reliable service after struggling with numerous waves of DDoS attacks. Many customers have stuck with the provider given this is the first major outage they say they've seen in years. CallCentric says they've filed a report on the attacks with the FBI, and have urged users to use the company's DNS SRV servers instead of their DNS A servers. The latest Twitter statement from the company suggests that the attacks are ongoing. CallCentric says that they'll share more detail when they've completed analyzing the nature of the attacks.

"We appreciate everyone's patience with us," says the company. "We would not wish this kind of scenario on any other provider or business."

view: topics flat text 
Post a:

David
Now accepting new patients
Premium,VIP
join:2002-05-30
Granite City, IL
kudos:78
Reviews:
·DIRECTV
·AT&T Midwest
·magicjack.com
·Google Voice
·AT&T Southwest

hire anonymous to find out.

I hear they have recently did some work for a young teenage girl finding her stalker for free. I am sure if you pay them, they could probably find out really quick!

Everyone hates that law enforcement may employ criminals, but sometimes, some of the best information comes from them!

Metatron2008
Premium
join:2008-09-02
Stockbridge, GA

Re: hire anonymous to find out.

Hire a bunch of script kiddies in their parents basements?

And how do you know it isn't anon doing the attacks?

David
Now accepting new patients
Premium,VIP
join:2002-05-30
Granite City, IL
kudos:78
Reviews:
·DIRECTV
·AT&T Midwest
·magicjack.com
·Google Voice
·AT&T Southwest

Re: hire anonymous to find out.

said by Metatron2008:

Hire a bunch of script kiddies in their parents basements?

And how do you know it isn't anon doing the attacks?

Well I guess you can choose to do something or choose to do nothing. I can't imagine a script kiddie in a basement would turn down a silly thing such as money. Oh right, I forgot this is the "entitlement" generation

my bad
--
If you have a topic in the direct forum please reply to it or a post of mine, I get a notification when you do this.
Koetting Ford, Granite City, illinois... YOU'RE FIRED!!

Metatron2008
Premium
join:2008-09-02
Stockbridge, GA

Re: hire anonymous to find out.

Well if you are gonna hire somebody, you could start with 'real' hackers.

Simba7
I Void Warranties

join:2003-03-24
Billings, MT

Hang in there..

I sucks getting DDoS'd. I could only imagine the admins since it has been going on for weeks. Just hang in there.

I am curious on why CC is getting DDoS'd. Their website is having some major issues coming up, too. It just sits there.

Frink
Professor
Premium
join:2000-07-13
Scotch Plains, NJ

Firewalls

Shouldn't this company have Firewalls performing SYN & UDP Flood protection?

rmarynowski

join:1999-08-26
Tinley Park, IL

Re: Firewalls

The DDOS attack is using the SIP protocol.

Frink
Professor
Premium
join:2000-07-13
Scotch Plains, NJ

Re: Firewalls

Yea I read that, but SIP uses TCP and UDP ports to communicate the protocol, so SYN & UDP protections should be effective, no?

espaeth
Digital Plumber
Premium,MVM
join:2001-04-21
Minneapolis, MN
kudos:2
Reviews:
·Vitelity VOIP

Re: Firewalls

said by Frink:

Yea I read that, but SIP uses TCP and UDP ports to communicate the protocol, so SYN & UDP protections should be effective, no?

No.

The problem is large amounts of legitimate-looking traffic causing CPU resource contention, much the same as the SSL attacks that took down BoA / Citibank / etc a few months back.

Frink
Professor
Premium
join:2000-07-13
Scotch Plains, NJ

Re: Firewalls

Proper SYN and UDP floods are comprised of legitimate traffic, the abnormal rates are what can be detected and suppressed via Firewall Screen features.

espaeth
Digital Plumber
Premium,MVM
join:2001-04-21
Minneapolis, MN
kudos:2
Reviews:
·Vitelity VOIP

Re: Firewalls

said by Frink:

Proper SYN and UDP floods are comprised of legitimate traffic, the abnormal rates are what can be detected and suppressed via Firewall Screen features.

If you have 600,000 bots sending a single request once every 60 seconds (consistent with SIP registration), that still breaks down to about 10,000 registration requests a second.

Highly distributed attacks aren't easy to filter.

Frink
Professor
Premium
join:2000-07-13
Scotch Plains, NJ

Re: Firewalls

That's true, if they are that many sources. However another Firewall protection that can be used in those instances are Session Limit protections.
watice

join:2008-11-01
New York, NY

Re: Firewalls

Not really practical when the amount of traffic is that high. I think they're switching srv records and nullrouting ips and praying the attacks don't renew?

cdru
Go Colts
Premium,MVM
join:2003-05-14
Fort Wayne, IN
kudos:7
said by Frink:

Shouldn't this company have Firewalls performing SYN & UDP Flood protection?

It's just a hunch, but I'm going to bet that they probably have tried most of the ordinary anti-DDOS countermeasures. Callcentric isn't exactly a fly by night operation ran off of some VPS.
zerog

join:2002-02-10
Carrollton, TX
kudos:1

that sucks

Love callcentric - will not be abandoning them.
So who are the "usual suspects" for this, rival telco providers? Is it a smokescreen by attackers for other activities?

A sustained, targeted attack like this has a more nefarious purpose at it's heart.
openbox9
Premium
join:2004-01-26
japan
kudos:2

Re: that sucks

Ironically this started a couple of days after I initiated service with CC. I'll hang with them for now based on the reviews I've read. Hopefully the team continues mitigating the attack.

said by zerog:

So who are the "usual suspects" for this, rival telco providers?

I seriously doubt that. Identities won't remain hidden forever and it would be corporate suicide for competitors to stoop to this level. My guess is that it's a test run (or demonstration?) to see what is capable.
josephf

join:2009-04-26

Law Enforcement

Is this a type of scenario that law enforcement would bother getting involved in and seriously investigate?

dcurrey
Premium
join:2004-06-29

Re: Law Enforcement

Thanks to the FCC requiring e911 you would think this would fall under willful disruption of emergency services. Hence it should be a major crime.

But then again department of homeland security is more interested in copyright violations.
josephf

join:2009-04-26

Re: Law Enforcement

Another question is whether CallCentric is a large enough entity for law enforcement to seriously notice.
openbox9
Premium
join:2004-01-26
japan
kudos:2

Re: Law Enforcement

What does size matter when criminality is involved?
josephf

join:2009-04-26
Reviews:
·VoicePulse

Re: Law Enforcement

If someone illegally breaks into your Mom & Pop website or Gmail account, and defaces and/or deletes all its content, violating numerous anti-wiretapping and other statues, it is highly unlikely the FBI will dedicate any of their resources to track down the perpetrator. Perhaps your local police department will do you a favor and write-up a report; but they probably lack the necessary resources -- both technical and manpower -- to do much more.

The question is whether CallCentric is on anyone's radar. Outside of the VoIP enthusiast world and small business', they are a relatively unknown company in the wide world out there.
openbox9
Premium
join:2004-01-26
japan
kudos:2

Re: Law Enforcement

Have ma and pop actually notified the FBI? My guess is that ma and pop, at best, file a police report. More likely, ma and pop hire some two-bit "security guy" to fix their problem and then put the website back online.
said by josephf:

The question is whether CallCentric is on anyone's radar.

I'm guessing, yes. The FBI and DHS would be foolish not to consider this a threat to our national security if it were to escalate beyond one provider.
josephf

join:2009-04-26
Reviews:
·VoicePulse

Re: Law Enforcement

Even if Ma and Pa did notify the FBI it is unlikely the FBI would put much manpower into an investigation.

And when CallCentric called the FBI, the operator taking the report never heard of CallCentric, does not know whether CC is a mom and pop operation or just some entrepreneur dipping his toes in the business. So she simply files a report as she does the dozens of reports she gets every week of some hacker breaking into some website or business network.
openbox9
Premium
join:2004-01-26
japan
kudos:2

Re: Law Enforcement

How many people does it take to investigate a single webserver breach? One? Not a lot of manpower is really required?

I'm sure CC's involvement with the FBI is a little more than calling an operator and dropping off a few details about the act.
josephf

join:2009-04-26
Reviews:
·VoicePulse

Re: Law Enforcement

said by openbox9:

How many people does it take to investigate a single webserver breach? One? Not a lot of manpower is really required?

DDoS attacks are notoriously difficult to investigate and trace. Especially considering the likely international origins (from places such as China and Russia) and intermediaries of such attacks.
openbox9
Premium
join:2004-01-26
japan
kudos:2

Re: Law Enforcement

said by josephf:

DDoS attacks are notoriously difficult to investigate and trace.

Of course, and they're relatively easy to deploy and fairly effective at doing their job, as witnessed by CC and many others.

Linklist
Premium
join:2002-03-03
Longport, NJ
kudos:5
said by josephf:

said by openbox9:

How many people does it take to investigate a single webserver breach? One? Not a lot of manpower is really required?

DDoS attacks are notoriously difficult to investigate and trace. Especially considering the likely international origins (from places such as China and Russia) and intermediaries of such attacks.

It could just be one more target of the Iranian attacks on US banks and other US companies.
--
»www.gop.com/2012-republican-platform_home/
»www.gop.com/2012-republican-plat···onalism/
openbox9
Premium
join:2004-01-26
japan
kudos:2
ICE is only one part of DHS and has no relevance in these criminal attacks. DHS has a mandate to support and protect our nation's critical infrastructure, telecommunications being a big piece of that.
josephf

join:2009-04-26

Re: Law Enforcement

Who mentioned ICE? And DHS is unlikely to investigate every time someone attacks a small VoIP provider.
openbox9
Premium
join:2004-01-26
japan
kudos:2

Re: Law Enforcement

dcurrey See Profile mentioned copyright infringement as DHS' only worry...which falls under ICE's purview.
civicturbo

join:2009-11-08
USA
Reviews:
·MetConnect
That gives me an idea Call Centric should immediately add e911 service to all plan levels including the free one. Then as long as the attack continues the attackers off Call Centric are guilty of wantantly disrupting 911 emergency service. This could help CC get more FBI and police help.

I love call centric I use the free plan and its great I have also used ipKall with a free line for a free DID number and it worked great. It pisses me off that some jerks would do this to a decent provider. If they need to attack someone why can't it be one of the greedy bastard Wallstreet companies.
SayNoToPOTS

join:2012-10-12
Fair Lawn, NJ

Unfair Reporting

Karl,

CallCentric Victim of Devastating Month-Long DDoS Attack???

Last time I checked, 14 days was 2 weeks and not one month.... Sigh...
openbox9
Premium
join:2004-01-26
japan
kudos:2

Re: Unfair Reporting

Maybe he meant since the beginning of the month? This isn't the first sensational headline around here. Sadly, not much different than any "news" outlet these days.

Karl Bode
News Guy
join:2000-03-02
kudos:33
Fixed, thanks. Yes, I was intending to mean since the beginning of the month. Not trying to be sensationalist whatsoever.
neftv

join:2000-10-01
Broomall, PA

MTU

Doesn't changing the MTU of the WAN of the router alleviate DDoS attacks to some degree? Or am I thinking of something else?

Davesworld

join:2007-10-30
Everett, WA

Re: MTU

said by neftv:

Doesn't changing the MTU of the WAN of the router alleviate DDoS attacks to some degree? Or am I thinking of something else?

Sure, raise the MTU until packet fragmentation occurs, then the connection is useless to everyone including CallCentric and it's customers.

You certainly do not want to lower the MTU below ideal either. I don't see how that would do anything good here.
quixote

join:2009-12-05

Longtime Callcentric customer

Six years? Seven? Eight? This is the first problem they've ever had, and it's kind of hard to say it's their fault. I'm waiting with bated breath to find out who did it and why. They seem like a totally harmless company who actually tries to give users a good deal.

The curious thing is, I never noticed the problem on my own voip lines. I'm not a business, just personal use, so very light usage. And one day people at the other end were breaking up so badly I had to give up talking to them. I assumed it was just my crappy broadband, and maybe it was. I don't know if that's one of the symptoms they've been having. It's never happened before, but it could still be crappy Roadrunner getting even crappier.

I found out about it because I was trying some new settings and logged on to my account where all the big red messages about DDOS were on the first screen.

Sly
Premium
join:2004-02-20
Chuckey, TN
kudos:1

Re: Longtime Callcentric customer

I had the same problems with crappy audio. It's not your broadband, it's the DDoS attack. Things have gotten better lately though. I hope they find who is doing this...

USA001

@ptd.net

Just Porting Now!

I was just porting my first number to them and am hearing all of these issues. Maybe time to go elsewhere?

I think it is time the US bump any country found guilty of this crap off the net. These people have no jobs and endless time to sit and make others lives more difficult.

goodguys

@fluor.com

Partial workaround

I have a friend who is a long time customer. Callcentric is a great company and I feel very sorry for them - they don't deserve this treatment.

At the same, when you rely on your phone, this is a problem. My friend's solution is to forward incoming calls to a cell phone. You can do this on the CC website, and it seems reliable.
civicturbo

join:2009-11-08
USA

Bugsy would "handle" this back in the day in Nevada

CallCentric is a good company and dosent deserve this. I hope whoever is responsible gets "A tour of our beutiful Desert" it would be just Karma.

anonname

@rr.com

It's not an attack idiots

This is not an attack it's a routing loop within their own network.

This kind of thing is easy to isolate and they are looking in the wrong place.

Try hiring some real network engineers.

JAEB

@comcast.net

IMBECILITY

Callcentric is clueless about how to fix this. Any online service that can't address such issues in a couple of days deserves to go under.

Their excuse for support is a joke. I have had four open tickets for a least three days. Two of them have been answered with the exact same language, and they are different issues.

They can't even do something like tweet something once a day.

I have had it with these clowns and plan to close my account in the morning.

comet48

@comcast.net

Re: IMBECILITY

It's worse today than it was when it first started. I'm going to transfer my number to another service. Any recomendations? I use the Freepbx version of asterisk.

bleeby

@rr.com

still down

This cleared up a bit over the weekend and I was able to telephone another CC user a few times. But by the end of the weekend it was rapidly deteriorating again. During this week it's been largely unusable. This has been going on for 3 weeks: read, for the last 3 weeks this service has been unusable more than 95% of the time. I'm guessing they're victims of an extortion attempt by sophisticated cyber-criminals. But that's just my guess. I don't have the technical competence to determine whether that or some mismanagement on their part is to blame.

I've been more than happy with their service thus far. I'd like to stay with them but I can't live without a phone for 3 weeks. I've set up an alternatire SIP provider for the time being.

geoforce

@totalbb.net.tw

Re: still down

Callcentric has been nothing but a choke and die scenario for two months now. Last month I had far more people telling me my phone is not working than my total calls. I don't believe they even experienced a DoS attack. I think it is a cheap operation that did not have enough hardware to cover failures. They blew out all their passwords at one point. I find the people at Callcentric to be incompetent. A DoS does not last two months. These people are incompetent. Plain and simple.

headshot12

join:2001-04-02
Plano, TX

Re: still down

I think they're more insolvent than incompetent. Combating a DDoS is obviously not one of their core competencies. They should have hired an outside firm to develop a battle plan and execute it. I'm guessing they just plain don't have the funds to do this. This is what happens when a company doesn't have a good disaster recovery plan in place.
On second thought, maybe they are incompetent for not having a DRP. A shame, 'cause I really like the service and business model.
--
The natural progress of things is for liberty to yield and government to gain ground. - Thomas Jefferson
bb_user

join:2008-07-18
Toronto, ON

And Now CallCentric is *OFF* Due no Disaster Plan (Sandy)

As of 20h55 EST last night apparent Callcentric employee and DSLReports member IScream is reporting CallCentric has turned off (literally) *ALL* their services - i.e. not even DNS records for callcentric.com are in place.

Heart goes out to Callcentric employees, their families and everyone impacted by Sandy...

But...It appears CallCentric have no disaster recovery plan - up to and including keeping their customers informed. Last post to Twitter was 12+ hours ago... and of course they can't post to their website!

See talk in these forums starting with IScreams post...

»Re: CallCentric tech issues today?

This would be great time for a U.S. West Cost, Canadian or European VOIP provider to purchase some NYC based redundancy at a knock down price!

Sunday, 26-May 05:35:11 Terms of Use & Privacy | feedback | contact | Hosting by nac.net - DSL,Hosting & Co-lo
over 13.5 years online © 1999-2013 dslreports.com.