We recently discussed why some ISPs have a difficult time quickly booting infected phish-website hosting or spam spewing bots from their networks. We decided to continue this discussion with an interview with Canadian cable provider Cogeco's Network Security Administrator, who around here goes by the nom de plume Krispy.

BBR: We've noticed a significant difference from ISP to ISP when it comes to their abuse department response efficiency and speed. What could account for the sluggishness at many of these major and deep pocketed ISPs?

Krispy: I fully believe the sluggishness - or downright inaction - shown by some providers is due to bureaucratic red-tape and costs. Bureaucracy, as we know, moves slowly - botnets do not, and the learning curve required for many executive managers to understand this threat and what it is doing to their network - sucking up bandwidth unnecessarily, causing congestion that results in inbound 'slow-speed' calls, etc - takes far too long.

Network security people need to be able to move quickly in order to try and stay in step with the miscreants and the miscreants often change direction quickly and they have little regard for timelines, process management or change control.

BBR: So a big reason for the lag is that caring - costs?

Krispy: The cost factor is probably the biggest detriment to effective botnet management. First and foremost network security and finance have historically been at odds in most companies - it's difficult for finance to sanction funds for something that *appears* to bring in no revenue and also *appears* to actually frustrate customers.

Many security departments are severely understaffed (or lacking competent technical resources due to lack of adequate dept funding) so day-to-day life generally consists of just trying to stay above water and 50 steps behind the threats with little to no time left for doing statistical analysis and reporting on the data that could clearly illustrate cost savings of a clean and relatively threat free network.

BBR: So what's the status quo on this front?

Krispy: Many customer service centers operate by adhering to a variety of call handling variables including call handle time and calls per agent, sadly these variables generally leave little breathing room for the agent to fully explore and assist the customer in resolving the issue and when it comes to botnets the issues can often be extremely difficult to resolve even for security professionals.

However I personally believe that taking the time to train employees, secure the customer's system, resolve their issues and educate the customer is a win-win situation for all involved as this results in higher employee morale, cleaner network wasting less resources on unnecessary traffic and I feel the personal attention to the customer can foster a feeling of loyalty that is rare in the ISP customer base.

That said I do not blame the customer service centres as they merely respond to the requests of the executive management, if they were instructed to spend as much time as necessary to resolve an issue they would but sadly this direction and/or support from executive management is generally absent in most ISPs.

BBR: So this is largely a top-level issue at most ISPs?

Krispy: Yes, due to the lack of technical understanding of the issue and it's effects management doesn't necessarily champion the cries from the network security department, which results in less resources and much frustration.

BBR: So it must take a thick skin to man these departments...

Krispy: Yeah, It takes a special kind of person to work in network security as you're generally the "paranoid crazy in the corner rant & raving and asking for all sorts of resources" - that is until something happens and then you're the most important person in the company...well for a little while anyway!

The job also requires the individual to occasionally stand-up to, disagree with or push back on executive management as generally management lack the in-depth technical resources and information that can curtail malicious activity on their networks.

BBRThere has been a slow improvement across the industry on the security front though, has there not? But Spam seems to be getting more attention that botnets from ISP executives...

Krispy: In my opinion the reason we see many more ISPs actioning spam reports these days moreso then in the past was because blacklists, anti-spam, anti-virus and negative PR impacted their business and their customers complained en masse. Even non-technical customers call in to complain about spam in their inbox or that they can't email their friend at ISP X.

In addition to general net pressure (ie: RBLs) most governments have begun to hoist the anti-spam flag and while it's a little too late for my liking, better late then never I guess. These days most ISP CEOs know what spam is and how it can impact their business - how many know what a botnet is and how it affects their bottom-line?

Botnets do not prevail for lack of caring on the part of security depts in most ISPs, it's due to lack of (knowledgeable) human, financial or technical resources and the lack of support from upper management. The ISP world lives on the bleeding edge and the race to put out the newest, fastest, most appealing products often utilizes most of the ISPs resources so poorly understood issues like security are oft placed in the trunk - they don't even make it to the back seat!

BBR: We've been reading more and more about a walled garden approach to tackling botnets, can you comment on this?

Krispy: Yeah, we're sorta playing around with our own home grown walled garden - which is to say we use it for a small number of threats right now but i have hopes to expand this to all threats if i ever find the time to work on it more.

Basically the concept of a walled garden is that the provider limits the customer's online activity to whatever services or pages or sites that it wants. Theoretically we could allow access to microsoft updates but not BBR, or access to @cogeco.ca email but no other email, etc. The idea is that you limit the customer's ability to infect/annoy others on the net by only letting them go certain places until their machine is cleaned.

Eventually I would like to have all threats managed this way as this lets the customer have more freedom to resolve issues themselves without having to call in plus it reduces call volume. It also causes less customer frustration as they can reconnect themselves.

BBR: It sounds like creativity is an absolute asset in your line of work...

Krispy: I have all kinds of plans but little time, some of my hairbrained schemes include using a combo of monitoring applications, anaylzers and the walled garden concept - for example the monitoring application sees an infected machine trying to talk to known botnet controller and tells our analyzer to take a more in-depth look at the infected machine's traffic for further analysis and to analyze packets for botnet control packets.

If present, then do the walled garden thing to the customer so that their machine won't be participating in attacks or spam or whatever.

BBR: As far as the customer is concerned, doesn't the success of these systems to some degree depend on them?

Krispy: Some customers just don't read the page no matter how big, bold and colorful you make it. I have re-designed the RECONNECT ME button so many times and still some people just don't bother to read so call in confused.

Also, these days some customers are actually listening to information about phishing so are leery of any site they are redirected to so won't click the button or any links within. I can't really blame them and i just haven't had enough time to sit and contemplate an easy solution to this.

BBR: So, walled gardens and total automation are an industry trend?

Krispy: Personally I feel this is the way we'll end up. Many ISPs already use the concept of walled gardens but not for abuse issues. For example many ISPs let customer's self-provision themselves, Cogeco does this - on the Cogeco network if you were to swap your modem at a storefront (because your modem got hit by lightning or something) you'd bring home the new modem, plug it in and connect your PC and you'd see authentication page and that's the only website or internet traffic you can connect to until authenticated and the new modem is added to the customers account.

Another way we currently use walled garden is that when new customers are signed up, the field tech will connect the modem and PC and then bring customer to the PC where they are confronted with a webpage (again, this is the only page/service they can use). This site displays our Terms of Service/Acceptable Use Policy and the customer has ability to read the agreement and then must click "I AGREE". Once this is done the customer can use their connection normally.

BBR: Other ISPs are exploring this walled garden system for a slew of new tasks as well?

Krispy: I know alot of ISPs use the self-provisioning walled garden, and a few are beginning to use it for TOS/AUP acceptance as well so it can be done for abuse but, as we discussed, security is the black sheep in many ISPs so often little development is done on it which, in my opinion, is ridiculous. It's an effective way of containing threats with minimum customer frustration, PLUS it can reduce inbound calls.

BBR: Where do you see this issue headed?

Krispy: I predict that as awareness of this issue grows ISP management will begin to pay attention and security depts will have to smile pleasantly as they are made aware of an issue they've been aware of for quite some time! Governments will eventually step in however as I initially stated, bureaucracy, as we know, moves slowly - botnets do not.