Our unique antivirus testing: How we did it in Security

said by Steve

When I was running the WildList Organization, the vast majority of the work involved verifying the viability of every virus sample received, then replicating out more samples, then verifying the viability of every single replicant. Replicants often had to be rejected. Doing this every month for over a decade, one learns just how extremely buggy viruses are.

Therefore, my statement, that samples should be suspected before antivirus products are suspected, is based on years of testing both viruses and antivirus products. The claim is not based on conjecture or opinion.

BTW. I currently work for an anti-spyware company, not an antivirus company. I work in future technologies research, not public relations.

Regards,
Joe Wells
Chief Scientist, Security Research
Sunbelt Software]]> http://www.dslreports.com/forum/remark,16767086 Wed, 23 Aug 2006 20:42:23 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16766950 anon :

quote:
In my personal opinion, even if the variants are all viable and malicious, the test results based on that files are useless and misleading for the readers, as they do not tell about how good or bad av products are in detecting new malware in the real world, as they are just artificially created slightly variants of old malware, which is not want you are going to encounter usually (in contrary to what CR stated, like also the contrary when they said that test results for this kind of goal do not exist already).

Wise words from someone who really knows about testing!

quote:
I still hope that we all will get in near future from somewhere/someone more details on the files they used, so it will be easier to argue aout the test.

mmmm...Don't hold your breath - no way CR will provide the info you (and all of us for that matter) the needed goods. ]]> http://www.dslreports.com/forum/remark,16766950 Wed, 23 Aug 2006 20:20:50 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16766640 Steve :

said by joewells

:

A fundamental truism in antivirus testing is that, if an antivirus product does not detect a virus sample, then automatically suspect the virus sample; not the antivirus product.

Well we just took a detour back down self-serving lane: you may well have the numbers to back this up, but it sounds so self-congratulatory, that it looks like you took off your technical hat and put on your PR hat.

I'm actually likely to believe this when it comes from Joe Random Idiot making a claim against the A/V product, but I think that Consumer Reports probably gave a bit more care.

Steve
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site]]> http://www.dslreports.com/forum/remark,16766640 Wed, 23 Aug 2006 19:37:10 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16766349 IBK : In my personal opinion, even if the variants are all viable and malicious, the test results based on that files are useless and misleading for the readers, as they do not tell about how good or bad av products are in detecting new malware in the real world, as they are just artificially created slightly variants of old malware, which is not want you are going to encounter usually (in contrary to what CR stated, like also the contrary when they said that test results for this kind of goal do not exist already).
I still hope that we all will get in near future from somewhere/someone more details on the files they used, so it will be easier to argue aout the test.

[P.S.: i saw that many peoples have many different opinions and views on how CR tested and the results - the above is just my personal opinion that i share with you, nothing more)]]> http://www.dslreports.com/forum/remark,16766349 Wed, 23 Aug 2006 18:58:37 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16766221 joewells :

said by Steve

I guess much of it comes down to looking at the "threats" (if CR made them available), to find out which were real threats and which were just steaming piles of bits.

A fundamental truism in antivirus testing is that, if an antivirus product does not detect a virus sample, then automatically suspect the virus sample; not the antivirus product. Unlike AV products, viruses don't go through extensive quality assurance, alpha, and beta testing. Therefore, the likelyhood of error source leans strongly toward the virus.

Oddly, many testers seem ignorant of this simple fact; and thus fall prey to a fallacious false assumption.

Regards,

Joe Wells
Chief Scientist, Security Research
Sunbelt Software]]> http://www.dslreports.com/forum/remark,16766221 Wed, 23 Aug 2006 18:41:00 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16763812 AB :

said by ghost16825

:

The key word here is variant I think. There's also a) unanswered questions about CR's exact methodology and I also think b) confusion about exactly how separate tests were weighted . . . etc. etc.

Excellent points all, ghost16825, posted with pensive logic and erudition. As have been most of the other comments in this thread (mine possibly excepted). ;)
Which would still seem to bring us around again to 'Lucy, you got some splainin' to do'!]]> http://www.dslreports.com/forum/remark,16763812 Wed, 23 Aug 2006 12:30:18 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16762408 ghost16825 : The key word here is variant I think. There's also a) unanswered questions about CR's exact methodology and I also think b) confusion about exactly how separate tests were weighted

First round: To see how quickly software makers update their signature lists, we gave all of the products Internet access. Then we spent weeks closely monitoring each product and noted how early, if at all, the manufacturer equipped it to detect newly discovered viruses.

Assumption: Newly discovered means exactly that, not create by CR

Round 2: To pit the software against novel threats not identified on signature lists, we created 5,500 new virus variants derived from six categories of known viruses, the kind you’d most likely encounter in real life.

Then we infected our lab computer with each of 185 of them to see whether the products could better detect viruses that were actively executing, based on their behavior.

Round 3:Finally, to see how often the antivirus software raised false alarms...

Question: What does this all mean?

Here's the Webster's definition of variant:

Variant
Adjective

1. Differing from a norm or standard; "a variant spelling".
Noun

1. An event that departs from expectations.

2. (biology) a group of organisms within a species that differ in trivial ways from similar groups; "a new strain of microorganisms".

3. A variable quantity that is random.

4. Something a little different from others of the same type; "an experimental version of the night fighter"; "an emery wheel is a modern variant of the grindstone".

Were they:
1) Brand new viruses using some new and fancy infection method
2) Just modified existing viruses already detected by AV vendors
3) Brand new viruses which exhibited behaviour based on 'common' infection methods, no fancy infection method
4) A mixture of some or all

Even with their technical consultants helping them, I really doubt it was choice 1. Everyone seems to think that the method used was Choice 2, but Choice 3 seems much more likely given the wording. If this is the case it really would depend on what these infection vectors were, and the test cases chosen are really important. If they are not, they're testing AV software with 'bad-like' behaviour rather than malicious executable content. If it was Choice 2 I really don't have a problem.

said by bluezanetti

:

Sure 5,500 variants were created. Were they created from 600, 60, or 6 parent samples? That matters. Details do matter. New approaches are fine, but I'd like a better sense of underlying details before embracing the product of that new approach.

I totally agree.

said by eburger68

:

...The bottom line is, you don't compound the problem by writing these things yourself, even with the best of intentions, because intentions will matter very little if the thing escapes from the lab.

One AV researcher that I know ...

You're talking about something in the class of new viruses not largely based on existing code, which may or may not be a new infection vector.

said by eburger68

:

To return to the original issue -- the full issue -- which is whether CR had any justification -- be it practical, methodological, or ethical -- to create 5500 new viruses for testing, I hope that it is becoming clear that even if one considers CR's actions but a minor or negligible transgression, that there simply was no practical or methodological justification for them.

It really depends on their test cases. On the whole, CR has justification, even if their testing methodology turns out to be absurd.

said by eburger68

:

However mistaken they might ultimately prove to be, I fail to see how modifying existing threats from today to create new variants of those threats advances the cause of anticipating tomorrow's threats either.

Now you're talking about modification of existing viruses, it's pretty easy to see how. If you have an AV which loudly talks about heuristic this and heuristic that, yet doesn't detect practically a clone of a virus which it already detects (perhaps rebased, or replaced assembly instructions with other equivalent ones) than that is pretty shocking performance, and testing labs have a right to report it. I'm not talking about two executables with wildly varying characteristics, but clones of each other in both form and behaviour.

said by SnowyOne

:

I'm not saying that activity does or doesn't happen. I'm saying it can be manipulated which doesn't say much for the staus quo.

said by eburger68

:

But if the company has the "close enough" detections today, it makes no sense for them to withhold them from regular release defs. Indeed, it makes more sense for them to include the "close enough" defs in public release defs today because there's no practical way (save special arrangement with the tester, which would call into question the validity of the test) for the AV company to guarantee that the testing entity uses the "special sauce" defs instead of public release defs. In such an event, there is no "manipulation" to speak of.

Perhaps SnowyOne is talking about something along these lines:

»(Old) Interesting AV claims/Clam response times

Here's another hypothetical situation:

Say CR decided to test right after W32BagleA came out. (Let's not include later variants like W32BagleAZ since it is debatable whether they have more in common with this virus or another virus class). Say CR modified W32BagleA and created something like the form of W32BagleB and furthermore, let's say no-one bothered to create variants of W32BagleA anywhere in the world. Would it have be right of them to test? Absolutely! Is it massively dangerous of them to do this in case it leaked out? I doubt it. What if the original file was super malicious? It is doubtful that the variant would greatly contribute to the damage already inflicted by the original.]]> http://www.dslreports.com/forum/remark,16762408 Wed, 23 Aug 2006 07:51:48 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16759030 SpannerITWks : Joe Wells

Hi,

I did read the links provided by others to your articles, and commented in earlier posts on several matters.

-

Well i was also wondering about just how many of those 5500 variants could Actually be of ANY danger in some way/s. I would guess that all of them wern't that smart, any anyware near 5500. But of course as yet we don't know, i think sooner or later we will though, one way or another !

But my point was and still is, if even a small number of them, or even one for that matter, were able to slip through the net, then a nasty is a nasty is a nasty, no matter how minor the variation. And depending where the variation/s in the code took place, the effect it could have would be real. If the alteration/s were Very clever then this might have disasterous consequences indeed.

I really think the whole concept of altered code needs to reassessed now, as a consequence of this test, and anyway. As the knowledge of this test is in the public domain, i'll bet there are any number of people out there, and not just sciddies either, who will be actively working on new minor variants. And i also believe much more clever stuff piggybacking etc on things that have gone before.

Take as an example the very current situation with for eg the - www.google.com - nasty. This is reincarnating itself at a furious pace almost daily now. I have several samples of it and they are Exactly the same file size, but different nasties.

I don't expect to see an improvement any day soon, and not just with - www.google.com - et al. Well i do actually, but only in improved nastiness !

Spanner
--
I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks
/SpannerITWks]]> http://www.dslreports.com/forum/remark,16759030 Tue, 22 Aug 2006 22:58:57 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16760382 Steve :

said by joewells

:

But first, to give you a basis for understanding my concern, let me state this:

I appreciate that another actual expert is stepping up; I'm certainly not one, and am glad to get that perspective.

My key concern is actually irrelevant as to whether or not creating 5500 viruses is ethical or unethical. What I am most interested in, is knowing is whether or not those 5500 viruses were verified as valid threats.

Which is exactly what you should be commenting on. You most likely have your own private view, but sticking to the tech stuff means you're going to retain your audience here.

Therefore, if any of the 5500 programs were actually not viable viruses, then a good antivirus product would be penalized for correctly recognizing them as non-viral.

Bravo - this is an outstanding argument for why CR's tests are not valid, and I'm very strongly persuaded by this line of reasoning.

I guess much of it comes down to looking at the "threats" (if CR made them available), to find out which were real threats and which were just steaming piles of bits.

Thank you.

Steve
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site]]> http://www.dslreports.com/forum/remark,16760382 Tue, 22 Aug 2006 21:54:57 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16752482 joewells : I seem to be getting quoted here a bit. So I guess I’ll go ahead and speak up.

One key concern I have with the CU testing has been touched upon, still I’d like to elaborate on it.

But first, to give you a basis for understanding my concern, let me state this:

Since 1991, I have worked professionally on several antivirus products (Certus AV, Novi, Norton, IBM, V-Find, Fortinet, etc.).

Moreover, I’ve also designed and/or performed antivirus tests for publication (PC World, PC Magazine, Tech TV, etc.).

In addition, I’ve also had several scientific papers and technical articles published related to antivirus testing.

Now, in 1993 I started a cooperative effort to qualify and quantify the actual clear-and-present-danger nature of the virus threat; in order to best protect users of all antivirus products. I did this (with in cooperation many other antivirus developers and testing organizations) through the WildList Organization. And a big part of that joint effort was intended to improve and empower scientifically based antivirus testing.

It is also important for you to understand that a lot of this work in test design is based on my knowing precisely how various antivirus products work. They are not simple signature-based grunt scanners; and they haven’t been since the late 1980s. They are all precision, detection engines; often referred to as scalpel scanners to reflect the precision analysis they perform in detection and verification.

So where I stand in this controversy is on top of well over a decade of working hard to establish a strong and fair scientific foundation for antivirus testing, in order to best serve users.

Hey. I’ve worked hard, for a long time, to establish solid, effective, beneficial antivirus testing criteria and methodology. That is where I stand.

Now I’ll shift gears.

My key concern is actually irrelevant as to whether or not creating 5500 viruses is ethical or unethical. What I am most interested in, is knowing is whether or not those 5500 viruses were verified as valid threats. How were the created? Who tested them to verify their viability?

Keep in mind that antivirus scalpel scanners are precision detection machines. So I’m wondering is: if the 5500 samples were not all verified as viable, then some or many may not have been viable or may even have been corrupted. And if they weren’t viable, then technically, they weren’t viruses; and a good antivirus scanner would not detect them as such.

Therefore, if any of the 5500 programs were actually not viable viruses, then a good antivirus product would be penalized for correctly recognizing them as non-viral.

That said, please don’t conclude that I am in anyway inferring that, if the 5500 were all functional, then the test was good. From where I stand, it wasn’t good.

Good testing best serves users by testing reality. That has been my mantra for years.

Joe Wells
Chief Scientist, Security Research
Sunbelt Software

Founder
WildList Organization International]]> http://www.dslreports.com/forum/remark,16752482 Mon, 21 Aug 2006 20:12:40 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16751755 SpannerITWks : Well the links that swhx7 gave, Thanx, and i'm quoting further from below, do indeed seem to confirm and validate my earlier points about testing with variants, even if the varience is only " minor " ! A few others have also agreed it may have merit too.

So where this all headed now i wonder, and who ya gonna call ? Not ghostbusters anyway, well maybe Strider Ghostbuster + the like lol.

-

Why popular antivirus apps 'do not work'

On Wednesday, the general manager of Australia's Computer Emergency Response Team (AusCERT), Graham Ingram, described how the threat landscape has changed -- along with the skill of malware authors.

-

"The most popular brands of antivirus on the market have an 80 percent miss rate. So if you are running these pieces of software, eight out of 10 pieces of malicious code are going to get in," said Ingram.

Although Ingram didn't mention any of the leading losers by name, Gartner's figures for 2005 show that Symantec is the clear leader with 53.6 percent of the market. McAfee and Trend own 18.8 percent and 13.8 percent of the market respectively.

"We are getting code of a quality that is probably worthy of software engineers. Not application developers but software engineers," said Ingram.

One vendor Ingram did mention was Russian outfit Kaspersky, which in the same tests managed to block around 90 percent of new malware.

»www.zdnet.com.au/blogs/securifyt···9,00.htm

Botnet Eavesdropping: Inside the Mocbot (MS06-040) Attack

"This was just a minor variant of something that was out there for months but the majority of scanners were missing it," he said.

Even more worrisome is the fact that the attack included the use of botnet instructions to download the second-stage Trojan executable.

"In this case, it was a spam proxy Trojan, but what if it was a rootkit? The rookits are getting so good these days that the programs we typically rely on to find and clean machines just can't see them. There is still the possibility that the spammers could slip in a rootkit to hide things forever," he said.

»www.eweek.com/article2/0,1895,2004922,00.asp

Spanner
--
I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks
/SpannerITWks]]> http://www.dslreports.com/forum/remark,16751755 Mon, 21 Aug 2006 19:48:31 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16749201 swhx7 : Related:

quote:
The top three antivirus programs -- from Symantec, McAfee, and Trend Micro -- are less likely to detect new viruses and worms than less popular programs, because virus writers specifically test their work against those programs.

zdnet article; above quotation from Schneier's Cryptogram.

This one is about a worm rather than a virus but has some relevant factoids: Eweek "Botnet Eavesdropping: Inside the Mocbot (MS06-040) Attack"

quote:
In the initial stages of the Mocbot attack, only one-third of anti-virus scanners tested by Stewart's research team were detecting the malware.
"This was just a minor variant of something that was out there for months but the majority of scanners were missing it," he said.
* * *
The lesson? "Don't get infected in the first place," Stewart said.

]]> http://www.dslreports.com/forum/remark,16749201 Mon, 21 Aug 2006 10:46:46 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16748637 Steve :

said by bluezanetti

:

To my own eye, there are internal inconsistencies in the results provided, and that's all I have to go on. Failing to get a sense of some clarification, it appears to me to be just another example of poorly executed intellectual self-abuse posing as an objective evaluation.

... which is a perfectly fair judgement, one that may well be the correct one.

I hope nobody thinks that I'm trying to claim that CR got it right: I have no idea, I'm not an expert on this and haven't read the whole report.

But I smell BS in the A/V industry reaction, and in my mind, that sheds light on something I otherwise wouldn't have thought much about.

Steve
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site]]> http://www.dslreports.com/forum/remark,16748637 Mon, 21 Aug 2006 08:37:14 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16748311 SpannerITWks : eburger68

So you're a 24/7-365 kinda guy 2 hey !

Nice to hear you say what you did, and no offense taken, was just wondering. OK i'm on standby.

Thanx,

Spanner
--
I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks
/SpannerITWks]]> http://www.dslreports.com/forum/remark,16748311 Mon, 21 Aug 2006 05:49:13 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16746972 bluezanetti :

said by Steve :
Obviously, I can't decide on this for others, and you'd be right if you're talking about a scientific paper, but the guy with spending money in his pocket makes this call.

I guess that's why it seems to matter a bit to me.

To my own eye, there are internal inconsistencies in the results provided, and that's all I have to go on. Failing to get a sense of some clarification, it appears to me to be just another example of poorly executed intellectual self-abuse posing as an objective evaluation.

And, recall, I'm a subscriber and I generally give CR the benefit of any doubt.

Blue]]> http://www.dslreports.com/forum/remark,16746972 Sun, 20 Aug 2006 22:21:42 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16746418 Steve :

said by eburger68

:

Burden of proof isn't for you to determine

Oh yes it is: if you want to convince me, I get to decide what counts.

I might be mistaken, I most certainly may be unscientific, but this is not about deciding whether God exists or whether to approve the new miracle drug: it's about where I spend my money for A/V products. The consumer decides.

Obviously, I can't decide on this for others, and you'd be right if you're talking about a scientific paper, but the guy with spending money in his pocket makes this call.

Steve
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site]]> http://www.dslreports.com/forum/remark,16746418 Sun, 20 Aug 2006 20:32:19 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16746366 eburger68 : Steve:

You wrote:

said by Steve

:

Besides: if (for whatever reason), I accept their findings, then the burden is on the A/V industry to show that it's not valid.

Sorry I can't respond to everything, but this bit begs for a quick response.

Burden of proof isn't for you to determine; it's determined by the the testing process itself. You might as well say:

"If I accept the idea that little green men from an invisible planet have visited Earth, are responsible for kidnapping human beings and processing them into cosmic dog food, and that a few captured little green men are being held in Area 51, then the burden of proof is on my critics to prove me wrong."

You make the claims, you're responsible for backing up those claims with sufficient evidence and reasoning.

Apologies to all for not being able to respond to all the intelligent points that folks have made, but I've got too much work waiting to be done. It's been an interesting and productive conversation nonetheless.

Best,

Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior]]> http://www.dslreports.com/forum/remark,16746366 Sun, 20 Aug 2006 20:20:15 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16746307 eburger68 : SpannerITWks:

Sorry, but I've had an enormous amount of work to do today (yes, I normally work weekends). Given my short schedule I had to pick and choose whom I would respond to. swhx7's post was new and esp. meaty, whereas you and I had already had a few go-arounds.

No slight or offense intended was intended. If I can work in a response over the next day or so, I will.

Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior]]> http://www.dslreports.com/forum/remark,16746307 Sun, 20 Aug 2006 20:06:30 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16746217 SpannerITWks : eburger68

Did you just forget to reply to me, or ?

Spanner]]> http://www.dslreports.com/forum/remark,16746217 Sun, 20 Aug 2006 19:46:07 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16746208 Steve :

said by eburger68

:

It is CR's burden of proof to establish that those artificially generated samples actually represent credible, potential threats that resemble what real malware authors actally might produce. It's their burden of proof to establish that those virsues are even minimally functional, let alone malicious.

We're not talking a court of law, we're talking about advice to the consumer. The industry can whine all it wants about a "burden of proof", but most others will be satisfied to accept CR's tests as an honest effort at evaluation.

Besides: if (for whatever reason), I accept their findings, then the burden is on the A/V industry to show that it's not valid.

To demand that AVs detect every existant (or potential) piece of malware in the world is not only unreasonable but counterproductive.

This is a fair point: malware is not exactly about bits, but about intentions, and reading intentions into a pile of bits you've never seen before is a very hard problem.

No, it's ethical consideration, despite what Steve says. It may not be the most burning ethical question in the world at the moement, but an ethical question it is. CR took actions which were not only methodologically unsound and unnecessary, but which constituted practices that it should have known could cause harm to others in a number of different ways -- and that such harm might ultimately actually be to the trustworthiness and integrity of the AV industry itself.

Says the industry, but I don't buy it. Ethical considerations are normally based on whether other people are harmed, and the industry seems to be breathless about all this "what might happen down the line?" considerations.

As if they're so smart that only they know that keeping a handle on this by responsible parties is impossible, and they're doing us a public service by forestalling this evil behavior.

I don't buy it.

Steve
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site]]> http://www.dslreports.com/forum/remark,16746208 Sun, 20 Aug 2006 19:43:52 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16745862 eburger68 : swhx7:

You wrote:

said by swhx7

:

- My assumptions about the CR method are based on the description in the article which said that they created variants with just enough difference to evade detection, and that it was the kind of technique that actual virus writers use. My interpretation was that the changes were such as to foil signatures without affecting functionality. This was the secondary article linked above; I haven't seen the original.

But even those measely few details aren't enough. Until bonified experts get a chance to validate the samples, we just don't know, and we shouldn't be making assumptions about the qualities, nature, and performance of those 5500 samples.

said by swhx7

:

- The idea that the author is responsible for all subsequent possessors is like the claim that citizens are responsible for everything the government does just because they get to vote once in a while: it is unreasonable to the point of absurdity. If no one could ever contract out of responsibility for anything, the economy and judicial systems would collapse with all the lawsuits and we'd have to be self-sufficient farmers with guns.

That's really not a useful analogy, because the standard for holding someone responsible for kicking off a potential chain of events is, "Was it foreseeable that X, Y, and Z would happen?"

Given the level of complexity at which governments operate and the our recognition that governments must inevitably deal with a whole range of unforeseen events, and that governments are comprised of thousands if not millions of fallible human beings, it isn't reasonable to hold citizens responsible for every last action of a government because there was no possible way for the citizens to have foreseen all the potential consequences of electing a particular government.

The situation that we're dealing with is quite a bit different.

Q. If I create lab viruses for use in a test designed to be scientifically valid, is it foreseeable that that independent experts would be required to validate the samples, thus compelling me to redistribute my creations?

A. Absolutely that is foreseeable by anyone with a rudimentary working knowledge of testing and what makes a test valid.

Q. If I re-distribute those viruses to third parties, is it foreseeable that -- viruses being what they are, and human being fallible creatures -- that further re-distribution might occur and an escape might occur at some point?

A. Yes, absolutely. Even if I don't regard the chances of an escape to be very high, it doens't take too much thinking through to realize that an escape remains in the cards as one possible outcome.

At this point, it is reasonable to hold me responsible for the consequences of actions that I could have foreseen and anticipated.

said by swhx7

:

- Testing with new variants is appropriate only for products that are claimed to protect against new variants (or suspicious code patterns/behavior, or other threats beyond those currently known). If your concept of anti-virus is something that's supposed to detect only those viruses which are already identified, then we agree that the CR method is bad. But vendors advertise more than that.

Any product that touts its heuristic or behavioral detection capabilities is essentially touting its ability to catch new, previously unknown viruses. And as been established by a number of AV authorities, there are other ways to test the effectiveness of heuristics.

said by swhx7

:

- My statement: "whatever [AV software] already does against future/unknown viruses should be equally effective against anything testers might come up with" does not suggest an impossible standard of perfection. It merely suggests that if a company claims to detect, for example, 20% of new variants, and a tester makes some new variants by the same means the virus-writers use, then the product should detect 20% of them. Why not?

Because in this case your theoretical 20 percent is meaningless. Twenty percent of what? Total new threats in the wild? But your lab viruses isn't part of the wild.

Twenty percent of the lab's daily creation? The tester's? Twenty percent of the total number of lab viruses created today (but how would we know that that number is?).

As I said, there's always going to be someone out there with a piece of malware that is capable of evading current detetion engines and sigs. The key question, though, will alwasy be: is the threat actually a threat?

said by swhx7

:

- Synthetic testing is valid only if it simulates real threats - you are right about that. I just don't buy the claim that there's no way it can ever do so.

Fine. But the burden of proof is on those who contend that these types of lab variants would yield useful results obtainable thhrough no other means. Thus far I have't seen anyone come close to meeting it; certainly CR hasn't.

said by swhx7

:

Stepping back to the broader picture, your arguments come across as "the status quo must be protected". But the status quo is a seemingly pointless whack-a-mole race that's not getting any better. This little tempest about testing methods is just a diversion from the fact that we need a whole new approach to countering malware.

I make no claim to defend the entire status quo within the field of information security or even the state of AV testing and research. There are plenty of things that need fixing and changing within these worlds. I simply ask that those who would step past a prohibition that was specifically erected to forestall certain bad things from happening prvide a sufficient justification for doing so. Again, CR hasn't been able to do this.

Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior]]> http://www.dslreports.com/forum/remark,16745862 Sun, 20 Aug 2006 18:35:12 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16745419 swhx7 : I'm not going to argue most of this. It's a long and interesting thread already and I'll just leave it for readers to decide which points are valid.

But a few particulars:

- My assumptions about the CR method are based on the description in the article which said that they created variants with just enough difference to evade detection, and that it was the kind of technique that actual virus writers use. My interpretation was that the changes were such as to foil signatures without affecting functionality. This was the secondary article linked above; I haven't seen the original.

- The idea that the author is responsible for all subsequent possessors is like the claim that citizens are responsible for everything the government does just because they get to vote once in a while: it is unreasonable to the point of absurdity. If no one could ever contract out of responsibility for anything, the economy and judicial systems would collapse with all the lawsuits and we'd have to be self-sufficient farmers with guns.

- Testing with new variants is appropriate only for products that are claimed to protect against new variants (or suspicious code patterns/behavior, or other threats beyond those currently known). If your concept of anti-virus is something that's supposed to detect only those viruses which are already identified, then we agree that the CR method is bad. But vendors advertise more than that.

- My statement: "whatever [AV software] already does against future/unknown viruses should be equally effective against anything testers might come up with" does not suggest an impossible standard of perfection. It merely suggests that if a company claims to detect, for example, 20% of new variants, and a tester makes some new variants by the same means the virus-writers use, then the product should detect 20% of them. Why not?

- Synthetic testing is valid only if it simulates real threats - you are right about that. I just don't buy the claim that there's no way it can ever do so.

Stepping back to the broader picture, your arguments come across as "the status quo must be protected". But the status quo is a seemingly pointless whack-a-mole race that's not getting any better. This little tempest about testing methods is just a diversion from the fact that we need a whole new approach to countering malware.]]> http://www.dslreports.com/forum/remark,16745419 Sun, 20 Aug 2006 17:07:26 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16744933 eburger68 : swhx7:

You wrote:

said by swhx7

:

Retrospective testing may be better, but that does not mean synthetic testing is invalid.

As I argued with several earlier posters, it's important to keep in mind burden of proof requirements. We know with retrospective testing that what is tested represents actual, real world threats developed by actual malware authors and released into the wild.

With synthetic lab viruses we don't, not only because it is not within the power of CR to predict the future (as McAfee pointed out), but because no validation has been done on those lab created viruses.

It is CR's burden of proof to establish that those artificially generated samples actually represent credible, potential threats that resemble what real malware authors actually might produce. It's their burden of proof to establish that those virsues are even minimally functional, let alone malicious.

And to meet that burden of proof CR is going to have to supply copies of those samples to an independent body for verification.

So far, they haven't even come close to meeting their burden of proof. Lacking any proof of the validity of their testing, the test cannot be assumed to be valid. Had CR doen proper retrospective testing against Wild List viruses, they wouldn't have this validity problem.

said by swhx7

:

The CR procedure is close to what actual virus writers do, and therefore a realistic simulation. I think others above have made this point too but it gets lost in the barrage.

You don't know that, I don't know that, and it's a fair bet that even CR doesn't know that. Why? Because CR created an enormous quantity of virus variants in a short period of time, and they have disclosed nothing meaningful about the virus variants they created, how they created them, or even let an independent body validate those newly created variants. See bluezanetti's post above for some of the crucial questions that would need to be answered.

It's important not to make assumptions about what you don't know. (And, btw, the fact that you don't know is no reflection on you; it's a reflection on CR.)

said by swhx7

:

Presumably, as someone stated above, any "good" virus concepts from the lab will be independently created in the wild soon if they haven't been already. This means that the lab-creation method is a valid test and AV companies ought to be using it. If definitions for the lab creations would not detect real viruses based on the same concepts, then AV products need improvement.

Once again you're assuming that you know what's sitting there in CR's lab -- you don't. If none of us know what the nature of those variants in CR's lab is, then it will be well nigh impossible to say whether those same viruses have been recreated in the wild.

said by swhx7

:

As Steve and others have pointed out, (b) is an argument for careful procedures, but does not support a prohibition. The creator must practice good containment and properly select the recipients. After the transfer, responsibility shifts to properly selected recipients.

This whole argument falls apart in the last sentence, where you attempt to shift responsibility entirely to another party -- which your argument has to, if it is to have a hope of being credible. By why would the transfer of the samples absolve the original author of responsibility, esp. when that author deliberately and knowingly created those viruses in circumstances that he or she should have known would require transfer to another party/ No, the original author remains responsible all down the line, because it was the author's actions that brought the new viruses into the world and created the circumstances that compelled others to consider accepting their transfer. And with each transfer and new possessor, the risk for escape increases.

said by swhx7

:

If testers are going to use the "new virus creation" method, their better ethical choice is to share samples with independent experts but not with vendors.

As a matter of course, this will not be practical, as it flies in the face of scientifically valid testing procedures. What you're essentially arguing is that that testers should test AV products yet refuse to disclose the test bed to the vendors. No vendor is going to be satisfied with that arrangement, and understandably so.

Moreover, withholding lab created viruses from teh vendors only increases the pressure on the vendors to start cooking up similar viruses in their own labs to compensate for or recreate what they've been denied by testing bodies.

said by swhx7

:

For-profit businesses can't be relied on to give impartial evaluations of anything. It's probably best for the independent meta-reviewers to destroy the samples after evaluating the tests.

Same problems as above, but the proposal to destroy the lab viruses soon after testing does raise the question: how sson after testing? How much chnance for independent experts to examine the test bed must be given before destruction begins?

And, btw, how is one to ensure that a tester or independent expert friendly to one of the vendors doesn't leak the samples but to one or two of the vendors but not the rest of the industry?

said by swhx7

:

Regarding (b), if an AV company is not trying to counter future, unknown (0 day) viruses, then it should advertise as being for known viruses only, and make speed the selling point. If it is tring to protect comprehensively, then whatever it already does against future/unknown viruses should be equally effective against anything testers might come up with. Again this means the CR test is valid.

You've essentially set an impossible standard. It will always be possible for someone somewhere to create some new form of malware that can slip past existing sigs and detection schemes. They key question, though, will always be: does that malware represent an actual threat to users -- a threat that is actually spreading in the wild.

What you're essentially demanding is that we return to the days before the Wild List, when testers were running tests against exotic zoo viruses created in by some hacker in Thailand and that existed nowhere but on the hard drive of said hacker. But who cares if the AVs miss that one -- it ain't a real threat.

To demand that AVs detect every existant (or potential) piece of malware in the world is not only unreasonable but counterproductive. For more on the pitfalls of this demand and what happened when the AV industry got sucked into an earlier cycle attempting to meet this demand, see the several papers by Joe Wells that have been cited in this thread.

said by swhx7

:

If creating malware in the lab helps to make the product better, then it's a good thing and businesses and consumers ought to be in favor of it.

And what's the standard for "better" here? Better protection against real threats in the wild; better protection against lab viruses created by the industry itself? Could we even distringuish what protion of the price of an AV defs subscription represented improved protection against real, in the wild threats, and what protion merely represented protection against threats created by teh AV industry itself?

said by swhx7

:

What would outrage the public is AV vendors releasing viruses, which is a very different proposition.

Oh, that would surely outrage the public. But I sincerely doubt that the public would be happy with the alternatives:

1) the AV industry creating hundreds of thousands of viable threats in the lab but refusing to release definitions for them;

2) the AV industry creating hundreds of thousands of viable threats in the lab, adding those threats into news defs, and charging users to for those definitions.

As I said several times earlier, the only clear winner in this scenario is the sales deptartments of the AV companies themselves.

said by swhx7

:

Anyway, as Steve pointed out, this is a business consideration, not ethical.

said by swhx7

:

Finally ((d)), if anti-virus companies feel compelled to inflate their numbers, this is a defect of capitalism, not of testing procedures.

This is a useless form of fatalism that attempts to shield human actors from being responsible for the consequences of their actions. We can do better than to fob off our failures on "capitalism." We can investigate and recognize causes that might feed or set the stage for such decision-making -- causes that were preventable and that had a human hand behind them.

said by swhx7

:

If lab-created viruses, if they come into wider use, would have to be added to the numbers, that's a problem for the anti-virus companies, but for the public it would merely give a more accurate picture of what's really going on.

No, it would be a problem for the public. Just why is it that you think the public would be so sanguine about being forced to pay for protection against threats created by the AV industry itself?

Here's a hypothetical: let's imagine that the time is three weeks ago, well before anyone had learned of CR's decision to create lab viruses for testing. Let's say that it came to light that members of the AV industry -- testers and researchers primarily, but a few companies as well -- had gotten into the practice of creating and using lab viruses, and that as a consequence AV companies were scrambling to add these into their detections. And, conveniently enough, this situation came to light right about the time that AV corps are announcing yet another round of price hikes for AV subscription renewals.

What do you think the reaction among forum members here at DSLR/BBR would be? I think the reaction is fairly predictable: outrage on a scale that we haven't seen for some time, with a barn-burner of a discussion being filled up with vitriolic deununciations of a corrupt AV industry creating the very threats it was selling protection for. And more than a few wouldn't hesitate to call it a racket, a scam, and demand that the heads of AV companies be thrown in jail.

said by swhx7

:

I think lab creation of viruses will be necessary to make better products. If this causes loss of confidence in the anti-virus industry, it's too bad.

Well, that's a rather sanguine expectation. Just who did you think would be using those "better products" if a widepsread loss of confidence in the AV industry occurred?

said by swhx7

:

The existence of AV as an industry is mainly a symptom of the wide reliance on an OS with Swiss-cheese security and a culture of software requiring root. Exposing weaknesses of security products will only hasten the adoption of a more Unix-like privilege regime. A that's a good thing.

Absolustely no one in this thread is opposing research into the "weaknesses of security programs" or the underlying OS. The question has always been been how to do it responsibly and how to do it in a way that didn't cause security companies to focus their time and energy pursuing chimerical threats of their own making.

Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior]]> http://www.dslreports.com/forum/remark,16744933 Sun, 20 Aug 2006 15:25:10 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16744117 swhx7 : Eric, I've basically ageed with Steve in this thread, but was trying to articulate exactly why. So I've been trying to piece together the whole argument on your side. Although these are paraphrases at best, I'm placing them in quotation boxes to distinguish who's who.

quote:
1. Testing AV products with lab-made samples is invalid:
(a) For testing detection of "known" viruses, only real ITW ones are a valid test.
(b) For testing detection of new/unknown viruses, the new ones created might not be the same as the ones the real-world authors will create, and might not be real threats, but retrospective testing is necessarily realistic.

Retrospective testing may be better, but that does not mean synthetic testing is invalid. The CR procedure is close to what actual virus writers do, and therefore a realistic simulation. I think others above have made this point too but it gets lost in the barrage.

Presumably, as someone stated above, any "good" virus concepts from the lab will be independently created in the wild soon if they haven't been already. This means that the lab-creation method is a valid test and AV companies ought to be using it. If definitions for the lab creations would not detect real viruses based on the same concepts, then AV products need improvement.

quote:
2. Testing AV products with lab-made samples, in addition to being invalid, is unethical because:
(a) If samples are not released, there can't be independent validation of the testing procedures.
(b) If samples are released to experts, the samples might leak out.

As Steve and others have pointed out, (b) is an argument for careful procedures, but does not support a prohibition. The creator must practice good containment and properly select the recipients. After the transfer, responsibility shifts to properly selected recipients.

quote:
3. Further argument that it's unethical:
(a) If testers of AV productes use lab-made samples and share with AV vendors, AV vendors have to add the samples to their definitions.
(b) If testers use lab-made samples and don't share with AV vendors, then the vendors are justified in adopting the same practice, or have to, to try to counter the new lab-virus population.
(c) In either case, the public would object once they found out that protection they're buying is partly against viruses created by AV vendors, product reviewers or both.
(d) Marketing incentives are such that vendors would have to add the lab viruses to the numbers they advertise.

If testers are going to use the "new virus creation" method, their better ethical choice is to share samples with independent experts but not with vendors. For-profit businesses can't be relied on to give impartial evaluations of anything. It's probably best for the independent meta-reviewers to destroy the samples after evaluating the tests.

Regarding (b), if an AV company is not trying to counter future, unknown (0 day) viruses, then it should advertise as being for known viruses only, and make speed the selling point. If it is tring to protect comprehensively, then whatever it already does against future/unknown viruses should be equally effective against anything testers might come up with. Again this means the CR test is valid.

And (c) ...

said by eburger68

:

[W]ould you personally feel fine shelling out money each year for a subscription to an anti-virus product's definitions when those definitions were in part necessary in order to cover viruses that AV researchers, testers, and companies were themselves cooking up in the lab?

How do you think others would react to the same proposition -- that they had to pay for protection from viruses created by parts of the AV industry itself?

If creating malware in the lab helps to make the product better, then it's a good thing and businesses and consumers ought to be in favor of it. What would outrage the public is AV vendors releasing viruses, which is a very different proposition. (Personally I don't use anti-virus.) Anyway, as Steve pointed out, this is a business consideration, not ethical.

Finally ((d)), if anti-virus companies feel compelled to inflate their numbers, this is a defect of capitalism, not of testing procedures. If lab-created viruses, if they come into wider use, would have to be added to the numbers, that's a problem for the anti-virus companies, but for the public it would merely give a more accurate picture of what's really going on.

I think lab creation of viruses will be necessary to make better products. If this causes loss of confidence in the anti-virus industry, it's too bad.

The existence of AV as an industry is mainly a symptom of the wide reliance on an OS with Swiss-cheese security and a culture of software requiring root. Exposing weaknesses of security products will only hasten the adoption of a more Unix-like privilege regime. A that's a good thing.]]> http://www.dslreports.com/forum/remark,16744117 Sun, 20 Aug 2006 12:41:01 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16743642 SpannerITWks : eburger68

My points regarding NG Malware were in response to yours, not CR. You seem to think that NG is sometime way in the future, whereas i'm saying the future can be anything from a millisecond upwards. In other words, as i already said, those NG's could land @ any time, and/or be here already, but they havn't been discovered yet, but they still would be NG. If they are NG then they are, whether they are based on x amount of old code, or not !

Actually the vendors would probably want to keep " their " zoos all to themselves anyway.

Well it depends who wrote that Sexy story, a crap writer wouldn't help of course. If you compare it to for eg one on cars. Super fast, very low fuel consumption, very reliable, lots of novel and new safety features, passes all the available tests and meets and exceeds all standards, comfortable to use, great looking etc etc. That sounds Sexy to me anyway !

The software-buying public from now on is what i was saying, not todays. The more Correct info that is put out there the better, and that goes hand in hand with the above Proper Sexy story.

Re the in house Malware -

If one of the vendors created some super duper ZOO nasty/exploit, or was sent it by a " friend " etc, what would you expect them to do about it. Realise that often parallel nasty, and non nasty etc, inventing does take place, and if they can think it up so can others, and do something about it. Or just sit on it, and wait for the Real parallel nasty to surface and possibly trash peoples PC's, and then react to it ?

Actually maybe the preventative types of codes i'm alluding to would be more suitably applicable to HIPS etc type software than AV etc ! But still, positive action would need to be taken Straight after discovery, whether aquisition of this new knowledge was in house and/or via external means.

So if you look at it that way, laterally, i think it's not very difficult to comprehend the superimposition of incorporating preventative code in various different, but complementary, ways into security products. After all prevetion is Top priority, NOT clean up after the fact !

I wouldn't be at all surprised to learn that reverse engineering, either auotomatically and/or by hands, does go on by vendors looking at competitors Defs. I'm NOT saying they then incorporate these into their Defs, but just as an exercise in " cos we can " to learn how the others do it ! Maybe they don't do it every day, but if they do or ever have, i wouldn't be very surprised.

Also remember the Malware coders spend a Lot of time reverse engineering, as well as forward engineering, so i would say it's in security vendors interests to do whatever they legally can, ie NOT stealing, to be steps ahead, and hence offer users better protection.

Spanner
--
I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks
/SpannerITWks]]> http://www.dslreports.com/forum/remark,16743642 Sun, 20 Aug 2006 10:50:41 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16743229 bluezanetti :

said by sybille :
My argument isn't really about Linux, that's just an example to show why lab viruses in themselves do not seem so dangerous for users from my point of view. And so it doesn't seem so problematic to me that people in the Consumer Reports labs have written some lab viruses in order to compare different AV programs.

I'd tend to agree, although I believe that reasonable people can disagree on this point and I do see a long term problem of operational malware being created simply for testing purposes. As with the products themselves, it will lead to an escalating arms race of functionality, except now it is in the arena of test malware.

My own position is rather simple - does this first step of synthetic malware creation effectively compromise the remainder of the test protocol and render the results suspect? I don't have enough information to know that, although seeing the relative detection rankings of KAV vs. F-Secure makes me wonder how much internal stress testing and reality checks were applied to the results given that F-Secure is KAV engined. It's a single observation, but it's hanging out there, lurking large, and completely contrary to a priori expectation.

Blue]]> http://www.dslreports.com/forum/remark,16743229 Sun, 20 Aug 2006 09:00:18 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16743197 sybille : If "lab viruses" were so potentially dangerous for users (as opposed to for the AV industry), then I wonder why we haven't seen more problems stemming from the existence of proof-of-concept lab viruses for the GNU/Linux operating system?

In fact, there have been a number of such lab viruses for Linux. In 2001, Peeling and Satchell noted that essentially all of the viruses for Linux were of the laboratory variety:

said by »www.govtalk.gov.uk/documents/Qin···tware%22 :
There are about 60,000 viruses known for Windows, 40 or so for the Macintosh, about 5 for commercial Unix versions and perhaps 40 for Linux. Most of the Windows viruses are not important, but many hundreds have caused widespread damage. Two or three of the Macintosh viruses were widespread enough to be of importance. None of the Unix or Linux viruses became widespread – most were confined to the laboratory. (p. 21)

It is interesting to note that more proof-of-concept viruses have been developed since that report was written, but no Linux viral epidemic has occurred as a result. To me, this suggests that the mere existence of such lab viruses does not present a grave danger for the computer user.

On the other hand, I expect that the existence of lab viruses would be very threatening to the AV industry, especially if these viruses are kept hidden by competing companies who consider them proprietary trade secrets. But isn't this one of the problematic consequences of the proprietary software model in general, that knowledge is hidden so that it can be used to increase profits? This is an issue for any kind of proprietary code, not just lab viruses, so it is hard for me to conclude that lab viruses are a particularly unusual or troubling case.

Really, though, I'm not too worried about whether lab viruses are dangerous for the AV industry. I don't think they need me to be worrying about their endeavors - I'm sure they have quite a large investment in doing so themselves.

Incidentally, my remarks have nothing to do with the issue of whether Linux is impervious to viruses or why there are not more in-the-wild viruses for Linux, etc., etc. An interesting discussion of those issues can be found at Rick Moen's linuxmafia page: »linuxmafia.com/~rick/faq/index.p···ge=virus

My argument isn't really about Linux, that's just an example to show why lab viruses in themselves do not seem so dangerous for users from my point of view. And so it doesn't seem so problematic to me that people in the Consumer Reports labs have written some lab viruses in order to compare different AV programs.]]> http://www.dslreports.com/forum/remark,16743197 Sun, 20 Aug 2006 08:45:28 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16742601 AB :

said by eburger68

:

You're imagining a software-buying public that is 180 degrees the opposite of the public we know and understand today. Show me that consumers are finally rejecting the bloat and burden of NIS and NAV en masse, and I might start to think you were on to something.

Good point. After all, Microsoft has 95% of the OS market, don't they? And many people are on pins & needles waiting for Vista. Go figure.

said by eburger68

said by SpannerITWks

:

General point -

Can't we get a rep from CR to post on here ?

I think this is the one proposal that everyone in this thread could agree on.

Eric L. Howes

Yep. In the immortal words of Ricky Ricardo-- Lucy, you got some 'splainin' to do!]]> http://www.dslreports.com/forum/remark,16742601 Sun, 20 Aug 2006 02:30:56 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16742524 eburger68 : SpannerITWks:

You wrote:

said by SpannerITWks

:

Who's talking about Radmin whatever code, and just Bots ? I clearly stated All Malware.

It was an example, not an exhaustive inventory of potential malware threats. Just and example to illustrate a point.

said by SpannerITWks

:

I mean how many lines of code actually need to be changed and/or added to an " old " nasty to make it NG ? I don't think you me or anyone can truly say can we !

You're right. We don't. And neither does CR, I reckon. Nor did they claim that this was the goal of their testing. So, let's stop speculating about it ourselves, huh?

said by SpannerITWks

:

Re the vendors cooking up nasties -

I already explained about that, and why, and ONLY keeping the clever stuff, and why it would make sense for them to ONLY do that.

If we could expect only scrupulous, ethical researchers to be involved in the internal decision-making at AV companies as to what to keep and what to throw out, then your hope might be justified. Unfortunately, there are other folks that would be involved in decisions like that: marketers, advertisers, middle managers, clueless senior execs even. And in the worst kind of competitive environment -- one where suspicion and mistrust ruled the day -- the premium would be on numbers, not the determination of what constituted the "clever stuff."

said by SpannerITWks

:

If people just go on thinking in todays terms regarding users, and their often understandable, in the True sense " ignorance " about software PC's etc, imagining that they won't get more knowledge through various channels, then i believe that's a mistake. All it would take is for the media to do a SEXY story on bloat etc, and the cat would well and truly be out of the bag forever.

A "sexy story on bloat"? Somehow I'm thinking that story wouldn't make it off the editor's desk. Much more likely that the editor would go with the muckraking story that I imagine in my response to Wildcatboy above. "Coverup," hidden threats, and virus researchers gone mad sells copy -- a sexy story on bloat doesn't.

said by SpannerITWks

:

And they would be comparing products, not just on price, but also lightness/speed etc etc, and about flipping time too !

You're imagining a software-buying public that is 180 degrees the opposite of the public we know and understand today. Show me that consumers are finally rejecting the bloat and burden of NIS and NAV en masse, and I might start to think you were on to something.

said by SpannerITWks

:

The media love SEXY stories, and that would definately be one, and it would be picked by other media outlets and spread. Also by word of mouth with the punters too. The vendors would be putting Big Bold statements on their boxes about how slim and lightning fast it was, as well as being effective in this n that. Hey i see a whole new marketing statergy evolving out of all this. Don't forget to bung Spanner a few $ for the idea, will you vendors !

What numbers tell the more compelling story to users looking for "comprehensive protection" against the plethora of threats in the computing universe?

* The number of viruses detected by Product A vs Product B, or...

* the benchmarked scan speeds of Product A vs. Product B?

If I'm a careful, cautious consumer without too much knowledge of the relative risks of "in the wild" viruses vs "lab viruses" (or even an inkling that such a division exists in the malware world -- a threat is a threat, isn't it?) -- then I'd rather be safe and slow than sorry.

said by SpannerITWks

:

Here's another idea -

You asked how would the "real crap" be seperated from the "potentially clever stuff" Like this -

Why couldn't some software be written that analysises AV's etc Defs and identifies Everything in there. Then whatever you want could be removed, left in, or even added to ! All it would take is a once and for all analysis, with i imagine humans intervention and approval, of ALL the zoo type stuff, then you'ld know what you want included or not. Afer that it would just be a matter of updating any new zoos by the same methods. Didn't say it would be easy, i dunno it might be, i'm not a coder, but you asked how it could be done ! You might not approve, but others might, and even take the idea up and design such a system. Bet you'ld like a copy of it hey !

Why couldn't someone do that? Well, there are these folks called lawyers. Major AV companies tend to hire a good number of them -- esp. those with backgrounds in intellectual property law. IP lawyers, as a general rule, don't look favorably on who folks who start reversing the copyrighted, patent protected software of their employers. In fact, they tend to frown on that kind of thing. You get the drift...

It's worth repeating at this point that even if you DID manage to set up an effective regime in which AV companies released defs only for the "potentially clever stuff," you'd still be asking the software buying public to swallow the proposition that they had to pay for protection againt malware created by the industry itself. I simply don't think that kind situation would be sustainable.

said by SpannerITWks

:

General point -

Can't we get a rep from CR to post on here ?

I think this is the one proposal that everyone in this thread could agree on.

Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior]]> http://www.dslreports.com/forum/remark,16742524 Sun, 20 Aug 2006 02:01:38 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16742379 eburger68 : Wildcatboy:

You wrote:

said by Wildcatboy

:

The same way it's done today. The AV industry is not proactive by nature. They simply react to the threats presented to them. If any of those 5000 viruses get out in the wild, they'll see it and they'll add the signature. If they don't make it out, the AV industry will do what they have been doing. Nothing.

But the standard for what goes into definitions in that state of affairs wouldn't be limited to what's "in the wild." In the AV dystopia that we're postulating here, the "in the wild" standard has been partially or largely abandoned, as we would be in a situation where what goes into definitions would be comprised to some (a large?) degree of lab viruses -- "potential threats" cooked up in the lab by researchers, testing entities, and (in the worst case scenario) the AV companies themselves.

I suppose one could propose a industry standard that no one added lab viruses to definitions unless there was some reasonable suspicion or confirmation that an escape had taken place. That industry agreement could even provide for sample sharing of lab viruses among vendors.

But even after such an agreement had been brokered, there would be serious pressure on the AV industry to add those lab viruses to definitions on the grounds that:

a) they were legitimate, potential threats (as SpannerITWks has argued here);

b) the best policy is always to be proactive, not reactive in response to known potential threats.

It doesn't take too much to imagine a muckraking series of articles in the mainstream media that gravely informed readers that the AV industry had hundreds of thousands of dangerous viruses in its lab but was refusing to offer its customers protection against those viruses. Throwing gasoline on the fire, these muckrakers inform the public that the industry doesn't actually know the precise number, nature, and disposition of these lab viruses, and that no one can therefore guarantee that an escape of some sort hadn't already taken place.

What do you think the response of users and customers -- the public at large -- would be to those kinds of revelations? We've got large numbers of users and consumers currently demanding that cookies be detected and removed as serious threats. Would these folks be easily persuaded that 300,00 viruses sitting in a lab somewhere posed no threat to them?

Let me clear: I don't think the single set of tests conducted by CR is sufficient to bring about the scenario outlined above. Obviously, it wouldn't be (though I wouldn't be surprised to hear that some corporate clients started asking for similar, CR-like lab virus variant testing from the commercial testing companies they hired to perform comparative testing in advance of a major software licensing purchase). What I worry about is a situation in which CR blows off the criticism of the AV industry, conducts more of these kinds of tests, and effectively forces the hand of other testing entities to keep up methodologically. From there it is a race to the bottom.

No, the only folks who benefit from a scenario in which lab virus creation becomes widely accepted are the sales departments of the AV companies themselves. And I have to believe that users and customers would eventually become justifiably embittered at having to pay for protection against potential threats created by the industry itself.

Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior]]> http://www.dslreports.com/forum/remark,16742379 Sun, 20 Aug 2006 01:18:26 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16742351 SpannerITWks : eburger68

Hi,

Who's talking about Radmin whatever code, and just Bots ? I clearly stated All Malware. And my example, extreme on purpose cos ya never know, about changing 1 Bit to make it a killer still holds up. It doesn't matter if it's 20 years old or from 2 mins ago, if some " old " code gets reworked into something tasty, well it's still new.

You're not telling us that the vendors would ignore it as old and not include in its Defs are you ? And if it turned out that a small amount of code, 1 Bit or whatever, turned it into an NG type nasty because that change actually enabled it to assume that status, then it would be.

I mean how many lines of code actually need to be changed and/or added to an " old " nasty to make it NG ? I don't think you me or anyone can truly say can we ! The NG aspect refers to it being capable of infiltrating by, unknown to us at this moment, holes in our systems, and/or bugs in software, that are exploited very craftily, and/or stealthily, and/or maybe very hard not only to discover, but also to remove, if possible !

As i said NG could be here right now, and we wouldn't know it, and it does NOT need to be 100% brand new code. All it takes is what it takes to accomlish it, whether it's millions of lines of code or a helluva lot less. Actually less is Much better, and i think that's the way a lot of new stuff will be headed.

Re the vendors cooking up nasties -

I already explained about that, and why, and ONLY keeping the clever stuff, and why it would make sense for them to ONLY do that. If people just go on thinking in todays terms regarding users, and their often understandable, in the True sense " ignorance " about software PC's etc, imagining that they won't get more knowledge through various channels, then i believe that's a mistake. All it would take is for the media to do a SEXY story on bloat etc, and the cat would well and truly be out of the bag forever. And they would be comparing products, not just on price, but also lightness/speed etc etc, and about flipping time too !

The media love SEXY stories, and that would definately be one, and it would be picked by other media outlets and spread. Also by word of mouth with the punters too. The vendors would be putting Big Bold statements on their boxes about how slim and lightning fast it was, as well as being effective in this n that. Hey i see a whole new marketing statergy evolving out of all this. Don't forget to bung Spanner a few $ for the idea, will you vendors !

Here's another idea -

You asked how would the "real crap" be seperated from the "potentially clever stuff" Like this -

Why couldn't some software be written that analysises AV's etc Defs and identifies Everything in there. Then whatever you want could be removed, left in, or even added to ! All it would take is a once and for all analysis, with i imagine humans intervention and approval, of ALL the zoo type stuff, then you'ld know what you want included or not. Afer that it would just be a matter of updating any new zoos by the same methods. Didn't say it would be easy, i dunno it might be, i'm not a coder, but you asked how it could be done ! You might not approve, but others might, and even take the idea up and design such a system. Bet you'ld like a copy of it hey !

-

General point -

Can't we get a rep from CR to post on here ?

Spanner
--
I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks
/SpannerITWks]]> http://www.dslreports.com/forum/remark,16742351 Sun, 20 Aug 2006 01:11:33 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16742319 Blackbird :

said by Wildcatboy

:

... The AV industry is not proactive by nature. They simply react to the threats presented to them. If any of those 5000 viruses get out in the wild, they'll see it and they'll add the signature. If they don't make it out, the AV industry will do what they have been doing. Nothing.

In my mind, I'm trying to figure out how any AV system could be truly effective "proactively" in the broadest sense of the word.
-- Is it possible to know in advance that some coding which "exploits" a particular behavior of some software (perhaps manipulating a previously unknown vulnerability in it) is not, in fact, a legitimate callup of that functionality? In advance, how does one make the determination of evil intent, especially when one may not even know the vulnerability exists?
-- Is it feasible for any AV to determine (and test for) every permutation and combination of bits in all variants that might ever come to exist for just one basic signature (especially with bit-padding, etc) of a common, "known" virus?
-- If the AV analyzes for "suspicious" behavior in candidate code, how is one to define as suspicious a truly novel form of exploitive behavior - one not previously seen before?

It seems to me that unless the AV can accomplish all these with a high degree of accuracy, there can never be effective proactivity, even in theory. Which is to say that all AV's, of necessity, would have to predominate on the side of reactivity - just by the nature of the problem.
--
If God wanted us to work with electrons, He'd make them big enough to see...]]> http://www.dslreports.com/forum/remark,16742319 Sun, 20 Aug 2006 01:02:25 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16742243 Wildcatboy :

said by eburger68

:

In that kind of a state of affairs, just how would you propose to suss out the "serious potential threats" from the "negligible potential threats"?

The same way it's done today. The AV industry is not proactive by nature. They simply react to the threats presented to them. If any of those 5000 viruses get out in the wild, they'll see it and they'll add the signature. If they don't make it out, the AV industry will do what they have been doing. Nothing.
--
You can catch the Devil, but you can't hold him long.]]> http://www.dslreports.com/forum/remark,16742243 Sun, 20 Aug 2006 00:41:09 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16742182 eburger68 : SnowyOne:

Don't try to play victim with me. Your post was a grammatical mess that was easily misread, and the fault for that lies entirely on your side.

If you want to respond to part where I actually answered your actual hypothetical, then please do so. If you want to continue playing victim, then I'll consider this exchange with you finished.

Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior]]> http://www.dslreports.com/forum/remark,16742182 Sun, 20 Aug 2006 00:25:10 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16742145 eburger68 : SpannerITWks:

I'm sorry, but I remain unconvinced that twiddling with existing viruses in the manner that CR reports doing is likely to produce anything as radical as a next generation threat. And, no, there is no contradiction whatsoever in allowing that next gen threats may re-use some old code, but in a way that completely surpasses the capabilities and functionality of the malware from which that code was originally taken. Example: your typical bot-net will use some old Radmin code, but the bot-net software we're seeing today (servers, clients, and admin consoles) are far beyond your old Radmin setup.

You wrote:

quote:
Don't need to mate, i use Free AV, and others are free to do what they like, including paying for zoo. But if i was shelling out $ for one, which i probably will sooner or later as i'm not against paying @ all, then Yes.

Yes, you'd be willing to pay for protection from viruses cooked up by the AV industry itself? I'm having hard time believing this. Maybe it would be the case in your situation, but I seriously doubt that many others would be happy with that state of affairs --- in fact, I would expect many would charge the AV industry with running a fraudulent protection racket.

Remember: we've already got people right now who are convinced the AV industry is releasing viruses to create a market, and they are outraged. I would be, too, if I were convinced the industry was actually engaged in those kinds of shenanigans.

quote:
But here's da fing, if that situation arose in the future, cos it would be the future, and not 1823 or whatever, the media/forums etc are in full knowledge of the Bad 'ole days of the numbers game, not to mention bloat which gets talked about a lot more than ever before. And the vendors know it. So it wouldn't take too many of them to start playing those worn out games, before they got tripped up. So i imagine they would kick out the real crap, and just include the potentially clever stuff.

You need to crack your history books -- or at least review the history of the AV industry, because the very media that was supposed to be watchdogging the AV industry got sucked into the "numbers game" to a large extent. One only has to look at the raft of complaints in this very forum about the (in)accuracy of mainstream news media coverage of security issues to doubt that the media would be able to rein in this problem.

But the more interesting question is how you would propose the media and forums in their watchdog role separate the "real crap" from the "potentially clever stuff"?

We're going to be facing a situation in which many organizations, including AV companies, might have an undetermined number lab viruses -- the nature and seriousness remaining unknown in any great detail. Some sharing of samples might have occurred between some entitites, but as the AV companies view their research data as proprietary and essential to maintaining a competitive edge, data about these lab viruses would likely remain in a wretched state, though rumours would carry the day.

In that kind of a state of affairs, just how would you propose to suss out the "serious potential threats" from the "negligible potential threats"?

Eric L. Howes
--
Microsoft MVP

Sunbelt Software

Spyware Warrior]]> http://www.dslreports.com/forum/remark,16742145 Sun, 20 Aug 2006 00:16:47 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16742073 SnowyOne :

said by eburger68

:

No, I figured out that you hadn't written a grammtically coherent sentence that one could make heads or tails of in one pass.

That answers why you wrote it, not why you decided to keep it in your post

said by eburger68

:

I wrongly assumed that I had parsed the thing correctly,

Again, That answers why you wrote it, not why you decided to keep it in your post

said by eburger68

:

but given its general incoherence I guess I should have been suspicious.

Instead of putting so much focus & energy into being "suspicious", you could have just answered the question.

said by eburger68

:

I won't make that mistake again.

We all make mistakes. Standing up to them is what separates the sincere from the insincere. Putting the blame on things such as "grammtically coherent sentence that one could make heads or tails of in one pass." is neither sincere or genuine.

said by eburger68

:

At any rate, I did give you a proper answer once I deciphered what it was you were trying to say.

I'll just assume this is your way of somewhat accepting responsibilty of the fact that your previous answers weren't proper.

said by eburger68

:

You're welcome to respond only to the last part.
Eric L. Howes

Since when did you think you have the authority to dictate what I may or may not respond to?]]> http://www.dslreports.com/forum/remark,16742073 Sun, 20 Aug 2006 00:03:36 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16742012 eburger68 : EGeezer:

You wrote:

quote:
It's good that these concerns have been raised by critics of the CU tests - I'd like to see such standards of disclosure of details applied to all tests - including those of CU, the industry media and the vendors' own tests. I think it would be at once revealing and informative to the prospective customer to have them published.

And in the next post...

quote:
Links to the details will be helpful, as would links to the vendor's own tests that back up their claims of superiority in their ads and websites.

First, I have never pretended and am not about to start to defend tests conducted by the vendors themselves or conducted under contract for the vendors by testing entities -- esp. those that are used for marketing purposes. Those tests are so obviously compromisd that one would have to be a fool to stand up for those tests.

Second, for those interested in how reputable, independent testing entities conduct and report their testing, see the following:

AV-comparatives
»www.av-comparatives.org/

AV-comparatives - sample report
»www.av-comparatives.org/seiten/e···rt01.pdf

AV-comparatives - methodology
»www.av-comparatives.org/seiten/e···logy.pdf

Virus Bulletin/VB100
»www.virusbtn.com/vb100/index

VB100 - Latest Comparative
»www.virusbtn.com/vb100/latest_co···ve/index

VB100 - Procedures
»www.virusbtn.com/vb100/about/100···dure.xml

VB100 - The WIld List (i.e., the test bed)
»www.virusbtn.com/resources/wildl···ndex.xml

* Note: to access most VB100 material, you will need to go through an online registration, which is free.

As most reputable tests rely in whole or part on the Wild List, see also:

»www.wildlist.org/

PC World conducted an anti-virus test in January that was designed to test just how well AV programs responded to "unknowns":

»www.pcworld.com/article/id,12416···cle.html

The test was conducted by AV-Test.org, and the methodology (which involved 9 separate tests) is described here:

»www.pcworld.com/article/id,12416···cle.html

Notice the use of retrospective testing.

Here's another PC World AV test conducted by Joe Wells:

»pcworld.about.com/magazine/1909p···5803.htm

Methodology and test bed are described 3/4 down the page in the "How We Tested" section.

Note that I am less than satisfied with many of the major tech-mag tests, including the AV testing, because so often the methodology and test bed are not completely disclosed. When tech-mags do use the Wild List, though, one can at least visit the Wild List page to learn what malware was included in any particular edition of the Wild List, which makes the test bed fairly transparent. (Getting samples will be another task, though.)

And see the home page for AV-Test.org for a handy cross-reference table of names used by all the major AV corps for the threats on the Wild List:

»www.av-test.org/

Keep in mind that recognized researchers and experts as well as the tested AV companies themselves can typically get access to the samples that were used in testing plus more detailed reports of test results.

Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior]]> http://www.dslreports.com/forum/remark,16742012 Sat, 19 Aug 2006 23:47:18 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16741877 eburger68 : SnowyOne:

You asked:

said by SnowyOne

:

So you figured out that you didn't understand what I was saying & decided to post to your misunderstanding anyway?
Am I supposed to respond to a former misunderstanding of yours?? Why else would you insist keeping it in your post?
To embarress? To intimidate? To silence?

No, I figured out that you hadn't written a grammtically coherent sentence that one could make heads or tails of in one pass. I wrongly assumed that I had parsed the thing correctly, but given its general incoherence I guess I should have been suspicious. I won't make that mistake again.

At any rate, I did give you a proper answer once I deciphered what it was you were trying to say. You're welcome to respond only to the last part.

Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior]]> http://www.dslreports.com/forum/remark,16741877 Sat, 19 Aug 2006 23:20:42 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16741776 sheiny :

said by alexeck

:

The CORRECT way to test heuristics is extremely simple: Turn off definitions for all the products being tested, and test against new virus strains after a few weeks or months. That's the only honest, correct approach, as it a) tests against the real-world, b) doesn't turn you into a virus creator/distributor and c) actually gets you the right results.

This approach neglects the fact that black-hats possess an extremely important bit of knowledge: the current state of AV detections. Whenever they wish they can test new viruses or strains in the lab until the have an undetected variant.They can test against as wide a sample of AVs as they choose. Hence the terrible performance as seen in the AV-comparatives.org tests.

Pro-active AV detection is a delusion. It assumes black-hats will not use the knowledge they have. Such assumptions are both foolish and dangerous.

The one thing retrospective tests will never be able to tell us is how high the bar has been raised. Is it as trivial as as hex editing an existing virus, or a much more complex task.

The only way to quantify the level of pro-active protection an AV provides is to assume the role of the black-hat and try to bypass it. This is similar to the tasks of penetration testers and vulnerability researchers.

If I am told a new layer has been added to protect me, I have also been given the right to test that layer. In this sense, the AV vendors not only allow but * require * that new viruses are created.

Only then will I be able to make an informed decision about the risks I face. The right to make such decisions trumps any ethic dreamt up by AV vendors.

The way to avoid these dilemmas is simple: Don't claim proactive protection. Instead, focus on response times to new virus outbreaks. The best AVs will be able to approach zero hour response times, and no reputable researcher will release viruses to test this.

Most importantly, customers will be able to ask what, if any, steps they need to take to cover the lag.]]> http://www.dslreports.com/forum/remark,16741776 Sat, 19 Aug 2006 22:53:56 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16741762 SnowyOne :

said by eburger68

:

SnowyOne:
Sorry, but I still don't follow.

said by SnowyOne

:

Definitions based on "Heuristic" detection.

Errrmmm, but heuristic detections of one sort or another are a standard feature in most AVs today. How do these "close enough" detections differ from normal heuristic detections?

said by SnowyOne

:

Actual detections are based on actual behavior
"Heuristic detections are based on among other things anticipated behaviour.

As I said, heuristic detections are already standard fare in most AV detections these days.

said by SnowyOne

:

I encounter them frequently. Their usually called "False Positives"

False positives are not deliberately withheld from or placed into defintions, as you've speculated that "close enough" defintions would be. Moreover, false positives in AV detections can result both from standard signature detections and heurisitic detections (though heuristics are certainly more prone to fps).

said by SnowyOne

:

To manipulate results.

But if the company has the "close enough" detections today, it makes no sense for them to withhold them from regular release defs. Indeed, it makes more sense for them to include the "close enough" defs in public release defs today because there's no practical way (save special arrangement with the tester, which would call into question the validity of the test) for the AV company to guarantee that the testing entity uses the "special sauce" defs instead of public release defs. In such an event, there is no "manipulation" to speak of.

Which brings me to my last question, which you apparently have no answer for:

said by Eric L. Howes :
3) What makes you think that an AV company could ensure that its "custom" defs with the nifty "close-enough" special sauce would be picked up and used by the testing entities in lieu of defintions publicly available from the AV company?

You say all of that & then conveniently discover

said by eburger68

:

But, wait -- things might actually be a bit different than they originally looked...

So you figured out that you didn't understand what I was saying & decided to post to your misunderstanding anyway?
Am I supposed to respond to a former misunderstanding of yours?? Why else would you insist keeping it in your post?
To embarress? To intimidate? To silence?]]> http://www.dslreports.com/forum/remark,16741762 Sat, 19 Aug 2006 22:51:52 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16741697 SpannerITWks : eburger68

Who said Anything about " Minor variants " just now, not me boss ! I said possibly deadly too. And even if it was, as an extreme example, just 1 bit of code changed that turned it into a killer, it would still class as a minor variation. But nobody would be going round saying for eg, oh well it's only a MR so we don't worry about it etc etc.

And minor and/or major variants to today's threats ARE indeed tomorrows threats, just by definition alone ! In fact a new variation or brand new nasty could quite easily be picked by some coder today, and re-coded to make it into yet another variation, and release it the same day.

Pardon me, but you seem to contradict yourself here.

" even if they re-use some amount of code from exisiting threats, are by definition next generation type threats that are likely to be different to require a radical re-think of detection, blocking, and remediation strategies. "

Then this -

" There's nothing radical about new variants to existing threats. "

Well that's the whole point/s i'm making, " next generation type threats that are likely to be different to require a radical re-think of detection, blocking, and remediation strategies "

Next generation isn't always going to be in some far off time, giving vendors stacks of time to prepare for it at a leisurely pace, those NG's could arrive next week, in a few days, in fact it/they could be out there right now !

Re - " asked you earlier today "

Sorry i didn't realise you were asking me directly ! Anyways.

Don't need to mate, i use Free AV, and others are free to do what they like, including paying for zoo. But if i was shelling out $ for one, which i probably will sooner or later as i'm not against paying @ all, then Yes.

But here's da fing, if that situation arose in the future, cos it would be the future, and not 1823 or whatever, the media/forums etc are in full knowledge of the Bad 'ole days of the numbers game, not to mention bloat which gets talked about a lot more than ever before. And the vendors know it. So it wouldn't take too many of them to start playing those worn out games, before they got tripped up. So i imagine they would kick out the real crap, and just include the potentially clever stuff.

Spanner
--
I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks
/SpannerITWks]]> http://www.dslreports.com/forum/remark,16741697 Sat, 19 Aug 2006 22:40:47 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16741657 EGeezer : Links to the details will be helpful, as would links to the vendor's own tests that back up their claims of superiority in their ads and websites.
--
This space for rent]]> http://www.dslreports.com/forum/remark,16741657 Sat, 19 Aug 2006 22:29:14 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16741646 EGeezer :

said by bluezanetti

:

If that evaluation does possess a systematic bias, its purpose is lost. Sure 5,500 variants were created. Were they created from 600, 60, or 6 parent samples? That matters. Were the parents relatively new forms or rather aged examples? That matters. What ranges of behaviors were sampled in assembling the parent set? CR states that they were derived from 6 categories. It would be nice to know the broad details here. That matters. Details do matter.

It's good that these concerns have been raised by critics of the CU tests - I'd like to see such standards of disclosure of details applied to all tests - including those of CU, the industry media and the vendors' own tests. I think it would be at once revealing and informative to the prospective customer to have them published.
--
This space for rent]]> http://www.dslreports.com/forum/remark,16741646 Sat, 19 Aug 2006 22:26:01 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16741642 eburger68 : EGeezer:

You wrote:

said by EGeezer

:

I agree, we've not seen the details on the CU tests - However, the details on the "industry" tests that critics feel CU should provide is not being included in the "industry" reports either. So, I have no reason to place no more credibility on their tests than I would on CU's.

Hogwash. Look at any of the testing done by recognized, independent testing entities and you'll see that they provide a wealth of detail about their methodology and test bed. Moreover, these independent testing entities will provide the actual samples used in the test to other recognized experts as well as the AV companies whose products were tested.

CR has provided nothing even approaching what the standard AV testing entities do.

Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior]]> http://www.dslreports.com/forum/remark,16741642 Sat, 19 Aug 2006 22:25:43 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16741627 eburger68 : SnowyOne:
Sorry, but I still don't follow.

said by SnowyOne

:

Definitions based on "Heuristic" detection.

Errrmmm, but heuristic detections of one sort or another are a standard feature in most AVs today. How do these "close enough" detections differ from normal heuristic detections?

said by SnowyOne

:

Actual detections are based on actual behavior
"Heuristic detections are based on among other things anticipated behaviour.

As I said, heuristic detections are already standard fare in most AV detections these days.

said by SnowyOne

:

I encounter them frequently. Their usually called "False Positives"

said by SnowyOne

:

To manipulate results.

said by Eric L. Howes :
3) What makes you think that an AV company could ensure that its "custom" defs with the nifty "close-enough" special sauce would be picked up and used by the testing entities in lieu of defintions publicly available from the AV company?

But, wait -- things might actually be a bit different than they originally looked...

You wrote:

quote:
Maybe because I never said it? I've bold faced the part that should not have been attributed to me.

Here's what you originally wrote:

quote:
There's nothing in the world that could stop me from including "close enough" definitions in todays update of virus that I won't put ITW until 2 months down the road just to bolster my AV standings in tests using the "standard technique".

So, uh, am I to understand that you're postulating a scenario in which an AV company releases a virus of its own unqiue creation "into the wild?" If so, this is not a flaw unique to retrospective testing -- it would potentially affect ALL test scenarios that rely on viruses gathered from the wild -- with one set of exceptions: those tests based on official "Wild List" viruses, which are regularly dsistributed to major AV companies on a regular basis in order to ensure that all AV companies's products enjoy equal access to the the set of viruses that shall be considered legitimate targets for testing. Indeed, fraud and manipulation on the part of an unethical AV company is one of the outcomes that teh "Wild List" was set up to eliminate.

So, if a testing entity ensures that it tests against viruses certified by the "Wild List," there's very little chance for Nefarious AV to manipulate the test results. The retrospective test would theoretically be vulnerable to manipulation, but the best response to this scenario is to vigrously enforce the prohibition against lab created viruses that might actually be used by an AV company to perpetrate the fraud. And, it should go without saying, this kind of behavior on the part of an AV company would expose the company to ruin if the behavior were discovered -- its reputation would disappear overnight, customers would flee to other products, and the company itself might be prosecuted for felonious actions.

In other words, this scenario provides even more reasons why we don't want anyone in the AV industry to get into the business of creating lab viruses.

Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior]]> http://www.dslreports.com/forum/remark,16741627 Sat, 19 Aug 2006 22:21:35 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16741572 bluezanetti :

said by Steve :
The "standard technique" of retrospective testing is actually not so bad: turn off updates, and a month or two later find out which new viruses the A/V picks up. This is really a great test that satisfies most of what matters using undeniably real-world examples.

But this all smells like the security precautions taken in the "war on terror", which is fighting yesterday's threats. It's not bothering me at all that somebody tried a different approach.

Steve,

It may be a different approach, but rather than focusing on yesterday's threats as you put it, they focus on threats that are arguably non-existent and will hopefully never see the light of day. I don't know about you, but if I were to choose between the two approaches, I would certainly opt for the standard retrospective methodology since it has some objective bearing on reality. I realize it is not a panacea, but it does seem to provide a reasonable balance.

The test executed by CR has one goal only - to provide an unbiased evaluation of product performance. If that evaluation does possess a systematic bias, its purpose is lost. Sure 5,500 variants were created. Were they created from 600, 60, or 6 parent samples? That matters. Were the parents relatively new forms or rather aged examples? That matters. What ranges of behaviors were sampled in assembling the parent set? CR states that they were derived from 6 categories. It would be nice to know the broad details here. That matters. Details do matter. New approaches are fine, but I'd like a better sense of underlying details before embracing the product of that new approach. I assume that the online information (I do have suscriber access) is all the information provided, and it is rather lean on details although I'm sure it's appropriate to their main audience. Finally, there are also some key results that seem suspect (e.g. detection on KAV rated Very Good while F-Secure is a simple Good) on the face of it. That does make me wonder...

Blue]]> http://www.dslreports.com/forum/remark,16741572 Sat, 19 Aug 2006 22:10:02 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16741490 eburger68 : SpannerITWorks:

You wrote:

said by SpannerITWks

:

You do, why ?

Unless a Malware/Exploit coder is starting with a clean slate to write a brand new from the ground up nasty, guess where some the code comes from, to a greater or lesser degree ? Yep, previous stuff of course, whether it's theirs, and/or someone elses !

Minor variants to existing threats don't count as "tomorrow's threats" -- they're minor variants to today's threats. Tomorrow's threats, even if they re-use some amount of code from exisiting threats, are by definition next generation type threats that are likely to be different to require a radical re-think of detection, blocking, and remediation strategies. There's nothing radical about new variants to existing threats.

Btw, I'm still curious to get an answer to the questions I asked you earlier today after you heartily enthused over the prospect of the AV industry dipping into the lab virus cookie jar;

quote:
So, my question, though, remains: would you personally feel fine shelling out money each year for a subscription to an anti-virus product's definitions when those definitions were in part necessary in order to cover viruses that AV researchers, testers, and companies were themselves cooking up in the lab?

How do you think others would react to the same proposition -- that they had to pay for protection from viruses created by parts of the AV industry itself?

Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior]]> http://www.dslreports.com/forum/remark,16741490 Sat, 19 Aug 2006 21:53:46 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16741459 EGeezer : I agree, we've not seen the details on the CU tests - However, the details on the "industry" tests that critics feel CU should provide is not being included in the "industry" reports either. So, I have no reason to place no more credibility on their tests than I would on CU's.

At this point it seems to be "industry VS. CU", more particularly, "vendors who came out lower than they'd like VS CU". There's little advantage to the industry in the criticism, whose vendors claims are rife with "we're number one" marketing blurbs, with little or no independent technical detail provided in their company sites to back up their claims.
--
This space for rent]]> http://www.dslreports.com/forum/remark,16741459 Sat, 19 Aug 2006 21:46:51 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16741437 SnowyOne :

said by eburger68

:

1) What are "close enough" defintions,

Definitions based on "Heuristic" detection.

said by eburger68

:

and how do they differ from actual defintions?

Actual detections are based on actual behavior
"Heuristic detections are based on among other things anticipated behaviour.

said by eburger68

:

Have you ever encountered any "close enough" definitions from an AV vendor that you could describe for us?

I encounter them frequently. Their usually called "False Positives"

said by eburger68

:

Why do think an AV company would even consider withholding "close enough" (ITW) definitions until two months down the road?

To manipulate results.

EDIT:

said by eburger68

:

Why do think an AV company would even consider withholding "close enough" (ITW)definitions until two months down the road?

Maybe because I never said it? I've bold faced the part that should not have been attributed to me.]]> http://www.dslreports.com/forum/remark,16741437 Sat, 19 Aug 2006 21:44:06 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16741413 SpannerITWks : eburger68

Quote

" I fail to see how modifying existing threats from today to create new variants of those threats advances the cause of anticipating tomorrow's threats either. "

You do, why ?

Unless a Malware/Exploit coder is starting with a clean slate to write a brand new from the ground up nasty, guess where some the code comes from, to a greater or lesser degree ? Yep, previous stuff of course, whether it's theirs, and/or someone elses !

If the Anti brigade are hip to manipulating previous stuff, surely they Must, if the're top class anyway i would have thought, be able to patch stuff together with bits n pieces of old n new to create a NEW one, it ain't 100% original, but could still be deadly. Even if it wasn't the killer of all time, it's still NEW. Oh and size of company does NOT matter when it comes 2 brains, and being able to think both logically + laterally, ya know just like the best Malware coders are capable of.

So that's why i believe it does matter, and it's in their interests, and ultimately ALL of ours, that they should investigate these avenues. I bet it would be a stack of fun + a very challenging + rewarding too, in more ways than 1.

Spanner
--
I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks
/SpannerITWks]]> http://www.dslreports.com/forum/remark,16741413 Sat, 19 Aug 2006 21:39:47 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16741348 eburger68 : SnowyOne:

You wrote:

said by SnowyOne

:

I see a glaring problem with the "standard technique".
There's nothing in the world that could stop me from including "close enough" definitions in todays update of virus that I won't put ITW until 2 months down the road just to bolster my AV standings in tests using the "standard technique".
I'm not saying that activity does or doesn't happen. I'm saying it can be manipulated which doesn't say much for the staus quo.

I'm sorry, but I don't follow any of the logic here.

1) What are "close enough" defintions, and how do they differ from actual defintions? Have you ever encountered any "close enough" definitions from an AV vendor that you could describe for us?

2) Why do think an AV company would even consider withholding "close enough" (ITW) definitions until two months down the road?

3) What makes you think that an AV company could ensure that its "custom" defs with the nifty "close-enough" special sauce would be picked up and used by the testing entities in lieu of defintions publicly available from the AV company?

In short, this scenario relies on too many questionable assumptions to be considered a valid objection to retrospective testing.

Best,

Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior]]> http://www.dslreports.com/forum/remark,16741348 Sat, 19 Aug 2006 21:20:25 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16741300 SnowyOne :

said by Steve

:

The "standard technique" of retrospective testing is actually not so bad: turn off updates, and a month or two later find out which new viruses the A/V picks up. This is really a great test that satisfies most of what matters using undeniably real-world examples.

I'll take a little walk on the wild side here.
I see a glaring problem with the "standard technique".
There's nothing in the world that could stop me from including "close enough" definitions in todays update of virus that I won't put ITW until 2 months down the road just to bolster my AV standings in tests using the "standard technique".
I'm not saying that activity does or doesn't happen. I'm saying it can be manipulated which doesn't say much for the staus quo.]]> http://www.dslreports.com/forum/remark,16741300 Sat, 19 Aug 2006 21:09:54 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16741270 eburger68 : Hi All:

Rather than start a new post for each of the several interesting new responses that arrived since I last checked in, let me combine a responses of my own into one post.

funchords, you wrote:

quote:
Please provide any evidence that they made a choice. Otherwise, the best we can assume is that they took an independent approach without knowledge of the "vast corpus."

Fair enough, but I don't see that ignorance is any better in this situation than single-minded stubbornness or hubris. Even if they didn't know, they should have known -- and one of the most important reasons for this is provided by you in a later post...

funchords, you wrote:

quote:
It's an established principle in software testing that one continuously re-examines ones approach, re-evaluates the effectiveness of the testing, creatively explores for additional information not available from the everyday "comprehensive" suite, and adjusts to the current and future environment.

This is a fine principle in the abstract. Problem is, you've already admitted that CR was, at best, ignorant of the prior history and established methodologies of AV testing. One can't re-examine, re-evaluate and creatively explore if one isn't starting from an historically informed base of knowledge. IF CR simply struck out on their own without any knowledge of previously established methodologies and lessons of AV testing, then there's little to be gained from this exercise unless you want the AV industry itself to take up the practice of lab virus creation. For the potential consequences of that, see my earlier posts.

funchords, you wrote:

quote:
The level of protesting seems too shrill. By getting high-and-mighty, the industry has lowered itself a bit in my eyes.

So send the head of Virus Bulletin an etiquette book.

However large a presence the AV industry may have in specialist forums like these, the AV industry's presence among the general population (beyond that flashing tray icon on people's computers) is next to nothing compared to Consumer Reports, which is currently blasting these test results into the homes of millions of Americans, the vast majority of whom will, even after this noisy controversy, have little idea that a group of veteran AV experts thinks these tests have signficant problems. All the general public will see is "newsy" little items like this:

»news10now.com/content/features/t···SecID=97

To get its message out in the face of that kind of massive publicity, the AV industry is going to have to shout to be heard by the mainstream news media.

Steve, you wrote:

quote:
This is one of those sentiments that sounds good, and ought to be applied generally, but is not the kind of universal prohibition which admits no exceptions in any case; I don't see the need for the hysteria over this.

C'mon, Steve. CR's critics have offered plenty of sound reasons for rejecting CR's course of action -- even you've allowed as much. And although you might be less than impressed by the ethical implications that CR's critic see here, those ethical objections aren't "hysteria" -- they're grounded in the experience and knowledge of the critics.

Moreover, an admonition to allow for exceptions is a nice bit of advice in the abstract, but the burden for establishing the justification for a particular exception in this case falls on those who would advocate for the exception. So far I haven't seen anything that comes close to establishing the need for an exception in this case. I've heard plenty of noise about the "high-handed," "whiney" behavior of the AV industry, plenty of worshipful paeans to the inerrancy of CR, but next to nothing in the way of response to the specific points made by Joe Wells (in several quoted/cited texts), IBK, or bluezanetti as to why lab viruses are a methodologically unsound and even counterproductive means of testing.

I'm always willing to entertain exceptions to the rule, but ultimately the burden is on those advocating the exception.

Steve, you wrote:

quote:
Their badness comes from the harm they do to others, not to some inherent badness of the bits themselves.

(Warning: slightly OT) - This is one of the more tiresome shibboleths of tech-geek culture -- the inherent "neutrality" of technology -- one that really should have been retired years ago, as it offers almost zero insight into the problems and challenges that software programs and other technologies pose to real human environments, which is where we always encounter the bits. The "inherent" quality (or non-quality) of the "bits themselves" (whatever that might be) is of zero interest or relevance because we never encounter or deal with only "the bits themselves." One might as well say that the badness that criminals do comes from the harm they do others, not to some inherent badness of the cells themselves that make up the criminal. Technology is never "neutral." It may have multiple, complex consequences when introduced into a particular environment, but it is never neutral because technologies always lend themselves to particular uses, not all uses, and affect the surrounding environment in particular ways, not all possible ways.

SpannerITWks, you wrote:

quote:
1 - " It is claimed that created viruses were the kind you’d most likely encounter in real life which is, of course, something the testers cannot know. "

( And something Igor Muttik or anybody outside of the inner sanctum can't know, as they don't know if they don't have access to them )

And that is not an adequate response, because it is not the burden of those of us outside of the "inner sanctum" to establish that the viruses aren't the kind that users would most likely encounter in real life. Far from it. It is the burden of those inside the inner sanctum (at present CR itself and the few advisors they hired) -- those who made the claim in the first place, and who have thus far offered nothing in the way of evidence to back up that claim.

EGeezer, you wrote:

quote:
Since no one has specifically reviewed the variants created by the CU testers, we really don't know whether or not the variants would be typical of the dozens that are created daily by the "production" malware coders(Per David Emm of Kapersky, quoted in one of the SANS linked articles, Kapersky adds over 200 signatures a day).

Yes, exactly...

quote:
However, I'd guess they have sufficient history and have retained the expertise to extrapolate and create reasonable variations in a well secured environment. Until credible experts come up with analyses of the CU variants that discredit CU's tests, I'll give CU the benefit of the doubt based on their past history of providing accurate testing and successful defence of challenges.

Errrmmm, but this doesn't follow, because the burden of proof is entirely backwards here. Based on your trust (faith?) in CR, you've essentially thrust the burden of proof on the critics of the test, not the authors of the test, which is where the burden of proof properly lies. Still worse, you've put the critics into a potentially impossible bind -- if CR refuses to release the necessary data to allow critics to prod and poke, then those critics will have been denied the ability to establish their case. That's a rather convenient "heads-I-win, tails-you-lose" proposition for CR to be offering its critics.

No, the burden is on CR to establish the validity and meaningfulness of its testing, not on its critics to disprove it. If CR fails to do the minimal amount of work to establish the validity and meaningfulness of its testing, then the testing is invalid and has no meaning, and your faith or trust in CR should play no role in your evaluation of that testing.

Steve, you wrote:

quote:
But this all smells like the security precautions taken in the "war on terror", which is fighting yesterday's threats. It's not bothering me at all that somebody tried a different approach.

I don't think the AV industry is claiming that retrospective testing is sufficient to the job of anticipating tomorrow's threats and planning for them. Most respectable AV companies that I know of (Kaspersky, for example) have entire teams of researchers dedicated to anticipating what tomorrow's threats might look like. However mistaken they might ultimately prove to be, I fail to see how modifying existing threats from today to create new variants of those threats advances the cause of anticipating tomorrow's threats either.

Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior]]> http://www.dslreports.com/forum/remark,16741270 Sat, 19 Aug 2006 21:02:17 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16741121 Steve : The more I read about this, the less I think about these whiney A/V vendors.

said by McAfee/Avert blog :

It is claimed that created viruses were “the kind you’d most likely encounter in real life” which is, of course, something the testers cannot know.

Well how about that: calling CR to tasks for not having a crystal ball.

The "standard technique" of retrospective testing is actually not so bad: turn off updates, and a month or two later find out which new viruses the A/V picks up. This is really a great test that satisfies most of what matters using undeniably real-world examples.

But this all smells like the security precautions taken in the "war on terror", which is fighting yesterday's threats. It's not bothering me at all that somebody tried a different approach.

Steve
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site]]> http://www.dslreports.com/forum/remark,16741121 Sat, 19 Aug 2006 20:25:16 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16740778 t2contra : Can someone post the test results?]]> http://www.dslreports.com/forum/remark,16740778 Sat, 19 Aug 2006 19:11:31 EDT SANS NewsBite article http://www.dslreports.com/forum/remark,16740698 EGeezer : Since no one has specifically reviewed the variants created by the CU testers, we really don't know whether or not the variants would be typical of the dozens that are created daily by the "production" malware coders(Per David Emm of Kapersky, quoted in one of the SANS linked articles, Kapersky adds over 200 signatures a day).

However, I'd guess they have sufficient history and have retained the expertise to extrapolate and create reasonable variations in a well secured environment. Until credible experts come up with analyses of the CU variants that discredit CU's tests, I'll give CU the benefit of the doubt based on their past history of providing accurate testing and successful defence of challenges.

That being said, Here's a note from SANS;

said by SANS Newsletter and editorial :

--Consumer Reports Creates 5,500 Viruses For Tests
(16 August 2006)
Consumer Reports is under fire from the anti-virus community for sponsoring the creation of 5,500 new viruses to test anti-virus products. Zone Alarm Internet Security Suite scored high in the test for both virus and spyware. Spybot Search and Destroy scored well for spyware.

»www.computerworld.com/action/art···_topic17

»cbs4boston.com/consumer/local_st···410.html

Special Tip: A great discussion on Microsoft Office security and vulnerabilities has been posted on SecurityFocus:
»www.securityfocus.com/infocus/1874

[Editor's Note (Paller): This controversy is especially problematic for the leading AV companies because they have traditionally not done well in finding and blocking new viruses quickly. But for goodness sakes, if they don't do well at finding and blocking new viruses, why ae we buying them? They should stop complaining and instead thank Jeff Fox and the editors at Consumer Reports for helping to do important product improvement research for them.

--
This space for rent]]> http://www.dslreports.com/forum/remark,16740698 Sat, 19 Aug 2006 18:55:52 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16740610 SpannerITWks : IBK

Re the Mcafee blog

1 - " It is claimed that created viruses were the kind you’d most likely encounter in real life which is, of course, something the testers cannot know. "

( And something Igor Muttik or anybody outside of the inner sanctum can't know, as they don't know if they don't have access to them )

3 - 4 - ( Already covered those )

Re Conclusion

" there was no need to create that virus variants, as the test based on these self-made variants do not show/tell to the user how good AVs are in detecting new viruses. It only tells about how much of the self-made files created for testing - and that you will never encounter - were detected "

( Why doesn't it help users to determine if an AV can detect, or not new Malware. As i've already said, if it's new then it IS new, and if it could do damage, no matter how small, it's still unwanted and would need sorting. How can anyone say that ALL 5500 were crap, they might be, but we don't know do we, yet ! How can it be stated 100% that nobody will EVER encounter Anything similar.

Why do people deferentiate between those 5500 in a closed lab, and scriddies or worse coding something equally, let's say crap but still capable of damage, or something not crap but lethal ! Why does it matter where they were created, and by whom. If it's new and can do damage then it's fair game for testing AV's detection capabilities.

Any chance of putting a bit more white inbetween the black in future, thanx. )

Spanner
--
I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks
/SpannerITWks]]> http://www.dslreports.com/forum/remark,16740610 Sat, 19 Aug 2006 18:38:10 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16740345 IBK : Hi,
the following is only my opinion as, nothing more:
I think the main point on why CR is currently in the spotlight due what they did is not due the high risk their samples could pose to real world. The main points (I refer now to the points of mcafee blog because i think they were the first to note it) are that 1. the variants they created are NOT variants that you will encounter in real life (its long ago since a saw a scriptkiddie variant of e.g. loveletter and not what goes in daily [much more new malware appears, not so silly varaints], 2. true, writing viruses is generally considered not a good idea (but the AVIEN letter was an example, about a topic were an university wanted to teach how to write viruses to its students to teach how to protect against them - all students failed in real world about this, as no one is working in any av related job), 3. testing methods to measure how good or bad av software is at detecting new malware are discussed since long time and since some years (6?) the retrospective method is considered to give the best real-world results (and that's true, if the test is done accuratly and a bit perfectionated to avoid some influences). It is known that AV-Test.org (Andreas Marx) does retrospective tests and publishes the results in many magazines. And it should be also known that AV-Comparatives does such testing publicly available (still for free - that's more user-friendly). So I can only think that they wanted to make something spectacular, but failed in doing some research about the topic before they acted (well, it is probably also the fault of the peoples they engaged to do the test. They are most probably very good about other security tests in enviroments they provide, but probably not very informed about antivirus testing). The point 4 in the mcafee blog is (for those that did not noticied it) a sarcastic phrase (see ;)).
Conclusion: there was no need to create that virus variants, as the test based on these self-made variants do not show/tell to the user how good AVs are in detecting new viruses. It only tells about how much of the self-made files created for testing - and that you will never encounter - were detected, making all the test senseless and not useful for anyone. CR will not write that in their article, but even if they would state that, most readers would anyway get to their own conclusion and believe in the printed scores.
AV vendors (also those that scored top) are imo very sad/upset that magazines still make home-made invalid tests and deliver to users wrong information (what happenend since long time and still happens in some magazines) instead of e.g. asking independent organizations like av-test.org, virusbtn, icsa, wcl etc. in helping doing the tests (or performing the tests for them). I do not list av-comparatives because as I publish the results up-to-date for free on the website to the users, I do not think that anyone would want to wait for several months for seeing it published in some magazine when it will be already outdated (usually [but not always] most tests in magazines are at least already some months old).]]> http://www.dslreports.com/forum/remark,16740345 Sat, 19 Aug 2006 17:44:35 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16740177 SpannerITWks : No Stevie baby, i was responding to alexeck " Just don't create viruses "

I assumed he did actually mean ALL malware though, as i mentioned earlier on in the thread.

Your right, hysteria won't help anybody, especially the users !

Spanner
--
I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks
/SpannerITWks]]> http://www.dslreports.com/forum/remark,16740177 Sat, 19 Aug 2006 17:02:52 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16740155 Steve :

said by SpannerITWks

:

How about Trojans + Rootkits etc then, is that OK lol ?

I assume you were replying to me in spite of your Topic Reply.

I don't have any problem creating any software on a test basis for legitimate research and testing purposes as long as one takes precautions that they do not leak. It's harder to do this than it looks, but it's not beyond the ability of mankind to get this right.

Their badness comes from the harm they do to others, not to some inherent badness of the bits themselves.

Steve
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site]]> http://www.dslreports.com/forum/remark,16740155 Sat, 19 Aug 2006 16:57:30 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16740139 SpannerITWks : How about Trojans + Rootkits etc then, is that OK lol ?

Spanner]]> http://www.dslreports.com/forum/remark,16740139 Sat, 19 Aug 2006 16:53:53 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16740121 Steve :

said by alexeck

:

Sure, that's fine. Just don't create viruses.

This is one of those sentiments that sounds good, and ought to be applied generally, but is not the kind of universal prohibition which admits no exceptions in any case; I don't see the need for the hysteria over this.

Steve
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site]]> http://www.dslreports.com/forum/remark,16740121 Sat, 19 Aug 2006 16:50:30 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16740089 alexeck :

said by funchords :
While agreed-on standards for testing are useful, they are also the minimum and they quickly decrease in value over time. You better serve the customer by doing more than just the (minimum) standard testing.

Sure, that's fine. Just don't create viruses. ]]> http://www.dslreports.com/forum/remark,16740089 Sat, 19 Aug 2006 16:45:11 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16740069 alexeck :

said by funchords :
Please provide any evidence that they made a choice. Otherwise, the best we can assume is that they took an independent approach without knowledge of the "vast corpus."

Well, they could start with Virus Bulletin, which has been doing this for many years. »www.virusbtn.com/]]> http://www.dslreports.com/forum/remark,16740069 Sat, 19 Aug 2006 16:41:28 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16739994 funchords :

said by Steve

:

It may well be that CR committed the crime in question, but the A/V industry is doing a terrible job in the witness stand.

Agreed. I have no beef with the AV industry. So far, McAfee (which ranks about center in the AV Comparitors list) hasn't failed me -- and I'm a rough customer. To me, anyway, mediocracy seems to be pretty damn good. That speaks well of the Industry.

"The industry" (whoever they are) has judged that Consumer Reports made a mistake. Fine. They may even choose to ignore CRs methods and findings. Fine. They may even write a letter to the editor explaining why. Fine. I think that all of those conclusions and actions are rational.

The level of protesting seems too shrill. By getting high-and-mighty, the industry has lowered itself a bit in my eyes.
--
Robb Topolski -= funchords.com =- Hillsboro, Oregon USA
~ Keeper of the D-Link FAQ ~ Did you Search? ~ More features, Free! Join BBR! ~]]> http://www.dslreports.com/forum/remark,16739994 Sat, 19 Aug 2006 16:27:01 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16739922 funchords :
said by alexeck :

We need standardized testing for all types of security products. This debate should be done in a reasoned, scientific fashion, with broad representation in the community and industry to come out with a clear, comprehensive method of testing. That is the only real way to serve the consumer.
No, it's not.

As a professional software tester (not of security products), allow me to both agree and disagree.

While agreed-on standards for testing are useful, they are also the minimum and they quickly decrease in value over time. You better serve the customer by doing more than just the (minimum) standard testing.

It's an established principle in software testing that one continuously re-examines ones approach, re-evaluates the effectiveness of the testing, creatively explores for additional information not available from the everyday "comprehensive" suite, and adjusts to the current and future environment.
--
Robb Topolski -= funchords.com =- Hillsboro, Oregon USA
~ Keeper of the D-Link FAQ ~ Did you Search? ~ More features, Free! Join BBR! ~]]> http://www.dslreports.com/forum/remark,16739922 Sat, 19 Aug 2006 16:11:18 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16739850 funchords :
said by alexeck :

CR chose to ignore a vast corpus of research, debate and analysis by the academic and security research community.
Hey Alex!

Please provide any evidence that they made a choice. Otherwise, the best we can assume is that they took an independent approach without knowledge of the "vast corpus."

--Robb (formerly of Q.O.S.) :) --wonk!--
--
Robb Topolski -= funchords.com =- Hillsboro, Oregon USA
~ Keeper of the D-Link FAQ ~ Did you Search? ~ More features, Free! Join BBR! ~]]> http://www.dslreports.com/forum/remark,16739850 Sat, 19 Aug 2006 15:57:44 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16739566 bluezanetti :
said by SpannerITWks :
And it's just as easily debated that it does ! Cos you n me both + everybody else don't know do we, it's just speculation after all.
Quite true. Of course, had CR followed the protocol noted above, debate on this point would be moot, albeit replaced by other points of possible contention.

My overriding point is that the route pursued by CR is one that could inject unintentional bias into the test. Who knows if measures were taken to minimize this eventuality, I certainly have no insight into that point.

Blue]]> http://www.dslreports.com/forum/remark,16739566 Sat, 19 Aug 2006 14:58:37 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16739527 AB :
said by SpannerITWks :

. . Anybody can test whatever they like, ther's no law against it, and as long as the tests are competent and the results are fair, and it's beneficial in some way/s to the users, so what !

. . just because we havn't been given the title " experts " or call ourselves that ?

Spanner
Hey, I think I'm starting to see the light here!
All those pimply-faced 17-year-old Russian scriptkiddies are just 'testers', testing what they like! They certainly don't have the title "experts", do they? And it's ultimately beneficial to the community because other so-called 'experts' get to play around with their handiwork to find out how to stop it. Yes! We've been dogging these people, when in reality, we owe them our deepest gratitude!
Please, allow me to be the first--
Thank you, pimply-faced Russian scriptkiddies, for helping to make the computing world a better, safer place!]]> http://www.dslreports.com/forum/remark,16739527 Sat, 19 Aug 2006 14:51:34 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16739435 SpannerITWks : Quote bluezanetti -

" However, it is just as easily argued that no matter how controlled their testbed, there's no assurance the test sampling bears any resemblence to emerging malware threats at play today due to the very dynamic nature of the challenge. "

And it's just as easily debated that it does ! Cos you n me both + everybody else don't know do we, it's just speculation after all. If out of 5500 brand new nasties they havn't written some super duper stuff that really challenges AV's, then yes it would be a bit limp, but it's to be hoped they did. We might find out sooner rather than later as the " noise " increases from various sectors. They could open up their secret Treasure Trove to Trusted peeps for evaluation, that would sort it one way or the other, or maybe even inbetween !

I suggest that a group of Interested parties could be invited to bring their testing laptops to a pre determined SECURE location, and under the watchfull eyes of a number of agreed by all parties peeps, conduct their own tests ALL at once ! Then publish the results either as a joint release, and/or individually for all to see. If people Really want it to happen it can and will, so start making connections and make it happen, then they will know and so will we.

-

As a general observation, comments made towards somebody/thing etc like this for eg " Consumer Reports, better known for reviewing cars, lawn-mowers and appliances " only seem to be posted to demean them in some way/s. Just because they test those things, they also test a range of other things too, and why shouldn't they be ( allowed ) to if they want to. Anybody can test whatever they like, there's no law against it, and as long as the tests are competent and the results are fair, and it's beneficial in some way/s to the users, so what !

For one eg -

I've thrown just every FW test there is going at my FW + Apps, and published the results for all to see, more than once. People were and are free to challenge them and my methods etc, and ask questions etc, which they did, and i was happy to respond. Happy in more ways than 1 too, as i pass 99% of them on my PC. Now am i or others to be disbelieved over tests they do such as those, just because we havn't been given the title " experts " or call ourselves that ?

Spanner

edit typo Only
--
I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks
/SpannerITWks]]> http://www.dslreports.com/forum/remark,16739435 Sat, 19 Aug 2006 14:32:24 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16739171 bluezanetti :
said by alexeck :
The CORRECT way to test heuristics is extremely simple: Turn off definitions for all the products being tested, and test against new virus strains after a few weeks or months. That's the only honest, correct approach, as it a) tests against the real-world, b) doesn't turn you into a virus creator/distributor and c) actually gets you the right results.
Absolutely.

I can see that an organization such as CR could feel that generating a synthetic testbed of samples would provide a more expedient and controlled testing platform than would an active harvest of malware over a defined period (e.g. the basic protocol followed at av-comparatives.org). Many of CR's evaluations follow this scheme of devising a synthetic challenge and performing classical challenge/response testing. However, it is just as easily argued that no matter how controlled their testbed, there's no assurance the test sampling bears any resemblence to emerging malware threats at play today due to the very dynamic nature of the challenge. The validity of this portion of the test results reside in appropriate choices being made in the creation of the synthetic testbed. Poor choices there will completely skew the final results. In a sense, the test results could range anywhere from an accurate reflection to a completely inverted ranking of current performance and there's really no way to get an independent sense of where things lay.

Even the av-comparatives.org test results have to be closely inspected owing to the noise associated with small sample testbed. In the case of CR, it is noise with an unknown level of unintended sampling bias.

Blue]]> http://www.dslreports.com/forum/remark,16739171 Sat, 19 Aug 2006 13:32:03 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16739135 Steve :
said by eburger68 :

Even if the AV vendors themselves somehow managed to refrain, we could still well wind up in a situation where AV companies were forced to contemplate whether to start adding lab viruses to their defintions. And who would benefit? Surely not ordindary users and consumers.
This is a fair point, but it's not the argument that was made in the early parts of this thread. It was originally that CR's creation of these was bad in and of itself, but now it's because it might lead to the industry into screwing the consumer. Those aren't the same things!

CR generally thinks outside the box without regard for what the industry being reviewed thinks, and I believe that's good for the consumer. The louder the industry wails, the more I think they may be onto something.

Should we let the lawn-mower industry define the tests for what makes a good lawn mower? How about car companies? etc.

It may well be that CR committed the crime in question, but the A/V industry is doing a terrible job in the witness stand.

Look at all these signatures! Look at how much we're gnashing our teeth! I'm not going to believe CR when they rate a gas grill!

I'm an educated, technical consumer with a reasonable nose for BS, and this all comes off as incredibly disingenuous to me. But I could be wrong.

Steve
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site]]> http://www.dslreports.com/forum/remark,16739135 Sat, 19 Aug 2006 13:23:41 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16739082 EGeezer : Great discussion here -

From my own perspective, not knowing how valid the CU "variations" are, I couldn't speculate on the validity of the tests. However, I feel that testing against variations provides value by providing a fresh angle, and Consumer reports has done pretty well in the past in providing consumer level information on product quality. It seems that this test, if the variations were reasonably close to what malware writers do to circumvent detection, would lean more favorably toward those who have better behavior-based or heuristic engines, and at a disadvantage to those relying more on signature tables. The difficulty I see is if the tests themselves don't employ multiple techniques, but are positioned as be-all end-all tests.

In the end, CU is little different from other AV testers in that few have the same results in ranking. At best, they generally correspond to each other. The use different reference samples, test methods and tools, so their results will be different.

As for judging validity of the tests based on sources, I'd consider the reputation of the source and how successful they've been as well as their methods, credentials and affiliations. One should naturally consider whether the tester has an ax to grind or a stake in the outcomes, then research against other sources.

As for the vendors. the FUD and hysteria built into the detection messages is problematic. They appear to have been written by marketing types, not by technical writers, and with the goal of promoting the product rather than providing an accurately positioned description of the object detected.

I have no issues with a product detecting cookies on a scan, but having it do so in the manner of the screaming carpet salesmen on late night TV is in insult to the educated and a disservice to the technically uninformed. Vendors need to more accurately position the characteristics of privacy and security-related objects they detect, isolate or remove.
--
This space for rent]]> http://www.dslreports.com/forum/remark,16739082 Sat, 19 Aug 2006 13:09:43 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16739066 eburger68 : Steve:

You wrote:

said by Steve :

But since Consumer Reports is not in the A/V industry, they don't have any of those incentives (they're not selling an A/V product), so these reasonable proscriptions on industry behavior do not apply here.
I'm afraid I have to disagree once again. CR is widely respected and influential -- that we've seen even from some of the posts in this thread. If this testing turns out not to be a one-off situation with CR -- that is, if CR were to start routinely using lab viruses in their widely followed testing -- then pressure on the AV industry itself (including independent researchers, testing bodies, consultants, and the AV vendors themselves) to do the same would inevitably increase. It would likely start with other research entities, but it would likely spread to other parts of the AV industry.

Even if the AV vendors themselves somehow managed to refrain, we could still well wind up in a situation where AV companies were forced to contemplate whether to start adding lab viruses to their defintions. And who would benefit? Surely not ordindary users and consumers.

No, CR is not an island unto itself. Many here have championed CR for having the wherewithal to force industries to think and behave differently. This power can be a benefit in some circumstances. It can also pose dangers if that influence unintentionally forces an industry down a path it has no business going down. One can't celebrate the influence of CR on the one hand and not contemplate the potential consequences of its actions on the other.

So, is the AV industry "circling the wagons." Perhaps only to protect itself from a potential trend that it long ago recognized as dangerous.

Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior]]> http://www.dslreports.com/forum/remark,16739066 Sat, 19 Aug 2006 13:05:49 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16739033 alexeck : Folks,

I see arguments supporting CR and against CR.

Here's the simple truth: CR chose to ignore a vast corpus of research, debate and analysis by the academic and security research community. They decided to go their own way and have severely undermined their credibility by making a major error, and possibly others.

It's an established principle in security research that you NEVER create your own antivirus strains for testing purposes. There are a number of reasons which I discuss in my most recent blog posting »tinyurl.com/msclw .

The CORRECT way to test heuristics is extremely simple: Turn off definitions for all the products being tested, and test against new virus strains after a few weeks or months. That's the only honest, correct approach, as it a) tests against the real-world, b) doesn't turn you into a virus creator/distributor and c) actually gets you the right results.

Why CR couldn't simply follow this time-honored approach is a bit confusing.

Arguing that the AV community is biased in this regard is patently false reasoning. The arguments against CR are across the spectrum, from the pure research side to the antivirus community.

If CR had simply followed standard testing methods, all would be fine and no one would care. It would actually be a service to the community.

But the problem is a bigger one: We need standardized testing for all types of security products. This debate should be done in a reasoned, scientific fashion, with broad representation in the community and industry to come out with a clear, comprehensive method of testing. That is the only real way to serve the consumer.

Alex]]> http://www.dslreports.com/forum/remark,16739033 Sat, 19 Aug 2006 12:57:58 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16738999 AB :
said by Steve :

. . I certainly agree that it's dangerous, may well agree that it's unnecessary, (which would follow that there are more effective methods). . (sic)

Even if they somehow got in the wild accidently, that's about "negligence" . . .

Steve
'Nuff said, perhaps??
]]> http://www.dslreports.com/forum/remark,16738999 Sat, 19 Aug 2006 12:51:34 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16738987 Steve :
said by eburger68 :

The concerns that I've expressed -- that, in a worst case scenario, the industry and its customers could be drawn into enervating cycle lab virus creation and virus definition building in response to customer demands -- is not an idle one. We've seen these kinds of cycles before.
Ah, so now the fog lifts.

This is not so much an egalitarian "do not hurt others" concern, but a worry that the industry will be unable to restrain itself; that's a different concern that does not speak in any way to the ethical behavior of Consumer Reports.

It would indeed be unethical for an A/V company to create viruses ("for testing") and then include them in a product with a claim that they protected against more stuff than the other guys (who don't have the synthetic tests). This is hyping against threats that do not really exist.

So the objection is not about "creating test viruses" but "creating test viruses and using them for marketing": the latter creates a whole cycle of bad incentives at the expense of the consumer.

That is unethical, and I'm pretty sure that there's essentially 100% agreement on that point.

But since Consumer Reports is not in the A/V industry, they don't have any of those incentives (they're not selling an A/V product), so these reasonable proscriptions on industry behavior do not apply here.

The more I look at this, the more I believe Consumer Reports was not unethical in any way, even remotely.

So putting aside the ethical issue, we're left with something that's somewhat easier for us to talk about: the technical merit of their testing methodology.

But that doesn't make it completely easy: the onlooker must be on the lookout for a circle-the-wagons reaction by the industry — this happens all the time — and I'd be surprised if there were none of that here.

I, like others, am content to evaluate both the evidence and the warrants, sniffing out the good and spurious claims.

But the more I see about "ethics", the more I think it's a circle-the-wagons reaction, and to accept A/V claims with more and more grains of salt.

Steve
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site]]> http://www.dslreports.com/forum/remark,16738987 Sat, 19 Aug 2006 12:48:50 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16738963 eburger68 : Steve:

You wrote:

said by Steve :

I believe that these practices are a substantial disservice to the customer, and border on unethical.
I couldn't agree more. The trick, of course, is how do we compel an industry addicted in many ways to the "numbers game" to come together and give up this marketing "crack"?

I should note that the difficulty only increases when one realizes that potential customers are being exposed on a daily basis to insanely unethical scaremongering pitches for "rogue" anti-spyware products and the like that are being marketed by malware pushers themselves -- borderline criminal elements who aren't likely to see any benefit to sitting down at a roundtable on industry ethics and best practices.

Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior]]> http://www.dslreports.com/forum/remark,16738963 Sat, 19 Aug 2006 12:43:27 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16738948 funchords :
said by eburger68 :

First, give part of Joe Wells' paper here a read:

"Lies, Damn Lies, and Marketing Perfidious Priorities"
»vx.netlux.org/lib/ajw01.html#p3

Joe rehearses one these cycles from the early years of the AV industry, when it was sucked into a competitive arms race over the number of "all known viruses" and the comparative detection rates of AV products. Bad research and analysis feeds opportunistic, competitive marketing, which feeds user fears and customer demands, which in turn feeds product testing and research, which feeds...

Yep, similar to Processors and MIPS, wireless network theoretical datarates, pharmaceuticals and "restless-knee syndrome," ... the list goes on forever.
--
Robb Topolski -= funchords.com =- Hillsboro, Oregon USA
~ Keeper of the D-Link FAQ ~ Did you Search? ~ More features, Free! Join BBR! ~]]> http://www.dslreports.com/forum/remark,16738948 Sat, 19 Aug 2006 12:38:25 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16738937 eburger68 : GeekNJ:

You wrote:

said by GeekNJ :

And on a bit of a related thought, I've personally always felt that the A/V industry itself was possibly responsible for the scare and even creation of viruses in order to pump up the "We protect you from xxx,000 nasties". Of course, how many of those nasties have never been "in the wild"? :uhh:
Yes, you and a number of other folks, as I noted in an earlier post. If this truly is a concern of yours, then the very last thing you want to encourage is the creation and use of lab viruses by anyone in the industry or even connected with the industry -- and that includes CR, because if a widely respected and influential testing entity like CR begins routinely creating and using lab viruses, then the pressure will only increase on others in the industry to start doing the same. At some point, AV companies could very well be compelled by customers or circumstances to start loading up their definitions (and selling subscriptions to them) with these lab viruses.

And who would benefit from such an eventuality? The only possible beneficiaries that I see are the sales departments of AV companies.

This is one of the quagmires that the "Wild List" was created to forestall -- to compel the industry to focus on, research, test against, and target actual viruses that posed real threats to users "in th wild."

Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior]]> http://www.dslreports.com/forum/remark,16738937 Sat, 19 Aug 2006 12:36:49 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16738898 Steve :
said by eburger68 :

If you're game, I would like to get your thoughts on the most recent issues we've been mulling over here -- namely, what the potential effect on the AV industry and its products/definitions would be if the use of lab-created viruses were to become commonplace among testing entities and researchers, esp. if it were concluded that a "hole in the industry" had been discovered.
Now this does get into an interesting area, one which would get into arms races that don't serve the consumer. Generally: one should only protect against threats, and test viruses in a lab are not threats (EICAR is not a threat, but it's an exception).

Many of the players in the A/V industry are also involved the spyware and personal firewall industries, and I think this gives us a peek into the future as to what to expect.

Personal Firewall vendors are in this ever-growing competition to detect more stuff, which is why we have these bogus "YOU ARE BEING ATTACKED" popups. OMG! It's a PING!

I staff a couple of abuse desks, and we get reports all the time about benign behavior, but because of self-serving WE CATCH MORE STUFF! WE PROTECT YOU! crap from the firewall industry, the consumer is frightend and at least two people waste a lot of time.

Adware vendors likewise are in this "we detect more stuff", which leads to outlandish claims about "5,000 objects detected". Yah, but most of them are cookies, and the rest are mostly benign. We see these threads here all the time.

I believe that these practices are a substantial disservice to the customer, and border on unethical.

Steve
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site]]> http://www.dslreports.com/forum/remark,16738898 Sat, 19 Aug 2006 12:29:02 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16738887 eburger68 : Hi All:

A quick followup to my last few posts regarding the potential effects of the widespread use of lab viruses on the AV industry. The concerns that I've expressed -- that, in a worst case scenario, the industry and its customers could be drawn into enervating cycle lab virus creation and virus definition building in response to customer demands -- is not an idle one. We've seen these kinds of cycles before.

First, give part of Joe Wells' paper here a read:

"Lies, Damn Lies, and Marketing Perfidious Priorities"
»vx.netlux.org/lib/ajw01.html#p3

Joe rehearses one these cycles from the early years of the AV industry, when it was sucked into a competitive arms race over the number of "all known viruses" and the comparative detection rates of AV products. Bad research and analysis feeds opportunistic, competitive marketing, which feeds user fears and customer demands, which in turn feeds product testing and research, which feeds...

You get the picture.

We've even seen a similar phenomenon in the anti-spyware industry with respect to cookie detection. Having talked to a large number of folks from various anti-spyware companies, I can tell you that none of them (at least that I know of) regards cookies as anywhere near the same kind of threat as executable adware, spyware, or malware. And most that I've talked to have expressed a desire to do something different with the cookie detection in their products. Some would like to drop it altogether. Others would like to handle it differently, so that cookies weren't presented alongside executable malware in a manner that suggested that the two were roughly similar types of threats.

So why don't things change within the anti-spyware industry? Because everyone's afraid of the consequences of being the first to act (beyond Microsoft, which dropped cookie detection from the GIANT product that it acquired). Any anti-spyware company out there can tell you about the angry calls and emails they get from customers that Product A failed to detect a few cookies that Product B detected. In short, fearful customers are demanding cookie protection and frequently see no difference between cookies, viruses, spyware, and adware. And the anti-spyware companies, much as they might gnash their teeth over the detection of cookies, continue to provide it (and even, in some cases, hype it) out of fear that their product will take a hit in sales and reputation should they be perceived as "soft on cookies."

So, my concerns do have a basis in actual situations that we've encountered before.

Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior]]> http://www.dslreports.com/forum/remark,16738887 Sat, 19 Aug 2006 12:26:48 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16738886 funchords :
said by GeekNJ :

And on a bit of a related thought, I've personally always felt that the A/V industry itself was possibly responsible for the scare and even creation of viruses in order to pump up the "We protect you from xxx,000 nasties". Of course, how many of those nasties have never been "in the wild"? :uhh:

We've all wondered about that at some time or another. We've all heard the true stories of firemen that try to boost their careers by starting fires in order to be the hero that saves a life or building.

But, I think we're pretty safe from that possibility, because:

The industry is both old enough and large enough that, if this were happening, a current or ex-employee whistleblower would have appeared by now.

The number of competitors is large enough to identify one competitor that is constantly adding threats to their definitions that nobody else has ever seen.

In both of the above, such an allegation against a specific company would be a death sentence. Even the allegation of fabricating viruses to sell AV product could kill an AV company. As an example, we can look back to the tainting of anti-adware companies that started to become chummy with certain software companies.
--
Robb Topolski -= funchords.com =- Hillsboro, Oregon USA
~ Keeper of the D-Link FAQ ~ Did you Search? ~ More features, Free! Join BBR! ~]]> http://www.dslreports.com/forum/remark,16738886 Sat, 19 Aug 2006 12:26:39 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16738824 funchords :
said by Steve :

so I'm actually going to be able to appreciate technical arguments about why they have gone down the wrong road, it's ineffective, it's too dangerous, etc.

(to all:)

If the danger could be sufficiently reduced, would it be worthwhile to explore their technique?

It occurs to me that one could engineer these test threats such that,
- if spread beyond the intended test bed (a specific set of machines), they would be inert and/or self-delete. (A large set of antipiracy-style techniques exist.)
- if successfully executed, their behavior is sufficiently mitigated. For spam variants, .com could be replaced by .invalid. For disk writes, any changes made by the test virus are eventually reversed by the test virus as part of its execution. Etcetera.

This way, the AV industry could test heuristic behavior protection with less risk.

This does nothing for the "it is unnecessary" theory, which several smart people maintain. And, although I'm not as educated as these AV professionals are on this topic, I am pursuaded that they are probably right. But if the danger were sufficiently reduced, perhaps they would grant one another permission to test that theory provided they followed certain safety practices.

Just a thought -- and an example of a possible way to learn from what CR has done, regardless if you agree with it.
--
Robb Topolski -= funchords.com =- Hillsboro, Oregon USA
~ Keeper of the D-Link FAQ ~ Did you Search? ~ More features, Free! Join BBR! ~]]> http://www.dslreports.com/forum/remark,16738824 Sat, 19 Aug 2006 12:13:39 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16738801 GeekNJ : I wonder what those folks here that think CR did something irresponsible would think of an individual who finds a vulnerability in an OS or product and then, after contacting the company, doesn't receive an adequate response. They then post their findings "in the wild". I think that's more dangerous then what CR did, yet the latter happens all the time and is typically how some areas of the software industry need to be treated in order to react.

I think CR's testing is fine and will likely (or more appropriately hopefully) result in A/V vendors better addressing potential issues because, like Steve, I think there's a need for those not "in the business" to challenge the business.

And on a bit of a related thought, I've personally always felt that the A/V industry itself was possibly responsible for the scare and even creation of viruses in order to pump up the "We protect you from xxx,000 nasties". Of course, how many of those nasties have never been "in the wild"? :uhh:
--
Tweaked your connection? | Mail Parse | Speed Converter]]> http://www.dslreports.com/forum/remark,16738801 Sat, 19 Aug 2006 12:08:52 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16738730 eburger68 : Steve:

You wrote:

said by Steve :

Now, I happen to be much more educated about the A/V industry than the average reader of CR, so I'm actually going to be able to appreciate technical arguments about why they have gone down the wrong road, it's ineffective, it's too dangerous, etc.

But in the back of my mind, I'm still reserving the possibility that they really found a hole in the industry, and the industry doesn't like it.

I just don't know yet.
OK, fair enough. But if this does turn out to be a "hole in the industry," credit for finding it will have to go to CNET, not Consumer Reports, as CNET did this kind of testing 6 long years before Consumer Reports. And AV veterans will likely know of instances pre-dating even CNET.

If you're game, I would like to get your thoughts on the most recent issues we've been mulling over here -- namely, what the potential effect on the AV industry and its products/definitions would be if the use of lab-created viruses were to become commonplace among testing entities and researchers, esp. if it were concluded that a "hole in the industry" had been discovered.

Would customers demand that these lab viruses be added to definitions? Would they be justified in doing so?

Would AV companies be justified in adding lab viruses to definitions? Even if they weren't justified, would they be compelled to do so?

Obviously, the answers here would involve some amount of speculation, but these are serious issues nonetheless.

Eric L. Howes
--
Microsoft MVP

Sunbelt Software

Spyware Warrior]]> http://www.dslreports.com/forum/remark,16738730 Sat, 19 Aug 2006 11:51:13 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16738617 Steve : However, to shed some light on CR's sense of security advice, I got this mailer from them some years about offering their online service (I'm sure it was from their marketing department, not the testing group).

I did send this into them for the "Selling It" column, but they never printed it; I was hoping they would have a sense of humor about it and realize their error, but they didn't.

Steve
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site

Good advice?
]]> http://www.dslreports.com/forum/remark,16738617 Sat, 19 Aug 2006 11:25:22 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16738536 Steve :
said by eburger68 :

For the sake of clarity, let's narrow the focus of our disagreement to this one statement which, so far as I can tell, motivates and undergirds your entire argument:

quote:
It just smells self serving to me.

How is the demand that entities that presume to do quality, responsible AV testing follow a simple ethical rule that one shalt not create malware onesself "self serving"?
Agreeing on a set of best practices sounds like the kind of thing an industry ought to do, but the whole aura of "that's unethical" is self-important, high-and-mighty chest puffing, and it just really turns me off.

"Ethics" is about right and wrong, and CR was not unethical in any way, in spite of all the industry wailing. It's not wrong for a responsible, competent party to create test code in a lab environment in order to learn something about A/V coverage.

I certainly agree that it's dangerous, may well agree that it's unnecessary, (which would follow that there are more effective methods), but if this has not harmed anybody else, there's no ethical violation if this was all done in good faith.

Even if they somehow got in the wild accidently, that's about "negligence", not "ethics".

My knee-jerk reaction in a situation like thia is to side with Consumer Reports and not with the industry being reviewed. As a CR reader for many years, I've seen time and time again when the industry in question wailed about the reviews: it was unfair, that's not how you test that kind of thing, etc.

It's just happened before that CR used out-of-the-box thinking to think about an industry differently than the industry has. Sometimes this finds something important, sometimes it doesn't, but wails from the industry sound the same in either case.

The chest-puffing about "ethics" has that same ring to me.

Now, I happen to be much more educated about the A/V industry than the average reader of CR, so I'm actually going to be able to appreciate technical arguments about why they have gone down the wrong road, it's ineffective, it's too dangerous, etc.

But in the back of my mind, I'm still reserving the possibility that they really found a hole in the industry, and the industry doesn't like it.

I just don't know yet.

Steve
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site]]> http://www.dslreports.com/forum/remark,16738536 Sat, 19 Aug 2006 11:08:56 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16738318 eburger68 : SpannerITWks:

You wrote:

said by SpannerITWks :

What's the difference between someone sending a vendor X amount of New REAL nasties that they have discovered, that NO vendor has, and the vendors testing their AntiNasty with them, and then releasing Defs for them, and testing with those Specially written ones ?

Isn't that Exactly what would happen if those Specially written nasties got out somehow ?

Either a nasty IS a Nasty capable of doing whatever it can, or it isn't, then it ain't a nasty is it !

I believe those Specially written nasties are as valid as any others, that are new and discovered for the 1st time. Otherwise let's all pretend that ANY new nasties are completely irrelevent, and therefore we don't need protecting from it/them. Don't think so somehow !
You've essentially elaborated the logic that would cause customers of AV companies to demand that lab viruses be added to AV definitions, and the logic that AV companies might be forced to bow to, if lab viruses became commonplace enough.

So, my question, though, remains: would you personally feel fine shelling out money each year for a subscription to an anti-virus product's definitions when those definitions were in part necessary in order to cover viruses that AV researchers, testers, and companies were themselves cooking up in the lab?

How do you think others would react to the same proposition -- that they had to pay for protection from viruses created by parts of the AV industry itself?

Eric L. Howes
--
Microsoft MVP

Sunbelt Software

Spyware Warrior]]> http://www.dslreports.com/forum/remark,16738318 Sat, 19 Aug 2006 10:15:39 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16738031 Mele20 : It doesn't really matter what you think. What matters is what the readers and supporters of CR think. I think they will conclude that CR has not done anything unethical or rash or stupid. In fact, I venture to say that most CR readers will say "right on"! CR has taken on the AV vultures and the pundits who think they are such great advisors. The public will be behind CR and that is how it should be.

I don't believe they did anything wrong or that there is anything wrong with how they tested. I think you and Symantec, etc. are bloating this all out of proportion. I also think the AV vendors probably do create viruses just so they can stay in business. I'm a cynic especially when it comes to AV vendors and security "consultants". The typical reader of CR doesn't like their sacred cow attacked and won't allow anyone who does so to get away unscathed.

You have gone on ad nauseum in this thread and I wonder why "thou doest protest so much"? Are you a shill for the AV companies? If someone else, just about anyone else (other than IBK who also has a very obvious vested interest and the AV vendors you quote for support), had said some of what you have said, it might have been credible but your saying it is just plain laughable. I'll take CR's way any day over yours and I think the general public feels as I do. We don't need your "pontificating"...we trust CR....we don't trust you or the AV vendors ...all with heavily vested interest. That is it in a nutshell.

Anyhow this is a tempest in a teapot. The typical CR reader will never see this thread and would dismiss it if they did. So, it doesn't matter what you think. It matters that the public reveres CR, believes in its integrity and abilities to properly assess what ever is under the microscope for that issue. You and the AV companies seem to be blind because it makes no sense for vested interests to attack an impeccable entity like CR and expect to have the public not spit in their face.
--
"If you want to do DRM on a PC then you need to treat the user as the enemy." Ross Anderson in "`Trusted Computing' Frequently Asked Questions"]]> http://www.dslreports.com/forum/remark,16738031 Sat, 19 Aug 2006 08:40:26 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16737991 SpannerITWks : What's the difference between someone sending a vendor X amount of New REAL nasties that they have discovered, that NO vendor has, and the vendors testing their AntiNasty with them, and then releasing Defs for them, and testing with those Specially written ones ?

Isn't that Exactly what would happen if those Specially written nasties got out somehow ?

Either a nasty IS a Nasty capable of doing whatever it can, or it isn't, then it ain't a nasty is it !

I believe those Specially written nasties are as valid as any others, that are new and discovered for the 1st time. Otherwise let's all pretend that ANY new nasties are completely irrelevent, and therefore we don't need protecting from it/them. Don't think so somehow !

The same goes for some bug etc in software that could be exploitable. What should someone do that discovers it, vendor or otherwise, nothing, or get busy with da fizzy and write some code to fix it, or pass the info on to the vendor if it's not their forte ?

Err not really too difficult to answer is it ! Cos if they don't someone out there will take advantage of it sooner or later, as they continue to do, almost weekly these days. And where would and does that leave MOST users out there, yeah right up **** street without a paddle that's where, as it frequently does !

Spanner

edit typo Only
--
I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks
/SpannerITWks]]> http://www.dslreports.com/forum/remark,16737991 Sat, 19 Aug 2006 08:22:02 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16737872 eburger68 : SnowyOne:

You wrote:

said by SnowyOne :

I've had my opinion change thanks to the well argued points in this thread & that's to the credit of every poster.
said by eburger68 :

Let's say for the sake of discussion that we all decide that it is perfectly OK for testing entities like CR, AV-Comparatives, Virus Bulletin, as well as PC Magazine and PC World to start creating and using lab viruses in their testing. And let's assume that this practice becomes rather commonplace in AV testing -- perhaps not universal, but routine enough that it doesn't shock those in the AV industry.
That assumes AV-Comparatives, Virus Bulletin, as well as PC Magazine and PC World all have the same level of history, credibility & impartiality that CR has. They don't.
I haven't read the full article yet either, but I can say that if someone were to write a new chapter in product testing CR would be my odds on favorite to be the one that gets it right.
But as a practical matter, it's not going to work out that CR somehow gets a "pass" for creating and using lab viruses and everyone else in the industry itself refrains. If CR is seen to use lab viruses for testing, and CR has the influence that folks in this thread have attributed to it, then others will be compelled to join the virus-creation game as well. One shouldn't let a misplaced affection for CR cloud one's understanding as to what the likely consequences of its irresponsible actions could be. You are not going to be in a position to determine who is allowed to create lab viruses for research and who is not.

As for the reputation of CR, even CR recognized that it was incapable of doing this kind of testing without the assistance of what it presumed to be experts from the world of computer security -- see CR's own discussion of how it tested (link in first post of this thread).

Moreover, as IBK pointed out earlier, there is nothing new or trailblazing about the testing done by CR, contrary to CR's own self-serving claims (there's that word again!). It was only CR's own ignorance of AV testing that allowed it to make such claims.

Finally, there will be no new "chapters in product testing" that get written as a result of this affair -- save the one that documents how an otherwise respected testing entity stumbled badly because it failed to heed the advice of those who had gone before -- because CR has not and is not likely to be offering the raw data from its testing up to other testing entities for peer review. (We'll set aside for the minute the fact that such peers would be ethically obligated and would likely refuse to take delivery of the data, for the all the reasons that IBK outlined in an earlier post.)

I'm sorry folks, but if you're starting from the position that CR's testing is "holy ground" that puts their ethics and behavior beyond scrutiny, or that CR's impartiality is of such paramount significance that it overrides demonstrable flaws in their testing methodology, then this discussion has exited the realm of the rational analysis of empirical testing.

Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior]]> http://www.dslreports.com/forum/remark,16737872 Sat, 19 Aug 2006 07:27:21 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16737547 SnowyOne : I've had my opinion change thanks to the well argued points in this thread & that's to the credit of every poster.
said by eburger68 :

Let's say for the sake of discussion that we all decide that it is perfectly OK for testing entities like CR, AV-Comparatives, Virus Bulletin, as well as PC Magazine and PC World to start creating and using lab viruses in their testing. And let's assume that this practice becomes rather commonplace in AV testing -- perhaps not universal, but routine enough that it doesn't shock those in the AV industry.
That assumes AV-Comparatives, Virus Bulletin, as well as PC Magazine and PC World all have the same level of history, credibility & impartiality that CR has. They don't.
I haven't read the full article yet either, but I can say that if someone were to write a new chapter in product testing CR would be my odds on favorite to be the one that gets it right.
I also believe if they conclude that they screwed up they'll openly admit it.]]> http://www.dslreports.com/forum/remark,16737547 Sat, 19 Aug 2006 03:04:54 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16737216 funchords : Eric,

Apparently you don't think that we hear your point.

I hear your point, and I heard it before your last two messages. You are repeating it over and over.

It's a settled matter in the AV community that it is bad to create malware merely for testing:
- because it somewhat artifically and needlessly inflates the number of known threats that perhaps need a signature
- because the originator is compelled to release them to others for test and prevention research, and by doing such loses control of them

I think, and have thought all the while, that these are reasonable positions, reasonably arrived at. I understand them and I think they are valid.

To which I add, without contradicting:
- CR has done what it has done. Let's not simply dismiss it, but let's see if there's anything to learn from the results that they obtained with their method.
- Even if one disagrees with their method, let's not attack CU's ethics. They have no dog in this fight. No products to sell or advertise. They may have done something that was ill advised, but they have not shown bias nor have they behaved outrageously.
--
Robb Topolski -= funchords.com =- Hillsboro, Oregon USA
~ Keeper of the D-Link FAQ ~ Did you Search? ~ More features, Free! Join BBR! ~]]> http://www.dslreports.com/forum/remark,16737216 Sat, 19 Aug 2006 01:13:42 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16737117 eburger68 : Funchords:

One other thought, if you'll allow me. You wrote:

said by funchords :

As a reader, I expect the Consumers Union to come up with a reasonable way to compare one product against another, describe what they did, and to report what they found. As far as I'm concerned, they did that. I think it's a benefit to everyone that they took a different approach. And I think the jury is out as to whether their method, and their results, turn out to be right.
So, let's take this idea and run with it -- and imagine the kind of malware and anti-malware landscape that results.

Let's say for the sake of discussion that we all decide that it is perfectly OK for testing entities like CR, AV-Comparatives, Virus Bulletin, as well as PC Magazine and PC World to start creating and using lab viruses in their testing. And let's assume that this practice becomes rather commonplace in AV testing -- perhaps not universal, but routine enough that it doesn't shock those in the AV industry.

Now, knowing that all these testing entities were in possession of thousands upon thousands of virus variants, would you or would you not want those testing entitities to turn over the samples to the AV companies so that the companies could generate signatures for those variants and play around with them for developmental purposes?

Signatures would be important, because although heuristics, HIPS, and behavioral analysis of malware is improving, we have yet to see a preventative technology emerge (save that trusty pair of wire cutters) that can guarantee zero infections, zero infestations in a networked environment. There will be a market for remediation tools for some time to come.

So, the AV companies start getting an influx of lab viruses to shove into their definitions, which then swell with artificial malware that everyone hopes (but no one can guarantee) would stay within the confines of the lab.

Or maybe they don't, for whatever reason -- perhaps the testing entities refuse on principle to supply any more than general descriptions of how their own lab variants were created.

Would you then allow that the anti-virus companies themselves would be justified in creating lab viruses, if only to attempt to replicate the kinds of viruses that they know exist somewhere in the world and which, some cases, might have been leaked to less responsible entities, raising a real question as to whether such lab viruses aren't fair game for the definitions that AV companies create?

At this point I ask our readers to consider what their reaction would be were they to find out that the yearly anti-virus subscriptions being sold to consumers functioned, in part, to grant them access to definition updates for viruses and malware that the AV companies themselves -- along with AV testing entities like CR -- were cooking up in their own labs?

Can you imagine the hue and outcry? We already have a few skeptics around here who aren't thoroughly convinced that the AV industry isn't just hiping, if not actually playing a surreptitious role in the creation of the very threats they sell protection for. Can you imagine what the reaction of these and other folks would be once they realized that virus definition subscriptions were, in part, payments to the AV industry for protection against viruses created in those very same AV companies' own labs?

No, down that road lies madness. That's partly why AV industry experts insist that AV entities -- be they product vendors, researchers, or testing entities -- refrain from creating malware themselves -- because the widespread acceptance of lab-virus creation woud quickly involve the AV industry in a terrible ethical and practical quandry.

Eric L. Howes
--
Microsoft MVP

Sunbelt Software

Spyware Warrior]]> http://www.dslreports.com/forum/remark,16737117 Sat, 19 Aug 2006 00:48:20 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16736916 eburger68 : Steve:

For the sake of clarity, let's narrow the focus of our disagreement to this one statement which, so far as I can tell, motivates and undergirds your entire argument:

quote:
It just smells self serving to me.

How is the demand that entities that presume to do quality, responsible AV testing follow a simple ethical rule that one shalt not create malware onesself "self serving"?

How is the demand that those who would presume to test AV products against "unknown" variants take the safer, saner, and more scientifically valid approach of conducting retrospective testing "self serving"?

"Self serving" in what way? Because to prefer retrospective testing over the creation of lab viruses would impose some kind of onerous burden on prospective new testers so that the AV industry could keep the testing game all to itself? Is retrospective testing really THAT onerous and difficult to conduct?

Because lab-created viruses might produce more valid results that would allow non-standard products not favored by industry insiders to rise to the top of test results? Is Bit Defender not an established player in the AV industry? How about Kaspersky? KAV placed a respectable third in this testing, and even they protested.

The AV industry has an extensive body of literature on testing, and, if anything, the recommendations and admonitions you'll find there-in often make testing easier to perform as well as more reliable. Indeed, CR could have saved itself quite a lot of headache and expense (and given its readers more reliable test results) had they not resorted to synthetic virus creation.

Really and truly I don't get the "self-serving" charge here. If you're going to make it, you ought to at least be able to explain what the industry hopes to gain by insisting on such an ethical standard. Thus far, I haven't seen anything beyond a flip, empty accusation.

Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior]]> http://www.dslreports.com/forum/remark,16736916 Sat, 19 Aug 2006 00:05:33 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16736793 eburger68 : Funchords:

You wrote:

said by funchords :

The Consumers Union is an independent body. The assumption up front is that they are objective.

I would be more inclined to ask an independent body to confirm tests that might be subject to bias.
When evaluating the validity of testing, there is more to consider than just "bias" and "objectivity" -- words which are too often pushed into service to stand in the place of other considerations. One can be independent, perfectly unbiased, and as "objective" as one could ever hope to be and still produce hopelessly invalid, meaningless tests. Lack of bias is no guarantee of the quality of a test.

said by funchords :

There are plenty of examples in the medical community where high-risk or limited quantity test items are not shared. That fact doesn't make the testing invalid.
And here we are talking about anti-virus testing of mass consumer products, testing that could have and should have been performed with the plethora (nay, the tidal wave) of valid samples in the wild. The kinds of medical studies that you're referring to are the exception, not the rule, and are conducted nonetheless because there is no other practical way to obtain the uniquely valuable data available only in such rare circumstances. CR's testing of AV products doesn't even come close to fitting those criteria, esp. in light of the alternative methodologies that had been known and practiced for years as well as the wide availability of "in the wild" malware to test against.

said by funchords :

Medical and psychological studies involve groups of individual subjects, with a description of why these individuals were interesting from a test perspective. These studies are not invalid.
Again, bad comparison. The studies you're referring to are constrained by unique characteristics of the subjects -- human beings, whose qualities obviously forego the question of handling like common lab samples. And to compensate for these limitations, the medical and psychological communities set up and run numerous similar experiments on various populations while striving to control for the inevitable differences among their subjects. As with your earlier example, what you're describing are unavoidable constraints -- constraints which aren't a part of the situation we're dealing with here.

But even if we were to accept that the comparison was apt, CR has done nothing to release data on the virus variants that they created. Nothing even resembling the kind of data usually supplied for the subjects of medical and psychological studies has been forthcoming from CR, so even on the basis of this comparison CR's AV testing must be considered invalid.

said by funchords :

The AV Comparitor web-site I visited last night approached the "0 day" threat differently. To summarize, they used a version of the AV product that was several weeks old, and tested it using more recent viruses that could not have been added into definition updates yet.

That's a plausible approach, but does suffer the very same "fortune telling" sin that the Avert Blog was complaining about.
Not at all, because this kind of retrospective testing is conducted actual viruses and malware in the wild, not against lab malware that one organization speculated might be in the wild some day.

said by funchords :

I haven't read the CR article, yet. But if the results are very different than AV Comparitor's results, then hopefully this starts a debate that improves the latter. CR isn't going to review AV products in its next issue, but AV Comparitor will.
How would AV Comparatives benefit from CR's testing when no crticial data about this test bed of lab viruses has been disclosed by CR to say nothing of the samples themselves? Again, the dilemma: to be useful in any way to the wider AV community, CR would have to start distributing those samples, which fairly demolishes the claim that these things were guaranteed to stay within the safe confines of one lab.

said by funchords :

Every time I see someone question the ethics of the Consumers Union, I have an emotional response. (And it's not just you, the industry blogs are all doing it.) This is an organization that has decades of behavior beyond reproach. The industry is walking on their customer's holy ground -- they would be well advised to behave themselves.
A testing entity is not "holy ground," and to regard a testing entity as anything like that does a disservice to the testing entity as well as the public it is supposed to serve.
--
Microsoft MVP
Sunbelt Software
Spyware Warrior]]> http://www.dslreports.com/forum/remark,16736793 Fri, 18 Aug 2006 23:39:55 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16736719 Steve :
said by eburger68 :

And I, too, would expect CR to be able to make the distinction. But the point is, once they redistribute, those viruses are effectively beyond their control.
I'm just unmoved. We're not talking about nuclear secrets or actual ebola virus: they're just bits, and we have no evidence that they have gone one walkabout. It's a big "so what?" to me.
Bad comparison. Justin didn't create any malware and isn't re-distributing any malware that wasn't already in the wild -- that's the key difference.
Yes, this is a difference, but it's not that big of a one to me.
1) Here the experts are urging the advice on each other -- the very folks one would expect to be the exceptions to the rule.
Industries do this all the time, and it means nothing about the genuine-ness of their motives.

You have to have a barber's license to cut hair in California, and this "imposing a requirement on themselves" was ostensibly done to protect the consumer, but was actually done to increase the barriers of entry into the field and to reduce competition.

Big payroll companies are behind the push for expensive SAS70 audits (technology audits by CPAs, which pretty much fills you in on their utility), mainly to impose costs on the little guys. This is an industry imposing rules on itself in order to increase the barriers of entry into the field and to reduce competition.

I don't believe that this is behind the sentiment going on here — your heroic efforts are informed by motives which are beyond reproach — but industries tend to look at things from their own point of view. They may not be the same as mine.
And the fact that the creation of 5500 new viruses lacks any practical or methodological justification makes the ethical lapse even more glaring. If they could mount a credible defense that such steps were necssary to allow some unique and innovative testing to proceed -- testing that might shed real light on the capaibilities of the tested AV apps and/or the behavior of malware -- then we might be looking at one of those ethical "corner cases."
Whether you're right or wrong on this, many people see the hue and cry about the ethics as a smokescreen, and it hurts the cause to focus on it. It just smells self serving to me.
But we're not. We're just looking at some run-of-the-mill irresponsible behavior by an otherwise respected testing entity that should have known better -- and all in the name of some rather unimpressive testing that shed light on little of anything except the organization's own ignorance and carelessness.
... and whether the objection on ethical grounds is well founded or not, that casts no light on whether CR was actually competent or incompetent in this matter.

I am more than willing to accept that they have received bad advice from their "experts", have gotten in far beyond their competence, and did an all-around bad job.

The "ethics" are just a side show.

Steve
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site]]> http://www.dslreports.com/forum/remark,16736719 Fri, 18 Aug 2006 23:27:38 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16736613 eburger68 :
said by Steve :

I am unmoved by this argument: I think it's possible to distinguish between a known virus researcher and Ivan J. Trojanovic - that kind of distinction goes on all the time by those with common sense.
And I, too, would expect CR to be able to make the distinction. But the point is, once they redistribute, those viruses are effectively beyond their control.

said by Steve :

Data point: DSLR has a "Malware Archive" forum where people post samples of badware, but only those on a trusted list can download them for research. I'm on that list, you probably are too. Is Justin "distributing" malware? Or just using his head in an effort at public service?
Bad comparison. Justin didn't create any malware and isn't re-distributing any malware that wasn't already in the wild -- that's the key difference. If he or anyone else affiliated with DSLR/BBR did start creating malware on their own and re-distributing it for the sake of prodding AV companies to bolster their ability to detect variants (which I wouldn't expect DSLR/BBR to do, obviously) then the same objections would apply.

said by Steve :

Sure, but advice that applies generally does not always apply specifically in every possible case.
Two points:

1) Here the experts are urging the advice on each other -- the very folks one would expect to be the exceptions to the rule.

2) In order to argue for an exception to the rule, one would have to mount a fairly strong case that the exception was justified on the grounds that the sought-after results were practically obtainable through no other means and that the potential risks were far outweighed by the unique benefits that would incur. In this situation CR can't even come close to making such an argument. Their only possible justification was expedience, to say noting of their own ignorance of established AV testing methodologies.

said by Steve :

Example: The recognized experts also strongly urge people to run antivirus software on their desktops, but I never have. I'm very well educated in virus infection vectors, am extraordinarily careful, and have never had an infection in almost 30 years of using a computer. I'm not the only in this forum who believes this.
Again, the analogy/comparison doesn't apply, because your example involves general advice given to the general population. In this situation, the recognized authorities came to the conclusions they did regarding the ethical behavior of other experts, not the general population (my dad is an unlikely target for such admonitions, as it's rather unlikely he'd ever feel the urge to pull together a malware zoo and begin experimenting on it).

said by Steve :

The validity of the CR testing will stand or fall on the merits of the methodology, not the ethics. I'm so confident that the smart people like you will point out why they have reached an unwarranted conclusion that I simply do not care about the "ethics" issue. You'll demolish them without it, so to me it's just a distraction.
And the fact that the creation of 5500 new viruses lacks any practical or methodological justification makes the ethical lapse even more glaring. If they could mount a credible defense that such steps were necssary to allow some unique and innovative testing to proceed -- testing that might shed real light on the capaibilities of the tested AV apps and/or the behavior of malware -- then we might be looking at one of those ethical "corner cases."

But we're not. We're just looking at some run-of-the-mill irresponsible behavior by an otherwise respected testing entity that should have known better -- and all in the name of some rather unimpressive testing that shed light on little of anything except the organization's own ignorance and carelessness.

Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior]]> http://www.dslreports.com/forum/remark,16736613 Fri, 18 Aug 2006 23:07:10 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16736462 funchords :
said by eburger68 :

And when independent bodies ...
The Consumers Union is an independent body. The assumption up front is that they are objective.

I would be more inclined to ask an independent body to confirm tests that might be subject to bias.

said by eburger68 :

...demanded copies of those viruses in order to validate the test bed, which is a basic requirement of scientifically valid testing, what then? If you provide them then you have effectively become a virus distributor. And can you vouch for the safety of the parties to whom you distributed the samples?

If you refuse, then your test's results are invalid. End of story.

There are plenty of examples in the medical community where high-risk or limited quantity test items are not shared. That fact doesn't make the testing invalid.

Medical and psychological studies involve groups of individual subjects, with a description of why these individuals were interesting from a test perspective. These studies are not invalid.

said by eburger68 :

To return to the original issue -- the full issue -- which is whether CR had any justification -- be it practical, methodological, or ethical -- to create 5500 new viruses for testing
I say yes, you say no.

The AV Comparitor web-site I visited last night approached the "0 day" threat differently. To summarize, they used a version of the AV product that was several weeks old, and tested it using more recent viruses that could not have been added into definition updates yet.

That's a plausible approach, but does suffer the very same "fortune telling" sin that the Avert Blog was complaining about.

I haven't read the CR article, yet. But if the results are very different than AV Comparitor's results, then hopefully this starts a debate that improves the latter. CR isn't going to review AV products in its next issue, but AV Comparitor will.

said by eburger68 :
in order to establish and defend the methodological integrity of the testing, CR would unavoidably risk compounding its ethical lapses
Every time I see someone question the ethics of the Consumers Union, I have an emotional response. (And it's not just you, the industry blogs are all doing it.) This is an organization that has decades of behavior beyond reproach. The industry is walking on their customer's holy ground -- they would be well advised to behave themselves.

To me, there is no ethical question here. The question is whether there was an effective methodology. Did they make a mistake?

It is possible to have an ineffective methodology, and to have taken uninformed dangerous risks, and still be ethical.

I think their methodology is plausible and that the "danger" exists but is being overblown. For me, I think it's done and it is interesting. I'm not sure it is the same choice that I would make, had I been CR's tester.

The highly emotional response (ranting) by the AV industry is not serving them at all. I would expect them to recognize that Consumer Reports reviews lawn mowers and hair dryers and everything else, and might not return to a set of products for several years.

As a reader, I expect the Consumers Union to come up with a reasonable way to compare one product against another, describe what they did, and to report what they found. As far as I'm concerned, they did that. I think it's a benefit to everyone that they took a different approach. And I think the jury is out as to whether their method, and their results, turn out to be right.
--
Robb Topolski -= funchords.com =- Hillsboro, Oregon USA
~ Keeper of the D-Link FAQ ~ Did you Search? ~ More features, Free! Join BBR! ~]]> http://www.dslreports.com/forum/remark,16736462 Fri, 18 Aug 2006 22:41:07 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16736414 Steve :
said by eburger68 :

And when independent bodies demanded copies of those viruses in order to validate the test bed, which is a basic requirement of scientifically valid testing, what then? If you provide them then you have effectively become a virus distributor. And can you vouch for the safety of the parties to whom you distributed the samples?
I am unmoved by this argument: I think it's possible to distinguish between a known virus researcher and Ivan J. Trojanovic - that kind of distinction goes on all the time by those with common sense.

Data point: DSLR has a "Malware Archive" forum where people post samples of badware, but only those on a trusted list can download them for research. I'm on that list, you probably are too. Is Justin "distributing" malware? Or just using his head in an effort at public service?
If you refuse, then your test's results are invalid.
I agree that reproducible tests are not really valid in the scientific study sense, but the consumer won't care about much of that: many people trust CR to be unbiased (which I believe they are here) and expert (which they may not be), and are happy to just accept their conclusions.

When I'm shopping for a bbq grill or a dishwasher, I usually get what they like without digging in too much to just how they got their answer. They are smart about this, I'm pretty dumb, and am better off in the long run to just defer to their judgement.

But yes: if nobody can reproduce their results, then it really casts doubt on just how they filled in those little circles.
But the point here is that even the recognized experts in the field strongly advise against this type of behavior.
Sure, but advice that applies generally does not always apply specifically in every possible case.

Example: The recognized experts also strongly urge people to run antivirus software on their desktops, but I never have. I'm very well educated in virus infection vectors, am extraordinarily careful, and have never had an infection in almost 30 years of using a computer. I'm not the only in this forum who believes this.

This doesn't mean that I'm "evidence" against the expert advice, it doesn't mean that I recommend others take this course, and it doesn't mean that I object to the advice (I don't - I urge it strongly of others).

It just means that there are corner cases in most maxims.
And as has been pointed out now, the issue of lab security goes beyond one's own security precautions, but the precautions of those to whom one might be obligated to share these synthethic viruses in order to establish the scientific credibility of the testing.
The validity of the CR testing will stand or fall on the merits of the methodology, not the ethics. I'm so confident that the smart people like you will point out why they have reached an unwarranted conclusion that I simply do not care about the "ethics" issue. You'll demolish them without it, so to me it's just a distraction.
And I don't expect you or the other established security professionals would take kindly to having their own views characterized and dismissed in this manner.
I dismiss them because they are unnecessary: you can fully make your case without the self-serving don't-try-this-at-home arguments.

Steve
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site]]> http://www.dslreports.com/forum/remark,16736414 Fri, 18 Aug 2006 22:33:09 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16736083 eburger68 : Steve:

You wrote:

said by Steve :

Just because it's easy to accidently infect the world with ebola doesn't mean that nobody has the ability to create a virus test environment with proper precautions against leakage: I'm quite sure that both you and I would be able to construct such an environment.
And when independent bodies demanded copies of those viruses in order to validate the test bed, which is a basic requirement of scientifically valid testing, what then? If you provide them then you have effectively become a virus distributor. And can you vouch for the safety of the parties to whom you distributed the samples?

If you refuse, then your test's results are invalid. End of story. In which case, just what was the value of creating those viruses in the first place -- esp. given that there is no shortage of viruses and other malware to test against. Nor is there a lack of methodological alternatives to accomplish the same goals with real viruses in the wild.

And, by the way, we haven't even broached the subject of how Consumer Reports internally validated those lab-created viruses? Did they execute them in order to verify that the changes they had made to the pre-existing variants hadn't rendered the synthetic variants non-executable or the payload null? Did they diligently execute every single one of those 5500 new viruses? These are important questions because if CR failed to validate the viruses internally, then they have no reason to know that they weren't testing against non-viruses -- i.e., non-threats, which the tested AV apps would be perfectly justified in NOT detecting becaue the threats weren't real.

said by Steve :

I am not arguing for the value of synthetic viruses like this (I don't know, I'm not an expert), or that CR actually did so in a safe manner, but it's not out of the question that they were aware of this issue and retained the proper experts to make sure that this didn't happen.
But the point here is that even the recognized experts in the field strongly advise against this type of behavior. And as has been pointed out now, the issue of lab security goes beyond one's own security precautions, but the precautions of those to whom one might be obligated to share these syntethic viruses in order to establish the scientific credibility of the testing.

said by Steve :

It's perfectly fair to object to the testing methodologies on their merits, and to urge others not to play with fire, but this discussion about the "ethics" of this kind of testing smells incredibly self-serving: "We're the only ones who know how to do that"
This is a bit disappointing, Steve. You've been in these forums as long as I have, and on more than a few occasions you've drawn on your own impressive professional knowledge and experience to argue emphatically that such-and-such action, behavior, process, or decision was muddle-headed, improper, dangerous, or even unethical. And you have been quite justified in doing so, given the depth of experience and expertise that you bring to the table. To so casually dismiss the judgments of recognized experts in the AV field is not what I would have expected. And I don't expect you or the other established security professionals would take kindly to having their own views characterized and dismissed in this manner.

To return to the original issue -- the full issue -- which is whether CR had any justification -- be it practical, methodological, or ethical -- to create 5500 new viruses for testing, I hope that it is becoming clear that even if one considers CR's actions but a minor or negligible transgression, that there simply was no practical or methodological justification for them. Moreover, in order to establish and defend the methodological integrity of the testing, CR would unavoidably risk compounding its ethical lapses. These aren't neatly separable issues.

Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior]]> http://www.dslreports.com/forum/remark,16736083 Fri, 18 Aug 2006 21:39:25 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16735511 Steve :
said by eburger68 :

This is just a variant of the "guns don't kill people, people kill people" argument.
No, it really isn't.

Just because it's easy to accidently infect the world with ebola doesn't mean that nobody has the ability to create a virus test environment with proper precautions against leakage: I'm quite sure that both you and I would be able to construct such an environment.

I am not arguing for the value of synthetic viruses like this (I don't know, I'm not an expert), or that CR actually did so in a safe manner, but it's not out of the question that they were aware of this issue and retained the proper experts to make sure that this didn't happen.

It's perfectly fair to object to the testing methodologies on their merits, and to urge others not to play with fire, but this discussion about the "ethics" of this kind of testing smells incredibly self-serving: "We're the only ones who know how to do that"

Steve
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site]]> http://www.dslreports.com/forum/remark,16735511 Fri, 18 Aug 2006 20:10:17 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16735267 IBK : av-test.org and av-comparatives provide retrospective tests to see how well av products protect against new _real-world_ viruses/malware. (and the results etc. can be seen by anyone for free without having to pay a subscription fee).
btw, (something i wanted to tell since long time) remember that the test of CR was done most probably months ago, as usually printed magazines (articles etc.) are prepared around 30 days prior. considering that they engaged other peoples to do the test and that they needed to provide them enough time for doing this and then to write the article, the test must been had done months ago. So they could have - instead of creating new variants of old viruses - making a retrospective test which would deliver valid results. Of course that is more time consuming, but they would not need to create new virus variants.
so they have now those variants on a CD in a safe. ok. AV vendors of course would like to know what kind of files were used in this test, in order to see if the results are true (even if from the method etc. invalid). Now the dilemma: if they do not give the samples to the AV vendors, they can not check if the samples works etc., and if they send the new variants to the AV vendors, the AV vendors are obligated to add those variants to their databases (= and that is not good, because if AV vendors have to add the viruses created for testing reasons, the scanning speed may be affected. So for who is all this of help? Not for the users and not for the vendors. Probably just for CR as they get publicity and new paying subscribers).
sidenote: they state that there are no independent tests to measure how well av software is against new threats, which is absolutly wrong. If they would know a little bit on this materia, they would know or have read about methods to do such tests and would also know or see (by using e.g. google) that such tests exist (like I said before provided by av-test.org which publishes those results on various magazines around the world and av-comparatives which has the results/report publicly available online for free).
p.s.: this is just my opinion.]]> http://www.dslreports.com/forum/remark,16735267 Fri, 18 Aug 2006 19:27:48 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16735146 AB :
said by HMS1 :

The danger increases with the number of viruses only if your behavior is careless. Instead focus on the possible ways they can get into the computer and be executed with privileges. The number of ways this can happen is far smaller than the number of viruses, and the list of infection routes remains far more consistent than the spectrum of viruses, and it is much more in your control.
HMS, you may well be correct in what you say. However, I look at this way-- It doesn't matter to me if I live in a secure, gated, guarded, moated castle. When I open the curtains to look outside, my preference is to see green pastures and children playing, not a teeming horde of ne'er-do-wells looking for a way to breach the ramparts!
So when I hear about researchers thinking up 5500 new ways to infect me, I don't like it! Let 'em do their research on the OLD stuff. Now, that may not be very scientific, nor am I saying it's right, but that's my opinion anyway! ;)]]> http://www.dslreports.com/forum/remark,16735146 Fri, 18 Aug 2006 19:05:51 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16735030 HMS1 :
said by AB :

And it is, btw, 'common sense' (as well as mathematics) that says I am much more likely to be exposed to a virus if there are are 105,500 of them out there than I would be if there were only a couple of dozen!

The danger increases with the number of viruses only if your behavior is careless. Instead focus on the possible ways they can get into the computer and be executed with privileges. The number of ways this can happen is far smaller than the number of viruses, and the list of infection routes remains far more consistent than the spectrum of viruses, and it is much more in your control.]]> http://www.dslreports.com/forum/remark,16735030 Fri, 18 Aug 2006 18:48:55 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16734987 HMS1 : It's not a "blame the user" argument. I'll try to say it another way (and btw I was revising my 2nd post, trying to be clearer, just when you were posting).

The point is, if the addition of a new virus makes a difference in the administrator's or protection vendor's strategy, then the strategy is inadequate in the first place. We already know the outer limits of what viruses can do (viz. what the user account allows), and we already know how they get into the LAN or local system (email, junkware etc.). The only difference a new virus makes is some new variation of what they do to the system once infection is already underway.

Following up on the guns analogy, it's as if the whole approach to prevention of shootings is listing all the various types of bullets, and then complaining if someone makes a new kind of bullet, and saying it increases the risk. Instead you just have to keep the guns out of the courthouse or airport. Then it doesn't matter what kind of bullets they use.]]> http://www.dslreports.com/forum/remark,16734987 Fri, 18 Aug 2006 18:40:16 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16734980 AB :
said by eburger68 :

. . People being what they are -- which is to say fallible, gullible, ignorant, lazy, and prone to error --
Hey! Watch it, buddy! I resemble that remark! :D
. . it can be expected that the introduction of new viruses into the environment does increase the risk of people executing those viruses, if even accidentally. (sic)
And, by the way, in making this argument I am most certainly not slighting efforts to reduce the opportunities or chances for users to run untrusted code. We can do both: keep less open gasoline lying around AND keep people away from the gasoline.

Eric L. Howes
Again, Amen!
I run a secure PC, I use common sense when I surf. And it is, btw, 'common sense' (as well as mathematics) that says I am much more likely to be exposed to a virus if there are are 105,500 of them out there than I would be if there were only a couple of dozen!
And I believe there is a saying-- "The road to Hell is paved with good intentions." ;)]]> http://www.dslreports.com/forum/remark,16734980 Fri, 18 Aug 2006 18:39:27 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16734788 eburger68 : HMS:

This is just a variant of the "guns don't kill people, people kill people" argument. This time it's, "viruses don't pose threats; people's executing of viruses poses threats," as if people were some optional, extraneous component of the threat environment.

People being what they are -- which is to say fallible, gullible, ignorant, lazy, and prone to error -- it can be expected that the introduction of new viruses into the environment does increase the risk of people executing those viruses, if even accidentally.

One can blame the people or users for being lazy, ignorant, and all the things that people can tend to be in their more error-prone modes of being, but the fact remains that the introduction of new viruses into an environment where fallible users (and researchers) can access them increases the risk of harm being done.

And, by the way, in making this argument I am most certainly not slighting efforts to reduce the opportunities or chances for users to run untrusted code. We can do both: keep less open gasoline lying around AND keep people away from the gasoline.

Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior]]> http://www.dslreports.com/forum/remark,16734788 Fri, 18 Aug 2006 18:05:41 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16734121 HMS1 : Maybe the point in my first paragraph was unclear. What I meant was: adding some new viruses does not make any significant difference to internet (in)security. They don't pose any danger merely by existing. Viruses by definition cannot do anything without user interaction.

If a user runs a virus, the harm done depends on the particular virus. But how would the differences between viruses make any difference in policies or defenses? Policies must be against any/all untrusted code, without knowing in advance what it will be. And defenses must be against any possible virus, not only a "known" list. Signature-based anti-virus is a dead end.

Putting it another way, a user's risk is the same with or without a new batch of viruses being loose. With or without any addition to the virus pool, the potential harm includes whatever can be done on the user's account, and the spectrum of what's in the wild must be assumed to be whatever the authors can, in principle, create. These factors do not change with addition of new viruses.

The problem is users running untrusted code, not whether the range of viruses is (big number) or (big number + small number).]]> http://www.dslreports.com/forum/remark,16734121 Fri, 18 Aug 2006 16:27:29 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16733374 AB :
said by eburger68 :

. . . the issue lies in creating new viruses that could, despite the best efforts of their creators, escape into the wild, and this kind of thing is a regular occurrence in the virus-writing world. The bottom line is, you don't compound the problem by writing these things yourself, even with the best of intentions, because intentions will matter very little if the thing escapes from the lab.

Eric L. Howes
Amen, Brother!]]> http://www.dslreports.com/forum/remark,16733374 Fri, 18 Aug 2006 14:33:15 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16733297 eburger68 :
said by HMS1 :

I don't see any problem with CR having created a bunch of new viruses. Obviously the bad guys can do the same whenever they want (the Bagle guys were cranking out the variations for a while, for example). CR's action did not increase this ability of malicious actors, nor would refraining have restrained the malicious actors.
The ethical issue doesn't involve "increasing" or "restraining" the ability the ability of bad guys to create viruses or other malware.

No, the issue lies in creating new viruses that could, despite the best efforts of their creators, escape into the wild, and this kind of thing is a regular occurrence in the virus-writing world. The bottom line is, you don't compound the problem by writing these things yourself, even with the best of intentions, because intentions will matter very little if the thing escapes from the lab.

One AV researcher that I know has received source code for viruses that authorities like the FBI and Secret Service have uncovered in raids. This researcher refuses on principle even to compile the code himself for the purpose of lab analysis. The issue for him is just that clear-cut -- just that serious.

said by HMS1 :

Retrospective testing is fine for anti-virus. But if you're working on heuristic detection, you may have good reason to write some new malware to test against. There may be some preexisting program that fits the description of what you want, or there may not be, or it may be too much trouble to find one.
The beautiful thing is that you already provided all the reasons why it's not even necessary to write new viruses back up in your first paragraph:

quote:
Obviously the bad guys can do the same whenever they want (the Bagle guys were cranking out the variations for a while, for example).

Precisely. There is no shortage of viruses and variants in the wild to analyze -- all the more reason why it's not necessary to create new ones. If researchers are having so many problems finding enough viruses to analyze that they're tempted to start creating them, then those researchers aren't doing a proper job of it.

Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior]]> http://www.dslreports.com/forum/remark,16733297 Fri, 18 Aug 2006 14:24:27 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16732085 HMS1 : I don't see any problem with CR having created a bunch of new viruses. Obviously the bad guys can do the same whenever they want (the Bagle guys were cranking out the variations for a while, for example). CR's action did not increase this ability of malicious actors, nor would refraining have restrained the malicious actors.

Retrospective testing is fine for anti-virus. But if you're working on heuristic detection, you may have good reason to write some new malware to test against. There may be some preexisting program that fits the description of what you want, or there may not be, or it may be too much trouble to find one.

There is an underlying contradiction in the rhetoric against virus-writing. If lab-created viruses are ineffective for testing, then they must be harmless when released. If they are harmful when released, then they are good for testing. If the real-world virus writers haven't taken a particular approach, but you can think of it, then they have thought of it too, or will soon. It is contradictory - and suspicious - to maintain that lab-created viruses are so dangerous that they must not be created, yet so unrepresentative that they are no good for crafting defenses.

Statements such as the Avien one and the critques of the CR method make me wary of some sort of initiative to get virus-writing outlawed or restricted by licensing. There are hints too of a lobby for legal limits on instruction about malware writing.

Any such laws would be unjust and harmful. There should never be any prohibition on writing any kind of software, at all, ever (only spreading malware to unwilling parties should be illegal). We should be wary of such self-serving proposals which would impair freedom of speech and make computer users dangerously dependent on a privileged industry.

The right approach is to focus on improving the castle walls, not quibbling about how we study the Huns' weapons.]]> http://www.dslreports.com/forum/remark,16732085 Fri, 18 Aug 2006 11:12:55 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16731980 eburger68 : Funchords:

You wrote:

said by funchords :

What has been conventional wisdom in past years may not necessarily apply to strategies for combatting today's 0-day malware.
Maybe, maybe not. But it's not enough just to suggest in the abstract that conventional wisdom may be wrong -- anyone can do that. The challenge here is to offer concrete, logical reasons why the 0-day malware landscape of today would somehow invalidate the principles agreed upon by the AV industry -- principles based on hard-won experience with malware.

So far I haven't heard any that directly address the arguments offered by experienced AV experts.

Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior]]> http://www.dslreports.com/forum/remark,16731980 Fri, 18 Aug 2006 10:53:32 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16730987 SpannerITWks : eburger68 gave some links, Thanx, which i read, and the following in " " are quotes from them.

Re - Open Letter

" If a product does not report a simulated virus as being infected, it's right. And if a program does report a simulated virus as being infected, it's wrong. Thus, using simulated viruses in a product review inverts the test results. It grossly misrepresents the truth of the matter because:

* It rewards the product that incorrectly reports a non-virus as infected. * It penalizes a product that correctly recognizes the non-virus as not infected.

Competent, credible antivirus product reviewers today recognize the need to reflect the real world in their testing. To do so, they focus detection testing on the real-world threat, using real viruses. They focus on viruses reported by the WildList Organization International. True, some may also include other viruses in testing, but they still use real viruses, not simulated ones. "

»cybersoft.com/whitepapers/papers···er.shtml

So REAL Malware is obviously Totally acceptable for testing purposes ! Even if the vendors don't yet have a sample of it/them, non the less a NEW real as yet undiscovered nasty by them is 100% valid.

-

Joe Wells = The WildList

A Radical New Approach to Virus Scanning

Don't expect this paper to be about a virus problem. To the contrary, it's actually about your having an antivirus problem.

-

" unless otherwise stated, virus scanning specifically refers to methods of detecting known viruses - as, for example, by using signatures.

-

Of course not, the virus problem has been getting progressively worse. What it does mean is that the number of "all known viruses" is far outstripping the number of wild viruses. It means the increase is almost entirely in zoo viruses.

-

What I theorized and IBM proved about trends in DOS file virus extinction effects you directly. Members of an endangered species are becoming increasingly rare in the wild. No wonder nearly all of them are found only in zoos - they simply can't cut it anymore.

-

Zoo detection, in particular a large polymorphic library, is not required for a good certification scheme. Rather, a threat library which determines whether the product is capable of providing protection from all types of self-replicating code should be used.

-

So why are we discussing zoo viruses if they're not a threat? What's the point?

The point is this: There are tens of thousands of viruses you'll never get. There should be no reason for us to discuss them. But we must discuss them. Because, whether you realize it or not, those tens of thousands of viruses do affect you. They do have a direct impact on you. And their effect is detrimental.

-

In the real world, zoo viruses are not a problem. Wild viruses are.

-

But even if you haven't dealt with viruses at all, you don't have to be an expert to intelligently evaluate evidence.

Let me illustrate. Suppose you served on a jury. The trial involves complex medical issues. Would you have to be a neurosurgeon to weigh the evidence presented? Of course not. You would listen to the evidence presented, and evaluate it fairly. You might have to ask for clarifications (I certainly would.), but that does not make you unqualified.

-

Of course, many experts think they are authorities. They actually believe that their expert opinion is more than just opinion-it is truth. Similarly, many people do view experts as being authorities.

( Paraphrasing by me ) Many experts who do testing have doctorate degrees. Others don't have any degrees at all. Who should you believe, others or real experts?

-

Making someone else appear inferior, somehow make you appear superior. "

»vx.netlux.org/lib/ajw01.html

OK so zoo are off limits fine, but appear, or did @ one time, to outnumber real Malware. So why did/do they waste so much time on testing them then ?

" In the real world, zoo viruses are not a problem. Wild viruses are. "

Absolutely, and not just viruses of course, but All forms of Malware. And Real Wild = REAL even if the vendors don't have them yet. If they are out there then the potential for infiltration/infection and/or damage is also Very REAL.

Re the earlier link - Public letter concerning the Writing of Viruses & How it Does Not Teach about Virus Prevention.

" It is not necessary and it is not useful to write computer viruses to learn how to protect against them. " So why bother with writing ALL those zoos ?

-

" cannot know what viruses we are going to face in future "

»www.avertlabs.com/research/blog/?p=71

Exactly !

-

Naturally no AV without heuristics etc is going to going to detect something that is not in it's defs !

I still believe that coding New workable Malware of all various types can definately help the vendors design better Stiffer more resistant products, why wouldn't it ? And it certainly ain't gonna do any harm is it, unlike the Real Malware that's already been released before you read this, and later on today, and tomorrow etc etc, that the vendors don't know about yet, which means neither does your AV hence you !

Spanner
--
I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks
/SpannerITWks]]> http://www.dslreports.com/forum/remark,16730987 Fri, 18 Aug 2006 05:50:02 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16730859 funchords :
said by eburger68 :

have all offered sound reasons -- methodological, practical, and ethical -- for not relying on lab-created viruses for anti-virus testing. The principles here have been rather settled in the AV community for some time.
Fine, but the virus landscape has been recently changing. What has been conventional wisdom in past years may not necessarily apply to strategies for combatting today's 0-day malware.

As an outsider to the AV community, I don't see a major crime here. The magazine's approach seems to be a reasonable one.
--
Robb Topolski -= funchords.com =- Hillsboro, Oregon USA
~ Keeper of the D-Link FAQ ~ Did you Search? ~ More features, Free! Join BBR! ~]]> http://www.dslreports.com/forum/remark,16730859 Fri, 18 Aug 2006 04:11:33 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16730784 SnowyOne : What's done has been done & there's no turning the clock back regardless of whether one agrees or disagrees with the bonehead methodology.
What I'm not sure of is would CR be acting responsibly if they gave every AV/whatever vendor a copy of each & every file they created? If some or all were to go ITW, wouldn't the AV/whatever vendors be in a better position to minimize the damage by already having the definitions covered? On the other hand, the more who have access, the more chance of leaks. Opinions?]]> http://www.dslreports.com/forum/remark,16730784 Fri, 18 Aug 2006 03:35:16 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16730700 eburger68 : Steve:

The "real report" is simply table of ranked apps coupled with basic information (price, company) and 10 columns of bubbles to cover features and test results (bubbles can be empty, partially filled-in, entirely filled-in). The rough rankings were posted in this forum very recently:

»Consumer Reports Best Tools Stop Viruses/Spam/Spyware

The issue is on the newsstands now.

quote:
On one hand, they're going to be the subject-matter experts, which clearly gives them an edge over Consumer Reports, but on the other hand, the industry likely wishes to measure their own products in terms of how they think of the problem. That may or may not be how the consumer looks at it.

McAfee and the other AV industry experts that have been quoted so far in news stories (VirusBulletin, Sophos, and Kaspersky) have all offered sound reasons -- methodological, practical, and ethical -- for not relying on lab-created viruses for anti-virus testing. The principles here have been rather settled in the AV community for some time.

For an elaboration on why the creation of viruses for testing is not only methodologically unsound, but practically unnecessary, and ethically dubious, see this this Open Letter from the AV community (authored by Joe Wells) to CNET:

»cybersoft.com/whitepapers/papers···er.shtml

Note that a number of the letter's signatories are not affiliated with any AV company. And the author of the letter has proven himself quite capable of being a fierce critic of the industry's own practices and habits -- see for example:

»vx.netlux.org/lib/ajw01.html

If one is going to speculate that this griping about CR's AV testing is just the AV industry covering its own backside, then we'll really need just a bit more than the speculation -- the arguments against the creation and use of lab viruses have been on the table for some time now. What would be the arguments in favor of lab viruses? In what way did consumers benefit from Consumer Reports' creation of 5500 new virus variants, none of which was an actual "in the wild" virus?

For those who are wondering, "If not for the use of lab-created viruses, how could researchers test the ability of AV products to handle new and hitherto unknown viruses?", McAfee's complaint contains the answer: retrospective testing, a procedure that is well established and has the advantage of testing against "in the wild" viruses, not lab-created viruses:

»www.avertlabs.com/research/blog/?p=71

Best,

Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior]]> http://www.dslreports.com/forum/remark,16730700 Fri, 18 Aug 2006 03:02:12 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16729192 Steve :
said by Mele20 :

Is that from the article itself? It doesn't sound like a quote with the bad grammar and the repeated words).
The blame for the sloppy syntax is with me - long day, eyes getting blurry. My fault.

Industry self-policing (which is what "industry-standard tests" are) can be good or it can be bad, but one can't forget that they have their own constituency.

On one hand, they're going to be the subject-matter experts, which clearly gives them an edge over Consumer Reports, but on the other hand, the industry likely wishes to measure their own products in terms of how they think of the problem. That may or may not be how the consumer looks at it.

It's certainly not out of the question, however, that CR just did a poor job on this; I look forward to seeing the real report when it arrives too.

Steve
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site]]> http://www.dslreports.com/forum/remark,16729192 Thu, 17 Aug 2006 21:26:45 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16729135 AB :
said by Steve :

. . Consumer Reports has a different constituency than does the A/V industry, and it's not out of the question - in principle - to for one to wonder if if the A/V industry's tests are there to serve the industry and not the consumer.

Steve
I don't have any hard facts to put into evidence here, but as I recall it, the A/V industry stats tend to show an average detection rate of about 80-85%. Not too shabby, yet not exactly something to be boasting about, either.
One might think they would pad those stats up a bit higher if the tests they run were merely of self-serving interest, no?]]> http://www.dslreports.com/forum/remark,16729135 Thu, 17 Aug 2006 21:17:09 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16729089 Mele20 :
said by Steve :

Consumer Reports has a different constituency than does the A/V industry, and it's not out of the question - in principle - to for one to wonder if if the A/V industry's tests are there to serve the industry and not the consumer.

Steve

Is that from the article itself? It doesn't sound like a quote with the bad grammar and the repeated words). I can't read the article or comment here because I am not a subscriber to Consumer Reports. If that is from the article, I don't see any particular objection...but I haven't been able to read the article..just other people's comments about it so I can't really comment intelligently until I am able to read the article when my library gets a copy of the September issue.
--
"If you want to do DRM on a PC then you need to treat the user as the enemy." Ross Anderson in "`Trusted Computing' Frequently Asked Questions"]]> http://www.dslreports.com/forum/remark,16729089 Thu, 17 Aug 2006 21:11:32 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16728996 Steve :
said by Sunbelt's blog :

Publications need to use industry-standardized methods for testing. Organizations like Virus Bulletin have been doing this for years. Why can';t publications follow their lead?
I don't have any opinion about the particular approach taken or conclusions reached, but this statement is one I can take some exception with.

Consumer Reports has a different constituency than does the A/V industry, and it's not out of the question - in principle - for one to wonder if the A/V industry's tests are there to serve the industry and not the consumer.

Steve

Edit - fixed yucky writing (and not even drinking yet!)
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site]]> http://www.dslreports.com/forum/remark,16728996 Thu, 17 Aug 2006 20:59:47 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16728243 AB :
said by SpannerITWks :

. . . You will see how being able to write RK's and dissasemble them etc, and write detectors enables both sides to have a greater understanding of each others tactics etc. Thereby enabling them to design better RK's + detectors.

So i do believe it's definately worthwhile to as much inside knowledge as possible about how the other side Really works, because that IS what they do, every day !
Sure, knowing how the other half lives, what they do, is good and will help people better understand how to fight the malware more effectively. But ya gotta write 5500 NEW variants to do that? I don't think so!
This is a disaster waiting to happen. Let's hope it won't.
And the first variant found in the wild that can be directly linked back to this research, I hope to see one massive class-action lawsuit.
And btw, is 'Consumer Reports' really the organization we want leading this research? While I understand that this is in fact a consumer issue, I'm just not so sure these are the people I want in the vanguard of this somewhat shaky business.]]> http://www.dslreports.com/forum/remark,16728243 Thu, 17 Aug 2006 19:18:46 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16728088 SpannerITWks : Blackbird SR

Sure i get your murder analogy Thanx !

But people might be interested in looking @ this thread - »forum.sysinternals.com/forum_pos···003&PN=1 - to see just how cat + mouse actually works in REAL life.

Yes real life, because in there are Real Rootkit coders with Real RK's that are out there right now being used to hide nasties and being used by 3rd parties for crime. Also in there are various well known RK detector guys n girls combatting those and other RK's.

You will see how being able to write RK's and dissasemble them etc, and write detectors enables both sides to have a greater understanding of each others tactics etc. Thereby enabling them to design better RK's + detectors.

So i do believe it's definately worthwhile to as much inside knowledge as possible about how the other side Really works, because that IS what they do, every day !

Spanner
--
I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks
/SpannerITWks]]> http://www.dslreports.com/forum/remark,16728088 Thu, 17 Aug 2006 18:54:51 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16727994 Cudni : 5,500 "new" viruses released to the wild is not even a fraction scary as just one of the biological viruses stored in both civilian and military labs escaping

Cudni
--
Some are born to failure, others achieve it, all deserve it.
Help yourself so God can help you.
MVP, Microsoft Windows Security 2006]]> http://www.dslreports.com/forum/remark,16727994 Thu, 17 Aug 2006 18:41:34 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16727898 Blackbird : I think what they mean is that CU's constructing new variants from 6 categories of known viruses only shows how various AVs will respond to new, unknown virus variants constructed using the same techniques employed by CU. Those techniques were intended by CU to create large numbers of virus variants based on existing virus structures and ideas... they were not created to exploit new-found security holes nor were they created using novel virus-structure techniques. While CU's variants may be "new", they are not necessarily representative of what many actual virus writers will do in creating their malware in the real world. Until now. Now there are 5,500 'new' viruses on CU's lab computers and some (likely) documented recipes in CU's files of how each was created from existing virus categories - all for the script kiddies and other baddies to sniff out as only they can. And we can all hope and pray that CU's internal data/info security is better than was their reasoning in following such a path in the first place.

Thoroughly understanding viruses and how they are written does not equate to actually writing them. Writing them may or may not make one more expert in combating them. One certainly does not need to commit murder (nor many other things in life and the technical world) to understand how it is done and to combat it.

edit: phrasing in middle of para 1
--
If God wanted us to work with electrons, He'd make them big enough to see...]]> http://www.dslreports.com/forum/remark,16727898 Thu, 17 Aug 2006 18:29:50 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16725929 AB : Quoted from Consumer Reports article:
----------------------------------------------------------------------------------
To be safe from online infection, you need protection from current viruses, which number 100,000 . . (sic) . .

To pit the software against novel threats not identified on signature lists, we created 5,500 new virus variants . . .
----------------------------------------------------------------------------------
Cute. Very nice.
In one fell swoop, they have increased the known virus count by 5.5%. That's an excellent day's work there, Dr. Consumerstein.
And you can practically write it in stone that some of these will soon be finding their way OUT of the lab.
If the disease doesn't kill you, the cure will.

Potential quote from a Consumer Reports spokesperson in a couple of months: "Whoops! Sorry!" ]]> http://www.dslreports.com/forum/remark,16725929 Thu, 17 Aug 2006 14:09:50 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16725494 SpannerITWks : - An open and sincere letter to the AV etc peeps -

I clicked on the avertlabs link - »www.avertlabs.com/research/blog/?p=71 - (you can read an open letter on the AVIEN site about that).

Which gets you to here -

»www.avien.org/publicletter.htm - Public letter concerning the Writing of Viruses & How it Does Not Teach about Virus Prevention. Originally published: May 30th, 2003 Last updated: August 11, 2006 7:14 PM

" The more than 100 signatories of this public letter, all security professionals with years of experience in dealing with computer viruses, and who work in all sectors, wish to express their whole-hearted support of the following principle:

It is not necessary and it is not useful to write computer viruses to learn how to protect against them. "

Signed:

etc -

Among the people signing their names to it are a number of well known figures. Whether ALL of them who originally signed still agree with Everything on there is open to question, but let's say they do for now !

Of course you don't have to be " able " to write nasties to write code to detect them per se. But, i've got a number of nasties in my collection that ALL the vendors listed on Jottis + VirusTotal did NOT detect when i submitted them ? These included Rootkits/Trojans/Exploits/Keyloggers etc.

So how can this be if the signed statement above is Totally correct ? Either they can detect new nasties and variations, or they can't ! And based on my tests they can NOT and did NOT on those occasions.

If they Actually mean detecting whilst being run etc ok. But they do NOT All do that either, whether normally and/or heuristically. If they say don't need to know how to write nasties, and in ALL their variations/conotations, how can they Totally understand and prevent vectors etc being compromised and therfore computers getting infected. If they were 100% right about their claims, then NOBODY would EVER get infected with ANYTHING, but hey guess what, err yes that's right, they DO, and daily with ALL sorts of crap, including brand new stuff and variations.

So what Exactly do they mean when they say " It is not necessary and it is not useful to write computer viruses to learn how to protect against them. " Because if they DO know, they are NOT putting that knowledge into practice ?

Spanner
--
I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks
/SpannerITWks]]> http://www.dslreports.com/forum/remark,16725494 Thu, 17 Aug 2006 13:09:35 EDT Re: Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16725385 KyeU : I hope one day they don't go:

"OH NO! WE CREATED A MONSTER!"]]> http://www.dslreports.com/forum/remark,16725385 Thu, 17 Aug 2006 12:54:30 EDT Our unique antivirus testing: How we did it http://www.dslreports.com/forum/remark,16725030 Cudni : from
»www.consumerreports.org/cro/elec···ting.htm
"...
To pit the software against novel threats not identified on signature lists, we created 5,500 new virus variants derived from six categories of known viruses, the kind you’d most likely encounter in real life.
.."

not everybody thinks that was/is good idea
»sunbeltblog.blogspot.com/
»www.avertlabs.com/research/blog/?p=71

Cudni
--
Some are born to failure, others achieve it, all deserve it.
Help yourself so God can help you.
MVP, Microsoft Windows Security 2006]]> http://www.dslreports.com/forum/remark,16725030 Thu, 17 Aug 2006 12:03:26 EDT