how-to block ads
|
SpannerITWks
Premium
join:2005-04-22
3 edits | reply to Cudni
Re: Our unique antivirus testing: How we did it
Quote bluezanetti -
" However, it is just as easily argued that no matter how controlled their testbed, there's no assurance the test sampling bears any resemblence to emerging malware threats at play today due to the very dynamic nature of the challenge. "
And it's just as easily debated that it does ! Cos you n me both + everybody else don't know do we, it's just speculation after all. If out of 5500 brand new nasties they havn't written some super duper stuff that really challenges AV's, then yes it would be a bit limp, but it's to be hoped they did. We might find out sooner rather than later as the " noise " increases from various sectors. They could open up their secret Treasure Trove to Trusted peeps for evaluation, that would sort it one way or the other, or maybe even inbetween !
I suggest that a group of Interested parties could be invited to bring their testing laptops to a pre determined SECURE location, and under the watchfull eyes of a number of agreed by all parties peeps, conduct their own tests ALL at once ! Then publish the results either as a joint release, and/or individually for all to see. If people Really want it to happen it can and will, so start making connections and make it happen, then they will know and so will we.
-
As a general observation, comments made towards somebody/thing etc like this for eg " Consumer Reports, better known for reviewing cars, lawn-mowers and appliances " only seem to be posted to demean them in some way/s. Just because they test those things, they also test a range of other things too, and why shouldn't they be ( allowed ) to if they want to. Anybody can test whatever they like, there's no law against it, and as long as the tests are competent and the results are fair, and it's beneficial in some way/s to the users, so what !
For one eg -
I've thrown just every FW test there is going at my FW + Apps, and published the results for all to see, more than once. People were and are free to challenge them and my methods etc, and ask questions etc, which they did, and i was happy to respond. Happy in more ways than 1 too, as i pass 99% of them on my PC. Now am i or others to be disbelieved over tests they do such as those, just because we havn't been given the title " experts " or call ourselves that ?
Spanner
edit typo Only
--
I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks
/SpannerITWks | |
AB
Premium
join:2006-04-04
Leesburg, VA
| said by SpannerITWks  :. . Anybody can test whatever they like, ther's no law against it, and as long as the tests are competent and the results are fair, and it's beneficial in some way/s to the users, so what ! . . just because we havn't been given the title " experts " or call ourselves that ? Spanner Hey, I think I'm starting to see the light here!
All those pimply-faced 17-year-old Russian scriptkiddies are just 'testers', testing what they like! They certainly don't have the title "experts", do they? And it's ultimately beneficial to the community because other so-called 'experts' get to play around with their handiwork to find out how to stop it. Yes! We've been dogging these people, when in reality, we owe them our deepest gratitude!
Please, allow me to be the first--
Thank you, pimply-faced Russian scriptkiddies, for helping to make the computing world a better, safer place! | |
bluezanetti
Premium
join:2003-10-04
| reply to SpannerITWks
said by SpannerITWks :
And it's just as easily debated that it does ! Cos you n me both + everybody else don't know do we, it's just speculation after all.
Quite true. Of course, had CR followed the protocol noted above, debate on this point would be moot, albeit replaced by other points of possible contention.
My overriding point is that the route pursued by CR is one that could inject unintentional bias into the test. Who knows if measures were taken to minimize this eventuality, I certainly have no insight into that point.
Blue | |
funchords
Hello
Premium,MVM
join:2001-03-11
Washington, DC
 ·Verizon Online DSL
 ·Skype
| reply to alexeck
said by alexeck :CR chose to ignore a vast corpus of research, debate and analysis by the academic and security research community. Hey Alex!
Please provide any evidence that they made a choice. Otherwise, the best we can assume is that they took an independent approach without knowledge of the "vast corpus."
--Robb (formerly of Q.O.S.)  --wonk!--
--
Robb Topolski -= funchords.com =- Hillsboro, Oregon USA
~ Keeper of the D-Link FAQ ~ Did you Search? ~ More features, Free! Join BBR! ~ | |
funchords
Hello
Premium,MVM
join:2001-03-11
Washington, DC
·Verizon Online DSL
·Skype
| reply to alexeck
said by alexeck :We need standardized testing for all types of security products. This debate should be done in a reasoned, scientific fashion, with broad representation in the community and industry to come out with a clear, comprehensive method of testing. That is the only real way to serve the consumer. No, it's not.
As a professional software tester (not of security products), allow me to both agree and disagree.
While agreed-on standards for testing are useful, they are also the minimum and they quickly decrease in value over time. You better serve the customer by doing more than just the (minimum) standard testing.
It's an established principle in software testing that one continuously re-examines ones approach, re-evaluates the effectiveness of the testing, creatively explores for additional information not available from the everyday "comprehensive" suite, and adjusts to the current and future environment.
--
Robb Topolski -= funchords.com =- Hillsboro, Oregon USA
~ Keeper of the D-Link FAQ ~ Did you Search? ~ More features, Free! Join BBR! ~ | |
funchords
Hello
Premium,MVM
join:2001-03-11
Washington, DC
·Verizon Online DSL
·Skype
| reply to Steve
said by Steve :It may well be that CR committed the crime in question, but the A/V industry is doing a terrible job in the witness stand. Agreed. I have no beef with the AV industry. So far, McAfee (which ranks about center in the AV Comparitors list) hasn't failed me -- and I'm a rough customer. To me, anyway, mediocracy seems to be pretty damn good. That speaks well of the Industry.
"The industry" (whoever they are) has judged that Consumer Reports made a mistake. Fine. They may even choose to ignore CRs methods and findings. Fine. They may even write a letter to the editor explaining why. Fine. I think that all of those conclusions and actions are rational.
The level of protesting seems too shrill. By getting high-and-mighty, the industry has lowered itself a bit in my eyes.
--
Robb Topolski -= funchords.com =- Hillsboro, Oregon USA
~ Keeper of the D-Link FAQ ~ Did you Search? ~ More features, Free! Join BBR! ~ | |
alexeck
join:2004-12-20
Clearwater, FL
1 edit | reply to funchords
said by funchords :
Please provide any evidence that they made a choice. Otherwise, the best we can assume is that they took an independent approach without knowledge of the "vast corpus."
Well, they could start with Virus Bulletin, which has been doing this for many years. »www.virusbtn.com/ | |
alexeck
join:2004-12-20
Clearwater, FL
| reply to funchords
said by funchords :
While agreed-on standards for testing are useful, they are also the minimum and they quickly decrease in value over time. You better serve the customer by doing more than just the (minimum) standard testing.
Sure, that's fine. Just don't create viruses. | |
Steve
I'm a PC, so shut up
Consultant
join:2001-03-10
Yorba Linda, CA
| said by alexeck : Sure, that's fine. Just don't create viruses. This is one of those sentiments that sounds good, and ought to be applied generally, but is not the kind of universal prohibition which admits no exceptions in any case; I don't see the need for the hysteria over this.
Steve
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site | |
SpannerITWks
Premium
join:2005-04-22 | reply to Cudni
How about Trojans + Rootkits etc then, is that OK lol ?
Spanner | |
Steve
I'm a PC, so shut up
Consultant
join:2001-03-10
Yorba Linda, CA
| said by SpannerITWks :How about Trojans + Rootkits etc then, is that OK lol ? I assume you were replying to me in spite of your Topic Reply.
I don't have any problem creating any software on a test basis for legitimate research and testing purposes as long as one takes precautions that they do not leak. It's harder to do this than it looks, but it's not beyond the ability of mankind to get this right.
Their badness comes from the harm they do to others, not to some inherent badness of the bits themselves.
Steve
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site | |
SpannerITWks
Premium
join:2005-04-22
| No Stevie baby, i was responding to alexeck " Just don't create viruses "
I assumed he did actually mean ALL malware though, as i mentioned earlier on in the thread.
Your right, hysteria won't help anybody, especially the users !
Spanner
--
I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks
/SpannerITWks | |
IBK
join:2003-06-20
Austria
| reply to Cudni
Hi,
the following is only my opinion as, nothing more:
I think the main point on why CR is currently in the spotlight due what they did is not due the high risk their samples could pose to real world. The main points (I refer now to the points of mcafee blog because i think they were the first to note it) are that 1. the variants they created are NOT variants that you will encounter in real life (its long ago since a saw a scriptkiddie variant of e.g. loveletter and not what goes in daily [much more new malware appears, not so silly varaints], 2. true, writing viruses is generally considered not a good idea (but the AVIEN letter was an example, about a topic were an university wanted to teach how to write viruses to its students to teach how to protect against them - all students failed in real world about this, as no one is working in any av related job), 3. testing methods to measure how good or bad av software is at detecting new malware are discussed since long time and since some years (6?) the retrospective method is considered to give the best real-world results (and that's true, if the test is done accuratly and a bit perfectionated to avoid some influences). It is known that AV-Test.org (Andreas Marx) does retrospective tests and publishes the results in many magazines. And it should be also known that AV-Comparatives does such testing publicly available (still for free - that's more user-friendly). So I can only think that they wanted to make something spectacular, but failed in doing some research about the topic before they acted (well, it is probably also the fault of the peoples they engaged to do the test. They are most probably very good about other security tests in enviroments they provide, but probably not very informed about antivirus testing). The point 4 in the mcafee blog is (for those that did not noticied it) a sarcastic phrase (see  ).
Conclusion: there was no need to create that virus variants, as the test based on these self-made variants do not show/tell to the user how good AVs are in detecting new viruses. It only tells about how much of the self-made files created for testing - and that you will never encounter - were detected, making all the test senseless and not useful for anyone. CR will not write that in their article, but even if they would state that, most readers would anyway get to their own conclusion and believe in the printed scores.
AV vendors (also those that scored top) are imo very sad/upset that magazines still make home-made invalid tests and deliver to users wrong information (what happenend since long time and still happens in some magazines) instead of e.g. asking independent organizations like av-test.org, virusbtn, icsa, wcl etc. in helping doing the tests (or performing the tests for them). I do not list av-comparatives because as I publish the results up-to-date for free on the website to the users, I do not think that anyone would want to wait for several months for seeing it published in some magazine when it will be already outdated (usually [but not always] most tests in magazines are at least already some months old). | |
SpannerITWks
Premium
join:2005-04-22
| IBK
Re the Mcafee blog
1 - " It is claimed that created viruses were the kind you’d most likely encounter in real life which is, of course, something the testers cannot know. "
( And something Igor Muttik or anybody outside of the inner sanctum can't know, as they don't know if they don't have access to them )
3 - 4 - ( Already covered those )
Re Conclusion
" there was no need to create that virus variants, as the test based on these self-made variants do not show/tell to the user how good AVs are in detecting new viruses. It only tells about how much of the self-made files created for testing - and that you will never encounter - were detected "
( Why doesn't it help users to determine if an AV can detect, or not new Malware. As i've already said, if it's new then it IS new, and if it could do damage, no matter how small, it's still unwanted and would need sorting. How can anyone say that ALL 5500 were crap, they might be, but we don't know do we, yet ! How can it be stated 100% that nobody will EVER encounter Anything similar.
Why do people deferentiate between those 5500 in a closed lab, and scriddies or worse coding something equally, let's say crap but still capable of damage, or something not crap but lethal ! Why does it matter where they were created, and by whom. If it's new and can do damage then it's fair game for testing AV's detection capabilities.
Any chance of putting a bit more white inbetween the black in future, thanx. )
Spanner
--
I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks
/SpannerITWks | |
EGeezer
Go Bobcats
Premium
join:2002-08-04
Country!
·Callcentric
 ·RoadRunner Cable
| reply to Cudni
SANS NewsBite article
Since no one has specifically reviewed the variants created by the CU testers, we really don't know whether or not the variants would be typical of the dozens that are created daily by the "production" malware coders(Per David Emm of Kapersky, quoted in one of the SANS linked articles, Kapersky adds over 200 signatures a day).
However, I'd guess they have sufficient history and have retained the expertise to extrapolate and create reasonable variations in a well secured environment. Until credible experts come up with analyses of the CU variants that discredit CU's tests, I'll give CU the benefit of the doubt based on their past history of providing accurate testing and successful defence of challenges.
That being said, Here's a note from SANS;
said by SANS Newsletter and editorial : --Consumer Reports Creates 5,500 Viruses For Tests (16 August 2006) Consumer Reports is under fire from the anti-virus community for sponsoring the creation of 5,500 new viruses to test anti-virus products. Zone Alarm Internet Security Suite scored high in the test for both virus and spyware. Spybot Search and Destroy scored well for spyware. » www.computerworld.com/action/art···_topic17» cbs4boston.com/consumer/local_st···410.htmlSpecial Tip: A great discussion on Microsoft Office security and vulnerabilities has been posted on SecurityFocus: » www.securityfocus.com/infocus/1874[Editor's Note (Paller): This controversy is especially problematic for the leading AV companies because they have traditionally not done well in finding and blocking new viruses quickly. But for goodness sakes, if they don't do well at finding and blocking new viruses, why ae we buying them? They should stop complaining and instead thank Jeff Fox and the editors at Consumer Reports for helping to do important product improvement research for them.
--
This space for rent | |
t2contra
join:2002-05-27 | reply to Cudni
Re: Our unique antivirus testing: How we did it
Can someone post the test results? | |
Steve
I'm a PC, so shut up
Consultant
join:2001-03-10
Yorba Linda, CA
| reply to Cudni
The more I read about this, the less I think about these whiney A/V vendors.said by McAfee/Avert blog :It is claimed that created viruses were “the kind you’d most likely encounter in real life” which is, of course, something the testers cannot know. Well how about that: calling CR to tasks for not having a crystal ball.
The "standard technique" of retrospective testing is actually not so bad: turn off updates, and a month or two later find out which new viruses the A/V picks up. This is really a great test that satisfies most of what matters using undeniably real-world examples.
But this all smells like the security precautions taken in the "war on terror", which is fighting yesterday's threats. It's not bothering me at all that somebody tried a different approach.
Steve
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site | |
eburger68
Premium,MVM
join:2001-04-28
2 edits | reply to Cudni
Hi All:
Rather than start a new post for each of the several interesting new responses that arrived since I last checked in, let me combine a responses of my own into one post.
funchords, you wrote:
quote:
Please provide any evidence that they made a choice. Otherwise, the best we can assume is that they took an independent approach without knowledge of the "vast corpus."
Fair enough, but I don't see that ignorance is any better in this situation than single-minded stubbornness or hubris. Even if they didn't know, they should have known -- and one of the most important reasons for this is provided by you in a later post...
funchords, you wrote:
quote:
It's an established principle in software testing that one continuously re-examines ones approach, re-evaluates the effectiveness of the testing, creatively explores for additional information not available from the everyday "comprehensive" suite, and adjusts to the current and future environment.
This is a fine principle in the abstract. Problem is, you've already admitted that CR was, at best, ignorant of the prior history and established methodologies of AV testing. One can't re-examine, re-evaluate and creatively explore if one isn't starting from an historically informed base of knowledge. IF CR simply struck out on their own without any knowledge of previously established methodologies and lessons of AV testing, then there's little to be gained from this exercise unless you want the AV industry itself to take up the practice of lab virus creation. For the potential consequences of that, see my earlier posts.
funchords, you wrote:
quote:
The level of protesting seems too shrill. By getting high-and-mighty, the industry has lowered itself a bit in my eyes.
So send the head of Virus Bulletin an etiquette book.
However large a presence the AV industry may have in specialist forums like these, the AV industry's presence among the general population (beyond that flashing tray icon on people's computers) is next to nothing compared to Consumer Reports, which is currently blasting these test results into the homes of millions of Americans, the vast majority of whom will, even after this noisy controversy, have little idea that a group of veteran AV experts thinks these tests have signficant problems. All the general public will see is "newsy" little items like this:
»news10now.com/content/features/t···SecID=97
To get its message out in the face of that kind of massive publicity, the AV industry is going to have to shout to be heard by the mainstream news media.
Steve, you wrote:
quote:
This is one of those sentiments that sounds good, and ought to be applied generally, but is not the kind of universal prohibition which admits no exceptions in any case; I don't see the need for the hysteria over this.
C'mon, Steve. CR's critics have offered plenty of sound reasons for rejecting CR's course of action -- even you've allowed as much. And although you might be less than impressed by the ethical implications that CR's critic see here, those ethical objections aren't "hysteria" -- they're grounded in the experience and knowledge of the critics.
Moreover, an admonition to allow for exceptions is a nice bit of advice in the abstract, but the burden for establishing the justification for a particular exception in this case falls on those who would advocate for the exception. So far I haven't seen anything that comes close to establishing the need for an exception in this case. I've heard plenty of noise about the "high-handed," "whiney" behavior of the AV industry, plenty of worshipful paeans to the inerrancy of CR, but next to nothing in the way of response to the specific points made by Joe Wells (in several quoted/cited texts), IBK, or bluezanetti as to why lab viruses are a methodologically unsound and even counterproductive means of testing.
I'm always willing to entertain exceptions to the rule, but ultimately the burden is on those advocating the exception.
Steve, you wrote:
quote:
Their badness comes from the harm they do to others, not to some inherent badness of the bits themselves.
(Warning: slightly OT) - This is one of the more tiresome shibboleths of tech-geek culture -- the inherent "neutrality" of technology -- one that really should have been retired years ago, as it offers almost zero insight into the problems and challenges that software programs and other technologies pose to real human environments, which is where we always encounter the bits. The "inherent" quality (or non-quality) of the "bits themselves" (whatever that might be) is of zero interest or relevance because we never encounter or deal with only "the bits themselves." One might as well say that the badness that criminals do comes from the harm they do others, not to some inherent badness of the cells themselves that make up the criminal. Technology is never "neutral." It may have multiple, complex consequences when introduced into a particular environment, but it is never neutral because technologies always lend themselves to particular uses, not all uses, and affect the surrounding environment in particular ways, not all possible ways.
SpannerITWks, you wrote:
quote:
1 - " It is claimed that created viruses were the kind you’d most likely encounter in real life which is, of course, something the testers cannot know. "
( And something Igor Muttik or anybody outside of the inner sanctum can't know, as they don't know if they don't have access to them )
And that is not an adequate response, because it is not the burden of those of us outside of the "inner sanctum" to establish that the viruses aren't the kind that users would most likely encounter in real life. Far from it. It is the burden of those inside the inner sanctum (at present CR itself and the few advisors they hired) -- those who made the claim in the first place, and who have thus far offered nothing in the way of evidence to back up that claim.
EGeezer, you wrote:
quote:
Since no one has specifically reviewed the variants created by the CU testers, we really don't know whether or not the variants would be typical of the dozens that are created daily by the "production" malware coders(Per David Emm of Kapersky, quoted in one of the SANS linked articles, Kapersky adds over 200 signatures a day).
Yes, exactly...
quote:
However, I'd guess they have sufficient history and have retained the expertise to extrapolate and create reasonable variations in a well secured environment. Until credible experts come up with analyses of the CU variants that discredit CU's tests, I'll give CU the benefit of the doubt based on their past history of providing accurate testing and successful defence of challenges.
Errrmmm, but this doesn't follow, because the burden of proof is entirely backwards here. Based on your trust (faith?) in CR, you've essentially thrust the burden of proof on the critics of the test, not the authors of the test, which is where the burden of proof properly lies. Still worse, you've put the critics into a potentially impossible bind -- if CR refuses to release the necessary data to allow critics to prod and poke, then those critics will have been denied the ability to establish their case. That's a rather convenient "heads-I-win, tails-you-lose" proposition for CR to be offering its critics.
No, the burden is on CR to establish the validity and meaningfulness of its testing, not on its critics to disprove it. If CR fails to do the minimal amount of work to establish the validity and meaningfulness of its testing, then the testing is invalid and has no meaning, and your faith or trust in CR should play no role in your evaluation of that testing.
Steve, you wrote:
quote:
But this all smells like the security precautions taken in the "war on terror", which is fighting yesterday's threats. It's not bothering me at all that somebody tried a different approach.
I don't think the AV industry is claiming that retrospective testing is sufficient to the job of anticipating tomorrow's threats and planning for them. Most respectable AV companies that I know of (Kaspersky, for example) have entire teams of researchers dedicated to anticipating what tomorrow's threats might look like. However mistaken they might ultimately prove to be, I fail to see how modifying existing threats from today to create new variants of those threats advances the cause of anticipating tomorrow's threats either.
Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior | |
SnowyOne
Premium
join:2003-04-05
Kailua, HI
·RoadRunner Cable
 ·Clearwire Wireless
| reply to Steve
said by Steve :The "standard technique" of retrospective testing is actually not so bad: turn off updates, and a month or two later find out which new viruses the A/V picks up. This is really a great test that satisfies most of what matters using undeniably real-world examples. I'll take a little walk on the wild side here.
I see a glaring problem with the "standard technique".
There's nothing in the world that could stop me from including "close enough" definitions in todays update of virus that I won't put ITW until 2 months down the road just to bolster my AV standings in tests using the "standard technique".
I'm not saying that activity does or doesn't happen. I'm saying it can be manipulated which doesn't say much for the staus quo. | |
eburger68
Premium,MVM
join:2001-04-28
| SnowyOne:
You wrote:
said by SnowyOne :I see a glaring problem with the "standard technique". There's nothing in the world that could stop me from including "close enough" definitions in todays update of virus that I won't put ITW until 2 months down the road just to bolster my AV standings in tests using the "standard technique". I'm not saying that activity does or doesn't happen. I'm saying it can be manipulated which doesn't say much for the staus quo. I'm sorry, but I don't follow any of the logic here.
1) What are "close enough" defintions, and how do they differ from actual defintions? Have you ever encountered any "close enough" definitions from an AV vendor that you could describe for us?
2) Why do think an AV company would even consider withholding "close enough" (ITW) definitions until two months down the road?
3) What makes you think that an AV company could ensure that its "custom" defs with the nifty "close-enough" special sauce would be picked up and used by the testing entities in lieu of defintions publicly available from the AV company?
In short, this scenario relies on too many questionable assumptions to be considered a valid objection to retrospective testing.
Best,
Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior | |
|
