SnowyOne
Premium
join:2003-04-05
Kailua, HI | reply to alexeck
Re: Our unique antivirus testing: How we did it
I believe this is the correct link
»sunbeltblog.blogspot.com/ |
|
SpannerITWks
Premium
join:2005-04-22
1 edit | reply to Cudni
Alex Eckelberry
Bit of an oops on your own link lol.
Well this is a completely different topic, as this thread is called - Our unique antivirus testing: How we did it - Not - Our unique antispyware testing: How we did it -
" In addition to antivirus programs, Consumer Reports tested antispyware applications. "
But anyways, i see where you're coming from, as well as FL, lucky you !
" And even more surprisingly, even though Consumer Reports used the Spycar testing methodology, they never even contacted the authors of Spycar for advice or feedback. "
Who says they, or anybody else Have to, it's not a legal requirement ? Maybe it should be from now on though lol. As long as the testers remain independent from ANY final decision making, then communicating with the Test files authors, might be acceptable, as long as this IS clearly stated within the article, and about Exactly what info was exchanged !
" So, Consumer Reports
a) Ignored the instructions of the Spycar authors and used the simulator as the sole method of testing.
b) Ignored the instructions by the Spycar authors to not use Spycar to test scan and remove functionality. "
That's different, in This case, but i wouldn't just advocate blindly Obeying + accepting what someone said, Whoever they are, just because " they " said ! But i agree, about these particular AS tests, hardly ANYwhere near thorough @ all. Useful as an extra series of tests to compliment a much more demanding batch, as quite a number of those SpyCar tests can actually get through onto many peoples PC's.
At least they responded to you, and next time they test Anti's, somehow i think things will be a lot different from the last batch !
Spanner
edit typo Only
--
I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks
/SpannerITWks |
|
Steve
I'm a PC, so shut up
Consultant
join:2001-03-10
Yorba Linda, CA
| said by SpannerITWks  : Who says they, or anybody else Have to, it's not a legal requirement ? There are two kinds of "explicit-instructions" that one might ignore:
1) For-their-own-good instructions, such as those attempting to keep you from selling their product or using it in a published benchmark. EULAs are mostly about for-their-own-good instructions.
2) For-our-own-good instructions, such as a limitation of how much information one actually can get from using Spycar in this manner. Prescription drugs and power tools have lots of for-our-own-good instructions.
Instructions of the #1 type can usually be ignored without much consequence, but #2 can only be ignored if one really knows what one is doing.
I'll leave it as an exercise to the reader as to which is likely to apply in this case.
Steve
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site |
|
SpannerITWks
Premium
join:2005-04-22
| reply to Cudni
Steve
Sure, but who should decide, the notifiers or the testers !
It may have been instructive to seek out info etc from a variety of external sources, including SpyCar, but the decisions, rightly or wrongly, Must solely rest with the testers, Every time !
Spanner
--
I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks
/SpannerITWks |
|
Steve
I'm a PC, so shut up
Consultant
join:2001-03-10
Yorba Linda, CA
| said by SpannerITWks :but the decisions, rightly or wrongly, Must solely rest with the testers, Every time ! Well duh - I think everybody agrees with that much.
We're not talking about whether Consumer Reports should go to jail for "violating explicit instructions", but whether their results should be taken seriously or not.
Steve
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site |
|
SpannerITWks
Premium
join:2005-04-22
1 edit | Steve
Ah well duh, that's where i take issue with phrases like " violating explicit instructions " etc.
It makes it sound like an Order from " them " ! I presume you didn't Actually mean it as such, but it does sound a bit draconian when stated like that.
Spanner
edit typo Only
--
I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks
/SpannerITWks |
|
SpannerITWks
Premium
join:2005-04-22
| reply to Cudni
Found this via a link on - »sunbeltblog.blogspot.com/
-
The AV Doth Protest Too much (Consumer Reports)
" At XXXXX we have a few honeypot boxes that we use to capture malware that is actually in the wild (none of this we found it in our lab). We then run it through an engine that uses 27 different AV products to try and identify the malware. The results obviously vary but out of the 27 it is common to only have 2 or 3 products actually identify the code.
It seems clear that catching old malware is easy and catching new malware is hard, even new malware that is a slight variation on old.
So the efficacy of current AV must be proportional to the churn rate of malware. The faster virus writers are able to make modifications, the more likely they are to be successful. "
»www.matasano.com/log/433/the-av-···reports/
Also found this Very illuminating article on there about the much appaulded " by some " Retrospective testing.
-
Ignore Igor Muttik’s Retrospective Antivirus Testing Method
-
You can make up your own mind about whether a virus born out of modifications to an existing virus is a more serious threat than any of the thousands of historical curiousities and QA test lab anomalies that get replayed during a retrospective test. Personally, I look at the genealogy of other forms of malware - shellcode, bots, worms, and exploit tools - and I notice that the most malicious attackers tend not to write things from scratch, and I think the ISE guys can make a good case for having designed the most relevant test in the industry.
Etc -
»www.matasano.com/log/category/malware/
Spanner
--
I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks
/SpannerITWks |
|
eburger68
Premium,MVM
join:2001-04-28
1 edit | SpannITWks:
You wrote:
said by SpannerITWks :You can make up your own mind about whether a virus born out of modifications to an existing virus is a more serious threat than any of the thousands of historical curiousities and QA test lab anomalies that get replayed during a retrospective test. I'm thinking there's some confusion here over retrospective testing -- at least not as it's practiced by reputable, independent AV testing entities.
Proper retrosptective testing does not test against:
a) "historical curiosities": by defintion, the threats included in a retrospective test are NEWER than than definitions/sigs being tested against. Moreover, they're usually selected by from the Wild List, which ensures that they are current, reasonably prevalent, and actually in the wild.
b)"QA test lab anomalies": again, proper retrospective testing uses samples selected from the Wild List -- meaning that they are in the wild and reasonably prevalent.
Indeed, the entire purpose of the Wild List is to encourage and pressure testers to test against real threats that are current, prevalent, and in the wild, NOT against "historical curiosities" and "QA test lab anomalies" -- those are the very enemies of the Wild List, the kinds of things that testers were often using before the advent of the Wild List.
If you're worried about testing against "QA test lab anomalies," your efforts would be better directed to protesting the use of lab viruses that no independent expert has validated and that have never been in the wild. Those are the epitome of "QA test lab anomalies."
Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior |
|
SpannerITWks
Premium
join:2005-04-22
1 edit | eburger68
Yes i did wrote what i did wrote ! I take full resposibility for posting what i did, even though i was Obviously quoting from the links i provided, which i'm sure people including yourself must realise !
Well the folks over @ - www.matasano.com - seem to have different views on several matters, including Variants + Retrospective testing. They don't appear to be fresh out of dipers to me anyway !
So whos data + info etc are we now all expected to accept as the gospel as far as testing is concerned ? It's not that straightforward anymore, and even if the majority of those 5500 Variants turn out to be not much to crow about, it has certainly opened up a giant can of worms.
I don't think things will be the same again from now on, in many ways. But ya know what, i believe in the long run it will have been a good all round shake up for everyone, and ultimately be of service to users. Might make it harder for vendors, but hey so what, it's the users that want + have a right to expect the best possible + effective products, and it's they who pay for it after all !
Spanner
edit - By the way i watched your video, the one with the smoked tuna fish sandwiches in, nice looking set up you have there in FL !
--
I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks
/SpannerITWks |
|
gourbi
@85.195.x.x
| reply to Cudni
Re: Our pathetic antivirus testing: How we screwed it up
Several years ago a guy called Rodzilla launched an attack that hammered the mighty CNet into submission after a couple of its wannabee virus experts created a few new virus variants for their worthless anti-virus program tests.
Creating 5,500 new virus variants is several orders of magnitude more stupid and worthless.
Consumer Reports needs a brain transplant. |
|
ghost16825
Use security metrics
Premium
join:2003-08-26
| said by gourbi :
Several years ago a guy called Rodzilla launched an attack that hammered the mighty CNet into submission after a couple of its wannabee virus experts created a few new virus variants for their worthless anti-virus program tests.
Creating 5,500 new virus variants is several orders of magnitude more stupid and worthless.
Consumer Reports needs a brain transplant.
I fail to see the connection, unless you're implying that the number of created viruses is directly related to the severity of the DDoS attack. (...and you believe this presents more of a global threat than the viruses themselves)
--
The previous signature has been removed due to recent and continuing website "ownership" issues. |
|
