ghost16825
Use security metrics
Premium
join:2003-08-26
| reply to gourbi
Re: Our pathetic antivirus testing: How we screwed it up
said by gourbi :
Several years ago a guy called Rodzilla launched an attack that hammered the mighty CNet into submission after a couple of its wannabee virus experts created a few new virus variants for their worthless anti-virus program tests.
Creating 5,500 new virus variants is several orders of magnitude more stupid and worthless.
Consumer Reports needs a brain transplant.
I fail to see the connection, unless you're implying that the number of created viruses is directly related to the severity of the DDoS attack. (...and you believe this presents more of a global threat than the viruses themselves)
--
The previous signature has been removed due to recent and continuing website "ownership" issues. |
|
gourbi
@85.195.x.x
| reply to Cudni
Several years ago a guy called Rodzilla launched an attack that hammered the mighty CNet into submission after a couple of its wannabee virus experts created a few new virus variants for their worthless anti-virus program tests.
Creating 5,500 new virus variants is several orders of magnitude more stupid and worthless.
Consumer Reports needs a brain transplant. |
|
SpannerITWks
Premium
join:2005-04-22
1 edit | reply to eburger68
Re: Our unique antivirus testing: How we did it
eburger68
Yes i did wrote what i did wrote ! I take full resposibility for posting what i did, even though i was Obviously quoting from the links i provided, which i'm sure people including yourself must realise !
Well the folks over @ - www.matasano.com - seem to have different views on several matters, including Variants + Retrospective testing. They don't appear to be fresh out of dipers to me anyway !
So whos data + info etc are we now all expected to accept as the gospel as far as testing is concerned ? It's not that straightforward anymore, and even if the majority of those 5500 Variants turn out to be not much to crow about, it has certainly opened up a giant can of worms.
I don't think things will be the same again from now on, in many ways. But ya know what, i believe in the long run it will have been a good all round shake up for everyone, and ultimately be of service to users. Might make it harder for vendors, but hey so what, it's the users that want + have a right to expect the best possible + effective products, and it's they who pay for it after all !
Spanner
edit - By the way i watched your video, the one with the smoked tuna fish sandwiches in, nice looking set up you have there in FL !
--
I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks
/SpannerITWks |
|
eburger68
Premium,MVM
join:2001-04-28
1 edit | reply to SpannerITWks
SpannITWks:
You wrote:
said by SpannerITWks  :You can make up your own mind about whether a virus born out of modifications to an existing virus is a more serious threat than any of the thousands of historical curiousities and QA test lab anomalies that get replayed during a retrospective test. I'm thinking there's some confusion here over retrospective testing -- at least not as it's practiced by reputable, independent AV testing entities.
Proper retrosptective testing does not test against:
a) "historical curiosities": by defintion, the threats included in a retrospective test are NEWER than than definitions/sigs being tested against. Moreover, they're usually selected by from the Wild List, which ensures that they are current, reasonably prevalent, and actually in the wild.
b)"QA test lab anomalies": again, proper retrospective testing uses samples selected from the Wild List -- meaning that they are in the wild and reasonably prevalent.
Indeed, the entire purpose of the Wild List is to encourage and pressure testers to test against real threats that are current, prevalent, and in the wild, NOT against "historical curiosities" and "QA test lab anomalies" -- those are the very enemies of the Wild List, the kinds of things that testers were often using before the advent of the Wild List.
If you're worried about testing against "QA test lab anomalies," your efforts would be better directed to protesting the use of lab viruses that no independent expert has validated and that have never been in the wild. Those are the epitome of "QA test lab anomalies."
Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior |
|
SpannerITWks
Premium
join:2005-04-22
| reply to Cudni
Found this via a link on - »sunbeltblog.blogspot.com/
-
The AV Doth Protest Too much (Consumer Reports)
" At XXXXX we have a few honeypot boxes that we use to capture malware that is actually in the wild (none of this we found it in our lab). We then run it through an engine that uses 27 different AV products to try and identify the malware. The results obviously vary but out of the 27 it is common to only have 2 or 3 products actually identify the code.
It seems clear that catching old malware is easy and catching new malware is hard, even new malware that is a slight variation on old.
So the efficacy of current AV must be proportional to the churn rate of malware. The faster virus writers are able to make modifications, the more likely they are to be successful. "
»www.matasano.com/log/433/the-av-···reports/
Also found this Very illuminating article on there about the much appaulded " by some " Retrospective testing.
-
Ignore Igor Muttik’s Retrospective Antivirus Testing Method
-
You can make up your own mind about whether a virus born out of modifications to an existing virus is a more serious threat than any of the thousands of historical curiousities and QA test lab anomalies that get replayed during a retrospective test. Personally, I look at the genealogy of other forms of malware - shellcode, bots, worms, and exploit tools - and I notice that the most malicious attackers tend not to write things from scratch, and I think the ISE guys can make a good case for having designed the most relevant test in the industry.
Etc -
»www.matasano.com/log/category/malware/
Spanner
--
I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks
/SpannerITWks |
|
SpannerITWks
Premium
join:2005-04-22
1 edit | reply to Steve
Steve
Ah well duh, that's where i take issue with phrases like " violating explicit instructions " etc.
It makes it sound like an Order from " them " ! I presume you didn't Actually mean it as such, but it does sound a bit draconian when stated like that.
Spanner
edit typo Only
--
I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks
/SpannerITWks |
|
Steve
I'm a PC, so shut up
Consultant
join:2001-03-10
Yorba Linda, CA
| reply to SpannerITWks
said by SpannerITWks :but the decisions, rightly or wrongly, Must solely rest with the testers, Every time ! Well duh - I think everybody agrees with that much.
We're not talking about whether Consumer Reports should go to jail for "violating explicit instructions", but whether their results should be taken seriously or not.
Steve
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site |
|
SpannerITWks
Premium
join:2005-04-22
| reply to Cudni
Steve
Sure, but who should decide, the notifiers or the testers !
It may have been instructive to seek out info etc from a variety of external sources, including SpyCar, but the decisions, rightly or wrongly, Must solely rest with the testers, Every time !
Spanner
--
I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks
/SpannerITWks |
|
Steve
I'm a PC, so shut up
Consultant
join:2001-03-10
Yorba Linda, CA
| reply to SpannerITWks
said by SpannerITWks : Who says they, or anybody else Have to, it's not a legal requirement ? There are two kinds of "explicit-instructions" that one might ignore:
1) For-their-own-good instructions, such as those attempting to keep you from selling their product or using it in a published benchmark. EULAs are mostly about for-their-own-good instructions.
2) For-our-own-good instructions, such as a limitation of how much information one actually can get from using Spycar in this manner. Prescription drugs and power tools have lots of for-our-own-good instructions.
Instructions of the #1 type can usually be ignored without much consequence, but #2 can only be ignored if one really knows what one is doing.
I'll leave it as an exercise to the reader as to which is likely to apply in this case.
Steve
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site |
|
SpannerITWks
Premium
join:2005-04-22
1 edit | reply to Cudni
Alex Eckelberry
Bit of an oops on your own link lol.
Well this is a completely different topic, as this thread is called - Our unique antivirus testing: How we did it - Not - Our unique antispyware testing: How we did it -
" In addition to antivirus programs, Consumer Reports tested antispyware applications. "
But anyways, i see where you're coming from, as well as FL, lucky you !
" And even more surprisingly, even though Consumer Reports used the Spycar testing methodology, they never even contacted the authors of Spycar for advice or feedback. "
Who says they, or anybody else Have to, it's not a legal requirement ? Maybe it should be from now on though lol. As long as the testers remain independent from ANY final decision making, then communicating with the Test files authors, might be acceptable, as long as this IS clearly stated within the article, and about Exactly what info was exchanged !
" So, Consumer Reports
a) Ignored the instructions of the Spycar authors and used the simulator as the sole method of testing.
b) Ignored the instructions by the Spycar authors to not use Spycar to test scan and remove functionality. "
That's different, in This case, but i wouldn't just advocate blindly Obeying + accepting what someone said, Whoever they are, just because " they " said ! But i agree, about these particular AS tests, hardly ANYwhere near thorough @ all. Useful as an extra series of tests to compliment a much more demanding batch, as quite a number of those SpyCar tests can actually get through onto many peoples PC's.
At least they responded to you, and next time they test Anti's, somehow i think things will be a lot different from the last batch !
Spanner
edit typo Only
--
I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks
/SpannerITWks |
|
SnowyOne
Premium
join:2003-04-05
Kailua, HI | reply to alexeck
I believe this is the correct link
»sunbeltblog.blogspot.com/ |
|
alexeck
join:2004-12-20
Clearwater, FL | reply to Cudni
It gets worse, folks, as I've blogged here »snipurl.com/vg57
For the antispyware testing, CR solely relied on Spycar, against the explicit instructions of the Spycar authors.
Alex Eckelberry |
|
SpannerITWks
Premium
join:2005-04-22
| reply to Cudni
" They tend to attract certain people when the thread is about to end. "
Yeah i've noticed that too !
Good news about the link after all,Thanx.
Spanner
--
I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks
/SpannerITWks |
|
Wildcatboy
Premium,Mod
join:2000-10-30
Toronto, ON
Host:
Security Product V..
Security
1 edit | reply to Cudni
FYI:
There's nothing wrong with the link to the weblog or the web site itself. IBK and I mutually agreed that the thread would be better off if the short contents of the post in the weblog is quoted here to keep the thread self contained.
Except a couple of people assumed there's a ban of some sort on the link and took it upon themselves to play heroes and challenge it, hence the deleted posts.
Let's get back to the main subject now please. Although I assume we might still have one or two people who may not want to stop and you may see further deletions but that's the nature of most popular threads. They tend to attract certain people when the thread is about to end.
--
You can catch the Devil, but you can't hold him long. |
|
IBK
join:2003-06-20
Austria
2 edits | reply to zorry
Re: Our unique antivirus testing: How we did it
said by zorry :
mmmm...Don't hold your breath - no way CR will provide the info you (and all of us for that matter) the needed goods.
those who know the url to the weblog on av-comparatives can read some of my comments there. I replied before here with a link to my weblog, by I forgot that I am here not allowed to put links to my website (as I am the owner of that website).
edit: plz do not post the url here, just ignore it atm (many points are already in this thread) |
|
joewells
join:2006-08-21
Clearwater, FL
| reply to Steve
Re: Our unique antivirus testing: How we did it
said by Steve :Well we just took a detour back down self-serving lane: you may well have the numbers to back this up, but it sounds so self-congratulatory, that it looks like you took off your technical hat and put on your PR hat.
Steve
When I was running the WildList Organization, the vast majority of the work involved verifying the viability of every virus sample received, then replicating out more samples, then verifying the viability of every single replicant. Replicants often had to be rejected. Doing this every month for over a decade, one learns just how extremely buggy viruses are. Therefore, my statement, that samples should be suspected before antivirus products are suspected, is based on years of testing both viruses and antivirus products. The claim is not based on conjecture or opinion. BTW. I currently work for an anti-spyware company, not an antivirus company. I work in future technologies research, not public relations. Regards, Joe Wells Chief Scientist, Security Research Sunbelt Software |
|
