Our unique antivirus testing: How we did it Page 7

Author	All Replies
swhx7 Premium join:2006-07-23 Elbonia ·RoadRunner Cable	reply to Cudni Re: Our unique antivirus testing: How we did it Related: quote: The top three antivirus programs -- from Symantec, McAfee, and Trend Micro -- are less likely to detect new viruses and worms than less popular programs, because virus writers specifically test their work against those programs. zdnet article; above quotation from Schneier's Cryptogram. This one is about a worm rather than a virus but has some relevant factoids: Eweek "Botnet Eavesdropping: Inside the Mocbot (MS06-040) Attack" quote: In the initial stages of the Mocbot attack, only one-third of anti-virus scanners tested by Stewart's research team were detecting the malware. "This was just a minor variant of something that was out there for months but the majority of scanners were missing it," he said. * * * The lesson? "Don't get infected in the first place," Stewart said.
	to forum · permalink · · 2006-08-21 10:46:46 ·
SpannerITWks Premium join:2005-04-22	reply to Cudni Well the links that swhx7 gave, Thanx, and i'm quoting further from below, do indeed seem to confirm and validate my earlier points about testing with variants, even if the varience is only " minor " ! A few others have also agreed it may have merit too. So where this all headed now i wonder, and who ya gonna call ? Not ghostbusters anyway, well maybe Strider Ghostbuster + the like lol. - Why popular antivirus apps 'do not work' On Wednesday, the general manager of Australia's Computer Emergency Response Team (AusCERT), Graham Ingram, described how the threat landscape has changed -- along with the skill of malware authors. - "The most popular brands of antivirus on the market have an 80 percent miss rate. So if you are running these pieces of software, eight out of 10 pieces of malicious code are going to get in," said Ingram. Although Ingram didn't mention any of the leading losers by name, Gartner's figures for 2005 show that Symantec is the clear leader with 53.6 percent of the market. McAfee and Trend own 18.8 percent and 13.8 percent of the market respectively. "We are getting code of a quality that is probably worthy of software engineers. Not application developers but software engineers," said Ingram. One vendor Ingram did mention was Russian outfit Kaspersky, which in the same tests managed to block around 90 percent of new malware. »www.zdnet.com.au/blogs/securifyt···9,00.htm Botnet Eavesdropping: Inside the Mocbot (MS06-040) Attack "This was just a minor variant of something that was out there for months but the majority of scanners were missing it," he said. Even more worrisome is the fact that the attack included the use of botnet instructions to download the second-stage Trojan executable. "In this case, it was a spam proxy Trojan, but what if it was a rootkit? The rookits are getting so good these days that the programs we typically rely on to find and clean machines just can't see them. There is still the possibility that the spammers could slip in a rootkit to hide things forever," he said. »www.eweek.com/article2/0,1895,2004922,00.asp Spanner -- I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks /SpannerITWks
	to forum · permalink · · 2006-08-21 19:48:31 ·
joewells join:2006-08-21 Clearwater, FL	reply to Cudni I seem to be getting quoted here a bit. So I guess I’ll go ahead and speak up. One key concern I have with the CU testing has been touched upon, still I’d like to elaborate on it. But first, to give you a basis for understanding my concern, let me state this: Since 1991, I have worked professionally on several antivirus products (Certus AV, Novi, Norton, IBM, V-Find, Fortinet, etc.). Moreover, I’ve also designed and/or performed antivirus tests for publication (PC World, PC Magazine, Tech TV, etc.). In addition, I’ve also had several scientific papers and technical articles published related to antivirus testing. Now, in 1993 I started a cooperative effort to qualify and quantify the actual clear-and-present-danger nature of the virus threat; in order to best protect users of all antivirus products. I did this (with in cooperation many other antivirus developers and testing organizations) through the WildList Organization. And a big part of that joint effort was intended to improve and empower scientifically based antivirus testing. It is also important for you to understand that a lot of this work in test design is based on my knowing precisely how various antivirus products work. They are not simple signature-based grunt scanners; and they haven’t been since the late 1980s. They are all precision, detection engines; often referred to as scalpel scanners to reflect the precision analysis they perform in detection and verification. So where I stand in this controversy is on top of well over a decade of working hard to establish a strong and fair scientific foundation for antivirus testing, in order to best serve users. Hey. I’ve worked hard, for a long time, to establish solid, effective, beneficial antivirus testing criteria and methodology. That is where I stand. Now I’ll shift gears. My key concern is actually irrelevant as to whether or not creating 5500 viruses is ethical or unethical. What I am most interested in, is knowing is whether or not those 5500 viruses were verified as valid threats. How were the created? Who tested them to verify their viability? Keep in mind that antivirus scalpel scanners are precision detection machines. So I’m wondering is: if the 5500 samples were not all verified as viable, then some or many may not have been viable or may even have been corrupted. And if they weren’t viable, then technically, they weren’t viruses; and a good antivirus scanner would not detect them as such. Therefore, if any of the 5500 programs were actually not viable viruses, then a good antivirus product would be penalized for correctly recognizing them as non-viral. That said, please don’t conclude that I am in anyway inferring that, if the 5500 were all functional, then the test was good. From where I stand, it wasn’t good. Good testing best serves users by testing reality. That has been my mantra for years. Joe Wells Chief Scientist, Security Research Sunbelt Software Founder WildList Organization International
	to forum · permalink · · 2006-08-21 20:12:40 ·
Steve I'm a PC, so shut up Consultant join:2001-03-10 Yorba Linda, CA	said by joewells : But first, to give you a basis for understanding my concern, let me state this: I appreciate that another actual expert is stepping up; I'm certainly not one, and am glad to get that perspective. My key concern is actually irrelevant as to whether or not creating 5500 viruses is ethical or unethical. What I am most interested in, is knowing is whether or not those 5500 viruses were verified as valid threats. Which is exactly what you should be commenting on. You most likely have your own private view, but sticking to the tech stuff means you're going to retain your audience here. Therefore, if any of the 5500 programs were actually not viable viruses, then a good antivirus product would be penalized for correctly recognizing them as non-viral. Bravo - this is an outstanding argument for why CR's tests are not valid, and I'm very strongly persuaded by this line of reasoning. I guess much of it comes down to looking at the "threats" (if CR made them available), to find out which were real threats and which were just steaming piles of bits. Thank you. Steve -- Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site
	to forum · permalink · · 2006-08-22 21:54:57 ·
SpannerITWks Premium join:2005-04-22	reply to joewells Joe Wells Hi, I did read the links provided by others to your articles, and commented in earlier posts on several matters. - Well i was also wondering about just how many of those 5500 variants could Actually be of ANY danger in some way/s. I would guess that all of them wern't that smart, any anyware near 5500. But of course as yet we don't know, i think sooner or later we will though, one way or another ! But my point was and still is, if even a small number of them, or even one for that matter, were able to slip through the net, then a nasty is a nasty is a nasty, no matter how minor the variation. And depending where the variation/s in the code took place, the effect it could have would be real. If the alteration/s were Very clever then this might have disasterous consequences indeed. I really think the whole concept of altered code needs to reassessed now, as a consequence of this test, and anyway. As the knowledge of this test is in the public domain, i'll bet there are any number of people out there, and not just sciddies either, who will be actively working on new minor variants. And i also believe much more clever stuff piggybacking etc on things that have gone before. Take as an example the very current situation with for eg the - www.google.com - nasty. This is reincarnating itself at a furious pace almost daily now. I have several samples of it and they are Exactly the same file size, but different nasties. I don't expect to see an improvement any day soon, and not just with - www.google.com - et al. Well i do actually, but only in improved nastiness ! Spanner -- I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks /SpannerITWks
	to forum · permalink · · 2006-08-22 22:58:57 ·
ghost16825 Use security metrics Premium join:2003-08-26 1 edit	reply to Cudni The key word here is variant I think. There's also a) unanswered questions about CR's exact methodology and I also think b) confusion about exactly how separate tests were weighted First round: To see how quickly software makers update their signature lists, we gave all of the products Internet access. Then we spent weeks closely monitoring each product and noted how early, if at all, the manufacturer equipped it to detect newly discovered viruses. Assumption: Newly discovered means exactly that, not create by CR Round 2: To pit the software against novel threats not identified on signature lists, we created 5,500 new virus variants derived from six categories of known viruses, the kind you’d most likely encounter in real life. Then we infected our lab computer with each of 185 of them to see whether the products could better detect viruses that were actively executing, based on their behavior. Round 3:Finally, to see how often the antivirus software raised false alarms... Question: What does this all mean? Here's the Webster's definition of variant: Variant Adjective 1. Differing from a norm or standard; "a variant spelling". Noun 1. An event that departs from expectations. 2. (biology) a group of organisms within a species that differ in trivial ways from similar groups; "a new strain of microorganisms". 3. A variable quantity that is random. 4. Something a little different from others of the same type; "an experimental version of the night fighter"; "an emery wheel is a modern variant of the grindstone". Were they: 1) Brand new viruses using some new and fancy infection method 2) Just modified existing viruses already detected by AV vendors 3) Brand new viruses which exhibited behaviour based on 'common' infection methods, no fancy infection method 4) A mixture of some or all Even with their technical consultants helping them, I really doubt it was choice 1. Everyone seems to think that the method used was Choice 2, but Choice 3 seems much more likely given the wording. If this is the case it really would depend on what these infection vectors were, and the test cases chosen are really important. If they are not, they're testing AV software with 'bad-like' behaviour rather than malicious executable content. If it was Choice 2 I really don't have a problem. said by bluezanetti : Sure 5,500 variants were created. Were they created from 600, 60, or 6 parent samples? That matters. Details do matter. New approaches are fine, but I'd like a better sense of underlying details before embracing the product of that new approach. I totally agree. said by eburger68 : ...The bottom line is, you don't compound the problem by writing these things yourself, even with the best of intentions, because intentions will matter very little if the thing escapes from the lab. One AV researcher that I know ... You're talking about something in the class of new viruses not largely based on existing code, which may or may not be a new infection vector. said by eburger68 : To return to the original issue -- the full issue -- which is whether CR had any justification -- be it practical, methodological, or ethical -- to create 5500 new viruses for testing, I hope that it is becoming clear that even if one considers CR's actions but a minor or negligible transgression, that there simply was no practical or methodological justification for them. It really depends on their test cases. On the whole, CR has justification, even if their testing methodology turns out to be absurd. said by eburger68 : However mistaken they might ultimately prove to be, I fail to see how modifying existing threats from today to create new variants of those threats advances the cause of anticipating tomorrow's threats either. Now you're talking about modification of existing viruses, it's pretty easy to see how. If you have an AV which loudly talks about heuristic this and heuristic that, yet doesn't detect practically a clone of a virus which it already detects (perhaps rebased, or replaced assembly instructions with other equivalent ones) than that is pretty shocking performance, and testing labs have a right to report it. I'm not talking about two executables with wildly varying characteristics, but clones of each other in both form and behaviour. said by SnowyOne : I'm not saying that activity does or doesn't happen. I'm saying it can be manipulated which doesn't say much for the staus quo. said by eburger68 : But if the company has the "close enough" detections today, it makes no sense for them to withhold them from regular release defs. Indeed, it makes more sense for them to include the "close enough" defs in public release defs today because there's no practical way (save special arrangement with the tester, which would call into question the validity of the test) for the AV company to guarantee that the testing entity uses the "special sauce" defs instead of public release defs. In such an event, there is no "manipulation" to speak of. Perhaps SnowyOne is talking about something along these lines: »(Old) Interesting AV claims/Clam response times Here's another hypothetical situation: Say CR decided to test right after W32BagleA came out. (Let's not include later variants like W32BagleAZ since it is debatable whether they have more in common with this virus or another virus class). Say CR modified W32BagleA and created something like the form of W32BagleB and furthermore, let's say no-one bothered to create variants of W32BagleA anywhere in the world. Would it have be right of them to test? Absolutely! Is it massively dangerous of them to do this in case it leaked out? I doubt it. What if the original file was super malicious? It is doubtful that the variant would greatly contribute to the damage already inflicted by the original.
	to forum · permalink · · 2006-08-23 07:51:48 ·
AB Premium join:2006-04-04 Leesburg, VA	said by ghost16825 : The key word here is variant I think. There's also a) unanswered questions about CR's exact methodology and I also think b) confusion about exactly how separate tests were weighted . . . etc. etc. Excellent points all, ghost16825, posted with pensive logic and erudition. As have been most of the other comments in this thread (mine possibly excepted). Which would still seem to bring us around again to 'Lucy, you got some splainin' to do'!
	to forum · permalink · · 2006-08-23 12:30:18 ·
joewells join:2006-08-21 Clearwater, FL	reply to Steve said by Steve : I guess much of it comes down to looking at the "threats" (if CR made them available), to find out which were real threats and which were just steaming piles of bits. A fundamental truism in antivirus testing is that, if an antivirus product does not detect a virus sample, then automatically suspect the virus sample; not the antivirus product. Unlike AV products, viruses don't go through extensive quality assurance, alpha, and beta testing. Therefore, the likelyhood of error source leans strongly toward the virus. Oddly, many testers seem ignorant of this simple fact; and thus fall prey to a fallacious false assumption. Regards, Joe Wells Chief Scientist, Security Research Sunbelt Software
	to forum · permalink · · 2006-08-23 18:41:00 ·
IBK join:2003-06-20 Austria	reply to Steve In my personal opinion, even if the variants are all viable and malicious, the test results based on that files are useless and misleading for the readers, as they do not tell about how good or bad av products are in detecting new malware in the real world, as they are just artificially created slightly variants of old malware, which is not want you are going to encounter usually (in contrary to what CR stated, like also the contrary when they said that test results for this kind of goal do not exist already). I still hope that we all will get in near future from somewhere/someone more details on the files they used, so it will be easier to argue aout the test. [P.S.: i saw that many peoples have many different opinions and views on how CR tested and the results - the above is just my personal opinion that i share with you, nothing more)
	to forum · permalink · · 2006-08-23 18:58:37 ·
Steve I'm a PC, so shut up Consultant join:2001-03-10 Yorba Linda, CA	reply to joewells said by joewells : A fundamental truism in antivirus testing is that, if an antivirus product does not detect a virus sample, then automatically suspect the virus sample; not the antivirus product. Well we just took a detour back down self-serving lane: you may well have the numbers to back this up, but it sounds so self-congratulatory, that it looks like you took off your technical hat and put on your PR hat. I'm actually likely to believe this when it comes from Joe Random Idiot making a claim against the A/V product, but I think that Consumer Reports probably gave a bit more care. Steve -- Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site
	to forum · permalink · · 2006-08-23 19:37:10 ·
zorry @megawebservers.com	reply to Cudni quote: In my personal opinion, even if the variants are all viable and malicious, the test results based on that files are useless and misleading for the readers, as they do not tell about how good or bad av products are in detecting new malware in the real world, as they are just artificially created slightly variants of old malware, which is not want you are going to encounter usually (in contrary to what CR stated, like also the contrary when they said that test results for this kind of goal do not exist already). Wise words from someone who really knows about testing! quote: I still hope that we all will get in near future from somewhere/someone more details on the files they used, so it will be easier to argue aout the test. mmmm...Don't hold your breath - no way CR will provide the info you (and all of us for that matter) the needed goods.
	to forum · permalink · 2006-08-23 20:20:50 ·
joewells join:2006-08-21 Clearwater, FL	reply to Steve said by Steve : Well we just took a detour back down self-serving lane: you may well have the numbers to back this up, but it sounds so self-congratulatory, that it looks like you took off your technical hat and put on your PR hat. Steve When I was running the WildList Organization, the vast majority of the work involved verifying the viability of every virus sample received, then replicating out more samples, then verifying the viability of every single replicant. Replicants often had to be rejected. Doing this every month for over a decade, one learns just how extremely buggy viruses are. Therefore, my statement, that samples should be suspected before antivirus products are suspected, is based on years of testing both viruses and antivirus products. The claim is not based on conjecture or opinion. BTW. I currently work for an anti-spyware company, not an antivirus company. I work in future technologies research, not public relations. Regards, Joe Wells Chief Scientist, Security Research Sunbelt Software
	to forum · permalink · · 2006-08-23 20:42:23 ·
IBK join:2003-06-20 Austria 2 edits	reply to zorry Re: Our unique antivirus testing: How we did it said by zorry : mmmm...Don't hold your breath - no way CR will provide the info you (and all of us for that matter) the needed goods. those who know the url to the weblog on av-comparatives can read some of my comments there. I replied before here with a link to my weblog, by I forgot that I am here not allowed to put links to my website (as I am the owner of that website). edit: plz do not post the url here, just ignore it atm (many points are already in this thread)
	to forum · permalink · · 2006-08-24 09:35:28 ·
Wildcatboy Premium,Mod join:2000-10-30 Toronto, ON Host: Security Product V.. Security 1 edit	reply to Cudni Re: Our unique antivirus testing: How we did it FYI: There's nothing wrong with the link to the weblog or the web site itself. IBK and I mutually agreed that the thread would be better off if the short contents of the post in the weblog is quoted here to keep the thread self contained. Except a couple of people assumed there's a ban of some sort on the link and took it upon themselves to play heroes and challenge it, hence the deleted posts. Let's get back to the main subject now please. Although I assume we might still have one or two people who may not want to stop and you may see further deletions but that's the nature of most popular threads. They tend to attract certain people when the thread is about to end. -- You can catch the Devil, but you can't hold him long.
	to forum · permalink · · 2006-08-24 16:55:19 ·
SpannerITWks Premium join:2005-04-22	reply to Cudni " They tend to attract certain people when the thread is about to end. " Yeah i've noticed that too ! Good news about the link after all,Thanx. Spanner -- I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks /SpannerITWks
	to forum · permalink · · 2006-08-24 17:10:33 ·
alexeck join:2004-12-20 Clearwater, FL	reply to Cudni It gets worse, folks, as I've blogged here »snipurl.com/vg57 For the antispyware testing, CR solely relied on Spycar, against the explicit instructions of the Spycar authors. Alex Eckelberry
	to forum · permalink · · 2006-08-25 10:50:18 ·


Sunday, 29-Nov 04:00:33	Terms of Use \| Privacy Policy \| Hosting by www.nac.net - DSL,Hosting & Co-lo \| feedback \| contact over 10 years online! © 1999-2009 dslreports.com. republican-creole page compression OFF