how-to block ads
|
Cudni
La Merma - Vigilado
Premium,MVM
join:2003-12-20
Someshire
| Our unique antivirus testing: How we did it
from
»www.consumerreports.org/cro/elec···ting.htm
"...
To pit the software against novel threats not identified on signature lists, we created 5,500 new virus variants derived from six categories of known viruses, the kind you’d most likely encounter in real life.
.."
not everybody thinks that was/is good idea
»sunbeltblog.blogspot.com/
»www.avertlabs.com/research/blog/?p=71
Cudni
--
Some are born to failure, others achieve it, all deserve it.Help yourself so God can help you.MVP, Microsoft Windows Security 2006 | |
KyeU
join:2003-12-31
Canada | I hope one day they don't go:
"OH NO! WE CREATED A MONSTER!" | |
SpannerITWks
Premium
join:2005-04-22
| reply to Cudni
- An open and sincere letter to the AV etc peeps -
I clicked on the avertlabs link - »www.avertlabs.com/research/blog/?p=71 - (you can read an open letter on the AVIEN site about that).
Which gets you to here -
»www.avien.org/publicletter.htm - Public letter concerning the Writing of Viruses & How it Does Not Teach about Virus Prevention. Originally published: May 30th, 2003 Last updated: August 11, 2006 7:14 PM
" The more than 100 signatories of this public letter, all security professionals with years of experience in dealing with computer viruses, and who work in all sectors, wish to express their whole-hearted support of the following principle:
It is not necessary and it is not useful to write computer viruses to learn how to protect against them. "
Signed:
etc -
Among the people signing their names to it are a number of well known figures. Whether ALL of them who originally signed still agree with Everything on there is open to question, but let's say they do for now !
Of course you don't have to be " able " to write nasties to write code to detect them per se. But, i've got a number of nasties in my collection that ALL the vendors listed on Jottis + VirusTotal did NOT detect when i submitted them ? These included Rootkits/Trojans/Exploits/Keyloggers etc.
So how can this be if the signed statement above is Totally correct ? Either they can detect new nasties and variations, or they can't ! And based on my tests they can NOT and did NOT on those occasions.
If they Actually mean detecting whilst being run etc ok. But they do NOT All do that either, whether normally and/or heuristically. If they say don't need to know how to write nasties, and in ALL their variations/conotations, how can they Totally understand and prevent vectors etc being compromised and therfore computers getting infected. If they were 100% right about their claims, then NOBODY would EVER get infected with ANYTHING, but hey guess what, err yes that's right, they DO, and daily with ALL sorts of crap, including brand new stuff and variations.
So what Exactly do they mean when they say " It is not necessary and it is not useful to write computer viruses to learn how to protect against them. " Because if they DO know, they are NOT putting that knowledge into practice ?
Spanner
--
I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks
/SpannerITWks | |
AB
Premium
join:2006-04-04
Leesburg, VA
| reply to Cudni
Quoted from Consumer Reports article:
----------------------------------------------------------------------------------
To be safe from online infection, you need protection from current viruses, which number 100,000 . . (sic) . .
To pit the software against novel threats not identified on signature lists, we created 5,500 new virus variants . . .
----------------------------------------------------------------------------------
Cute. Very nice.
In one fell swoop, they have increased the known virus count by 5.5%. That's an excellent day's work there, Dr. Consumerstein.
And you can practically write it in stone that some of these will soon be finding their way OUT of the lab.
If the disease doesn't kill you, the cure will.
Potential quote from a Consumer Reports spokesperson in a couple of months: "Whoops! Sorry!" | |
Blackbird
Built for Speed
Premium
join:2005-01-14
Fort Wayne, IN
 ·Verizon Online DSL
1 edit | reply to SpannerITWks
I think what they mean is that CU's constructing new variants from 6 categories of known viruses only shows how various AVs will respond to new, unknown virus variants constructed using the same techniques employed by CU. Those techniques were intended by CU to create large numbers of virus variants based on existing virus structures and ideas... they were not created to exploit new-found security holes nor were they created using novel virus-structure techniques. While CU's variants may be "new", they are not necessarily representative of what many actual virus writers will do in creating their malware in the real world. Until now. Now there are 5,500 'new' viruses on CU's lab computers and some (likely) documented recipes in CU's files of how each was created from existing virus categories - all for the script kiddies and other baddies to sniff out as only they can. And we can all hope and pray that CU's internal data/info security is better than was their reasoning in following such a path in the first place.
Thoroughly understanding viruses and how they are written does not equate to actually writing them. Writing them may or may not make one more expert in combating them. One certainly does not need to commit murder (nor many other things in life and the technical world) to understand how it is done and to combat it.
edit: phrasing in middle of para 1
--
If God wanted us to work with electrons, He'd make them big enough to see... | |
Cudni
La Merma - Vigilado
Premium,MVM
join:2003-12-20
Someshire
| reply to Cudni
5,500 "new" viruses released to the wild is not even a fraction scary as just one of the biological viruses stored in both civilian and military labs escaping
Cudni
--
Some are born to failure, others achieve it, all deserve it.Help yourself so God can help you.MVP, Microsoft Windows Security 2006 | |
SpannerITWks
Premium
join:2005-04-22
| reply to Blackbird
Blackbird SR
Sure i get your murder analogy Thanx !
But people might be interested in looking @ this thread - »forum.sysinternals.com/forum_pos···003&PN=1 - to see just how cat + mouse actually works in REAL life.
Yes real life, because in there are Real Rootkit coders with Real RK's that are out there right now being used to hide nasties and being used by 3rd parties for crime. Also in there are various well known RK detector guys n girls combatting those and other RK's.
You will see how being able to write RK's and dissasemble them etc, and write detectors enables both sides to have a greater understanding of each others tactics etc. Thereby enabling them to design better RK's + detectors.
So i do believe it's definately worthwhile to as much inside knowledge as possible about how the other side Really works, because that IS what they do, every day !
Spanner
--
I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks
/SpannerITWks | |
AB
Premium
join:2006-04-04
Leesburg, VA
| said by SpannerITWks  :. . . You will see how being able to write RK's and dissasemble them etc, and write detectors enables both sides to have a greater understanding of each others tactics etc. Thereby enabling them to design better RK's + detectors. So i do believe it's definately worthwhile to as much inside knowledge as possible about how the other side Really works, because that IS what they do, every day ! Sure, knowing how the other half lives, what they do, is good and will help people better understand how to fight the malware more effectively. But ya gotta write 5500 NEW variants to do that? I don't think so!
This is a disaster waiting to happen. Let's hope it won't.
And the first variant found in the wild that can be directly linked back to this research, I hope to see one massive class-action lawsuit.
And btw, is 'Consumer Reports' really the organization we want leading this research? While I understand that this is in fact a consumer issue, I'm just not so sure these are the people I want in the vanguard of this somewhat shaky business. | |
Steve
I'm a PC, so shut up
Consultant
join:2001-03-10
Yorba Linda, CA
1 edit | reply to Cudni
said by Sunbelt's blog :
Publications need to use industry-standardized methods for testing. Organizations like Virus Bulletin have been doing this for years. Why can';t publications follow their lead? I don't have any opinion about the particular approach taken or conclusions reached, but this statement is one I can take some exception with.
Consumer Reports has a different constituency than does the A/V industry, and it's not out of the question - in principle - for one to wonder if the A/V industry's tests are there to serve the industry and not the consumer.
Steve
Edit - fixed yucky writing (and not even drinking yet!)
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site | |
Mele20
Premium
join:2001-06-05
Hilo, HI
| said by Steve :Consumer Reports has a different constituency than does the A/V industry, and it's not out of the question - in principle - to for one to wonder if if the A/V industry's tests are there to serve the industry and not the consumer. Steve Is that from the article itself? It doesn't sound like a quote with the bad grammar and the repeated words). I can't read the article or comment here because I am not a subscriber to Consumer Reports. If that is from the article, I don't see any particular objection...but I haven't been able to read the article..just other people's comments about it so I can't really comment intelligently until I am able to read the article when my library gets a copy of the September issue.
--
"If you want to do DRM on a PC then you need to treat the user as the enemy." Ross Anderson in "`Trusted Computing' Frequently Asked Questions" | |
AB
Premium
join:2006-04-04
Leesburg, VA
| reply to Steve
said by Steve :. . Consumer Reports has a different constituency than does the A/V industry, and it's not out of the question - in principle - to for one to wonder if if the A/V industry's tests are there to serve the industry and not the consumer. Steve I don't have any hard facts to put into evidence here, but as I recall it, the A/V industry stats tend to show an average detection rate of about 80-85%. Not too shabby, yet not exactly something to be boasting about, either.
One might think they would pad those stats up a bit higher if the tests they run were merely of self-serving interest, no? | |
Steve
I'm a PC, so shut up
Consultant
join:2001-03-10
Yorba Linda, CA
| reply to Mele20
said by Mele20 : Is that from the article itself? It doesn't sound like a quote with the bad grammar and the repeated words). The blame for the sloppy syntax is with me - long day, eyes getting blurry. My fault.
Industry self-policing (which is what "industry-standard tests" are) can be good or it can be bad, but one can't forget that they have their own constituency.
On one hand, they're going to be the subject-matter experts, which clearly gives them an edge over Consumer Reports, but on the other hand, the industry likely wishes to measure their own products in terms of how they think of the problem. That may or may not be how the consumer looks at it.
It's certainly not out of the question, however, that CR just did a poor job on this; I look forward to seeing the real report when it arrives too.
Steve
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site | |
eburger68
Premium,MVM
join:2001-04-28
1 edit | reply to Cudni
Steve:
The "real report" is simply table of ranked apps coupled with basic information (price, company) and 10 columns of bubbles to cover features and test results (bubbles can be empty, partially filled-in, entirely filled-in). The rough rankings were posted in this forum very recently:
»Consumer Reports Best Tools Stop Viruses/Spam/Spyware
The issue is on the newsstands now.
quote:
On one hand, they're going to be the subject-matter experts, which clearly gives them an edge over Consumer Reports, but on the other hand, the industry likely wishes to measure their own products in terms of how they think of the problem. That may or may not be how the consumer looks at it.
McAfee and the other AV industry experts that have been quoted so far in news stories (VirusBulletin, Sophos, and Kaspersky) have all offered sound reasons -- methodological, practical, and ethical -- for not relying on lab-created viruses for anti-virus testing. The principles here have been rather settled in the AV community for some time.
For an elaboration on why the creation of viruses for testing is not only methodologically unsound, but practically unnecessary, and ethically dubious, see this this Open Letter from the AV community (authored by Joe Wells) to CNET:
»cybersoft.com/whitepapers/papers···er.shtml
Note that a number of the letter's signatories are not affiliated with any AV company. And the author of the letter has proven himself quite capable of being a fierce critic of the industry's own practices and habits -- see for example:
»vx.netlux.org/lib/ajw01.html
If one is going to speculate that this griping about CR's AV testing is just the AV industry covering its own backside, then we'll really need just a bit more than the speculation -- the arguments against the creation and use of lab viruses have been on the table for some time now. What would be the arguments in favor of lab viruses? In what way did consumers benefit from Consumer Reports' creation of 5500 new virus variants, none of which was an actual "in the wild" virus?
For those who are wondering, "If not for the use of lab-created viruses, how could researchers test the ability of AV products to handle new and hitherto unknown viruses?", McAfee's complaint contains the answer: retrospective testing, a procedure that is well established and has the advantage of testing against "in the wild" viruses, not lab-created viruses:
»www.avertlabs.com/research/blog/?p=71
Best,
Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior | |
SnowyOne
Premium
join:2003-04-05
Kailua, HI
 ·RoadRunner Cable
 ·Clearwire Wireless
| What's done has been done & there's no turning the clock back regardless of whether one agrees or disagrees with the bonehead methodology.
What I'm not sure of is would CR be acting responsibly if they gave every AV/whatever vendor a copy of each & every file they created? If some or all were to go ITW, wouldn't the AV/whatever vendors be in a better position to minimize the damage by already having the definitions covered? On the other hand, the more who have access, the more chance of leaks. Opinions? | |
funchords
Hello
Premium,MVM
join:2001-03-11
Washington, DC
·Verizon Online DSL
 ·Skype
| reply to eburger68
said by eburger68 :have all offered sound reasons -- methodological, practical, and ethical -- for not relying on lab-created viruses for anti-virus testing. The principles here have been rather settled in the AV community for some time. Fine, but the virus landscape has been recently changing. What has been conventional wisdom in past years may not necessarily apply to strategies for combatting today's 0-day malware.
As an outsider to the AV community, I don't see a major crime here. The magazine's approach seems to be a reasonable one.
--
Robb Topolski -= funchords.com =- Hillsboro, Oregon USA
~ Keeper of the D-Link FAQ ~ Did you Search? ~ More features, Free! Join BBR! ~ | |
SpannerITWks
Premium
join:2005-04-22
| reply to Cudni
eburger68 gave some links, Thanx, which i read, and the following in " " are quotes from them.
Re - Open Letter
" If a product does not report a simulated virus as being infected, it's right. And if a program does report a simulated virus as being infected, it's wrong. Thus, using simulated viruses in a product review inverts the test results. It grossly misrepresents the truth of the matter because:
* It rewards the product that incorrectly reports a non-virus as infected. * It penalizes a product that correctly recognizes the non-virus as not infected.
Competent, credible antivirus product reviewers today recognize the need to reflect the real world in their testing. To do so, they focus detection testing on the real-world threat, using real viruses. They focus on viruses reported by the WildList Organization International. True, some may also include other viruses in testing, but they still use real viruses, not simulated ones. "
»cybersoft.com/whitepapers/papers···er.shtml
So REAL Malware is obviously Totally acceptable for testing purposes ! Even if the vendors don't yet have a sample of it/them, non the less a NEW real as yet undiscovered nasty by them is 100% valid.
-
Joe Wells = The WildList
A Radical New Approach to Virus Scanning
Don't expect this paper to be about a virus problem. To the contrary, it's actually about your having an antivirus problem.
-
" unless otherwise stated, virus scanning specifically refers to methods of detecting known viruses - as, for example, by using signatures.
-
Of course not, the virus problem has been getting progressively worse. What it does mean is that the number of "all known viruses" is far outstripping the number of wild viruses. It means the increase is almost entirely in zoo viruses.
-
What I theorized and IBM proved about trends in DOS file virus extinction effects you directly. Members of an endangered species are becoming increasingly rare in the wild. No wonder nearly all of them are found only in zoos - they simply can't cut it anymore.
-
Zoo detection, in particular a large polymorphic library, is not required for a good certification scheme. Rather, a threat library which determines whether the product is capable of providing protection from all types of self-replicating code should be used.
-
So why are we discussing zoo viruses if they're not a threat? What's the point?
The point is this: There are tens of thousands of viruses you'll never get. There should be no reason for us to discuss them. But we must discuss them. Because, whether you realize it or not, those tens of thousands of viruses do affect you. They do have a direct impact on you. And their effect is detrimental.
-
In the real world, zoo viruses are not a problem. Wild viruses are.
-
But even if you haven't dealt with viruses at all, you don't have to be an expert to intelligently evaluate evidence.
Let me illustrate. Suppose you served on a jury. The trial involves complex medical issues. Would you have to be a neurosurgeon to weigh the evidence presented? Of course not. You would listen to the evidence presented, and evaluate it fairly. You might have to ask for clarifications (I certainly would.), but that does not make you unqualified.
-
Of course, many experts think they are authorities. They actually believe that their expert opinion is more than just opinion-it is truth. Similarly, many people do view experts as being authorities.
( Paraphrasing by me ) Many experts who do testing have doctorate degrees. Others don't have any degrees at all. Who should you believe, others or real experts?
-
Making someone else appear inferior, somehow make you appear superior. "
»vx.netlux.org/lib/ajw01.html
OK so zoo are off limits fine, but appear, or did @ one time, to outnumber real Malware. So why did/do they waste so much time on testing them then ?
" In the real world, zoo viruses are not a problem. Wild viruses are. "
Absolutely, and not just viruses of course, but All forms of Malware. And Real Wild = REAL even if the vendors don't have them yet. If they are out there then the potential for infiltration/infection and/or damage is also Very REAL.
Re the earlier link - Public letter concerning the Writing of Viruses & How it Does Not Teach about Virus Prevention.
" It is not necessary and it is not useful to write computer viruses to learn how to protect against them. " So why bother with writing ALL those zoos ?
-
" cannot know what viruses we are going to face in future "
»www.avertlabs.com/research/blog/?p=71
Exactly !
-
Naturally no AV without heuristics etc is going to going to detect something that is not in it's defs !
I still believe that coding New workable Malware of all various types can definately help the vendors design better Stiffer more resistant products, why wouldn't it ? And it certainly ain't gonna do any harm is it, unlike the Real Malware that's already been released before you read this, and later on today, and tomorrow etc etc, that the vendors don't know about yet, which means neither does your AV hence you !
Spanner
--
I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks
/SpannerITWks | |
eburger68
Premium,MVM
join:2001-04-28
1 edit | reply to funchords
Funchords:
You wrote:
said by funchords :What has been conventional wisdom in past years may not necessarily apply to strategies for combatting today's 0-day malware. Maybe, maybe not. But it's not enough just to suggest in the abstract that conventional wisdom may be wrong -- anyone can do that. The challenge here is to offer concrete, logical reasons why the 0-day malware landscape of today would somehow invalidate the principles agreed upon by the AV industry -- principles based on hard-won experience with malware.
So far I haven't heard any that directly address the arguments offered by experienced AV experts.
Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior | |
HMS1
join:2006-01-14
Austin, TX
| reply to Cudni
I don't see any problem with CR having created a bunch of new viruses. Obviously the bad guys can do the same whenever they want (the Bagle guys were cranking out the variations for a while, for example). CR's action did not increase this ability of malicious actors, nor would refraining have restrained the malicious actors.
Retrospective testing is fine for anti-virus. But if you're working on heuristic detection, you may have good reason to write some new malware to test against. There may be some preexisting program that fits the description of what you want, or there may not be, or it may be too much trouble to find one.
There is an underlying contradiction in the rhetoric against virus-writing. If lab-created viruses are ineffective for testing, then they must be harmless when released. If they are harmful when released, then they are good for testing. If the real-world virus writers haven't taken a particular approach, but you can think of it, then they have thought of it too, or will soon. It is contradictory - and suspicious - to maintain that lab-created viruses are so dangerous that they must not be created, yet so unrepresentative that they are no good for crafting defenses.
Statements such as the Avien one and the critques of the CR method make me wary of some sort of initiative to get virus-writing outlawed or restricted by licensing. There are hints too of a lobby for legal limits on instruction about malware writing.
Any such laws would be unjust and harmful. There should never be any prohibition on writing any kind of software, at all, ever (only spreading malware to unwilling parties should be illegal). We should be wary of such self-serving proposals which would impair freedom of speech and make computer users dangerously dependent on a privileged industry.
The right approach is to focus on improving the castle walls, not quibbling about how we study the Huns' weapons. | |
eburger68
Premium,MVM
join:2001-04-28
2 edits | said by HMS1 :I don't see any problem with CR having created a bunch of new viruses. Obviously the bad guys can do the same whenever they want (the Bagle guys were cranking out the variations for a while, for example). CR's action did not increase this ability of malicious actors, nor would refraining have restrained the malicious actors. The ethical issue doesn't involve "increasing" or "restraining" the ability the ability of bad guys to create viruses or other malware.
No, the issue lies in creating new viruses that could, despite the best efforts of their creators, escape into the wild, and this kind of thing is a regular occurrence in the virus-writing world. The bottom line is, you don't compound the problem by writing these things yourself, even with the best of intentions, because intentions will matter very little if the thing escapes from the lab.
One AV researcher that I know has received source code for viruses that authorities like the FBI and Secret Service have uncovered in raids. This researcher refuses on principle even to compile the code himself for the purpose of lab analysis. The issue for him is just that clear-cut -- just that serious.
said by HMS1 :Retrospective testing is fine for anti-virus. But if you're working on heuristic detection, you may have good reason to write some new malware to test against. There may be some preexisting program that fits the description of what you want, or there may not be, or it may be too much trouble to find one. The beautiful thing is that you already provided all the reasons why it's not even necessary to write new viruses back up in your first paragraph:
quote:
Obviously the bad guys can do the same whenever they want (the Bagle guys were cranking out the variations for a while, for example).
Precisely. There is no shortage of viruses and variants in the wild to analyze -- all the more reason why it's not necessary to create new ones. If researchers are having so many problems finding enough viruses to analyze that they're tempted to start creating them, then those researchers aren't doing a proper job of it.
Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior | |
AB
Premium
join:2006-04-04
Leesburg, VA
| said by eburger68 :. . . the issue lies in creating new viruses that could, despite the best efforts of their creators, escape into the wild, and this kind of thing is a regular occurrence in the virus-writing world. The bottom line is, you don't compound the problem by writing these things yourself, even with the best of intentions, because intentions will matter very little if the thing escapes from the lab. Eric L. Howes Amen, Brother! | |
|
