how-to block ads
|
SpannerITWks
Premium
join:2005-04-22
| reply to Cudni
Re: Our unique antivirus testing: How we did it
eburger68
Hi,
Who's talking about Radmin whatever code, and just Bots ? I clearly stated All Malware. And my example, extreme on purpose cos ya never know, about changing 1 Bit to make it a killer still holds up. It doesn't matter if it's 20 years old or from 2 mins ago, if some " old " code gets reworked into something tasty, well it's still new.
You're not telling us that the vendors would ignore it as old and not include in its Defs are you ? And if it turned out that a small amount of code, 1 Bit or whatever, turned it into an NG type nasty because that change actually enabled it to assume that status, then it would be.
I mean how many lines of code actually need to be changed and/or added to an " old " nasty to make it NG ? I don't think you me or anyone can truly say can we ! The NG aspect refers to it being capable of infiltrating by, unknown to us at this moment, holes in our systems, and/or bugs in software, that are exploited very craftily, and/or stealthily, and/or maybe very hard not only to discover, but also to remove, if possible !
As i said NG could be here right now, and we wouldn't know it, and it does NOT need to be 100% brand new code. All it takes is what it takes to accomlish it, whether it's millions of lines of code or a helluva lot less. Actually less is Much better, and i think that's the way a lot of new stuff will be headed.
Re the vendors cooking up nasties -
I already explained about that, and why, and ONLY keeping the clever stuff, and why it would make sense for them to ONLY do that. If people just go on thinking in todays terms regarding users, and their often understandable, in the True sense " ignorance " about software PC's etc, imagining that they won't get more knowledge through various channels, then i believe that's a mistake. All it would take is for the media to do a SEXY story on bloat etc, and the cat would well and truly be out of the bag forever. And they would be comparing products, not just on price, but also lightness/speed etc etc, and about flipping time too !
The media love SEXY stories, and that would definately be one, and it would be picked by other media outlets and spread. Also by word of mouth with the punters too. The vendors would be putting Big Bold statements on their boxes about how slim and lightning fast it was, as well as being effective in this n that. Hey i see a whole new marketing statergy evolving out of all this. Don't forget to bung Spanner a few $ for the idea, will you vendors !
Here's another idea -
You asked how would the "real crap" be seperated from the "potentially clever stuff" Like this -
Why couldn't some software be written that analysises AV's etc Defs and identifies Everything in there. Then whatever you want could be removed, left in, or even added to ! All it would take is a once and for all analysis, with i imagine humans intervention and approval, of ALL the zoo type stuff, then you'ld know what you want included or not. Afer that it would just be a matter of updating any new zoos by the same methods. Didn't say it would be easy, i dunno it might be, i'm not a coder, but you asked how it could be done ! You might not approve, but others might, and even take the idea up and design such a system. Bet you'ld like a copy of it hey !
-
General point -
Can't we get a rep from CR to post on here ?
Spanner
--
I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks
/SpannerITWks | |
eburger68
Premium,MVM
join:2001-04-28
1 edit | reply to Wildcatboy
Wildcatboy:
You wrote:
said by Wildcatboy  :The same way it's done today. The AV industry is not proactive by nature. They simply react to the threats presented to them. If any of those 5000 viruses get out in the wild, they'll see it and they'll add the signature. If they don't make it out, the AV industry will do what they have been doing. Nothing. But the standard for what goes into definitions in that state of affairs wouldn't be limited to what's "in the wild." In the AV dystopia that we're postulating here, the "in the wild" standard has been partially or largely abandoned, as we would be in a situation where what goes into definitions would be comprised to some (a large?) degree of lab viruses -- "potential threats" cooked up in the lab by researchers, testing entities, and (in the worst case scenario) the AV companies themselves.
I suppose one could propose a industry standard that no one added lab viruses to definitions unless there was some reasonable suspicion or confirmation that an escape had taken place. That industry agreement could even provide for sample sharing of lab viruses among vendors.
But even after such an agreement had been brokered, there would be serious pressure on the AV industry to add those lab viruses to definitions on the grounds that:
a) they were legitimate, potential threats (as SpannerITWks has argued here);
b) the best policy is always to be proactive, not reactive in response to known potential threats.
It doesn't take too much to imagine a muckraking series of articles in the mainstream media that gravely informed readers that the AV industry had hundreds of thousands of dangerous viruses in its lab but was refusing to offer its customers protection against those viruses. Throwing gasoline on the fire, these muckrakers inform the public that the industry doesn't actually know the precise number, nature, and disposition of these lab viruses, and that no one can therefore guarantee that an escape of some sort hadn't already taken place.
What do you think the response of users and customers -- the public at large -- would be to those kinds of revelations? We've got large numbers of users and consumers currently demanding that cookies be detected and removed as serious threats. Would these folks be easily persuaded that 300,00 viruses sitting in a lab somewhere posed no threat to them?
Let me clear: I don't think the single set of tests conducted by CR is sufficient to bring about the scenario outlined above. Obviously, it wouldn't be (though I wouldn't be surprised to hear that some corporate clients started asking for similar, CR-like lab virus variant testing from the commercial testing companies they hired to perform comparative testing in advance of a major software licensing purchase). What I worry about is a situation in which CR blows off the criticism of the AV industry, conducts more of these kinds of tests, and effectively forces the hand of other testing entities to keep up methodologically. From there it is a race to the bottom.
No, the only folks who benefit from a scenario in which lab virus creation becomes widely accepted are the sales departments of the AV companies themselves. And I have to believe that users and customers would eventually become justifiably embittered at having to pay for protection against potential threats created by the industry itself.
Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior | |
eburger68
Premium,MVM
join:2001-04-28
| reply to SpannerITWks
SpannerITWks:
You wrote:
said by SpannerITWks :Who's talking about Radmin whatever code, and just Bots ? I clearly stated All Malware. It was an example, not an exhaustive inventory of potential malware threats. Just and example to illustrate a point.
said by SpannerITWks :I mean how many lines of code actually need to be changed and/or added to an " old " nasty to make it NG ? I don't think you me or anyone can truly say can we ! You're right. We don't. And neither does CR, I reckon. Nor did they claim that this was the goal of their testing. So, let's stop speculating about it ourselves, huh?
said by SpannerITWks :Re the vendors cooking up nasties - I already explained about that, and why, and ONLY keeping the clever stuff, and why it would make sense for them to ONLY do that. If we could expect only scrupulous, ethical researchers to be involved in the internal decision-making at AV companies as to what to keep and what to throw out, then your hope might be justified. Unfortunately, there are other folks that would be involved in decisions like that: marketers, advertisers, middle managers, clueless senior execs even. And in the worst kind of competitive environment -- one where suspicion and mistrust ruled the day -- the premium would be on numbers, not the determination of what constituted the "clever stuff."
said by SpannerITWks :If people just go on thinking in todays terms regarding users, and their often understandable, in the True sense " ignorance " about software PC's etc, imagining that they won't get more knowledge through various channels, then i believe that's a mistake. All it would take is for the media to do a SEXY story on bloat etc, and the cat would well and truly be out of the bag forever. A "sexy story on bloat"? Somehow I'm thinking that story wouldn't make it off the editor's desk. Much more likely that the editor would go with the muckraking story that I imagine in my response to Wildcatboy above. "Coverup," hidden threats, and virus researchers gone mad sells copy -- a sexy story on bloat doesn't.
said by SpannerITWks :And they would be comparing products, not just on price, but also lightness/speed etc etc, and about flipping time too ! You're imagining a software-buying public that is 180 degrees the opposite of the public we know and understand today. Show me that consumers are finally rejecting the bloat and burden of NIS and NAV en masse, and I might start to think you were on to something.
said by SpannerITWks :The media love SEXY stories, and that would definately be one, and it would be picked by other media outlets and spread. Also by word of mouth with the punters too. The vendors would be putting Big Bold statements on their boxes about how slim and lightning fast it was, as well as being effective in this n that. Hey i see a whole new marketing statergy evolving out of all this. Don't forget to bung Spanner a few $ for the idea, will you vendors ! What numbers tell the more compelling story to users looking for "comprehensive protection" against the plethora of threats in the computing universe?
* The number of viruses detected by Product A vs Product B, or...
* the benchmarked scan speeds of Product A vs. Product B?
If I'm a careful, cautious consumer without too much knowledge of the relative risks of "in the wild" viruses vs "lab viruses" (or even an inkling that such a division exists in the malware world -- a threat is a threat, isn't it?) -- then I'd rather be safe and slow than sorry.
said by SpannerITWks :Here's another idea - You asked how would the "real crap" be seperated from the "potentially clever stuff" Like this - Why couldn't some software be written that analysises AV's etc Defs and identifies Everything in there. Then whatever you want could be removed, left in, or even added to ! All it would take is a once and for all analysis, with i imagine humans intervention and approval, of ALL the zoo type stuff, then you'ld know what you want included or not. Afer that it would just be a matter of updating any new zoos by the same methods. Didn't say it would be easy, i dunno it might be, i'm not a coder, but you asked how it could be done ! You might not approve, but others might, and even take the idea up and design such a system. Bet you'ld like a copy of it hey ! Why couldn't someone do that? Well, there are these folks called lawyers. Major AV companies tend to hire a good number of them -- esp. those with backgrounds in intellectual property law. IP lawyers, as a general rule, don't look favorably on who folks who start reversing the copyrighted, patent protected software of their employers. In fact, they tend to frown on that kind of thing. You get the drift...
It's worth repeating at this point that even if you DID manage to set up an effective regime in which AV companies released defs only for the "potentially clever stuff," you'd still be asking the software buying public to swallow the proposition that they had to pay for protection againt malware created by the industry itself. I simply don't think that kind situation would be sustainable.
said by SpannerITWks :General point - Can't we get a rep from CR to post on here ? I think this is the one proposal that everyone in this thread could agree on.
Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior | |
AB
Premium
join:2006-04-04
Leesburg, VA
| said by eburger68 :You're imagining a software-buying public that is 180 degrees the opposite of the public we know and understand today. Show me that consumers are finally rejecting the bloat and burden of NIS and NAV en masse, and I might start to think you were on to something. Good point. After all, Microsoft has 95% of the OS market, don't they? And many people are on pins & needles waiting for Vista. Go figure.
said by eburger68 :said by SpannerITWks :General point - Can't we get a rep from CR to post on here ? I think this is the one proposal that everyone in this thread could agree on. Eric L. Howes Yep. In the immortal words of Ricky Ricardo-- Lucy, you got some 'splainin' to do! | |
sybille
Not only "just visiting"
Premium
join:2004-04-06
France
| reply to Cudni
Re: Our unique antivirus testing: How we did it
If "lab viruses" were so potentially dangerous for users (as opposed to for the AV industry), then I wonder why we haven't seen more problems stemming from the existence of proof-of-concept lab viruses for the GNU/Linux operating system?
In fact, there have been a number of such lab viruses for Linux. In 2001, Peeling and Satchell noted that essentially all of the viruses for Linux were of the laboratory variety:
said by »www.govtalk.gov.uk/documents/Qin···tware%22 :
There are about 60,000 viruses known for Windows, 40 or so for the Macintosh, about 5 for commercial Unix versions and perhaps 40 for Linux. Most of the Windows viruses are not important, but many hundreds have caused widespread damage. Two or three of the Macintosh viruses were widespread enough to be of importance. None of the Unix or Linux viruses became widespread – most were confined to the laboratory. (p. 21)
It is interesting to note that more proof-of-concept viruses have been developed since that report was written, but no Linux viral epidemic has occurred as a result. To me, this suggests that the mere existence of such lab viruses does not present a grave danger for the computer user.
On the other hand, I expect that the existence of lab viruses would be very threatening to the AV industry, especially if these viruses are kept hidden by competing companies who consider them proprietary trade secrets. But isn't this one of the problematic consequences of the proprietary software model in general, that knowledge is hidden so that it can be used to increase profits? This is an issue for any kind of proprietary code, not just lab viruses, so it is hard for me to conclude that lab viruses are a particularly unusual or troubling case.
Really, though, I'm not too worried about whether lab viruses are dangerous for the AV industry. I don't think they need me to be worrying about their endeavors - I'm sure they have quite a large investment in doing so themselves.
Incidentally, my remarks have nothing to do with the issue of whether Linux is impervious to viruses or why there are not more in-the-wild viruses for Linux, etc., etc. An interesting discussion of those issues can be found at Rick Moen's linuxmafia page: »linuxmafia.com/~rick/faq/index.p···ge=virus
My argument isn't really about Linux, that's just an example to show why lab viruses in themselves do not seem so dangerous for users from my point of view. And so it doesn't seem so problematic to me that people in the Consumer Reports labs have written some lab viruses in order to compare different AV programs. | |
bluezanetti
Premium
join:2003-10-04
| said by sybille :
My argument isn't really about Linux, that's just an example to show why lab viruses in themselves do not seem so dangerous for users from my point of view. And so it doesn't seem so problematic to me that people in the Consumer Reports labs have written some lab viruses in order to compare different AV programs.
I'd tend to agree, although I believe that reasonable people can disagree on this point and I do see a long term problem of operational malware being created simply for testing purposes. As with the products themselves, it will lead to an escalating arms race of functionality, except now it is in the arena of test malware.
My own position is rather simple - does this first step of synthetic malware creation effectively compromise the remainder of the test protocol and render the results suspect? I don't have enough information to know that, although seeing the relative detection rankings of KAV vs. F-Secure makes me wonder how much internal stress testing and reality checks were applied to the results given that F-Secure is KAV engined. It's a single observation, but it's hanging out there, lurking large, and completely contrary to a priori expectation.
Blue | |
SpannerITWks
Premium
join:2005-04-22
| reply to Cudni
eburger68
My points regarding NG Malware were in response to yours, not CR. You seem to think that NG is sometime way in the future, whereas i'm saying the future can be anything from a millisecond upwards. In other words, as i already said, those NG's could land @ any time, and/or be here already, but they havn't been discovered yet, but they still would be NG. If they are NG then they are, whether they are based on x amount of old code, or not !
Actually the vendors would probably want to keep " their " zoos all to themselves anyway.
Well it depends who wrote that Sexy story, a crap writer wouldn't help of course. If you compare it to for eg one on cars. Super fast, very low fuel consumption, very reliable, lots of novel and new safety features, passes all the available tests and meets and exceeds all standards, comfortable to use, great looking etc etc. That sounds Sexy to me anyway !
The software-buying public from now on is what i was saying, not todays. The more Correct info that is put out there the better, and that goes hand in hand with the above Proper Sexy story.
Re the in house Malware -
If one of the vendors created some super duper ZOO nasty/exploit, or was sent it by a " friend " etc, what would you expect them to do about it. Realise that often parallel nasty, and non nasty etc, inventing does take place, and if they can think it up so can others, and do something about it. Or just sit on it, and wait for the Real parallel nasty to surface and possibly trash peoples PC's, and then react to it ?
Actually maybe the preventative types of codes i'm alluding to would be more suitably applicable to HIPS etc type software than AV etc ! But still, positive action would need to be taken Straight after discovery, whether aquisition of this new knowledge was in house and/or via external means.
So if you look at it that way, laterally, i think it's not very difficult to comprehend the superimposition of incorporating preventative code in various different, but complementary, ways into security products. After all prevetion is Top priority, NOT clean up after the fact !
I wouldn't be at all surprised to learn that reverse engineering, either auotomatically and/or by hands, does go on by vendors looking at competitors Defs. I'm NOT saying they then incorporate these into their Defs, but just as an exercise in " cos we can " to learn how the others do it ! Maybe they don't do it every day, but if they do or ever have, i wouldn't be very surprised.
Also remember the Malware coders spend a Lot of time reverse engineering, as well as forward engineering, so i would say it's in security vendors interests to do whatever they legally can, ie NOT stealing, to be steps ahead, and hence offer users better protection.
Spanner
--
I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks
/SpannerITWks | |
swhx7
Premium
join:2006-07-23
Elbonia
 ·RoadRunner Cable
| reply to Cudni
Eric, I've basically ageed with Steve in this thread, but was trying to articulate exactly why. So I've been trying to piece together the whole argument on your side. Although these are paraphrases at best, I'm placing them in quotation boxes to distinguish who's who.
quote:
1. Testing AV products with lab-made samples is invalid:
(a) For testing detection of "known" viruses, only real ITW ones are a valid test.
(b) For testing detection of new/unknown viruses, the new ones created might not be the same as the ones the real-world authors will create, and might not be real threats, but retrospective testing is necessarily realistic.
Retrospective testing may be better, but that does not mean synthetic testing is invalid. The CR procedure is close to what actual virus writers do, and therefore a realistic simulation. I think others above have made this point too but it gets lost in the barrage.
Presumably, as someone stated above, any "good" virus concepts from the lab will be independently created in the wild soon if they haven't been already. This means that the lab-creation method is a valid test and AV companies ought to be using it. If definitions for the lab creations would not detect real viruses based on the same concepts, then AV products need improvement.
quote:
2. Testing AV products with lab-made samples, in addition to being invalid, is unethical because:
(a) If samples are not released, there can't be independent validation of the testing procedures.
(b) If samples are released to experts, the samples might leak out.
As Steve and others have pointed out, (b) is an argument for careful procedures, but does not support a prohibition. The creator must practice good containment and properly select the recipients. After the transfer, responsibility shifts to properly selected recipients.
quote:
3. Further argument that it's unethical:
(a) If testers of AV productes use lab-made samples and share with AV vendors, AV vendors have to add the samples to their definitions.
(b) If testers use lab-made samples and don't share with AV vendors, then the vendors are justified in adopting the same practice, or have to, to try to counter the new lab-virus population.
(c) In either case, the public would object once they found out that protection they're buying is partly against viruses created by AV vendors, product reviewers or both.
(d) Marketing incentives are such that vendors would have to add the lab viruses to the numbers they advertise.
If testers are going to use the "new virus creation" method, their better ethical choice is to share samples with independent experts but not with vendors. For-profit businesses can't be relied on to give impartial evaluations of anything. It's probably best for the independent meta-reviewers to destroy the samples after evaluating the tests.
Regarding (b), if an AV company is not trying to counter future, unknown (0 day) viruses, then it should advertise as being for known viruses only, and make speed the selling point. If it is tring to protect comprehensively, then whatever it already does against future/unknown viruses should be equally effective against anything testers might come up with. Again this means the CR test is valid.
And (c) ...
said by eburger68 : [W]ould you personally feel fine shelling out money each year for a subscription to an anti-virus product's definitions when those definitions were in part necessary in order to cover viruses that AV researchers, testers, and companies were themselves cooking up in the lab?
How do you think others would react to the same proposition -- that they had to pay for protection from viruses created by parts of the AV industry itself? If creating malware in the lab helps to make the product better, then it's a good thing and businesses and consumers ought to be in favor of it. What would outrage the public is AV vendors releasing viruses, which is a very different proposition. (Personally I don't use anti-virus.) Anyway, as Steve pointed out, this is a business consideration, not ethical.
Finally ((d)), if anti-virus companies feel compelled to inflate their numbers, this is a defect of capitalism, not of testing procedures. If lab-created viruses, if they come into wider use, would have to be added to the numbers, that's a problem for the anti-virus companies, but for the public it would merely give a more accurate picture of what's really going on.
I think lab creation of viruses will be necessary to make better products. If this causes loss of confidence in the anti-virus industry, it's too bad.
The existence of AV as an industry is mainly a symptom of the wide reliance on an OS with Swiss-cheese security and a culture of software requiring root. Exposing weaknesses of security products will only hasten the adoption of a more Unix-like privilege regime. A that's a good thing. | |
eburger68
Premium,MVM
join:2001-04-28
1 edit | swhx7:
You wrote:
said by swhx7 :Retrospective testing may be better, but that does not mean synthetic testing is invalid. As I argued with several earlier posters, it's important to keep in mind burden of proof requirements. We know with retrospective testing that what is tested represents actual, real world threats developed by actual malware authors and released into the wild.
With synthetic lab viruses we don't, not only because it is not within the power of CR to predict the future (as McAfee pointed out), but because no validation has been done on those lab created viruses.
It is CR's burden of proof to establish that those artificially generated samples actually represent credible, potential threats that resemble what real malware authors actually might produce. It's their burden of proof to establish that those virsues are even minimally functional, let alone malicious.
And to meet that burden of proof CR is going to have to supply copies of those samples to an independent body for verification.
So far, they haven't even come close to meeting their burden of proof. Lacking any proof of the validity of their testing, the test cannot be assumed to be valid. Had CR doen proper retrospective testing against Wild List viruses, they wouldn't have this validity problem.
said by swhx7 :The CR procedure is close to what actual virus writers do, and therefore a realistic simulation. I think others above have made this point too but it gets lost in the barrage. You don't know that, I don't know that, and it's a fair bet that even CR doesn't know that. Why? Because CR created an enormous quantity of virus variants in a short period of time, and they have disclosed nothing meaningful about the virus variants they created, how they created them, or even let an independent body validate those newly created variants. See bluezanetti's post above for some of the crucial questions that would need to be answered.
It's important not to make assumptions about what you don't know. (And, btw, the fact that you don't know is no reflection on you; it's a reflection on CR.)
said by swhx7 :Presumably, as someone stated above, any "good" virus concepts from the lab will be independently created in the wild soon if they haven't been already. This means that the lab-creation method is a valid test and AV companies ought to be using it. If definitions for the lab creations would not detect real viruses based on the same concepts, then AV products need improvement. Once again you're assuming that you know what's sitting there in CR's lab -- you don't. If none of us know what the nature of those variants in CR's lab is, then it will be well nigh impossible to say whether those same viruses have been recreated in the wild.
said by swhx7 :As Steve and others have pointed out, (b) is an argument for careful procedures, but does not support a prohibition. The creator must practice good containment and properly select the recipients. After the transfer, responsibility shifts to properly selected recipients. This whole argument falls apart in the last sentence, where you attempt to shift responsibility entirely to another party -- which your argument has to, if it is to have a hope of being credible. By why would the transfer of the samples absolve the original author of responsibility, esp. when that author deliberately and knowingly created those viruses in circumstances that he or she should have known would require transfer to another party/ No, the original author remains responsible all down the line, because it was the author's actions that brought the new viruses into the world and created the circumstances that compelled others to consider accepting their transfer. And with each transfer and new possessor, the risk for escape increases.
said by swhx7 :If testers are going to use the "new virus creation" method, their better ethical choice is to share samples with independent experts but not with vendors.
As a matter of course, this will not be practical, as it flies in the face of scientifically valid testing procedures. What you're essentially arguing is that that testers should test AV products yet refuse to disclose the test bed to the vendors. No vendor is going to be satisfied with that arrangement, and understandably so.
Moreover, withholding lab created viruses from teh vendors only increases the pressure on the vendors to start cooking up similar viruses in their own labs to compensate for or recreate what they've been denied by testing bodies.
said by swhx7 :For-profit businesses can't be relied on to give impartial evaluations of anything. It's probably best for the independent meta-reviewers to destroy the samples after evaluating the tests. Same problems as above, but the proposal to destroy the lab viruses soon after testing does raise the question: how sson after testing? How much chnance for independent experts to examine the test bed must be given before destruction begins?
And, btw, how is one to ensure that a tester or independent expert friendly to one of the vendors doesn't leak the samples but to one or two of the vendors but not the rest of the industry?
said by swhx7 :Regarding (b), if an AV company is not trying to counter future, unknown (0 day) viruses, then it should advertise as being for known viruses only, and make speed the selling point. If it is tring to protect comprehensively, then whatever it already does against future/unknown viruses should be equally effective against anything testers might come up with. Again this means the CR test is valid. You've essentially set an impossible standard. It will always be possible for someone somewhere to create some new form of malware that can slip past existing sigs and detection schemes. They key question, though, will always be: does that malware represent an actual threat to users -- a threat that is actually spreading in the wild.
What you're essentially demanding is that we return to the days before the Wild List, when testers were running tests against exotic zoo viruses created in by some hacker in Thailand and that existed nowhere but on the hard drive of said hacker. But who cares if the AVs miss that one -- it ain't a real threat.
To demand that AVs detect every existant (or potential) piece of malware in the world is not only unreasonable but counterproductive. For more on the pitfalls of this demand and what happened when the AV industry got sucked into an earlier cycle attempting to meet this demand, see the several papers by Joe Wells that have been cited in this thread.
said by swhx7 :If creating malware in the lab helps to make the product better, then it's a good thing and businesses and consumers ought to be in favor of it. And what's the standard for "better" here? Better protection against real threats in the wild; better protection against lab viruses created by the industry itself? Could we even distringuish what protion of the price of an AV defs subscription represented improved protection against real, in the wild threats, and what protion merely represented protection against threats created by teh AV industry itself?
said by swhx7 :What would outrage the public is AV vendors releasing viruses, which is a very different proposition.
Oh, that would surely outrage the public. But I sincerely doubt that the public would be happy with the alternatives:
1) the AV industry creating hundreds of thousands of viable threats in the lab but refusing to release definitions for them;
2) the AV industry creating hundreds of thousands of viable threats in the lab, adding those threats into news defs, and charging users to for those definitions.
As I said several times earlier, the only clear winner in this scenario is the sales deptartments of the AV companies themselves.
said by swhx7 :Anyway, as Steve pointed out, this is a business consideration, not ethical. No, it's ethical consideration, despite what Steve says. It may not be the most burning ethical question in the world at the moement, but an ethical question it is. CR took actions which were not only methodologically unsound and unnecessary, but which constituted practices that it should have known could cause harm to others in a number of different ways -- and that such harm might ultimately actually be to the trustworthiness and integrity of the AV industry itself.
said by swhx7 :Finally ((d)), if anti-virus companies feel compelled to inflate their numbers, this is a defect of capitalism, not of testing procedures. This is a useless form of fatalism that attempts to shield human actors from being responsible for the consequences of their actions. We can do better than to fob off our failures on "capitalism." We can investigate and recognize causes that might feed or set the stage for such decision-making -- causes that were preventable and that had a human hand behind them.
said by swhx7 :If lab-created viruses, if they come into wider use, would have to be added to the numbers, that's a problem for the anti-virus companies, but for the public it would merely give a more accurate picture of what's really going on. No, it would be a problem for the public. Just why is it that you think the public would be so sanguine about being forced to pay for protection against threats created by the AV industry itself?
Here's a hypothetical: let's imagine that the time is three weeks ago, well before anyone had learned of CR's decision to create lab viruses for testing. Let's say that it came to light that members of the AV industry -- testers and researchers primarily, but a few companies as well -- had gotten into the practice of creating and using lab viruses, and that as a consequence AV companies were scrambling to add these into their detections. And, conveniently enough, this situation came to light right about the time that AV corps are announcing yet another round of price hikes for AV subscription renewals.
What do you think the reaction among forum members here at DSLR/BBR would be? I think the reaction is fairly predictable: outrage on a scale that we haven't seen for some time, with a barn-burner of a discussion being filled up with vitriolic deununciations of a corrupt AV industry creating the very threats it was selling protection for. And more than a few wouldn't hesitate to call it a racket, a scam, and demand that the heads of AV companies be thrown in jail.
said by swhx7 :I think lab creation of viruses will be necessary to make better products. If this causes loss of confidence in the anti-virus industry, it's too bad. Well, that's a rather sanguine expectation. Just who did you think would be using those "better products" if a widepsread loss of confidence in the AV industry occurred?
said by swhx7 :The existence of AV as an industry is mainly a symptom of the wide reliance on an OS with Swiss-cheese security and a culture of software requiring root. Exposing weaknesses of security products will only hasten the adoption of a more Unix-like privilege regime. A that's a good thing. Absolustely no one in this thread is opposing research into the "weaknesses of security programs" or the underlying OS. The question has always been been how to do it responsibly and how to do it in a way that didn't cause security companies to focus their time and energy pursuing chimerical threats of their own making.
Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior | |
swhx7
Premium
join:2006-07-23
Elbonia
·RoadRunner Cable
| reply to Cudni
I'm not going to argue most of this. It's a long and interesting thread already and I'll just leave it for readers to decide which points are valid.
But a few particulars:
- My assumptions about the CR method are based on the description in the article which said that they created variants with just enough difference to evade detection, and that it was the kind of technique that actual virus writers use. My interpretation was that the changes were such as to foil signatures without affecting functionality. This was the secondary article linked above; I haven't seen the original.
- The idea that the author is responsible for all subsequent possessors is like the claim that citizens are responsible for everything the government does just because they get to vote once in a while: it is unreasonable to the point of absurdity. If no one could ever contract out of responsibility for anything, the economy and judicial systems would collapse with all the lawsuits and we'd have to be self-sufficient farmers with guns.
- Testing with new variants is appropriate only for products that are claimed to protect against new variants (or suspicious code patterns/behavior, or other threats beyond those currently known). If your concept of anti-virus is something that's supposed to detect only those viruses which are already identified, then we agree that the CR method is bad. But vendors advertise more than that.
- My statement: "whatever [AV software] already does against future/unknown viruses should be equally effective against anything testers might come up with" does not suggest an impossible standard of perfection. It merely suggests that if a company claims to detect, for example, 20% of new variants, and a tester makes some new variants by the same means the virus-writers use, then the product should detect 20% of them. Why not?
- Synthetic testing is valid only if it simulates real threats - you are right about that. I just don't buy the claim that there's no way it can ever do so.
Stepping back to the broader picture, your arguments come across as "the status quo must be protected". But the status quo is a seemingly pointless whack-a-mole race that's not getting any better. This little tempest about testing methods is just a diversion from the fact that we need a whole new approach to countering malware. | |
eburger68
Premium,MVM
join:2001-04-28
1 edit | swhx7:
You wrote:
said by swhx7 :- My assumptions about the CR method are based on the description in the article which said that they created variants with just enough difference to evade detection, and that it was the kind of technique that actual virus writers use. My interpretation was that the changes were such as to foil signatures without affecting functionality. This was the secondary article linked above; I haven't seen the original. But even those measely few details aren't enough. Until bonified experts get a chance to validate the samples, we just don't know, and we shouldn't be making assumptions about the qualities, nature, and performance of those 5500 samples.
said by swhx7 :- The idea that the author is responsible for all subsequent possessors is like the claim that citizens are responsible for everything the government does just because they get to vote once in a while: it is unreasonable to the point of absurdity. If no one could ever contract out of responsibility for anything, the economy and judicial systems would collapse with all the lawsuits and we'd have to be self-sufficient farmers with guns. That's really not a useful analogy, because the standard for holding someone responsible for kicking off a potential chain of events is, "Was it foreseeable that X, Y, and Z would happen?"
Given the level of complexity at which governments operate and the our recognition that governments must inevitably deal with a whole range of unforeseen events, and that governments are comprised of thousands if not millions of fallible human beings, it isn't reasonable to hold citizens responsible for every last action of a government because there was no possible way for the citizens to have foreseen all the potential consequences of electing a particular government.
The situation that we're dealing with is quite a bit different.
Q. If I create lab viruses for use in a test designed to be scientifically valid, is it foreseeable that that independent experts would be required to validate the samples, thus compelling me to redistribute my creations?
A. Absolutely that is foreseeable by anyone with a rudimentary working knowledge of testing and what makes a test valid.
Q. If I re-distribute those viruses to third parties, is it foreseeable that -- viruses being what they are, and human being fallible creatures -- that further re-distribution might occur and an escape might occur at some point?
A. Yes, absolutely. Even if I don't regard the chances of an escape to be very high, it doens't take too much thinking through to realize that an escape remains in the cards as one possible outcome.
At this point, it is reasonable to hold me responsible for the consequences of actions that I could have foreseen and anticipated.
said by swhx7 :- Testing with new variants is appropriate only for products that are claimed to protect against new variants (or suspicious code patterns/behavior, or other threats beyond those currently known). If your concept of anti-virus is something that's supposed to detect only those viruses which are already identified, then we agree that the CR method is bad. But vendors advertise more than that. Any product that touts its heuristic or behavioral detection capabilities is essentially touting its ability to catch new, previously unknown viruses. And as been established by a number of AV authorities, there are other ways to test the effectiveness of heuristics.
said by swhx7 :- My statement: "whatever [AV software] already does against future/unknown viruses should be equally effective against anything testers might come up with" does not suggest an impossible standard of perfection. It merely suggests that if a company claims to detect, for example, 20% of new variants, and a tester makes some new variants by the same means the virus-writers use, then the product should detect 20% of them. Why not? Because in this case your theoretical 20 percent is meaningless. Twenty percent of what? Total new threats in the wild? But your lab viruses isn't part of the wild.
Twenty percent of the lab's daily creation? The tester's? Twenty percent of the total number of lab viruses created today (but how would we know that that number is?).
As I said, there's always going to be someone out there with a piece of malware that is capable of evading current detetion engines and sigs. The key question, though, will alwasy be: is the threat actually a threat?
said by swhx7 :- Synthetic testing is valid only if it simulates real threats - you are right about that. I just don't buy the claim that there's no way it can ever do so. Fine. But the burden of proof is on those who contend that these types of lab variants would yield useful results obtainable thhrough no other means. Thus far I have't seen anyone come close to meeting it; certainly CR hasn't.
said by swhx7 :Stepping back to the broader picture, your arguments come across as "the status quo must be protected". But the status quo is a seemingly pointless whack-a-mole race that's not getting any better. This little tempest about testing methods is just a diversion from the fact that we need a whole new approach to countering malware. I make no claim to defend the entire status quo within the field of information security or even the state of AV testing and research. There are plenty of things that need fixing and changing within these worlds. I simply ask that those who would step past a prohibition that was specifically erected to forestall certain bad things from happening prvide a sufficient justification for doing so. Again, CR hasn't been able to do this.
Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior | |
Steve
I'm a PC, so shut up
Consultant
join:2001-03-10
Yorba Linda, CA
| reply to eburger68
said by eburger68 :It is CR's burden of proof to establish that those artificially generated samples actually represent credible, potential threats that resemble what real malware authors actally might produce. It's their burden of proof to establish that those virsues are even minimally functional, let alone malicious. We're not talking a court of law, we're talking about advice to the consumer. The industry can whine all it wants about a "burden of proof", but most others will be satisfied to accept CR's tests as an honest effort at evaluation.
Besides: if (for whatever reason), I accept their findings, then the burden is on the A/V industry to show that it's not valid. To demand that AVs detect every existant (or potential) piece of malware in the world is not only unreasonable but counterproductive. This is a fair point: malware is not exactly about bits, but about intentions, and reading intentions into a pile of bits you've never seen before is a very hard problem.No, it's ethical consideration, despite what Steve says. It may not be the most burning ethical question in the world at the moement, but an ethical question it is. CR took actions which were not only methodologically unsound and unnecessary, but which constituted practices that it should have known could cause harm to others in a number of different ways -- and that such harm might ultimately actually be to the trustworthiness and integrity of the AV industry itself. Says the industry, but I don't buy it. Ethical considerations are normally based on whether other people are harmed, and the industry seems to be breathless about all this "what might happen down the line?" considerations.
As if they're so smart that only they know that keeping a handle on this by responsible parties is impossible, and they're doing us a public service by forestalling this evil behavior.
I don't buy it.
Steve
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site | |
SpannerITWks
Premium
join:2005-04-22 | reply to Cudni
eburger68
Did you just forget to reply to me, or ?
Spanner | |
eburger68
Premium,MVM
join:2001-04-28
| reply to Cudni
SpannerITWks:
Sorry, but I've had an enormous amount of work to do today (yes, I normally work weekends). Given my short schedule I had to pick and choose whom I would respond to. swhx7's post was new and esp. meaty, whereas you and I had already had a few go-arounds.
No slight or offense intended was intended. If I can work in a response over the next day or so, I will.
Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior | |
eburger68
Premium,MVM
join:2001-04-28
1 edit | reply to Steve
Steve:
You wrote:
said by Steve :Besides: if (for whatever reason), I accept their findings, then the burden is on the A/V industry to show that it's not valid. Sorry I can't respond to everything, but this bit begs for a quick response.
Burden of proof isn't for you to determine; it's determined by the the testing process itself. You might as well say:
"If I accept the idea that little green men from an invisible planet have visited Earth, are responsible for kidnapping human beings and processing them into cosmic dog food, and that a few captured little green men are being held in Area 51, then the burden of proof is on my critics to prove me wrong."
You make the claims, you're responsible for backing up those claims with sufficient evidence and reasoning.
Apologies to all for not being able to respond to all the intelligent points that folks have made, but I've got too much work waiting to be done. It's been an interesting and productive conversation nonetheless.
Best,
Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior | |
Steve
I'm a PC, so shut up
Consultant
join:2001-03-10
Yorba Linda, CA
| said by eburger68 :Burden of proof isn't for you to determine Oh yes it is: if you want to convince me, I get to decide what counts.
I might be mistaken, I most certainly may be unscientific, but this is not about deciding whether God exists or whether to approve the new miracle drug: it's about where I spend my money for A/V products. The consumer decides.
Obviously, I can't decide on this for others, and you'd be right if you're talking about a scientific paper, but the guy with spending money in his pocket makes this call.
Steve
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site | |
bluezanetti
Premium
join:2003-10-04
| said by Steve :
Obviously, I can't decide on this for others, and you'd be right if you're talking about a scientific paper, but the guy with spending money in his pocket makes this call.
I guess that's why it seems to matter a bit to me.
To my own eye, there are internal inconsistencies in the results provided, and that's all I have to go on. Failing to get a sense of some clarification, it appears to me to be just another example of poorly executed intellectual self-abuse posing as an objective evaluation.
And, recall, I'm a subscriber and I generally give CR the benefit of any doubt.
Blue | |
SpannerITWks
Premium
join:2005-04-22
| reply to Cudni
eburger68
So you're a 24/7-365 kinda guy 2 hey !
Nice to hear you say what you did, and no offense taken, was just wondering. OK i'm on standby.
Thanx,
Spanner
--
I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks
/SpannerITWks | |
Steve
I'm a PC, so shut up
Consultant
join:2001-03-10
Yorba Linda, CA
| reply to bluezanetti
said by bluezanetti :To my own eye, there are internal inconsistencies in the results provided, and that's all I have to go on. Failing to get a sense of some clarification, it appears to me to be just another example of poorly executed intellectual self-abuse posing as an objective evaluation. ... which is a perfectly fair judgement, one that may well be the correct one.
I hope nobody thinks that I'm trying to claim that CR got it right: I have no idea, I'm not an expert on this and haven't read the whole report.
But I smell BS in the A/V industry reaction, and in my mind, that sheds light on something I otherwise wouldn't have thought much about.
Steve
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site | |
|
