how-to block ads
|
HMS1
join:2006-01-14
Austin, TX
2 edits | reply to Cudni
Re: Our unique antivirus testing: How we did it
Maybe the point in my first paragraph was unclear. What I meant was: adding some new viruses does not make any significant difference to internet (in)security. They don't pose any danger merely by existing. Viruses by definition cannot do anything without user interaction.
If a user runs a virus, the harm done depends on the particular virus. But how would the differences between viruses make any difference in policies or defenses? Policies must be against any/all untrusted code, without knowing in advance what it will be. And defenses must be against any possible virus, not only a "known" list. Signature-based anti-virus is a dead end.
Putting it another way, a user's risk is the same with or without a new batch of viruses being loose. With or without any addition to the virus pool, the potential harm includes whatever can be done on the user's account, and the spectrum of what's in the wild must be assumed to be whatever the authors can, in principle, create. These factors do not change with addition of new viruses.
The problem is users running untrusted code, not whether the range of viruses is (big number) or (big number + small number). | |
eburger68
Premium,MVM
join:2001-04-28
| reply to Cudni
HMS:
This is just a variant of the "guns don't kill people, people kill people" argument. This time it's, "viruses don't pose threats; people's executing of viruses poses threats," as if people were some optional, extraneous component of the threat environment.
People being what they are -- which is to say fallible, gullible, ignorant, lazy, and prone to error -- it can be expected that the introduction of new viruses into the environment does increase the risk of people executing those viruses, if even accidentally.
One can blame the people or users for being lazy, ignorant, and all the things that people can tend to be in their more error-prone modes of being, but the fact remains that the introduction of new viruses into an environment where fallible users (and researchers) can access them increases the risk of harm being done.
And, by the way, in making this argument I am most certainly not slighting efforts to reduce the opportunities or chances for users to run untrusted code. We can do both: keep less open gasoline lying around AND keep people away from the gasoline.
Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior | |
AB
Premium
join:2006-04-04
Leesburg, VA
| said by eburger68  :. . People being what they are -- which is to say fallible, gullible, ignorant, lazy, and prone to error -- Hey! Watch it, buddy! I resemble that remark! 
. . it can be expected that the introduction of new viruses into the environment does increase the risk of people executing those viruses, if even accidentally. (sic)
And, by the way, in making this argument I am most certainly not slighting efforts to reduce the opportunities or chances for users to run untrusted code. We can do both: keep less open gasoline lying around AND keep people away from the gasoline.
Eric L. Howes Again, Amen!
I run a secure PC, I use common sense when I surf. And it is, btw, 'common sense' (as well as mathematics) that says I am much more likely to be exposed to a virus if there are are 105,500 of them out there than I would be if there were only a couple of dozen!
And I believe there is a saying-- "The road to Hell is paved with good intentions."  | |
HMS1
join:2006-01-14
Austin, TX
| reply to Cudni
It's not a "blame the user" argument. I'll try to say it another way (and btw I was revising my 2nd post, trying to be clearer, just when you were posting).
The point is, if the addition of a new virus makes a difference in the administrator's or protection vendor's strategy, then the strategy is inadequate in the first place. We already know the outer limits of what viruses can do (viz. what the user account allows), and we already know how they get into the LAN or local system (email, junkware etc.). The only difference a new virus makes is some new variation of what they do to the system once infection is already underway.
Following up on the guns analogy, it's as if the whole approach to prevention of shootings is listing all the various types of bullets, and then complaining if someone makes a new kind of bullet, and saying it increases the risk. Instead you just have to keep the guns out of the courthouse or airport. Then it doesn't matter what kind of bullets they use. | |
HMS1
join:2006-01-14
Austin, TX
| reply to AB
said by AB :And it is, btw, 'common sense' (as well as mathematics) that says I am much more likely to be exposed to a virus if there are are 105,500 of them out there than I would be if there were only a couple of dozen!
The danger increases with the number of viruses only if your behavior is careless. Instead focus on the possible ways they can get into the computer and be executed with privileges. The number of ways this can happen is far smaller than the number of viruses, and the list of infection routes remains far more consistent than the spectrum of viruses, and it is much more in your control. | |
AB
Premium
join:2006-04-04
Leesburg, VA
| said by HMS1 :The danger increases with the number of viruses only if your behavior is careless. Instead focus on the possible ways they can get into the computer and be executed with privileges. The number of ways this can happen is far smaller than the number of viruses, and the list of infection routes remains far more consistent than the spectrum of viruses, and it is much more in your control. HMS, you may well be correct in what you say. However, I look at this way-- It doesn't matter to me if I live in a secure, gated, guarded, moated castle. When I open the curtains to look outside, my preference is to see green pastures and children playing, not a teeming horde of ne'er-do-wells looking for a way to breach the ramparts!
So when I hear about researchers thinking up 5500 new ways to infect me, I don't like it! Let 'em do their research on the OLD stuff. Now, that may not be very scientific, nor am I saying it's right, but that's my opinion anyway! | |
IBK
join:2003-06-20
Austria
| reply to Cudni
av-test.org and av-comparatives provide retrospective tests to see how well av products protect against new _real-world_ viruses/malware. (and the results etc. can be seen by anyone for free without having to pay a subscription fee).
btw, (something i wanted to tell since long time) remember that the test of CR was done most probably months ago, as usually printed magazines (articles etc.) are prepared around 30 days prior. considering that they engaged other peoples to do the test and that they needed to provide them enough time for doing this and then to write the article, the test must been had done months ago. So they could have - instead of creating new variants of old viruses - making a retrospective test which would deliver valid results. Of course that is more time consuming, but they would not need to create new virus variants.
so they have now those variants on a CD in a safe. ok. AV vendors of course would like to know what kind of files were used in this test, in order to see if the results are true (even if from the method etc. invalid). Now the dilemma: if they do not give the samples to the AV vendors, they can not check if the samples works etc., and if they send the new variants to the AV vendors, the AV vendors are obligated to add those variants to their databases (= and that is not good, because if AV vendors have to add the viruses created for testing reasons, the scanning speed may be affected. So for who is all this of help? Not for the users and not for the vendors. Probably just for CR as they get publicity and new paying subscribers).
sidenote: they state that there are no independent tests to measure how well av software is against new threats, which is absolutly wrong. If they would know a little bit on this materia, they would know or have read about methods to do such tests and would also know or see (by using e.g. google) that such tests exist (like I said before provided by av-test.org which publishes those results on various magazines around the world and av-comparatives which has the results/report publicly available online for free).
p.s.: this is just my opinion. | |
Steve
I'm a PC, so shut up
Consultant
join:2001-03-10
Yorba Linda, CA
1 edit | reply to eburger68
said by eburger68 :This is just a variant of the "guns don't kill people, people kill people" argument. No, it really isn't.
Just because it's easy to accidently infect the world with ebola doesn't mean that nobody has the ability to create a virus test environment with proper precautions against leakage: I'm quite sure that both you and I would be able to construct such an environment.
I am not arguing for the value of synthetic viruses like this (I don't know, I'm not an expert), or that CR actually did so in a safe manner, but it's not out of the question that they were aware of this issue and retained the proper experts to make sure that this didn't happen.
It's perfectly fair to object to the testing methodologies on their merits, and to urge others not to play with fire, but this discussion about the "ethics" of this kind of testing smells incredibly self-serving: "We're the only ones who know how to do that"
Steve
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site | |
eburger68
Premium,MVM
join:2001-04-28
2 edits | Steve:
You wrote:
said by Steve :Just because it's easy to accidently infect the world with ebola doesn't mean that nobody has the ability to create a virus test environment with proper precautions against leakage: I'm quite sure that both you and I would be able to construct such an environment. And when independent bodies demanded copies of those viruses in order to validate the test bed, which is a basic requirement of scientifically valid testing, what then? If you provide them then you have effectively become a virus distributor. And can you vouch for the safety of the parties to whom you distributed the samples?
If you refuse, then your test's results are invalid. End of story. In which case, just what was the value of creating those viruses in the first place -- esp. given that there is no shortage of viruses and other malware to test against. Nor is there a lack of methodological alternatives to accomplish the same goals with real viruses in the wild.
And, by the way, we haven't even broached the subject of how Consumer Reports internally validated those lab-created viruses? Did they execute them in order to verify that the changes they had made to the pre-existing variants hadn't rendered the synthetic variants non-executable or the payload null? Did they diligently execute every single one of those 5500 new viruses? These are important questions because if CR failed to validate the viruses internally, then they have no reason to know that they weren't testing against non-viruses -- i.e., non-threats, which the tested AV apps would be perfectly justified in NOT detecting becaue the threats weren't real.
said by Steve :I am not arguing for the value of synthetic viruses like this (I don't know, I'm not an expert), or that CR actually did so in a safe manner, but it's not out of the question that they were aware of this issue and retained the proper experts to make sure that this didn't happen. But the point here is that even the recognized experts in the field strongly advise against this type of behavior. And as has been pointed out now, the issue of lab security goes beyond one's own security precautions, but the precautions of those to whom one might be obligated to share these syntethic viruses in order to establish the scientific credibility of the testing.
said by Steve :It's perfectly fair to object to the testing methodologies on their merits, and to urge others not to play with fire, but this discussion about the "ethics" of this kind of testing smells incredibly self-serving: "We're the only ones who know how to do that" This is a bit disappointing, Steve. You've been in these forums as long as I have, and on more than a few occasions you've drawn on your own impressive professional knowledge and experience to argue emphatically that such-and-such action, behavior, process, or decision was muddle-headed, improper, dangerous, or even unethical. And you have been quite justified in doing so, given the depth of experience and expertise that you bring to the table. To so casually dismiss the judgments of recognized experts in the AV field is not what I would have expected. And I don't expect you or the other established security professionals would take kindly to having their own views characterized and dismissed in this manner.
To return to the original issue -- the full issue -- which is whether CR had any justification -- be it practical, methodological, or ethical -- to create 5500 new viruses for testing, I hope that it is becoming clear that even if one considers CR's actions but a minor or negligible transgression, that there simply was no practical or methodological justification for them. Moreover, in order to establish and defend the methodological integrity of the testing, CR would unavoidably risk compounding its ethical lapses. These aren't neatly separable issues.
Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior | |
Steve
I'm a PC, so shut up
Consultant
join:2001-03-10
Yorba Linda, CA
| said by eburger68 :And when independent bodies demanded copies of those viruses in order to validate the test bed, which is a basic requirement of scientifically valid testing, what then? If you provide them then you have effectively become a virus distributor. And can you vouch for the safety of the parties to whom you distributed the samples? I am unmoved by this argument: I think it's possible to distinguish between a known virus researcher and Ivan J. Trojanovic - that kind of distinction goes on all the time by those with common sense.
Data point: DSLR has a "Malware Archive" forum where people post samples of badware, but only those on a trusted list can download them for research. I'm on that list, you probably are too. Is Justin "distributing" malware? Or just using his head in an effort at public service?If you refuse, then your test's results are invalid. I agree that reproducible tests are not really valid in the scientific study sense, but the consumer won't care about much of that: many people trust CR to be unbiased (which I believe they are here) and expert (which they may not be), and are happy to just accept their conclusions.
When I'm shopping for a bbq grill or a dishwasher, I usually get what they like without digging in too much to just how they got their answer. They are smart about this, I'm pretty dumb, and am better off in the long run to just defer to their judgement.
But yes: if nobody can reproduce their results, then it really casts doubt on just how they filled in those little circles.But the point here is that even the recognized experts in the field strongly advise against this type of behavior. Sure, but advice that applies generally does not always apply specifically in every possible case.
Example: The recognized experts also strongly urge people to run antivirus software on their desktops, but I never have. I'm very well educated in virus infection vectors, am extraordinarily careful, and have never had an infection in almost 30 years of using a computer. I'm not the only in this forum who believes this.
This doesn't mean that I'm "evidence" against the expert advice, it doesn't mean that I recommend others take this course, and it doesn't mean that I object to the advice (I don't - I urge it strongly of others).
It just means that there are corner cases in most maxims. And as has been pointed out now, the issue of lab security goes beyond one's own security precautions, but the precautions of those to whom one might be obligated to share these synthethic viruses in order to establish the scientific credibility of the testing. The validity of the CR testing will stand or fall on the merits of the methodology, not the ethics. I'm so confident that the smart people like you will point out why they have reached an unwarranted conclusion that I simply do not care about the "ethics" issue. You'll demolish them without it, so to me it's just a distraction.And I don't expect you or the other established security professionals would take kindly to having their own views characterized and dismissed in this manner. I dismiss them because they are unnecessary: you can fully make your case without the self-serving don't-try-this-at-home arguments.
Steve
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site | |
funchords
Hello
Premium,MVM
join:2001-03-11
Washington, DC
 ·Verizon Online DSL
 ·Skype
1 edit | reply to eburger68
said by eburger68 :And when independent bodies ... The Consumers Union is an independent body. The assumption up front is that they are objective.
I would be more inclined to ask an independent body to confirm tests that might be subject to bias.
said by eburger68 :...demanded copies of those viruses in order to validate the test bed, which is a basic requirement of scientifically valid testing, what then? If you provide them then you have effectively become a virus distributor. And can you vouch for the safety of the parties to whom you distributed the samples? If you refuse, then your test's results are invalid. End of story. There are plenty of examples in the medical community where high-risk or limited quantity test items are not shared. That fact doesn't make the testing invalid.
Medical and psychological studies involve groups of individual subjects, with a description of why these individuals were interesting from a test perspective. These studies are not invalid.
said by eburger68 :To return to the original issue -- the full issue -- which is whether CR had any justification -- be it practical, methodological, or ethical -- to create 5500 new viruses for testing I say yes, you say no.
The AV Comparitor web-site I visited last night approached the "0 day" threat differently. To summarize, they used a version of the AV product that was several weeks old, and tested it using more recent viruses that could not have been added into definition updates yet.
That's a plausible approach, but does suffer the very same "fortune telling" sin that the Avert Blog was complaining about.
I haven't read the CR article, yet. But if the results are very different than AV Comparitor's results, then hopefully this starts a debate that improves the latter. CR isn't going to review AV products in its next issue, but AV Comparitor will.
said by eburger68 :in order to establish and defend the methodological integrity of the testing, CR would unavoidably risk compounding its ethical lapses Every time I see someone question the ethics of the Consumers Union, I have an emotional response. (And it's not just you, the industry blogs are all doing it.) This is an organization that has decades of behavior beyond reproach. The industry is walking on their customer's holy ground -- they would be well advised to behave themselves.
To me, there is no ethical question here. The question is whether there was an effective methodology. Did they make a mistake?
It is possible to have an ineffective methodology, and to have taken uninformed dangerous risks, and still be ethical.
I think their methodology is plausible and that the "danger" exists but is being overblown. For me, I think it's done and it is interesting. I'm not sure it is the same choice that I would make, had I been CR's tester.
The highly emotional response (ranting) by the AV industry is not serving them at all. I would expect them to recognize that Consumer Reports reviews lawn mowers and hair dryers and everything else, and might not return to a set of products for several years.
As a reader, I expect the Consumers Union to come up with a reasonable way to compare one product against another, describe what they did, and to report what they found. As far as I'm concerned, they did that. I think it's a benefit to everyone that they took a different approach. And I think the jury is out as to whether their method, and their results, turn out to be right.
--
Robb Topolski -= funchords.com =- Hillsboro, Oregon USA
~ Keeper of the D-Link FAQ ~ Did you Search? ~ More features, Free! Join BBR! ~ | |
eburger68
Premium,MVM
join:2001-04-28
| reply to Steve
said by Steve :I am unmoved by this argument: I think it's possible to distinguish between a known virus researcher and Ivan J. Trojanovic - that kind of distinction goes on all the time by those with common sense. And I, too, would expect CR to be able to make the distinction. But the point is, once they redistribute, those viruses are effectively beyond their control.
said by Steve :Data point: DSLR has a "Malware Archive" forum where people post samples of badware, but only those on a trusted list can download them for research. I'm on that list, you probably are too. Is Justin "distributing" malware? Or just using his head in an effort at public service? Bad comparison. Justin didn't create any malware and isn't re-distributing any malware that wasn't already in the wild -- that's the key difference. If he or anyone else affiliated with DSLR/BBR did start creating malware on their own and re-distributing it for the sake of prodding AV companies to bolster their ability to detect variants (which I wouldn't expect DSLR/BBR to do, obviously) then the same objections would apply.
said by Steve :Sure, but advice that applies generally does not always apply specifically in every possible case. Two points:
1) Here the experts are urging the advice on each other -- the very folks one would expect to be the exceptions to the rule.
2) In order to argue for an exception to the rule, one would have to mount a fairly strong case that the exception was justified on the grounds that the sought-after results were practically obtainable through no other means and that the potential risks were far outweighed by the unique benefits that would incur. In this situation CR can't even come close to making such an argument. Their only possible justification was expedience, to say noting of their own ignorance of established AV testing methodologies.
said by Steve :Example: The recognized experts also strongly urge people to run antivirus software on their desktops, but I never have. I'm very well educated in virus infection vectors, am extraordinarily careful, and have never had an infection in almost 30 years of using a computer. I'm not the only in this forum who believes this. Again, the analogy/comparison doesn't apply, because your example involves general advice given to the general population. In this situation, the recognized authorities came to the conclusions they did regarding the ethical behavior of other experts, not the general population (my dad is an unlikely target for such admonitions, as it's rather unlikely he'd ever feel the urge to pull together a malware zoo and begin experimenting on it).
said by Steve :The validity of the CR testing will stand or fall on the merits of the methodology, not the ethics. I'm so confident that the smart people like you will point out why they have reached an unwarranted conclusion that I simply do not care about the "ethics" issue. You'll demolish them without it, so to me it's just a distraction. And the fact that the creation of 5500 new viruses lacks any practical or methodological justification makes the ethical lapse even more glaring. If they could mount a credible defense that such steps were necssary to allow some unique and innovative testing to proceed -- testing that might shed real light on the capaibilities of the tested AV apps and/or the behavior of malware -- then we might be looking at one of those ethical "corner cases."
But we're not. We're just looking at some run-of-the-mill irresponsible behavior by an otherwise respected testing entity that should have known better -- and all in the name of some rather unimpressive testing that shed light on little of anything except the organization's own ignorance and carelessness.
Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior | |
Steve
I'm a PC, so shut up
Consultant
join:2001-03-10
Yorba Linda, CA
| said by eburger68 : And I, too, would expect CR to be able to make the distinction. But the point is, once they redistribute, those viruses are effectively beyond their control. I'm just unmoved. We're not talking about nuclear secrets or actual ebola virus: they're just bits, and we have no evidence that they have gone one walkabout. It's a big "so what?" to me.Bad comparison. Justin didn't create any malware and isn't re-distributing any malware that wasn't already in the wild -- that's the key difference. Yes, this is a difference, but it's not that big of a one to me.1) Here the experts are urging the advice on each other -- the very folks one would expect to be the exceptions to the rule. Industries do this all the time, and it means nothing about the genuine-ness of their motives.
You have to have a barber's license to cut hair in California, and this "imposing a requirement on themselves" was ostensibly done to protect the consumer, but was actually done to increase the barriers of entry into the field and to reduce competition.
Big payroll companies are behind the push for expensive SAS70 audits (technology audits by CPAs, which pretty much fills you in on their utility), mainly to impose costs on the little guys. This is an industry imposing rules on itself in order to increase the barriers of entry into the field and to reduce competition.
I don't believe that this is behind the sentiment going on here — your heroic efforts are informed by motives which are beyond reproach — but industries tend to look at things from their own point of view. They may not be the same as mine.And the fact that the creation of 5500 new viruses lacks any practical or methodological justification makes the ethical lapse even more glaring. If they could mount a credible defense that such steps were necssary to allow some unique and innovative testing to proceed -- testing that might shed real light on the capaibilities of the tested AV apps and/or the behavior of malware -- then we might be looking at one of those ethical "corner cases." Whether you're right or wrong on this, many people see the hue and cry about the ethics as a smokescreen, and it hurts the cause to focus on it. It just smells self serving to me.But we're not. We're just looking at some run-of-the-mill irresponsible behavior by an otherwise respected testing entity that should have known better -- and all in the name of some rather unimpressive testing that shed light on little of anything except the organization's own ignorance and carelessness. ... and whether the objection on ethical grounds is well founded or not, that casts no light on whether CR was actually competent or incompetent in this matter.
I am more than willing to accept that they have received bad advice from their "experts", have gotten in far beyond their competence, and did an all-around bad job.
The "ethics" are just a side show.
Steve
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site | |
eburger68
Premium,MVM
join:2001-04-28
| reply to funchords
Funchords:
You wrote:
said by funchords :The Consumers Union is an independent body. The assumption up front is that they are objective. I would be more inclined to ask an independent body to confirm tests that might be subject to bias. When evaluating the validity of testing, there is more to consider than just "bias" and "objectivity" -- words which are too often pushed into service to stand in the place of other considerations. One can be independent, perfectly unbiased, and as "objective" as one could ever hope to be and still produce hopelessly invalid, meaningless tests. Lack of bias is no guarantee of the quality of a test.
said by funchords :There are plenty of examples in the medical community where high-risk or limited quantity test items are not shared. That fact doesn't make the testing invalid. And here we are talking about anti-virus testing of mass consumer products, testing that could have and should have been performed with the plethora (nay, the tidal wave) of valid samples in the wild. The kinds of medical studies that you're referring to are the exception, not the rule, and are conducted nonetheless because there is no other practical way to obtain the uniquely valuable data available only in such rare circumstances. CR's testing of AV products doesn't even come close to fitting those criteria, esp. in light of the alternative methodologies that had been known and practiced for years as well as the wide availability of "in the wild" malware to test against.
said by funchords :Medical and psychological studies involve groups of individual subjects, with a description of why these individuals were interesting from a test perspective. These studies are not invalid. Again, bad comparison. The studies you're referring to are constrained by unique characteristics of the subjects -- human beings, whose qualities obviously forego the question of handling like common lab samples. And to compensate for these limitations, the medical and psychological communities set up and run numerous similar experiments on various populations while striving to control for the inevitable differences among their subjects. As with your earlier example, what you're describing are unavoidable constraints -- constraints which aren't a part of the situation we're dealing with here.
But even if we were to accept that the comparison was apt, CR has done nothing to release data on the virus variants that they created. Nothing even resembling the kind of data usually supplied for the subjects of medical and psychological studies has been forthcoming from CR, so even on the basis of this comparison CR's AV testing must be considered invalid.
said by funchords :The AV Comparitor web-site I visited last night approached the "0 day" threat differently. To summarize, they used a version of the AV product that was several weeks old, and tested it using more recent viruses that could not have been added into definition updates yet. That's a plausible approach, but does suffer the very same "fortune telling" sin that the Avert Blog was complaining about. Not at all, because this kind of retrospective testing is conducted actual viruses and malware in the wild, not against lab malware that one organization speculated might be in the wild some day.
said by funchords :I haven't read the CR article, yet. But if the results are very different than AV Comparitor's results, then hopefully this starts a debate that improves the latter. CR isn't going to review AV products in its next issue, but AV Comparitor will. How would AV Comparatives benefit from CR's testing when no crticial data about this test bed of lab viruses has been disclosed by CR to say nothing of the samples themselves? Again, the dilemma: to be useful in any way to the wider AV community, CR would have to start distributing those samples, which fairly demolishes the claim that these things were guaranteed to stay within the safe confines of one lab.
said by funchords :Every time I see someone question the ethics of the Consumers Union, I have an emotional response. (And it's not just you, the industry blogs are all doing it.) This is an organization that has decades of behavior beyond reproach. The industry is walking on their customer's holy ground -- they would be well advised to behave themselves. A testing entity is not "holy ground," and to regard a testing entity as anything like that does a disservice to the testing entity as well as the public it is supposed to serve.
--
Microsoft MVP
Sunbelt Software
Spyware Warrior | |
eburger68
Premium,MVM
join:2001-04-28
1 edit | reply to Steve
Steve:
For the sake of clarity, let's narrow the focus of our disagreement to this one statement which, so far as I can tell, motivates and undergirds your entire argument:
quote:
It just smells self serving to me.
How is the demand that entities that presume to do quality, responsible AV testing follow a simple ethical rule that one shalt not create malware onesself "self serving"?
How is the demand that those who would presume to test AV products against "unknown" variants take the safer, saner, and more scientifically valid approach of conducting retrospective testing "self serving"?
"Self serving" in what way? Because to prefer retrospective testing over the creation of lab viruses would impose some kind of onerous burden on prospective new testers so that the AV industry could keep the testing game all to itself? Is retrospective testing really THAT onerous and difficult to conduct?
Because lab-created viruses might produce more valid results that would allow non-standard products not favored by industry insiders to rise to the top of test results? Is Bit Defender not an established player in the AV industry? How about Kaspersky? KAV placed a respectable third in this testing, and even they protested.
The AV industry has an extensive body of literature on testing, and, if anything, the recommendations and admonitions you'll find there-in often make testing easier to perform as well as more reliable. Indeed, CR could have saved itself quite a lot of headache and expense (and given its readers more reliable test results) had they not resorted to synthetic virus creation.
Really and truly I don't get the "self-serving" charge here. If you're going to make it, you ought to at least be able to explain what the industry hopes to gain by insisting on such an ethical standard. Thus far, I haven't seen anything beyond a flip, empty accusation.
Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior | |
eburger68
Premium,MVM
join:2001-04-28
1 edit | reply to funchords
Funchords:
One other thought, if you'll allow me. You wrote:
said by funchords :As a reader, I expect the Consumers Union to come up with a reasonable way to compare one product against another, describe what they did, and to report what they found. As far as I'm concerned, they did that. I think it's a benefit to everyone that they took a different approach. And I think the jury is out as to whether their method, and their results, turn out to be right. So, let's take this idea and run with it -- and imagine the kind of malware and anti-malware landscape that results.
Let's say for the sake of discussion that we all decide that it is perfectly OK for testing entities like CR, AV-Comparatives, Virus Bulletin, as well as PC Magazine and PC World to start creating and using lab viruses in their testing. And let's assume that this practice becomes rather commonplace in AV testing -- perhaps not universal, but routine enough that it doesn't shock those in the AV industry.
Now, knowing that all these testing entities were in possession of thousands upon thousands of virus variants, would you or would you not want those testing entitities to turn over the samples to the AV companies so that the companies could generate signatures for those variants and play around with them for developmental purposes?
Signatures would be important, because although heuristics, HIPS, and behavioral analysis of malware is improving, we have yet to see a preventative technology emerge (save that trusty pair of wire cutters) that can guarantee zero infections, zero infestations in a networked environment. There will be a market for remediation tools for some time to come.
So, the AV companies start getting an influx of lab viruses to shove into their definitions, which then swell with artificial malware that everyone hopes (but no one can guarantee) would stay within the confines of the lab.
Or maybe they don't, for whatever reason -- perhaps the testing entities refuse on principle to supply any more than general descriptions of how their own lab variants were created.
Would you then allow that the anti-virus companies themselves would be justified in creating lab viruses, if only to attempt to replicate the kinds of viruses that they know exist somewhere in the world and which, some cases, might have been leaked to less responsible entities, raising a real question as to whether such lab viruses aren't fair game for the definitions that AV companies create?
At this point I ask our readers to consider what their reaction would be were they to find out that the yearly anti-virus subscriptions being sold to consumers functioned, in part, to grant them access to definition updates for viruses and malware that the AV companies themselves -- along with AV testing entities like CR -- were cooking up in their own labs?
Can you imagine the hue and outcry? We already have a few skeptics around here who aren't thoroughly convinced that the AV industry isn't just hiping, if not actually playing a surreptitious role in the creation of the very threats they sell protection for. Can you imagine what the reaction of these and other folks would be once they realized that virus definition subscriptions were, in part, payments to the AV industry for protection against viruses created in those very same AV companies' own labs?
No, down that road lies madness. That's partly why AV industry experts insist that AV entities -- be they product vendors, researchers, or testing entities -- refrain from creating malware themselves -- because the widespread acceptance of lab-virus creation woud quickly involve the AV industry in a terrible ethical and practical quandry.
Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior | |
funchords
Hello
Premium,MVM
join:2001-03-11
Washington, DC
·Verizon Online DSL
·Skype
| Eric,
Apparently you don't think that we hear your point.
I hear your point, and I heard it before your last two messages. You are repeating it over and over.
It's a settled matter in the AV community that it is bad to create malware merely for testing:
- because it somewhat artifically and needlessly inflates the number of known threats that perhaps need a signature
- because the originator is compelled to release them to others for test and prevention research, and by doing such loses control of them
I think, and have thought all the while, that these are reasonable positions, reasonably arrived at. I understand them and I think they are valid.
To which I add, without contradicting:
- CR has done what it has done. Let's not simply dismiss it, but let's see if there's anything to learn from the results that they obtained with their method.
- Even if one disagrees with their method, let's not attack CU's ethics. They have no dog in this fight. No products to sell or advertise. They may have done something that was ill advised, but they have not shown bias nor have they behaved outrageously.
--
Robb Topolski -= funchords.com =- Hillsboro, Oregon USA
~ Keeper of the D-Link FAQ ~ Did you Search? ~ More features, Free! Join BBR! ~ | |
SnowyOne
Premium
join:2003-04-05
Kailua, HI
 ·RoadRunner Cable
 ·Clearwire Wireless
| reply to eburger68
I've had my opinion change thanks to the well argued points in this thread & that's to the credit of every poster.
said by eburger68 :Let's say for the sake of discussion that we all decide that it is perfectly OK for testing entities like CR, AV-Comparatives, Virus Bulletin, as well as PC Magazine and PC World to start creating and using lab viruses in their testing. And let's assume that this practice becomes rather commonplace in AV testing -- perhaps not universal, but routine enough that it doesn't shock those in the AV industry. That assumes AV-Comparatives, Virus Bulletin, as well as PC Magazine and PC World all have the same level of history, credibility & impartiality that CR has. They don't.
I haven't read the full article yet either, but I can say that if someone were to write a new chapter in product testing CR would be my odds on favorite to be the one that gets it right.
I also believe if they conclude that they screwed up they'll openly admit it. | |
eburger68
Premium,MVM
join:2001-04-28
1 edit | SnowyOne:
You wrote:
said by SnowyOne :I've had my opinion change thanks to the well argued points in this thread & that's to the credit of every poster. said by eburger68 :Let's say for the sake of discussion that we all decide that it is perfectly OK for testing entities like CR, AV-Comparatives, Virus Bulletin, as well as PC Magazine and PC World to start creating and using lab viruses in their testing. And let's assume that this practice becomes rather commonplace in AV testing -- perhaps not universal, but routine enough that it doesn't shock those in the AV industry. That assumes AV-Comparatives, Virus Bulletin, as well as PC Magazine and PC World all have the same level of history, credibility & impartiality that CR has. They don't. I haven't read the full article yet either, but I can say that if someone were to write a new chapter in product testing CR would be my odds on favorite to be the one that gets it right. But as a practical matter, it's not going to work out that CR somehow gets a "pass" for creating and using lab viruses and everyone else in the industry itself refrains. If CR is seen to use lab viruses for testing, and CR has the influence that folks in this thread have attributed to it, then others will be compelled to join the virus-creation game as well. One shouldn't let a misplaced affection for CR cloud one's understanding as to what the likely consequences of its irresponsible actions could be. You are not going to be in a position to determine who is allowed to create lab viruses for research and who is not.
As for the reputation of CR, even CR recognized that it was incapable of doing this kind of testing without the assistance of what it presumed to be experts from the world of computer security -- see CR's own discussion of how it tested (link in first post of this thread).
Moreover, as IBK pointed out earlier, there is nothing new or trailblazing about the testing done by CR, contrary to CR's own self-serving claims (there's that word again!). It was only CR's own ignorance of AV testing that allowed it to make such claims.
Finally, there will be no new "chapters in product testing" that get written as a result of this affair -- save the one that documents how an otherwise respected testing entity stumbled badly because it failed to heed the advice of those who had gone before -- because CR has not and is not likely to be offering the raw data from its testing up to other testing entities for peer review. (We'll set aside for the minute the fact that such peers would be ethically obligated and would likely refuse to take delivery of the data, for the all the reasons that IBK outlined in an earlier post.)
I'm sorry folks, but if you're starting from the position that CR's testing is "holy ground" that puts their ethics and behavior beyond scrutiny, or that CR's impartiality is of such paramount significance that it overrides demonstrable flaws in their testing methodology, then this discussion has exited the realm of the rational analysis of empirical testing.
Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior | |
SpannerITWks
Premium
join:2005-04-22
1 edit | reply to Cudni
What's the difference between someone sending a vendor X amount of New REAL nasties that they have discovered, that NO vendor has, and the vendors testing their AntiNasty with them, and then releasing Defs for them, and testing with those Specially written ones ?
Isn't that Exactly what would happen if those Specially written nasties got out somehow ?
Either a nasty IS a Nasty capable of doing whatever it can, or it isn't, then it ain't a nasty is it !
I believe those Specially written nasties are as valid as any others, that are new and discovered for the 1st time. Otherwise let's all pretend that ANY new nasties are completely irrelevent, and therefore we don't need protecting from it/them. Don't think so somehow !
The same goes for some bug etc in software that could be exploitable. What should someone do that discovers it, vendor or otherwise, nothing, or get busy with da fizzy and write some code to fix it, or pass the info on to the vendor if it's not their forte ?
Err not really too difficult to answer is it ! Cos if they don't someone out there will take advantage of it sooner or later, as they continue to do, almost weekly these days. And where would and does that leave MOST users out there, yeah right up **** street without a paddle that's where, as it frequently does !
Spanner
edit typo Only
--
I Only Know What I Know, But I'm Learning all The Time - Stay Safe - Spanner intheWorks
/SpannerITWks | |
|
