how-to block ads
|
Mele20
Premium
join:2001-06-05
Hilo, HI
| reply to eburger68
Re: Our unique antivirus testing: How we did it
It doesn't really matter what you think. What matters is what the readers and supporters of CR think. I think they will conclude that CR has not done anything unethical or rash or stupid. In fact, I venture to say that most CR readers will say "right on"! CR has taken on the AV vultures and the pundits who think they are such great advisors. The public will be behind CR and that is how it should be.
I don't believe they did anything wrong or that there is anything wrong with how they tested. I think you and Symantec, etc. are bloating this all out of proportion. I also think the AV vendors probably do create viruses just so they can stay in business. I'm a cynic especially when it comes to AV vendors and security "consultants". The typical reader of CR doesn't like their sacred cow attacked and won't allow anyone who does so to get away unscathed.
You have gone on ad nauseum in this thread and I wonder why "thou doest protest so much"? Are you a shill for the AV companies? If someone else, just about anyone else (other than IBK who also has a very obvious vested interest and the AV vendors you quote for support), had said some of what you have said, it might have been credible but your saying it is just plain laughable. I'll take CR's way any day over yours and I think the general public feels as I do. We don't need your "pontificating"...we trust CR....we don't trust you or the AV vendors ...all with heavily vested interest. That is it in a nutshell.
Anyhow this is a tempest in a teapot. The typical CR reader will never see this thread and would dismiss it if they did. So, it doesn't matter what you think. It matters that the public reveres CR, believes in its integrity and abilities to properly assess what ever is under the microscope for that issue. You and the AV companies seem to be blind because it makes no sense for vested interests to attack an impeccable entity like CR and expect to have the public not spit in their face.
--
"If you want to do DRM on a PC then you need to treat the user as the enemy." Ross Anderson in "`Trusted Computing' Frequently Asked Questions" | |
eburger68
Premium,MVM
join:2001-04-28
1 edit | reply to SpannerITWks
SpannerITWks:
You wrote:
said by SpannerITWks  :What's the difference between someone sending a vendor X amount of New REAL nasties that they have discovered, that NO vendor has, and the vendors testing their AntiNasty with them, and then releasing Defs for them, and testing with those Specially written ones ? Isn't that Exactly what would happen if those Specially written nasties got out somehow ? Either a nasty IS a Nasty capable of doing whatever it can, or it isn't, then it ain't a nasty is it ! I believe those Specially written nasties are as valid as any others, that are new and discovered for the 1st time. Otherwise let's all pretend that ANY new nasties are completely irrelevent, and therefore we don't need protecting from it/them. Don't think so somehow ! You've essentially elaborated the logic that would cause customers of AV companies to demand that lab viruses be added to AV definitions, and the logic that AV companies might be forced to bow to, if lab viruses became commonplace enough.
So, my question, though, remains: would you personally feel fine shelling out money each year for a subscription to an anti-virus product's definitions when those definitions were in part necessary in order to cover viruses that AV researchers, testers, and companies were themselves cooking up in the lab?
How do you think others would react to the same proposition -- that they had to pay for protection from viruses created by parts of the AV industry itself?
Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior | |
Steve
I'm a PC, so shut up
Consultant
join:2001-03-10
Yorba Linda, CA
| reply to eburger68
said by eburger68 :For the sake of clarity, let's narrow the focus of our disagreement to this one statement which, so far as I can tell, motivates and undergirds your entire argument: quote:
It just smells self serving to me.
How is the demand that entities that presume to do quality, responsible AV testing follow a simple ethical rule that one shalt not create malware onesself "self serving"? Agreeing on a set of best practices sounds like the kind of thing an industry ought to do, but the whole aura of "that's unethical" is self-important, high-and-mighty chest puffing, and it just really turns me off.
"Ethics" is about right and wrong, and CR was not unethical in any way, in spite of all the industry wailing. It's not wrong for a responsible, competent party to create test code in a lab environment in order to learn something about A/V coverage.
I certainly agree that it's dangerous, may well agree that it's unnecessary, (which would follow that there are more effective methods), but if this has not harmed anybody else, there's no ethical violation if this was all done in good faith.
Even if they somehow got in the wild accidently, that's about "negligence", not "ethics".
My knee-jerk reaction in a situation like thia is to side with Consumer Reports and not with the industry being reviewed. As a CR reader for many years, I've seen time and time again when the industry in question wailed about the reviews: it was unfair, that's not how you test that kind of thing, etc.
It's just happened before that CR used out-of-the-box thinking to think about an industry differently than the industry has. Sometimes this finds something important, sometimes it doesn't, but wails from the industry sound the same in either case.
The chest-puffing about "ethics" has that same ring to me.
Now, I happen to be much more educated about the A/V industry than the average reader of CR, so I'm actually going to be able to appreciate technical arguments about why they have gone down the wrong road, it's ineffective, it's too dangerous, etc.
But in the back of my mind, I'm still reserving the possibility that they really found a hole in the industry, and the industry doesn't like it.
I just don't know yet.
Steve
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site | |
Steve
I'm a PC, so shut up
Consultant
join:2001-03-10
Yorba Linda, CA
| 
Good advice? |
I did send this into them for the "Selling It" column, but they never printed it; I was hoping they would have a sense of humor about it and realize their error, but they didn't.
Steve
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site | |
eburger68
Premium,MVM
join:2001-04-28
1 edit | reply to Steve
Steve:
You wrote:
said by Steve :Now, I happen to be much more educated about the A/V industry than the average reader of CR, so I'm actually going to be able to appreciate technical arguments about why they have gone down the wrong road, it's ineffective, it's too dangerous, etc. But in the back of my mind, I'm still reserving the possibility that they really found a hole in the industry, and the industry doesn't like it. I just don't know yet. OK, fair enough. But if this does turn out to be a "hole in the industry," credit for finding it will have to go to CNET, not Consumer Reports, as CNET did this kind of testing 6 long years before Consumer Reports. And AV veterans will likely know of instances pre-dating even CNET.
If you're game, I would like to get your thoughts on the most recent issues we've been mulling over here -- namely, what the potential effect on the AV industry and its products/definitions would be if the use of lab-created viruses were to become commonplace among testing entities and researchers, esp. if it were concluded that a "hole in the industry" had been discovered.
Would customers demand that these lab viruses be added to definitions? Would they be justified in doing so?
Would AV companies be justified in adding lab viruses to definitions? Even if they weren't justified, would they be compelled to do so?
Obviously, the answers here would involve some amount of speculation, but these are serious issues nonetheless.
Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior | |
GeekNJ
Premium
join:2000-09-23
Waldwick, NJ
| reply to Cudni
I wonder what those folks here that think CR did something irresponsible would think of an individual who finds a vulnerability in an OS or product and then, after contacting the company, doesn't receive an adequate response. They then post their findings "in the wild". I think that's more dangerous then what CR did, yet the latter happens all the time and is typically how some areas of the software industry need to be treated in order to react.
I think CR's testing is fine and will likely (or more appropriately hopefully) result in A/V vendors better addressing potential issues because, like Steve, I think there's a need for those not "in the business" to challenge the business.
And on a bit of a related thought, I've personally always felt that the A/V industry itself was possibly responsible for the scare and even creation of viruses in order to pump up the "We protect you from xxx,000 nasties". Of course, how many of those nasties have never been "in the wild"? 
--
Tweaked your connection? | Mail Parse | Speed Converter | |
funchords
Hello
Premium,MVM
join:2001-03-11
Washington, DC
 ·Verizon Online DSL
 ·Skype
| reply to Steve
said by Steve :so I'm actually going to be able to appreciate technical arguments about why they have gone down the wrong road, it's ineffective, it's too dangerous, etc. (to all:)
If the danger could be sufficiently reduced, would it be worthwhile to explore their technique?
It occurs to me that one could engineer these test threats such that,
- if spread beyond the intended test bed (a specific set of machines), they would be inert and/or self-delete. (A large set of antipiracy-style techniques exist.)
- if successfully executed, their behavior is sufficiently mitigated. For spam variants, .com could be replaced by .invalid. For disk writes, any changes made by the test virus are eventually reversed by the test virus as part of its execution. Etcetera.
This way, the AV industry could test heuristic behavior protection with less risk.
This does nothing for the "it is unnecessary" theory, which several smart people maintain. And, although I'm not as educated as these AV professionals are on this topic, I am pursuaded that they are probably right. But if the danger were sufficiently reduced, perhaps they would grant one another permission to test that theory provided they followed certain safety practices.
Just a thought -- and an example of a possible way to learn from what CR has done, regardless if you agree with it.
--
Robb Topolski -= funchords.com =- Hillsboro, Oregon USA
~ Keeper of the D-Link FAQ ~ Did you Search? ~ More features, Free! Join BBR! ~ | |
funchords
Hello
Premium,MVM
join:2001-03-11
Washington, DC
·Verizon Online DSL
·Skype
1 edit | reply to GeekNJ
said by GeekNJ :And on a bit of a related thought, I've personally always felt that the A/V industry itself was possibly responsible for the scare and even creation of viruses in order to pump up the "We protect you from xxx,000 nasties". Of course, how many of those nasties have never been "in the wild"? We've all wondered about that at some time or another. We've all heard the true stories of firemen that try to boost their careers by starting fires in order to be the hero that saves a life or building.
But, I think we're pretty safe from that possibility, because:
The industry is both old enough and large enough that, if this were happening, a current or ex-employee whistleblower would have appeared by now.
The number of competitors is large enough to identify one competitor that is constantly adding threats to their definitions that nobody else has ever seen.
In both of the above, such an allegation against a specific company would be a death sentence. Even the allegation of fabricating viruses to sell AV product could kill an AV company. As an example, we can look back to the tainting of anti-adware companies that started to become chummy with certain software companies.
--
Robb Topolski -= funchords.com =- Hillsboro, Oregon USA
~ Keeper of the D-Link FAQ ~ Did you Search? ~ More features, Free! Join BBR! ~ | |
eburger68
Premium,MVM
join:2001-04-28
| reply to Cudni
Hi All:
A quick followup to my last few posts regarding the potential effects of the widespread use of lab viruses on the AV industry. The concerns that I've expressed -- that, in a worst case scenario, the industry and its customers could be drawn into enervating cycle lab virus creation and virus definition building in response to customer demands -- is not an idle one. We've seen these kinds of cycles before.
First, give part of Joe Wells' paper here a read:
"Lies, Damn Lies, and Marketing Perfidious Priorities"
»vx.netlux.org/lib/ajw01.html#p3
Joe rehearses one these cycles from the early years of the AV industry, when it was sucked into a competitive arms race over the number of "all known viruses" and the comparative detection rates of AV products. Bad research and analysis feeds opportunistic, competitive marketing, which feeds user fears and customer demands, which in turn feeds product testing and research, which feeds...
You get the picture.
We've even seen a similar phenomenon in the anti-spyware industry with respect to cookie detection. Having talked to a large number of folks from various anti-spyware companies, I can tell you that none of them (at least that I know of) regards cookies as anywhere near the same kind of threat as executable adware, spyware, or malware. And most that I've talked to have expressed a desire to do something different with the cookie detection in their products. Some would like to drop it altogether. Others would like to handle it differently, so that cookies weren't presented alongside executable malware in a manner that suggested that the two were roughly similar types of threats.
So why don't things change within the anti-spyware industry? Because everyone's afraid of the consequences of being the first to act (beyond Microsoft, which dropped cookie detection from the GIANT product that it acquired). Any anti-spyware company out there can tell you about the angry calls and emails they get from customers that Product A failed to detect a few cookies that Product B detected. In short, fearful customers are demanding cookie protection and frequently see no difference between cookies, viruses, spyware, and adware. And the anti-spyware companies, much as they might gnash their teeth over the detection of cookies, continue to provide it (and even, in some cases, hype it) out of fear that their product will take a hit in sales and reputation should they be perceived as "soft on cookies."
So, my concerns do have a basis in actual situations that we've encountered before.
Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior | |
Steve
I'm a PC, so shut up
Consultant
join:2001-03-10
Yorba Linda, CA
| reply to eburger68
said by eburger68 : If you're game, I would like to get your thoughts on the most recent issues we've been mulling over here -- namely, what the potential effect on the AV industry and its products/definitions would be if the use of lab-created viruses were to become commonplace among testing entities and researchers, esp. if it were concluded that a "hole in the industry" had been discovered. Now this does get into an interesting area, one which would get into arms races that don't serve the consumer. Generally: one should only protect against threats, and test viruses in a lab are not threats (EICAR is not a threat, but it's an exception).
Many of the players in the A/V industry are also involved the spyware and personal firewall industries, and I think this gives us a peek into the future as to what to expect.
Personal Firewall vendors are in this ever-growing competition to detect more stuff, which is why we have these bogus "YOU ARE BEING ATTACKED" popups. OMG! It's a PING!
I staff a couple of abuse desks, and we get reports all the time about benign behavior, but because of self-serving WE CATCH MORE STUFF! WE PROTECT YOU! crap from the firewall industry, the consumer is frightend and at least two people waste a lot of time.
Adware vendors likewise are in this "we detect more stuff", which leads to outlandish claims about "5,000 objects detected". Yah, but most of them are cookies, and the rest are mostly benign. We see these threads here all the time.
I believe that these practices are a substantial disservice to the customer, and border on unethical.
Steve
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site | |
eburger68
Premium,MVM
join:2001-04-28
2 edits | reply to GeekNJ
GeekNJ:
You wrote:
said by GeekNJ :And on a bit of a related thought, I've personally always felt that the A/V industry itself was possibly responsible for the scare and even creation of viruses in order to pump up the "We protect you from xxx,000 nasties". Of course, how many of those nasties have never been "in the wild"? Yes, you and a number of other folks, as I noted in an earlier post. If this truly is a concern of yours, then the very last thing you want to encourage is the creation and use of lab viruses by anyone in the industry or even connected with the industry -- and that includes CR, because if a widely respected and influential testing entity like CR begins routinely creating and using lab viruses, then the pressure will only increase on others in the industry to start doing the same. At some point, AV companies could very well be compelled by customers or circumstances to start loading up their definitions (and selling subscriptions to them) with these lab viruses.
And who would benefit from such an eventuality? The only possible beneficiaries that I see are the sales departments of AV companies.
This is one of the quagmires that the "Wild List" was created to forestall -- to compel the industry to focus on, research, test against, and target actual viruses that posed real threats to users "in th wild."
Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior | |
funchords
Hello
Premium,MVM
join:2001-03-11
Washington, DC
·Verizon Online DSL
·Skype
| reply to eburger68
said by eburger68 :First, give part of Joe Wells' paper here a read: "Lies, Damn Lies, and Marketing Perfidious Priorities" » vx.netlux.org/lib/ajw01.html#p3Joe rehearses one these cycles from the early years of the AV industry, when it was sucked into a competitive arms race over the number of "all known viruses" and the comparative detection rates of AV products. Bad research and analysis feeds opportunistic, competitive marketing, which feeds user fears and customer demands, which in turn feeds product testing and research, which feeds... Yep, similar to Processors and MIPS, wireless network theoretical datarates, pharmaceuticals and "restless-knee syndrome," ... the list goes on forever.
--
Robb Topolski -= funchords.com =- Hillsboro, Oregon USA
~ Keeper of the D-Link FAQ ~ Did you Search? ~ More features, Free! Join BBR! ~ | |
eburger68
Premium,MVM
join:2001-04-28
| reply to Steve
Steve:
You wrote:
said by Steve :I believe that these practices are a substantial disservice to the customer, and border on unethical. I couldn't agree more. The trick, of course, is how do we compel an industry addicted in many ways to the "numbers game" to come together and give up this marketing "crack"?
I should note that the difficulty only increases when one realizes that potential customers are being exposed on a daily basis to insanely unethical scaremongering pitches for "rogue" anti-spyware products and the like that are being marketed by malware pushers themselves -- borderline criminal elements who aren't likely to see any benefit to sitting down at a roundtable on industry ethics and best practices.
Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior | |
Steve
I'm a PC, so shut up
Consultant
join:2001-03-10
Yorba Linda, CA
| reply to eburger68
said by eburger68 :The concerns that I've expressed -- that, in a worst case scenario, the industry and its customers could be drawn into enervating cycle lab virus creation and virus definition building in response to customer demands -- is not an idle one. We've seen these kinds of cycles before. Ah, so now the fog lifts.
This is not so much an egalitarian "do not hurt others" concern, but a worry that the industry will be unable to restrain itself; that's a different concern that does not speak in any way to the ethical behavior of Consumer Reports.
It would indeed be unethical for an A/V company to create viruses ("for testing") and then include them in a product with a claim that they protected against more stuff than the other guys (who don't have the synthetic tests). This is hyping against threats that do not really exist.
So the objection is not about "creating test viruses" but "creating test viruses and using them for marketing": the latter creates a whole cycle of bad incentives at the expense of the consumer.
That is unethical, and I'm pretty sure that there's essentially 100% agreement on that point.
But since Consumer Reports is not in the A/V industry, they don't have any of those incentives (they're not selling an A/V product), so these reasonable proscriptions on industry behavior do not apply here.
The more I look at this, the more I believe Consumer Reports was not unethical in any way, even remotely.
So putting aside the ethical issue, we're left with something that's somewhat easier for us to talk about: the technical merit of their testing methodology.
But that doesn't make it completely easy: the onlooker must be on the lookout for a circle-the-wagons reaction by the industry — this happens all the time — and I'd be surprised if there were none of that here.
I, like others, am content to evaluate both the evidence and the warrants, sniffing out the good and spurious claims.
But the more I see about "ethics", the more I think it's a circle-the-wagons reaction, and to accept A/V claims with more and more grains of salt.
Steve
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site | |
AB
Premium
join:2006-04-04
Leesburg, VA
| reply to Steve
said by Steve :. . I certainly agree that it's dangerous, may well agree that it's unnecessary, (which would follow that there are more effective methods). . (sic) Even if they somehow got in the wild accidently, that's about "negligence" . . . Steve 'Nuff said, perhaps?? | |
alexeck
join:2004-12-20
Clearwater, FL
1 edit | reply to Cudni
Folks,
I see arguments supporting CR and against CR.
Here's the simple truth: CR chose to ignore a vast corpus of research, debate and analysis by the academic and security research community. They decided to go their own way and have severely undermined their credibility by making a major error, and possibly others.
It's an established principle in security research that you NEVER create your own antivirus strains for testing purposes. There are a number of reasons which I discuss in my most recent blog posting »tinyurl.com/msclw .
The CORRECT way to test heuristics is extremely simple: Turn off definitions for all the products being tested, and test against new virus strains after a few weeks or months. That's the only honest, correct approach, as it a) tests against the real-world, b) doesn't turn you into a virus creator/distributor and c) actually gets you the right results.
Why CR couldn't simply follow this time-honored approach is a bit confusing.
Arguing that the AV community is biased in this regard is patently false reasoning. The arguments against CR are across the spectrum, from the pure research side to the antivirus community.
If CR had simply followed standard testing methods, all would be fine and no one would care. It would actually be a service to the community.
But the problem is a bigger one: We need standardized testing for all types of security products. This debate should be done in a reasoned, scientific fashion, with broad representation in the community and industry to come out with a clear, comprehensive method of testing. That is the only real way to serve the consumer.
Alex | |
eburger68
Premium,MVM
join:2001-04-28
| reply to Steve
Steve:
You wrote:
said by Steve :But since Consumer Reports is not in the A/V industry, they don't have any of those incentives (they're not selling an A/V product), so these reasonable proscriptions on industry behavior do not apply here. I'm afraid I have to disagree once again. CR is widely respected and influential -- that we've seen even from some of the posts in this thread. If this testing turns out not to be a one-off situation with CR -- that is, if CR were to start routinely using lab viruses in their widely followed testing -- then pressure on the AV industry itself (including independent researchers, testing bodies, consultants, and the AV vendors themselves) to do the same would inevitably increase. It would likely start with other research entities, but it would likely spread to other parts of the AV industry.
Even if the AV vendors themselves somehow managed to refrain, we could still well wind up in a situation where AV companies were forced to contemplate whether to start adding lab viruses to their defintions. And who would benefit? Surely not ordindary users and consumers.
No, CR is not an island unto itself. Many here have championed CR for having the wherewithal to force industries to think and behave differently. This power can be a benefit in some circumstances. It can also pose dangers if that influence unintentionally forces an industry down a path it has no business going down. One can't celebrate the influence of CR on the one hand and not contemplate the potential consequences of its actions on the other.
So, is the AV industry "circling the wagons." Perhaps only to protect itself from a potential trend that it long ago recognized as dangerous.
Eric L. Howes
--
Microsoft MVP
Sunbelt Software
Spyware Warrior | |
EGeezer
Go Bobcats
Premium
join:2002-08-04
Country!
·Callcentric
 ·RoadRunner Cable
·AT&T CallVantage
1 edit | reply to eburger68
Great discussion here -
From my own perspective, not knowing how valid the CU "variations" are, I couldn't speculate on the validity of the tests. However, I feel that testing against variations provides value by providing a fresh angle, and Consumer reports has done pretty well in the past in providing consumer level information on product quality. It seems that this test, if the variations were reasonably close to what malware writers do to circumvent detection, would lean more favorably toward those who have better behavior-based or heuristic engines, and at a disadvantage to those relying more on signature tables. The difficulty I see is if the tests themselves don't employ multiple techniques, but are positioned as be-all end-all tests.
In the end, CU is little different from other AV testers in that few have the same results in ranking. At best, they generally correspond to each other. The use different reference samples, test methods and tools, so their results will be different.
As for judging validity of the tests based on sources, I'd consider the reputation of the source and how successful they've been as well as their methods, credentials and affiliations. One should naturally consider whether the tester has an ax to grind or a stake in the outcomes, then research against other sources.
As for the vendors. the FUD and hysteria built into the detection messages is problematic. They appear to have been written by marketing types, not by technical writers, and with the goal of promoting the product rather than providing an accurately positioned description of the object detected.
I have no issues with a product detecting cookies on a scan, but having it do so in the manner of the screaming carpet salesmen on late night TV is in insult to the educated and a disservice to the technically uninformed. Vendors need to more accurately position the characteristics of privacy and security-related objects they detect, isolate or remove.
--
This space for rent | |
Steve
I'm a PC, so shut up
Consultant
join:2001-03-10
Yorba Linda, CA
| reply to eburger68
said by eburger68 :Even if the AV vendors themselves somehow managed to refrain, we could still well wind up in a situation where AV companies were forced to contemplate whether to start adding lab viruses to their defintions. And who would benefit? Surely not ordindary users and consumers. This is a fair point, but it's not the argument that was made in the early parts of this thread. It was originally that CR's creation of these was bad in and of itself, but now it's because it might lead to the industry into screwing the consumer. Those aren't the same things!
CR generally thinks outside the box without regard for what the industry being reviewed thinks, and I believe that's good for the consumer. The louder the industry wails, the more I think they may be onto something.
Should we let the lawn-mower industry define the tests for what makes a good lawn mower? How about car companies? etc.
It may well be that CR committed the crime in question, but the A/V industry is doing a terrible job in the witness stand.
Look at all these signatures! Look at how much we're gnashing our teeth! I'm not going to believe CR when they rate a gas grill!
I'm an educated, technical consumer with a reasonable nose for BS, and this all comes off as incredibly disingenuous to me. But I could be wrong.
Steve
--
Stephen J. Friedl • Unix Wizard • Microsoft Security MVP • Tustin, California USA • my web site | |
bluezanetti
Premium
join:2003-10-04
| reply to alexeck
said by alexeck :
The CORRECT way to test heuristics is extremely simple: Turn off definitions for all the products being tested, and test against new virus strains after a few weeks or months. That's the only honest, correct approach, as it a) tests against the real-world, b) doesn't turn you into a virus creator/distributor and c) actually gets you the right results.
Absolutely.
I can see that an organization such as CR could feel that generating a synthetic testbed of samples would provide a more expedient and controlled testing platform than would an active harvest of malware over a defined period (e.g. the basic protocol followed at av-comparatives.org). Many of CR's evaluations follow this scheme of devising a synthetic challenge and performing classical challenge/response testing. However, it is just as easily argued that no matter how controlled their testbed, there's no assurance the test sampling bears any resemblence to emerging malware threats at play today due to the very dynamic nature of the challenge. The validity of this portion of the test results reside in appropriate choices being made in the creation of the synthetic testbed. Poor choices there will completely skew the final results. In a sense, the test results could range anywhere from an accurate reflection to a completely inverted ranking of current performance and there's really no way to get an independent sense of where things lay.
Even the av-comparatives.org test results have to be closely inspected owing to the noise associated with small sample testbed. In the case of CR, it is noise with an unknown level of unintended sampling bias.
Blue | |
|
