Security → Investigation in Progress by AV Comparatives

Investigation in progress

AV-C has uncovered an infringement of the testing agreement by one of the vendors participating in its tests. It has been found that a product submitted for testing by the vendor had been specifically engineered for the major testing labs, including AV-C; public availability of this version was limited. A second vendor is also being investigated for similar reasons. When this analysis is complete, AV-C will announce the measures it will take against the vendor(s) found to be in breach of contract.

»weblog.av-comparatives.o ··· g/?p=501

Sad really whoever it is as we Consumers trust these vendors with our Data and the Protection of such Data, it will cause said vendors to suffer as people are not so forgiving on the WWW.

TH

Thanks, and please keep us posted!

Well this is no shocker to the more informed and industry workers.

AV tests have never been truly independent when money changes hands but more of purchased advertising material for the AV vendors that are willing to pay for it.

Even AVC who are probably the most trustworthy out of a shady business area will not burn their paying customers(as IBK knows they wont pay him another Euro if that occurred so he wont do anything to the detriment of his own business model) Have you ever seen anything but stella detection rates from AVC paying customers ?

Another example is to look at any commissioned/sponsored tests masquerading as being done by "independent" testing companies.

The customer pays the lab to test their software against their stated shortlist of competing softwares and stipulates how they are to be tested and against what etc

In exchange they get in return advertising material that they can use to make themselves look better against their piers when trying to promote their products.

So its no big shocker that some actors will cheat....No one ever want to buy #6 rated software

"28/04/2015 UPDATE:
AV-Test, Virus Bulletin and AV-Comparatives had a conference call today. Further collaborative investigation is now in progress. We will give additional information in a joint statement with the other labs. We will keep you up to date.
»weblog.av-comparatives.o ··· g/?p=501

This is becoming a real who-dunnit.
I wonder if the guilty party(s) are going to mea-culpa or try to defend the switcheroo.

If the pseudo private build is that much better than the public release why keep it under wraps?
But then again anyone that would game a test bed has already proved they make bad decisions.

Not sure if that is the real case.
I have found what was set up for some was not good for others and definitely pre-release versions were always scrutinized by sales requirements and support needs.

Everyone knows a default set-up security tool is not as good as it can be, and there is an area whereby what is a strong setup isn't necessarily good for the average joe and hence for support needs, certain switches are turned off or white lists are included for ease of end-user support.

If this is all it is I'm not worried about the results being published at all.

If it is an engine mod or signature tweak to make it work better purely for test results then I think it needs publishing and standards put in place.
I was under the understanding the software tested was a download version off the main downloads page tested in most cases, not a specifically handed over version. (Except where beta product is assessed at the cost of companies to help evolve the product before commercial release which I also see no problem with.)

Either way, I am interested in what is presented from this bold statement, as it is from various testers, not one specific test company and they all stand to affect specific company sales and their own credibility as testing companies.

Probably because way too many false positives because of lowered detection thresholds.

30 April 2015
»weblog.av-comparatives.o ··· g/?p=504

quote:

---

Testing bodies AVComparatives, AVTEST and Virus Bulletin comment on allegations of inappropriate behavior

---

Read more there and on the pdf file there.

 
From the PDF cited in the weblog link

---

...Today, three of the world’s most renowned and trusted security testing bodies, AV-Comparatives, AV-TEST and Virus Bulletin, stand united to censure security vendor Qihoo 360 after finding the firm submitted products for comparative and certification testing which behaved significantly differently from those made available to its users and customers. The three testing bodies will revoke all certifications and rankings awarded to the company's products so far this year, and going forward will insist on more open and fair dealings to ensure users are provided with the most accurate information possible. ...

---

I just read the PDF. Why am I not surprised which is the culprit?

Okay, report on all vendors please...not 1 or maybe 2.
Shessh, it feels like a kindergarten class.

The issue is commented and/or discussed at several places.
Some of them:

Graham Cluley:
»grahamcluley.com/2015/05 ··· t-cheat/

quote:

---

Chinese anti-virus vendor Qihoo 360 has been stripped of all of the certifications awarded to it this year by the three leading anti-malware testing agencies, after being found to have broken the rules.

In a joint statement issued by AV-Comparatives, AV-Test.org and Virus Bulletin, Qihoo was found guilty of attempting to game detection tests.

---

Neil J. Rubenking at PCMag:
»www.pcmag.com/article2/0 ··· 8,00.asp

quote:

---

Antivirus testing labs help vendors and consumers alike by putting products through rigorous testing. However, those results go by the wayside when a vendor (Qihoo, in this case) cheats by submitting one product for testing and supplying another to consumers.

Many years ago, PCMag Labs ran into testing troubles with some hardware vendors. They'd submit units that were cranked up to win speed tests, while selling consumers units running at a reasonable speed. Apparently, test cheating hasn't gone out of fashion. Today, three extremely prominent antivirus testing labs censured antivirus vendor Qihoo 360 for similar shell game chicanery.

---

Wilders Security Forums:
»www.wilderssecurity.com/ ··· .375769/
PS: IBK of AV-Comparatives has also posted there.

Qihoo 360 statement regarding cheating in lab test / May 2, 2015

said by Qihoo 360 :

---

April 30th, Qihoo 360 received comments from its industry partners with allegation of inappropriate behaviour on the benchmarking processes in test labs. We regret that this behaviour has resulted into such comments from these labs, who we recognize as reference for security benchmarking. However, we hereby offer our perspective to the alleged comments:

---

»blog.360totalsecurity.co ··· ab-test/

"The allegation highlights that the default configuration of the product available for the public, differs from the configuration used by the labs for testing. This configuration was explicitly declared upon submission of the tests, and was thereafter confirmed by the test labs."

Clever.
Their saying it's default of the testers.

 
So they're not taking responsibility for what they did.

Seems to me I'd want an a/v vendor who would take responsibility for their actions, instead of trying to put the blame on others.

I would never have used that AV company anyway.

Glad it wasn't one of the "big names". That would be a dilemma.

May 2, 2015

Qihoo 360 withdraw from testing by AV-C. Reason: "traditional anti-virus evaluation criteria behind".

Translation of Qihoo's reasons for withdrawing: »www.hihuadu.com/2015/05/ ··· 812.html

Original link: »tech.163.com/15/0502/21/ ··· 5BF.html

 
I looked at the company briefly when I switched a/v products recently.

Once I saw it was based in China, I took them off my list of vendors to consider.

We don't trust the Chinese or Russians, they don't trust us either, probably with good reason all around.

My strategy is to use them all, eventually one of them will find something the other one is attempting to hide.

I use the real-time scanner made in US and 3 On-Demand scanners. One made in Russia, One Made in a former eastern block nation and one made in China.

That way I have all of my bases covered.

From: »www.amtso.org/PR20150506
Why we cannot tolerate unethical behavior in the anti-malware industry
Anti-Malware Testing Standards Organization

quote:

---

6 May 2015, San Francisco

When you read an anti-malware product review, you expect an honest representation of the security product. The reviewer also expects that the software submitted for review has not been tampered or falsified from the product available to customers in any way. When these rules of engagement are not met, hard-earned reputations become tarnished and customers are put at unnecessary risk. In the last few weeks, the dishonest actions of a few security vendors has not only impacted the reputation of respected security testing bodies, but impacted the industry as a whole.

How? By e.g. "submitting a different product for review than what was actually offered their users" or by "having optimizations in the product only to perform better in a performance test".

This situation is not unlike someone buying a car based on a review highlighting its great NCAP rating for safety, only to find that the model purchased does not even include an airbag. Not only are the reputations of the car manufacturer, sellers and testing bodies are all negatively impacted, but the security of the purchaser is also put at risk.

If the security product delivered to consumers or businesses differs from the one reviewed by a reputable testing body, it could give the buyer a false sense of security. The results for the user could vary from malicious attackers accessing sensitive data to total disruption of the system and user experience.

---

Follow link above to read entire article.

Also: »plus.google.com/10038386 ··· mWbP89Qn
Statement regarding Tencent products in recent Windows tests

quote:

---

AV-TEST GmbH
Shared publicly - May 5, 2015

Statement regarding Tencent products in recent Windows tests

After in-depth investigations, certain optimizations have been identified in Tencent products which are clearly designed to improve their ratings in AV-TEST's performance testing. These optimizations, which have been found in all recent public versions of the products, provide minimal benefit to normal users and could even degrade the level of protection offered by the products.

All three testing labs involved in these investigations - AV-TEST, AV-Comparatives and Virus Bulletin - expect participants in their tests to behave in an open and ethical manner at all times, and consider this sort of "gaming" of tests to be unhelpful to both developers and users. The labs will be imposing stricter controls on participants to reduce opportunities for such actions, and will revoke all affected certifications and awards granted so far in 2015.

---