Wednesday, November 30, 2011

What is an Independent Antivirus Tester?

  With all the talk about Comodo and AV-C lately many people are requesting that products are tested from an Independent Antivirus Testing corp. However what is an independent antivirus testing group? AV-C "claims" to be an Independent Antivirus testing group, is it really independent?

  First we must define was an independent company is. I did a quick search online and the definitions I found where the following: 1. not influenced or controlled by others in matters of opinion, conduct, etc

  So how can a antivirus testing group be independent and how does it compare to antivirus testers such as av-c. First off I believe the test should not be commissioned or sponsored by an Antivirus vendor. The vendor should not be able to pick who is going to be up against or provide samples for them to be testing against. Avast wrote a blog post about it a while ago saying how they could hire a company and have them test their product against 10 samples and they could have an 100 percent detection using some randomly named "independent" testing company.

 Now let's compare this to AV-C. With AV-C you do not get to give them the samples to test against you, you don't get to pick who you were testing against, and no AV vendor sponsors the test. Everyone pays to get in but no one pays more then another. Plus the price to get in does not give you an advantage. Pbust said it best here on Wilders Security Forum:

"Paying testers is not the problem as long as all vendors pay the same fee structure and all get the same information, options for configuration and the facts are disclosed. Its a leveled playing field." -Pbust @ Panda Security

  The problems occur when payment is different from one vendor to the rest. Like for example sponsored tests where the sponsoring vendor chooses or provides the methodology, tested vendors, defines cases which should be tested and even goes as far as providing the samples. If you want to complain about non-independent tests then go after the sponsored tests. Those sponsored tests are the easiest to manipulate so that they say exactly what you want them to say.

  Now as you can see I mentioned a price to get in. This is the big debate for an independent AV testing group, how should they be able to get funds? However it goes beyond that how should any AV testing group get funded? Yeah I know many of you are thinking "if they are independent why should they get paid at all?" well I want to ask you is if the vendor is not paying a set amount how else can a tester get fairly funded? Every tester has to get funded some way or another. Magazine testers make money from advertising sales, some testing groups get paid to test a product a certain way, how can a AV testing group be 100 cash independent yet still have enough cash to do a lot of advance testing?

 Another big question is how do you hold a AV testing group accountable and know they are being truthful? Do you force them to turn over each sample they test with? Personally I can't see one simple way something can be put into place to make sure the results are truthful. Sometimes you just have to have trust in the tester.

  So to close I want to ask you, how should AV testing be done "fairly" in your opinion? Leave your thoughts in the comment section below.

1 comment:

  1. Just took a look at the Comodo forums and would like to point out that their CEO is misleading people in suggesting AMTSO audit testing organisations – they do not. AMTSO do not provide any testing certifications either – that is to say, there are no AMTSO certified tests – fact.

    Whilst it’s a good idea to try to agree valid and useful testing standards and processes, if “independence” is important then AMTSO membership must be discounted. The AMTSO board is made up of board members / directors of the large security vendors. It is not valid that an organisation which is primarily composed and run by the software vendors should dictate testing standards. It is important to work with vendors as this helps testing organisations understand products better and thereby generate better quality testing, however, the vendors role should be nonexecutive.

    The fact of the matter is, many industry insiders are becoming increasingly dissatisfied with the kind of testing AVC and others provide. These kinds of tests are only used as marketing collateral by vendors, they do not generally accurately assess a products performance in the real world.

    It is even the case that some AMTSO members recognise that the organisation needs to change if it is to genuinely contribute to the propagation of better testing.

    Nothing is perfect in this world and there are therefore no perfect tests or testing organisations. The kind of unprofessional ranting exhibited by the Comodo CEO in their forums is certainly not the way forward and only serves to mislead people.