First we must define was an independent company is. I did a quick search online and the definitions I found where the following: 1. not influenced or controlled by others in matters of opinion, conduct, etc
So how can a antivirus testing group be independent and how does it compare to antivirus testers such as av-c. First off I believe the test should not be commissioned or sponsored by an Antivirus vendor. The vendor should not be able to pick who is going to be up against or provide samples for them to be testing against. Avast wrote a blog post about it a while ago saying how they could hire a company and have them test their product against 10 samples and they could have an 100 percent detection using some randomly named "independent" testing company.
Now let's compare this to AV-C. With AV-C you do not get to give them the samples to test against you, you don't get to pick who you were testing against, and no AV vendor sponsors the test. Everyone pays to get in but no one pays more then another. Plus the price to get in does not give you an advantage. Pbust said it best here on Wilders Security Forum:
"Paying testers is not the problem as long as all vendors pay the same fee structure and all get the same information, options for configuration and the facts are disclosed. Its a leveled playing field." -Pbust @ Panda Security
The problems occur when payment is different from one vendor to the rest. Like for example sponsored tests where the sponsoring vendor chooses or provides the methodology, tested vendors, defines cases which should be tested and even goes as far as providing the samples. If you want to complain about non-independent tests then go after the sponsored tests. Those sponsored tests are the easiest to manipulate so that they say exactly what you want them to say.
Now as you can see I mentioned a price to get in. This is the big debate for an independent AV testing group, how should they be able to get funds? However it goes beyond that how should any AV testing group get funded? Yeah I know many of you are thinking "if they are independent why should they get paid at all?" well I want to ask you is if the vendor is not paying a set amount how else can a tester get fairly funded? Every tester has to get funded some way or another. Magazine testers make money from advertising sales, some testing groups get paid to test a product a certain way, how can a AV testing group be 100 cash independent yet still have enough cash to do a lot of advance testing?
Another big question is how do you hold a AV testing group accountable and know they are being truthful? Do you force them to turn over each sample they test with? Personally I can't see one simple way something can be put into place to make sure the results are truthful. Sometimes you just have to have trust in the tester.
So to close I want to ask you, how should AV testing be done "fairly" in your opinion? Leave your thoughts in the comment section below.