Get on-the-go access to the latest insights featured on our Trustworthy Computing blogs.
AV-Test just published the results of their most recent antimalware vendor testing, and they didn't grant Microsoft Security Essentials and Microsoft Forefront Endpoint Protection their "AV-Test Certified" status.
We conduct a rigorous review of the results whenever test results warrant it. We take the protection of our customers very seriously, and the investments we make to do these reviews is an example of that commitment.
Our review showed that 0.0033 percent of our Microsoft Security Essentials and Microsoft Forefront Endpoint Protection customers were impacted by malware samples not detected during the test. In addition, 94 percent of the malware samples not detected during the test didn't impact our customers.
The antimalware world is challenging, for both antimalware companies protecting their customers and for independent testing organizations trying to determine the efficacy of antimalware products. We choose to meet that challenge by prioritizing our protection work based on prevalence and customer impact measures, as Dennis Batchelder discussed in his recent blog post on Customer-focused prioritization. It is also difficult for independent antimalware testing organizations to devise tests that are consistent with the real-world conditions that customers live in; AV-Test shared some of the difficulties and shortfalls in many of the independent industry tests in a presentation they gave at the AVAR (Association of Anti-Virus Asia Researchers) Security Conference in 2012. We agree with them, it is difficult to get the tests right.
This post reviews AV-Test's results and their approach. In-depth details are provided below, but here are some key upfront data points to keep in mind:
Here's how AV-Test does their scoring:
%Weight of score
The 1.5 protection score is the score we focused on. Here's a breakdown of what goes into that score:
% Not detected
% Weight of score
Malware seen for the first time, not to be confused with a previously undisclosed vulnerability
Malware that appeared in the wild over the last 2-3 months
Widespread malware according to AV-Test data
During the test, our products didn't detect 28 of the 0-day malware samples, and 9 percent of the recent malware samples. AV-Test uses a minimum bar in their scoring: our results for these two areas fell under that bar. The missed samples in both of these sections were where we focused our analysis, as we wanted to ensure we weren't missing anything impactful to our customers.
When we did our review, we found that our customer-focused processes had already added signatures that protected against 4 percent of the missed samples. These files affected 0.003 percent of our customers.
For the remainder of the missed files, we used a retrospective analysis to see if any of our customers encountered these files. We were looking for files that slipped through because of gaps in our telemetry or file collection process. And we found that 2 percent of these files existed across 0.003 percent of our customers.
The other 94 percent of the samples don't represent what our customers encounter. When we explicitly looked for these files, we could not find them on our customers' machines.
In December 2012, we processed 20 million new potentially malicious files, and, using telemetry and customer impact to prioritize those files, added protection that blocked 4 million different malicious files on nearly 3 million computers. Those 4 million files could have been customer-impacting if we had not prioritized them appropriately.
We continually evaluate and look at ways to improve our processes. We know from feedback from customers that industry testing is valuable, and their tests do help us improve. We're committed to reducing our 0.0033 percent margin to zero.
Joe BlackbirdProgram ManagerMicrosoft Malware Protection Center