Van's FSS/Antigen Blog

make sure your engines have updated after October.

SP2 has a mapper (packaged with the engines) requirement that was released two months before SP2 came out.

we have seen a case that had a customer not updating for a year and this caused the engines to not load after the upgrade to service pack 2.

If you have written into support lately you should have seen something like this in the signature of the engineer you were working with

In 10 days we will be removing these engines from our update servers.

One thing that customers need to be aware of

If they have the “Box Set” license that comes with Antigen, CAVet, Microsoft, Norman, Sophos

they will be running with only the Microsoft updated engine as of the first.

The solution to this is to request a new license file that will unlock the Kaspersky, VirusBuster and Command engine and Cloud Mark for Anti-Spam.

The instructions for the license is in the link above.l

Issue:

Constant StatisticsManagerServer event id 100 on the passive node of a 2003 cluster (maybe on a SCC cluster in 2008 as well)

Cause:

Antigen Statistics Service needs to access the statistics.xml located in the %data% folder of the Antigen install.

On a passive node this xml file is located on the shared drive that is controlled by the active node. This causes a failure to start for the service.

The service is starting because something is making a call to it.

In most cases there is monitoring software that loads up our Scan counters for performance monitor.

Other issues could stem from FSSMC collecting scan data from the passive node.

Workaround:

Monitoring software. This is expected behavior and the process loading these counters need to be configured to not monitor Antigen on passive nodes.

FSSMC: If you are using FSSMC to monitor the passive node you can try re-deploying the agent to the passive node.

We have seen the following issue a few times in the last two weeks

Issue:

After installing Antigen or recycling the server where we are installed with, Exchange fails to start up. Error is dependency failed to start.

Trying to start Antigen Monitor results in a “%t is not a valid win32 application” message.

Cause:

Our paths are not wrapped in quotes in the registry.

Antigen is installed into a path with spaces in it (Program files)

A file matching the first word in the install path is in the root directory.

Example:

0 byte file called program in the root of c:

Analysis:

Without wrapping you paths in quotes you run into windows guessing what you mean when you pass it a command

For example

C:\program files\Microsoft antigen\exchange\antigenservice.exe

In a normal system would launch antienservice.

In a system with a file called program in the c:\ folder it would launch

C:\program

with the argument files\Microsoft antigen\exchange\antigenservice.exe

Solution:

Remove the file that matches the word before the first space (In my example the 0 byte program file)

You can also re-install in a path with no spaces (c:\antigen)

If you remove the file in the root of the drive and it comes back you can use process explorer to trace the creation of the file and then find out what program is dropping files in your root directory.

We should also be implementing a fix for this in the next hotfix rollup.

I had a good question today from one of my customers

What is the background scan.

here is my response.

You should never use a background scan.

Ever…

Background scanning was a clever idea back in the day when VSAPI 1.0 was released.

You have a background scan running in VSAPI that goes through every file in your database.

Then when that file is accessed, if it has been scanned with the latest definitions it skips the scan on access and the file is allowed to pass.

Problem is.. back in 1999-2000 when that came out, virus definitions updated on average once a week.

Today you can have engines that update every 1-4 hours. And with 5 engines.. you get the point.

The scan never finishes and basically acts as a resource pit.

We support it, because VSAPI still supports it.

The following error is logged every 60 minutes on Forefront Server for Exchange SP2

This is a informational warning but the way it is worded might cause some concern.

Log Name:      Application

Source:        FSEAgent

Date:          8/28/2008 2:13:21 AM

Event ID:      8048

Task Category: (8)

Level:         Error

Keywords:      Classic

User:          N/A

Computer:      XXXXXXXXXXXXXX

Description:

1 messages have been archived and purged due to an error while scanning. Please ensure that mail is not queuing.

This error stems from mail being stuck in the Undeliverable folder of our archive folder.

The default path is c:\Program files (x86)\Microsoft Forefront for Exchange\Archive\Undeliverable\

There should be a file stuck in that folder or under a subfolder.

Removing this message will stop the error.

Missed a week.

This is for the following error

Event Type:        Error

Event Source:    GetEngineFiles

Event Category:                Engine Error

Event ID:              6012

Date:                     #/##/####

Time:                     #:##:## ##

User:                     N/A

Computer:          Testlab01

Description:

Microsoft Forefront Server Security encountered an error while performing a scan engine update.

   Scan Engine: Microsoft

   Error Code: 0x80070102

   Description: Unable to acquire the scan engine update mutex within the    designated timeout period.

This is a timeout and there is a KB (http://support.microsoft.com/kb/939411)  that discusses increasing the download scan timeout but it does not take into account the other issues you will see with longer download times. Our mutex timeout is hardcoded at 3 minutes.

If you have increased your InternetDownloadtimeout to match the longest download times you see in your programlog you need to take into consideration that you can run into a situation where you have scan engine updates timing out because we timeout waiting for the current download.

For example

if the following engines are set to download at these times

Ahlab at 07 min after the hour Repeat every 1 hours

Kaspersky at 09 min after the hour Repeat every 1 hours

Norman at 34 min after the hour Repeat every 1 hours

Microsoft at 39 min after the hour Repeat every 1 hours

Vbuster at 48 min after the hour Repeat every 1 hours

Sophos at 44 min after the hour Repeat every 1 hours

CAVet at 49 min after the hour Repeat every 1 hours

Command at 54 min after the hour Repeat every 1 hours

And the engines take 15-25 minutes to download

This is never going to work correctly.

If it takes 15-30 minutes for you to download an engine you need to make sure that no engine downloads in that timeframe.

Here is a suggestion (1800 minutes for the download setting)

8:00 AM Ahnlab update Repeat every 4 hours

8:32 AM Kaspersky update Repeat every 4 hours

9:04 AM Norman Update Repeat every 4 hours

9:36 AM Microsoft Update Repeat every 4 hours

10:08 AM VBuster Update Repeat every 4 hours

10:40 AM Sophos Update Repeat every 4 hours

11:12 AM CaVet Update Repeat every 4 hours

11:44 AM Command update Repeat every 4 hours

This would ensure that you never have two engines fighting for the download at the same time. If your downloads are faster you can repeat the update faster.

Let me know if this helps you out.

Another FSCIMC hung in a start pending state issue.

Before this was tied to execution policy being set to restricted or locally signed

This time certificates are to blame.

If for some reason your Microsoft root authority certificate expires you can expect to see FSCIMC and Edge transport fail to start with the execution policy at the default setting.

We require at least remote signed but if there is an issue with contacting the CA and you have an invalid or expired certificate you will have to lower your exchange power shell script permissions to unrestricted.

This will allow you to start up but the real fix would be to fix your certificates as other issues will occur with updates not able to verify the digital signatures.

This issue occurs because there is a popup warning of an invalid certificate. Since we are running as system we never see that popup and sit and wait for something to happen.

If you remove our produce you will then see a pop-up generated by edge transport with the same information about an invalid certificate.

*Update*

MSAV has been updated and has resolved this issue.

Command engine has resolved this issue as well.

*Update*

Looks like there is a false detect issue with some engines and PDF files.

Antigen-Command - 5.1.0.5 2009-08-20 09:40 PDF/CollabExpl.C!Camelot

These seem to be false detects as this is the result of files built at a customer site.

We are currently investigating the cause as this effects our Microsoft engine and Command

This issue deals with Antigen 9.x and clusters with Multiple Active Nodes.

You do not need the cluster resource on a single Active cluster.

There is a fix in Antigen 9.1 RU5 and Antigen 9 Service Pack 2 that addresses an issue when moving Active nodes to a passive node.

Below is an example of what you see when you have the issue resolved in RU5 or SP2

When the first Node is moved over to the passive node we set a registry key that identifies the database path for the engines and databases. In this example Server1 has its databases on G:\AntigenCluster

The store is then moved back to the original node and then server2 is moved to the passive.

Sever2’s database path is in F:\AntigenCluster

Most of the time Server2 will move the database path to the Passive cluster and everything works. Sometimes this process fails. You end up with a wrong database path and in this example server2 is on the passive with the database path still set to G:\AntigenCluster

This will cause us to initialize with no scan engines and no settings for our scanjobs.

The symptoms of this issue is as follows

End users cannot open or send mail.

Messages queue in Local delivery

The fix requires you to do one of the following to to create the cluster resource

1. Full uninstall (Pause passive nodes, uninstall from all active nodes and then passive nodes) and run a fresh install.

2. Upgrade to SP2 and do the following. Run Antutil.exe /disable and then Antutil.exe /enable on each active node (Pause passive nodes, bring exchange resources offline, leave CMS up.)

Both methods Require downtime

I have been looking for a good issue to post this week.

So here is a post for all those 8.0 customers looking to upgrade to 9 by the time 8.x for Exchange reaches its end of life at the end of the year.

Mail Queues after an upgrade from 8.0 to 9.x

Symptoms:

After upgrading to 9.x from 8.x Mail on SMTP and Real-time does not flow.

Cause:

8.x did not have the Microsoft scan engine

9.x only comes with the Microsoft scan engine.

On an upgrade the Microsoft Scan engine is not selected. If you incorrectly entered your proxy server address or have an issue pulling down updates you are left with no engines and mail flow is stopped pending updates.

Quick Solution:

Open the Forefront Console and select the Microsoft Scan engine for all scan jobs.

After that you should have mail flow again and you can fix your updating issue.

We have seen a few cases of licensed builds coming online as expired evaluations as people upgrade to Antigen Service Pack 2 for exchange.

I believe this is somehow triggered on Active Passive server installs where the Active node fail’s over to the Passive during our install. This leaves the AntigenCluster folder on the shared drive missing the licence.cfg file.

I have yet to reproduce this but the lack of a license file and other files that are created during the install on the shared drive is leading me to that conclusion.

I would suggest always pausing the passive node while installing on the active node to avoid any issues with the server failing over during the install. This should probably be added to the online cluster install steps.

We are currently seeing a failure to allocate memory error

Error in the programlog:

Failed to allocate memory for local stream 0x8007000e

Mail flow issues occur at the same time as these memory allocation errors. The mail flow issue is a result of the memory issue and not a cause.

we are looking to collect more data to find the root cause of this issue.

http://support.microsoft.com/kb/971063

This should be a much better engine choice. In the coming weeks I will be sharing any experiences we have with this engine.

One suggestion:

If you switch over to the Cloudmark engine you need to disable the Starengine service and disable updates for Spamcure. (I would then delete the Spamcure engine folder)

Spamcure will spool up its service on an update. Since this is not being used, you end up wasting 350-500MB of physical ram.

This is for everyone running Rollup 3 (You are running Rollup 3 right?)

Rollup 3 has added extra diagnostics that can/will effect performance on high load systems.

If you are running rollup 3 your program log will look something like this.

This logging is not needed, and makes the program log unusable and can cause timeouts.

The Fix

The following registry key will remove these messages. SP2 will also resolve this when it comes out in July

create the following key and set it to DWORD 0

HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432node\Microsoft\Forefront Server Security\Exchange Server\DiagnosticLoggingLevel

This will stop the diagnostic messages.