Forefront Server Security Management Console, Templates, and Revisions

neilcar — Fri, 11 Jul 2008 23:57:00 GMT

Sometimes, working in support, you come across a best practice or a bit of knowledge that is well-known to some people...but that bit of knowledge has never actually been documented. Today was one of those days.

While working in an environment with multiple Exchange Server 2003 servers running Antigen 9.1 Hotfix Rollup 3, we had to reinstall Antigen on one of the servers. We installed Antigen 9.1 and tested to make sure that mail was flowing after the install (it was). We then configured Antigen, including re-installing the FSSMC agent, redeploying the template for this server, and disabling Antigen performance counters.

At that point, things went off the rails. When we opened the Antigen admin console, it told us "You have 4 days left on your evaluation". Confused, we tried various things, including rebooting the box; however, every time, the console mocked us with it's eval message.

Since we were working in a maintenance window and we had run out of time, we made a decision to disable Antigen temporarily and investigate further. We took the template to a non-production Antigen 9.1 server and applied it. After applying it, we opened the admin console and we were greeted with "You have 4 days left on your evaluation".

At this point, we knew we were onto something. After working with our sustained engineering team to investigate further, we found out the root cause was something that is an FSSMC best practice even though I don't know that it's been written down before:

You should never apply a template created with a later version of Antigen or Forefront Server to an earlier version.

In this case, the template had been created in Antigen 9.1 Hotfix Rollup 3. As long as we applied it to servers running rollup 3 (or later), everything was A-OK; however, when we applied it to an Antigen 9.1 with no hotfix rollup on it, we ran into trouble.

The trouble, in this case, is that the schema for scanjobs was changed to add some additional options into the scanjobs. The template includes this new information but, once it's applied, the older code doesn't know how to handle it. This resulted in memory corruption which caused the false eval notice.

The takeaway here is that, if you're running a mixture of patch levels for your Antigen or Forefront Server servers, you have to be sure that the templates you are deploying with FSSMC were created in the earliest patch level you have in production. This will mean that you can't take advantage of any settings that are added in later patch levels but it also means that you won't run into issues like the one we wrestled with today.

Alternately, you could create templates for each patch level but I think that would end up being more difficult to manage.

Antigen 9.1 Hotfix Rollup 3 and Performance Monitor

neilcar — Wed, 09 Jul 2008 22:56:30 GMT

While investigating an issue where mail was queuing in the Exchange Information Store, we discovered an issue that affects customers running Antigen 9.1 Hotfix Rollup 3 when there are performance monitoring tools such as Perfmon, Perfwiz, and the MOM client running. This issue will manifest itself as mail queuing (and never un-queueing), particularly immediately after the store is restarted. In this particular instance, we were seeing this happen when we failed from one cluster node to another. This could also occur in non-cluster environments and it could occur if scanjobs are restarted for other reasons (such as scan timeouts).

Additionally, you may see entries in ProgamLog.txt similar to the following:

"ERROR: scanjobs.cpp::ConfigScanJobFile(): AddNewScanJob() Failed 0x80030021"
"ERROR: scanjobs.cpp::CheckScanJobs(): ConfigScanJobFile() failed. hr[0x80030021]"

"ERROR: Unexpected, RetrieveScanJobIdentifier could not find the index"
"ERROR: Problems retrieving ScanJob identifier from RegisterMonitor"
"ERROR: antigenvsapi.cpp::VSAPINavigatorThread(): RegisterMonitor() returned 8000ffff"

You may also see instances where you open the Antigen Administrator console and scanjobs are not visible.

The root cause of this is a regression in the Antigen performance counters DLL that results in Antigen services being unable to access the configuration information for scanjobs; thus, when the server is in this state, scanning processes cannot be started and the admin console cannot access scanjob configuration information.

These symptoms will not occur in all instances.

Recommendations:

If a server is having this issue, you should be able to resolve the immediate issue by stopping all applications that are performing performance monitoring and restarting Exchange services.

If you are running services/applications that gather performance data on your Exchange Server with Antigen 9.1 Hotfix Rollup 3, you can mitigate this in the short-term by disabling Antigen performance counters. The following steps will disable those counters:

At c:\program files\microsoft antigen for exchange\
Enter command: antigenpmsetup -uninstall
You will also have to restart any application that loads performance counters. Rebooting the server will accomplish this; however, short of that, you can run 'tlist -m antigenpmdll.dll' to get a list. (Tlist is part of the debuggers package.)

This will be resolved in Rollup 4 when it is released. After Rollup 4 is available, we recommend re-enabling Antigen performance counters by running 'antigenpmsetup -install'.

Neil Carpenter's Blog : Forefront

Forefront Server Security Management Console, Templates, and Revisions

Antigen 9.1 Hotfix Rollup 3 and Performance Monitor