1. Introduction

 

This post is about a scenario where Firewall Administrator notices that the application log in the ISA Server computer is full of events 14197 saying:

 

Event ID 14197

Source Microsoft Web Proxy

Type Error

Description ISA Server failed to write content to cache file. The error code in the Data area of the event properties indicates the cause of the failure.

 

He notices that after some time with this event one of the following symptoms happen:

·         The Firewall Service stops responding and when he tries to restarts it hangs with starting status for awhile before the status change to started.

·         Sometimes the Firewall Service quits without any message.

·         Sometimes it is not possible to restart firewall service because it hangs in stopping status.

 

I’m very familiar with those events and most of the time (9 out 10) this is caused by a third party software scanning or locking the cache file, the article http://support.microsoft.com/kb/887311 explains that in details. For this particular scenario the most common third party application that can cause this is the antivirus. When you have file scan antivirus installed on ISA Server (or TMG) you need to use the article Considerations when using antivirus software on ISA Server to create an exclusion list of the folders, files and process that shouldn’t be scanned by the antivirus.

 

So, if I follow the recommendations from that article I should be good, right? You should, as long as the exclusions are REALLY…. REALLY in place. I’m emphasizing the “really” because of some recent unfortunate experiences that I had; let me tell you a story to chill the bones about a thing that I saw…oops, that a song.

 

2. The Tales

 

Recently I worked in at least three different scenarios very similar where the firewall administrator created the exclusion list by following that article, however the exclusions were added though AV policy deployment located in the central antivirus server management. The reason why this policy was deployed via central location was because the local client AV installed on ISA didn’t allow local changes; everything was done by this AV Server. However even after making those exclusions the event 14197 was still happening. Of course I heard the most common argument in such scenarios: I told you that this was not a problem on our AV, this is an ISA issue for sure.

 

Ok, at that point I couldn’t really argue and had to think that every piece of software is innocent until we prove otherwise. So I followed this line of think I went to a path of collecting evidence to see who was doing that. Hunting time !!

 

3. Preparing the Environment

 

One way to collect evidences that there is a process other than ISA Server process is to audit the folder where the cache is located, by default %systemdrive%\urlcache. To do that you need to follow the steps below:

 

1. Enable local audit for object access (Start / Run / secpol.msc):

 

 

 

2. Enable Auditing for the URLCache folder for the following users:

 

 

 

Notice that the column access says special, this is because I’m choosing the following type of access:

 

 

 

The reason why I’m selecting those users is because:

·         Most of the ISA Server services run under System account and Firewall Service runs under Network Service account.

·         The Antivirus that I’m running on my lab (MS FCS) runs under System account.

·         I’m selecting everyone because I want to catch manual attempts (from users) to access this folder.

 

After finishing that you just need to wait for the next occurrence.

 

3. Collecting Evidences

 

In order to show what you will see in the event viewer when a process try to access the URLCache folder I decided to build a lab and use Forefront Client Security without the ISA folder exclusions. After implementing the actions above here what I got when FCS scanned the URLCache folder:

 

Event Type: Success Audit

Event Source:     Security

Event Category:   Object Access

Event ID:   560

Date:       1/30/2010

Time:       5:00:22 AM

User:       NT AUTHORITY\SYSTEM

Computer:   ISACONTN1

Description:

Object Open:

      Object Server:    Security

      Object Type:      File

      Object Name:      C:\urlcache\Dir1.cdat

      Handle ID:  1772

      Operation ID:     {0,6542847}

      Process ID: 1772

      Image File Name:  C:\Program Files\Microsoft Forefront\Forefront System\Client\AntiMalware\MsMpEng.exe

      Primary User Name:      ISACONTN1$

      Primary Domain:   CONTOSO

      Primary Logon ID: (0x0,0x3E7)

      Client User Name: -

      Client Domain:    -

      Client Logon ID:  -

      Accesses:   ReadAttributes

           

      Privileges: -

      Restricted Sid Count:   0

      Access Mask:      0x80

 

This is the evidence that you need in order to prove that there is another process accessing this file!!

 

At this point you probably are asking: OK, I got all that, but on your real scenarios, why the folder exclusion was not working? Well, that I don’t technically know since it was a third party AV that was failing to push the policy from the central location to the AV agent installed on ISA Server, hence the folder exclusion was not taking effect.