Introduction:

In this scenario I am going to share a very unique issue which I came across while trying to join one of the TMG servers to the Stand alone array.

In this case we had two Enterprise Edition TMG servers installed on an Appliance.

 

Scenario:

We had two Enterprise Edition TMG servers and we were trying to join one of the servers to the Stand alone array pointing it to the other TMG server as the Array Manager.

But when we ran the ‘Join Array’ Wizard it failed with an error ‘KEYSET DOES NOT EXIST’ on the TMG server which we were trying to make the Array Member.

 

Troubleshooting:

Looking at the above error message it seems that TMG is trying to access some Folder/File which is either missing on the server or it does not have access to it.

So, we ran the Process Monitor on the TMG server while trying to join it to the Array. We filtered the Process Monitor file to show the results related to wspsrv.exe (Microsoft Forefront TMG Firewall Service) process.

And in the filtered trace we could see the following files being accessed by wspsrv.exe process:

2:02:36.6488085 AM        wspsrv.exe         2756       CreateFile           C:\ProgramData               NAME COLLISION            Desired Access: Read Data/List Directory, Synchronize, Disposition: Create, Options: Directory, Synchronous IO Non-Alert, Open Reparse Point, Attributes: N, ShareMode: Read, Write, AllocationSize: 0

2:02:36.6489655 AM        wspsrv.exe         2756       CreateFile           C:\ProgramData               SUCCESS              Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened

2:02:36.6489971 AM        wspsrv.exe         2756       QueryBasicInformationFile          C:\ProgramData               SUCCESS                CreationTime: 7/14/2009 8:50:08 AM, LastAccessTime: 5/5/2011 12:58:49 AM, LastWriteTime: 5/5/2011 12:58:49 AM, ChangeTime: 5/5/2011 12:58:49 AM, FileAttributes: HDNCI

2:02:36.6490214 AM        wspsrv.exe         2756       CloseFile              C:\ProgramData               SUCCESS             

2:02:36.6491280 AM        wspsrv.exe         2756       CreateFile           C:\ProgramData\Microsoft         NAME COLLISION                Desired Access: Read Data/List Directory, Synchronize, Disposition: Create, Options: Directory, Synchronous IO Non-Alert, Attributes: S, ShareMode: Read, Write, AllocationSize: 0

2:02:36.6492088 AM        wspsrv.exe         2756       CreateFile           C:\ProgramData\Microsoft\Crypto          NAME COLLISION                Desired Access: Read Data/List Directory, Synchronize, Disposition: Create, Options: Directory, Synchronous IO Non-Alert, Attributes: S, ShareMode: Read, Write, AllocationSize: 0

2:02:36.6493243 AM        wspsrv.exe         2756       CreateFile           C:\ProgramData\Microsoft\Crypto\RSA                NAME COLLISION          Desired Access: Read Data/List Directory, Synchronize, Disposition: Create, Options: Directory, Synchronous IO Non-Alert, Attributes: S, ShareMode: Read, Write, AllocationSize: 0

2:02:36.6494460 AM        wspsrv.exe         2756       CreateFile           C:\ProgramData\Microsoft\Crypto\RSA                NAME COLLISION          Desired Access: Read Data/List Directory, Synchronize, Disposition: Create, Options: Directory, Synchronous IO Non-Alert, Attributes: S, ShareMode: Read, Write, AllocationSize: 0

2:02:36.6496038 AM        wspsrv.exe         2756       CreateFile           C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys                NAME COLLISION            Desired Access: Read Data/List Directory, Synchronize, Disposition: Create, Options: Directory, Synchronous IO Non-Alert, Attributes: S, ShareMode: Read, Write, AllocationSize: 0

2:02:36.6498710 AM        wspsrv.exe         2756       CreateFile                C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\a63f7ad5b2228889fc41ae79c417446b_c49c7513-8b86-4d0e-90bf-4805804a9318         SUCCESS              Desired Access: Generic Write, Read Attributes, Disposition: Open, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: S, ShareMode: None, AllocationSize: n/a, OpenResult: Opened

2:02:36.6499060 AM        wspsrv.exe         2756       QueryStandardInformationFile                C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\a63f7ad5b2228889fc41ae79c417446b_c49c7513-8b86-4d0e-90bf-4805804a9318         SUCCESS              AllocationSize: 4,096, EndOfFile: 1,467, NumberOfLinks: 1, DeletePending: False, Directory: False

2:02:36.6499310 AM        wspsrv.exe         2756       WriteFile                C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\a63f7ad5b2228889fc41ae79c417446b_c49c7513-8b86-4d0e-90bf-4805804a9318         SUCCESS              Offset: 0, Length: 1,467, Priority: Normal

2:02:36.6499949 AM        wspsrv.exe         2756       CloseFile                C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\a63f7ad5b2228889fc41ae79c417446b_c49c7513-8b86-4d0e-90bf-4805804a9318         SUCCESS             

2:02:36.6501323 AM        wspsrv.exe         2756       CreateFile                C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\a63f7ad5b2228889fc41ae79c417446b_c49c7513-8b86-4d0e-90bf-4805804a9318         SUCCESS              Desired Access: Read Attributes, Delete, Disposition: Open, Options: Non-Directory File, Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened

2:02:36.6501620 AM        wspsrv.exe         2756       QueryAttributeTagFile                C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\a63f7ad5b2228889fc41ae79c417446b_c49c7513-8b86-4d0e-90bf-4805804a9318         SUCCESS              Attributes: SA, ReparseTag: 0x0

2:02:36.6501855 AM        wspsrv.exe         2756       SetDispositionInformationFile                C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\a63f7ad5b2228889fc41ae79c417446b_c49c7513-8b86-4d0e-90bf-4805804a9318         SUCCESS              Delete: True

2:02:36.6502116 AM        wspsrv.exe         2756       CloseFile                C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\a63f7ad5b2228889fc41ae79c417446b_c49c7513-8b86-4d0e-90bf-4805804a9318         SUCCESS             

2:02:36.6506463 AM        wspsrv.exe         2756       CreateFile           C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys                SUCCESS              Desired Access: Read Data/List Directory, Synchronize, Disposition: Open, Options: Directory, Synchronous IO Non-Alert, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened

2:02:36.6506747 AM        wspsrv.exe         2756       QueryDirectory                C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\a63f7ad5b2228889fc41ae79c417446b_* NO SUCH FILE                Filter: a63f7ad5b2228889fc41ae79c417446b_*

2:02:36.6507028 AM        wspsrv.exe         2756       CloseFile              C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys                SUCCESS             

As you can see in the above logs Firewall service is trying to access a particular file in the C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys folder on that server and due to some reason it was not able to access it and hence was showing the message as ‘NO SUCH FILE’.

So now it was pretty clear that wspsrv.exe was looking for a ‘machine key’ file in C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys folder but it was not able to find it or it did not have access to it.

As the Microsoft Forefront TMG Firewall Service runs under the NetworkService account, we tried to give Permissions to the NetworkService account on the C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\a63f7ad5b2228889fc41ae79c417446b file.  But while trying that we got an error ‘Access Denied’.

This was most likely due to the fact that the SYSTEM service account is typically the ‘owner’ of the machine key file in question, and therefore we did not have permissions to give Read permission to the Network Service account.

So it looked to be some permissions issue with the server itself.  We checked back with the Appliance vendor and came to know that the boxes were hardened. It looks as though the vendors hardening process may have changed the default permissions on the MachineKeys folder, it’s contents or both. They replaced that server with a new one and when we tried to join the TMG to the array on this new server, it got joined fine.

NOTE: The above troubleshooting was done on the TMG server which we were trying to join to the array and make it an Array Member.

CONCLUSION:

This was not a failure of the product. Problem was due to misconfiguration of server and overzealous hardening by the hardware vendor. TMG was working fine but not allowed to do what it should be able to in default installation.

Author:

Nitin Singh

Security Support Escalation Engineer

Microsoft CSS Forefront Security Edge Team

Technical Reviewers:

Billy Price

Security Sr. Support Escalation Engineer

Microsoft CSS Forefront Security Edge Team

Richard Barker

Security Sr. Support Escalation Engineer

Microsoft CSS Forefront Security Edge Team