Forefront TMG Product Team Blog

KB: A recurring monthly report job does not run when expected on an array in Forefront Threat Management Gateway 2010

J.C. Hornbeck — Tue, 16 Apr 2013 16:09:42 GMT

Just wanted to let you know about a new KB article we published today. This one is a TMG article that talks about an issue where a recurring monthly report job is not created when you expect it.

You can find the complete article here:

KB2830886 - A recurring monthly report job does not run when expected on an array in Forefront Threat Management Gateway 2010 (http://support.microsoft.com/kb/2830886)

J.C. Hornbeck | Knowledge Engineer | Microsoft GBS Management and Security Division

Get the latest System Center news on Facebook and Twitter:

System Center All Up: http://blogs.technet.com/b/systemcenter/
System Center – Configuration Manager Support Team blog: http://blogs.technet.com/configurationmgr/
System Center – Data Protection Manager Team blog: http://blogs.technet.com/dpm/
System Center – Orchestrator Support Team blog: http://blogs.technet.com/b/orchestrator/
System Center – Operations Manager Team blog: http://blogs.technet.com/momteam/
System Center – Service Manager Team blog: http://blogs.technet.com/b/servicemanager
System Center – Virtual Machine Manager Team blog: http://blogs.technet.com/scvmm

Windows Intune: http://blogs.technet.com/b/windowsintune/
WSUS Support Team blog: http://blogs.technet.com/sus/
The AD RMS blog: http://blogs.technet.com/b/rmssupp/

App-V Team blog: http://blogs.technet.com/appv/
MED-V Team blog: http://blogs.technet.com/medv/
Server App-V Team blog: http://blogs.technet.com/b/serverappv

The Forefront Endpoint Protection blog : http://blogs.technet.com/b/clientsecurity/
The Forefront Identity Manager blog : http://blogs.msdn.com/b/ms-identity-support/
The Forefront TMG blog: http://blogs.technet.com/b/isablog/
The Forefront UAG blog: http://blogs.technet.com/b/edgeaccessblog/

Forefront Unified Access Gateway 2010 Service Pack 3 Rollup 1 is available for download

J.C. Hornbeck — Mon, 15 Apr 2013 16:26:43 GMT

We are happy to announce that Rollup 1 for Forefront UAG 2010 Service Pack 3 has been released.

UAG 2010 Service Pack 3 Rollup 1 is available as a hotfix download from Microsoft Support as an update to UAG 2010 Service Pack 3.

This important update contains 8 new fixes for reported issues as well as enhanced context tracing to more easily filter trace data per session.

For details, please visit KB 2827350: Description of Rollup 1 for Forefront Unified Access Gateway 2010 Service Pack 3

Please download the Forefront Unified Access Gateway (UAG) 2010 Service Pack 3 package now, and learn more about UAG 2010 SP3 by visiting our TechNet Library.

J.C. Hornbeck | Knowledge Engineer | Microsoft GBS Management and Security Division

Get the latest System Center news on Facebook and Twitter:

Windows Intune: http://blogs.technet.com/b/windowsintune/
WSUS Support Team blog: http://blogs.technet.com/sus/
The AD RMS blog: http://blogs.technet.com/b/rmssupp/

App-V Team blog: http://blogs.technet.com/appv/
MED-V Team blog: http://blogs.technet.com/medv/
Server App-V Team blog: http://blogs.technet.com/b/serverappv

TMG stopped processing web proxy requests

Forefront TMG Team — Tue, 09 Apr 2013 14:16:24 GMT

This post is about an issue I worked on several days ago.

Symptom:

========

My customer had a TMG array with two nodes running with NLB. The problem they faced was that from time to time some TMG node couldn't process traffic anymore: requests to the virtual IP (VIP) failed and only rebooting the TMG machine eliminated the issue.

IE was configured to use the NLB virtual IP (VIP) as proxy address. When the issue happened users couldn’t browse internet pages - the browser didn't show any error page, it just stayed with a blank page.

Troubleshooting:

=============

First of all, I looked at the TMG Web proxy log and found an error logged there with the result code of 11001 (WSAHOST_NOT_FOUND) for the request for “contoso.com”, which we tried to browse in order to reproduce the issue.

Then I took a look at a captured network monitor trace in order to find an appropriate network conversation between the client and TMG:

Time Delta time Source Destination Protocol Length Info

…

11:07:26.566261000 1.376436000 CLIENT_IP TMG_VIP HTTP 433 GET http://contoso.com/ HTTP/1.1

11:07:26.570933000 0.004672000 TMG_VIP CLIENT_IP HTTP 1440 HTTP/1.1 502 Proxy Error ( The host was not found. ) (text/html)

…

In a web proxy scenario, the web proxy is supposed to resolve the name, however, by some reason the node didn't manage to resolve the remote host.

As the next step, I filtered out the trace for DNS traffic and didn't find DNS packets on the node at all – this looked very suspicious.

Therefore I looked at Netstat and its output showed around ~62000 lines similar to the followings, whereas 2580 was the pid of wspsrv.exe:

…

UDP 0.0.0.0:4002 *:* 2580

UDP 0.0.0.0:4003 *:* 2580

UDP 0.0.0.0:4004 *:* 2580

UDP 0.0.0.0:4005 *:* 2580

UDP 0.0.0.0:4006 *:* 2580

UDP 0.0.0.0:4007 *:* 2580

UDP 0.0.0.0:4020 *:* 2580

UDP 0.0.0.0:4021 *:* 2580

UDP 0.0.0.0:4022 *:* 2580

UDP 0.0.0.0:4023 *:* 2580

UDP 0.0.0.0:4024 *:* 2580

UDP 0.0.0.0:4025 *:* 2580

UDP 0.0.0.0:4026 *:* 2580

…

So TMG seemed to use up all UDP ports – hence there was no free UDP port left to use as source port for dns queries. This explained why name resolution failed and why TMG returned 502.

Why was such a great amount of ports used by TMG?

My guess was that it might be doing it on behalf of a client. Therefore, I looked at the Firewall log which showed that there were huge amount of UDP requests from a single client to different external hosts.

Luckily TMG firewall client was installed on the client machine which gave us an application name:

…

7:27:51 Establish CLINET2_IP 2058 EXTERNAL_HOST_1 54150 0x0 Access Internet UDP domain\user smax4pnp.exe:3:5.1 TMG1 Unidentified IP Traffic

7:27:52 Establish CLINET2_IP 2033 EXTERNAL_HOST_2 9362 0x0 Access Internet UDP domain\user smax4pnp.exe:3:5.1 TMG1 Unidentified IP Traffic

7:27:53 Terminate CLINET2_IP 2026 EXTERNAL_HOST_3 59866 0x80074e20 Access Internet UDP domain\user smax4pnp.exe:3:5.1 TMG1 Unidentified IP Traffic

…

So TMG used up all the available ports due to the excessive amount of request from this client/application.

In order to resolve the issue the customer disabled the client that has been causing this huge UDP traffic.

Author:

Vasily Kobylin, Senior Support Engineer, EMEA Forefront Edge Team

Reviewer:

Balint Toth, Senior Support Escalation Engineer, EMEA Forefront Edge Team

How to configure the TMG Service Account to avoid problem with logging on SQL Server

Forefront TMG Team — Tue, 02 Apr 2013 11:26:56 GMT

One of the features introduced with TMG Service Pack 2 is to run the Firewall Service with a Domain account, this allow users to authenticate with Kerberos when using NLB.
Find more information about this feature here: http://technet.microsoft.com/en-us/library/hh454304.aspx

However you should pay attention when specifying the account name to avoid problems with logging to SQL Server, either local or remote.

The account specified is used by TMG to configure the service and also to create the Login in SQL Server.
For the TMG Firewall service to start any format is fine, but for SQL Server only the format domainName\loginName is valid.

For example if you want to use the account TMGSvc in the domain CONTOSO you have to enter CONTOSO\TMGSvc.

Using the UPN (User Principal Name) format or the FQDN (Fully Qualified Domain Name) does not work.
For example you cannot use TMGSvc@Contoso.com or Contoso.com\TMGSvc

The SQL Server documentation for the CREATE LOGIN command has the following note:

"When you are creating logins that are mapped from a Windows domain account, you must use the pre-Windows 2000 user logon name in the format [<domainName>\<loginName>]."

If you try using an invalid format you will see the Log Status as Disconnected and your LLQ folder growing:

Author:
Gianni Bragante
Support Engineer - Microsoft Forefront Edge Security Team

Reviewer:
Lars Bentzen
Sr. Escalation Engineer - Microsoft Forefront Edge Security Team

Clients Are Not Prompted to Choose a Certificate When Authenticating to ISA/TMG

Forefront TMG Team — Wed, 06 Mar 2013 15:14:30 GMT

Recently I have been seeing an increasing number of cases with the same symptom especially in the military and the government sector and even in contractors for the government. In these highly secure environments clients largely rely on the use of a “smart” card known as Common Access Cards (CAC) for authentication to their various types of servers and services.

Symptom

Your Internet Security and Acceleration Server (ISA) or Forefront Threat Management Gateway 2010 (TMG) Server is publishing resources internally/externally and your Web Listener is configure to use SSL Client Certificate Authentication. When clients navigate to the site that is published they would normally be prompted to choose their client certificate. Some or all of the clients are not being prompted to choose the certificate. On the ISA/TMG server, you may see a Warning in your Event Log with an Event ID of 36885.

Event Type: Warning
Event Source: Schannel
Event Category: None
Event ID: 36885
Date: date
Time: time
User:
Computer: COMPUTERNAME
Description: When asking for client authentication, this server sends a list of trusted certificate authorities to the client. The client uses this list to choose a client certificate that is trusted by the server. Currently, this server trusts so many certificate authorities that the list has grown too long. This list has thus been truncated. The administrator of this machine should review the certificate authorities trusted for client authentication and remove those that do not really need to be trusted.

Cause

This issue is caused when there are too many trusted certificate authorities in the Certificate Store on ISA/TMG. This is particularly common for servers that need a long list of Department of Defense (DoD) Certificate Authorities. When the list grows beyond 12,228 bytes (the maximum size the current Schannel security package supports) the list will be truncated. If the client doesn't receive the root CA that it needs because it has been truncated, it will not prompt to choose the certificate.

Resolution

There are a few workarounds for this but the one that is easiest to implement and seems to fit the needs of most organizations is below.

On the server or servers that are running ISA/TMG you will need to set the following registry entry to 0 (false):

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL

Value name: SendTrustedIssuerList
Value type: REG_DWORD
Value data: 0

By default the value is 1 (true).

For other possible workarounds please see this KB:

http://support.microsoft.com/kb/933430

Conclusion

Troubleshooting SSL Client Certificate issues can be tricky and time consuming. This issue was certainly difficult to identify the first time I saw it. Hopefully the information I have given you here can save you time, money, and aggravation.

Author:

Keith Abluton:

Security Support Escalation Engineer - MSD Security Team

Reviewer:

Richard Barker

Sr. Security Support Escalation Engineer - MSD Security Team

Access to remote FTP server through TMG 2010 may fail with error 550 (Access Denied)

Forefront TMG Team — Tue, 05 Mar 2013 10:00:00 GMT

Hi everybody!

In this article we will see how to troubleshoot an issue with accessing an FTP server behind TMG 2010.

Imagine we have the following situation: a client PC on an internal corporate network want to access a remote FTP server through TMG 2010 using an FTP client such as, for example, FileZilla.

The way the FTP is configured (authentication, encryption, ecc…) is out of interest for this case.

On the TMG server, we’ve created an access rule allowing “Read-Only” outbound requests for the FTP protocol:

When we try to connect to our remote FTP server using, for example, FileZilla, we may face the following error:

FTP connection issues through ISA/TMG could be related to many different aspects.

In the following article it’s possible to find a resolution for many of the most common problems:

http://technet.microsoft.com/en-us/library/bb794745.aspx

The problem we’re focusing on in this article, however, is not included in the above troubleshooting guide and depends on a specific by-design behavior of TMG server.

Basically, in our case we see that the connection attempt is failing due to a “550-Access Denied” error after having performed a MLSD command.

What is MLSD exactly ?

Here we can find a description of what MLSD is used for:

http://tools.ietf.org/html/draft-ietf-ftpext-mlst-16#section-7

As we can see from the above:

The MLST and MLSD commands are intended to standardize the file and directory information returned by the Server-FTP process. These commands differ from the LIST command in that the format of the replies is strictly defined although extensible.

In the default configuration of the TMG FTP Access filter in “Read-Only Mode”, the filter will only allow a specific subset of FTP commands. The MLSD command is not included in this set of “Read-Only” commands. FTP clients using LIST command will not experience this problem, since LIST is an allowed command.

Its easy to resolve the problem by allowing write-permissions in the FTP-Filter advanced properties of our access rule:

Now, granting write rights is not always a good choice, and most of the times this is not allowed nor suggested.

Nevertheless, a workaround exists for this situation: in fact, it’s possible to add the MLDS command in the “allowed-commands list” of the “Read-only” TMG FTP filter.

The following MSDN article explains how to configure add-ins:

http://msdn.microsoft.com/en-us/library/dd435753.aspx

Specifically:

FTP Access Filter

FTP Access Filter is an application filter that is installed with Forefront TMG. It enables FTP protocols. When running in read-only mode, FTP Access Filter blocks all commands in the control channel except the following commands: ABOR, ACCT, CDUP, CWD /0, FEAT, HELP, LANG, LIST, MODE, NLST, NOOP, PASS, PASV, PORT, PWD /0, QUIT, REIN, REST, RETR, SITE, STRU, SYST, TYPE, USER, XDUP, XCWD, XPWD, SMNT. This should block any writing to the server side. The default list of allowed commands can be replaced by a customized list that is written to the collection of vendor parameters sets (FPCVendorParametersSets) associated with the filter. The Firewall service must restarted for the new settings to take effect.

The above article provides a script example through which it is possible to customize FTP filter list. This way, it will be possible to keep the filter configured in Read-Only mode, and also allow the FileZilla connection to work as expected.

Hope this can be useful!

Let's see you back with the next topic!!

Author:
Daniele Gaiulli

Support Engineer – EMEA Forefront Edge

Reviewer:
Philipp Sand

Support Escalation Engineer – EMEA Forefront Edge

You can remotely manage the Enterprise Policy, but not the Array Policy

Forefront TMG Team — Fri, 08 Feb 2013 15:10:27 GMT

I’ll try to elaborate on the issue using as many illustrations and snapshots as possible. When I came across this issue, it was quite surprising.

32-bit Remote Management Client

In the TMG environment, we are using a single EMS (Enterprise Management Server) with a single Array. There are two TMG nodes joined to this array. To manage the environment we are using a Windows 7 32-bit machine with a 32-bit client. Please use this link download the 32-bit client (TMG_ENU_Management_x86.exe). Note that you will need to login with a Microsoft Live Id and register in order to download.

Once downloaded, install the client and connect to the EMS server using its FQDN. Make sure EMS is configured to allow remote management, refer to the below mentioned articles.

· About Forefront TMG roles and permissions - http://technet.microsoft.com/en-us/library/dd897006.aspx#BKMK_RolesAndPermissions

· Configuring roles and permissions - http://technet.microsoft.com/en-us/library/dd441007.aspx

The relevant Users on the list should be able to gain access to the TMG EMS server for administration.

After assigning the correct set of permissions and remote access to TMG EMS server, you can remotely access the Enterprise Policy and make allowed changes.

But while accessing Array nothing displays, it doesn’t even shows the arrays created in the enterprise. Refer to below mentioned Snips.

Here in this snip we can see that Enterprise policy is displayed.

Here you can see the focus is on “Arrays”, but no policies are displayed.

Let’s check and compare the version on TMG EMS server and then on this client.

This is the version number from TMG EMS server which is updated to latest i.e. SP2 RU2

This is the version number from client which is updated to SP1 UP1.

Cause

The major cause of this is due to version mismatch between the management console and the TMG enterprise. For example the TMG enterprise is at SP2 RollUP2 update level which is build number 7.0.9193.540 and the TMG management console on remote 32bit machine is at RTM which is build number 7.0.7734.100 (refer to the article).

Solution

This can be resolved by updating the TMG RTM management console on 32bit Remote machine. Refer to the links mentioned below to download and install the relevant updates.

· SP1

· SP1 UP1

· SP2

This completely depends on the version level of the TMG environment. Check the article mentioned below for all the relevant TMG versions.

http://blogs.technet.com/b/keithab/archive/2011/09/27/forefront-tmg-2010-service-pack-rollup-and-version-number-reference.aspx

Match the version with the updates to the level TMG EMS server is on. There are no rollups released for the MMC.

After updating the client to SP2 we were able to access the array policies on the client machine.

64-bit Remote Management Client

Now goes the story for 64-bit setup. Just to mention there is no separate TMG mmc console installation msp available. For this installation, the TMG 2010 ISO/DVD is used.

NOTE: The following 4 steps outline the default MMC install using the install media.

You may have followed these steps, believing that you would be able to manage TMG EMS remotely using the MMC.

1. TMG setup from installation ISO/DVD by starting Preparation Tools first.

2. On Welcome screen accepted the terms for installation.

3. On Installation type dialog box selected Forefront TMG Management.

4. Wizard finishes its work.

After a default installation of the MMC (from the install media), you may be surprised to find out it doesn’t work . Because TMG is at SP2 or above update level and the MMC installation is at RTM level.

There are updates available which can be used to bring the MMC to the same update level as the TMG EMS server is at. But the procedure used for 32bit installation doesn’t work for 64bit.

I know there are a lot of questions surfacing, but I have the answer.

Because the 64bit mmc is installed straight from the install media, we’ll have to update the installation itself to SP2/ relevant to your environment.

To do this, we’ll need to create a “TMG 2010 Slipstream” installation, in which we update the TMG installation MSI itself.

Steps for TMG 2010 Slipstream installation.

1. If the TMG 2010 MMC console was previously installed directly from the install media, you’ll need to uninstall it from Control Panel >> Programs and Feature.

2. Copy all the contents from TMG 2010 ISO/DVD to a folder on HDD. In this example, we will copy the contents to C:\TMG.

3. Download the following TMG 2010 updates; making sure you download all 64bit versions. Use the following links:

a. SP1 - http://www.microsoft.com/en-us/download/details.aspx?id=16734

b. UP1 - http://www.microsoft.com/en-us/download/details.aspx?id=11445

c. SP2 - http://www.microsoft.com/en-us/download/details.aspx?id=27603

4. Once you have downloaded all three files, copy the files to C:\TMG\FPC

5. The UP1 and SP2 are in .exe format, therefore we will need to extract the msp files so they can be used for slipstreaming TMG 2010.

6. Open a command prompt with elevated privileges and, in the C:\TMG\FPC folder, execute the following commands.

a. SP1 Update1 - TMG-KB2288910-amd64-ENU.exe /t TMGSP1U1

b. You’ll get a dialog after completion.

Click ok to close.

c. SP2 - TMG-KB2555840-amd64-ENU.exe /t TMGSP2

d. You’ll get a dialog after completion.

Click ok to close.

7. Below is the snip for commands and folders I used.

8. Under C:\TMG\FPC, there should be two new folders called TMGSP1UP1 and TMGSP2. Both of these folders will contain the extracted msp file.

9. Copy the msp files to FPC folder.

10. Now let’s create slipstream for TMG 2010. Follow the commands and make sure you update it to the same level as TMG EMS.

a. SP1 - msiexec /a ms_fpc_server.msi /p tmg-kb981324-amd64-enu.msp

This will initiate installation wizard, which will slipstream the TMG2010 installation with SP1

b. SP1UP1 - msiexec /a MS_FPC_Server.msi /p TMG-KB2288910-amd64-ENU.msp

This will initiate installation wizard, which will slipstream the TMG2010 installation with SP1UP1.

c. SP2 - msiexec /a MS_FPC_Server.msi /p TMG-KB2555840-amd64-ENU.msp

This will initiate installation wizard, which will slipstream the TMG2010 installation with SP2.

11. Next, you can delete the following highlighted files and folders from C:\TMG\FPC.

12. Once deleted, the C:\TMG\FPC folder appear as follows:

13. Now create an ISO/DVD of the entire C:\TMG folder. Make sure you do not create and ISO/DVD out of only FPC folder.

14. Now this ISO/DVD can be used to install TMG mmc console on a 64-bit client machine using the steps mentioned below.

a. Start TMG setup from installation ISO/DVD by starting Preparation Tools first.

b. On Welcome screen click next and accept the terms for installation.

c. On Installation type dialog box select Forefront TMG Management only.

d. Let wizard to finish its work and then click Finish. This will start mmc installation wizard.

e. Once installation finishes you can access the array policies as well, provided that appropriate permissions are assigned.

Thanks for reading through, I hope I was able to clear your doubts and provide a solution. If you are still facing the issue then I would recommend opening a case with Microsoft CSS.

Author:

Vivek Kumar Sharma

Support Engineer – MSD Security Division

Reviewers:

Junaid Jan

Security Support Escalation Engineer – MSD Security Division

TMG SP2 Rollup 3 available

Forefront TMG Team — Thu, 17 Jan 2013 14:14:00 GMT

We are happy to announce the availability of Rollup 3 for Forefront Threat Management Gateway (TMG) 2010 Service Pack 2 (SP2). TMG SP2 Rollup 3 is available for download here: Rollup 3 for Forefront Threat Management Gateway (TMG) 2010 Service Pack 2

Please see KB Article ID: 2735208 for details of the fixes included in this rollup. The Build Number for this update is: 7.0.9193.575

To install this update, you must be running Forefront Threat Management Gateway 2010 Service Pack 2.

For more information about Forefront Threat Management Gateway 2010 SP2, please see the following Microsoft website:

Download information for Forefront TMG 2010 SP2

Thank you,

Forefront TMG Team

TMG services hang at startup due to third party service

Forefront TMG Team — Wed, 02 Jan 2013 13:37:33 GMT

This post is, once again, about an issue I worked on few days back. Before I start discussing the issue, and how I resolved it, I would like to outline the objective of this post.

The objective of this post is to make TMG administrators aware of issues like this; and what can be done to resolve them. Discovering the root cause of this issue required a User Mode Dump analysis.

Performing a User Mode Dump analysis requires “Symbol” files (which are private). My goal is not to provide specific instruction on User Mode Dump analysis, but instead to show what kind of information can be gathered, and how it can be used, to help troubleshoot boot-time “service issues” on a TMG server.

For those that are not familiar with dump analysis terms like process, threads and its stack, I will elaborate further as I explain the steps.

Issue:

TMG server admin was rebooting the server and at the time of reboot TMG services were hanging and were not starting. A similar issue was reported pre TMG sp2 but it was fixed post sp2. In this scenario TMG was updated to latest build i.e. TMG sp2 RU2.

Troubleshooting:

Some background: It should be noted that quite a bit of troubleshooting had taken place prior to my involvement in the case. This includes the steps in the following Knowledge Base article:

Forefront Threat Management Gateway 2010 services do not start as expected when the FTMG 2010 servers are in a workgroup array

During startup, the following System Event was logged…

_____________________________________________________________________________________________

Log Name:      System
Source:        Service Control Manager
Date:          09/11/2012 17:42:30
Event ID:      7022
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      server1
Description:
The Microsoft Forefront TMG Firewall service hung on starting.

Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
    <Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" EventSourceName="Service Control Manager" />
    <EventID Qualifiers="49152">7022</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8080000000000000</Keywords>
    <TimeCreated SystemTime="2012-11-09T17:42:30.378163900Z" />
    <EventRecordID>344470</EventRecordID>
    <Correlation />
    <Execution ProcessID="716" ThreadID="720" />
    <Channel>System</Channel>
    <Computer>server1</Computer>
    <Security />
</System>
<EventData>
    <Data Name="param1">Microsoft Forefront TMG Firewall</Data>
</EventData>
</Event>

_____________________________________________________________________________________________

Data collection:

During the course of troubleshooting we collected a User Mode Dump while reproducing the issue.

User mode dumps collection reference: http://msdn.microsoft.com/en-us/library/ff420662.aspx

Data analysis:

Note: The approach taken in this post is very similar to guidelines given in the following link about debugging a deadlock as we were in a scenario similar to a deadlock: http://msdn.microsoft.com/en-us/library/windows/hardware/ff540592(v=vs.85).aspx

In the dump, I found following critical section was locked :

Note: For more information about critical section and locked critical section, please refer to: http://msdn.microsoft.com/en-us/library/windows/hardware/ff541979(v=vs.85).aspx

Then I located the owning thread of this locked critical section. In following snapshot we can see the stack of this thread. The stack is read from bottom to top. From this call stack it appears that wspsrv (firewall service) is trying to load a filter called XSISAPI. It appears TMG has deferred its filters’ startup until this filter (i.e. XSISAPI)is loaded.

I then checked the module for this filter (i.e. XSISAPI) and found that it’s a filter called “Afaria” from Sybase.

Solution:

We configured the XSISAPI filter service to delayed start. After this change, the TMG services started normally after reboot.

Author:

Suraj Singh:

Security Support Escalation Engineer - MSD Security Team

Reviewer:

Richard Barker

Sr. Security Support Escalation Engineer - MSD Security Team

TMG sources outgoing packets with Secondary IP addresses

Forefront TMG Team — Mon, 31 Dec 2012 14:03:23 GMT

Hello Everyone! We’ve seen a few cases lately dealing with TMG servers sourcing outgoing packets with secondary IP addresses that have been added to the NICs. This could cause issues in communications between nodes or possibly other issues. One such example that I have seen come across is where a customer had a TMG server being utilized as an internal firewall behind a 3^rd party Edge firewall. Clients were utilizing the TMG server as their proxy server. When the http requests left the external interface of the TMG server the packets were sourced with a secondary IP address of the External NIC on the TMG instead of the primary address of that NIC. When the Edge firewall received the packets it dropped them because its rules were configured to only allow packets out when sourced with the primary IP address of the TMG’s external interface. This of course broke internet connectivity for all internal clients.

The question at hand is… “Why is the TMG server sourcing packets with a secondary address instead of the primary address of the NIC?”

The answer to that question deals with the differentiation between the Network Stack in Server 2008 and above and Server 2003 and below. Server 2003\XP and below were based off the Weak Host Model. Basically, in a Weak Host Model the primary address of the adapter with a route that most closely matches the target IP address is used as the Source IP Address.

In server 2008\Vista and above we re-architected the Network stack. It is based on the Strong Host Model. In the Strong Host Model the concept of a Primary IP Address doesn’t exist. To determine the IP address that is utilized it looks at the routing table to decide the proper NIC to utilize, then uses the process defined in RFC 3484 to choose the source IP for outbound packets. Here is a basic breakdown of how the windows implementation of RFC 3484 chooses an IP address:

Windows Source IP V4 address selection:

- Rule 1 Prefer same address (applies)

- Rule 2 Prefer appropriate scope (applies)

- Rule 3 Avoid deprecated addresses (applies)

- Rule 4 - Prefer home addresses - does not apply to IP v4

- Rule 5 Prefer outgoing Interfaces (applies)

- Rule 6 Prefer matching label - does not apply to IP v4

- Rule 7 Prefer public addresses - does not apply to IP v4

- Rule 8a: Use longest matching prefix with the next hop IP address. (not in RFC!)

"If CommonPrefixLen(SA, D) > CommonPrefixLen(SB, D), then prefer SA. Similarly, if CommonPrefixLen(SB, D) > CommonPrefixLen(SA, D), then prefer SB."

*This says that the IP with the most high order bits that match the destination of the next hop will be used.

Note: Rule 8 - Use longest matching Prefix is similar to rule 8a except the match is with the destination IP address rather than the next hop IP address.

For example, use the following addresses as an example of choosing the longest matching prefix:

TMG Servers External IP Addresses:

192.168.1.14/24
&
192.168.1.68/24

Default Gateway:

192.168.1.127

Convert these addresses into binary:

192.168.1.14 = 11000000 10101000 00000001 00001110
192.168.1.68 = 11000000 10101000 00000001 01000100
192.168.1.127 = 11000000 10101000 00000001 01111111

The 192.168.1.68 address has more matching high order bits with the gateway address 192.168.1.127. This would cause the server to utilize what was originally defined as the “secondary” as the Source IP address of outgoing packets.

*For more information on RFC 3484 please refer to the following link: http://www.ietf.org/rfc/rfc3484.txt . Please note that IPv6 is referenced in RFC. Windows utilize the same process for IPv4 sourcing.

You can also review the following TechNet article for supported document details on the above information:

The functionality for source IP address selection in Windows Server 2008 and in Windows Vista differs from the corresponding functionality in earlier versions of Windows
http://support.microsoft.com/kb/969029

So now we know why your TMG server may be sourcing your packets with what you call your “Secondary IP Address”. It isn’t TMG at all. It is the default behavior of the server itself. Your server version is Server 2008 or above. The question is…

“Can I configure my Server 2008 or above in a way that it will only utilize the first IP address as a Source address?”

The answer to that is YES! There is actually a Netsh command that can be utilized to add IP addresses. In that command you use the “SkipAsSource” flag and it will no longer use the IP address you are adding as a Source IP Address. This means that you will have to temporarily remove the IP Address you are having the issues with then re-add them utilizing the Netsh command (This means you will have to have a maintenance window!). Here are examples of the command lines you will use:

Server 2008:
Netsh int ipv4 add address <Interface Name> <ip address> <subnet mask> skipassource=true

Server 2008 R2:

Netsh int ipv4 add address <Interface Name> <ip address> skipassource=true

* For details and prerequisites to utilize these commands please refer to the following articles:

All IP addresses are registered on the DNS servers when the IP addresses are assigned to one network adapter on a computer that is running Windows Server 2008 SP2 or Windows Vista SP2
http://support.microsoft.com/default.aspx?scid=kb;EN-US;975808

IP addresses are still registered on the DNS servers even if the IP addresses are not used for outgoing traffic on a computer that is running Windows 7 or Windows Server 2008 R2
http://support.microsoft.com/default.aspx?scid=kb;en-US;2386184

Keep in mind that I gave only one specific example where this may be causing an issue. There may be other problems you run into where the Netsh entry I listed may help you out. No telling… it may not even be on your TMG servers. Maybe you see the issue on your UAG servers, or any other server for that fact.

I hope the information provided helps out!

Author

Brett Crane - Sr Security Support Escalation Engineer, Microsoft CTS Forefront Security Edge Team

Reviewer

Richard Barker - Sr Security Support Escalation Engineer, Microsoft CTS Forefront Security Edge Team