Some of you may have already heard and know what NLB is and how it works as described in the general Network Load Balancing Overview [http://technet.microsoft.com/en-us/library/cc725946.aspx].
An integral part of a TMG NLB solution is Bi-direction affinity, which is well described at the following link:
Bi-Directional Affinity in ISA Server [http://blogs.technet.com/b/isablog/archive/2008/03/12/bi-directional-affinity-in-isa-server.aspx].
Bi-directional affinity creates multiple instances of Network Load Balancing (NLB) on the same host, which work in tandem to ensure that responses from published servers are routed through the appropriate ISA servers in a cluster. Bi-directional affinity is commonly used when NLB is configured with Internet Security and Acceleration (ISA) servers. If bi-directional affinity is not consistent across all NLB hosts or if NLB fails to initialize bi-directional affinity, the NLB cluster will remain in the converging state until a consistent teaming configuration is detected.
Bi-directional affinity is a crucial thing if you enable NLB on multiple interfaces, as it ensures a single client to work through the same node and have consistent data flow.
By default when a client connects to an NLB interface a hash algorithm based on the packet source IP (client) is computed in the NLB driver to decide which NLB node should handle this request. On the way back (the server responses to the client) the source IP is the server IP (not the client IP) and without BDA it may be handled by another TMG NLB node – which would discard the server response, not having seen the client request. Hence, a mechanism is needed to guarantee that client/server packets are handled by the same host in the array.
Bidirectional affinity ensures the server responses are handled by the same TMG NLB node as the original client request. The mechanism ensuring this functionality is implemented as so-called hook rules:
http://technet.microsoft.com/en-us/library/dd348817(v=ws.10).aspx
Filter hooks help to direct traffic in a Network Load Balancing (NLB) cluster by filtering network packets. If the filter hooks are not properly configured, the NLB cluster will continue to converge and operate normally, however, the server application that is running with NLB will not be able to properly register the hooks.
The essential logic of the hook rules is the following:
At each packet, NLB calls out to the registered drivers (in this case fweng) whether they want to modify how the hash is created.
For example, when a client sends a SYN packet, NLB “asks” TMG’s fweng driver how it should calculate the hash. TMG will tell (depending on its hook rules) to use for example the client source IP for hashing (which is the default behavior). The calculated hash instructs NLB for example that the first node should handle the traffic and pass the SYN to the backend server.
When the server responds to the internal NLB of the TMG array, NLB calls out again to ask TMG how the hash is supposed to be calculated.
Based on the same hook rule set and seeing the packet direction, TMG tells NLB to hash based on the destination IP, which is again the client IP, so the packet will be handled by the same node as the original packet.
If the hook rule was not present , TMG would tell to use the default behavior (use the source IP), which would result in calculating the hash based on the source (server) IP, possibly yielding that a different node is supposed to handle the traffic, which is not what we want.
If you have a TMG array with several nodes and NLB is enabled, then TMG service creates hook rules at start.
These rules can be checked by executing netsh tmg show nlb from an elevated command prompt, which yields similar output as can be seen below.
Notice the rules have a source and destination range, and a direction based on which TMG can decide what to tell NLB when its called: whether to hash on source (forward) or destination IP (reverse).
A potential problem occurs if hook rules are missing. In this post, we are going to explore a potential cause for missing hook rules.
The below is from a sample test lab which we built based on a particular issue we got reported by a customer:
In the example above we can see only rules which were created and hash based on source (forward direction) for outgoing scenario.
You however do not see any reverse rules, indicating that some rules may be missing .
I took those rules from my TMG Array with Internal 10.0.0.0/24, DMZ 192.168.0.0/24 and External 172.20.1.0/24 networks.
Let's imagine a scenario where we have created a publishing rule for a server from DMZ with a listener on External network and you have configured the rule to be half NAT (request appears from client).
Because there is no specific rule for the range external network -> DMZ, DMZ -> external, in both directions we use the default behavior to hash based on the source IP.
As described above, because the hook rule is missing, this may or may not work depending on client IP/published server twin. If the NLB hash algorithm gives the same NLB node ID for both the client and the server IP , it will work. Otherwise, client and server packets will be serviced by different hosts, and the published server responses will be dropped with the error 0xc0040017 FWX_E_TCP_NOT_SYN_PACKEP_DROPPED.
The reason of the issue is lack of some NLB hook rules.
These rules are created at the startup based on network rules. In a real world scenario, when we may have a lot of subnets, it's quite easy to miss a network rule between two networks.
In this case, this is exactly what happened - there was no network relationship defined between External and DMZ, hence the appropriate rule was never created
Once we add the network rule, the hook rules will be created. I created the rule from External to DMZ network with Route relationship. In the output below you can see how hook rules changed.
Now we have appropriate rules for processing requests from External to DMZ network back and forth, ensuring that we hash using the same IP in both directions. Hence, we should not get error 0xc0040017 FWX_E_TCP_NOT_SYN_PACKEP_DROPPED any more.
If you see the above error, make sure to check whether the appropriate hook rules are present – one of the root causes for missing hook rules can be the missing network relationship definition.
Author:
Vasily Kobylin, Senior Support Engineer, Microsoft EMEA Forefront Edge
Reviewer:
Balint Toth, Senior Support Escalation Engineer, Microsoft EMEA Forefront Edge
Franck Heilmann, Senior Escalation Engineer, Microsoft EMEA Forefront Edge
In a previous article we explained how to create a self-signed CNG certificate, suitable for the HTTPS Inspection feature, which can be used to inspect sites using an SHA-256 certificate.
In this article we will explain how to generate a similar certificate using your internal CA based on Windows 2008 R2.
Using a certificate issued by your own CA, rather than a self-signed certificate, has the advantage that you will not have an additional certificate to distribute to the web proxy clients.Bearing in mind that certificates have a limited validity period and that this eases operation maintenances, since you need only to change only the Subordinate CA certificate on TMG.
Arguably it can be said that if you set a validity period long enough, as some tenth of years, this would not be an issue.But the truth is that what is considered secure today, might be easily compromised in a couple of years and for that reason, having an easy way to replace your certificates also improves your overall security.The ability to quickly move to a larger key length or better hash algorithm at any time is the key to an always up to date and secure PKI infrastructure.
Now let's go ahead and create the certificate.Begin by opening the Certificate Authority administration console, right click on Certificate Templates then Manage.
First step is duplicating the “Subordinate Certification Authority” template.
In Compatibility settings select Windows Server 2008:
Type a display name for the template:
Put the checkbox on “CA certificate manager approval” if you prefer to. If you do you will have to approve the request once submitted.
In the Extensions tab select Key usage then Edit:
Put the checkbox on “Certificate signing” only
Ensure the local Administrator, or your administrative user/group, has Enroll permissions:
Right click on Certificate Templates, New, Certificate Template to Issue:
Select then newly created template then click OK:
The new template will then appear as available:
Then open MMC, add the Certificates snap-in for the Local computer.
Open the Personal container, right click on Certificates, All Tasks, Advanced Operations, Create Custom request:
Select Active Directory Enrollment Policy then click Next:
Select the template then click Next:
Click on Details then Properties:
Enter a Common name then click Add:
Enter a friendly name for the certificate:
In the Private key tab ensure the CSP “Microsoft Software Key Storage Provider” is listed:
Ensure “Mark private key exportable is selected”:
In Select Hash Algorithm select “sha256”:
Then click OK, you will be asked to save the request to a file:
Open an elevated command prompt and execute the command:
certreq -submit -config "<ServerName\CAName>" "<CertificateRequest.req>" "<CertificateResponse.cer>"
The last parameter is required if you have not requested an approval before issuing the certificate.
If no approval is required a .cer will be retrieved, otherwise you will have to refer to the RequestID that will be displayed:
If the approval is required, once the request has been approved in the CA console, enter the following command to retrieve the .cer file:
certreq -retrieve -config "<ServerName\CAName>" <RequestID> "<CertificateResponse.cer>"
Then complete the request with the following command:
certreq -accept -config "<ServerName\CAName>" "<CertificateResponse.cer>"
At this point you will find the new certificate in the Personal store of the local computer and you can export it:
Ensure you export the private key:
Do not select any of the PFX options:
Provide the PFX password when requested and save the file.
Then export the certificate again, this time without the private key:
Click Next then select the DER format:
Click Next then save the .CER file.
Notice that the certificate you have just generated might be signed using the SHA1 Hash algorithm, although we have requested a certificate using SHA256 Hash Algorithm. This happens due to the fact that in some operating systems the Subordinate Certification Authority template forces the key hashing algorithm to be SHA1. This however does not prevent TMG from issuing (signing) certificates with SHA256 hash algorithms.
Also note that Microsoft is phasing out weaker certificates such as SHA1 certificates and certificates with keys under 1024 bits in length. So expect at some point in time this to change.
So don’t be surprised and go ahead with the next steps.
Copy the PFX and the CER files to the TMG box, open the HTTPS Inspection configuration and import the certificate form the PFX file:
Then save and apply the configuration
On the TMG server open an MMC console and add the Certificates snap-in for the local computer.Then import the .cer file in the Intermediate certificate store.You will need to restart the TMG server.
Importing the certificate in the Intermediate store is necessary for the TMG server to send the intermediate certificate in the certificate chain so that the client is able to build the trust chain.
This is further explained bellow.
Now you can try from a client browsing a site using a CNG certificate, such as Twitter.You will see the certificate being signed by the new certificate from your CA:
Notice that this certificate is signed using SHA256:
Since the Subordinate CA Certificate (“TMG HTTPS Inspection CNG Ent.CA” in this case) is not published anywhere the clients can’t, on its own find the certificate SubCA certificate.
For a normal certificate issuing CA you would be able to publish the SubCA Certificate and publish to either a LDAP or HTTP location and the clients would be able to look it up, by downloading the certificate from the AIA Extension in the certificate.
In the case of TMG issued certificates, for HTTP inspection, these don’t have an AIA extension.
TMG will send both the certificate for the URL being accessed on the browser (or other client) and the Subordinate CA, configured in TMG to issue these certificates.
Otherwise the client would end up with a certificate that do not built up to a trusted root, having a "gap" in the chain. So as you can see bellow TMG will send the two certificates on “Server Hello” handshake.
The certificate generated, on the fly by TMG, for the HTTPS site you are visiting, will appear as being issued by an intermediate CA represented by the certificate you generated.
That is why it is essential to add the SubCA to the “Intermediate Certification Authorities” store for the “Local Computer”. Otherwise the TMG issued certificates would need to have the AIA sections which would require the intermediate certificate (the one you have just generated) to be published to a AIA location. This would require further configuration on your environment, so TMG provides some help in the Chain validation process and sends the intermediate CA in the "Certificate" section of the SSL handshake as we can see in the “Server Hello” handshake on the network capture above.
Your “Subordinate CA” (TMG HTTPS Inspection CNG Ent.CA) will then have an AIA Extension and from there up to the Root CA. As you can see from the image
This will then allow to build up a valid certificate chain ending up in your Internal CA and starting in the leaf certificate issued by TMG for HTTP Inspection. The bellow picture shows the expected Certificate chain.
Author:Gianni Bragante Support Engineer – Microsoft Forefront Edge Security Team
Luis SousaSupport Engineer - Microsoft PKI/AD Team
Reviewer:Philipp Sand Sr. Support Escalation Engineer – Microsoft Forefront Edge Security Team
We are happy to announce the availability of Rollup 5 for Forefront Threat Management Gateway (TMG) 2010 Service Pack 2 (SP2). TMG SP2 Rollup 5 is available for download here: Rollup 5 for Forefront Threat Management Gateway (TMG) 2010 Service Pack 2
Please see KB Article ID: 2954173 for details of the fixes included in this rollup. The Build Number for this update is: 7.0.9193.644
To install this update, you must be running Forefront Threat Management Gateway 2010 Service Pack 2.
For more information about Forefront Threat Management Gateway 2010 SP2, please see the followingMicrosoft website: Microsoft Forefront Threat Management Gateway 2010 Service Pack 2
Download information for Forefront TMG 2010 SP2
Thank you,
Forefront TMG Team
If you have configured HTTPS Inspection on TMG you may not be able to access some sites such as Twitter and Dropbox. In this scenario clients will get a blank page and in the TMG logs you will see the error 0x8009000a
This happens when:
Web site that are using certificate with suite-B algorithm
TMG certificate used by HTTPS inspection feature to sign the certificate that will be sent to client is not compatible with suite-B certificate.
To resolve the problem you need to install a CNG certificate for the HTTPS Inspection feature
This certificate can be issued by your internal CA or it can be a self-signed certificate that you can generate with a script (see below).
This certificate must have the “Certificate signing” key usage You can read more about CNG certificates in this article: http://msdn.microsoft.com/en-us/library/windows/desktop/bb931380(v=vs.85).aspx
As example here is a PowerShell script that create a self-signed CNG certificate:
#SCRIPT - Generate Self-signed CNG Certificates for Certificate signing purpose, This will be used by TMG Https Inpection #AUTHOR - Franck Heilmann - Microsoft Corporation #VERSION - 1.1 #$ErrorActionPreference = "Stop" Write-Host "`n WARNING: This script sample is provided AS-IS with no warranties and confers no rights." -ForegroundColor red Write-Host "`n This script sample will generate self-signed CNG Authority certificate to be used by TMG HTTPS Inspection feature" Write-Host " in the Local Computer Personal certificate store.`n Private is can be exported. As well the .cer and .pfx files will be save ini the provided directory`n`n" $outputDir = Read-Host "`nEnter directory path where certificate will be saved" $Subject = Read-Host "`nEnter the Subject for certificate " $password = Read-Host -assecurestring "`nPfx password" $ValidateDays = Read-Host "`nCertificate Validity days: (please enter number of days)" #Generate cert in local computer My store $name = new-object -com "X509Enrollment.CX500DistinguishedName.1" $name.Encode("CN=$Subject", 0) # The Key $key = new-object -com "X509Enrollment.CX509PrivateKey.1" $key.ProviderName = "Microsoft Software Key Storage Provider" # CNG is Software Key Storage "Microsoft RSA SChannel Cryptographic Provider" $key.KeySpec = 0 # was 1 but 0 looks needed for CNG http://msdn.microsoft.com/en-us/library/aa379409(VS.85).aspx $key.keyUsage =0xfffff # Set the key usage to all usage http://msdn.microsoft.com/en-us/library/windows/desktop/aa379410(v=vs.85).aspx $key.Length = 2048 $key.SecurityDescriptor = "D:PAI(A;;0xd01f01ff;;;SY)(A;;0xd01f01ff;;;BA)(A;;0x80120089;;;NS)" # Allow Write NT AUTHORITY\SYSTEM Allow Write BUILTIN\Administrators Allow Write NT AUTHORITY\NETWORK SERVICE $key.MachineContext = 1 $key.ExportPolicy = 1 # Allow private key to be exported XCN_NCRYPT_ALLOW_EXPORT_FLAG 0x00000001 http://msdn.microsoft.com/en-us/library/windows/desktop/aa379002(v=vs.85).aspx $key.Create() Write-Host "`nPrivate Key created ......" #The certificate itself $cert = new-object -com "X509Enrollment.CX509CertificateRequestCertificate.1" # Interface for self signed cert request http://msdn.microsoft.com/en-us/library/windows/desktop/aa377124(v=vs.85).aspx $cert.InitializeFromPrivateKey(2, $key, "") $cert.Subject = $name $cert.Issuer = $cert.Subject $today =get-date $cert.NotBefore = $today.AddDays(-1) # yesterday $cert.NotAfter = $cert.NotBefore.AddDays($ValidateDays) # Add Key usage to the certificate, this is needed as TMG chek this during certificate import $KeyUsage = new-object -com "X509Enrollment.CX509ExtensionKeyUsage.1" $Keyusage.InitializeEncode(0x4) #0x4 XCN_CERT_KEY_CERT_SIGN_KEY_USAGE http://msdn.microsoft.com/en-us/library/windows/desktop/aa379410(v=vs.85).aspx $cert.X509Extensions.Add($keyusage)
$BasicConstraints = new-object -com "X509Enrollment.CX509ExtensionBasicConstraints.1"
$BasicConstraints.InitializeEncode(1,0) #set isCA to true, max path of 0 means it can't create subordinate CAs, only endpoint certs
$cert.X509Extensions.Add($BasicConstraints)$cert.Encode() Write-Host "`nCertificate created ......" $enrollment = new-object -com "X509Enrollment.CX509Enrollment.1" $enrollment.InitializeFromRequest($cert) $certdata = $enrollment.CreateRequest(0) $enrollment.InstallResponse(2, $certdata, 0, "") Write-Host "`nCNG self signed installed in the Computer certificate local store" #Create the ".pfx" and ".cer" version by exporting the just inserted certificate $store = new-object System.Security.Cryptography.X509Certificates.X509Store "My","LocalMachine" $store.Open("ReadOnly") $certs = $store.Certificates $cerPath = $outputDir+ "\"+ $Subject+ ".cer" $pfxPath = $outputDir + "\" + $Subject + ".pfx" foreach ($cert in $certs) { # write-host $cert.Subject if($cert.Subject -like ("CN="+ $Subject)) { $ExportCert = $cert.Export(1) #http://msdn.microsoft.com/fr-fr/library/system.security.cryptography.x509certificates.x509certificate2.aspx 1=.cer 3=.fx [System.IO.File]::WriteAllBytes(($cerPath), $ExportCert) Write-Host "`nCertificate .cer exported to: " $cerPath $PFXStrData =$cert.Export(3,$password) [System.IO.File]::WriteAllBytes($pfxPath, $PFXStrData) Write-Host "`nCertificate with private Key .pfx exported to: " $pfxPath } } $store.close() #now Import it to Local computer Root store $RootCert = new-object System.Security.Cryptography.X509Certificates.X509Certificate2 $cerPath $RootStore = new-object System.Security.Cryptography.X509Certificates.X509Store "Root","LocalMachine" $RootStore.Open("ReadWrite") $RootStore.Add($RootCert) Write-Host "`nCertificate installed in Local computer - Trusted root" $RootStore.close() Write-Host "`nDone ... `n" -ForegroundColor Green
You will have to copy the above lines into a new file and then save it into a file name with .ps1 extension, such as CNG-HTTPSi.ps1 on a computer running Windows 2008 R2 SP1 or later.
Then follow these steps:
Open a PowerShell window as administrator
Ensure local scripts can be executed by running the command: Set-ExecutionPolicy Set-ExecutionPolicy RemoteSigned Find more information about execution policy in this article: http://technet.microsoft.com/en-us/library/ee176949.aspx
Execute the script with the command ./CNG-HTTPSi.ps1
Enter the path where the certificates will be saved
Enter the subject, such as ”CNG HTTPS Inspection”
Enter the pfx password
Enter the validity of the certificate in days, for example 730 for two years
In the specified folder you will find a .cer and a .pfx files
Deploy the cer file in the Trusted Root container to all clients and the TMG servers, you can deploy the certificate manually or using Active directory Find more information in this article: http://technet.microsoft.com/en-us/library/dd441069
Once the certificate is deployed you can reconfigure HTTP Inspection on TMG importing the pfx certificate generated by the script
Save and apply the configuration then you will be able to reach the site
Gianni Bragante Support Engineer – Microsoft Forefront Edge Security Team
Franck HeilmannSr. Escalation Engineer - Microsoft Forefront Edge Security Team
Philipp Sand Sr. Support Escalation Engineer – Microsoft Forefront Edge Security Team
You may face an issue with a certificate assigned to a listener that suddenly becomes invalid and therefore the incoming SSL connection are dropped.
Restarting the service you will show the following error:
Event Source: Microsoft Firewall Event ID: 14060 Description: Description: Cannot load an application filter Web Proxy Filter ({4CB7513E-220E-4C20-815A-B67BAA295FF4}). FilterInit failed with code 0x80092004. To attempt to activate this application filter again, stop and restart the Firewall service.
The problem can be caused by the permission on private keys of the certificate store becoming corrupted. This may be affecting one or more certificates. In these cases deleting the bad certificate and re-importing can help to resolve the the problem most of the times.
Find more information in this article: http://blogs.technet.com/b/isablog/archive/2009/03/10/unable-to-start-microsoft-firewall-service-in-isa-server-2006.aspx
In some cases you may have lost the original PFX file or forgot the password and need the fix the issue using a different approach. In this article we will discuss how to better diagnose the issue and try to fix it, this may or may not work in your environment depending on the entity of the damage.
You can identify the invalid certificates by opening the TMG console, even if the Firewall service is not running, and try to assign the right certificate to all of your listeners. By unselecting the checkbox “Show only valid certificates”, you will see a message similar to that in the screenshot below:
In the properties of the listener, when selecting a certificate, you may get the status “Private key handle error” or “Invalid key”
You can try fixing the issue from the Certificates console:
Execute MMC
Add the Certificate snap-in for the Local computer
In the Personal store right click on the certificate, All task, Manage Private keys
If you can assign Full control to the local Administrators group and to SYSTEM
Then go back to the TMG console and select the certificate, it should appear valid
Save and apply the configuration and try to start the Firewall service
However you may be unable to assign the permission from the certificates console, you may get an Access denied error. In this case you will have to identify the file with the certificate’s private key, the file is located in the folder c:\ProgramData\Microsoft\Crypto\RSA\MachineKeys
To troubleshoot the issue, you can use Process Monitor from SysInternals (DOWNLOAD: http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx).Once downloaded and extracted follow these steps:
Close all running programs, just keep the Certificates MMC open
Start Process Monitor, a capture will start automatically
On the Certificate MMC go to Manage Private keys as described above
Once you get the Access Denied error go back to Process Monitor and press CTRL-E to stop the capture
Click on Tools then Process Tree
Scroll down to mmc.exe, right click then Add process to Include filter, then click Close
The events will be filtered and only those generated by MMC will be displayed
Scroll down in the list and you will see some rows generated while trying to access to the private key file and ACESS DENIED in the result column, right click on any of them then Jump To …
The right folder and file will automatically be opened and you will be able to assign the permissions. You may also need to take ownership of the file in order to do that.
It is important to always re-select the certificate in the TMG console, this way the additional permissions required by the TMG Firewall service will be assigned automatically.
Depending on the entity of the damage you may need to follow the above steps for all the certificates affected by the issue.
If some of you recently tried to publish RDP protocol through TMG server, and suddenly lost the possibility to perform TS connections to the TMG server itself, you may find this post useful!
In TMG 2010, a System Policy rule exists allowing RDP traffic from a white-list of workstations to the TMG server itself.
Thanks to this, it’s generically possible to connect to our TMG server from allowed PCs.
If you perform the following command, you’ll be able to see the TCP listener bound to the Remote Desktop Services (RDS) service accepting incoming RDP connections on default port 3389:
Netstat –ano | findstr :3389
Now, let’s assume that you have to publish RDP protocol externally through TMG, so that TS connections from external devices toward an internal server can be established, while the internal server’s IP address is masked.
You can do this by creating a specific “Non-Web server publishing rule” in TMG:
If you run again the netstat command mentioned above after you performed this configuration, you will see no differences at all, and everything should be working as expected.
Let’s now assume you have to reboot the TMG server.
After the reboot, if you run the netstat, you’ll see that the situation has changed:
Now the TMG server is listening for RDP connection only on the IP address which has been configured within TMG’s Server Publishing rule. Moreover, we can see that the listener is now associated to the PID of the FW service itself, and we no longer have entries related to the RDS service.
If you test TS connectivity to the TMG server from one of the allowed internal workstations, it will now fail.
This happens because of a socket conflict generated between the FW service and the RDS service, and it is an expected behavior.
In order to allow remote access while publishing RDP, basically 2 solutions exist:
You can publish the RDP server on different port than 3389, using non web server publishing rule and creating a “customized” RDP protocol for the new considered port.
You can just change the RDS on the TMG to listen on a different port or adapter
To apply this second solution, you can open Remote Desktop Session Host Configuration > Right-Click on RDP-TCP > Network Adapter > then select the internal NIC.
After this action, just restart the RDS service.
If you now run a new netstat, you’ll see the following situation:
You will now have the RDS service listening on the IP addresses bound to the internal NIC, while the FW service is listening on the external side for RDP publishing.
With this configuration, it’s again possible to establish TS connections to the TMG server even when the server publishes the RDP protocol.
As always…I hope you can find this useful!!
Daniele Gaiulli Support Engineer – Microsoft Forefront Edge Security Team
recently we were working with customers, who did run into issues where the AD LDS database grew very large and they ended up with a full disc on System Drive.
The AD LDS database files can be found in %windir%\ADAM. The file which did grow quite big is the adamntds.dit.
The AD LDS on TMG is used to save and distribute the TMG configuration. Back in ISA days, you could only find it for Enterprise Edition Servers, in TMG every server has AD LDS installed, but only the Master Node or the EMS has a running and active AD LDS instance running. The AD LDS instance is called ISASTGCTRL.
When analyzing the customers environment, we found, that they were applying multiple configuration changes every hour, by running a custom script which modified the configuration in a way, that some elements were deleted and added.
By design AD LDS keeps track on all deleted records though. This lead to the scenario where the database file grew more and more, as all deleted records were not permanently deleted by only flagged as “to be deleted”.
When we did run the command
Ldifde –m –f exportfpc2.ldf –s localhost –t 2171 –d “CN=FPC2”
we did see a quite small export file of the configuration, which can be some MB depending on the configuration.
When you execute ldifde and add the –X parameter to also show deleted or tombstone objects like this:
Ldifde.exe –x -m -f exportfpc2.ldf -s localhost -t 2171 -d "CN=FPC2" you might see that the output file can grow very large indeed.
The reason for this behavior is, that the AD LDS is configured with a default Tombstone lifetime of 180 days. Therefore all objects remain in the database file until this tombstone lifetime had been exceeded.
Please be aware, that the Tombstone lifetime also controls the amount of days, which can be between a sync of the AD LDS for the case you have multiple AD LDS Server, e.g. in a EMS scenario.
Please visit these sites or some useful information about the tombstone lifetime:
Scenario Overview for Restoring Deleted Active Directory Objects - http://technet.microsoft.com/en-us/library/dd379542(v=WS.10).aspx
Appendix A: Additional Active Directory Recycle Bin Tasks - http://technet.microsoft.com/en-us/library/dcf3431a-c562-447f-a591-4742d2bcdbfa(v=ws.10)#BKMK_1
Depending on your backup strategy for AD LDS you might want to decrease this value if you face the issue, that the AD LDS is using all your Systemdrive disc space. As a rule of thumb the Tombstone Lifetime should not be smaller than the period of days between every full backup of AD LDS. E.g. Full Backup every 7 days -> Tombstone should be at least 8+ days. The minimum value you can define is 3 days.
You can modify the Tombstone lifetime like this:
On TMG where AD LDS is running, open ADSI Edit and connect to the local AD LDS -> localhost:2171 in Well-known-Context: “Configuration”
In Services->Windows NT -> Directory Services – right-click Directory Services and search for tombstoneLifetime. By default this is set to 180, depending on the customers backup strategy they can decrease this value. As a rule of thumb the Tombstone Lifetime should not be smaller than the period of days between every full backup of AD LDS. E.g. Full Backup every 7 days -> Tombstone should be 8+ days.
After the change, you need to restart the ISASTGCTRL to apply the change.
To manually trigger a garbage collection, you need to open ldp.exe on the host and connect to localhost:2171 again. Please make sure to connect using a local admin or AD LDS admin.
Next choose Browse -> Modify.
In Attribute add dogarbagecollection with value „1“. Next click Enter and afterwards run.
Depending on the size of the DB, the garbage collection might not be able to clean all records at once / it might take some days to completely clean up the records.
To monitor the garbage collection, you can increase the tracing using by changing the registry.
Important This section, method, or task contains steps that tell you how to modify the registry. However, serious problems might occur if you modify the registry incorrectly. Therefore, make sure that you follow these steps carefully. For added protection, back up the registry before you modify it. Then, you can restore the registry if a problem occurs. For more information about how to back up and restore the registry, click the following article number to view the article in the Microsoft Knowledge Base:
How to back up and restore the registry in Windows 322756 (http://support.microsoft.com/kb/322756/)
The RegKey you need to look for is HKLM\SYSTEM\CCS\services\ADAM_ISASTGCTRL\Diagnostics you can change the value of the key „6 Garbage collection” to 5 for purposes of monitoring/troubleshooting.
Afterwards you can monitor the events in Application and Services -> ADAM (ISASTGCTRL).
Please be aware the garbage collection is usually triggered once a day, therefore you do not need to trigger the garbage collection manually.
Afterwards you can check the size of the database files on your disc again, and you will realize, that the size did not change… at least it should not grow any more now
In order to free up some disc space, you will need to perform an offline defragmentation of the AD LDS as described here: http://support.microsoft.com/kb/232122 or http://support.microsoft.com/kb/930832
If you decide to use esentutl, the command looks like this: esentutl.exe /d adamntds.dit
Hope that helps!
Philipp Sand
Microsoft CSS Forefront Security Edge Team
We are happy to announce the availability of Rollup 4 for Forefront Threat Management Gateway (TMG) 2010 Service Pack 2 (SP2). TMG SP2 Rollup 4 is available for download here: Rollup 4 for Forefront Threat Management Gateway (TMG) 2010 Service Pack 2
Please see KB Article ID: 2870877 for details of the fixes included in this rollup. The Build Number for this update is: 7.0.9193.601
For more information about Forefront Threat Management Gateway 2010 SP2, please see the followingMicrosoft website:
In these days we received a considerable number of support requests asking for more info about SSL/TLS Renegotiation and the risk it introduces of being exposed to DoS attacks and malicious code injections.
The requests in object were focused on ISA/TMG products, considering they are used as reverse proxy for web publishing purposes, but the below considerations can be considered valid for every kind of Windows server/client supporting SSL/TLS connections.
First, what is exactly SSL/TLS Renegotiation?
"TLS [as defined in RFC 5246] allows either the client or the server to initiate a renegotiation -- a new handshake that establishes new cryptographic parameters. Unfortunately, although the new handshake is carried out using the cryptographic parameters established by the original handshake, there is no cryptographic binding between the two. This creates the opportunity for an attack in which the attacker who can intercept a client's transport layer connection can inject traffic of his own as a prefix to the client's interaction with the server"
The above definition is taken from RFC 5746.
The following is a graphic representation of a basic SSL/TLS Handshake:
Under certain circumstances, the client could be asking the server a renegotiation of the SSL/TLS parameters using the same TCP socket:
If the SSL/TLS is not secure (as per RFC 5746 recommendations) a MITM could use the renegotiation to send the server malicious data, pretending to be "good" user.
What chances do we have to mitigate this issue?
You should make sure, that the following security hotfix is installed:
http://support.microsoft.com/kb/980436
This fix is making the system compliant with RFC 5746, mitigating the risk of malicious data injection.
To provide backward compatibility, this security update works in the following modes: STRICT and COMPATIBLE
Compatible mode
o If this security update is applied to the server, and the server is in compatible mode, the server allows all clients to set up and renegotiate Transport Layer Security (TLS) sessions. This occurs whether the clients are updated or are not updated by using this security update.
o Similarly, if this security update is applied to the client, and the client is in compatible mode, the client can set up and renegotiate TLS sessions with all the servers for which this security update is applied or is not applied.
Strict mode
o If this security update is applied to the server, and the server is in strict mode, the server allows only those clients to which this security update is applied to set up and renegotiate TLS sessions. The server does not allow the clients to which this security update is not applied to set up the TLS session. In this case, the server terminates such requests from the clients.
Similarly, if this security update is applied to the client, and the client is in strict mode, the client can set up and renegotiate TLS sessions with all the servers for which this security update is applied. The clients cannot set up TLS sessions at all with servers for which this security update is not applied. The client cannot move ahead with a TLS negotiation attempt with such servers.
By default, this security update enables the TLS or Secure Sockets Layer (SSL) client or server to stay in compatible mode. An administrator can use the AllowInsecureRenegoClients and the AllowInsecureRenegoServers entry DWORD values in the following registry path to enable strict mode on the client or on the server:
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL
The following table shows how these DWORD values can be used:
DWORD
Value = zero
Value = nonzero
AllowInsecureRenegoClients
Strict Server
Compatible Server
AllowInsecureRenegoServers
Strict Client
Compatible Client
Malicious data injection is not the only problem related to Renegotiation: it can be in fact used to perform DoS attacks against a server.
When a new SSL/TLS connection is being negotiated, the server will typically spend significantly more CPU resources than the client. A malicious user, leveraging the Renegotiation, could be able to enhance the server's CPU usage causing DoS.
In order to have a better mitigation for both malicious data injection and DoS attacks, the best option would be to reject the client-initiated SSL/TLS renegotiation at all.
The following Microsoft Security Advisory explains how:
http://support.microsoft.com/kb/977377/en-us
As reported in the article, the behavior can be modified by changing the value of the following registry key:
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\DisableRenegoOnServer
Notes
· If the DisableRenegoOnClient subkey is present and has any nonzero value:
o The client will not initiate renegotiation. o The client will not respond to renegotiation.
o The client will not initiate renegotiation.
o The client will not respond to renegotiation.
· If the DisableRenegoOnClient subkey is missing or is present and has a zero value:
o The client will initiate renegotiation. o The client will respond to renegotiation.
o The client will initiate renegotiation.
o The client will respond to renegotiation.
· If the DisableRenegoOnServer subkey is present and has any nonzero value:
o Server initiated renegotiation is not allowed. o The server will not respond to renegotiation requests from the client.
o Server initiated renegotiation is not allowed.
o The server will not respond to renegotiation requests from the client.
· If the DisableRenegoOnServer subkey is missing or is present and has a zero value:
o Server initiated renegotiation is allowed. o The server will respond to renegotiation requests from the client.
o Server initiated renegotiation is allowed.
o The server will respond to renegotiation requests from the client.
Of course, this may have an impact on the use of specific applications requiring SSL/TLS renegotiation feature.
The KB article underlines the following:
o After you install this security update, you cannot use the legacy provisioning service parameter (–UseLegacyProvisioningService) when you create a federation trust with the Microsoft Federation Gateway. The security update will prevent the federation trust from working correctly. This problem will occur if you install this security update on a computer that is running Exchange Server 2010 or Exchange Server 2010 Service Pack 1 before you have created a federation trust. To avoid this problem, you must create the federation trust before you install this security update. For more information about how to create a federation trust by using the –UseLegacyProvisioningService parameter, visit the following Microsoft webpage: Create a Federation Trust
o When you install this update on a computer that has the Microsoft Online Single Sign-In services client installed, you may experience the following issues:
· The Sign In client cannot obtain user configuration information. This only affects new users who are running the Sign In client for the first time. The Sign In client cannot obtain information to configure Outlook. If the Sign In client has already run and configured the applications, there are no additional issues with the Sign In client. · Outlook users cannot see free/busy information. Therefore, this update also affects existing Outlook users. To resolve this problem, set the DisableRenegoOnClient registry entry to a value of 0 (zero), and then restart the computer.
· The Sign In client cannot obtain user configuration information. This only affects new users who are running the Sign In client for the first time. The Sign In client cannot obtain information to configure Outlook. If the Sign In client has already run and configured the applications, there are no additional issues with the Sign In client.
· Outlook users cannot see free/busy information. Therefore, this update also affects existing Outlook users. To resolve this problem, set the DisableRenegoOnClient registry entry to a value of 0 (zero), and then restart the computer.
o This update disables TLS/SSL renegotiation, common protocol functionality that is required for specific applications. This may cause this software to no longer function as expected. If any side effects are experienced, customers should uninstall the workaround to resolve the issue. The following software has been tested by Microsoft and that has been found to experience problems when you install this update:
· Windows 7 DirectAccess: The IP HTTPS interface will not function. · Exchange ActiveSync: Does not function when it uses certificate client authentication. · Internet Information Services (IIS): In certain configurations, IIS using certificate client authentication, including certificate mapping scenarios, will be affected. Site-wide client certificate authentication will not be affected and will continue to function. · Internet Explorer: When you browse Web sites that require client certificate authentication, but not site-wide client certificate authentication, you may not successfully be able to connect.
· Windows 7 DirectAccess: The IP HTTPS interface will not function.
· Exchange ActiveSync: Does not function when it uses certificate client authentication.
· Internet Information Services (IIS): In certain configurations, IIS using certificate client authentication, including certificate mapping scenarios, will be affected. Site-wide client certificate authentication will not be affected and will continue to function.
· Internet Explorer: When you browse Web sites that require client certificate authentication, but not site-wide client certificate authentication, you may not successfully be able to connect.
Of course, it's not possible to predict the implications of disabling client-initiated renegotiation for various applications: this solution should be appropriately tested in each specific environment.
From a security point of view, though, this is the recommended way to mitigate all the above described problems.
Hope this can help!
Daniele Gaiulli
Support Engineer – Microsoft Forefront Edge Security Team
Sr. Support Escalation Engineer – Microsoft Forefront Edge Security Team
If the Firewall service crashes a number of times within a short time period it does not automatically restart after the 4th crash. If you review the Service Control Manager settings for the Firewall service appears to be configured to restart after all failures.
After each of the first three failures, you will see this error in the event log:
Log Name: System Source: Service Control Manager Date: 3/4/2013 1:36:24 PM Event ID: 7031 Level: Error Computer: TMG.domain.local Description: The Microsoft Forefront TMG Firewall service terminated unexpectedly. It has done this 3 time(s). The following corrective action will be taken in 60000 milliseconds: Restart the service.
This is inline with the expected behavior.
However, after the fourth failure the service will no longer restart and you will see this error in the event log:
Log Name: System Source: Service Control Manager Date: 3/4/2013 1:45:34 PM Event ID: 7034 Level: Error Computer: TMG.domain.local Description: The Microsoft Forefront TMG Firewall service terminated unexpectedly. It has done this 4 time(s).
The behavior may appear inconsistent and unexpected but it is actually by design.
During the TMG installation, the service is configured to only automatically restart after the first 3 crashes in a 24 hour period in order to raise the attention of the system administrator that something is going wrong with this service that needs investigating. This can be considered similar to IIS Rapid Fail Protection to avoid a situation where we are restarting and then crashing straight way
By checking the service configuration in the registry key, HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\fwsrv we can see the following:
The number of configured recovery actions is actually four, the first three being "Restart the service" and the fourth being "Do nothing", this results in the behavior described above.
The Windows Service Control Manager UI is limited to displaying only the first 3 actions and therefore gives the wrong impression of the configured actions.
If you have good reasons to configure the service to restart for subsequent failures you can do so by running the following command at an elevated command prompt:
Sc.exe failure fwsrv reset= 86400 actions= restart/60000/restart/60000/restart/60000
This configures 3 restart actions to restart the service after 60 seconds. The last action will be used to determine the behavior of subsequent crashes.
To revert to the default TMG behavior please run the following command from an elevated command prompt:-
Sc.exe failure fwsrv reset= 86400 actions= restart/60000/restart/60000/restart/60000//
This will re-configure Service Control Manager to restart the Firewall service for the first 3 crashes but to then take no action for the 4th and subsequent crashes.
Gianni Bragante
Support Engineer - Microsoft Forefront Edge Security Team
Ian Parramore
Sr. Escalation Engineer - Microsoft Forefront Edge Security Team
Here’s some info on an interesting support issue I worked the other day. If you happen to run into this one day, maybe this will help you get it resolved.
Issue:
We have a website published through ISA 2006. The site is configured for both HTTP and HTTPS access from the ISA server. When a user connects to the site over HTTP, the site comes up fine.
But when he tries over HTTPS, he gets a ‘page cannot be displayed’.
Troubleshooting and Resolution:
We started with live logging on the ISA console while doing a repro of the issue. We were seeing ‘Failed Connection Attempts’ for the traffic coming from the test machine used for the repro, with the error message: Error 64 “The specified network name is no longer available”
This error is very generic and there can be multiple reasons which would translate to this error code.The most common one is when the backend server is performing a dirty TCP connection reset.
So, to check this further, we collected a network monitor trace on the internal NIC of ISA server.
We filtered down to the traffic that is of interest to us.
So this clearly indicates that the backend server is Resetting the TCP connection prematurely and this is triggering the ‘64 Error’.
Investigating further, we identified that the backend device is a 3rd party load balancer. And for some unknown reasons, the ISA server was failing at the SSL handshake stage.
So, we had the 3rd party support team collect a dump of the SSL settings on the Load Balancer and identified the following:
Then, we went back to the Network Monitor trace (the earlier screenshot) and compared this with the ciphers advertised by ISA server in the client hello. RSA_WITH_RC4_128_MD5 is not part of the Cipher list sent by the ISA server.
Due to this, the 2 peers are not able to successfully choose a common encryption scheme and the SSL handshake fails.
After identifying this, we had the 3rd party vendor enable additional Ciphers which are accepted by ISA server.
Once we did this, the published site was accessible from the internet.
The issue was resolved!!
Hope this would be helpful when you are troubleshooting website accessibility issues through ISA server…especially with 3rd party load balancers in the infrastructure.
Karthik Divakaran
Security Support Engineer - Microsoft Forefront Edge Team
Reviewers:
Suraj Singh
Security Support Escalation Engineer - Microsoft Forefront Edge Team
Richard Barker
Security Sr. Support Escalation Engineer – Microsoft Forefront Edge Team
Just wanted to let you know about a new KB article we published today. This one is a TMG article that talks about an issue where a recurring monthly report job is not created when you expect it.
You can find the complete article here:
KB2830886 - A recurring monthly report job does not run when expected on an array in Forefront Threat Management Gateway 2010 (http://support.microsoft.com/kb/2830886)
J.C. Hornbeck | Knowledge Engineer | Microsoft GBS Management and Security Division
Get the latest System Center news on Facebook and Twitter:
System Center All Up: http://blogs.technet.com/b/systemcenter/ System Center – Configuration Manager Support Team blog: http://blogs.technet.com/configurationmgr/ System Center – Data Protection Manager Team blog: http://blogs.technet.com/dpm/ System Center – Orchestrator Support Team blog: http://blogs.technet.com/b/orchestrator/ System Center – Operations Manager Team blog: http://blogs.technet.com/momteam/ System Center – Service Manager Team blog: http://blogs.technet.com/b/servicemanager System Center – Virtual Machine Manager Team blog: http://blogs.technet.com/scvmm
Windows Intune: http://blogs.technet.com/b/windowsintune/ WSUS Support Team blog: http://blogs.technet.com/sus/ The AD RMS blog: http://blogs.technet.com/b/rmssupp/
The Forefront Endpoint Protection blog : http://blogs.technet.com/b/clientsecurity/ The Forefront Identity Manager blog : http://blogs.msdn.com/b/ms-identity-support/ The Forefront TMG blog: http://blogs.technet.com/b/isablog/ The Forefront UAG blog: http://blogs.technet.com/b/edgeaccessblog/
We are happy to announce that Rollup 1 for Forefront UAG 2010 Service Pack 3 has been released.
UAG 2010 Service Pack 3 Rollup 1 is available as a hotfix download from Microsoft Support as an update to UAG 2010 Service Pack 3.
This important update contains 8 new fixes for reported issues as well as enhanced context tracing to more easily filter trace data per session.
For details, please visit KB 2827350: Description of Rollup 1 for Forefront Unified Access Gateway 2010 Service Pack 3
Please download the Forefront Unified Access Gateway (UAG) 2010 Service Pack 3 package now, and learn more about UAG 2010 SP3 by visiting our TechNet Library.
This post is about an issue I worked on several days ago.
Symptom:
========
My customer had a TMG array with two nodes running with NLB. The problem they faced was that from time to time some TMG node couldn't process traffic anymore: requests to the virtual IP (VIP) failed and only rebooting the TMG machine eliminated the issue.
IE was configured to use the NLB virtual IP (VIP) as proxy address. When the issue happened users couldn’t browse internet pages - the browser didn't show any error page, it just stayed with a blank page.
Troubleshooting:
=============
First of all, I looked at the TMG Web proxy log and found an error logged there with the result code of 11001 (WSAHOST_NOT_FOUND) for the request for “contoso.com”, which we tried to browse in order to reproduce the issue.
Then I took a look at a captured network monitor trace in order to find an appropriate network conversation between the client and TMG:
Time Delta time Source Destination Protocol Length Info
…
11:07:26.566261000 1.376436000 CLIENT_IP TMG_VIP HTTP 433 GET http://contoso.com/ HTTP/1.1
11:07:26.570933000 0.004672000 TMG_VIP CLIENT_IP HTTP 1440 HTTP/1.1 502 Proxy Error ( The host was not found. ) (text/html)
In a web proxy scenario, the web proxy is supposed to resolve the name, however, by some reason the node didn't manage to resolve the remote host.
As the next step, I filtered out the trace for DNS traffic and didn't find DNS packets on the node at all – this looked very suspicious.
Therefore I looked at Netstat and its output showed around ~62000 lines similar to the followings, whereas 2580 was the pid of wspsrv.exe:
UDP 0.0.0.0:4002 *:* 2580
UDP 0.0.0.0:4003 *:* 2580
UDP 0.0.0.0:4004 *:* 2580
UDP 0.0.0.0:4005 *:* 2580
UDP 0.0.0.0:4006 *:* 2580
UDP 0.0.0.0:4007 *:* 2580
UDP 0.0.0.0:4020 *:* 2580
UDP 0.0.0.0:4021 *:* 2580
UDP 0.0.0.0:4022 *:* 2580
UDP 0.0.0.0:4023 *:* 2580
UDP 0.0.0.0:4024 *:* 2580
UDP 0.0.0.0:4025 *:* 2580
UDP 0.0.0.0:4026 *:* 2580
So TMG seemed to use up all UDP ports – hence there was no free UDP port left to use as source port for dns queries. This explained why name resolution failed and why TMG returned 502.
Why was such a great amount of ports used by TMG?
My guess was that it might be doing it on behalf of a client. Therefore, I looked at the Firewall log which showed that there were huge amount of UDP requests from a single client to different external hosts.
Luckily TMG firewall client was installed on the client machine which gave us an application name:
7:27:51 Establish CLINET2_IP 2058 EXTERNAL_HOST_1 54150 0x0 Access Internet UDP domain\user smax4pnp.exe:3:5.1 TMG1 Unidentified IP Traffic
7:27:52 Establish CLINET2_IP 2033 EXTERNAL_HOST_2 9362 0x0 Access Internet UDP domain\user smax4pnp.exe:3:5.1 TMG1 Unidentified IP Traffic
7:27:53 Terminate CLINET2_IP 2026 EXTERNAL_HOST_3 59866 0x80074e20 Access Internet UDP domain\user smax4pnp.exe:3:5.1 TMG1 Unidentified IP Traffic
So TMG used up all the available ports due to the excessive amount of request from this client/application.
In order to resolve the issue the customer disabled the client that has been causing this huge UDP traffic.
Vasily Kobylin, Senior Support Engineer, EMEA Forefront Edge Team
Balint Toth, Senior Support Escalation Engineer, EMEA Forefront Edge Team
One of the features introduced with TMG Service Pack 2 is to run the Firewall Service with a Domain account, this allow users to authenticate with Kerberos when using NLB. Find more information about this feature here: http://technet.microsoft.com/en-us/library/hh454304.aspx
However you should pay attention when specifying the account name to avoid problems with logging to SQL Server, either local or remote.
The account specified is used by TMG to configure the service and also to create the Login in SQL Server. For the TMG Firewall service to start any format is fine, but for SQL Server only the format domainName\loginName is valid.
For example if you want to use the account TMGSvc in the domain CONTOSO you have to enter CONTOSO\TMGSvc.
Using the UPN (User Principal Name) format or the FQDN (Fully Qualified Domain Name) does not work. For example you cannot use TMGSvc@Contoso.com or Contoso.com\TMGSvc
The SQL Server documentation for the CREATE LOGIN command has the following note:
"When you are creating logins that are mapped from a Windows domain account, you must use the pre-Windows 2000 user logon name in the format [<domainName>\<loginName>]."
If you try using an invalid format you will see the Log Status as Disconnected and your LLQ folder growing:
Author: Gianni Bragante Support Engineer - Microsoft Forefront Edge Security Team
Reviewer: Lars Bentzen Sr. Escalation Engineer - Microsoft Forefront Edge Security Team
Recently I have been seeing an increasing number of cases with the same symptom especially in the military and the government sector and even in contractors for the government. In these highly secure environments clients largely rely on the use of a “smart” card known as Common Access Cards (CAC) for authentication to their various types of servers and services.
Symptom
Your Internet Security and Acceleration Server (ISA) or Forefront Threat Management Gateway 2010 (TMG) Server is publishing resources internally/externally and your Web Listener is configure to use SSL Client Certificate Authentication. When clients navigate to the site that is published they would normally be prompted to choose their client certificate. Some or all of the clients are not being prompted to choose the certificate. On the ISA/TMG server, you may see a Warning in your Event Log with an Event ID of 36885.
Event Type: Warning Event Source: Schannel Event Category: None Event ID: 36885 Date: date Time: time User: Computer: COMPUTERNAME Description: When asking for client authentication, this server sends a list of trusted certificate authorities to the client. The client uses this list to choose a client certificate that is trusted by the server. Currently, this server trusts so many certificate authorities that the list has grown too long. This list has thus been truncated. The administrator of this machine should review the certificate authorities trusted for client authentication and remove those that do not really need to be trusted.
Cause
This issue is caused when there are too many trusted certificate authorities in the Certificate Store on ISA/TMG. This is particularly common for servers that need a long list of Department of Defense (DoD) Certificate Authorities. When the list grows beyond 12,228 bytes (the maximum size the current Schannel security package supports) the list will be truncated. If the client doesn't receive the root CA that it needs because it has been truncated, it will not prompt to choose the certificate.
Resolution
There are a few workarounds for this but the one that is easiest to implement and seems to fit the needs of most organizations is below.
On the server or servers that are running ISA/TMG you will need to set the following registry entry to 0 (false):
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL
Value name: SendTrustedIssuerList Value type: REG_DWORD Value data: 0
By default the value is 1 (true).
For other possible workarounds please see this KB:
http://support.microsoft.com/kb/933430
Conclusion
Troubleshooting SSL Client Certificate issues can be tricky and time consuming. This issue was certainly difficult to identify the first time I saw it. Hopefully the information I have given you here can save you time, money, and aggravation.
Keith Abluton:
Security Support Escalation Engineer - MSD Security Team
Sr. Security Support Escalation Engineer - MSD Security Team
Hi everybody!
In this article we will see how to troubleshoot an issue with accessing an FTP server behind TMG 2010.
Imagine we have the following situation: a client PC on an internal corporate network want to access a remote FTP server through TMG 2010 using an FTP client such as, for example, FileZilla.
The way the FTP is configured (authentication, encryption, ecc…) is out of interest for this case.
On the TMG server, we’ve created an access rule allowing “Read-Only” outbound requests for the FTP protocol:
When we try to connect to our remote FTP server using, for example, FileZilla, we may face the following error:
FTP connection issues through ISA/TMG could be related to many different aspects.
In the following article it’s possible to find a resolution for many of the most common problems:
http://technet.microsoft.com/en-us/library/bb794745.aspx
The problem we’re focusing on in this article, however, is not included in the above troubleshooting guide and depends on a specific by-design behavior of TMG server.
Basically, in our case we see that the connection attempt is failing due to a “550-Access Denied” error after having performed a MLSD command.
What is MLSD exactly ?
Here we can find a description of what MLSD is used for:
http://tools.ietf.org/html/draft-ietf-ftpext-mlst-16#section-7
As we can see from the above:
The MLST and MLSD commands are intended to standardize the file and directory information returned by the Server-FTP process. These commands differ from the LIST command in that the format of the replies is strictly defined although extensible.
In the default configuration of the TMG FTP Access filter in “Read-Only Mode”, the filter will only allow a specific subset of FTP commands. The MLSD command is not included in this set of “Read-Only” commands. FTP clients using LIST command will not experience this problem, since LIST is an allowed command.
Its easy to resolve the problem by allowing write-permissions in the FTP-Filter advanced properties of our access rule:
Now, granting write rights is not always a good choice, and most of the times this is not allowed nor suggested.
Nevertheless, a workaround exists for this situation: in fact, it’s possible to add the MLDS command in the “allowed-commands list” of the “Read-only” TMG FTP filter.
The following MSDN article explains how to configure add-ins:
http://msdn.microsoft.com/en-us/library/dd435753.aspx
Specifically:
FTP Access Filter
FTP Access Filter is an application filter that is installed with Forefront TMG. It enables FTP protocols. When running in read-only mode, FTP Access Filter blocks all commands in the control channel except the following commands: ABOR, ACCT, CDUP, CWD /0, FEAT, HELP, LANG, LIST, MODE, NLST, NOOP, PASS, PASV, PORT, PWD /0, QUIT, REIN, REST, RETR, SITE, STRU, SYST, TYPE, USER, XDUP, XCWD, XPWD, SMNT. This should block any writing to the server side. The default list of allowed commands can be replaced by a customized list that is written to the collection of vendor parameters sets (FPCVendorParametersSets) associated with the filter. The Firewall service must restarted for the new settings to take effect.
The above article provides a script example through which it is possible to customize FTP filter list. This way, it will be possible to keep the filter configured in Read-Only mode, and also allow the FileZilla connection to work as expected.
Hope this can be useful!
Let's see you back with the next topic!!
Author: Daniele Gaiulli
Support Engineer – EMEA Forefront Edge
Reviewer: Philipp Sand
Support Escalation Engineer – EMEA Forefront Edge
I’ll try to elaborate on the issue using as many illustrations and snapshots as possible. When I came across this issue, it was quite surprising.
32-bit Remote Management Client
In the TMG environment, we are using a single EMS (Enterprise Management Server) with a single Array. There are two TMG nodes joined to this array. To manage the environment we are using a Windows 7 32-bit machine with a 32-bit client. Please use this link download the 32-bit client (TMG_ENU_Management_x86.exe). Note that you will need to login with a Microsoft Live Id and register in order to download.
Once downloaded, install the client and connect to the EMS server using its FQDN. Make sure EMS is configured to allow remote management, refer to the below mentioned articles.
· About Forefront TMG roles and permissions - http://technet.microsoft.com/en-us/library/dd897006.aspx#BKMK_RolesAndPermissions
· Configuring roles and permissions - http://technet.microsoft.com/en-us/library/dd441007.aspx
The relevant Users on the list should be able to gain access to the TMG EMS server for administration.
After assigning the correct set of permissions and remote access to TMG EMS server, you can remotely access the Enterprise Policy and make allowed changes.
But while accessing Array nothing displays, it doesn’t even shows the arrays created in the enterprise. Refer to below mentioned Snips.
Here in this snip we can see that Enterprise policy is displayed.
Here you can see the focus is on “Arrays”, but no policies are displayed.
Let’s check and compare the version on TMG EMS server and then on this client.
This is the version number from TMG EMS server which is updated to latest i.e. SP2 RU2
This is the version number from client which is updated to SP1 UP1.
The major cause of this is due to version mismatch between the management console and the TMG enterprise. For example the TMG enterprise is at SP2 RollUP2 update level which is build number 7.0.9193.540 and the TMG management console on remote 32bit machine is at RTM which is build number 7.0.7734.100 (refer to the article).
Solution
This can be resolved by updating the TMG RTM management console on 32bit Remote machine. Refer to the links mentioned below to download and install the relevant updates.
· SP1
· SP1 UP1
· SP2
This completely depends on the version level of the TMG environment. Check the article mentioned below for all the relevant TMG versions.
http://blogs.technet.com/b/keithab/archive/2011/09/27/forefront-tmg-2010-service-pack-rollup-and-version-number-reference.aspx
Match the version with the updates to the level TMG EMS server is on. There are no rollups released for the MMC.
After updating the client to SP2 we were able to access the array policies on the client machine.
64-bit Remote Management Client
Now goes the story for 64-bit setup. Just to mention there is no separate TMG mmc console installation msp available. For this installation, the TMG 2010 ISO/DVD is used.
NOTE: The following 4 steps outline the default MMC install using the install media.
You may have followed these steps, believing that you would be able to manage TMG EMS remotely using the MMC.
1. TMG setup from installation ISO/DVD by starting Preparation Tools first.
2. On Welcome screen accepted the terms for installation.
3. On Installation type dialog box selected Forefront TMG Management.
4. Wizard finishes its work.
After a default installation of the MMC (from the install media), you may be surprised to find out it doesn’t work . Because TMG is at SP2 or above update level and the MMC installation is at RTM level.
There are updates available which can be used to bring the MMC to the same update level as the TMG EMS server is at. But the procedure used for 32bit installation doesn’t work for 64bit.
I know there are a lot of questions surfacing, but I have the answer.
Because the 64bit mmc is installed straight from the install media, we’ll have to update the installation itself to SP2/ relevant to your environment.
To do this, we’ll need to create a “TMG 2010 Slipstream” installation, in which we update the TMG installation MSI itself.
1. If the TMG 2010 MMC console was previously installed directly from the install media, you’ll need to uninstall it from Control Panel >> Programs and Feature.
2. Copy all the contents from TMG 2010 ISO/DVD to a folder on HDD. In this example, we will copy the contents to C:\TMG.
3. Download the following TMG 2010 updates; making sure you download all 64bit versions. Use the following links:
a. SP1 - http://www.microsoft.com/en-us/download/details.aspx?id=16734
b. UP1 - http://www.microsoft.com/en-us/download/details.aspx?id=11445
c. SP2 - http://www.microsoft.com/en-us/download/details.aspx?id=27603
4. Once you have downloaded all three files, copy the files to C:\TMG\FPC
5. The UP1 and SP2 are in .exe format, therefore we will need to extract the msp files so they can be used for slipstreaming TMG 2010.
6. Open a command prompt with elevated privileges and, in the C:\TMG\FPC folder, execute the following commands.
a. SP1 Update1 - TMG-KB2288910-amd64-ENU.exe /t TMGSP1U1
b. You’ll get a dialog after completion.
Click ok to close.
c. SP2 - TMG-KB2555840-amd64-ENU.exe /t TMGSP2
d. You’ll get a dialog after completion.
7. Below is the snip for commands and folders I used.
8. Under C:\TMG\FPC, there should be two new folders called TMGSP1UP1 and TMGSP2. Both of these folders will contain the extracted msp file.
9. Copy the msp files to FPC folder.
10. Now let’s create slipstream for TMG 2010. Follow the commands and make sure you update it to the same level as TMG EMS.
a. SP1 - msiexec /a ms_fpc_server.msi /p tmg-kb981324-amd64-enu.msp
This will initiate installation wizard, which will slipstream the TMG2010 installation with SP1
b. SP1UP1 - msiexec /a MS_FPC_Server.msi /p TMG-KB2288910-amd64-ENU.msp
This will initiate installation wizard, which will slipstream the TMG2010 installation with SP1UP1.
c. SP2 - msiexec /a MS_FPC_Server.msi /p TMG-KB2555840-amd64-ENU.msp
This will initiate installation wizard, which will slipstream the TMG2010 installation with SP2.
11. Next, you can delete the following highlighted files and folders from C:\TMG\FPC.
12. Once deleted, the C:\TMG\FPC folder appear as follows:
13. Now create an ISO/DVD of the entire C:\TMG folder. Make sure you do not create and ISO/DVD out of only FPC folder.
14. Now this ISO/DVD can be used to install TMG mmc console on a 64-bit client machine using the steps mentioned below.
a. Start TMG setup from installation ISO/DVD by starting Preparation Tools first.
b. On Welcome screen click next and accept the terms for installation.
c. On Installation type dialog box select Forefront TMG Management only.
d. Let wizard to finish its work and then click Finish. This will start mmc installation wizard.
e. Once installation finishes you can access the array policies as well, provided that appropriate permissions are assigned.
Thanks for reading through, I hope I was able to clear your doubts and provide a solution. If you are still facing the issue then I would recommend opening a case with Microsoft CSS.
Vivek Kumar Sharma
Support Engineer – MSD Security Division
Junaid Jan
Security Support Escalation Engineer – MSD Security Division
We are happy to announce the availability of Rollup 3 for Forefront Threat Management Gateway (TMG) 2010 Service Pack 2 (SP2). TMG SP2 Rollup 3 is available for download here: Rollup 3 for Forefront Threat Management Gateway (TMG) 2010 Service Pack 2
Please see KB Article ID: 2735208 for details of the fixes included in this rollup. The Build Number for this update is: 7.0.9193.575
For more information about Forefront Threat Management Gateway 2010 SP2, please see the following Microsoft website:
This post is, once again, about an issue I worked on few days back. Before I start discussing the issue, and how I resolved it, I would like to outline the objective of this post.
The objective of this post is to make TMG administrators aware of issues like this; and what can be done to resolve them. Discovering the root cause of this issue required a User Mode Dump analysis.
Performing a User Mode Dump analysis requires “Symbol” files (which are private). My goal is not to provide specific instruction on User Mode Dump analysis, but instead to show what kind of information can be gathered, and how it can be used, to help troubleshoot boot-time “service issues” on a TMG server.
For those that are not familiar with dump analysis terms like process, threads and its stack, I will elaborate further as I explain the steps.
TMG server admin was rebooting the server and at the time of reboot TMG services were hanging and were not starting. A similar issue was reported pre TMG sp2 but it was fixed post sp2. In this scenario TMG was updated to latest build i.e. TMG sp2 RU2.
Some background: It should be noted that quite a bit of troubleshooting had taken place prior to my involvement in the case. This includes the steps in the following Knowledge Base article:
Forefront Threat Management Gateway 2010 services do not start as expected when the FTMG 2010 servers are in a workgroup array
During startup, the following System Event was logged…
_____________________________________________________________________________________________
Log Name: System Source: Service Control Manager Date: 09/11/2012 17:42:30 Event ID: 7022 Task Category: None Level: Error Keywords: Classic User: N/A Computer: server1 Description: The Microsoft Forefront TMG Firewall service hung on starting.
Event Xml: <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" EventSourceName="Service Control Manager" /> <EventID Qualifiers="49152">7022</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x8080000000000000</Keywords> <TimeCreated SystemTime="2012-11-09T17:42:30.378163900Z" /> <EventRecordID>344470</EventRecordID> <Correlation /> <Execution ProcessID="716" ThreadID="720" /> <Channel>System</Channel> <Computer>server1</Computer> <Security /> </System> <EventData> <Data Name="param1">Microsoft Forefront TMG Firewall</Data> </EventData> </Event>
Data collection:
During the course of troubleshooting we collected a User Mode Dump while reproducing the issue.
User mode dumps collection reference: http://msdn.microsoft.com/en-us/library/ff420662.aspx
Data analysis:
Note: The approach taken in this post is very similar to guidelines given in the following link about debugging a deadlock as we were in a scenario similar to a deadlock: http://msdn.microsoft.com/en-us/library/windows/hardware/ff540592(v=vs.85).aspx
In the dump, I found following critical section was locked :
Note: For more information about critical section and locked critical section, please refer to: http://msdn.microsoft.com/en-us/library/windows/hardware/ff541979(v=vs.85).aspx
Then I located the owning thread of this locked critical section. In following snapshot we can see the stack of this thread. The stack is read from bottom to top. From this call stack it appears that wspsrv (firewall service) is trying to load a filter called XSISAPI. It appears TMG has deferred its filters’ startup until this filter (i.e. XSISAPI)is loaded.
I then checked the module for this filter (i.e. XSISAPI) and found that it’s a filter called “Afaria” from Sybase.
Solution:
We configured the XSISAPI filter service to delayed start. After this change, the TMG services started normally after reboot.
Suraj Singh:
Hello Everyone! We’ve seen a few cases lately dealing with TMG servers sourcing outgoing packets with secondary IP addresses that have been added to the NICs. This could cause issues in communications between nodes or possibly other issues. One such example that I have seen come across is where a customer had a TMG server being utilized as an internal firewall behind a 3rd party Edge firewall. Clients were utilizing the TMG server as their proxy server. When the http requests left the external interface of the TMG server the packets were sourced with a secondary IP address of the External NIC on the TMG instead of the primary address of that NIC. When the Edge firewall received the packets it dropped them because its rules were configured to only allow packets out when sourced with the primary IP address of the TMG’s external interface. This of course broke internet connectivity for all internal clients.
The question at hand is… “Why is the TMG server sourcing packets with a secondary address instead of the primary address of the NIC?”
The answer to that question deals with the differentiation between the Network Stack in Server 2008 and above and Server 2003 and below. Server 2003\XP and below were based off the Weak Host Model. Basically, in a Weak Host Model the primary address of the adapter with a route that most closely matches the target IP address is used as the Source IP Address.
In server 2008\Vista and above we re-architected the Network stack. It is based on the Strong Host Model. In the Strong Host Model the concept of a Primary IP Address doesn’t exist. To determine the IP address that is utilized it looks at the routing table to decide the proper NIC to utilize, then uses the process defined in RFC 3484 to choose the source IP for outbound packets. Here is a basic breakdown of how the windows implementation of RFC 3484 chooses an IP address:
Windows Source IP V4 address selection:
- Rule 1 Prefer same address (applies)
- Rule 2 Prefer appropriate scope (applies)
- Rule 3 Avoid deprecated addresses (applies)
- Rule 4 - Prefer home addresses - does not apply to IP v4
- Rule 5 Prefer outgoing Interfaces (applies)
- Rule 6 Prefer matching label - does not apply to IP v4
- Rule 7 Prefer public addresses - does not apply to IP v4
- Rule 8a: Use longest matching prefix with the next hop IP address. (not in RFC!)
"If CommonPrefixLen(SA, D) > CommonPrefixLen(SB, D), then prefer SA. Similarly, if CommonPrefixLen(SB, D) > CommonPrefixLen(SA, D), then prefer SB."
*This says that the IP with the most high order bits that match the destination of the next hop will be used.
Note: Rule 8 - Use longest matching Prefix is similar to rule 8a except the match is with the destination IP address rather than the next hop IP address.
For example, use the following addresses as an example of choosing the longest matching prefix:
TMG Servers External IP Addresses:
192.168.1.14/24 & 192.168.1.68/24
Default Gateway:
192.168.1.127
Convert these addresses into binary:
192.168.1.14 = 11000000 10101000 00000001 00001110 192.168.1.68 = 11000000 10101000 00000001 01000100 192.168.1.127 = 11000000 10101000 00000001 01111111
The 192.168.1.68 address has more matching high order bits with the gateway address 192.168.1.127. This would cause the server to utilize what was originally defined as the “secondary” as the Source IP address of outgoing packets.
*For more information on RFC 3484 please refer to the following link: http://www.ietf.org/rfc/rfc3484.txt . Please note that IPv6 is referenced in RFC. Windows utilize the same process for IPv4 sourcing.
You can also review the following TechNet article for supported document details on the above information:
The functionality for source IP address selection in Windows Server 2008 and in Windows Vista differs from the corresponding functionality in earlier versions of Windows http://support.microsoft.com/kb/969029
So now we know why your TMG server may be sourcing your packets with what you call your “Secondary IP Address”. It isn’t TMG at all. It is the default behavior of the server itself. Your server version is Server 2008 or above. The question is…
“Can I configure my Server 2008 or above in a way that it will only utilize the first IP address as a Source address?”
The answer to that is YES! There is actually a Netsh command that can be utilized to add IP addresses. In that command you use the “SkipAsSource” flag and it will no longer use the IP address you are adding as a Source IP Address. This means that you will have to temporarily remove the IP Address you are having the issues with then re-add them utilizing the Netsh command (This means you will have to have a maintenance window!). Here are examples of the command lines you will use:
Server 2008: Netsh int ipv4 add address <Interface Name> <ip address> <subnet mask> skipassource=true
Server 2008 R2:
Netsh int ipv4 add address <Interface Name> <ip address> skipassource=true
* For details and prerequisites to utilize these commands please refer to the following articles:
All IP addresses are registered on the DNS servers when the IP addresses are assigned to one network adapter on a computer that is running Windows Server 2008 SP2 or Windows Vista SP2 http://support.microsoft.com/default.aspx?scid=kb;EN-US;975808
IP addresses are still registered on the DNS servers even if the IP addresses are not used for outgoing traffic on a computer that is running Windows 7 or Windows Server 2008 R2 http://support.microsoft.com/default.aspx?scid=kb;en-US;2386184
Keep in mind that I gave only one specific example where this may be causing an issue. There may be other problems you run into where the Netsh entry I listed may help you out. No telling… it may not even be on your TMG servers. Maybe you see the issue on your UAG servers, or any other server for that fact.
I hope the information provided helps out!
Author
Brett Crane - Sr Security Support Escalation Engineer, Microsoft CTS Forefront Security Edge Team
Reviewer
Richard Barker - Sr Security Support Escalation Engineer, Microsoft CTS Forefront Security Edge Team
Sergio Medina here, and today I want to talk about a question we receive every now and then and explain what the solution is just in case some of you run into a similar issue. You could hit this issue when publishing an internal webserver that accepts only the username as a valid format. For example, when your web server is configured this way, if you try to log in directly to it using ‘DOMAIN\user’ as the format for the username you receive an error, however if you only include the ‘user’ part of the name in the user field, and provide a valid password of course, you log in successfully.
So let’s say you have an internal web server configured this way and you just configured your new web publishing rule in a Forefront Threat Management Gateway 2010 (TMG 2010) server/array. You used Form Based Authentication (FBA) as the authentication method in the web listener and HTTP Basic delegation in the web publishing rule.
Unfortunately, when you try to test your new publishing rule you find that it is not working as expected. If you analyze network traces between the TMG 2010 server and the webserver you discover that TMG 2010 is adding the NetBIOS domain name as part of the user name value:
Now we see the cause of the problem:
· The user introduces his/her username in the TMG form, as ‘user’ · TMG 2010 adds the NetBIOS domain name, sending the authentication in the format ‘DOMAIN\user’ · The webserver only accepts ‘user’ as a valid format so the request fails
So what could we do now?
a) Change webserver behavior, accepting also ‘DOMAIN\user’.
or
b) Change TMG behavior, sending only the username introduced by the user.
Solution a) is out of the scope of this blog article, so let’s see how to achieve solution b).
The trick here is to change the value of StripDomainFromCredentials, one of the internal properties of the web publishing rule. This property is not accessible using the Forefront TMG Management console so we need to change this is by using COM objects.
NOTE The IFPCWebPublishingProperties2::StripDomainFromCredentials Property is documented here: http://msdn.microsoft.com/en-us/library/ff827099(v=vs.85).aspx
So with a simple script we could change this property for any web publishing rule. Here are the steps:
1. Copy the following lines into a new file named StripDomain.vbs on the TMG 2010 server:
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' ' Microsoft Customer Technical Support ' Script for enabling StripDomainFromCredentials on a rule in TMG 2010 ' THIS CODE IS MADE AVAILABLE AS IS, WITHOUT WARRANTY OF ANY KIND. THE ENTIRE ' RISK OF THE USE OR THE RESULTS FROM THE USE OF THIS CODE REMAINS WITH THE ' USER. USE AND REDISTRIBUTION OF THIS CODE, WITH OR WITHOUT MODIFICATION, IS ' HEREBY PERMITTED. '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' Option Explicit Dim WebPublishingRuleName, newStripDomainFromCredentials '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' ' SET YOUR VALUES HERE ' Rule name to change WebPublishingRuleName = "Rule name" ' Set here custom values newStripDomainFromCredentials = True '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' ' Begin Dim Root, Array, Rules, Rule, intCompare Set Root = CreateObject("FPC.Root") Set Array = Root.GetContainingArray Set Rules = Array.ArrayPolicy.PolicyRules '''''''''''''''''''''''''''''''''' ' Look for the WebListener For Each Rule in Rules Wscript.Echo " Comparing Rule name |" & WebPublishingRuleName & "| with |" & Rule.Name & "|" intCompare = StrComp(WebPublishingRuleName, Rule.Name, vbTextCompare) If intCompare = 0 then Exit For End If Next Wscript.Echo Wscript.Echo "Found Rule with description: |" & Rule.Description & "|" '''''''''''''''''''''''''''''''''' ' Show values Wscript.Echo Wscript.Echo "***** CURRENT VALUES: " Wscript.Echo "** StripDomainFromCredentials = |" & Rule.WebPublishingProperties.StripDomainFromCredentials & "|" Wscript.Echo "***** NEW VALUES: " Wscript.Echo "** StripDomainFromCredentials = |" & newStripDomainFromCredentials & "|" '''''''''''''''''''''''''''''''''' ' Warning and ask to continue Dim strMessage WScript.Echo ' newline Wscript.Echo "Please check if the previous information is correct and you want to apply the changes" strMessage = "Press any key to continue or Ctrl+C to cancel" WScript.Echo ' newline WScript.StdOut.Write strMessage Do While Not WScript.StdIn.AtEndOfLine Input = WScript.StdIn.Read(1) Loop '''''''''''''''''''''''''''''''''' ' Set new values Rule.WebPublishingProperties.StripDomainFromCredentials = newStripDomainFromCredentials Wscript.Echo "***** CURRENT VALUES: " Wscript.Echo "** StripDomainFromCredentials = |" & Rule.WebPublishingProperties.StripDomainFromCredentials & "|" Rule.Save ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
2. Modify the script, changing the value WebPublishingRuleName to the name of the rule you want to modify.
3. Open a command prompt with elevated privileges (Run as Administrator)
4. Run the script using: cscript /nologo StripDomain.vbs
5. Follow the steps in the script.
6. In an Enterprise environment, wait until the configuration changes are applied (Monitoring > Configuration > all servers in the array with Synced Status)
After this, you should see the TMG 2010 server sending only the username, thus resolving our issue:
If you need to undo the change, just edit the script, change the following variable to False and execute it again:
newStripDomainFromCredentials = False
One last thing to mention is that this default behavior has changed from ISA 2006 to TMG 2010 so you might have published your webserver successfully using ISA 2006 but found this new behavior when using TMG 2010.
Hope this helps,
Author: Sergio Medina | Senior Support Engineer | Microsoft CSS Forefront Edge Team
Technical Reviewer: Ashish Kapila | Support Escalation Engineer | Microsoft CSS Forefront Edge Team
As you may know, there is a known security vulnerability for the authentication method MS-CHAPv2.
The following TechNet article provides some detailed information about it: Microsoft Security Advisory (2743314)
Unencapsulated MS-CHAP v2 Authentication Could Allow Information Disclosure
http://technet.microsoft.com/en-us/security/advisory/2743314
You may consider moving away from PPTP VPN connections which are configured to use this authentication method therefore.
But, what are the alternatives and how can you integrate these in TMG 2010 which is configured as a VPN Server?
As you can see, only VPN solutions that rely on PPTP in combination with MS-CHAP v2 as the sole authentication method are vulnerable to this issue. If you need to rely on PPTP connections due to down level clients’ requirements, you may consider changing the authentication method for such PPTP connection to PEAP-MSCHAPv2. I will illustrate later how this can be achieved with TMG 2010.
First of all I would like to mention that you should consider using SSTP or L2TP/IPSec connections instead of PPTP.
Both types are also supported and integrated in TMG 2010.
Why you should deprecate PPTP?
PPTP uses Microsoft Point to Point encryption (MPPE). MPPE uses RC4 stream cipher for encryption. There is no method for authentication of the cipher text stream and therefore the cipher text is vulnerable to a bit-flipping attack.
You could use an authentication method like EAP-TLS, PEAP-TLS or PEAP-MSCHAPv2 for a PPTP connection to address the mentioned vulnerability in MS-CHAPv2. But the encryption method does not change by doing that.
So let’s take a look at L2TP/IPsec and SSTP. The main advantage of Secure Socket Tunneling Protocol (SSTP) is that it is uses TCP port 443 as its server destination port since it relies on SSL which provides transport layer security. Therefore the SSTP packets can pass firewalls and proxies. Unfortunately there is no built-in SSTP support in down level or several 3rd party operating systems. That’s why the affected administrators need to decide between PPTP and L2TP/IPsec for compatibility reasons.
L2TP/IPsec is well-known and has been widely used for many years. If you need some in-depth information, please refer to the following article:
http://technet.microsoft.com/de-de/library/cc771298(v=ws.10).aspx
Because of that fact that there is no need to reinvent the wheel, I would like to point to Deb Shinder who did a pretty good job writing a step-by-step guide on how to implement L2TP/IPsec in TMG 2010.
http://www.isaserver.org/tutorials/Checking-Out-TMG-2010-Virtual-Private-Network-Server-Part3.html
Though its quality, Microsoft does not adhere for third party information.
Before we look at the configuration of PEAP-MSCHAPv2, let me clarify that PEAP-MSCHAPv2 is NOT affected by the mentioned vulnerability, because the MSCHAPv2 information is only transmitted within a secure channel in PEAP phase 2. The following article provides further information about the authentication method PEAP-MSCHAPv2 in general:
http://technet.microsoft.com/en-us/library/cc779326(v=WS.10).aspx One more word before you read ahead. PPTP PEAP-MSCHAPv2 requires a certificate with Server Authentication EKU on a NPS Server from either your organizations PKI or a public CA.
Now let’s focus on how to implement PEAP-MSCHAPv2 in TMG. You may expect that TMG supports PEAP-MSCHAPv2 as valid authentication method, since it is some sort of EAP variant with improved security and because EAP can be enabled in TMG as authentication method. Unfortunately this is not 100% correct. If you choose EAP it in the Remote Access Policy it will cause the VPN connection to fail.
A Windows 7 client will report the following error:
Why does this happen? The reason is that TMG configures the Routing and Remote Access Service (RRAS) and the Network Policy Service (NPS) which are included in the Windows Server operating system.
Doing this, TMG adds the following authentication method as a valid type to its local NPS configuration: “Microsoft: Smart Card or other certificate” which is EAP-TLS, but neither PEAP-MSCHAPv2 or PEAP-TLS.
If you change the NPS configuration on the TMG Server, it will only provide a temporary workaround. The reason is that TMG will overwrite the NPS configuration once its service is restarted.
How you can solve that:
You will need to configure TMG to use Radius Authentication. This will result in outgoing Radius packets to a remote Radius/NPS Server. This additional NPS Server can be configured to enable PEAP support.
As next step you will need to define the remote RADIUS Server(s)
This is it from TMG perspective. After you have applied the configuration, you will need to configure the remote NPS Server.
After you have added the required NPS role on your remote server, open the NPS MMC.
First of all you will need to define and enable a RADIUS client there. This will ensure that incoming RADIUS packets from this source will not be dropped.
Next you will need to create a ‘Connection Request Policy’ and a ‘Network Policy’ in the remote NPS Server.
The following article explains the differences between ‘Connection Request Policies’ and ‘Network Policies’.
http://technet.microsoft.com/en-us/library/cc772279(v=WS.10).aspx
When installing NPS, you will have some policies already. Either you can edit them or create some new ones. We will create new ones for this example. Let’s start with a new ‘Connection Request Policy’ by using the wizard.
Then you will have to define conditions when this policy applies. You have lots of possibilities to define granular rules which match your needs. These possibilities are illustrated in the linked article above. To make things as easy as possible for this guide, we only use a Day-and-Time condition. Basically the condition always applies.
Leave the default settings on the next screen. You could forward the requests once more (Radius Proxy feature), but that does not make sense for this example.
If you are using Network Access Protection (NAP) for your VPN clients, you will need to use the setting “Override network policy authentication settings”. If you don’t use NAP, leave the default settings which I do for this straight-forward example.
Leave the settings on the next page to its defaults and click ‘Next’. After you have checked the summary, click on ‘Finish’. That’s it for the ‘Connection request policy’.
Let’s move on to the network policy.
Right click on ‘Network Policies’ and choose ‘New’.
In the next screen define a suitable policy condition which defines, when the network policies is about to be applied for a connection request. Again, you have plenty of options to use granular conditions. This does not only make sense, but may also be required due to your infrastructure needs. For example you may want to use different authentication methods, you may need to create different policies for VPN, 802.1x, etc.
To keep things simple, we use the Day and Time restriction once more. Again, using it in this way, it will always apply this rule.
Skip the next screen by clicking on ‘Next’. This will bring you to the configuration of the valid EAP types.
You will need PEAP-MSCHAPv2 for this scenario. Uncheck the ones which are not required and add PEAP-MSCHAPv2 by clicking on ‘Add’.
Click on ‘Ok’.
You screen should look like this:
Highlight the PEAP entry in the list and click ‘Edit’:
Make sure that you have chosen a valid and trusted certificate. Besides add EAP-MSCHAP v2 as inner method.
Click ‘Ok’, click ‘Next’ three times and then ‘Finish’ completing the configuration.
That’s it. I hope this guide helps you to understand if you need to change your VPN type and if needed, how to do it.
I am happy to answer related questions and read your feedback in the comments below.
Frank Hennemann
Technical Review:
Markus Sarcletti
Microsoft CSS Windows Networking
This post is about an issue that I came across, while working on a case recently.
SCENARIO:
The scenario was a simple website publishing through ISA server 2006. While accessing the HTTPS website from a client, we were seeing a Failed Connection attempt in the ISA logs. More specifically, the error message was pointing to Error code 13, “the data is invalid”. First, we made sure the rule elements were configured correctly. Next, while reproducing the issue, we collected an ISA Data Packager with the “Web Proxy and Web Publishing” template selected…and also collected a decrypted SSL Network trace from the ISA server.
DATA ANALYSIS:
From the decrypted SSL trace, we could identify that the ISA server returned a response code of 400 (BAD REQUEST). Upon further inspection of the HTTP headers and payload (the request was a POST), we could not find any issue which was obvious. So we looked further into the ISA Tracing file, which is a component level trace collected by the ISA Data Packager.
In that trace, the request seemed to contain a Multi-Line header which is not allowed by ISA server, by default. Hence it is rejecting the request made by the client. The ISA tracing contained the following error:
ERROR:Multi line header - unsupported
We then revisited the decrypted SSL trace and observed that there was a MultiLine header found.
This trace example contains a multi-line header
In an attempt to reproduce the issue, we also used a web debugging tool to construct a similar HTTP POST request with the same payload and same headers and that worked as expected! When we checked the network trace from our repro, we could see that the header was not sent as a “Multi-line”.
This trace example does not contain a multi-line header
The following byte codes (highlighted above) can be indicative of a multi-line header:
0d – Carriage Return
0a – New line (Line Feed)
09 – Horizontal tab
More information about the byte codes and their corresponding mappings can be found in RFC2616.
RESOLUTION:
In order to resolve the issue, we opened the registry editor on the problematic server and navigated to HKEY_LOCAL_MACHINE\SOFTWARE\MICROSOFT\RAT\Stingray\Debug\Web Filters
and added a DWORD value APPEND_CONTINUATION_LINES and set the value to 1, and then rebooted the server.
The issue no longer occurred. The client was able to access the site successfully.
The solution to this issue has been addressed in detail in the following KB article:
http://support.microsoft.com/kb/941293
This fix appends the continuation tab, thereby enabling the ISA server to successfully read the headers and process it further.
I hope this post would be helpful in resolving issues that you may face while publishing web applications through ISA/TMG. But always remember not to make any changes to the registry unless you have isolated the root cause of the issue. Also, as a general best practice, take a backup of the registry before making modifications.
Security Support Engineer – MSD Security Team
Introduction
I have seen a number of cases where customers were installing TMG 2010 in a “hybrid” scenario. What I mean by this is that the EMS was part of the Domain but the Array Servers were in a workgroup. There are a couple of “gotchas” that I wanted to talk about today.
Assumptions
I am going to make a few of assumptions before I get started. First I am going to assume that you have already installed the TMG Enterprise Management Server (EMS) on a server that is a domain member. I am also going to assume that you have installed the TMG Array Member on a server that is in a workgroup. I recommend getting both of them to the latest Service Pack, Updates, and hotfixes before proceeding. You definitely want both the EMS and TMG Array server to be the same code level. Please refer here for version number information on TMG.
Certificates and Accounts
The first thing you want to do is request a Server Authentication certificate for your EMS. It needs to be issued to the Fully Qualified Domain Name (FQDN) of the EMS with the Private Key Exportable option checked. In my lab the EMS is called ems.fabrikam.com so I requested a Server Authentication certificate from my Certification Authority and installed it into the Certificate Store. Make sure you also export the .PFX file for the certificate, with the private key, and put the file somewhere handy on the EMS machine.
Next you will want to make sure that both the EMS and the TMG Array Server trust the CA that issued the Server Authentication certificate to the EMS. You can do this by importing the certificate for the CA into the Trusted Root Certificate Authorities branch in the Computer Store on each of those machines.
Another thing that is sometimes overlooked in the scenario is that mirrored accounts are needed on both the EMS and the TMG Array Server. For example, I just used the fabrikam/administrator account on the EMS and the local Administrator account on the TMG Array server. They both have the same password.
Keep in mind that if you have any firewalls that reside between the EMS and the TMG Array Servers you will initially want to allow ALL traffic between them. You can tweak this down later but it can cause you a lot of heartache with communications in TMG.
Create the New Array
On your EMS, open the TMG MMC, highlight the Arrays branch, and then on the far right-hand Tasks pane choose to Create New Array
Give your new array a name.
Type in the DNS name of the array.
Choose the Default Policy.
Click Next at the Array Policy Rule Types.
Complete the New Array Wizard.
TMG will create the new array and you should see that it was a success.
Apply this on the EMS.
Wait for the configuration changes to be saved.
In the TMG MMC on your EMS, there should now be a branch called Arrays. Below it should be the array that you just created.
Joining the New Array
Back on your TMG Array Server go into the TMG MMC and highlight the branch that says Forefront TMG (servername). On the far right-hand pane under the Tasks Tab, click Join Array.
You will see a welcome screen for the Join Array Wizard.
Under the Array Membership Type choose to “Join an array managed by an EMS Server”.
Give it the Fully Qualified Domain Name of your EMS. (Note: you will want to make sure name resolution is working properly on the TMG Array Server before you do this step).
The newly created array should come up as a choice.
Click Finish on the Completing the Join Array Wizard.
You should get a message that you successfully joined the array.
Give it a few minutes but you will probably notice that the configuration is not synching and you will get an error and a red X under the Configuration Status. The error reads “Forefront TMG Management cannot establish a connection with the Forefront TMG Computer.”
The Problem
So why isn’t the TMG Array Server able to communicate with the EMS? It seems like everything was set up correctly. TMG in a workgroup scenario relies on Authentication over SSL encrypted channel (LDAPS). That is the reason we requested the Server Authentication certificate for the EMS Server.
You can verify this by going into the MMC on your EMS, right-clicking on the top level branch of the array that you just created and choosing Properties.
Under the Configuration Storage branch the authentication type near the bottom should be set to “Authentication over SSL encrypted channel”
The problem is that the Server Authentication certificate was never bound to the ISASTGCTRL service.
You can verify this by creating a Certificates Snap-in MMC.
Choose to manage snap-in for a Service Account.
Select the Local Computer.
Select the ISASTGCTRL service and finish.
You should now see that there is not a certificate under the Personal branch.
Keep this open, you will refresh it in a few minutes.
To correct the certificate issue, download the TMG Cert Tool Pack from here.
Install the tool on the EMS but then move the ISACertTool.exe to the same directory where TMG is installed. Open an administrative command prompt and navigate to that directory. Run the command as explained below against the .PFX file you have for your EMS Server Authentication Certificate.
The syntax is listed in the DOC file that comes with it and is:
• ISACertTool /st file_name [/pswd password] [/keepcerts]
Where:
• /st file_name installs the exported certificate on the Configuration Storage server. File_name specifies the path and name of the exported .pfx certificate file.
• /pswd password specifies the password that may be required when installing the server certificate. It is only required if a password was specified during export of the certificate file.
• /keepcerts specifies that existing certificates should not be deleted. By default when you run ISACertTool.exe, all certificates in the ADAM_ISSTGCTRL local store are erased. To specify that existing certificates should not be deleted, specify the /keepcerts parameter.
After running this you should see a message that the Storage certificate installation was successful.
Go back to the Certificate Snap-in for the ISASTGCTRL Service and refresh the Personal Branch. You should now see the Server Authentication Certificate.
Now go back to Monitoring on your EMS and you should see that the TMG Server is successfully synching.
Setting TMG up where the EMS is in a domain and the TMG Array Servers are in a workgroup can be tricky. TMG in a workgroup relies on an SSL Encrypted channel and sometimes getting that to work correctly is not always straight-forward. In this article I have shown you a couple of the common pitfalls and how to correct them.
Keith Abulton – Security Support Escalation Engineer, Microsoft CTS Forefront Security Edge Team