website statistics
Yuri Diogenes's Blog - Site Home - TechNet Blogs

Yuri Diogenes's Blog

Thoughts from a Senior Technical Writer @ Microsoft Server and Cloud Division (Solutions Group) - Information Experience

Posts
  • Yuri Diogenes's Blog

    VPN users are unable to browse the Internet when connected to TMG and the web browser is configured to “automatically detect settings”

    • 0 Comments

    Consider the following scenario: Remote access VPN client users are unable to browse the Internet when connected to TMG and the web browser is configured to “automatically detect settings”. When connected, the WPAD record appears to be resolving to the IP address of the RRAS interface and not the interface of the TMG firewall.

    This problem can happen because RRAS interface is higher than the internal interface in the binding order of the OS. One quick fix for that will be to change the binding order to have the internal interface on the top. Another approach is to follow the steps below:

    1. Download the CarpNameSystem.js

    2. Open command promot in elevated privilege and run the command:

    cscript carpnamesystem.js /set: DNS

    3. Restart Firewall Sevice and run the command below in the workstation that is connecting remotely:

    del \wpad*.dat /s

  • Yuri Diogenes's Blog

    DRP- The Community Making a Difference

    • 0 Comments

    Disaster Recovery Plan, also known as DRP, this discipline is mainly concern about “Availability”, which is one of the main pillars in the Security Triad (Confidentiality, Integrity and Availability). The security principals (and common sense) determines that first and foremost we all need to make sure everyone is safe (human life is the top priority in any DRP). In an extreme situation, like our friends from Japan are living at this moment, there are more than just availability to be concern about: integrity and confidentiality might be gone for some business.  In order to assist the business in Japan to have some guidelines on what to do to be back in business the article below was created:

    http://social.technet.microsoft.com/wiki/contents/articles/windows-server-emergency-management-resources.aspx

    Here are some important points to notice in this article:

    image

     

    …and also the tags that we currently have:

    image

    There are much more to add, so make sure that you take some time to add valuable information to this article. This can be very useful for those that are desperate to put their business back on track.

  • Yuri Diogenes's Blog

    Chat about TMG on Talk TechNet Show

    • 0 Comments

    imageAs I announced in this post, yesterday I was on Talk TechNet Show with Keith Combs and Matt Hester. I had a great time talking about TMG and answering questions about the product. If you missed the show you can still listening the conversation by downloading the MP3 from here. I also want to say to the Forefront TMG Administrator’s Companion Book winner that I shipped the book today and it should be with you on Saturday Smile.

    BTW, don’t miss Tom Shinder’s interview on Talk TechNet tomorrow (Friday 11th), registration is still open here.

  • Yuri Diogenes's Blog

    Building Community Based Content with TechNet Wiki

    • 0 Comments

    Last week on MVP Summit was indeed quiet busy, but the results were great. Here an interview with David Tesar for the Technet Edge site where I talk about the value of building a community based content using TechNet Wiki:

     

     

    You can also download in WMV format.

  • Yuri Diogenes's Blog

    Troubleshooting Forefront TMG 2010 Performance issues Cheat Sheet

    • 0 Comments

    Last week I presented a session on MVP Summit in Redmond about Troubleshooting TMG Performance issues. During that presentation I said to the MVPs there that I will be writing a cheat sheet with some WinDBG commands that can be used while troubleshooting TMG performance issues. I thought about this type of document and concluded that this content can have a base framework but it should be expanded and enhanced by the community. Having said that, I decided to write this article in two places:

    • Microsoft TechNet Wiki, that you can access it from here.
    • A printable and downloadable PDF version that you can get it from here.

    Enjoy it !!

  • Yuri Diogenes's Blog

    Forefront TMG Update 1 Rollup 3 is now Available

    • 4 Comments

    The Update 1 Rollup 3 for Microsoft TMG 2010 is now available for you. This rollup address the following issues:

    KB article

    Title

    2501646

    FIX: "A security package specific error occurred" error when you run a recurring report on a Forefront TMG 2010 server that is managed by an EMS and that is in a workgroup

    2502685

    FIX: "0xc0360007 (STATUS_IPSEC_CLEAR_TEXT_DROP)" error when you try to access the internal IP address of a Forefront TMG 2010 server through an IPsec site-to-site network

    2472894

    "HTTP/1.1 502 - Error 11 Bad format" error when you access SSL websites that use SAN certificates in Forefront TMG Server 2010 if a non-English version of a Windows operating system is installed

    2501650

    FIX: "Page Cannot Be Displayed" error when you try to access a website that requires a client certificate authentication on a Forefront TMG client in Forefront TMG 2010 if HTTPS Inspection is enabled

    2501776

    FIX: "502 Proxy Error. An attempt was made to load a program with an incorrect format. (11)" error when you try to use a HTTPS URL through Forefront TMG 2010 if HTTPS inspection is enabled

    2498831

    How to configure the "HTTPS inspection caching in a forward proxy scenario" and "HTTPS inspection inclusion list" features in Forefront TMG 2010

    2498837

    An enterprise node is incorrectly added in Forefront TMG MMC after you install Forefront TMG 2010 SP1 Update 1

    2445386 

    "Sign in as a Different User" does not work on a SharePoint website that is published by Forefront TMG 2010

    2498835 

    PPTP or L2TP/IPsec connection is not reestablished between Forefront TMG 2010 servers

    2501777 

    FIX: "502 Proxy Error. An unknown error occurred while processing the certificate. (-2146893017)" error when you try to access a website over HTTPS in Forefront TMG 2010 if HTTPS inspection is enabled

    2497959

    Forefront TMG Firewall service may stop when users run desktop sharing software over HTTPS that is proxied by Forefront TMG 2010

    2500737

    "0xc0040446" or "0xc004041d" error if the primary IP address or DNS address uses 128.0.0.0/16, 191.255.0.0/16, or 223.255.255.0/24 in Forefront TMG 2010

    2497858

    SCOM logs many "Forefront TMG Server - Cache: Current Cache Fetches Average Ms Per Request error" error alerts from TMG Management Pack through Forefront TMG 2010

    2501755

    Mspadmin.exe may crash if you do not use SQL Server Express to log traffic in Forefront TMG 2010

    2502686

    Forefront TMG Firewall service might crash when WP_TRAFFIC tracing is enabled in Forefront TMG 2010

    2501782

    "0xc004039E" error when you use the "Allow user override" setting for a HTTP deny rule in an enterprise policy in Forefront TMG 2010

    2501780

    FIX: Forefront TMG Job Scheduler service (Isasched) stops responding on an array member server that is not a report server in Forefront TMG 2010

    As you can see there are a lot of fixes in this rollup, I particularly worked in many issues involving 2501650 and 2502686 while the hotfixes were not even ready. Due the nature of those issues I strong recommend you to download this update and plan the installation on your Forefront TMG. To install this update, you must have TMG 2010 SP1 and Update 1 already installed.

    Got get it at  http://support.microsoft.com/kb/2498770.

  • Yuri Diogenes's Blog

    Secure Access to your Cloud Services with Forefront TMG

    • 0 Comments

    Greetings!

    The article that I wrote for TechNet Magazine February issue is now available, you can access it from the link below: 

    image

    http://technet.microsoft.com/en-us/magazine/gg607680.aspx

    This article will give you a better picture of how Forefront TMG can assist your cloud migration by enhancing secure Internet access to the cloud services.

    Enjoy!

  • Yuri Diogenes's Blog

    Windows Security Survival Guide

    • 0 Comments

    Yesterday I post my first WiKi article, it is about Windows Security and the core Windows foundation to cover the security triad (Confidentiality, Integrity and Availability). Many IT Pros sometimes jump directly to try to hardening the system without first step back and analyze the business needs as well as how to cover the core security triad using built in resources available on Windows OS. This article will cover this discussion.

    You can access this article from the link below:

    http://social.technet.microsoft.com/wiki/contents/articles/windows-security-survival-guide.aspx

    The Microsoft TechNet WiKi is a great resource for exchanging experiences by writing content that you feel will be useful for the community. If you have a need and you look for an article and don’t find it, why not you write your own article under Microsoft TechNet? That’s the goal here, to make sure that you can help the community to succeed. Here are some articles that you should read before get involved on this:

    Get yourself engaged and enjoy working with such great community!!

  • Yuri Diogenes's Blog

    Talk TechNet with Keith Combs and Matt Hester – Episode 11: Yuri Diogenes on Forefront Threat Management Gateway

    • 0 Comments

    Registration is open for Episode 11 of Talk TechNet:

    https://msevents.microsoft.com/CUI/WebCastEventDetails.aspx?EventID=1032477768&EventCategory=4&culture=en-US&CountryCode=US

    Looking forward to chat with you about TMG as Secure Web Gateway.

  • Yuri Diogenes's Blog

    See you around Forefront Edge Community

    • 9 Comments

    I created this blog back in February 2008 and since that day I really tried to bring to you interesting troubleshooting techniques based on real scenarios. This blog was always something that I drove on my own free time (not that I have lot of free time), but I tried to managed my time in such way that posting here was part of my regular agenda. The numbers below show how much the traffic increased over the last couple of years in this blog and I would like to thank you all for contributing with that, is because I know you are reading that I feel energized to keep writing.

    image


    I can safely say that 90% of the posts that I wrote for this blog were related to ISA/TMG, which makes a lot of sense to me as I was working for CSS Forefront Edge Team. Yes, you read it right, I “was”. Starting Monday (Feb 14th) I will be fully dedicated to the Windows Security Team as a Technical Writer. As one of the co-authors of the Forefront TMG Administrator’s Companion Book and Forefront TMG Deployment Guide, I plan to keep writing about Forefront TMG here, but certainly will not be on the same frequency as before since I will be dedicated to Windows Security subject. From now on I will be more engaged in produce content that will be available in other locations, such as:

    http://technet.microsoft.com/en-us/windowsserver/windows-server-security.aspx

    http://social.technet.microsoft.com/wiki/contents/articles/wiki-it-security-portal.aspx

     

    There are some initiatives on the Forefront TMG space that I’m still engaged during this transition phase, which are:

    • MVP Summit 2011 – I hope to see all my MVP friends there, my presentation will be on Wednesday March 2nd (first two sessions in the morning).
    • Talk TechNet – I will be on Episode 11 of Talk TechNet to discuss about Forefront TMG as Secure Web Gateway. More details will soon be available here.
    • TechNet Magazine Article – a new article that I wrote for TechNet Magazine about using TMG to assist on BPOS deployment will be available on TechNet Magazine February issue (expected to be out by Feb 21st).

     

    Again, thanks a lot for visiting this blog and I hope to keep partnering with you in 2011, now in a broader way.

     

    Stay Safe!!

  • Yuri Diogenes's Blog

    Inbound TLS SMTP Traffic gets TCP Reset when published through TMG 2010

    • 0 Comments

    Consider a scenario where you are publishing a SMTP Server that uses TLS on Forefront TMG 2010, in such scenario TMG resets the connection to the SMTP client when the SMTP server closes its connection to TMG with a TCP FIN packet. This behavior can cause some specific SMTP client applications to report that message delivery failed even though messages are sent correctly. In this scenario you also will see the following entry on the Live Logging: Incoming SMTP Server    0x80074e24 FWX_E_CONNECTION_KILLED. This problem is documented for ISA Server 2006 in KB 959312. Recently we also experienced the same issue with TMG and the script from KB 959312 fixed the issue. After running this script on TMG you should see the message below on your command prompt windows (which should be opened in privileged mode):

    image

    After this change such behavior should go away…and yes, we will update this KB to include TMG.

    Note: only run this script on TMG if you are experiencing exactly the same behavior as explained in KB 959312.

  • Yuri Diogenes's Blog

    A New PAL for TMG

    • 3 Comments

    Back in 2009 I wrote this post about PAL (Performance Analysis of Logs Tool), during that time we didn’t have ISA/TMG template available. Good news is that we now have the TMG template available!!! Back in April 2010 we had our Security Summit in Porto (Portugal) where we started to elaborate a plan to make this happen.  My contribution was very little comparing with what those guys from PFE did, I pretty much assisted with some TMG perfmon counters/thresholds. Main kudos here goes to: Clint Huffman (the tool owner), Shaf Mahmood, Zbigniew Kukowski (content owner), Dirk-Jan van der Vecht and Luís Galvão.

    Download PAL from here, once you install it you will notice that TMG is now on the list, as shown below:

    image


    The current template is V2.4 and the template file is located here:

    image

    Now you you have a new PAL to help you out. Enjoy it !!

  • Yuri Diogenes's Blog

    Distributing the Authentication Load – A follow up on my previous post

    • 2 Comments

    Yesterday I published this post about an issue that caused TMG stop to respond and I want to clarify one key point here: TMG didn’t stop because was not able to handle the load, let’s be clear on that. Maybe it was not clear for some readers that don’t know about how TMG works, but the issue here was that the DC was not able to handle the gigantic amount of authentication request on the time speed that TMG was sending the requests and waiting for an answer. As a result of that TMG’s backlog started to grow and caused this behavior. Its plain simple: DC was not sized to handle that amount of authentication request. Again, not a TMG issue.

     

    Couple of things can be done to avoid that those incidents don’t fully affect your environment. Here are some key tips (nothing new, but maybe you missed):

    ·         Don’t create rules allowing ALL OUTBOND TRAFFIC as Protocol. This may cause issues as I explained in this post.

    ·         Make sure to use Internet Explorer 7 or higher to take advantage of Kerberos, which will distributed the authentication load among the DCs. Back in 2008 I wrote this article that explains in details all the advantages of using Kerberos for Proxy authentication.

     

    By using those practices you offload the authentication request to go from TMG to the DC and leave this task for the workstation (again read this article for more info), which dramatically impact the backlog (by lowering the utilization). Last but not least I want to say that it’s all about sizing: if the environment was sized to receive 20 x 100, it will have a negative impact if you see 2000 x 100. There is no magic here, in this case TMG was correctly sized, but as a secure firewall it couldn’t allow traffic to pass through without waiting for the DC to reply back saying that that request comes from a valid user, therefore it will fail safe and block the traffic from traversing the networks.

     

    BTW, for those of you that still believe that Hardware Firewall is better, I will let you with the wise words of my friend Tom Shinder about this old discussion: Tom Shinder on “Hardware” Firewalls.

     

    Enjoy Smile

  • Yuri Diogenes's Blog

    TMG Hangs and requires a manual restart

    • 2 Comments

    1. Introduction

     

    Yes, another case where TMG stops responding…bad, bad TMG right? NOT!!! Recently I worked in some scenarios where TMG was stopping every day, during the same time and required a manual restart. TMG Admin claimed that nothing really changed on TMG or on the client workstations, besides the environment was running rock solid for a long time and the issue started happened couple of weeks ago.

     

    2. Digging In

     

    After many sessions of data gathering (believe me, it can take more than one round to find out) using the usual approach to collect performance related data I found out the following trend when the issue was happening:

     

    image

     

    This is the Forefront TMG Firewall Packet Engine\Backloggged Packets counter, which should be 10 and it was 2,485.000 (WOW…just WOW). Notice this beautiful line going from 0 to 2K and worst, staying there forever. Now we know why TMG stops responding, but why backlogging is growing? Well, there are two core reasons: authentication and/or name resolution. On the user mode dump of wspsrv.exe we have hundreds of threads like this:

     

    0:000> ~27k

    Child-SP          RetAddr           Call Site

    00000000`11fbccc8 000007fe`fd82aa76 ntdll!ZwAlpcSendWaitReceivePort+0xa

    00000000`11fbccd0 000007fe`fd8ccb64 rpcrt4!NdrDllCanUnloadNow+0x31c6

    00000000`11fbcd90 000007fe`fd8ccd55 rpcrt4!Ndr64AsyncClientCall+0xe04

    00000000`11fbd050 000007fe`fc9e1f95 rpcrt4!NdrClientCall3+0xf5

    00000000`11fbd3e0 000007fe`fc9e1e74 dnsapi!DnsApiAlloc+0xdd1

    00000000`11fbd440 000007fe`fc9e60a6 dnsapi!DnsApiAlloc+0xcb0

    00000000`11fbd500 000007fe`fca0d012 dnsapi!DnsValidateName_W+0x186

    00000000`11fbd580 00000000`72cab68f dnsapi!DnsQuery_A+0x36

    00000000`11fbd5d0 00000000`72caaced msphlpr!COC_NameResolution_TargetImpl::FoundInNegativeCache+0x2b93

    00000000`11fbd6a0 00000000`72ca6efe msphlpr!COC_NameResolution_TargetImpl::FoundInNegativeCache+0x21f1

    00000000`11fbd960 00000001`3fa467ff msphlpr!ProxyGetHostByAddr+0x4a2

    00000000`11fbdce0 00000001`3fa48420 wspsrv!FwGapaGetConfig+0x4b4a3

    00000000`11fbddb0 00000001`3f8ded8d wspsrv!FwGapaGetConfig+0x4d0c4

    00000000`11fbe660 00000001`3f92e07c wspsrv+0x5ed8d

    00000000`11fbe730 00000001`3f92d79e wspsrv!IsChainingRequired+0x163cc

    00000000`11fbef90 00000001`3f8ea240 wspsrv!IsChainingRequired+0x15aee

    00000000`11fbf050 00000001`3f8f1c0a wspsrv+0x6a240

    00000000`11fbf1f0 00000001`3f8f11d2 wspsrv+0x71c0a

    00000000`11fbf270 00000001`3f9838bf wspsrv+0x711d2

    00000000`11fbf320 00000001`3f97d871 wspsrv!DeleteFwEngFilter+0x249b

    00000000`11fbf360 00000001`3fa1bedc wspsrv!IsChainingRequired+0x65bc1

    00000000`11fbf550 00000001`3f971a53 wspsrv!FwGapaGetConfig+0x20b80

    00000000`11fbf6a0 00000001`3f94185c wspsrv!IsChainingRequired+0x59da3

    00000000`11fbf780 00000001`3f9415ce wspsrv!IsChainingRequired+0x29bac

    00000000`11fbf7f0 00000000`771cf56d wspsrv!IsChainingRequired+0x2991e

    00000000`11fbf880 00000000`77403021 kernel32!BaseThreadInitThunk+0xd

    00000000`11fbf8b0 00000000`00000000 ntdll!RtlUserThreadStart+0x21

     

    …and lots more like this:

     

    0:000> ~14k

    Child-SP          RetAddr           Call Site

    00000000`021aece8 000007fe`fd3e10ac ntdll!ZwWaitForSingleObject+0xa

    00000000`021aecf0 00000001`3f90609b KERNELBASE!WaitForSingleObjectEx+0x9c

    00000000`021aed90 00000001`3f8ec283 wspsrv+0x8609b

    00000000`021aee60 00000001`3f8f1c37 wspsrv+0x6c283

    00000000`021af250 00000001`3f8f11d2 wspsrv+0x71c37

    00000000`021af2d0 00000001`3f9838bf wspsrv+0x711d2

    00000000`021af380 00000001`3f97d9fe wspsrv!DeleteFwEngFilter+0x249b

    00000000`021af3c0 00000001`3fa1bedc wspsrv!IsChainingRequired+0x65d4e

    00000000`021af5b0 00000001`3f971a53 wspsrv!FwGapaGetConfig+0x20b80

    00000000`021af700 00000001`3f94185c wspsrv!IsChainingRequired+0x59da3

    00000000`021af7e0 00000001`3f9415ce wspsrv!IsChainingRequired+0x29bac

    00000000`021af850 00000000`771cf56d wspsrv!IsChainingRequired+0x2991e

    00000000`021af8e0 00000000`77403021 kernel32!BaseThreadInitThunk+0xd

    00000000`021af910 00000000`00000000 ntdll!RtlUserThreadStart+0x21

     

    I cannot use the private symbols here (for obvious reasons – they are private Smile), but this function is dealing with re-injection and we had 50 threads performing this operation. This is a magic number, because 50 is the default value of re-injection threads on TMG as I explain here.

     

    3. Moving forward

     

    We now know why TMG box hangs, but why we have this gigantic amount of authentication if the environment didn’t suffer any change, the applications are the same, the users are the same…how’s that possible? Maybe a malware sending burst traffic from inside to outside? We didn’t know but we continued the investigation and concluded that the environment was clean from malware.

     

    We used netmon to understand from where the traffic was coming from and it was identified some IPs on the internal network that were sending this gigantic amount of traffic. We tracked that IPs and found the owners of those computers; they were contractors that were in the company performing a project. Guess what? They were using a P2P application to download “some stuff”. I started reviewing more info about this application and found the following statement on their website:

     

    “If you use a software firewall (e.g. ZoneAlarm) you will need to make sure Ares P2P gets full and unlimited access to the Internet.”
    Source:
    http://www.aresp2p.net

     

    What’s your reading on this? I will let you think about that.

     

    4. Conclusion

     

    This was not a TMG problem, TMG was hanging because it was waiting from DC/DNS to reply to the plethora of requests that it was sending through the network and as it couldn’t authenticate the users and couldn’t allow them just to go through. We really need to step back here and think about security in a broader manner, how do you validate the guest computers on your corporate network? Because here, the environment had a good security policy for domain joined computers by using software restriction policy and disallowing non-authorized applications to run on the corporate environment. But, it didn’t have a validation process for guest computers. I will suggest start by reading this “Protect Corporate Assets from Unmanaged Computers” and go from there. At the end of the day you don’t want to create a rule on your firewall allowing everything for all users just because there are some applications that need full Internet access, unless you are willing to take the risk for such action, are you?

     

    Stay safe!

    (Note: read the follow up of this post here).

  • Yuri Diogenes's Blog

    Unable to Connect to Microsoft Online Services (Lync) behind ISA Server

    • 0 Comments

    1. Introduction

     

    BPOS is growing in a fast pace and as IT Admins starts to use this service they need to adjust their Firewall in order to proper allow the traffic to traverse the on-premise clients to the cloud. Microsoft Online Services did a good job documenting what needs to be in place from the Firewall perspective to allow this traffic to correctly flow. Here are the main articles for this type of deployment:

     

    KB2410859 Firewall prevents users from using Microsoft Online Services Directory Synchronization, rich clients, or the Microsoft Online Services Identity Federation Management tool in Office 365

     

    KB2409256 You cannot connect to Lync Online, or certain features do not work, because an on-premises firewall blocks the connection

     

    Both articles mention ISA Server as an example and they also mention that for ISA you may need to use Firewall Client in order to make this deployment to work. If you use Firewall Client, nothing else needs to be done on the client workstation, however, if you don’t want to install Firewall Client you will need to edit the file Program Files\Microsoft Online Services\Sign In\SignIn.exe.config and add the entry below:

     

    <system.net>

        <defaultProxy useDefaultCredentials="true">

          <proxy  usesystemdefault="True" />

        </defaultProxy>

    </system.net>

    Source: http://technet.microsoft.com/en-us/library/ee832722.aspx

     

    2. Scenario

     

    Consider a scenario that you have all the implementations in place, rules are correctly configured on ISA Server as per KB2410859 and have Firewall Client on the workstation, however the issue persists and on ISA log you see access denied due anonymous request. When closely look to the detailed logging (Monitoring/Logging/Lower Pane) you see that no rules appear in there, which means that the request is getting processed in lower level mode (kernel).

     

    3. Solution

     

    The problem here was caused because the option below was enabled:

    image


    When you enable this option you might have issues with a variety of applications (not only BPOS), because this option completely disable Anonymous access for Web Proxy requests on the network. This application forces the user’s credential to be requested even before the firewall policy is starting to get evaluated. This is the reason why when you enable this option you receive the warning below:

    image

    As you can see on this warning window, this option can cause compatibly issue with applications such as Windows Update (and I found out that with BPOS too). In order to avoid compatibly problems, disable this option and make sure to control your user access via Firewall Policy. There are many other scenarios where we recommend to disable this option, see this article for more information. After disabling this option the user was able to login:

    Have a good migration to the Cloud!!

  • Yuri Diogenes's Blog

    Unable to join a new Node on an existing TMG 2010 Array

    • 3 Comments

    First of all, happy new year!! It took me a long time to come back here due many other projects going on…I’m actually feeling like I still in December as I didn’t really have time off during the holidays. Very different from Brazil (where I’m originally from), there things are slow during the holidays and keep going slow until Carnival (usually in February). This is actually very funny to me, because recently when I was writing my next book (about Security+ Certification in Portuguese) I told my editor: “let’s release the book in March” and he said: “This year Carnival is in March, so nobody will really read books in March, let’s release in April”, he got a good point for sure. But, since I moved to US I notice that the year really starts on January 2nd :).

    Anyway, here it goes the first post of this year and it is about a collaboration with a colleague of mine that was originally troubleshooting this issue. The problem here was when trying to join a new node to an existing TMG Array and the following error message appeared:

    image

    The user that was trying to join had permission on the Array level as shown below:

    image

    We also could see on ProcMon that this user was making the connection to the remote server while the issue was happening:

    image

    Unfortunately in this situation as the error message was showing right away, nothing was really useful in TMG Setup logs (located at %windir%\temp). Now what? Well, now you need to move to a more deep data gathering and use TMG Data Packager in both servers (EMS and Node that is trying to join). In this particular scenario it was possible to see the error “ldap_modify_s failed” followed by 0x80070005 (which is Access Denied) while trying to change some properties on ADAM (ADLDS). After reviewing the source code for this specific error at this moment of the failure it was possible to understand that in order to perform such action the user needs Enterprise level rights, in this case the user was not there as shown below:

    image

    Once we added the user in there (Enterprise Level) it was possible to join without any issue. So…when deploying TMG, make sure to remember that the user that is joining new members to the array need to have Enterprise Level permission.

    Note: If you decide to add a group there, remember the warning for the following window:

    image

  • Yuri Diogenes's Blog

    There is a XMas package for you that use TMG 2010…hey and for you that use ISA Server 2006 too

    • 8 Comments

    Today we are making publicly available the Software Update 1 Rollup 2 for Forefront Threat Management Gateway (TMG) 2010 Service Pack 1. This hotfix include resolution for the following issues:

     

    article

    Title

    2452980 (http://support.microsoft.com/kb/2452980)

    Upload speed through Forefront TMG 2010 is very slow on a high speed Internet connection

    2478286 (http://support.microsoft.com/kb/2478286)

    Connection does not time out after inactivity time elapses in an OWA 2010 client connected to Exchange Server 2010 if published by using Forefront TMG 2010

    2484988 (http://support.microsoft.com/kb/2484988)

    A DNS server publishing rule stops working for a DNS server that is published by using Forefront TMG 2010

    2478297 (http://support.microsoft.com/kb/2478297)

    User Activity reports that are created by Forefront TMG 2010 show a wrong value in the reported data range

     

    Notice that the first issue on this KB is the same that we were discussing on this TechNet thread here. So if you are facing such issue, make sure to install this update and run the script from KB2452980 (http://support.microsoft.com/kb/2452980). The other issue that we address on this rollup was raised from one of my customers as a problem, working with him I was able to reproduce the issue and after a long investigation we were able to find the root cause of the problem (in a great partnership with Exchange Team and TMG Developers), read KB2478286 (http://support.microsoft.com/kb/2478286)  for more details. The third issue that we address on this TMG rollup is a DNS publication that stops to work, see KB2484988 (http://support.microsoft.com/kb/2484988)  for more details. Last but not least a problem on the user activity report, simple stuff but that bothers for sure; see KB2478297 (http://support.microsoft.com/kb/2478297)  for more details.

     

    For ISA Server we are releasing the ISA Server 2006 hotfix package: December 2010, which includes the following updates:

     

    KB article

    Title

    2478307 (http://support.microsoft.com/kb/2478307)

    MAPI client does not connect to an Exchange server on an internal network through an ISA Server 2006-based VPN connection on a computer that is running Windows 7

    2481980 (http://support.microsoft.com/kb/2481980)

    Unexpected authentication prompts while you use an OWA website that is published by using ISA Server 2006 SP1 if RSA authentication and FBA are used

     

    Go get it and enjoy your holidays.

     

    Merry XMas !!

     

  • Yuri Diogenes's Blog

    Eternal Loop while Accessing a site Published by TMG 2010

    • 0 Comments

    Consider the following scenario:

    • You have a computer that is running Forefront TMG 2010.
    • You have a Web server that automatically redirects HTTP requests to Secure Socket Layer (SSL) requests.
    • You configure the Web listener to listen for HTTP requests and also to use bridging.
    • You configure the Web listener and the bridging for both HTTP and for SSL requests (HTTPS).

    In this scenario, when the Web server receives an HTTP request, it redirects the request to the TMG adding the https on the new location within the header as shown below:

    - GET Request sent from TMG to the internal Server:

    Http: Request, GET /default.aspx
        Command: GET
      + URI: /default.aspx
        ProtocolVersion: HTTP/1.1
        Via:  1.1 TMG
        Host:  contoso.com
        Accept:  */*
        Accept-Language:  en-us
        Connection:  Keep-Alive
        Accept-Encoding:  peerdist
        HeaderEnd: CRLF

    - Web Server reply with the new location:

    Http: Response, HTTP/1.1, Status: Moved temporarily, URL: /default.aspx
        ProtocolVersion: HTTP/1.1
        StatusCode: 302, Moved temporarily
        Reason: Found
        Cache-Control:  private
        Location: 
    https://contoso.com/default.aspx
        Server:  Microsoft-IIS/7.5
        XAspNetVersion:  2.0.50727
        XPoweredBy:  ASP.NET
        ContentLength:  149
        HeaderEnd: CRLF

    Problem: TMG receives the request with the new location and instead of sending this new location to the client workstation, it sends http://contoso.com/default.aspx (removing the “s”), client receives this 302 and send the request again, causing an eternal loop.

    Resolution: in order to fix this problem, use the resolution (method 2) from KB http://support.microsoft.com/kb/924373. Although the KB doesn’t have Forefront TMG 2010 listed, the same approach applies to TMG 2010 (yes, we will update the KB).

  • Yuri Diogenes's Blog

    Unable to Access a Published Apache Server behind ISA Server using SSL

    • 1 Comments

    Consider a scenario where you are publishing a third party web server, in this case an Apache Server that uses HTTPS through ISA Server 2006. Randomly the site doesn’t work, clients are unable to access it and when this happens the publishing rule test button shows the result below (error 0x80090326):

    SSLEncryptAlert_ErroronISA

    Notice that this error talks about a server certificate error, so clearly it is something during the SSL process.

     

    Reviewing the Data

     

    Using ISA Data Packager in repro mode (web proxy / web publishing template) was possible to collect simultaneous netmon traces from both NICs (internal and external). During those traces it was possible to see that the SSL handshake on the external interface using the certificate that was bound to the Web Listener on ISA was working just fine. Reviewing the SSL Handshake on the internal interface, while ISA was negotiating with the published server (Apache) we had a failure. Here it is the moment of the failure:

     

    ISA     APACHE    TCP   TCP:Flags=......S., SrcPort=24433, DstPort=443, PayloadLen=0, Seq=3108278462, Ack=0, Win=65535 (  ) = 65535

    APACHE    ISA     TCP   TCP:Flags=...A..S., SrcPort=443, DstPort=24433, PayloadLen=0, Seq=2120534540, Ack=3108278463, Win=5840 ( Scale factor not supported ) = 5840

    ISA     APACHE    TCP   TCP: Flags=...A...., SrcPort=24433, DstPort=443, PayloadLen=0, Seq=3108278463, Ack=2120534541, Win=65535 (scale factor 0x0) = 65535

     

    After finishing the TCP Handshake they start the SSL handshake and this is done by ISA sending the SSL Client Hello as shown below:

     

    ISA     APACHE TLS   TLS:TLS Rec Layer-1 HandShake

    TLSSSLData: Transport Layer Security (TLS) Payload Data

    - TLS: TLS Rec Layer-1 HandShake

      - TlsRecordLayer: TLS Rec Layer-1 HandShake

         ContentType: HandShake

       - Version: TLS 1.0

          Major: 3 (0x3)

          Minor: 1 (0x1)

         Length: 88 (0x58)

       - SSLHandshake: SSL HandShake ClientHello(0x01)

          HandShakeType: ClientHello(0x01)

          Length: 84 (0x54)

        - ClientHello: TLS 1.0

         + Version: TLS 1.0

         + RandomBytes:

           SessionIDLength: 16 (0x10)

           SessionID: Binary Large Object (16 Bytes)

           CipherSuitesLength: 22

         + TLSCipherSuites: TLS_RSA_WITH_RC4_128_MD5                { 0x00,0x04 }

         + TLSCipherSuites: TLS_RSA_WITH_RC4_128_SHA                { 0x00,0x05 }

         + TLSCipherSuites: TLS_RSA_WITH_3DES_EDE_CBC_SHA           { 0x00,0x0A }

         + TLSCipherSuites: TLS_RSA_WITH_DES_CBC_SHA                { 0x00,0x09 }

         + TLSCipherSuites: TLS_NTRU_NSS_WITH_AES_256_CBC_SHA       { 0x00, 0x64 }

         + TLSCipherSuites: TLS_NTRU_NSS_WITH_3DES_EDE_CBC_SHA      { 0x00, 0x62 }

         + TLSCipherSuites: TLS_RSA_EXPORT_WITH_RC4_40_MD5          { 0x00,0x03 }

         + TLSCipherSuites: TLS_RSA_EXPORT_WITH_RC2_CBC_40_MD5      { 0x00,0x06 }

         + TLSCipherSuites: TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA        { 0x00,0x13 }

         + TLSCipherSuites: TLS_DHE_DSS_WITH_DES_CBC_SHA             { 0x00,0x12 }

         + TLSCipherSuites: TLS_NTRU_NSS_WITH_AES_128_CBC_SHA       { 0x00, 0x63 }

           CompressionMethodsLength: 1 (0x1)

           CompressionMethods: 0 (0x0)

           ExtensionsLength: 5 (0x5)

     

    Right after that Apache sends a SSL Encrypt Alert error as shown below:

     

    APACHE      ISA     TLS   TLS:TLS Rec Layer-1 Encrypted Alert

      TLSSSLData: Transport Layer Security (TLS) Payload Data

    - TLS: TLS Rec Layer-1 Encrypted Alert

      - TlsRecordLayer: TLS Rec Layer-1 Encrypted Alert

         ContentType: Encrypted Alert

       - Version: TLS 1.0

          Major: 3 (0x3)

          Minor: 1 (0x1)

         Length: 2 (0x2)

         EncryptedData: Binary Large Object (2 Bytes)

    15 03 01 00 02 02 2F

    ....../

     

    By looking to the last two bytes of the hex value under Encrypted data we can find the meaning of the alert:

    o   2F in Hex = 47 in Decimal

    o   47 maps to illegal_parameter(47) error according to TLS RFC (http://www.ietf.org/rfc/rfc2246.txt?number=2246)

     

    Note: thanks to sudeepg for this nice approach reading SSL Encrypt Alert.

     

    Apache Server is saying that the TLS SSL Client Hello sent by ISA as an illegal parameter for this SSL negotiation.

     

    Conclusion

     

    This problem can happen because MS10-049 which is installed on ISA Server. As a temp workaround this update was removed and the issue got resolved on this particular scenario.  However, ultimately you should not remove this update; you should talk to the third party company web server admin and discuss the CVE-2009-3555 with him and how their product adequate for that. If you are publishing an IIS Server you might have this issue too, if you do, read the post below to see how to fix it:

    http://blogs.msdn.com/b/jpsanders/archive/2010/09/08/understanding-problems-with-ms10-049-kb-980436-and-ietf-rfc5746.aspx

     

     

  • Yuri Diogenes's Blog

    Software Update 2 for Forefront Threat Management Gateway (TMG) 2010 Service Pack 1 is now Available

    • 2 Comments

    Hi Folks, I just want to drop a quick note here about KB http://support.microsoft.com/kb/2433623/ that brings the list of the updates that are part of the new Software Update 2 for Forefront Threat Management Gateway (TMG) 2010 Service Pack 1.

    Enjoy it!!

  • Yuri Diogenes's Blog

    Hey DC, are you still there?

    • 0 Comments

    1. Introduction

     

    Yesterday I wrote about the “perfect storm” case where the root cause was disk bottleneck, today I would like to discuss a troubleshooting scenario for one of the 5 core scenarios that was presented yesterday, which is called “authentication”. Again, the symptom was exactly the same: ISA Server stops responding requests in random times of the day.  

     

    2. Data Gathering

     

    Use the same approach as was presented on yesterday’s blog, in addition, enable netlogon log while preparing the ISA Data Packager, it will be very useful in this type of scenario.

     

    3. Data Analysis

     

    Today I’m going to start the analysis by reviewing Perfmon and for such scenario I like to review some core counters:

     

    ·         Physical Disk\Average Disk Queue Length: to make sure that we don’t have disk bottleneck like yesterday’s blog.

    ·         Firewall Packet Engine\Backlogged Packets: you know, if this guy start growing substantially we have strong indications of authentication or DNS problem.

    ·         Web Proxy\Memory Pool for SSL Requests (%): you don’t want to see this value dropping, hitting zero and staying there. Usually when this occurs you will receive the event below:

     

    Log Name:      Application

    Source:        Microsoft Forefront TMG Web Proxy

    Event ID:      31212

    Task Category: None

    Level:         Warning

    Keywords:      Classic

    User:          N/A

    Description:

    The Forefront TMG Web Proxy memory pool that handles SSL connections is low. To specify a larger Web Proxy memory pool, set the ProxyVmemAlloc1pSize registry value in the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W3Proxy\Parameters registry key.

     

    Note: to address this issue, follow http://support.microsoft.com/kb/842438

     

    Those are the core counters, there are much more to look at and cross the data with the rest of the dataset that you might have it. You can see a full list of counters and what to expect on this post here from Tom Shinder.

     

    In this particular case the only counter that was presenting a suspicious (very suspicious) behavior was Backlogged Packets, as you can see below:

    image

     

    Notice the black line, how it goes from zero to 283..wow, that’s huge for sure. Remember that we should not be higher than 10 on this counter. Now, let’s review the dump file to see if the issue showed on perfmon is confirmed via the dump of the wspsrv.exe process:

     

    First step was to check if there was any thread locked in Critical Section, which in this case we haven’t. Since we haven’t then it will be interesting to know the information about the time consumed by each thread. To do that we use the runaway command as show below (just the top five):

     

    0:000> !runaway

     User Mode Time

      Thread       Time

      36:d9c       0 days 0:00:07.734

      30:d78       0 days 0:00:06.781

      37:da0       0 days 0:00:06.703

      21:d24       0 days 0:00:06.390

      29:d74       0 days 0:00:06.171

     

    Let’s take a look on what the first thread (36) is doing:

     

    0:000> ~36kb

    ChildEBP RetAddr  Args to Child             

    29e5ece8 7c827d29 77e61d1e 00004f89 00000000 ntdll!KiFastSystemCallRet

    29e5ecec 77e61d1e 00004f89 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc

    29e5ed5c 77c6a927 00004f89 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac

    29e5ed78 77c69cbf 2a50ca04 00000000 ffffffff rpcrt4!UTIL_WaitForSyncIO+0x20

    29e5ed9c 77c6a9fd 2a50c9cc 2a50ca04 29e5ede0 rpcrt4!UTIL_GetOverlappedResultEx+0x1d

    29e5edb8 77c6a9bb 2a50c9cc 2a50ca04 29e5ede0 rpcrt4!UTIL_GetOverlappedResult+0x17

    29e5edd8 77c69517 00000400 00000074 299626d0 rpcrt4!NMP_SyncSendRecv+0x73

    29e5ee00 77c6972f 00000000 299626d0 00000074 rpcrt4!OSF_CCONNECTION::TransSendReceive+0x7d

    29e5ee88 77c6969c 299626d0 2a5c9308 00000001 rpcrt4!OSF_CCONNECTION::SendFragment+0x2ae

    29e5eee0 77c69842 00000000 0000005c 29e5ef24 rpcrt4!OSF_CCALL::SendNextFragment+0x1e2

    29e5ef28 77c69aba 29e5efb0 29e5ef6c 00000001 rpcrt4!OSF_CCALL::FastSendReceive+0x148

    29e5ef44 77c69a3d 29e5efb0 29e5ef6c 29e5efb0 rpcrt4!OSF_CCALL::SendReceiveHelper+0x5b

    29e5ef74 77c7feb0 299626e8 29e5ef94 77c80845 rpcrt4!OSF_CCALL::SendReceive+0x41

    29e5ef80 77c80845 29e5efb0 7d1f7690 29e5f3a4 rpcrt4!I_RpcSendReceive+0x24

    29e5ef94 77ce415a 29e5efdc 29962744 00000000 rpcrt4!NdrSendReceive+0x2b

    29e5f384 7d1fac12 7d1f7690 7d1f998e 29e5f3a4 rpcrt4!NdrClientCall2+0x22e

    29e5f39c 7d1fadd3 2a408880 29e5f3d0 29e5f4c0 advapi32!LsarLookupSids2+0x1c

    29e5f41c 7d1fac5f 2a408880 00000001 29e5f4d0 advapi32!LsaICLookupSids+0xbd

    29e5f45c 7d1faf65 2a408880 00000001 29e5f4d0 advapi32!LsaLookupSids+0x41

    29e5f4c4 7d1faee6 00000000 275ce740 00000000 advapi32!LookupAccountSid+0x8b

     

    Notice (bottom to top) that we start this thread by retrieving the name of the account for this SID and the name of the first domain on which this SID is found using the function LookupAccountSid. After that we move to LsaLookupSids function, which looks up the names that correspond to an array of security identifiers (SIDs). This lookup process still going on until the Firewall Service (wspsrv.exe) leverages the function NdrClientCall2 to make a client side call using RPC. This RPC call keeps going back and forth until RPC gets stuck waiting for something.

     

    Note: for more information on how to debug RPC Stuck Call Problem read this article.

     

    By reviewing the other 4 threads on the top 5 threads from “runaway” output I notice that they all are doing the same type of operation and getting stuck on the same place. Here I already have enough data to say that we are hanging because we are waiting on authentication to come from the DC, but if you want to have the triple confirmation, open netlogon.log and you should see also events like these:

     

    [CRITICAL] CONTOSO : NlFinishApiClientSession: timeout call to \\dc01.contoso.com  Count: 1

    [CRITICAL] CONTOSO : NlpUserValidateHigher: denying access after status: 0xc0020017 1

    [SESSION] CONTOSO : NlSetStatusClientSession: Set connection status to c0020017

    [SESSION] CONTOSO : NlSetStatusClientSession: Unbind from server to \\dc01.contoso.com(TCP) 1.

     

    This keeps going on and on until we time out:

     

    [SESSION] CONTOSO : NlSetStatusClientSession: Unbind from server \\dc01.contoso.com (PIPE) 0.

    [SESSION] CONTOSO : NlSessionSetup: Session setup Failed

    [CRITICAL] I_NetLogonGetAuthData failed: (null) CONTOSO (Flags 0x1): 0xc000005e

    [LOGON] SamLogon: Network logon of CONTOSO\YuriDio from ISA02 Returns 0xC000005E

    [SESSION] I_NetLogonGetAuthData called: (null) CONTOSO (Flags 0x1) 

    [CRITICAL] I_NetLogonGetAuthData failed: (null) CONTOSO (Flags 0x1): 0xc000005e

     

    Notice that the error 0xC000005E means STATUS_NO_LOGON_SERVERS, which clearly states that we can’t get on hold of the DC for some reason.

     

    4. Conclusion

     

    In this case ISA/TMG performance was affected due the lack of response from the DC, in this particular case the DC was suffering with a bottleneck and causing authentication requests to queue up. Again, another good example of having the same symptom for a different root cause.  This is a very important concept to keep in mind: it is not because you have the same symptom that the root cause of the issue will be the same. Yesterday and today you have two posts where the symptom was the same but with a completely different root cause.

     

  • Yuri Diogenes's Blog

    We are all waiting for you Mr. Disk….are you there?

    • 0 Comments

    1. Introduction

     

    As more I deal with Performance issue on ISA Server (or TMG), more I realize that there are not really a lot of new things on this area to be explored. The reason why I say that there are not much of new things on this area is because I can easily map the top five core causes of ISA/TMG stop responding requests and causing the “server hanging” symptom, which are:

     

    ·         DNS – a wrong DNS configuration or a lack of response from the DNS Server can definitely cause issues on ISA. Please see the following related articles:

    http://blogs.technet.com/b/isablog/archive/2009/08/27/side-effects-of-incorrect-dns-configuration-on-isa-server-10060-connection-timeout-scenario.aspx

    http://blogs.technet.com/b/isablog/archive/2009/01/12/isa-server-2006-stops-answering-requests.aspx

     

    ·         Authentication – if the DC doesn’t answer, ISA can’t authenticate and as result new authentication requests will start to accumulate. The infamous 5783/5719 scenario is a good example of that.  Please see the following related articles:

    http://blogs.technet.com/b/yuridiogenes/archive/2008/06/05/isa-server-losing-secure-channel-with-the-dc-the-5783-nightmare.aspx

     

    ·         Logging – if we can’t log we will eventually stop responding. On ISA we go to lockdown mode, on TMG we start to write the LLQ files in the disk, which can fill up the disk and the server runs out of disk space, which will end up causing ISA/TMG stop responding.  Please see the following related articles:

    http://blogs.technet.com/b/yuridiogenes/archive/2008/08/06/intermittent-performance-problem-while-accessing-internet-through-isa-server-2006.aspx

     

    ·         Disk – this is one key element, because if we have disk bottleneck, everything else will fall apart. Please see the following related articles:

    http://blogs.technet.com/b/isablog/archive/2010/05/10/how-disk-bottleneck-can-affect-tmg-performance.aspx

     

    ·         Antivirus – well, yeah…this is true. There are many elements here that can go wrong, for example: some antivirus also introduces firewall modules and cause conflicts with ISA/TMG firewall kernel engine, which is something that I already explained in here.  Please see the following related articles:

    http://blogs.technet.com/b/isablog/archive/2008/03/11/isolating-problems-that-seems-to-be-related-to-the-isa-server-part-iii.aspx

    http://blogs.technet.com/b/yuridiogenes/archive/2009/07/18/isa-server-stop-answering-requests-and-firewall-service-hangs.aspx

     

    As you can see this is a long list and the scenario that I’m about to describe on this post is a combination of all elements above and I like to call it: the perfect storm. What’s the symptom? The usual: ISA Server stop responding request and to fix ISA Admin have to restart Firewall Service.

     

    2. Data Gathering

     

    On this type of scenario the most common action plan is to gather perfmon, dump of the wspsrv.exe process and ISA Data Packager in repro mode. Here are the core steps:

     

    a. Install ISABPA from www.isabpa.com

    b. Configure Performance Monitoring with the following objects:

     

    > ISA Server Firewall Packet Engine/*

    > ISA Server Firewall Service/*

    > ISA Server Web Proxy/*

    > Memory/*

    > Processor/*

    > Network Interface/*

    > Process/*

    > Physical Disk/*

    > Threads/*

     

    Note:  configure the maximum size file for 200MB, the refresh time to 15 seconds and configure Perfmon to stop when the log is full and create a new file (Schedule Tab).

     

    c. Install the DebugDiag (download from the link below):

    http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=28bd5941-c458-46f1-b24d-f60151d875a3

     

    When the issue happens, the following steps needs to be done in order to gather the correct data:

     

    a. Go to Start / Programs / Microsoft ISA Server / ISA Tools / ISA Data Packager

    b. On the option "Collect data using one of the following repro scenarios", select "Web Proxy and Web Publishing" and click Next;

    c. Click in Modify Options;

    d. In addition to the options that are already selected please select also:

    - ISA BPA

    - ISA Info

     

    e. Click in Start data collection

    f. The Data Packager will start to run. When the option "Press spacebar to start the capture" appears, press the spacebar and repro the issue by trying to connect from the client workstation.

    g. After you finishing testing then press space bar again in the ISA Data Packager console.

    h. When ISA Data Packager finishes collecting data, open DebugDiag.

    i. On the first Debugdiag screen (Select Rule Type) click cancel.

    j. Go to Processes tab and look for the wspsrv.exe process.

    l. While this window is open, go back to the workstation and try to connect again.

    m. While the workstation is trying to connect, go back to debugdiag window, right click on wspsrv.exe process and choose Create Full User Dump.

    n. Stop Perfmon counter.

     

    3. Data Analysis

     

    Having the correct data in hands you can start looking for obvious issues on the ISA BPA report. If there is nothing there that is relevant for this type of issue, move forward to analyze the perfmon or dump. In this particular case I’m going to review the dump first.

     

    First step was to check if there was any thread locked in Critical Section:

     

    0:000> !cs -l

    -----------------------------------------

    DebugInfo          = 0x000c0fe8

    Critical section   = 0x011c9024 (+0x11C9024)

    LOCKED

    LockCount          = 0x0

    WaiterWoken        = No

    OwningThread       = 0x0000113c

    RecursionCount     = 0x1

    LockSemaphore      = 0x0

    SpinCount          = 0x00000000

    -----------------------------------------

     

    Notice that this thread is locked and we have the address of the owning thread; let’s see which thread is that:

     

      45  Id: 1c38.113c Suspend: 1 Teb: 7ffa0000 Unfrozen

    ChildEBP RetAddr  Args to Child             

    36d8f354 7c827d29 77e61d1e 00000d28 00000000 ntdll!KiFastSystemCallRet

    36d8f358 77e61d1e 00000d28 00000000 36d8f39c ntdll!ZwWaitForSingleObject+0xc

    36d8f3c8 77e61c8d 00000d28 00004e20 00000000 kernel32!WaitForSingleObjectEx+0xac

    36d8f3dc 74cd2e3f 00000d28 00004e20 00004e20 kernel32!WaitForSingleObject+0x12

    36d8f408 6d56ddde 002fb5b0 0138d2a0 00000009 DBmsLPCn!ConnectionRead+0xaf

    36d8f428 6d5687fc 0138f2c0 0138d2a0 00000009 dbnetlib!WrapperRead+0x2c

    36d8f480 4e2597ce 0138f2c0 0138d2a0 0138d2a0 dbnetlib!ConnectionRead+0x519

    36d8f4b4 4e25982d 0138f2c0 0138d2a0 00000009 sqloledb!CDataSource::ConnectionRead+0x35

    36d8f500 4e252358 0138d06e 00000001 00000000 sqloledb!CDBConnection::GetBytes+0x269

    36d8f54c 4e2555c4 011da180 00000088 0000001e sqloledb!CDBConnection::ProcessTDSStream+0x157

    36d8f608 4e255691 011c5680 0000003d 011fb198 sqloledb!CStmt::ExecDirect+0x786

    36d8f620 4e254d32 011c5680 0000003d 00000000 sqloledb!CStmt::SQLExecDirect+0x28

    36d8f650 4e25517d 00000000 4e25321c 0000003d sqloledb!CCommand::ExecuteHelper+0x157

    36d8f6d4 4e254c4b 011d4888 00000000 4bbea778 sqloledb!CCommand::Execute+0x76b

    36d8f70c 4bbea64d 011e8550 00000000 4bbea778 sqloledb!CImpICommandText::Execute+0xdd

    36d8f74c 4bc0c79b 011c8d78 011fb22c 011f8738 msado15!CConnection::Execute+0x9d

    36d8f91c 4bbea4a7 011cfed8 00000000 011d16c8 msado15!_ExecuteAsync+0x19f

    36d8f930 4bbea385 011cfed8 ffffffff 00000000 msado15!ExecuteAsync+0x23

    36d8fa18 4bbea258 00000000 7c828200 00000000 msado15!CQuery::Execute+0xa5e

    36d8fa84 4bc21717 011d16c8 00000000 7c828200 msado15!CCommand::_Execute+0x153

     

    The yellow line in the second stack shows that the machine is submitting a SQL statement using the SQLExecDirect function. Now let’s see what SQL command is being executed:

     

    0:000> du 011c5680

    011c5680  "SELECT RTRIM(filename) FROM ISAL"

    011c56c0  "OG_20101107_FWS_000..sysfiles"

     

    Logs starting with FWS suffix represent the Firewall log; in this case ISA was querying the SQL database for this log. Now where is SQL located? According to ISAInfo collected by ISA Data Packager the Log was located on the D: drive, which was actually part of the same disk as C:, only in a different partition. Now it is time to review perfmon and see if we can match this with something going on from the disk perspective.

    Here it is sample of the time where the issue was happening:

    image

     

     

    The black line represents the Average Disk Queue Length that goes from zero to 26 (maximum should be 2 per spindle - in this case we just have 1 spindle) and got stuck there from 1:36PM to 1:39PM. During the same time we see the ISA Server Firewall Packet Engine\Backlogged Packets goes from zero to 113 (maximum should never be higher than 10). The logic here is the following:

    1.       ISA is trying to query a firewall log located on the SQL (MSDE in this case) database. ISA is waiting on SQL.

    2.       SQL is performing a reading operation for a piece of information located in disk. SQL is waiting on Disk.

    3.       Disk is having bottleneck and it is queuing up requests.

    4.       Since ISA can’t proceed (since is waiting on disk), ISA starts to accumulate requests (backlog starts to grow). ISA stop answering new requests.

    5.       Clients can’t browse.

     

    You might be thinking, but this is only for 3 minutes, I can live with that. Really? I doubt your helpdesk will not overflow of calls if nobody can browse Internet for 3 minutes.

     

    4. Conclusion

     

    I hope this post gives you a big picture of what goes behind an ISA (or TMG) performance issue in scenarios where ISA/TMG stops responding. There are much more elements that needs to be investigated other than ISA/TMG itself.

  • Yuri Diogenes's Blog

    Security baselines for Windows Server® 2008 R2 and Microsoft Office 2010 - setting packs for Windows® 7 and Windows® Internet Explorer® 8

    • 0 Comments

    The Microsoft Solution Accelerators team is pleased to announce the release of new resources that you can use in combination with the Microsoft Security Compliance Manager tool: the Windows Server 2008 R2 Security Baseline and the Office 2010 Security Baseline, and setting packs for Windows 7 and Internet Explorer 8.

    SCM graphic

    Together with the tool, these resources are designed to help organizations efficiently manage the security and compliance process for some of the most widely used Microsoft products.

    • The highly-anticipated security baselines for Windows Server 2008 R2 and Microsoft Office 2010 provide you with free Microsoft-recommended solutions to meet today's security challenges. In combination with best-practice guidance and the Security Compliance Manager tool, the baselines are designed to help you plan, deploy, and monitor the security of computers running Windows Server 2008 R2 and of Office 2010 applications. Both releases also include a setting pack (for Windows Server 2008 R2 and Office 2010, respectively) enabling you to define baselines that include settings outside the scope of the security baselines from Microsoft.
    • The Windows 7 and Windows Internet Explorer 8 setting packs, in combination with the Security Compliance Manager tool, will enable you to define baselines that include settings outside the scope of the security baselines from Microsoft. Use these new resources to define custom baselines, meet business-critical needs, and elevate the security of Windows 7 and Internet Explorer 8.

    The Security Compliance Manager works with the Microsoft Assessment and Planning (MAP) Toolkit and the Microsoft Deployment Toolkit (MDT) to help you plan, securely deploy, and manage new Microsoft technologies—easier, faster, and at less cost. Learn more.

    Next steps:

    First, learn more about the Security Compliance Manager tool. Next, learn more about the new security baselines and setting packs:

    Download the tool:

    • New users can access these resources by visiting the Microsoft Download Center to download the Security Compliance Manager tool.
    • Existing users can access the baseline and setting packs in the tool by clicking the Tools menu, and then clicking Check for Baselines.
    • Help spread the word: Tell your friends about these new security resources and the Security Compliance Manager tool.
    • Questions? Comments? Feedback? Tell it to the development team.
  • Yuri Diogenes's Blog

    TMG 2010 Overview for ISA Admins

    • 1 Comments

    Last week I delivered a level 200 presentation for a customer about TMG, a brief overview of TMG in a Secure Web Gateway Scenario and some demos (which are not available here, but I’m working on the videos to share it here). The goal of this presentation is to introduce TMG for ISA administrators of for new edge administrators, the agenda includes:

    • Business Needs and Desktop Security Challenges
    • TMG Value Proposition
    • TMG Feature Drill Down
    • Comparing ISA with TMG
    • Protection Against Web-based Threats
    • Integration with Exchange Edge and FPE

    Enjoy it !!

  • Yuri Diogenes's Blog

    Know more about the New Microsoft Press Book Deploying Microsoft Forefront Unified Access Gateway 2010

    • 0 Comments

    Today Microsoft Press published on their blog an overview of what this new book contains and how it is organized. For more information take a look at:

    http://blogs.msdn.com/b/microsoft_press/archive/2010/11/03/new-book-deploying-microsoft-forefront-unified-access-gateway-2010.aspx

    I also hope you enjoy it.

Page 6 of 17 (422 items) «45678»