Today ISSA released the ISSA Journal – September 2009 issue that contains an article that I wrote about unified threat management.

You can view the online version at:

https://www.issa.org/Library/Journals/2009/September/ISSA%20Journal%20September%202009.pdf

I was reading the Windows IT Pro Magazine of this month (September 2009) and there I found a nice article written by an Escalation Engineer here from Microsoft Texas (Michael Morales) where he describes how to use ProcDump to catch high CPU utilization. This is an amazing tool that can also help ISA Administrators, mainly for scenarios where we just can’t get the right data (most case dumps) because the issue is random and when it happens there is nobody available to execute a command (for example: launch DebugDiag and choose the option for manual dump the process).

For an ISA Server high CPU utilization scenario a simple example will be dump out the Firewall Service process two times when the CPU for wspsrv.exe is at or exceeds 90 percent for 5 seconds and store the dumps in the c:\dumps folder:

c:\procdump.exe -c 90 -s 5 -n 2 wspsrv.exe c:\dumps

Isn’t that cool?

Make sure to read the article from Michael Morales to fully understand how this tool works:

http://windowsitpro.com/article/articleid/102479/got-high-cpu-usage-problems-procdump-em.html

1. Introduction

When question that I always receive when working with Firewall Service crashing is: why is it crashing? When the answer is: due a third party application…then the next question is: how is that? I thought each process was running independently and one couldn’t crash the other, right? That’s correct; however you need to remember how things work on ISA core architecture. Let’s step back and review the following diagram:

Figure 1 – ISA Server architecture (from ISA Server 2006 Firewall Core Document)

Notice that Firewall Service (wspsrv.exe) runs in User Mode while Firewall Engine (fweng.sys) runs in kernel mode. While is true that each process has its own address space, security token, etc, it is also true that each process is composed by threads, where each thread can be executing a different set of instructions and interacting with different components. ISA Server 2006 allows third party application to build their proprietary Web Filter (ISA Server supports ISAPI filter development) and by doing so it will somehow interfere in the way that Web Proxy Filter acts by default.

2. Digging in

If you use Process Explorer (or ProcMon) to open the properties of wspsrv.exe process you will see that there are many threads in execution as shown Figure 2:

Figure 2 – Threads running in the context of wspsrv.exe process.

If you select one of those threads and click Stack you will see the stack content and the modules in use. A stack is a region of the memory that is used to temporarily store data; it is added and removed in a last-in-first-out base. When you choose the thread and click on the stack you can see what it is in execution on that thread on that moment. Having this foundation understanding let’s take a look in the following diagram to understand how wspsrv.exe process can be affected by a third party filter:

Figure 3 – Firewall service process and the threads that belongs to it.

As you can see in this diagram there are some threads within the wspsrv.exe process and I’m using the stack of two of them as example. First stack from thread 1988 just have Microsoft modules and for the purpose of this example let’s focus on the stack that belongs to the thread 1920 which has a third party module (MyWebFilter.dll) loaded into it.

If this module for some reason execute an operation that cause an unhandled exception we might compromised the whole thread and possible crash the process. If you do not have a debugger attached to the process you will not get a dump for the wspsrv.exe, the only thing that will happen is that Firewall Service will crash (process quits from the memory) and an event is registered in the event viewer saying that the Firewall Service crashed. If you want to catch this type of crash you need a debugger attached to the process, to do that you can use an article that I wrote some time back about that, check it out here.

3. Access Violation

For the purpose of this example let's assume that this fake third party filter module did cause Firewall Service to crash and since I did have DebugDiag attached to wspsrv.exe process I was able to catch the second chance crash. In this case here it is the result for this crash by a partial output from !analyze –v command:

0:040> !analyze -v

*******************************************************************************

*                                                                             *

*                        Exception Analysis                                   *

*                                                                             *

*******************************************************************************

FAULTING_IP:

ntdll!KiUserExceptionDispatcher+e

7c82857e 0ac0            or      al,al

EXCEPTION_RECORD:  102cf8cc -- (.exr 0x102cf8cc)

ExceptionAddress: 10161a50 (MyWebFilter.dll+0x00001a50)

   ExceptionCode: c0000005 (Access violation)

  ExceptionFlags: 00000000

NumberParameters: 2

   Parameter[0]: 00000000

   Parameter[1]: 10192068

Attempt to read from address 10192068

DEFAULT_BUCKET_ID: STATUS_STACKOVERFLOW

PROCESS_NAME:  wspsrv.exe

ERROR_CODE: (NTSTATUS) 0xc00000fd - A new guard page for the stack cannot be created.

READ_ADDRESS:  1016caac

NTGLOBALFLAG:  0

APPLICATION_VERIFIER_FLAGS:  0

IP_MODULE_UNLOADED:

MyFilter+1a50

10161a50 ??              ???

CONTEXT:  102cf8e8 -- (.cxr 0x102cf8e8)

eax=102cfe44 ebx=00000000 ecx=10192048 edx=f9b10046 esi=10192048 edi=102cfe38

eip=10161a50 esp=102cfbb4 ebp=102cfbdc iopl=0         nv up ei pl nz na pe nc

cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010206

MyWebFilter.dll+0x1a50:

10161a50 ??              ???

Resetting default scope

RECURRING_STACK: From frames 0x7 to 0xa

LAST_CONTROL_TRANSFER:  from 102cfe38 to 10161a50

IP_ON_STACK:

+102cfe38

102cfe38 5c              pop     esp

FRAME_ONE_INVALID: 1

STACK_TEXT: 

WARNING: Frame IP not in any known module. Following frames may be wrong.

102cfbb0 102cfe38 00000006 00000000 10192048 MyWebFilter.dll+0x1a50

102cfc2c 776bf813 77796898 00327277 00000000 0x102cfe38

102cfc30 77796898 00327277 00000000 00000000 ole32!COleStaticMutexSem::Release+0x1a

102cfe6c 7c83ac6c 00000000 0efd5400 0efd54a8 ole32!gComLock+0x18

102cfec8 7c83ca92 62251dc0 00000000 0efd5400 ntdll!RtlpWaitOrTimerCallout+0x74

102cfeec 7c83a857 0efd54a8 7c88b080 0ef88528 ntdll!RtlpAsyncWaitCallbackCompletion+0x37

102cff44 7c83aa3b 7c83ca5b 0efd54a8 00000000 ntdll!RtlpWorkerCallout+0x71

102cff64 7c83aab2 00000000 0efd54a8 0ef88528 ntdll!RtlpExecuteWorkerRequest+0x4f

102cff78 7c839f90 7c83a9fa 00000000 0efd54a8 ntdll!RtlpApcCallout+0x11

102cffb8 77e6482f 00000000 00000000 00000000 ntdll!RtlpWorkerThread+0x61

102cffec 00000000 7c839f2b 00000000 00000000 kernel32!BaseThreadStart+0x34

Let see our registers:

0:040> r

eax=00000000 ebx=00000000 ecx=1016caac edx=7c828786 esi=00000000 edi=00000000

eip=1016caac esp=102911b0 ebp=102911d0 iopl=0         nv up ei pl zr na pe nc

cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010246

MyWebFilter.dll+0xcaac:

1016caac ??              ???

Now let’s look at the EIP register (which points to where in the program the processor was currently executing the code):

0:040> r eip

eip=1016caac

Let’s dump it:

0:040> dd eip

1016caac  ???????? ???????? ???????? ????????

1016cabc  ???????? ???????? ???????? ????????

1016cacc  ???????? ???????? ???????? ????????

1016cadc  ???????? ???????? ???????? ????????

1016caec  ???????? ???????? ???????? ????????

1016cafc  ???????? ???????? ???????? ????????

1016cb0c  ???????? ???????? ???????? ????????

1016cb1c  ???????? ???????? ???????? ????????

Well, it doesn’t looks good since it is pointing to a bunch of question mark (either invalid or not accessible memory). Let’s see what memory address EIP was pointing to:

0:040> !address eip

    10160000 : 10160000 - 00040000

                    Type     00000000

                    Protect  00000001 PAGE_NOACCESS

                    State    00010000 MEM_FREE

                    Usage    RegionUsageFree

What this PAGE_NOACCESS means? Let’s see the definition from MSDN:

One strong hypothesis here (since we don’t have the code for the third party application to debug) is that this module tried to access an invalid memory address and therefore corrupted the stack causing the access violation. This was enough to cause the whole process (wspsrv.exe) to crash.

I already talked about Network Monitor Wizard (AKA NetWiz) here some months ago and this is an ongoing project that I have with some friends at Microsoft. Since our original code (written in C#) had some third party components we couldn’t really publish at CodePlex, therefore we completely re-wrote NetWiz with our own code (still in C#) and now we are making it available at http://netwiz.codeplex.com. The source code is also available at http://netwiz.codeplex.com/Release/ProjectReleases.aspx?ReleaseId=31511 and you are free to suggest improvements on this tool.

When I was system administrator back in 1999 one of the products that I used to work with on the daily basis was Exchange 5.5, I still remember the main components of the network that I used to manage: Windows NT 4, Proxy 2 and Exchange 5.5. I worked with Exchange Server from the version 5.0 to 2003, when 2007 was launched I was already 100% focus on ISA and didn’t have time to explore some features besides the integration with ISA Server. Now with TMG I will have a chance to start dealing with Exchange again to make sure that the SMTP Protection is correctly configured and also to troubleshooting possible mailflow issues (really fun!).

The new Email Protection on TMG is based on Exchange Edge and Forefront Security for Exchange and yesterday we just released a new article for the Tales from the Edge community site that has more details about how this works. Check it out at: http://technet.microsoft.com/en-us/library/ee338733.aspx

Sounds familiar? Interesting though is that in this case it was not an ISA issue (really??? J) and was actually fixed using the solution below from IE Team:

http://blogs.msdn.com/askie/archive/2009/07/17/slow-performance-in-internet-explorer-8-after-installing-the-skype-v4-1-application.aspx

One more issue that is not caused by ISA Server…and counting.

Although ISA Server 2006 has a great Sharepoint Wizard to configure Sharepoint/MOSS Publishing rule sometimes what was supposed to be simple can became a nightmare. Most of the times ISA gets beat first since it is the device that faces Internet and users from outside are the ones that are complaining about. However if there are so many elements that can cause issue of this nature that simply stick in mind that ISA is the culprit is just not realistic.

Recently a friend of mine from MOSS Team wrote an article that fixed an issue where has all the ingredients to make you think that ISA was causing the issue. The problem started because something call disk-based cache, AKA BLOB (see this article for more info). Here it is the scenario and when you read NLB, think about ISA NLB using Web Farm to publish MOSS with a configuration similar to this one for the Web Farm:

“One probable and common scenario is that on which you have multiples Web Front End (WFEs) and they are reached via Network Load Balancer (NLB). You did not add any host header when you created the web application and yet you can access the web application using both the local server name (eg. MOSS-WFE01) or the DNS entry (eg. http://myportal.contoso.com). However, when you add blob cache functionality to the site (see how to do this here), the cache does not seem to (and actually does not) work.”

Read more at http://blogs.msdn.com/rodneyviana/archive/2009/06/24/blobcache-will-not-work-if-the-request-url-is-not-in-the-alternate-access-mapping-list.aspx to fix this problem.

Note: another tip to avoid making tests while the MOSS content is in cache (IIS Cache not ISA) is to use the MOSS Support Explorer (also created by Rodney Viana from MOSS Team).

1. Introduction

One of the most painful issues to resolve on ISA Server is when the Firewall Service stops and doesn’t come up again. Many times this happens without a previous warning and most of the times is because ISA is failing to load something or to commit some kind of configuration that was made. In this particular scenario, Firewall administrator claims that he didn’t change anything and believe or not he didn’t. ISA Server was untouchable for months and one day, after installing a security patch on Windows and restart the server, Firewall Service didn’t start.

In situations like that is easy to blame the patch, because the first thing that comes in people’s mind is: well, if it was working and after installing a patch stopped working, it got be the patch. Although this makes sense (logically speaking) it might not be true (technically speaking). This particular case confirmed that: after firewall administrator uninstalled the patch (not really a good security recommendation) the issue persisted.

Let’s see the approach to fix this issue.

2. Starting from the Basics

Start from the simplest thing which is: review the event viewer. In this case here the sequence of events that I found it:

In this case these events are very generic and really don’t say much, but it gives us an idea of the sequence of failures that we have.

3. Going Further

On issues related to Firewall Service not starting, one thing that is very handy is understand what is happening during the time that Firewall Service is starting. Which files is he loading? To better see what is happening I used WinDBG to attach to Firewall Service. I did that on a working system to see the sequence that I have and repeated the same in the system that was broken. Here it is the steps that I used on my working system:

0. On the system that is working I stopped Firewall Service.

1. Open WinDBG (if you don’t have, download it here).

2. Started Firewall Service, open WinDBG, click on File / Attach to a Process, choose the wspsrv.exe process as shown below and click OK.

FGF TV just released this video that covers some highlights of the TMG presentation that I delivered in Brazil last month:

Why is it a bad request? If you haven’t seen this before, I see quiet often in some scenarios involving ISA Server and guess what: if I remove ISA Server from the picture it works just fine. Well, we all know that when you are wide open to Internet there is nobody inspecting your traffic anyway so you shall pass. Is the same thing as trying to get into your house without the key for the door, you can’t get in, but guess what, if you remove the door you can get in. Oh..so it got be the door the root cause of the problem, right? Better remove the door or look for your key?

In some scenarios when you receive the “400 Bad Request” is because is ISA Server is acting according to RFC 2616 and then sending the HTTP response to the client like this:

Here it is a piece of the HTTP header with a type of request that can be interpreted by ISA Server as invalid:

If the content type is multipart the HEX value might look like this:

The value 20 in HEX means space (see http://en.wikipedia.org/wiki/ASCII for more info), which makes perfect sense in this case since the next HTTP packet just continued with a bunch of more spaces at the end, which supposedly will extend to a third packet if ISA haven’t drop that:

Going back to the RFC 2616 (section 4.2) you will see that this is not a good practice and ISA Server does what is supposed to be done, drop the packet. The best way to fix that is to contact the web site administrator (or application developer) and ask him to fix that. However since in almost of all outbound access you don’t have control of the site that your internal client is trying to access, ISA Server has a workaround to accept that.  To workaround on this problem (since is not an ISA issue) you need should install ISA Server 2006 SP1 and after that make the registry changes suggested in the article below:

941293  Error message when you access a Web site through ISA Server 2006 or Microsoft Forefront Threat Management Gateway, Medium Business Edition: "HTTP 400 - Bad Request"

http://support.microsoft.com/default.aspx?scid=kb;EN-US;941293

1. Introduction

Error 64 can happen due many situations and I documented one of those situations last year and as you could see sometimes it is not easy to find out why this error happens. The issue that I’m about to describe here was identified while I was troubleshooting a third party application that uses TCP Port 80 to transmit files, but not using HTTP. What?? Yeah, I know. Although IANA has established port 80 for HTTP, anyone can create an application that uses port 80 to send whatever they want. This is fine, as long as you don’t try to use this application behind a Firewall that does application layer inspection and look to that traffic and say: what is that? This is not HTTP Protocol and it is using TCP Port 80…I shall block this traffic!

The firewall administrator was smart to understand that and what he did was, he created a custom protocol using port 80 and didn’t bind Web Proxy filtering to it. Fair enough, but doesn’t fully resolved this issue.

2. The Error

When the client (which had the 3^rd party application installed on his computer) started to transmit the file to the destination it received an error and didn’t proceed. Using Logging feature the Firewall Administrator saw the error below:

FGF University released a short version of the debate that we had on the University Live TV Program that was recorded last month when I was in Brazil. In this program we discussed many IT areas and the security concerns around those areas. The program (full version) will be live tomorrow (07/19) on Channel 14 (NET Brazil) and also Assembleia TV Channel (Brazil).

The problem that this post is going to discuss was related to a random issue where certain times of the day the ISA Server was stopping answering requests and when the firewall administrator tried to restart the firewall service the service didn’t start. The only event that we have prior to the issue happens was the one below:

Doing a quick assessment I could see that the Antivirus was scanning all folders, including ISA Folders (not good at all). As a troubleshooting step I disabled the AV but the issue persisted. Using ProcMon I could see that when ISA Storage process (ISAStg.exe) was trying to read a value in register the AV filter drive was still present in kernel mode and intercepting the request.  Here it is the sequence:

ISASTG process:

The stack for this process shows the AV filter drive (klif.sys):

Later on we fail to create the file:

We uninstalled the AV and the issue didn’t happen anymore. Since his environment had a requirement to have AV installed on ever single Windows machine we implemented the correct folder exclusion following the article “Considerations when using antivirus software on ISA Server” and the environment got stabilized.

Interesting side of this story is that this article was published exactly one year ago, one year later we still have firewall administrators not following such recommendation and therefore having unexpected downtimes.

FGF TV Channel just released the online version of the interview (in Portuguese) that I recorded last week about information security and also about TMG Book.

Couple of days ago I was assisting a friend to troubleshoot the infamous 5783 that was causing the authentication prompt issue that we all know about. In this case the problem was happening throughout the night, which was even odder because during the day when the traffic was really high the issue wasn’t happen. The employees on the third shift (which was no more than 20) were receiving authentication prompts randomly.

The question was: how to get data on this type of case? We don’t know what time it occurs and we don’t have IT people on that time to collect data. We installed a tool called Port Reporter tool that runs as a service and collects pretty much all the information about process and which port is using during that time. Read http://support.microsoft.com/kb/837243 for more information on how to use this amazing tool.

It boils down that the issue was a piece of malware on those workstations that were sending tons of request to an external URL and drastically affecting ISA Server’s performance.

I’m really happy for the invitation that I received from FGF (Faculdade Grande Fortaleza) Academic Director to deliver a presentation to the professors and students about information security emphasizing TMG role in the network protection. FGF is the university where I graduated and also where I was Professor back in 2003, there I taught two disciplines: Computer Networks and Operating System. It was an amazing time and I hope to see former students, coworkers and professors.

The presentation will be next Tuesday, June 30th, 6:40 PM at the university’s campus and on the same day I will be also participating in a TV Show (FGF TV Channel) to discuss about information security with other professors and students. For more information check the FGF’s web site at www.fgf.edu.br. A former student of mine also posted about the event in his web site: http://www.jorgebarata.eti.br/269.

Lately we had received some calls where ISA Server was not using the latest updates, which is fine although is not recommended. However when the subject is Service Pack then it might be a supportability blocker if ISA Server is not running within the supported Service Pack level. ISABPA does a great job in warning an ISA administrator that his ISA Server 2004 is not running with SP3 as shown below:

But today the issue is not only having the system with the latest update, is really a supportability matter. ISA Server 2004 SP2 is not supported since January 13th 2009 as shown in the table below:

From: http://support.microsoft.com/lifecycle/?p1=2108

Same applies to ISA Server 2006 RTM (without SP1), which the support will end July 14th 2009 as shown in the table below:

From: http://support.microsoft.com/lifecycle/?p1=11928

So if you are in an unsupported scenario (or about to get into this stage) make sure to plan your update as soon as possible to avoid supportability concerns when opening an incident with Microsoft CSS.

A friend of mine from Brazil (Paulo Oliveira) that already contributes a lot with ISA Community answering questions at www.isaserver.org has now his own blog. Paulo has a lot of skills on ISA Server and his contributions throughout the years demonstrate that. Take a look at Paulo’s blog at:

http://paulooliveirasilva.spaces.live.com/blog/

For all of you that were at SEBRAE - CE last Friday I would like to say THANK YOU VERY MUCH (MUITO OBRIGADO). Without your participation this event couldn’t happen. See you again next time.

1. Introduction

I was Proxy 2 administrator back in 1997 in a technology school, in 2000 I took my Proxy 2.0 exam and when ISA 2000 was released I was really like: WOW, that’s a huge change. It was indeed a great moving from a simple Proxy to a more robust Proxy with Firewall capabilities. But when I see TMG changes and I compare to ISA 2006, I have a great feeling that this is also a huge step towards an even better firewall with tremendous capabilities. There are so many good things on TMG that sometimes we overlook the hard work that the Product Team had to make the administration and management easier.

In this post I want really to emphasize some new features that are not related to security, but related to how a Firewall administrator’s experience was improved in this release.

2. More than a Getting Start Wizard

Do you know how many times I received a call where the firewall administrator was unable to make the basics? What basics? Allow secure web access for example. I had many situations in the past that creating a rule in the right manner was a nightmare for the administrator with less experience. The idea behind the getting start wizard is really to improve the administrator’s experience with the product and allow him to perform the essential configuration after install the product.

A buddy of mine (Daniel Mauser) from PFE (Premier Field Engineer) read my previous post about SSTP and sent me a note about his thoughts on the PKI side of the house (since is his specialty). The notes about the troubleshooting and planning phase from my previous post are:

·         For troubleshooting purpose we can disable the CRL Check on the client side (not recommended in the production, as he said: only for troubleshooting purpose). To do that follow http://technet.microsoft.com/en-us/library/dd458982.aspx

·         The certificate that I created had URLs for LDAP and HTTP for the CRL. Since the client workstation review those links in that order (top down), the LDAP will be checked first, since it can’t access the LDAP path it will try the HTTP path. This can cause performance issue on the client side. Make sure to change the search order in the CA prior to issue the certificate, this way the CA will issue certificates using the HTTP first.

Thanks Daniel for those tips.

My friend Tom Shinder is inspired this month; he already posted some great info in his Blog the last two weeks. From his recently posts I personally recommend you to review the following ones:

http://blogs.isaserver.org/shinder/2009/06/14/the-directaccess-challenge-nat-traversal/

http://blogs.isaserver.org/shinder/2009/06/14/direct-access-versus-directaccess-know-the-difference/

Another friend of mine that is also helping out the community is Richard Hicks, last month he posted a great article about SQL Logging. Check it out here:

http://tmgblog.richardhicks.com/2009/05/29/remote-sql-logging-with-microsoft-isa-server-2006/

Last but not least you have to read this post from Jason Jones about ADAM, very precious piece of information:

http://blog.msfirewall.org.uk/2009/05/using-adam-sites-tool-with-isa-server.html

Figure 2 – First error due the cert name.

That was pretty self explanatory, but just to confirm the name that I used to issue the certificate I got a netmon trace and got the subject name:

To quick fix this I edited my host file and created a manual entry there. But then right after that I got:

Figure 3 – Now is the CRL.

Looking to the properties of the certificate it was possible to see that the CRL was poiting to my internal CA: 

When I first planned this presentation I was thinking to talk about Forefront TMG Beta 2 features, but now with TMG Beta 3 available, I’m also going to cover some of the cool features available in this release. We already have 150 people enrolled, so if you didn’t enroll better to do it quickly because the venue is almost full. Enroll at:

http://msevents.microsoft.com/CUI/EventDetail.aspx?EventID=1032416163&Culture=pt-BR