FGF TV just released this video that covers some highlights of the TMG presentation that I delivered in Brazil last month:

Why is it a bad request? If you haven’t seen this before, I see quiet often in some scenarios involving ISA Server and guess what: if I remove ISA Server from the picture it works just fine. Well, we all know that when you are wide open to Internet there is nobody inspecting your traffic anyway so you shall pass. Is the same thing as trying to get into your house without the key for the door, you can’t get in, but guess what, if you remove the door you can get in. Oh..so it got be the door the root cause of the problem, right? Better remove the door or look for your key?

In some scenarios when you receive the “400 Bad Request” is because is ISA Server is acting according to RFC 2616 and then sending the HTTP response to the client like this:

Here it is a piece of the HTTP header with a type of request that can be interpreted by ISA Server as invalid:

If the content type is multipart the HEX value might look like this:

The value 20 in HEX means space (see http://en.wikipedia.org/wiki/ASCII for more info), which makes perfect sense in this case since the next HTTP packet just continued with a bunch of more spaces at the end, which supposedly will extend to a third packet if ISA haven’t drop that:

Going back to the RFC 2616 (section 4.2) you will see that this is not a good practice and ISA Server does what is supposed to be done, drop the packet. The best way to fix that is to contact the web site administrator (or application developer) and ask him to fix that. However since in almost of all outbound access you don’t have control of the site that your internal client is trying to access, ISA Server has a workaround to accept that.  To workaround on this problem (since is not an ISA issue) you need should install ISA Server 2006 SP1 and after that make the registry changes suggested in the article below:

941293  Error message when you access a Web site through ISA Server 2006 or Microsoft Forefront Threat Management Gateway, Medium Business Edition: "HTTP 400 - Bad Request"

http://support.microsoft.com/default.aspx?scid=kb;EN-US;941293

1. Introduction

Error 64 can happen due many situations and I documented one of those situations last year and as you could see sometimes it is not easy to find out why this error happens. The issue that I’m about to describe here was identified while I was troubleshooting a third party application that uses TCP Port 80 to transmit files, but not using HTTP. What?? Yeah, I know. Although IANA has established port 80 for HTTP, anyone can create an application that uses port 80 to send whatever they want. This is fine, as long as you don’t try to use this application behind a Firewall that does application layer inspection and look to that traffic and say: what is that? This is not HTTP Protocol and it is using TCP Port 80…I shall block this traffic!

The firewall administrator was smart to understand that and what he did was, he created a custom protocol using port 80 and didn’t bind Web Proxy filtering to it. Fair enough, but doesn’t fully resolved this issue.

2. The Error

When the client (which had the 3^rd party application installed on his computer) started to transmit the file to the destination it received an error and didn’t proceed. Using Logging feature the Firewall Administrator saw the error below:

FGF University released a short version of the debate that we had on the University Live TV Program that was recorded last month when I was in Brazil. In this program we discussed many IT areas and the security concerns around those areas. The program (full version) will be live tomorrow (07/19) on Channel 14 (NET Brazil) and also Assembleia TV Channel (Brazil).

The problem that this post is going to discuss was related to a random issue where certain times of the day the ISA Server was stopping answering requests and when the firewall administrator tried to restart the firewall service the service didn’t start. The only event that we have prior to the issue happens was the one below:

Doing a quick assessment I could see that the Antivirus was scanning all folders, including ISA Folders (not good at all). As a troubleshooting step I disabled the AV but the issue persisted. Using ProcMon I could see that when ISA Storage process (ISAStg.exe) was trying to read a value in register the AV filter drive was still present in kernel mode and intercepting the request.  Here it is the sequence:

ISASTG process:

The stack for this process shows the AV filter drive (klif.sys):

Later on we fail to create the file:

We uninstalled the AV and the issue didn’t happen anymore. Since his environment had a requirement to have AV installed on ever single Windows machine we implemented the correct folder exclusion following the article “Considerations when using antivirus software on ISA Server” and the environment got stabilized.

Interesting side of this story is that this article was published exactly one year ago, one year later we still have firewall administrators not following such recommendation and therefore having unexpected downtimes.

FGF TV Channel just released the online version of the interview (in Portuguese) that I recorded last week about information security and also about TMG Book.

Couple of days ago I was assisting a friend to troubleshoot the infamous 5783 that was causing the authentication prompt issue that we all know about. In this case the problem was happening throughout the night, which was even odder because during the day when the traffic was really high the issue wasn’t happen. The employees on the third shift (which was no more than 20) were receiving authentication prompts randomly.

The question was: how to get data on this type of case? We don’t know what time it occurs and we don’t have IT people on that time to collect data. We installed a tool called Port Reporter tool that runs as a service and collects pretty much all the information about process and which port is using during that time. Read http://support.microsoft.com/kb/837243 for more information on how to use this amazing tool.

It boils down that the issue was a piece of malware on those workstations that were sending tons of request to an external URL and drastically affecting ISA Server’s performance.