The process flow is very known by all IT professionals: user can’t access a web site and calls the Help Desk. First contact, initial troubleshooting, can’t fix and it calls the network admin. At that point, after troubleshoot his “piece” of the puzzle, it says: well, it got be our firewall that is not allowing that.
This chain started before I received this call where the user couldn’t open a web site that has some pages in MHT format. The error message was: page cannot be displayed. This issue has all elements to point ISA as the culprit:
· Traffic doesn’t work from all workstations behind the ISA.
· Accessing the web site from the ISA Server itself it works.
2. Narrowing Down
There is a very interesting book about troubleshooting called The Art Of Systematic Troubleshooting by Shawn Pinnock and if you read this book you will understand that one of the key elements of the troubleshooting technique is: ask questions. This was what I did; we need to know more and the following questions were asked:
· Does this ever worked?
o Answer: No.
· Is this a new ISA Server installation?
o Answer: Yes.
· Does the issue happen from IE 6 and IE7?
o Answer: I don’t know about IE7 we just have IE6.
· Does the issue happen only for this web site or all web sites?
o Answer: I don’t know, I just tested from this one..
· Can you access http://www.yuridiogenes.com.br/mht/primeirolugar.mht ?
o Answer: Let me try…no, got the same problem.
· Do you have any other workstation with IE7 that you can try?
o Answer: no, we don’t have IE7 at all.
3. Gathering Data
If those answers in mind, I decided to move on and get a sample data from the ISA Logging and netmon trace from both ISA interfaces (internal and external). When I got the data I could that things were clear on the ISA Server side and that although everything points to be an ISA issue, at that point I started to believe that it was not. Here is way:
Diagnostic Logging was showing that the data was transferred to the host:
Figure 1 – Diagnostic Logging showing the traffic.
Netmon trace shows the data being transferred to the client:
10.20.20.202 10.20.20.2 TCP TCP: Flags=....A..., SrcPort=1152, DstPort=HTTP Alternate(8080), Len=0, Seq=2767807327, Ack=3247783843, Win=65535 (scale factor not found)
10.20.20.2 10.20.20.202 HTTP HTTP: HTTP Payload
10.20.20.202 10.20.20.2 TCP TCP: Flags=....A..., SrcPort=1152, DstPort=HTTP Alternate(8080), Len=0, Seq=2767807327, Ack=3247788297, Win=65535 (scale factor not found)
10.20.20.202 10.20.20.2 TCP TCP: Flags=....A..., SrcPort=1152, DstPort=HTTP Alternate(8080), Len=0, Seq=2767807327, Ack=3247795597, Win=65535 (scale factor not found)
What seems to be is that ISA transfers all the data and the client doesn’t render the page correctly. Very odd, but definitely it didn’t seems to be an ISA Server issue. Customer than realized that he could try to repeat the test from the DC, that has Windows Sever 2003 with IE7 (yeah, I know he said he didn't have IE7) and the result was: it worked !!
4. Now What?
If you don’t keep asking most likely you will not have the information that you need. When I saw that a click came in and I asked: did you deploy those workstations through an image or something? The answer was: yes, we did.
Now things start to make more sense, we have a common problem in the OS or IE level. The next approach was to use FileMon and see what we were trying to access when we were open that page and the answer was clear:
16:27:10 iexplore.exe:3600 QUERY INFORMATION C:\WINDOWS\System32\inetcomm.dll NOT FOUND Attributes: Error
16:27:10 iexplore.exe:3600 QUERY INFORMATION C:\Arquivos de programas\Internet Explorer\INETCOMM.DLL NOT FOUND Attributes: Error
16:27:10 iexplore.exe:3600 QUERY INFORMATION C:\WINDOWS\system32\INETCOMM.DLL NOT FOUND Attributes: Error
16:27:10 iexplore.exe:3600 QUERY INFORMATION C:\WINDOWS\system\INETCOMM.DLL NOT FOUND Attributes: Error
As you can see we were missing the inetcomm.dll and this file is used by both Outlook Express and Outlook. Guess what: all the workstation didn’t have Outlooks Express. Besides that the template workstation was tightly hardening by the security admin. But what this has to do with MHT? Here it is the missing dot:
The ability to save a Web page as a Web archive file is provided by the Inetcomm.dll file (the Microsoft Internet Messaging API file)...
What we did was manually copy the inetcomm.dll from another XP SP2 (mine, I actually sent to him) to the System32 folder, re-register the inetcomm.dll. After that we were able to access the web page successfully.
To prove the point that not having the Outlook Express was not really the issue, we did two tests:
· Got a workstation and uninstalled the Outlook Express from it. The file was still there and we were able to access the system.
· Renamed the inetcomm.dll to inetcomm.old and ran the sfc /scannow. The Windows File Protection asked for the CD and replaced the file into the %systemroot%\system32\dllcache, after that we manually copy to %systemroot%\system32. Fixing the issue.
This means that somehow this file was manually removed during the hardening process that was done in the template workstation.
I have no doubt that it is really good to see people concern about security, however after hardening a system and deploy to hundreds of users it is VERY important to perform tests. Besides, there are supported ways to hardening a system without compromise the core functionality of it. For Windows XP, take a look into the Windows XP Security Guide.