Great news folks! Netmon Team just released Netmon 3.3 for public download. This release brings new enhancements from previous version, such as:
· Ability to capture on WWAN and Tunnel interfaces on Win7
· Right-click add to alias.
· Right click go to definition
· API Extension…and many others.
Don’t wait go grab your new tool at:
Also, remember that there are many netmon experts available at http://nmexperts.codeplex.com
Two weeks ago I worked in a case where customer was not able to watch a video stream behind ISA Server 2006 using Web Proxy Client. The interesting part was that he could watch it using the same workstation by enabling Firewall Client. I reproduced the issue in lab and the result was quiet interesting.
2. Is it pure HTTP? Really?
When we deal with an outbound scenario like this and the tests shown that only Firewall Client is capable to access the external resource the first question that comes up is: is this really a HTTP or HTTPs only type of traffic? Sometimes you inherit an environment that has third party application that you have no idea how it works internally, if it is Web Proxy or Winsock capable. Hopefully this was not the case and later on you will understand why.
3. Gathering Data
To better understand what was happening when the client was trying to establish a connection with the destination web site I used Netmon 3.2 and for my surprise, TCP port 80 was not the only port in as you can see below:
Figure 1 – Netmon Trace
Real destination IP Address
Real destination URL
SYN attempt to a port other than TCP 80
Notice the following in this picture:
· Frame 73: Client sends the HTTP GET Request to the destination web site, a SWF (Adobe Shockwave Flash file).
· Frame 77: A HTTP 304 response from the destination server through ISA Server (10.20.20.2)
· Frame 79: A HTTP GET request for the playlist (playlist.xml)
· Frame 84: A HTTP 200 response from the destination server through ISA Server (10.20.20.2)
· Frame 93: Out of nothing client sends the TCP SYN to the destination server (real IP in red) on port 1935. Since ISA Server does not have this protocol created and the rule was only allowing HTTP/HTTPs the answer for TCP SYN never arrived (three SYN requests without answer – frames 93, 102 and 122).
Based on this I had enough information to understand that this was not a simple HTTP request and that port 1935 was required for this communication. But why this was happening? This answer is on the KB below from Adobe:
4. Adjusting ISA Server
To comply with this request I created three protocols as shown below:
Figure 2 – TCP Port 1935.
Figure 3 – TCP Port 1626
Figure 4 – TCP Port 7070
After that I added those protocols as part of the web access policy to allow web proxy clients to successfully access this video.
This post is about a case where Windows Mobile devices were having problems when enabling Active Sync Direct Push feature. On the device we were receiving the error 0x80072eff. This error means:
# anonymous HRESULT: Severity: FAILURE (1), Facility 0x7, Code 0x2eff
# for hex 0x2eff / decimal 12031
To better illustrate the scenario and the troubleshooting line that I followed look the diagram below:
Figure 1 – Network Diagram and tests that were performed
To perform those tests I used Windows Mobile 6.1 Professional Emulator installed on a laptop so we have the flexibility to connect this laptop on each segment. In the above diagram each scenario has an specific result as shown below:
· Scenario 1 – DirectPush Fails.
· Scenario 2 – DirectPush Fails.
· Scenario 3 – DirectPush Works.
· Scenario 4 – DirectPush Works.
· Scenario 5 – DirectPush Works.
This pretty much isolates the problem to be either in the first hardware firewall or in the Internet border router. To see where the issue could be located we attached Netmon trace on ISA Server (internal and external interface as shown Figure 1) and also in the Exchange Server. The result was quiet interesting since all the communication went through just fine all the way from the external mobile device to the Exchange.
The only interesting part was found in the trace 1 (ISA External NIC) when the following packet sent from the hardware device to ISA:
HardFirewall ISAServer TCP TCP:Flags=.....R.., SrcPort=49985, DstPort=HTTPS(443), PayloadLen=0, Seq=2504173114, Ack=0, Win=0
A nice TCP Reset sent from the hardware firewall to ISA Server was causing the connection to be reset during the synchronization. To resolve this problem we implemented the following recommendation in the border hardware firewall (the one in front of ISA Server):
Enterprise firewall configuration for Exchange ActiveSync Direct Push Technology
Well, one more ISA Server in sandwich mode that caused days of troubleshooting and ISA Server was only a victim. Talking about ISA Server in sandwich mode, my previous post (see below) ISA Server was also in sandwich mode and that was the justification to open all ports since they have a hardware firewall to protect. Please DO NOT follow this example and create such rule or you will be owned as Shinder said in his post J.
Last week I said that 3 out of 5 cases where user says it is an ISA issue it ends up not being. Today I worked in another case that goes to that statistic. What makes me sad about this is the mindset that many Windows Admins, Network Administrators and users in general have when deal with an issue where ISA is in the middle. Sometimes they don’t even stress the troubleshooting because of the wrong thinking that: if works internally and doesn’t work externally, then of course is ISA blocking. No it is not like that.
Use your troubleshooting skills to narrow down the issue, research, try to repro, do above and beyond on the subject that you excel. If after all that you have enough technical argument to say it is an ISA issue, than it is fine. But point to ISA just because it is the device in the middle it is just a bad habit.
The issue in this case was that Windows Vista clients were unable to access resources through a VPN and the VPN Server was an ISA Server 2006. First point of argument in this case was: in the past (with Windows XP) the VPN Clients used to receive the IP address of the default gateway, now with Vista it appears as 0.0.0.0. Well, that’s not ISA and this is explained in here:
935269 The IP address of the default gateway for a dial-up connection in Windows Vista is 0.0.0.0
Second point of argument was: I correctly connect to my VPN Server, authenticate just fine but can’t access resources internally by name. If ISA Server rules are configured correctly, allowing the traffic (which in this case it was) then issue needs to be investigated in the client side. Guess what? It was a known issue with Vista and here it is how to resolve:
929853 You cannot access network resources and domain name resolution is not successful when you establish a VPN connection to the corporate network from a Windows Vista-based computer
I’m sure this won’t be the last case where Admins point fingers to ISA, but at least is one more that I’m glad to document here and share with you guys.
It is very common to a company have their security team conducting a port scan tests against servers and other devices within their network. When they perform this type of test against ISA Server the expected behavior can vary, because everything will depend on what ports are open, which services ISA Server is publishing to the internal and external network.
The other aspect that you need to consider is that when you are conducting such test against ISA Server, the IDS capabilities of ISA Server will trigger alerts that will look like this:
Figure 1 – Intrusion Detect Alert.
To cause this trigger to happen in my lab environment I installed software called Zenmap in the internal workstation ran the scan against ISA Server and the result was:
Figure 2 – Nmap Output
Notice that in this case I just have 3 ports open, you can see that better if you click in the Host Details tab, result will appears like this one below:
Figure 3 – Scan details
Why my ISA Server is listening in all ports?
Make no mistake about it; ISA Server doesn’t do that by itself. Your configuration will tell what ISA needs to do. If you configure your ISA Server to allow everything, it will allow everything; there is no magic (we hope we could have) that will make ISA Server to protect itself and the network that is supposed to protect if the configuration says otherwise.
So this whole thing that I’m explaining here is to tell this story about a call where this Security Admin was nervous that his port scan utility (he was using Nmap) was telling that ISA Server had all ports open. I could easily reproduce this behavior in lab and here it is how it looks like the scan result:
Figure 4 – All ports opened.
The question was: why ISA is listening in all ports?
Answer: because you are telling him to do that.
What happens is, if you create a firewall policy rule that allows all traffic to all networks, ISA Server will have this behavior, which is expected. Here it is the problematic rule that was allowing this to happen:
Figure 5 – You said to open all and ISA shall obey.
L no more comments…
Tired of reading huge BLG files and compare with triggers to see if the system is running as it should? I hear you my friend! It is time to automate that using PAL – Performance Analysis of Logs, a great tool created by a MSFTE that you can download from http://pal.codeplex.com and get the tips from Windows Performance Team on how to use it here http://blogs.technet.com/askperf/archive/2009/04/10/two-minute-drill-performance-analysis-of-logs-tool-pal.aspx.
Note: we are working to put ISA Server in this list below, stay tune.
As announced by Tom Shinder on his blog, here it goes:
I’m really pleased to use this space tonight (yeah, almost 9PM here in Texas) to advertise the forthcoming Portuguese version of Windows Server 2008 book written by William Stanek and having as technical review my friend Gilson Banin from Brazil. They will be hosting a presentation in Sao Paulo as you can see in the advertisement below. So if you are in the big SP make sure to stop by and appreciate this great event:
Today we are releasing the April 2009 Security Bulletin and this month we have a security update for ISA and TMG for a vulnerability that can cause Denial of Service. You can read more about this update on the ISA/TMG perspective reading the bulletin MS09-016:
For more information about MS April 2009 Security Bulletin watch the presentations below:
To download the updates for ISA/TMG use the following links:
ISA Server 2004 Standard Editionhttp://www.microsoft.com/downloads/details.aspx?FamilyID=adf623fa-2d74-4f2a-9835-4b8debdb0e1b
ISA Server 2004 Enterprise Editionhttp://www.microsoft.com/downloads/details.aspx?FamilyID=d1d55ab6-3de5-4811-9693-8d43f49f5fe8
ISA 2006 (Standard and Enterprise) http://www.microsoft.com/downloads/details.aspx?FamilyID=eda30bcc-0582-4f60-a4c5-ea5000b7c770
Note: those packages will be also available via Microsoft Update and WSUS.