Hi there,
In this blog post, I’ll be talking about another TMG problem where FTP over HTTP was failing through TMG server.
Let me first summarize the scenario:
- Internet Explorer clients need to connect to an external FTP site through TMG server
- Due to some other requirements, this FTP site needs to be accessed passively
FTP filter in TMG server already uses passive FTP when connecting to external FTP sites:
(Note: And this is the default behavior, please see http://blogs.technet.com/b/yuridiogenes/archive/2010/03/16/error-502-active-ftp-not-allowed-when-trying-to-list-files-in-a-ftp-session-behind-forefront-tmg-2010.aspx for more information.
That was also the case in my customer’s scenario but passive FTP connection to the target FTP server was still failing. After some troubleshooting, we found out that TMG server was trying to connect to the target FTP site actively even FTP filter was configured as above.
Normally, when you type ftp://target-FTP-Server-FQDN in the IE address bar and IE is configured to use a Proxy server, the connection request will be sent as an HTTP request to the Proxy server (and the FTP GET request will be inside that HTTP request), this is also called FTP over HTTP. So the request flow will be similar to below:
a) Client sends the request via FTP over HTTP to the Proxy server
b) Proxy server connects to the target FTP server via FTP procotol
After some further troubleshooting with TMG data packager and the network trace analysis, I found out that FTP filter wasn’t involved in when Proxy server receives FTP over HTTP traffic from clients and hence FTP filter setting doesn’t apply to FTP over HTTP requests.
The resolution was to set the NonPassiveFTPTransfer registry key on the TMG server and restart the firewall service:
Note: You can find more information about that registry key at http://support.microsoft.com/kb/300641 How to enable passive CERN FTP connections through ISA Server 2000, 2004, or 2006
As mentioned above, after the registry key is created, you’ll need to stop and then start firewall service from an elevated command prompt:
net stop fwsrv
net start fwsrv
To summarize; even though “NonPassiveFTPTransfer” registry key shouldn’t be needed for TMG server, the exact requirements are as follows:
a) If the internal client sends the FTP request directly through FTP procotol, there’s no need to change anything on TMG server side as the FTP filter will kick in and the FTP connection to the external FTP server will be initiated passively (Examples: Command prompt FTP client, 3rd party FTP client applications, IE which isn’t configured to use a Proxy server etc)
b) If the internal client sends the FTP request through FTP over HTTP procotol, then the changes mentioned above needs to be implemented on TMG server side in order for TMG server to initiate the outbound FTP connection passively (Example: IE which is configured to use a Proxy server)
Hope this helps
Thanks,
Murat
In this blog post, I’ll be talking about a TMG related issue. Actually it’s not an issue that stems from TMG itself but the way TMG server is configured (using authenticated rules on TMG server) triggers the problem.
This is already a known fact and we have a KB article that explains this issue (JVM applications cannot send authentication information when requested) and the workaround is to turn off authentication for the access rule that will allow the client’s connection to external networks:
http://support.microsoft.com/kb/925881/ An ISA server or Forefront Threat Management Gateway server requests credentials when client computers in the same domain use Internet Explorer to access Web sites that contain Java programs
So if you see all or some parts of a web page is not displayed correctly and you see Proxy authentication required or similar messages on the client side and you suspect that Java is involved somehow you’ll have to implement the steps mentioned at the above article.
But sometimes it may not be that clear which was the case in my scenario. The customer reported that videos at an on demand video conference site weren’t successfully viewed and the application running inside IE was displaying an unrelated error. I suspected that we were hitting the problem mentioned above and then requested the customer to configure a temp access rule to allow all outbound access for “All users”, then the videos started to play J
Then we changed the rule target to the target web site only (you can do this via a URL set (for HTTP/HTTPS access) or via domain name set (for any protocols), you can find more information below:
http://technet.microsoft.com/en-us/library/cc441706.aspx Processing domain name sets and URL sets
Since the customer was connecting to https://www.videoondemandwebsite.com , we have added this domain to the rule target. But afterwards the video access was still failing. Then we decided to collect more information on what kind of http activity was taking place on the client side. I asked the customer to install Fiddler on the client to see this activity (you can download the tool at http://www.fiddler2.com)
You’ll find below a sample screen shot taken from Fiddler which was taken when accessing Microsoft’s web site:
As you can see from the above output, even if you see a certain address in IE (www.microsoft.com in this example), the browser might need to connect to other related web sites to load some images, to get a script, etc etc. In the above example, browser also connects to ads1.msn.com or rad.msn.com ...
That was the case in my customer problem, even though the customer was connecting to https://www.videoondemandwebsite.com, the browser was connecting to a few other web sites like *.site1.com and *.site2.com. So we changed the relevant rule to cover these domain names as well and the problem was resolved:
*.videoondemandwebsite.com
*.site1.com
*.site2.com
I would like to talk about an issue that I have dealt with recently regarding Internet Explorer and displaying TMG error messages.
The problem reported was that newer IE versions (like 8 or 9) didn’t display the regular TMG error message which is displayed when the access rule allows certain users and the current user is not one of the allowed users (Error Code: 502 Proxy Error. The Forefront TMG denied the specified Uniform Resource Locator (URL). (12202)),instead the "Page not found" error was displayed and that was causing some help desk calls since the user thought that the target web site was not reachable based on the displayed error message whereas the real problem was user was not allowed to access the given web site.
IE6 didn’t have the same problem. Then we started investigating the problem from TMG perspective to make sure that it wasn’t something stemming from TMG server side. After some further troubleshooting (network traces), we found out that TMG was sending the regular error page back to the client but somehow it wasn’t displayed by the IE client.
Then we focused on the IE side. After some further investigation, I found out that it was the expected default behavior for newer Internet Explorer versions (8 and 9, we haven’t tested 7 but this might apply to 7 as well) for security reasons. You can find below more information about the vulnerability that could be exploited when IE uses Proxy servers to connect to target servers:
Pretty-Bad-Proxy: An Overlooked Adversary in Browsers’ HTTPS Deployments
Having said that, there’s a registry key which allows you to turn this enhanced security feature off in newer IE versions. You can see the details below on how to do this on the client machines:
http://msdn.microsoft.com/en-us/library/ms537184(VS.85).aspx Introduction to Feature Controls
- You’ll need to create the highlighted key at the given path on a client machine:
HKEY_LOCAL_MACHINE (or HKEY_CURRENT_USER) SOFTWARE Microsoft Internet Explorer Main FeatureControl FEATURE_SHOW_FAILED_CONNECT_CONTENT_KB942615 (Note: you’ll also need to create that registry key under “FeatureControl”)
Reg key name: Iexplore.exe
Type: REG_DWORD
Value: 0x00000001
You can also get some more information at http://msdn.microsoft.com/en-us/library/dd565641(VS.85).aspx#eventLog Event 1065 - Web Proxy Error Handling Changes
I would like to re-emphasize that it normally shouldn’t be turned off from security perspective, so please implement it at your own risk.
It has been sometime since I haven’t posted anything at the blog and finally got a chance to do so.
Recently I helped a colleague of mine to investigate a “Windows server connects to unknown destinations” problem. Our customer reported that they were unable to use their internet connections when they start up a certain Windows 2003 server in their network. After some troubleshooting at their side, they concluded that Windows 2003 server was generating excessive UDP activity towards random IP addresses at random times and they were asking for help with finding out the relevant process on the Windows 2003 server side that was generating that traffic.
We asked them to collect a network trace while the problem was visible. As the problem happened very frequently, it didn’t take them long to collect such a network trace. After analyzing the network trace collected, I found out the process that was source of the excessive traffic. Now let’s walk it through together:
a) First of all I tried to identify the excessive UDP traffic. It was like below:
...
303 16:19:48 10.08.2011 0.0000000 33.9062500 20.0.0.1 192.168.1.105 UDP UDP:SrcPort = 2976, DstPort = 10647, Length = 8200 {UDP:48, IPv4:47}
304 16:19:48 10.08.2011 0.0000000 33.9062500 20.0.0.1 192.168.1.105 IPv4 IPv4:[Fragment, Offset = 1480] Src = 20.0.0.1, Dest = 192.168.1.105, Next Protocol = UDP, Packet ID = 814, Total IP Length = 1500 {IPv4:47}
305 16:19:48 10.08.2011 0.0000000 33.9062500 20.0.0.1 192.168.1.105 IPv4 IPv4:[Fragment, Offset = 2960] Src = 20.0.0.1, Dest = 192.168.1.105, Next Protocol = UDP, Packet ID = 814, Total IP Length = 1500 {IPv4:47}
306 16:19:48 10.08.2011 0.0000000 33.9062500 20.0.0.1 192.168.1.105 IPv4 IPv4:[Fragment, Offset = 4440] Src = 20.0.0.1, Dest = 192.168.1.105, Next Protocol = UDP, Packet ID = 814, Total IP Length = 1500 {IPv4:47}
307 16:19:48 10.08.2011 0.0000000 33.9062500 20.0.0.1 192.168.1.105 IPv4 IPv4:[Fragment, Offset = 5920] Src = 20.0.0.1, Dest = 192.168.1.105, Next Protocol = UDP, Packet ID = 814, Total IP Length = 1500 {IPv4:47}
308 16:19:48 10.08.2011 0.0000000 33.9062500 20.0.0.1 192.168.1.105 IPv4 IPv4:[Fragment, Offset = 7400] Src = 20.0.0.1, Dest = 192.168.1.105, Next Protocol = UDP, Packet ID = 814, Total IP Length = 820 {IPv4:47}
309 16:19:48 10.08.2011 0.0000000 33.9062500 20.0.0.1 192.168.1.105 UDP UDP:SrcPort = 2976, DstPort = 10647, Length = 8200 {UDP:48, IPv4:47}
310 16:19:48 10.08.2011 0.0000000 33.9062500 20.0.0.1 192.168.1.105 IPv4 IPv4:[Fragment, Offset = 1480] Src = 20.0.0.1, Dest = 192.168.1.105, Next Protocol = UDP, Packet ID = 815, Total IP Length = 1500 {IPv4:47}
311 16:19:48 10.08.2011 0.0000000 33.9062500 20.0.0.1 192.168.1.105 IPv4 IPv4:[Fragment, Offset = 2960] Src = 20.0.0.1, Dest = 192.168.1.105, Next Protocol = UDP, Packet ID = 815, Total IP Length = 1500 {IPv4:47}
312 16:19:48 10.08.2011 0.0000000 33.9062500 20.0.0.1 192.168.1.105 IPv4 IPv4:[Fragment, Offset = 4440] Src = 20.0.0.1, Dest = 192.168.1.105, Next Protocol = UDP, Packet ID = 815, Total IP Length = 1500 {IPv4:47}
313 16:19:48 10.08.2011 0.0000000 33.9062500 20.0.0.1 192.168.1.105 IPv4 IPv4:[Fragment, Offset = 5920] Src = 20.0.0.1, Dest = 192.168.1.105, Next Protocol = UDP, Packet ID = 815, Total IP Length = 1500 {IPv4:47}
314 16:19:48 10.08.2011 0.0000000 33.9062500 20.0.0.1 192.168.1.105 IPv4 IPv4:[Fragment, Offset = 7400] Src = 20.0.0.1, Dest = 192.168.1.105, Next Protocol = UDP, Packet ID = 815, Total IP Length = 820 {IPv4:47}
315 16:19:48 10.08.2011 0.0000000 33.9062500 20.0.0.1 192.168.1.105 UDP UDP:SrcPort = 2976, DstPort = 10647, Length = 8200 {UDP:48, IPv4:47}
There were thousands of such UDP packets without meaningful data being sent to the very same target
NOTE: I deliberately changed the public IP address targeted to 192.168.1.105
b) After that, I tried to track back the starting point and I found the following in the network trace:
302 16:19:48 10.08.2011 21.7031250 33.9062500 httpd.exe 172.16.10.5 20.0.0.1 HTTP HTTP:Request, GET /webdav/shell.php, Query:act=phptools&host=192.168.1.105&time=120&port=3074 {HTTP:46, TCP:45, IPv4:20}
NOTE: I deliberately changed the source public IP address to 172.16.10.5 (from where the attack was initiated)
Note: I was lucky here as Network Monitor easily identified the relevant process. If you cannot see the process easily, you may want to try other methods given at the following post:
http://blogs.technet.com/b/nettracer/archive/2010/08/02/have-you-ever-wanted-to-see-which-windows-process-sends-a-certain-packet-out-to-network.aspx
c) After some research, I found out that this was a vulnerability with Webdav within Apache Web server
http://www.apachefriends.org/f/viewtopic.php?f=5&t=45534 [SOLVED] httpd.exe maxing internet connection...
http://www.opensc.ws/off-topic/11895-webdav-link-crawler-ip-scanner.html
So the attacker was using the customer’s Windows 2003 server as a victim to start DoS (Denial of service) attacks to other hosts...
d) After the customer upgraded the Apache Web server version to the latest one, the issue was resolved J
Long story short, always keep your all software up to date, not just the OS and its services running
Cheers,