This year I’m going to TechReady 8 in Seattle to present a TMG Beta 2 session with Mohit Saxena and Bala Natarajan. The session will be Wednesday (02/04) morning and for the MSFTEs that will be there and are interested in see some cool features from this beta version, stop by and we will be glad to talk to you. Due this trip I will be away from the blog for the next 10 days.
Last Tuesday night I was helping out a friend from my team that was handling a case where customer was unable to access Outlook Anywhere from outside network. As usual, everything works inside, so who’s to blame? Of course ISA Server, it is the only thing different, right? Will see…
To better isolate the problem we eliminated tests using Outlook Client and just tried to access the RPC URL using IE (example: https://mail.contoso.com:443/rpc/rpcproxy.dll) and the result was the error below:
Figure 1 – host not available error.
We used Fiddler and we got an interesting result, see below:
Figure 2 – Fiddler result.
Since this is a real traffic I’m hiding some of the legitimate URLs, but the point in the different colors are:
Expected traffic using the external URL (for example mail.contoso.com)
Non expected traffic using internal URL (for example mail.contoso.local)
What this means? This means that ISA is for same reason losing the host name during this conversation, which is exactly what error 64 means: "The specified network name is no longer available", which is a win32 error originally called ERROR_NETNAME_DELETED.
At that time the question was: who is changing this name and sending it to ISA? Since the answer was not on our side (we saw on netmon trace that CAS was doing that) we collaborated with an Engineer from the Exchange Team that after some other troubleshooting steps fixed the issue by using the following article:
Note: the error 400 mentioned in the above article is the same as the one that we received from the CAS server (by looking the netmon trace).
Very interesting case where “again” everything works internally but doesn’t work externally. But again we proved the point that ISA was not causing this issue with a very useful help (as usual) from the Exchange folks.
Microsoft Malware Protection Center Blog put together the latest update about Conficker worm, the attack vectors, how to prevent and how to clean the system. It is all consolidated in their blog that you can access from here: http://blogs.technet.com/mmpc/archive/2009/01/22/centralized-information-about-the-conficker-worm.aspx
Just a quick follow up on the article that I wrote for the ISA Team Blog about ISA stopping answering requests. Last week I was collaborating with Networking Team in another case where ISA was stopping answering because of delays in DNS response. They fixed the DNS issue by changing the registry keys SocketPoolSize and MaxUserPort in all internal DNS Servers using recommendations from KB956188.
Conclusion: keep yourself alert on slow browsing issues and make sure that your DNS is working properly prior to start troubleshooting ISA.
Check this out this nice tool that allows you to analyze IIS logs and see if the ASP pages were victim of SQL Injection attack:
IAG 2007 SP2 hits the ground running with many customers applying it and realizing that not only this service pack introduces lots of changes but it also has some UI changes. It’s all about having a better experience for the end user / administrator. In this post I’m going to talk about three majors UI enhancements:
· Getting Start Wizard
· Network Configuration
· Policy Editor
2. Getting Start Wizard and Network Configuration
The new getting start wizard has the same format of TMG (already in RTM with TMG MBE) Getting Start Wizard. The idea is to assist the administrator to correctly configure IAG 2007 by an organized set of procedures. You can access getting start wizard by choosing the option Getting Start Wizard in Admin’s menu as show below:
Figure 1 – Accessing Getting Started Wizard.
The first screen has the core steps that this wizard will guide you through:
Figure 2- Getting Start Wizard.
Instead of guide you through each window I will leave it open so you can explore this feature. The step by step is very intuitive and I doubt that you will get stuck while following this wizard. It is important to mention that prior to even execute this wizard is important that you have the following elements already defined:
· How your IAG Network Configuration will be used - what is it considered internal and external?
· Domain membership - what is the domain name that IAG will belong to?
· Trunk configuration - which IP are you going to use to create the trunk?
· Application – what application are you going to publish it?
What is interesting is that the first option in this Wizard also can be accessed individually by Admin menu and choosing the option Network Configuration. The screen below will appear:
Figure 3 – Network configuration.
Either here or in the Getting Start Wizard you can specify network configuration for you IAG 2007.
3. Policy Editor
The other UI change that SP2 introduced was the new Policy Editor. This new UI was improved to make it easier to the administrator to create new policies based on specific platform, such as: Windows, Mac, Linux and other (see square A in the figure below). It also allows you to create new policy from expression without having to use a different window as it was before (see square B in the figure below):
Figure 4 – New Policy Editor.
The goal of this post is just present you some of the new UI enhancements of IAG 2007 SP2 and how the product is getting more mature by offering a better user’s experience. Go ahead and try SP2, I’m sure you will not regret.
Quick post just to bring awareness about this new KB that explains how to manually remove Conficker. Follow the steps from:
The reason why I’m saying “demystifying” is because many people are still having wrong concepts and therefore making wrong assumptions about how networks are configured on ISA Server/TMG. Although this is well documented at TechNet (since it is a core concept), sometimes due the massive amount of information you feel like: ah…I already know all this, I don’t need to read it.
Wrong assumption and this makes me go back in the day that I was Professor in a university in Brazil. I was teaching Operating System using the classic Tanenbaum’s book about OS and I remember that there was a student that clearly thought he knew all that stuff. He didn’t attend that much and when he did attend he didn’t pay attention. Well, that’s fine, let’s give the benefit of the doubt and assume that he knows what he is doing. Six months later he comes to me saying that he needs help to better understand preemptive multitasking and confessed that he missed that class because he thought he knew and preferred to do other stuff on that day. Moral of the story: never think that you know everything, even if the subject is the same that you read or heard many times. The person that is writing or telling you something usually have a different perspective and insight of the same subject that can show you things that you didn’t realize before.
Sorry, off topic, but I couldn’t resist. Anyway, since I’m a lover of self explanatory pictures combined with a decent walkthrough I think that this is probably one of the most intuitive explanations about networks concept on ISA/TMG. I’m talking about the series of two articles written by my friend Tom Shinder that will make you digest all you need to know about networks on ISA.
Check it out at here:
This is another one of those cases where ISA Server Service mysterious crashes once a day, at the same time and nothing changed in the environment. This just make me really fell that the lack of communication between the teams that deals with technology is getting far beyond of what should exactly be. Many companies are investing money in putting Security in place by adding layers and layers of technology but they are still missing two important elements: process awareness and change control procedures. The absence of those elements can directly impact availability of the environment. Why availability? Well, I will tell you later when I finish this post.
2. Analyzing the Data
In this case ISA Server Service was crashing with the following errors:
Event Type: Error
Event Source: Microsoft ISA Server Web Proxy
Event Category: None
Event ID: 14197
Time: 2:58:03 AM
ISA Server was unable to write content to the cache file.
Event Source: Microsoft Firewall
Event ID: 14057
Time: 2:52:37 AM
The Firewall service stopped because an application filter module C:\Program Files\Microsoft ISA Server\w3filter.dll generated an exception code C0000005 in address 64754CD5 when function CompleteAsyncConnect was called. To resolve this error, remove recently installed application filters and restart the service.
The event 14057 is clear about one thing: this was an access violation exception (C0000005) on the filter module W3Filter.dll. Too broad, can be many things including issues with the filter itself, so we need to get a crash dump of this guy to better understand what is going on. Following the approach of one of my posts we can use DebugDiag to attach to wspsrv.exe and get the dump. After getting the dump you can use this other post as an example of how to analyze it. Unfortunately this is one of the cases where the public symbols don’t help that much as you can see below:
WARNING: Frame IP not in any known module. Following frames may be wrong.
2b37fe10 6476e6df 27441f80 647717fe 275a5558 0x3a6169
2b37fe24 64778438 00000001 2bf579a0 64703de0 W3Filter!DllUnregisterServer+0x45ede
2b37fe90 0046d701 275a5558 00000000 00000040 W3Filter!DllUnregisterServer+0x4fc37
2b37fefc 0046e461 00000000 00000000 00000000 wspsrv+0x6d701
2b37ff20 0046e568 2bf57818 0046e3d7 2b37ff50 wspsrv+0x6e461
2b37ff30 0046d4ba 00000000 00000000 00000000 wspsrv+0x6e568
2b37ff50 00455fd7 2bf578bc 00000000 00000000 wspsrv+0x6d4ba
2b37ff7c 00456c8e 2bf578bc 00000000 00000000 wspsrv+0x55fd7
2b37ffb8 77e64829 00000015 00000000 00000000 wspsrv+0x56c8e
2b37ffec 00000000 00456b26 00000015 00000000 kernel32!GetModuleHandleA+0xdf
6476e6df 8b4624 mov eax,dword ptr [esi+24h]
STACK_COMMAND: ~50s; .ecxr ; kb
The !analyze result showed above will make you under the impression that W3Filter.dll is the culprit and it is exactly the opposite, this guy is only a victim.
After deeply analyze the dump using the private symbols we got to a conclusion that someone was locking the cache file when the Web Filtering was trying to write to it. Guess who was locking it? Once upon a time there was a system administrator that was following a plan that he received from his management to install backup software in all Windows Servers, so he installed this backup software on ISA and configured a Job to run every night…
The backup software was backing up the whole server (all hard drivers) including the driver where the ISA Cache was located. For this reason customer was saying that the issue just happened when the ISA Server Cache was enabled, if they disabled the cache the issue didn’t happen. Well make sense and the recommendation to exclude cache from backup as not new, as a matter of fact the article that recommends this is out there since October 2004, which is the following one:
Event ID 5, event ID 14079, and event ID 14176 are logged in the Application log on your Internet Security and Acceleration Server computer
Now the answer for: Why Availability? Because the ISA Server service in this case was crashing due and addition of a new product in the ISA Box without testing it in a lab environment (where the change control procedure is?). The Windows OS maintenance was responsibility of the System Administrator that with all the good intentions configured the Backup Software to back it up the whole hard drive. However the Firewall Admin wasn’t aware of this addition since it was out of the scope of his duty (where the process awareness is?) and he swear since the begging that nothing change in the environment and ISA was crashing from nothing L. But, this story had a happy end at least, so let’s finish this post with a smiling face J.
Microsoft TechNet Magazine February 2009 Issue is online now and the Security Watch column brings an article about Malware Inspection on TMG MBE written by myself, Mohit Saxena and Jim Harrison:
For more information check:
If you read the article you will see at the end that we are writing the forthcoming MSPress TMG Book and we are glad to have onboard Tom Shinder as Tech Reviewer and bringing with him all the experience with all these years working in the ISA community and leading isaserver.org.
Yuri Diogenes, Mohit Saxena, Jim Harrison and Tom Shinder
This picture was taken last year in Seattle when we were in the TMG CTP3 Event for the TAP Program. We are enjoying working on this project and I hope you like the book when comes out.
Happy New Year everybody!
I hope you enjoyed your new years eve because now you might want to take a look on this worm that is causing lots of headaches to all IT Admins. MMPC (Microsoft Malware Protection Center) has a report about this malware and how to proceed to avoid infestation:
Good news is that ISA Server and TMG can block outbound requests for this worm and yesterday night (before midnight) our IR (Incident Response) Team in partnership with ISA Server Team brought together an action plan to allow ISA/TMG to block that. Jim Harrison automated this process by creating a script that you can use to create policies to block conficker and you can download from here:
Enjoy your day off and be sure to implement those actions ASAP.