Yuri Diogenes's Blog

Thanks for your participation

yuridio — Wed, 10 Feb 2010 00:19:36 GMT

I would like to say thanks for all partners that joined this morning to the troubleshooting performance issue on ISA Server session, it was great to share some ISA Perf best practices with you all. We had a great audience from many locations (US, Europe, Brazil and other places), which makes me really proud since I know that you have a time zone challenge to overcome in order to participate on that. Thanks for taking the time to watch this webcast.

For those partners that could not participate today, the offline recording version should be available on the Microsoft Partner Learning Center, so you still have a chance to watch on a time that is better for you.

Again, thanks a lot and hope to talk to you again soon.

It’s here

yuridio — Fri, 05 Feb 2010 06:36:31 GMT

I received my copy today directly from MSPress, if you pre ordered yours through Amazon you will be probably receiving yours pretty soon:

A collective thanks to all MSPress folks that make this great edition: Martin DelRe, Karen Szall, Carol Vu and Devon Musgrave. You guys rock !!

ISA Server triggers lots of 14197 events

yuridio — Sat, 30 Jan 2010 06:52:00 GMT

1. Introduction

This post is about a scenario where Firewall Administrator notices that the application log in the ISA Server computer is full of events 14197 saying:

Event ID 14197

Source Microsoft Web Proxy

Type Error

Description ISA Server failed to write content to cache file. The error code in the Data area of the event properties indicates the cause of the failure.

He notices that after some time with this event one of the following symptoms happen:

· The Firewall Service stops responding and when he tries to restarts it hangs with starting status for awhile before the status change to started.

· Sometimes the Firewall Service quits without any message.

· Sometimes it is not possible to restart firewall service because it hangs in stopping status.

I’m very familiar with those events and most of the time (9 out 10) this is caused by a third party software scanning or locking the cache file, the article http://support.microsoft.com/kb/887311 explains that in details. For this particular scenario the most common third party application that can cause this is the antivirus. When you have file scan antivirus installed on ISA Server (or TMG) you need to use the article Considerations when using antivirus software on ISA Server to create an exclusion list of the folders, files and process that shouldn’t be scanned by the antivirus.

So, if I follow the recommendations from that article I should be good, right? You should, as long as the exclusions are REALLY…. REALLY in place. I’m emphasizing the “really” because of some recent unfortunate experiences that I had; let me tell you a story to chill the bones about a thing that I saw…oops, that a song.

2. The Tales

Recently I worked in at least three different scenarios very similar where the firewall administrator created the exclusion list by following that article, however the exclusions were added though AV policy deployment located in the central antivirus server management. The reason why this policy was deployed via central location was because the local client AV installed on ISA didn’t allow local changes; everything was done by this AV Server. However even after making those exclusions the event 14197 was still happening. Of course I heard the most common argument in such scenarios: I told you that this was not a problem on our AV, this is an ISA issue for sure.

Ok, at that point I couldn’t really argue and had to think that every piece of software is innocent until we prove otherwise. So I followed this line of think I went to a path of collecting evidence to see who was doing that. Hunting time !!

3. Preparing the Environment

One way to collect evidences that there is a process other than ISA Server process is to audit the folder where the cache is located, by default %systemdrive%\urlcache. To do that you need to follow the steps below:

1. Enable local audit for object access (Start / Run / secpol.msc):

2. Enable Auditing for the URLCache folder for the following users:

Notice that the column access says special, this is because I’m choosing the following type of access:

The reason why I’m selecting those users is because:

· Most of the ISA Server services run under System account and Firewall Service runs under Network Service account.

· The Antivirus that I’m running on my lab (MS FCS) runs under System account.

· I’m selecting everyone because I want to catch manual attempts (from users) to access this folder.

After finishing that you just need to wait for the next occurrence.

3. Collecting Evidences

In order to show what you will see in the event viewer when a process try to access the URLCache folder I decided to build a lab and use Forefront Client Security without the ISA folder exclusions. After implementing the actions above here what I got when FCS scanned the URLCache folder:

Event Type: Success Audit

Event Source: Security

Event Category: Object Access

Event ID: 560

Date: 1/30/2010

Time: 5:00:22 AM

User: NT AUTHORITY\SYSTEM

Computer: ISACONTN1

Description:

Object Open:

Object Server: Security

Object Type: File

Object Name: C:\urlcache\Dir1.cdat

Handle ID: 1772

Operation ID: {0,6542847}

Process ID: 1772

Image File Name: C:\Program Files\Microsoft Forefront\Forefront System\Client\AntiMalware\MsMpEng.exe

Primary User Name: ISACONTN1$

Primary Domain: CONTOSO

Primary Logon ID: (0x0,0x3E7)

Client User Name: -

Client Domain: -

Client Logon ID: -

Accesses: ReadAttributes

Privileges: -

Restricted Sid Count: 0

Access Mask: 0x80

This is the evidence that you need in order to prove that there is another process accessing this file!!

At this point you probably are asking: OK, I got all that, but on your real scenarios, why the folder exclusion was not working? Well, that I don’t technically know since it was a third party AV that was failing to push the policy from the central location to the AV agent installed on ISA Server, hence the folder exclusion was not taking effect.

My SMTP Server is on the Blacklist, now what?

yuridio — Tue, 26 Jan 2010 19:27:37 GMT

1. Introduction

Recently the number of times that I received this question increased (not sure why), but the fact of the matter is that you can do something to resolve this problem (or at least identify the source of the problem), if you have the right tools. When your ISP informs you that you are in an email blacklist you can review the results by going to a site that provides real time blacklist results, such as http://www.mxtoolbox.com/blacklists.aspx.

2. Step 1 – Identifying

Next thing to do is identify the source of the problem, why your company got blacklisted? If you have ISA Server in the border of your network, what you can do is just watch your live logging creating a filter for SMTP Protocol coming from the internal network, as shown below:

If you have only one SMTP Server internally, then the only IP that you should see sending SMTP traffic is your SMTP server, if you start to see workstation’s IP address in the list then you need to investigate that further. Make sure to also get a netmon trace on ISA (internal and external adapter) at the same time or simple get an ISA Data Packager in repro mode using Web Proxy and Web Publishing template.

2. Step 2 – Containing

To contain the amount of SMTP traffic leaving your network you need to make sure that the only host that is capable to send SMTP traffic out (to the Internet) is your SMTP Server.

This is the reason why I get disappointed when I open a Firewall Policy rule and see a rule that allows All Outbound. If your environment doesn’t have such rule, the SMTP traffic that are not coming from the SMTP Server will not be able to leave your internal network in the first place.

Notice that I’m using step two as contention because on step one you are logging the real traffic, which means that you will have real data to analyze it later.

3. Step 3 – Remediating

After implementing this contention, you can now start working on the workstation and verify why it is sending traffic out to the Internet. At this point, you can:

· Get a netmon sample of the traffic, so you can see which process is sending out SMTP traffic.

· Unplug this workstation from the network

· Use an Antivirus (like Forefront Client Security or Microsoft Security Essentials) to scan the local workstation and see if it is infected.

After cleaning the system (last time that I worked in a case similar to this one, the piece of malware that was found in the workstation was Backdoor:Win32/Oderoor.gen!A.), one other thing that you can also do is to follow the procedures form the article Capturing a Trace at Boot Up and get a sample capture while the machine is starting up. With that you can see the traffic profile and make sure that there is no malicious attempt to go out right in the beginning of the system boot.

4. Conclusion

This is only a simple example of an incident response in case your SMTP is being blacklisted due a piece of malware that is running inside your network. A full description of Microsoft Incident Response recommendation read the article below:

Responding to IT Security Incidents

http://technet.microsoft.com/en-us/library/cc700825.aspx

Unable to Access OWA page published by UAG 2010

yuridio — Fri, 22 Jan 2010 14:59:08 GMT

1. Introduction

Consider a scenario where the UAG administrator just published OWA located on an Exchange Server 2010 and when external users click the OWA link from the portal they get the following page:

They users are able to access OWA internally without any problem and before implementing Exchange 2010, the users were able to access OWA using Exchange 2007.

2. Troubleshooting

Reviewing the Web Monitor log it is possible to notice the following errors:

As you can see the error message that is highlighted above says that the application Exchange 2010 of type ExchangePub2007 failed. This means that when the administrator created the Exchange publishing rule within the trunk he selected the wrong Exchange version (or in this case he just used the same application in the portal after replacing Exchange 2007 to Exchange 2010). This option is available on step 2 of the add application wizard as shown below:

3. Conclusion

When publishing Exchange through UAG 2010 make sure to choose the correct version on step 2 of the Add Application Wizard and if you currently have a trunk that is publishing Exchange 2007, do not change the rule to point to a new Exchange 2010 Server, you will need to create a new application publishing within the trunk to publish Exchange 2010.

Forefront TMG 2010 Case Studies

yuridio — Thu, 21 Jan 2010 21:45:11 GMT

It is very common for most customers to really wait until a product gets more mature, wait to see the trends and evaluate if they should migrate or not. Questions like: who is adopting this platform? what advantages I will have using this new product? why should I migrate my stable ISA environment to TMG? Those are very common and understandable questions and the fact of the matter is that there is nothing better than real world case scenarios to really answer those questions. If this is your case, then take a look on the Forefront TMG 2010 case studies page at http://www.microsoft.com/forefront/threat-management-gateway/en/us/case-studies.aspx , this should help you to make your decision.

Troubleshooting ISA Server 2006 Performance Issues – WebCast for MS Partners

yuridio — Wed, 20 Jan 2010 15:27:50 GMT

Next February 9th 11:30 AM CST I will be delivering a presentation about Troubleshooting ISA Server 2006 Performance issues for Microsoft Partners, if you are a partner and deal with ISA Server you should watch this presentation. Here it is the agenda with the core topics that will be covered:

What about understanding the issue first?
Source of the Performance Problems
Core Components
Data Gathering
Troubleshooting Sample Scenarios

Slow Internet Access
ISA stops answering requests

Advanced topics

The registration is open at https://training.partner.microsoft.com/learning/app/management/LMS_ActDetails.aspx?UserMode=0&ActivityId=573031

See you there !!

Can’t see a new published application in UAG 2010 Portal even after activate the changes

yuridio — Tue, 19 Jan 2010 22:29:24 GMT

Consider a scenario where you just added a new application on UAG 2010 portal, you activate the configuration and see the window below:

Right after click in Finish button we tried to access the portal and didn’t see the application in there. After less than a minute we tried again and the new application is still not there. Although the window above says that the configuration was saved, you should read the full message and realize that it says: “Synchronizing changes might take a few minutes”. In order to see if the synchronization is completed you should go to Forefront UAG Activation Monitor, in this case the window that we saw was the one below:

This means that the activation is not completed yet, after some time the activation was completed and the window showed the full log of what it was done during this process as shown below:

Notice that at the end of this log output there is a message saying that the activation completed successfully, at this point we tried to access the portal and the application was there.

OWASP Top Ten Project 2010 RC is available

yuridio — Tue, 19 Jan 2010 15:55:40 GMT

The Open Source Web Application Security Project (OWASP) has the Top Ten Project document currently in release candidate that can be downloaded from http://www.owasp.org/index.php/File:OWASP_T10_-_2010_rc1.pdf and the presentation that covers the main changes can be downloaded from http://www.owasp.org/images/a/a1/AppSec_DC_2009_-_OWASP_Top_10_-_2010_rc1.pptx. This document is very interesting from the application security standpoint and emphasizes some practices that are in place for years but are still playing a big role when mitigating today’s threats.

Very worth it to read it.

Where is fwengmon on Forefront TMG 2010?

yuridio — Mon, 18 Jan 2010 16:17:00 GMT

FWEngmon can be used in many circumstances and here are some great examples on how to use this tool:

http://blogs.technet.com/isablog/archive/2008/03/12/bi-directional-affinity-in-isa-server.aspx

http://blogs.technet.com/isablog/archive/2008/06/24/server-publishing-with-isa-server-2004-2006-and-route-relationship-between-networks.aspx

http://blogs.technet.com/isablog/archive/2007/06/25/rpc-over-http-logging-wildness.aspx

With Forefront TMG 2010 this tool is gone, but no worries, now it is actually much better since is part of the netsh command. Here it is an output of the command that shows the active sessions:

C:\>netsh tmg show connections

Active Sessions:

Source / Destination /

ID Protocol Source Proxy Dest. Proxy 2-way Timeout

-- -------- ----------- ------------ ----- -------

15583 TCP(6) 10.20.20.1:41099 10.20.20.10:445 Yes Yes

4518 TCP(6) 10.20.20.1:41130 10.20.20.10:135 Yes Yes

10.20.20.1:34635

4516 TCP(6) 10.20.20.1:41131 10.20.20.10:135 Yes Yes

10.20.20.1:41130

4522 TCP(6) 10.20.20.1:41132 10.20.20.10:49158 Yes Yes

10.20.20.1:34635

4520 TCP(6) 10.20.20.1:41133 10.20.20.10:49158 Yes Yes

10.20.20.1:41132

4525 TCP(6) 10.20.20.1:41135 10.20.20.10:135 Yes Yes

10.20.20.1:34635

4523 TCP(6) 10.20.20.1:41136 10.20.20.10:135 Yes Yes

10.20.20.1:41135

4529 TCP(6) 10.20.20.1:41137 10.20.20.10:49155 Yes Yes

10.20.20.1:34635

4527 TCP(6) 10.20.20.1:41138 10.20.20.10:49155 Yes Yes

10.20.20.1:41137

15602 UDP(17) 10.20.20.1:49014 10.20.20.10:389 Yes Yes

15603 UDP(17) 10.20.20.1:49015 10.20.20.10:389 Yes Yes

15605 UDP(17) 10.20.20.1:49016 10.20.20.10:389 Yes Yes

15606 UDP(17) 10.20.20.1:49017 10.20.20.10:389 Yes Yes

15601 TCP(6) 192.168.1.154:41129 192.168.1.45:445 Yes Yes

There are much more options available, just use the /? And you will see:

C:\>netsh tmg show /?

The following commands are available:

Commands in this context:

show all - Shows all available information.

show allowedrange - Shows current allowed IP ranges.

show connections - Shows connection element information.

show creations - Shows creation element information.

show global - Shows driver configuration information.

show holdpackets - Shows information about the hold packets in driver.

show nlbhookrules - Shows NLB hook rule and NLB server assigned ranges information.

show usermodepackets - Shows information about the hold packets currently being handled in user mode.

Now go ahead and start playing with this new built in toy.

A Preview of the new Forefront TMG 2010 Administrator’s Companion Book

yuridio — Wed, 13 Jan 2010 23:33:25 GMT

I know that you are anxious for this book to be out since TMG 2010 is already a reality today, I am too. The good news is that it is close, next month it will be available as announced this week on MS Press Blog. While you are waiting for this book to come out, we got a pretty cool preview for you, two full chapters so you can read it and get the feeling of the core book writing style. Besides those two sample chapters, here it is the final and full open cover for the book:

Access http://blogs.msdn.com/microsoft_press/archive/2010/01/13/forefront-tmg-2010-administrator-s-companion-sample-chapters.aspx for more details about the sample chapters and how to download it.

TMG Hardware Requirements

yuridio — Tue, 12 Jan 2010 18:18:16 GMT

Here it is something very cool to start this year: TMG Product Group released the hardware recommendations that applies to Forefront TMG 2010. This article also introduces my friend Tom Shinder (now MSFTE) as technical reviewer. I’m sure that from now on much more articles with his name will appear (welcome Tom !!). Read the full article at: http://blogs.technet.com/isablog/archive/2010/01/12/hardware-recommendations-for-forefront-tmg-2010.aspx

Happy TMG Deployment !!

Error “Object doesn’t support this property or method” while browsing a site published by ISA Server 2006

yuridio — Mon, 04 Jan 2010 18:12:00 GMT

Introduction

This post is about an interesting issue that I recently worked where firewall administrator published an internal web site through ISA Server 2006 and when it tries to load the main page some components didn’t load correctly and the error message below appeared:

The issue didn’t happen when browsing the web server directly (on the internal network) and the browser was able to load the whole page.

Troubleshooting

When we have client side error like this and we are using SSL, it is always a good idea to use HTTP Watch to read the content and see where is failing. It also important to get a working trace and a non working trace so you can compare both results. On the working HTTP Watch trace we see that some DLLs were being called by the application:

Since I couldn’t see this call on the non working trace I decided to try to access the DLL using the external URL (for example: https://www.contoso.com/app/myapp.dll) and it found the file but failed to download.

Different symptom but same root cause

With all these elements I could conclude that the issue was exactly the same one that I that I wrote on post Unable to Download an Office file from a Web Server Published through ISA Server 2006. Here it is the explanation why it was failing:

“In order for Internet Explorer to open documents in Office (or any out-of-process, ActiveX document server), Internet Explorer must save the file to the local cache directory and ask the associated application to load the file by using IPersistFile::Load. If the file is not stored to disk, this operation fails. When Internet Explorer communicates with a secure Web site through SSL, Internet Explorer enforces any no-cache request. If the header or headers are present, Internet Explorer does not cache the file. Consequently, Office cannot open the file.”
From http://support.microsoft.com/kb/316431

Notice the highlighted which emphasize the reason why it can also fail for other files other than office files. The article mentioned above explains how to fix the issue.

New TMG Articles in TechNet Library

yuridio — Wed, 23 Dec 2009 20:51:47 GMT

More great news from Forefront TMG Documentation Team, here it goes a series of new articles that were recently published on Microsoft TechNet Library about Forefront TMG 2010:

Enjoy it !!

Unsupported Configurations for Forefront TMG 2010

yuridio — Wed, 23 Dec 2009 15:31:45 GMT

Here it is a great XMas gift for you !!

After a long road and rounds of updates the Forefront TMG Team just released the new version of the famous Unsupported Configurations, this is great article to read PRIOR to deploy TMG. The reason why I’m emphasizing “prior” is because the worst thing that it can happen is to deploy TMG (or any software) and when a problem arise, you call support and you find out that the way that you did deploy (or configured) is not supported.

Read the full document here: http://technet.microsoft.com/en-us/library/ee796231.aspx

Happy Holidays !!