Recently, the support group worked on a Microsoft® Internet Security and Acceleration (ISA) Server case that was very interesting, mainly because the symptom described by the customer indicated that the support group should go in one troubleshooting direction. Later, the support group learned that other issues existed, and what the customer described initially was only part of the real problem.
The customer’s description: “We can’t receive e-mail messages from the Internet because ISA Server is not allowing traffic to pass on port 25.”
The following figure displays the customer's reported environment.
Figure 1 Customer environment
The customer previously had a Simple Mail Transfer Protocol (SMTP) relay on the perimeter network (also known as DMZ, demilitarized zone, and screened subnet) that was responsible for receiving messages from the Internet. Now the customer wants to directly publish the Microsoft Exchange server to allow the e-mail messages to go directly to it. The customer deleted the old rule associated with the SMTP relay and created a new mail publishing rule just for the SMTP protocol pointing to the Exchange server.
After this configuration change, the Exchange server was receiving e-mail messages from internal users and also was able to send e-mail messages to the Internet. The only problem was receiving e-mail messages from the Internet.
2.1. Initial tests
To confirm the customer’s problem, the following tests were performed.
Use Telnet on port 25 from an Internet host
Use Telnet on port 25 locally from the ISA Server computer itself
Ran the command netstat -nao from the ISA Server console to see who was listening on port 25
Correctly showed the ISA Server computer
Reviewed the MX record to see if it was pointing to the ISA Server computer
MX record was correct
These test results showed clearly that the issue was somehow on the ISA Server computer.
3. Collecting and analyzing data
To start data collection, the following components were configured:
· Enabled netmon trace on the external network interface card (NIC) of the ISA Server computer.
· Enabled ISA Server logging to log traffic where the destination port was equal to 25.
Next, a Telnet connection on port 25 from an Internet host to the external IP address of the ISA Server computer was established.
The netmon trace showed only that the external host was trying to connect and that the ISA Server computer did not answer, which was expected. But, ISA Server logging showed the following entry:
VALID_IP_ISA 10.10.10.20 25 SMTP Server Failed Connection Attempt SMTP IN 0x8007274d External DMZ - SRVISA1 Firewall
The following table describes each part of this log entry.
ISA Server computer external IP
IP address of the old SMTP relay server on the DMZ
Failed Connection Attempt
Name of the rule (the old rule that was previously deleted)
Target network interface
ISA Server computer name
Type of log
Importantly, this log entry shows that the ISA Server computer is actually allowing the traffic to pass, and that the problem is that the ISA Server computer is trying to communicate with the old SMTP relay (10.10.10.20), which no longer has the SMTP service.
Based on this, the fact that the error 0x8007274d is being received makes a lot of sense. This error means: "No connection could be made because the target machine actively refused it."
At this point, the support group knew that the ISA Server computer was allowing the traffic. Solving the issue was a matter of updating the configuration and verifying that the customer has a new rule in place to be used for this purpose.
4. Understanding the customer environment
The initial scenario explained by the customer was clear in most areas, however some key elements needed clarification. To better understand these elements, it was necessary to request more information from the customer as shown in the following Q and A:
· Is the ISA Server computer running on an array with more than one node?
· How many nodes?
· Are you using Network Load Balancing (NLB) in Integrated mode?
· Which server is the Configuration Storage server?
o The server where the capture was done (SRVISA1).
· Are these servers members of the corporate domain where the users are logging on?
o No. These servers are running in workgroup mode.
Based on the additional scenario information, the following conclusions were made:
· If there is a problem on the Configuration Storage server, it is possible that the rules are not up to date.
· When using ISA Server in a workgroup environment, a valid certificate to authenticate the array members is needed.
For more information about the difference between running ISA Server in workgroup and domain modes, see the white paper "Deployment Recommendations for ISA Server 2004 in a Workgroup or Domain" at the Microsoft Technet Web site.
5. Reviewing the requirements
Steps 1 through 5 from the article "Troubleshooting Configuration Storage Servers in ISA Server 2004 and ISA Server 2006 Enterprise Edition" at the Microsoft Technet Web site were performed. Based on the results of these steps and by reviewing the requirements for this kind of scenario, it was possible to see that SRVISA2 was not able to connect to the Configuration Storage server.
During step 5, it was verified that the certificate used to authenticate the array member was expired. By reviewing the logs, it was possible to see the following warning event:
Event Type: Warning
Event Source: Microsoft ISA Server Control
Event Category: None
Event ID: 21238
ISA Server cannot connect to the Configuration Storage server srvisa1.contoso.com for one of the following reasons:
- The Configuration Storage server is not available.
- General networking or authentication issues.
- A policy misconfiguration on the array.
To fix this issue, a request for a new certificate was sent to the certification authority (CA). For more information about how to import a new certificate to ISA Server, see the article "Obtain a server certificate (Enterprise Edition)" at the Microsoft Technet Web site.
After completing this procedure, the Configuration Storage server was restarted. A different error, 0x8009030d, was received. Error 0x8009030d means: “The credentials supplied to the package were not recognized". This is a security issue related to the access rights of the certificates. Only users who create or import the certificates have access to those certificates.
At this point, ISACertTool was used to import and register the certificate. For more information, see "ISACertTool for Internet Security and Acceleration (ISA) Server 2004 Enterprise Edition" at the Microsoft Download Center. This procedure worked correctly, and then both array members were able to replicate and the rule was updated.
The issue occurred almost one month after the certificate was expired for two reasons:
Customers can avoid this kind of situation, if they have a management system in place, such as Microsoft Operations Manager, which would warn them of this issue.
Special thanks to Alan Wood (ISA SEE from Product Support Services Texas) who was my mentor on this difficult, four hour case.
Support Engineer – Latin America Team – Platforms
Very good article!
The customer shows a number of deviations from ISA Firewall security best practices:
*CSS should not be on an array member
*The ISA Firewalls should be joined to the domain to enhance security
*They are allowing connections to the Exchange Server and not an SMTP relay - VERY UNSECURE!
Another good reason to use ISA servers joined to the domain, as opposed to workgroup mode!
I noticed that new functions (loggin tab) were installed but the Firewall Service still stopping