RO Exchange blogging @ EHL(R)O

IP Addresses are sometimes not shown in Anti-Spam log (get-agentlog). Why?

We have the following scenario: Exchange 2007 in place and Anti-Spam agents enabled on your HUB Servers. OK that’s fine, great move. During your normal administrative operations you come at the point where you have to check the Anti-Spam logs. So you open EMS, enter get-agentlog and... you discover that not all of your logged informations contain the IP address of the sending entity. Like in following example:

[PS] C:\Windows\System32>get-agentlog
Timestamp       : 2/8/2010 5:01:49 PM
SessionId       : 08CC76E6A839D299
IPAddress       : 10.10.10.250
MessageId       : <60a5876e-2ecb-4e77-b367-48ff2c24fd24@EX2K7.mara.mix>
P1FromAddress   : uwe@gmx.de
P2FromAddresses : {uwe@gmx.de}
Recipients      : {laura@mara.mix}
Agent           : Content Filter Agent
Event           : OnEndOfData
Action          : AcceptMessage
SmtpResponse    :
Reason          : SCL
ReasonData      : 6
Diagnostics     :

Timestamp       : 2/8/2010 5:03:40 PM
SessionId       : 08CC76E6A839D29A
IPAddress       :
MessageId       : <5167f33d-2541-44b2-bb9f-91e4c088d1d4@EX2K7.mara.mix>
P1FromAddress   : holger@aol.de
P2FromAddresses : {holger@aol.de}
Recipients      : {laura@mara.mix}
Agent           : Content Filter Agent
Event           : OnEndOfData
Action          : AcceptMessage
SmtpResponse    :
Reason          : SCL
ReasonData      : 6
Diagnostics     :

Why, would be your first question.  There are other logged informations there which shows up all the required fields and values. Some of you might get curious about this, some could get even worried. Btw. the behavior was „reproduced” on W2K8 SP2 and EX2K7 SP2.

So you might see this as a problem. As a matter of fact I had one of my customers, who raise this as a problem and he demand at least an explanation. Let’s have this behavior explained.

At the beginning I was curious, why not all of the logged information’s in Anti-Spam Log were “affected“ about this. So I filter out some Anti-Spam logs and I’ve get the conclusions that only specific sending systems were not logged in with their IP addresses. Particular my customer had his environment set up with 2 CAS and HUB Server on NLB.  I’ve tested it and could shortly exclude NLB solution as a reason for this behavior.

Based on further research I discovered that the corresponding field (IPAddress) is filled under two conditions:

1.      Either if the HUB Servers get the corresponding information when the session is established, so with other words only when this information is available through the session connection.

2.      The second possibility is to fill out the IPAddress field or bypass it based on the evaluation which is done with the Transport Settings, particular regarding the InternalSMTPServer parameter.

On the EMS à Organization level, on HUBà Global Settings à Transport Settings à Properties à on the “Message Delivery” tab you will find the place where you can configure the “Enter the IP addresses of internal SMTP servers. This IP addresses will be ignored by Sender ID and connection filtering”. Pretty obvious isn’t it? J

This setting corresponds in Power Shell to the Get-TransportConfig cmdlet:

[PS] C:\Windows\System32>Get-TransportConfig

ClearCategories                : True
ConvertDisclaimerWrapperToEml  : False
DSNConversionMode              : UseExchangeDSNs
GenerateCopyOfDSNFor           : {5.4.8, 5.4.6, 5.4.4, 5.2.4, 5.2.0, 5.1.4}
InternalSMTPServers            : {10.10.10.250}
JournalingReportNdrTo          : <>
MaxDumpsterSizePerStorageGroup : 18MB
MaxDumpsterTime                : 7.00:00:00
MaxReceiveSize                 : 10MB
MaxRecipientEnvelopeLimit      : 5000
MaxSendSize                    : 10MB
TLSReceiveDomainSecureList     : {}
TLSSendDomainSecureList        : {}
VerifySecureSubmitEnabled      : False
VoicemailJournalingEnabled     : True
HeaderPromotionModeSetting     : NoCreate
WritingBrandingInDSNEnabled    : True
Xexch50Enabled                 : True

Let’s summarize: we discover that in our enabled Anti-Spam log, not all IP addresses are showed as expected. Here in my example I reproduce the problem and you can observe that only mails which came from the 10.10.10.250 IP address are not showed up in the Anti-Spam log.

What does it mean actually InternalSMTPServer? By default all sending mails systems to en Exchange 2007 organization are handled as External. But there are scenarios where you, as an administrator, administer not only your Exchange organization, but also, for example, one or more smart-hosts. All incoming mail flow would come through these smart-hosts and because this, you don’t want the Anti-Spam agents to evaluate your incoming mails that came through smart-hosts, but rather the initial or the relay server which are sending to the smart-hosts. Considering this you will need to instruct Exchange to handle the “foreign” (foreign for Exchange) sending systems as Internal, which would allow then to your Anti-Spam agents to evaluate also the corresponding initial IP addresses of the sender. Another scenario in which you would configure the InternalSMTPServer is in organizations with one or more allowed relay systems configured.

To conclude the behavior is normal: Anti-Spam agents will ignore the internal sending systems regarding the IP address, but will do his job on all mails which came through regardless from which he receives them.  If you need to have all IP addresses listed in the Anti-Spam log, then you need to exclude all particular IP addresses as InternalSMTPServer. Like in following example configuration (actually the default one):

[PS] C:\Windows\System32>Get-TransportConfig

ClearCategories                : True
ConvertDisclaimerWrapperToEml  : False
DSNConversionMode              : UseExchangeDSNs
GenerateCopyOfDSNFor           : {5.4.8, 5.4.6, 5.4.4, 5.2.4, 5.2.0, 5.1.4}
InternalSMTPServers            : {}
JournalingReportNdrTo          : <>
MaxDumpsterSizePerStorageGroup : 18MB
MaxDumpsterTime                : 7.00:00:00
MaxReceiveSize                 : 10MB
MaxRecipientEnvelopeLimit      : 5000
MaxSendSize                    : 10MB
TLSReceiveDomainSecureList     : {}
TLSSendDomainSecureList        : {}
VerifySecureSubmitEnabled      : False
VoicemailJournalingEnabled     : True
HeaderPromotionModeSetting     : NoCreate
WritingBrandingInDSNEnabled    : True
Xexch50Enabled                 : True

Named Properties problems: A workaround

If you are already familiarized with the named property concept on Exchange 2003…well this usually means that you had to deal with all the trouble it may cause. If not, there are a couple of links that might help you in building an understanding of what we are talking about:

http://msexchangeteam.com/archive/2009/04/06/451003.aspx

http://msexchangeteam.com/archive/2009/06/12/451596.aspx

http://technet.microsoft.com/en-us/library/bb851492.aspx

To make a long story short, a named property is the way in which Exchange deals with all those X-headers that exist out there in the Internet and that are appended on a message by almost every application that may touch the mail (anti-spam, anti-virus, Outlook add-ons)

The need for creating these named properties derived from the following clear fact: Exchange talks MAPI, but out there, in the Internet, RFC822 is the standard for message creation and delivery.  Thus, once an Internet mail reaches an Exchange organization boundary, a conversion process must take place:  it’s during this conversion process that we transform rfc822 headers into MAPI properties and, of course, vice-versa.  

So far so good but here comes the problem: there is only so much named properties that can be created for each database, and sooner or later, we will hit that upper limit. Once the limit is reached, all sorts of problem will start to happen; the most common scenario is the following: users hosted on the database confronted with named properties exhaustion will not be able to receive mails. The store will throw a NDR for each mail destined to a user on that database, because he will not be able to convert the message headers into MAPI properties (i.e "The message reached the recipient's e-mail system, but delivery was refused.  Attempt to resend the message.  If it still fails, contact your system administrator. <servername.domain.com #5.2.1>"). This will happen for both internal and external generated messages, because Outlook is also a great consumer of named properties and he will also suffer from the lack of available named properties.  

The hard limit for named properties is set at 32,766 entries and recovering from hitting it is a pain: basically you will have to move out all the users from that database , scratch it and then move back the users onto this new ‘’blank ’’ database. (More info on this process: http://technet.microsoft.com/en-us/library/bb851495.aspx )

 Now what we would like to achieve is to find a way that will allow us to avoid such a troublesome scenario….and luckily we did. It’s actually quite simple:

The idea is to install an Exchange 2003 bridgehead server into your organization because in this way we can move the message conversion process from the server that hosts the mailbox onto the bridgehead. The bridgehead will be the first server in the Exchange organization that will deal with RFC822 headers and the conversion of these headers into named properties will take place here instead of the mailbox server.  In this way, once we will ran into named properties issues, we will have to deal with those on the databases hosted on the bridgehead server and not on those on the mailbox server. So instead of having to scratch a database with users, we will scratch instead a database on the bridgehead server that hosts no users.

 It’s safer, cleaner and less time consuming.

Enjoy.

Public Folder Replication Tip: Replication Message Size Limit value

When troubleshooting public folder replication one should always keep in mind that the replication process is message based.

If you find that you have some problems with the replication of public folders then your troubleshooting efforts should always start with these 2 awesome posts by Bill Long: http://msexchangeteam.com/archive/2006/01/17/417611.aspx and http://msexchangeteam.com/archive/2006/01/19/417737.aspx .

They are pretty comprehensive and they cover almost all the troubleshooting techniques you will be likely to apply when dealing with public folder replication issues.

I’ve said ‘’almost all’’ because recently I have just found another technique that proved to be pretty useful in the following scenario:

Let’s say you have a public folder for which we have 2 replicas on 2 different servers. Server A has the up to date replica but it seems that we have trouble replicating some old content to the server B replica (I had almost 50 old posts that would not replicate).

New content replicates just fine between the servers and also when you turn up diagnostic logging for Replication Incoming and Replication Outgoing on the servers you notice that when you force replication, server B sends out a backfill request, server A responds with a backfill response that contains the items that don’t replicate but the response never makes it back to server B and those items are never processed, thus never replicated. Looking in at the SMTP queues won’t help much because they are clear on both servers.

It looked like the message carrying over the items that needed to be replicated from server A to server B just vanished. In my particular case, the only lead I got came from the message tracking center, which I’ve used to track the mail flow between the 2 public folder databases; looking at this, I've noticed that the backfill response that contained the items to be replicated never left server A cause it had ended up in a “Advanced Queuing Failed To Deliver Message” error.

Now comes the tricky part: as looking at a SMTP trace didn’t really help isolating the problem, we decided to use a very interesting feature of public folder replication in order to try to find out what was wrong whit this message – this feature is the Replication message size limit value.

You should know that when replicating public folder data between servers, Exchange packs multiple content changes into one large replication message that is sent over to the replication partner server. By default we pack changes in chunks of 300 KB, thus composing 1 replication message that is ready to be transferred out (more info on this at http://technet.microsoft.com/en-us/library/aa997291(EXCHG.65).aspx ).

We decided to play on this feature by lowering the Replication Message Size Limit on the Public Folder store of server A to 1 KB, hoping that we will in this way send out one content change (or public folder post) per replication message. The goal was to see if by any chance we weren’t dealing with a corrupt post that would obstruct our replication process by rendering the whole replication message corrupt. This technique also should allow us to precisely identify the corrupt item because in the end, it would be the only one that would still not replicate.

So instead of packing multiple content changes (public folder posts) into one 300 KB replication message, we decided to force Exchange to replicate each content change (public folder post) one by one.

The result: we got all the posts replicated to server B except one particular item that would still not replicate.  This was our corrupt item ( aka bad guy) that prevent us replicated another 50 or so valid posts only because the whole replication message was saw as corrupt by Exchange and dropped as a result.

Stub objects remain on source mailbox servers after mailbox move

If you are moving mailboxes between servers in Exchange 2003 SP2 or Exchange 2007 and your clients are Outlook 2007, chances are you’ve already run into this problem before. The KB article 940012 documents this behavior for Exchange 2003 SP2 and a hotfix is available on the same page :

http://support.microsoft.com/kb/940012

However, even after applying the hotfix on both the source and the target server of the Move Mailbox operation, you may still encounter the problem for mailboxes used with Outlook 2007. This happens because Outlook 2007 creates a search folder called Tracked Mail Processing at the same level as Top of Information Store in the mailbox folder hierarchy.

When the folders of the mailbox located on the source server are deleted (recursively, all the child folders are deleted from the root folder), usually the Top of Information Store is deleted first , which leaves the Tracked Mail Processing folder with and unknown folder id (FID) and therefore impossible to remove.

What the hotfix in KB article 940012 actually does is make the events 9533, 10001 and 10002 visible so that the administrators become aware that they are facing this problem. The administrator then has to apply one of the workarounds:

1.       Move the mailbox back to the original mailbox store. This operation will fail but the disconnected object will be deleted.

2.       Wait for the Maintenance to run on the database where the disconnected mailbox is located. This will remove the object.

If you do not wish to wait, you can speed up this process by setting the deletion settings for the mailboxes for 0 days (Keep deleted mailboxes for …days) and then schedule the maintenance to run sooner (by default it runs between 2:00 AM and 6:00 AM every day)

Bulk change advanced properties of internet message format domains - Exchange 2003

In order to bulk enable the options "Allow out of office responses, Allow delivery reports, Allow non-delivery reports, Preserve senders display name on message" (as they are shown in the next figure) in the advanced tab in the properties of all domains you have listed in Internet Message Formats please proceed as follows:

Use ADSIEdit to export all distinguished names of the objects. To do so navigate to:

*** Configuration -> Services -> Microsoft Exchange -> Your Organization Name  -> Global Settings  -> Internet Message Formats

*** Right click on CN=Internet Message Formats and "Export List … "

Save this list as Tab Delimited txt file.

Open this file with Excel and delete the first row (name, class, distinguished name) and the first two columns (values for name and class).

After deletion ONLY the DNs should be in the file! Those should be all in the first column (A) starting from the first row (1):

Now for every domain we need to have value 25 for following attribute "msExchRoutingAcceptMessageType" which actually defines the settings on this options (see first figure). In order to find the right value, it is recommended to make the settings for one domain and to check the attribute using ADSIEdit.

In order to do so, for every domain the following syntax is needed:

dn: CN=a,CN=Internet Message Formats,CN=Global Settings,CN=Erste Organisation,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=timisoara,DC=lab

changetype: Modify

replace: msExchRoutingAcceptMessageType

msExchRoutingAcceptMessageType: 25

-

This can be achieved using a macro (vbs script) to insert these rows:

Sub insertrows()

[a1].Activate

' the script starts at cell A1

hasvalue = ActiveCell.Value

' uses the variable hasvalue to determine where to stop

Do While Not (hasvalue = "")

' it will stop where the next cell has no value

ActiveCell.Value = "dn: " + ActiveCell.Value

' to meet the neeed syntax dn: has to be added in front of the distinguished name

ActiveCell.Offset(1, 0).Select

' moving one cell down

Selection.Resize(rowsize:=1).Insert Shift:=xlDown

' inserting a new (first) row

ActiveCell.Value = "changetype: Modify"

' inserting the value in the first cell of this new row

ActiveCell.Offset(1, 0).Select

' moving one cell down

Selection.Resize(rowsize:=1).Insert Shift:=xlDown

' inserting a new (second) row

ActiveCell.Value = "replace: msExchRoutingAcceptMessageType"

' inserting the value in the first cell of this new row

ActiveCell.Offset(1, 0).Select

' moving one cell down

Selection.Resize(rowsize:=1).Insert Shift:=xlDown

' inserting a new (third) row

ActiveCell.Value = "msExchRoutingAcceptMessageType: 25"

' inserting the value in the first cell of this new row

ActiveCell.Offset(1, 0).Select

' moving one cell down

Selection.Resize(rowsize:=1).Insert Shift:=xlDown

' inserting a new (fourth) row

ActiveCell.Value = "-"

' inserting the value in the first cell of this new row

ActiveCell.Offset(1, 0).Select

' moving one cell down

Selection.Resize(rowsize:=1).Insert Shift:=xlDown

' inserting a new (fifth) row

ActiveCell.Value = ""

' inserting the value (actually an empty row) in the first cell of this new row

ActiveCell.Offset(1, 0).Select

' moving one cell down

hasvalue = ActiveCell.Value

' hasvalue is used to check if there is another row to loop

Loop

End Sub

Run the script, after which the excel sheet should look similar to the following figure:

Copy the first column by marking (click on top of the column A) and using CTRL+C. Paste everything into a txt file.

Import this file into the active directory (AD) using ldifde:

ldifde -i -f <text file name> -s <domain controller>

***************************************************************************

Disconnected mailboxes. Are they true mailboxes or... what?

Let's imagine this scenario:

In an Exchange 2007 organization there is an email address policy that is defined for every Exchange recipient, and some precanned filtering conditions.

Everybody knows the theory... The RUS API is called for all the cmdlets that modify the recipients, and the proper matching policies should apply.

Now, let's consider an AD user account that is not mail enabled, but which has attributes that match our policy.

If we create a new mailbox for that account on an Exchange 2007 server using the graphic console or Power Shell, everything will behave as expected: The email addresses are set instantly as the policy dictates.

But:

If we have an existing disconnected mailbox and we try to connect it the the account that we were talking about, we will face a surprise: The email address will not be properly set according to the policy, although the resulted object match the policy.

Later, if we modify that object, or we generally update that policy at organization level, the email will be set up correctly.

What is happening:

The actual command is connect-mailbox. Its subject is the disconnected mailbox AD object, which is not treated as an Exchange mailbox. It is something else, and therefore the policy as we defined will not apply at this stage. Later, further cmdlets have an Exchange recipient as subject and the RUS API will apply the policy as expected.

What we can do:

We can simply change the policy by selecting all recipients, not only Exchange mailboxes. This way the policy will match even the disconnected mailbox and everybody should be happy.

If we are too scrupulous:

We can use a custom OPATH expression to filter the policy:

[PS] Get-EmailAddressPolicy strategieX | Set-EmailAddressPolicy -RecipientFilter { ( Company -eq 'valeurX'  -and  ObjectClass -eq 'Mailbox'  -and  ObjectCategory -eq 'Person'  -and  Alias -ne $null  -and  ServerLegacyDN -ne $null ) }