Bill Long's Exchange Blog

New Version Of ExFolders Adds A Clear Permissions Bulk Operation

Bill Long [Exchange] — Mon, 04 Jun 2012 20:29:58 GMT

In October of last year, I updated ExFolders with a new Clear Permissions option in order to fix folders with non-canonical ACLs. I described that update in an old blog post you can find here: http://blogs.technet.com/b/bill_long/archive/2012/05/03/3460823.aspx.

Since then, one of the most requested features has been a way to run this on a whole subtree of folders. Unfortunately, it appears that customers are seeing ACL problems more often than I expected. So today, I’ve released a new build of ExFolders that includes a way to do this. In the latest version, when you go to Tools->Custom Bulk Operation and you hit Add, you’ll find an option to Clear Folder Permissions.

One of the reasons I didn’t add this feature initially is that it is such a destructive operation. Clearing the permissions on a whole tree of folders is not something to do casually without considering the consequences. In a mailbox, you are taking away permissions on folders that users have purposely shared out. In public folders, you are literally removing all access rights for everyone, making it impossible for your clients to access the public folders.

However, when you add the Clear Folder Permissions bulk operation, you’ll notice it has a checkbox, which is selected by default, called Restore previous permissions after clearing. When this is selected, ExFolders will clear the permissions and save those changes, and then it will attempt to set the permissions that were present before and save the changes again. This should allow it to fix non-canonical ACLs without requiring the administrator to set all the permissions back the way they were before.

Of course, as with any bulk permissions change, it’s a good idea to take a permissions export first just in case something goes wrong. However, I expect this option will make importing after a mass Clear Permissions unnecessary in most cases. And because this option is selected by default, I’m hoping that if someone accidentally runs this against a set of folders, the impact will be minimal.

How To Access System Folders From EWS Managed API

Bill Long [Exchange] — Wed, 30 Nov 2011 15:37:40 GMT

The EWS Managed API provides a simple way to get to the Public Folders – you simply bind to the Microsoft.Exchange.WebServices.Data.WellKnownFolderName.PublicFoldersRoot, like so:

$pfRootName = [Microsoft.Exchange.WebServices.Data.WellKnownFolderName]::PublicFoldersRoot
$pfRoot = [Microsoft.Exchange.WebServices.Data.Folder]::Bind($service, $pfRootName)

But what if you want to access the Free Busy folder, or the Offline Address Book folders? How do you get to the System Folders? The WellKnownFolderName enumeration doesn’t have a value for the System Folders Root.

Fortunately, there’s a relatively simple trick you can use to get there. EWS gives you the option of binding to a folder by ID. The ID of the Public Folders Root will always end with the number 1, and the ID of the System Folders Root will always end with the number 2. So you can simply bind to the Public Folders Root, get the ID, swap out the number 1 with a number 2, and voila! You have the ID of the System Folders root. Here’s how you do it:

$pfRootName = [Microsoft.Exchange.WebServices.Data.WellKnownFolderName]::PublicFoldersRoot
$pfRoot = [Microsoft.Exchange.WebServices.Data.Folder]::Bind($service, $pfRootName)
$pfRootIdString = $pfRoot.Id.ToString()
$pfRootIdBytes = [System.Convert]::FromBase64String($pfRootIdString)
$pfRootIdBytes[$pfRootIdBytes.Length - 4] = [byte]0x02
$systemRootIdBase64String = [System.Convert]::ToBase64String($pfRootIdBytes)
$systemRootId = new-object Microsoft.Exchange.WebServices.Data.FolderId($systemRootIdBase64String)
$systemRoot = [Microsoft.Exchange.WebServices.Data.Folder]::Bind($service, $systemRootId)

At that point you have the System Folders root in $systemRoot, and you can proceed from there. Hopefully this saves some of you some time. It was interesting trying to figure out how to do this!

Removing Unresolved SIDs in Exchange 2010

Bill Long [Exchange] — Sun, 23 Oct 2011 08:20:00 GMT

In this post, I'm going to describe how to remove unresolved SIDs from public folders in Exchange 2010. But first, let’s talk about what they are and why we care about them.

What are unresolved SIDs?

When you view the permissions on a file in Windows or an object in Active Directory, you get a nice friendly list of names, each one with its associated access rights displayed. However, the security descriptor does not actually contain the names of the users – it only contains their SIDs. When an application such as Explorer or ADUC displays the permissions, it attempts to resolve each SID to a name, so when you look at it you can make some sense of what you’re seeing.

An unresolved SID is a SID that no longer resolves, usually because the user has been deleted. If you look at an object in ADUC, it looks like this:

In Exchange, folders have security descriptors that are very similar to the ones in Windows, except that things are ordered differently. But you still have SIDs and rights, and when a SID doesn’t resolve anymore, you’ll see that in Outlook:

Why do we care about them?

Unresolved SIDs don’t cause a problem; they just look ugly. Some customers want to remove them for that reason alone, and some customers have other reasons. A lot of customers used PFDAVAdmin for this, but there was a specific reason PFDAVAdmin removed these.

Back in Exchange 2000 and Exchange 2003, it was possible to expose the Information Store database as a drive in Windows. This was the infamous M: drive. It became infamous because applications that were not Exchange-aware would access folders through the M: drive and mess up the permissions. Above, I briefly mentioned that things are ordered differently in an Exchange ACL than a Windows ACL. Applications that looked at the folders in M: and thought they were plain old Windows folders would reorder the stuff in the security descriptor to fit the Windows format. This meant it no longer matched the Exchange canonical format, and the permissions wouldn’t work as expected.

After an application had run through the M: drive and made all the ACLs non-canonical, it was a huge pain to go through and manually correct them. This was one of the main reasons I wrote the PFDAVAdmin tool. PFDAVAdmin provided an easy way to put all the ACLs back in canonical order.

One thing about Exchange canonical order is that the SIDs need to go in a particular order based on the object type. Users need to be at the top of the ACL, followed by groups. If you’re trying to reorder the ACL to make it canonical again, and you run into an unresolved SID, this presents a problem. Since you can’t resolve the SID, you don’t know what type of object it is, so where should it go?

There are a few ways you could approach this problem. For instance, you could just pretend it’s a user SID. If the object has been deleted, then the SID will never again resolve, so it won’t matter. If the SID is only unresolvable because of a temporary issue with a DC or something, then this could result in the ACL becoming non-canonical once the SID resolves again.

However, the approach I took with PFDAVAdmin was to have it remove the unresolved SIDs. I couldn’t be sure what they were and why they were unresolvable, and the whole point was to force the ACL into a canonical state, so this seemed like a sensible approach.

Why doesn’t ExFolders remove them?

When I rewrote the guts of PFDAVAdmin for Exchange 2010, I renamed it ExFolders. While the GUI looks almost identical, the code underneath is not the same and required a lot of rewriting. When it came time to look at the “Fix Folder DACL” code and decide whether to move it into ExFolders, I just didn’t see any need for it. The widespread non-canonical ACL issues from the Exchange 2000/2003 days were gone, so I left this functionality out of ExFolders entirely. Unresolved SID removal was a side-effect of fixing the DACL, so it likewise fell by the wayside.

How can I remove them now?

OK, so you want to remove them from your public folders in Exchange 2010, but there’s no tool to do that anymore. Fortunately, it’s easy to remove them with a simple script. I’ve included one below. This script will be far slower than PFDAVAdmin was, but it will get the job done.

If you run it with the following syntax, it will only report the unresolved SIDs:

.\Check-UnresolvedSIDs.ps1 -Server MyPFServer

If you want it to actually remove them, you must add the –Remove parameter:

.\Check-UnresolvedSIDs.ps1 -Server MyPFServer -Remove $true

The output will look something like this:

\Folder1: Checking folder
\Folder2: Checking folder
\Folder2 has unresolved SIDs:
     NT User:S-1-5-21-1016452943-3003584382-490080582-2161
\Folder3: Checking folder
\MyNewFolder: Checking folder
\SomeOtherFolder: Checking folder
\SomeOtherFolder has unresolved SIDs:
     NT User:S-1-5-21-1016452943-3003584382-490080582-1136
     NT User:S-1-5-21-1016452943-3003584382-490080582-1135

If you add the –Remove parameter, it will also tell you that it’s removing them.

Here’s the script. Enjoy!

# Check-UnresolvedSIDs.ps1
#
# This script will remove all unresolved SIDs from public folder permissions.
# This will take quite some time to run against a large hierarchy.

param([string]$Server, [bool]$Remove)

if ([System.String]::IsNullOrEmpty($Server))
{
"You must specify a server."
return
}

function CheckUnresolvedForFolder($folder)
{
    ($folder.Identity.ToString() + ": Checking folder")
    $unresolved = Get-PublicFolderClientPermission -Identity $folder -Server $Server| WHERE { $_.User -like "NT User:S-*" }
    if ($unresolved -ne $null)
    {
        ($folder.Identity.ToString() + " has unresolved SIDs:")
        foreach ($item in $unresolved)
        {
            ("     " + $item.User.ToString())
        }

        if ($Remove)
        {
            ("     Removing the unresolved SIDs...")
            foreach ($item in $unresolved)
            {
                $item | Remove-PublicFolderClientPermission -Identity $folder -Server $Server -Confirm:$false
            }
            ("     Unresolved SIDs were removed.")
        }
    }
}

function DoFolderRecursive($folder)
{
    CheckUnresolvedForFolder($folder)
    $subfolders = Get-PublicFolder -Identity $folder -GetChildren
    if ($subfolders -ne $null)
    {
        foreach ($subfolder in $subfolders)
        {
            DoFolderRecursive($subfolder)
        }
    }
}

$ipmSubtree = Get-PublicFolder "\" -Server $Server
DoFolderRecursive($ipmSubtree)
$nonIpmSubtree = Get-PublicFolder "\NON_IPM_SUBTREE" -Server $Server
DoFolderRecursive($nonIpmSubtree)

"Done!"

New Version of ExFolders Fixes Non-Canonical ACLs

Bill Long [Exchange] — Sun, 23 Oct 2011 07:07:00 GMT

When I wrote ExFolders, I thought the non-canonical Exchange ACL problems were permanently behind us in Exchange 2010. For this reason, ExFolders did not include any functionality to deal with non-canonical ACLs. It turns out I was overly optimistic.

In the last few weeks I’ve seen a couple of cases where customers ended up with non-canonical ACLs on public folders in Exchange 2010. They were seeing events like this:

Log Name: Application
Source: MSExchangeIS
Event ID: 9775
Task Category: General
Level: Warning
Description:
The mailbox '' contains a folder 'My Public Folder' with a security descriptor that violates the canonical format.

Trying to modify the permissions with Add-PublicFolderClientPermission would result in this error:

MapiExceptionNonCanonicalACL: Unable to modify table. (hr=0x80004005, ec=2409)

Each affected environment only had one folder in this state, so it does not appear to be a widespread problem, such as when something scanned the M: drive back in the Exchange 2000 days.

Even so, customers need a way to fix these problems. Since PFDAVAdmin fixed this for older versions of Exchange, it made sense to include this functionality in ExFolders for Exchange 2010.

Note, however, that what I’ve added to ExFolders is very different from the “Fix Folder DACLs” option in PFDAVAdmin. “Fix Folder DACLs” would run through your whole folder hierarchy looking for all kinds of problems and adjusting the DACL portion of the ACL.

What I’ve added in the October 20th, 2011 release of ExFolders for 2010 Sp1 is something much more limited in scope. When you right-click on a folder, you’ll see a “Clear Permissions” option. The previous build actually had this option as well, but it would not work on non-canonical ACLs. In this latest version, the “Clear Permissions” option accesses the raw security descriptor (similar to what PFDAVAdmin used to do) so it can clear it even when it’s non-canonical. You can use this to set one specific folder back into a good state and then manually set the permissions to whatever they should be.

This will be good enough to address these one-off non-canonical ACLs. I don’t expect we’ll be seeing widespread problems with this, but I’ll be keeping an eye on it. Hopefully there will be no need to introduce a bulk folder ACL fix again.

Investigating complex LDAP filters in Exchange

Bill Long [Exchange] — Thu, 06 Oct 2011 04:56:29 GMT

Customers migrating from Exchange 2003 to 2007 or 2010 often use my ConvertFrom-LdapFilter script to do very literal conversions from their old LDAP filters to the new OPATH filter syntax. In most cases, that works, but sometimes you’ll run across a filter like this:

My script will refuse to process this filter because of the bitwise filters on samAccountType and groupType. So what to do?

The first thing you should do is step back and say, “Who do I want this filter to match? What was the original intent of this filter?” If you’re not sure, you may be able to get an idea by breaking down the filter logic like this:

(&
    (&
        (&
            (|
                (&
                    (objectCategory=person)
                    (objectSid=*)
                    (!samAccountType:1.2.840.113556.1.4.804:=3)
                )
                (&
                    (objectCategory=person)
                    (!objectSid=*)
                )
                (&
                    (objectCategory=group)
                    (groupType:1.2.840.113556.1.4.804:=14)
                )
            )
        )
        (objectCategory=user)
        (memberOf=CN=SomeGroup,CN=Users,DC=contoso,DC=com)
    )
)

Now that we can see how all the matching conditions relate to each other, we have a couple of questions to answer before we can totally understand the filter. We need to know what these two conditions mean:

(!samAccountType:1.2.840.113556.1.4.804:=3)
(groupType:1.2.840.113556.1.4.804:=14)

Searching the KB, we can find that 1.2.840.113556.1.4.804 is the bitwise OR match condition, described in KB 269181. This means that if any one of the specified flags is found, it’s a match.

Looking at the samAccountType filter, we have a value of 3 in decimal. Converting this to binary shows that we have the two rightmost bits set:

00000011

So this bitwise OR filter matches on the 0x1 bit or the 0x2 bit. But the condition starts with a NOT (that’s what the ! means), so we’ll only match objects where neither of those bits are set. Now we need to look up what all these bits mean for the samAccountType filter. We can find those definitions here: http://msdn.microsoft.com/en-us/library/cc228417(v=PROT.13).aspx. A quick look at the list reveals that if we’re not matching anything with 0x1 or 0x2, that leaves 3 values that we will match on:

SAM_GROUP_OBJECT
SAM_ALIAS_OBJECT
SAM_USER_OBJECT

So basically, the whole purpose of that condition is to say, “only match on groups, contacts, or users”.

Now that we have that part figured out, we need to find out what this groupType condition means. Again, we have a bitwise OR. This time the flags are 14 in decimal, which gives us the following bits:

00001110

The 0x2, 0x4, and 0x8 bit are set. We can find the definitions of the groupType values here: http://msdn.microsoft.com/en-us/library/cc228506(v=PROT.13).aspx. This tells us that this condition matches groups that are either Universal, Resource, or Account groups. I have no idea what these flags mean (aside from Universal), but I don’t have a single group in my lab that isn��t one of these types.

Now that we understand the whole filter, we can explain the logic like this:

The object must match any one of the following conditions:

It has a SID and is either a user, group, or contact
It does not have a SID
It is a group that has either the Universal, Resource, or Account flag set

The object must also match both of the following conditions:

objectCategory is user
It’s a member of the specified group

At this point, it’s pretty clear that the whole idea of this filter is just to match members of a group. The filter is extremely overcomplicated, and the bitwise filters that we had to spend so much time investigating are largely pointless. This filter was almost certainly built using Exchange System Manager in Exchange 2000 or 2003. That management tool was infamous for its overly-complex filters, and it’s unlikely that a human would purposely make a filter this obtuse. If I was writing my own LDAP filter to accomplish this in Exchange 2003, I would simply use:

(&(mailnickname=*)(memberOf=CN=SomeGroup,CN=Users,DC=contoso,DC=com))

But now that we know what the filter is supposed to do, we can just replace it directly with a much more elegant OPATH filter and call it a day:

memberOfGroup -eq CN=SomeGroup,CN=Users,DC=contoso,DC=com

Setting the RecipientFilter on the policy to that value will essentially accomplish the same thing as that huge, messy LDAP filter, and it’s a whole lot easier to read and quickly ascertain what it does.

For more on translating LDAP filters to OPATH, see this article: http://technet.microsoft.com/en-us/library/cc164375.aspx.

Identifying Unresolved LegacyExchangeDNs via EWS and Powershell

Bill Long [Exchange] — Fri, 16 Sep 2011 17:42:47 GMT

I recently worked with a customer who had inadvertently deleted all their user accounts (and thus their Exchange mailboxes), and with no backup available, they had to recreate them. Talk about a nightmare! After they did so, they were able to get their email back, but they discovered that replying to email messages from before the problem resulted in a non-delivery report.

This is because of a fairly well-documented behavior. Email addresses in the From, To, and other fields are resolved to legacyExchangeDNs and stored in the message. When you reply to a message, we expect to be able to resolve that legacyExchangeDN. If we can’t, it causes an NDR. In various migration scenarios where the legacyExchangeDN of a user changes, we populate the user’s proxyAddresses with an X500 address that contains the old legacyExchangeDN. This allows the old value to resolve, preserving the ability to reply to old messages.

In this case, when the users were recreated, they got new legacyExchangeDNs, which broke the ability to reply to old email. We needed to somehow get the old leg DNs back, but we didn’t know what they all were, and having to manually poke around in mailboxes looking for them was not realistic.

To solve this problem, I wrote a script to scan a mailbox and output any unresolved legacyExchangeDNs. There are a few interesting parts to this script, so I figured I would post it in case others find it useful (and so I can easily refer back to it in the future).

The first interesting part is the way it gets the user’s password. I needed to get the user’s password without displaying it to the console. Unfortunately, EWS won’t seem to accept a SecureString, so I couldn’t use Get-Credential or similar approaches to ask the user for his password. To solve this, I had to adapt a routine I found in a couple of other blog posts so it would work in Powershell. You’ll find that near the top of the script. DJ found a much simpler way to accomplish this, so the 15ish lines of code that I had have been replaced with a single line that gets the password without showing it. Thanks DJ!

The second interesting part is that it demonstrates how to open another user’s mailbox folders. The script will ask you for one email address for authentication purposes, and then it will ask you for another user’s SMTP address, which is optional. If you enter the other user, the script will scan that user’s folders instead of the authenticating user’s folders.

The third interesting part is that EWS will not return the full recipient information as part of a FindItems() call – you only get back some basic information, which wasn’t enough to tell me if there was an unresolved legacyExchangeDN. To solve this, I had to actually bind to each individual email message. I did this using a specific property set of only a few properties that I was interested in, but it still made the script quite slow.

I wanted this script to work against both on-premises and Office 365 or BPOS environments. Unfortunately, Autodiscover to a cloud environment results in a redirect that can’t be handled in Powershell (or if it can, I haven’t figured out how). You have to create a callback to validate the redirection URL, and while I found that I could cast a script block to a delegate, I couldn’t get the script block to handle this properly. So if you’re connecting to the cloud, you have to manually specify your EWS URL. DJ also showed me how to make Autodiscover to the cloud work in Powershell, and I’ve updated the script below. Thanks again DJ! This script is really much better now thanks to him.

The script will scan the Inbox and any subfolders, as well as Sent Items and Deleted Items, looking for unresolved leg DNs. Any it finds will be written out to a CSV file. It caches what it finds so that duplicates are not written to the CSV.

Anyway, here’s the script!

# Scan-MailboxForLegDNs.ps1

Import-Module -Name "C:\Program Files\Microsoft\Exchange\Web Services\1.1\Microsoft.Exchange.WebServices.dll"

$hostName = Read-Host "Hostname for EWS endpoint (leave blank to attempt Autodiscover)"
$outputFile = Read-Host "Output file name"
$emailAddress = Read-Host "Email address for authentication"
$password = $host.ui.PromptForCredential("Credentials", "Please enter your password to authenticate to EWS.", $emailAddress, "").GetNetworkCredential().Password

# If specified, we'll try to open this mailbox instead of the one that authenticated
$otherMailboxSmtp = Read-Host "SMTP address of other mailbox (optional)"

# Initialize the output file
Set-Content -Path $outputFile -Value "Display Name,LegacyExchangeDN"

# Make variables for the properties to make them easier to type,
# then stick them into a PropertySet
$toRecipientsProperty = [Microsoft.Exchange.WebServices.Data.EmailMessageSchema]::ToRecipients
$ccRecipientsProperty = [Microsoft.Exchange.WebServices.Data.EmailMessageSchema]::CcRecipients
$fromProperty = [Microsoft.Exchange.WebServices.Data.EmailMessageSchema]::From
$subjectProperty = [Microsoft.Exchange.WebServices.Data.EmailMessageSchema]::Subject
$arrayOfPropertiesToLoad = @($toRecipientsProperty, $ccRecipientsProperty, $fromProperty, $subjectProperty)
$propertySet = new-object Microsoft.Exchange.WebServices.Data.PropertySet($arrayOfPropertiesToLoad)

# Here's where we'll store the ones we found to avoid duplicates in the CSV
$legDNsFound = new-object 'System.Collections.Generic.List[string]'

# This function checks an individual EmailAddress to see if it's an unresolved legacyExchangeDN
function CheckAddress($emailAddress)
{
    if ($emailAddress.RoutingType -eq "EX")
    {
        ("        Legacy DN: " + $emailAddress.Address)
        ("        Display Name: " + $emailAddress.Name)
        if (!($legDNsFound.Contains($emailAddress.Address.ToLower())))
        {
            $legDNsFound.Add($emailAddress.Address.ToLower())
            Add-Content -Path $outputFile -Value ($emailAddress.Name.Replace(",", ".") + "," + $emailAddress.Address)
        }
    }
}

# This function loops through the items in a folder
function ProcessFolder($folder)
{
    ("Scanning folder: " + $folder.DisplayName)
    $itemView = new-object Microsoft.Exchange.WebServices.Data.ItemView(100)
    while (($folderItems = $folder.FindItems($itemView)).Items.Count -gt 0)
    {
        foreach ($item in $folderItems)
        {
            if ($item.GetType() -eq [Microsoft.Exchange.WebServices.Data.EmailMessage])
            {
                $message = [Microsoft.Exchange.WebServices.Data.EmailMessage]::Bind($exchService, $item.Id, $propertySet)
                ("    " + $message.Subject)
                foreach ($emailAddress in $message.ToRecipients)
                {
                    CheckAddress($emailAddress)
                }

                foreach ($emailAddress in $message.CcRecipients)
                {
                    CheckAddress($emailAddress)
                }

                CheckAddress($message.From)
            }
        }

        $offset += $folderItems.Items.Count
        $itemView = new-object Microsoft.Exchange.WebServices.Data.ItemView(100, $offset)
    }
}

# This function recursively processes subfolders
function DoSubfoldersRecursive($folder)
{
    if ($folder.ChildFolderCount -gt 0)
    {
        $folderView = new-object Microsoft.Exchange.WebServices.Data.FolderView($folder.ChildFolderCount)
        $subfolders = $folder.FindFolders($folderView)
        foreach ($subfolder in $subfolders)
        {
            ProcessFolder($subfolder)
            DoSubfoldersRecursive($subfolder)
        }
    }
}

# Here's where we try to connect
# If a URL was specified we'll use that; otherwise we'll use Autodiscover
$exchService = new-object Microsoft.Exchange.WebServices.Data.ExchangeService([Microsoft.Exchange.WebServices.Data.ExchangeVersion]::Exchange2007_SP1)
$exchService.Credentials = new-object System.Net.NetworkCredential($emailAddress, $password, "")
if ($hostName -ne "")
{
    ("Using EWS URL:" + "https://" + $hostName + "/EWS/Exchange.asmx")
    $exchService.Url = new-object System.Uri(("https://" + $hostName + "/EWS/Exchange.asmx"))
}
elseif ($otherMailboxSmtp -ne "")
{
    ("Autodiscovering " + $otherMailboxSmtp + "...")
    $exchServce.AutoDiscoverUrl($otherMailboxSmtp, {$true})
}
else
{
    ("Autodiscovering " + $emailAddress + "...")
    $exchService.AutodiscoverUrl($emailAddress, {$true})
}

if ($exchService.Url -eq $null)
{
return
}

$mailbox = new-object Microsoft.Exchange.WebServices.Data.Mailbox($emailAddress)

# If some other mailbox was specified, open that one instead.
if ($otherMailboxSmtp -ne "")
{
$mailbox = new-object Microsoft.Exchange.WebServices.Data.Mailbox($otherMailboxSmtp)
}

# Create some variables for the folder names so they're easier to type
$inboxFolder = [Microsoft.Exchange.WebServices.Data.WellKnownFolderName]::Inbox
$sentItemsFolder = [Microsoft.Exchange.WebServices.Data.WellKnownFolderName]::SentItems
$deletedItemsFolder = [Microsoft.Exchange.WebServices.Data.WellKnownFolderName]::DeletedItems

# We'll bind to each folder by instantiating a FolderId that points to the mailbox we want.

# First, scan the inbox
$inboxId = new-object Microsoft.Exchange.WebServices.Data.FolderId($inboxFolder, $mailbox)
$inbox = [Microsoft.Exchange.WebServices.Data.Folder]::Bind($exchService, $inboxId)
ProcessFolder($inbox)

# Now any subfolders
DoSubfoldersRecursive($inbox)

# Now Sent Items
$sentItemsId = new-object Microsoft.Exchange.WebServices.Data.FolderId($sentItemsFolder, $mailbox)
$sentItems = [Microsoft.Exchange.WebServices.Data.Folder]::Bind($exchService, $sentItemsId)
ProcessFolder($sentItems)

# Now Deleted Items
$deletedItemsId = new-object Microsoft.Exchange.WebServices.Data.FolderId($deletedItemsFolder, $mailbox)
$deletedItems = [Microsoft.Exchange.WebServices.Data.Folder]::Bind($exchService, $deletedItemsId)
ProcessFolder($deletedItems)

"Done!"

The Effects Of Archival Stubs On Database Space Management

Bill Long [Exchange] — Fri, 02 Sep 2011 23:02:00 GMT

Update: The Exchange 2010 issue was resolved in SP2 RU1.

Recently, there have been some theories flying around the blogosphere about the way archive stubbing affects space reclamation in Exchange databases. Specifically, some have questioned whether Exchange will properly reclaim the space when the size of an item in the database shrinks. I contend that yes, Exchange will properly reclaim that space. But you don’t have to take my word for it. I will show you exactly how I tested it, so that you can test it yourself and draw your own conclusions.

First, however, I want to point out that we are currently investigating an issue with Exchange 2010 where database space is not reclaimed, but this issue does not necessarily have anything to do with archival or stubbing. We believe we have reproduced the behavior without those factors in the mix at all, and we are still trying to understand the behavior. This issue only affects Exchange 2010. As a result, I do not recommend trying to test this on Exchange 2010 at this time. The space may not be reclaimed as expected, and you could erroneously conclude that stubbing is at fault, when it’s actually due to the issue that we are investigating.

That said, let’s get started with the test. The point of my test was to prove whether we free up space in the database when a large message body is replaced with a small one. We can’t really test this on 2010 because of the outstanding issue we’re looking at, but it can easily be tested on 2007 or 2003. I decided to perform a simple test on Exchange 2007. Here’s what I did.

The Test

First, I created a brand new database on my Exchange 2007 server. On that database I put a single mailbox. Then, I wrote a simple script to send a bunch of messages with large bodies to that mailbox. Here’s the script:

# Send-SMTPMessages

param([string]$ServerName, [string]$RecipientAddress, [int]$NumberOfMessages)

$bodySB = new-object System.Text.StringBuilder
for ($x = 0; $x -lt 25; $x++) # 25 = 5000 byte body (25 iterations * 2 bytes for each character * 100 characters)
{
$foo = $bodySB.Append("1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890")
}

$smtpClient = new-object System.Net.Mail.SmtpClient($ServerName)

for ($x = 0; $x -lt $NumberOfMessages; $x++)
{
    ("Sending message " + $x.ToString())
    $message = new-object System.Net.Mail.MailMessage("sendsmtpmsgscript@contoso.com", $RecipientAddress, ("Message " + ($x+1).ToString()), $bodySB.ToString())
    $smtpClient.Send($message)
}

I ran the script with the following syntax:

.\Send-SMTPMessages MyServerName mytestmailbox@contoso.com 10000

As the script is written above, this will generate 10,000 messages (which I specified at the command line), with each of those messages having a 5000-byte message body. You can adjust the size of the message body by altering the number of iterations of the for loop, as I noted in the script. All those messages will end up in the Inbox of your test mailbox.

With that done, I let online defrag run to make sure there was no free space (I set the schedule and watched for the 1221 event), and then I dismounted the database and generated a space dump with eseutil /ms.

When you’re looking at space in an Exchange database, it’s important to understand how the database is structured. In all versions of Exchange prior to 2010, the database contained one big Msg table. That table held the bodies of every single message, for every single folder, for every single mailbox in the database. It was all stuffed into one huge table (attachments were stored in another huge table).

So, as expected, my space dump showed that the vast majority of the content of the database was in the Msg table. Here is a snippet of the actual output:

Name                   Type   ObjidFDP    PgnoFDP PriExt      Owned Available
Msg                     Tbl         19         50     2-m      11270         16
<Long Values>         LV          98         51     1-m      10046          7

“Owned” tells us how many pages belong to this table, and “Available” tells us how many of those pages are free. I had just created 10,000 new messages, so as you would expect, there was very little free space in the table.

With that done, I mounted the database and ran another script. I wrote this second script to simulate the act of stubbing. It iterates over every message in the Inbox and changes the body to simply say “Stubbed!”, reducing the body size from 5000 bytes to less than 100 (it actually comes out to something like 80 bytes, because this method of setting the body generates some HTML around the text). Here’s the script:

# Stub-AllMessagesInMailbox.ps1
# This script requires the EWS Managed API to be installed.

param($ServerName, $UserName, $Password, $DomainName)

Import-Module -Name "C:\Program Files\Microsoft\Exchange\Web Services\1.0\Microsoft.Exchange.WebServices.dll"

$exchService = new-object Microsoft.Exchange.WebServices.Data.ExchangeService([Microsoft.Exchange.WebServices.Data.ExchangeVersion]::Exchange2007_SP1)
$exchService.Credentials = new-object System.Net.NetworkCredential($UserName, $Password, $DomainName)
$exchService.Url = new-object System.Uri(("https://" + $ServerName + "/EWS/Exchange.asmx"))
$inbox = [Microsoft.Exchange.WebServices.Data.Folder]::Bind($exchService, [Microsoft.Exchange.WebServices.Data.WellKnownFolderName]::Inbox)

$offset = 0
$itemView = new-object Microsoft.Exchange.WebServices.Data.ItemView(100)
while (($inboxItems = $inbox.FindItems($itemView)).Items.Count -gt 0)
{
    foreach ($item in $inboxItems)
    {
        ("Stubbing item: " + $item.Subject)
        $item.Body = "Stubbed!"
        $item.Update([Microsoft.Exchange.WebServices.Data.ConflictResolutionMode]::AlwaysOverwrite)
    }

    $offset += $inboxItems.Items.Count
    $itemView = new-object Microsoft.Exchange.WebServices.Data.ItemView(100, $offset)
}

After all my items were “stubbed”, I immediately dismounted the database again and ran another space dump, without letting online defrag run. The result surprised me:

Name                   Type   ObjidFDP    PgnoFDP PriExt      Owned Available
Msg                     Tbl         19         50     2-m      11414         12
<Long Values>         LV          98         51     1-m      10054       7952

My Owned pages went up slightly, but I instantly had 7,952 pages free in the LV tree of the Msg table – about 62 MB. And this was before allowing online defrag to run. After online defrag I had even more free pages:

Name                   Type   ObjidFDP    PgnoFDP PriExt      Owned Available
Msg                     Tbl         19         50     2-m      11414         16
<Long Values>         LV          98         51     1-m      10050       8207

I repeated this test using message bodies of 2000 bytes and 7000 bytes, and the results were the same. I tried stubbing 5000-byte items down to 1078 bytes instead of 100 bytes, and the results were the same. I even tried stubbing only every 5th message instead of all the messages. In all my tests, Exchange 2007 freed a ton of pages when I reduced the size of the message bodies.

Conclusions From Testing

From this test, it is quite clear that after stubbing the items, space is reclaimed – and not just a small amount, either. In fact, we reclaimed more space than we would expect based on the size of the items. 5000 bytes x 10000 messages = roughly 50 MB, but we got back about 65 MB (8207 pages * 8192 bytes per page = 67,231,744 bytes). This is likely due to page fragmentation from before the stubbing. That is, if our items are 5k in size and we have 8k pages in Exchange 2007, we can’t fit 2 items on one page. As a result, there will be some unused space on these pages. After we stubbed, we were able to fit more items on a page more efficiently, and space wasted by fragmentation actually decreased.

Page fragmentation might also explain why the Msg table was about 90 MB in size before we stubbed, when we had only added 50 MB of message bodies. This is a great example of why you can’t just add up mailbox sizes to determine how big the database file should be. It has a lot to do with what sort of data is in the database.

So What’s The Deal With Stubbing?

A few years back, I worked on a series of Exchange 2003 cases where customers were using stub archival (also known as shortcuts), and they were concerned at the discrepancy between reported mailbox size and actual database size. Several of those customers sent me databases, and I spent a few months working with the archival software vendor to analyze the database contents and understand the reason for the difference. Our findings revealed that the mystery space was being taken up by two types of overhead.

The first type of overhead is properties that simply don't count toward the size of the message. When you look at the size of a message in Outlook, the size you see is not the total size of all the data that Exchange stores for that message. We store certain properties that do not count toward message size, and they aren’t counted against the user and aren’t reflected in the message size. These may be publicly documented properties such as PR_URL_NAME or other internal properties which are not publicly documented.

The second type of overhead is page fragmentation. A page in an Exchange 2003 database is 4k in size. Each record in the database has to be arranged onto these pages, and depending on how efficiently we are able to do so, there will be some space left on the page. This is page fragmentation, which results in empty space that cannot be reclaimed by online maintenance.

To the extent that an Exchange database is made up of many tiny items, these two kinds of overhead become more apparent. I worked with the vendor to set up a lab environment where we could test this theory with the vendor’s real archival software. We tested several scenarios, but I think the final test was the most interesting. We used LoadSim in a lab environment to fill a database with 1.4 million messages ranging in size from 1k to 4k with no attachments. Even before archiving, the database had about 30% overhead from fragmentation and 20% from properties that don't count toward message size. This showed that the behavior can be achieved even without archival by filling the database with lots of small items. After archiving everything and leaving shortcuts in place, those numbers went up to a combined 70% (because all the 4k items became much smaller shortcuts), although archiving did free up about 15% of the database. The same behavior would be expected from any archival product that is configured to leave shortcuts behind.

Exchange will continue to work just fine in this scenario. There's nothing inherently wrong with it. It's just that when you archive email messages and leave behind a shortcut, you are taking out the big properties like message body and attachments, and leaving a small stub - but all the overhead associated with the item remains. Although you do save disk space by taking out the big stuff, the remaining tiny items still have the same amount of overhead as a normal item. The ratio of overhead to actual email rises, so the database size will be considerably larger than what you would expect from adding up the reported mailbox sizes.

We concluded that if you want to reduce the overhead, the best approach is to expire the shortcuts after some time limit. To the extent you expire the shortcuts, you should see the overhead come down. Of course, another option is to let all the shortcuts stay in the database, knowing that your database is perfectly healthy despite the size difference between the mailbox sizes and the file size.

So what about Exchange 2007? If anything, I would expect page fragmentation to be somewhat less of an issue in 2007 than 2003 due to the larger page size. That should make it easier to arrange several stubbed items onto a single page. But again, this will depend a lot on what sort of items you put in the database.

In Conclusion

I avoided discussing Exchange 2010 here, because the issue with space reclamation which we are currently investigating makes it very difficult to test and draw any conclusions about how 2010 will behave.

However, my findings in testing Exchange 2007 are clear: When the items shrink, we reclaim a lot of space. Background cleanup is able to grab much of the space even before online defrag comes along. Maybe someone can find fault with my testing, or maybe someone can come up with a reproducible scenario where we don’t reclaim the space. If you can, I would love to hear from you.

Let me clearly state: At this time, there is no known issue with archival stubs beyond the increased overhead ratio I described above.

Hierarchy Replication Fails Due To Zero GUID

Bill Long [Exchange] — Fri, 24 Jun 2011 21:57:24 GMT

Exchange 2010 Sp1 Rollup Update 4 released a couple of days ago, and I want to briefly mention one of the fixes it includes. The fix I’m talking about is KB 2506049, “The hierarchy of a new public folder database on an Exchange Server 2010 SP1 server is not replicated”. You’ll know you’ve hit this problem if you get the very specific events mentioned in the article with error code 80070057. Specifically, we should see this in the 3079:

Unexpected replication thread error 80070057 on database "First Storage Group\Public Folder Store (<Server_Name>)".

EcReplFolderMessagesUnpack
EcReplMessageUnpack

If you’re not seeing that, then you’re not hitting this problem.

If you are hitting this problem, it means that your current PF databases which have hierarchy and content have a GUID in them somewhere of 00000000-0000-0000-0000-000000000000. It’s a GUID of all zeroes, and it’s not valid. Starting with Exchange 2010 SP1, we block these GUIDs from entering a new database. So, if you have this sitting around in your old PF databases, and you bring up a new clean database on 2010 SP1, the hierarchy fails to replicate because we refuse to let this GUID replicate in. You’ll never notice this problem until you try to create a new database on 2010 SP1 or later.

What I want to focus on, however, is the fix. The article states the following:

After you apply this update on all the computers that have public folder databases, follow these steps:

Unmount all public folder databases in the organization.

Run the following command on all the computers that have public folder databases in the organization:
isinteg -s <server> -test replidguid -fix

Mount all public folder databases.

Notes

The command must be run simultaneously on all the computers that have public folder databases in the organization.

Do not follow these steps if there are any public folder servers that are not running Update Rollup 4 for Exchange Server 2010 SP1, such as Exchange Server 2003, Exchange Server 2007, or earlier versions of Exchange Server 2010.

If you try to run the command on only a subset of computers that have public folder databases in the organization, public folder replication issues occur. Additionally, the issues cannot be resolved.

This command will strip the zero GUID out of the databases. Yes, the command must be run simultaneously on all public folder databases in the org. This means that all public folder servers must be on at least Exchange 2010 SP1 RU4. If you have any Exchange 2007 or 2003 public folder servers, you cannot use this fix.

Even if all your public folder databases are on Exchange 2010 SP1 RU4 or newer, you must dismount all of them at the same time and run this command against all of them. They must all be dismounted at the same time and must not be mounted until the command has been run.

To put it another way, a fixed database must not replicate with an unfixed database. In order to achieve this, all databases must be dismounted before you start running the isinteg command. Once the command completes, you can mount that fixed database if you want. But be very careful. It is probably safer to just leave them all down until you are sure they have all been fixed.

If you fix one or more databases and then accidentally mount one that has not been fixed, you run a high risk of introducing a form of logical corruption that cannot be fixed, ever, at all. When the article says that the issues cannot be resolved, we really mean it. The only option at that point is to pull the public folder data out to PSTs and start over with clean public folder databases everywhere.

Please use caution when using this fix, and make sure that when this isinteg command is run, all public folder databases have been dismounted and are not mounted again until they are fixed.

The good news is that this fix is extremely fast. This isinteg command should complete in less than a minute.

Do Not Use Remove-PublicFolder To Remove A Public Folder Database

Bill Long [Exchange] — Sat, 11 Jun 2011 05:30:00 GMT

About once a week, someone comes to me about a case where a customer has accidentally deleted all their public folders when they were just trying to get rid of one public folder store. This happens because of threads or blog posts where someone suggests using some variation of this command:

Get-PublicFolder "\" -Recurse -ResultSize Unlimited | Remove-PublicFolder

The first part of this command returns every public folder in the hierarchy (except for System Folders, which are under \Non_Ipm_Subtree). It returns every folder in the hierarchy, not just the ones that have replicas on a particular server. The second part of this command deletes those folders out of the hierarchy.

Yes, it deletes them. I don’t mean it just deletes them off of one particular store – that would be removing or deleting a replica. Remove-PublicFolder deletes the folder, just like if you go into Outlook, navigate to the folder, and choose Delete.

Of course, the deletion of the folder can take up to 15 minutes to replicate to the other public folder stores, so some customers get lucky. They run the command, then immediately run Remove-PublicFolderDatabase and delete the database before it has a chance to replicate. In that case, you only lose the directory objects for mail-enabled folders, so if you don’t have any mail-enabled folders, you may not even notice a problem.

In other cases, customers get very unlucky, and the change replicates immediately, wiping out every public folder on every public folder server throughout the entire organization.

If you get nothing else from this post, please just remember that Remove-PublicFolder deletes the public folder, and that change will replicate to all the other public folder stores.

So, what should you do if you are trying to remove a public folder database, and it’s telling you something like this:

Remove-PublicFolderDatabase : The public folder database specified contains folder replicas. Before deleting the public folder database, remove the folders or move the replicas to another public folder database.

In that case, you should read my post on the Exchange Team Blog from several years ago:

http://blogs.technet.com/b/exchange/archive/2007/11/13/clarifying-the-public-folder-store-removal-on-exchange-server-2007.aspx

The bottom line is that if you’re getting that error, you should use Get-PublicFolderStatistics to see which folders have not replicated off that store. The replica list on those folders needs to be changed so that you've added some other server, if there wasn't one already, and removed this server. This can be done with Set-PublicFolder, or MoveAllReplicas.ps1, or PFMC, or ExFolders, etc. Once you've changed the replica list, allow some time for replication.

Unfortunately, in many cases, customers change the replica list and wait, but the folders are still sticking in Get-PublicFolderStatistics. If those are folders you don’t care about, like OWAScratchPad and such, then feel free to delete them with Remove-PublicFolder (or Outlook, or MfcMapi, or PFMC, etc). If those are folders you do care about, you should confirm whether all the data you wanted replicated off. In most cases, they will be stuck because of a few bad/corrupt items that don’t want to replicate.

If you have folders that are stuck because of a few corrupt items, you have a few options. You can delete the bad items, which will allow the replica to gracefully remove itself via the normal replica removal process. You can dismount the database and delete it with ADSI Edit, losing any data that didn’t replicate off, and possibly breaking age limits for any folders that still had replicas there. But either of those options is probably better than piping the results of Get-PublicFolder to Remove-PublicFolder, which runs the risk of deleting the whole hierarchy throughout your environment.

Importing PFDAVAdmin or ExFolders Exports Without PFDAVAdmin or ExFolders

Bill Long [Exchange] — Tue, 26 Apr 2011 05:26:51 GMT

It’s almost May and I haven’t posted anything yet this year, so it’s definitely time to post a new script.

I recently worked with a customer that needed to export the public folder permissions from one Exchange organization and import them into another. The trick was that on the import side, some of the accounts were only mail-enabled – not mailbox-enabled. This creates a problem for ExFolders, because the API it uses does not consider mail-enabled accounts to be valid security principals. However, if you use the Add-PublicFolderClientPermission cmdlet to set the permissions, it has no problem with such an account.

I looked at changing ExFolders to better support this, but the changes would be complex and I just didn’t have that kind of time. Instead, I decided to write a script to import the permissions using the cmdlet. There are a few things to be aware of about the script.

First, the script will be much slower than using PFDAVAdmin or ExFolders to run the import. This is because the cmdlets only let you modify one user at a time, and unless you want to do a lot of complicated calculations, you need to remove the existing permissions for the user before adding the new ones. This means we end up running two cmdlets for every user permission. If you are importing permissions for 5 users on a folder, it will take 10 commands to assign those permissions.

Second, the script currently only works with exports that are in legacyExchangeDN format. If you used NT Account format (domain\user), this script can’t handle them. This means the legacyExchangeDNs from the old environment have to be resolvable in the new one. That is usually the case, since they are brought over as X500 addresses in most migrations.

And finally, this script works only with public folder permissions exports from PFDAVAdmin or ExFolders. If you exported replicas or properties or something, this script won’t help you.

Anyway, here is the script. Enjoy!

# Import-PFPermissions.ps1
#
# The purpose of this script is to import permissions from an export
# file generated using PFDAVAdmin or ExFolders.
#
# Syntax example:
#
# .\Import-PFPermissions C:\someimportfile.txt MYPFSERVER

param([string]$importFile, [string]$server)

function ConvertRightsString([string]$pfdavRightsString)
{
    $pfdavRights = $pfdavRightsString.Split(@(' '))
    $powershellRights = ""
    foreach ($right in $pfdavRights)
    {
        if ($right -eq "All")
        {
            $powershellRights += "Owner,"
        }
        elseif ($right -eq "Owner")
        {
            $powershellRights += "CreateItems,ReadItems,CreateSubfolders,FolderOwner,FolderVisible,EditOwnedItems,EditAllItems,DeleteOwnedItems,DeleteAllItems,"
        }
        elseif ($right -eq "Contact")
        {
            $powershellRights += "FolderContact,"
        }
        elseif ($right -eq "Create")
        {
            $powershellRights += "CreateItems,"
        }
        elseif ($right -eq "CreateSubfolder")
        {
            $powershellRights += "CreateSubfolders,"
        }
        elseif ($right -eq "Delete")
        {
            $powershellRights += "DeleteAllItems,"
        }
        elseif ($right -eq "DeleteOwn")
        {
            $powershellRights += "DeleteOwnedItems,"
        }
        elseif ($right -eq "Write")
        {
            $powershellRights += "EditAllItems,"
        }
        elseif ($right -eq "WriteOwn")
        {
            $powershellRights += "EditOwnedItems,"
        }
        elseif ($right -eq "o")
        {
            $powershellRights += "FolderOwner,"
        }
        elseif ($right -eq "Visible")
        {
            $powershellRights += "FolderVisible,"
        }
        elseif ($right -eq "Read")
        {
            $powershellRights += "ReadItems,"
        }
        else
        {
            $powershellRights += ($right + ",")
        }
    }

    $powershellRights = $powershellRights.TrimEnd(@(','))
    return $powershellRights
}

function RemoveFolderVisibleWithTrap([string]$folderPath, [string]$server, [string]$userString)
{
    trap [Exception]
    {
        continue
    }

    $returnedValue = Remove-PublicFolderClientPermission -Identity $folderPath -Server $server -User $userString -AccessRights FolderVisible
}

$fileReader = new-object System.IO.StreamReader($importFile)
while ($null -ne ($buffer = $fileReader.ReadLine()))
{
    if ($buffer.StartsWith("SETACL`t"))
    {
        $columns = $buffer.Split(@("`t"))

        # Resolve the folder path, which should always be the second column.
        $folderPath = $columns[1]
        if ($folderPath.StartsWith("Public Folders\"))
        {
            $folderPath = $folderPath.Substring(14)
        }
        elseif ($folderPath.StartsWith("System Folders\"))
        {
            $folderPath = ("\NON_IPM_SUBTREE" + $folderPath.Substring(14))
        }

        ("Updating folder: " + $folderPath)

        for ($x = 2; $x -lt $columns.Length; $x+=2)
        {
            $userString = $columns[$x]
            if ($userString.StartsWith("/o="))
            {
                $recipient = Get-Recipient $userString
                if ($recipient -ne $null)
                {
                    $userString = $recipient.Identity
                }
                else
                {
                    ("     Could not resolve: " + $userString)
                    continue
                }
            }
            elseif ($userString -ne "Default" -and $userString -ne "Anonymous")
            {
                ("     Skipping unrecognized user format: " + $userString)
                continue
            }
            ("     Processing user: " + $userString)
            ("          Checking for existing permissions...")
            $permsForUser = Get-PublicFolderClientPermission -Identity $folderPath -Server $server -User $userString
            if ($permsForUser -ne $null)
            {
                ("          Removing existing permissions...")
                $permsForUser | Remove-PublicFolderClientPermission -Identity $folderPath -Server $server -Confirm:$false
            }

            ("          Adding new permissions...")
            $rightsString = ConvertRightsString $columns[$x+1]
            ("          AccessRights: " + $rightsString)
            $rightsArray = $rightsString.Split(@(','))
            if ($permissionString -eq "None")
            {
                # This requires special handling. In the cmdlet, there is no way to grant a user no permissions,
                # denying the ability to even see the folder, because the None role includes the FolderVisible
                # right. When a user has None without folder visible, it still shows up as None, but IsRole is
                # false. We need to check for that situation here to make sure None is really None. In the
                # export, None really means no rights at all.
                #
                # So, what we do here, is first we set rights of None. Then, we try to remove FolderVisible,
                # and ignore any errors that happen. If it succeeds, we don't have FolderVisible anymore, and
                # if it fails, we didn't have FolderVisible in the first place, so it doesn't matter.
                $returnedValue = Add-PublicFolderClientPermission -Identity $folderPath -Server $server -User $userString -AccessRights $rightsArray
                RemoveFolderVisibleWithTrap $folderPath $server $userString
            }
            else
            {
                $returnedValue = Add-PublicFolderClientPermission -Identity $folderPath -Server $server -User $userString -AccessRights $rightsArray
            }
        }
    }
}

Canonical Ordering Of Mailbox Permissions In Exchange

Bill Long [Exchange] — Mon, 27 Dec 2010 19:30:00 GMT

An Access Control List, or ACL, is basically just a list of Access Control Entries, or ACEs. Each ACE contains a flag indicating whether it’s an Allow or a Deny ACE, a mask that shows which rights are being allowed or denied, and a SID to which those allow or deny rights are applied.

Evaluating an ACL to determine if a user has certain rights is pretty simple. We take the user’s token, which is a list of all SIDs that apply to this user (all groups he’s a member of), and then we iterate through the ACL looking at each ACE in the order in which it appears. We keep evaluating ACEs until we hit one of two conditions: either all rights requested have been granted, or any one right requested has been denied. At that point, evaluation stops, and we don’t even bother looking at the rest of the ACEs.

Because of this process, in order to achieve the desired behavior where denies take precedence over allows, the ACL must be in a specific order. If the ACL is ordered correctly, we say it is canonical. If the order is incorrect, we say it is non-canonical, because it does not follow the established rules for the ordering of ACEs in an ACL.

For example, let’s say we have the following very simple ACL, with only two ACEs:

Allow/Deny	Rights Mask	SID
Deny	0xFFFFFFFF	S-1-1-0
Allow	0xFFFFFFFF	S-1-1-0

The S-1-1-0 SID is the SID for the Everyone group. In this ACL, we have two ACEs. The first ACE denies all rights to the Everyone group. The second one grants all rights to the Everyone group. When we go through the process of checking a user’s rights, the first ACE we hit says that all rights are denied to Everyone. Because of that, we stop right there, and the evaluation result will be that all rights are denied. In this case, canonical order was followed, so we have the expected behavior – the user is denied access to the object.

Now imagine the ACEs are not in canonical order:

Allow/Deny	Rights Mask	SID
Allow	0xFFFFFFFF	S-1-1-0
Deny	0xFFFFFFFF	S-1-1-0

In this case, when we evaluate the ACL, the first ACE we hit says that all rights are allowed. The evaluation stops right there, and the result is that the user is allowed to access the object, despite the fact that we have also denied all rights to Everyone! So in this case, the ACL is non-canonical, and the result is that the permissions don’t function as expected – the user is allowed access.

Note that the above is a much simplified example just to illustrate why canonical order exists. For full information on canonical ordering of ACEs, see MSDN.

Canonical ordering is not enforced by Windows itself – it is enforced by the tools that are usually used to manage those ACLs. If you set permissions on an Active Directory object using Active Directory Users And Computers, those permissions will be canonical. If you set rights on a file using Explorer, the ACL on the file will be canonical.

However, it’s entirely possible for a programmer to write a tool that saves ACLs to these objects in non-canonical order. In fact, in some cases this has been done intentionally. Back in Exchange 2003, you could choose to hide distribution group membership. Doing so caused us to purposely put the ACL in non-canonical order. After hiding the membership, attempting to go to the Security tab in ADUC resulted in the message “Windows can not edit the permissions on ‘whatever’ because they have been written in a nonstandard format by another application. To enable editing, you must use the application to restore the permissions to a standard format.” This message was because the ACL was non-canonical.

Mailbox permissions require canonical ordering just like AD permissions or file permissions (public folder permissions have their own canonical order that is different from Windows canonical order – that’s a topic for another post). If you run Add-MailboxPermissions to modify the permissions and the ACL is not canonical, the cmdlet reports an error:

The ACL for the object "whatever" is not in canonical order (Deny/Allow/Inherited) and will be ignored.

This makes it impossible to edit the mailbox rights using the cmdlet.

I recently ran across a case where some mailboxes migrated from Exchange 2003 to Exchange 2007 were in this state. We were not able to determine what application made the ACLs non-canonical. To fix the problem, I wrote a simple application called FixMailboxSD and made the tool and the source available on Codeplex: http://fixmailboxsd.codeplex.com/. This uses the old CDOEXM interfaces from Exchange 2003 to put the mailbox rights back in canonical order.

So, if you need an example of how to play with mailbox rights from C# using CDOEXM, you can take a look at this simple tool. Or, if you run into this same problem in your environment, feel free to use the tool to correct the mailbox rights on any affected mailboxes.

Get-PublicFolderStatistics Can’t Find A Folder

Bill Long [Exchange] — Thu, 28 Oct 2010 18:36:00 GMT

On Exchange 2010, when you use the Get-PublicFolderStatistics command to look for the statistics of a particular folder, you may see an error like this one:

The cmdlet throws an InvalidOperationException saying that it “couldn’t find public folder”. However, if you look at the replica list, it clearly indicates that there is a replica on that server. This is because Get-PublicFolderStatistics is doing something sneaky in the background without warning you.

If you run Get-PublicFolderStatistics with no other parameters, you’ll see this message:

At the end of the list of results, it is warning you that it didn’t return the whole list of folders – it only returned the first 100. You have to use -ResultSize Unlimited to get the entire list.

When you run Get-PublicFolderStatistics and specify a particular Identity, the same thing is happening in the background, but it doesn’t warn you. By default, it only returns the first 100 folders and looks through those for a matching Identity. If the folder you specified doesn’t happen to be one of the first 100, the command fails to find it.

To make this work, you need to add -ResultSize Unlimited to the command:

With that command, it is able to search the entire list of folders to find the one the user specified.

Hopefully this helps anyone out there who is confused by the behavior of this cmdlet. On a recent case, quite a bit of time was spent before we understood why the command was failing to find the folder!

Cleaning Viruses In Public Folders Using OOM

Bill Long [Exchange] — Tue, 14 Sep 2010 07:26:00 GMT

In my previous post, I provided an example script that used EWS to delete items out of public folders. Of course, you can only use EWS if your public folders are on Exchange 2007 or 2010.

The example script in this post uses Outlook Object Model instead. This means it has to be run from a machine with Outlook installed (it should work with Outlook 2007 or Outlook 2010) as well as Powershell 2.0.

You might notice that this script is a little simpler than the EWS example. Most of that is due to the fact that Outlook Object Model doesn’t give us much control over how to retrieve the contents of a folder. For folders with lots of items, you may find that this script will run very slowly, if it works at all. The EWS version won’t have that problem since it is able to use ranged retrieval, whereas this one has to get the items in one big chunk.

# Delete-PublicFolderItems.ps1
#
# Syntax:
#
# .\Delete-PublicFolderItems \TopFolder\Subfolder $false
# This example runs against a single folder and does not recurse.
#
# .\Delete-PublicFolderItems "" $true
# This example runs against the entire public folder hierarchy.

param([string]$folderPath, [bool]$recurse)

# By default, this script only reports what it would have deleted.
# If you flip $readOnly to $false, it WILL DELETE ITEMS OUT OF YOUR
# PUBLIC FOLDERS. Make sure you've tested it first and you have a
# backup.
$readOnly = $true

$outlook = new-object -com Outlook.Application
$mapi = $outlook.GetNamespace("MAPI")
$pfStore = $null
foreach ($store in $mapi.Stores)
{
    if ($store.ExchangeStoreType -eq 2)
    {
        $pfStore = $store
    }
}

if ($pfStore -eq $null)
{
"Couldn't find public folder store."
return
}

$pfRoot = $pfStore.GetRootFolder()

# Define the function we'll use to traverse
# the PF tree if a path was specified.
function GetNamedFromCollection($name, $collection)
{
     foreach ($item in $collection)
     {
          if ($item.Name -eq $name -or $item.DisplayName -eq $name)
          {
               return $item
          }
     }
     return $null
}

$allPFs = GetNamedFromCollection "All Public Folders" $pfRoot.Folders
if ($allPFs -eq $null)
{
"Couldn't find All Public Folders folder."
return
}

$folderPath = $folderPath.Trim("\")
$folderPathSplit = $folderPath.Split("\")
$folder = $allPFs
if ($folderPath.Length -gt 0)
{
     "Traversing folder path..."
     for ($x = 0; $x -lt $folderPathSplit.Length; $x++)
     {
          $folder = GetNamedFromCollection $folderPathSplit[$x] $folder.Folders
     }

     if ($folder -eq $null)
     {
          ("Could not find folder: " + $folderPath)
          return
     }

("Found folder: " + $folder.FolderPath)
}

# Now let's define the function that'll do the real work

function DoFolder($folder)
{
    ("Scanning folder: " + $folder.FolderPath)

    try
    {
        foreach ($item in $folder.Items)
        {
            ##########################################################
            # Here's the most important part. This is where we
            # decide whether to delete the item.
            #
            # By default this script will delete any item that
            # contains the phrase "Here you have" in the subject.
            # If that isn't what you want, you need to edit
            # the criteria here.
            if ($item.Subject.Contains("Here you have"))
            {
                if ($readOnly)
                {
                    ("     Would have deleted item: " + $item.Subject)
                }
                else
                {
                    ("     Deleting item: " + $item.Subject)
                    $item.Delete()
                }
            }
        }
    }
    catch { "Error processing folder." }

    if ($recurse)
    {
        foreach ($subfolder in $folder.Folders)
        {
            try
            {
                DoFolder $subfolder
            }
            catch { "Error processing folder: " + $subfolder.FolderPath }
        }
    }
}

DoFolder $folder

"Done!"

Cleaning Viruses In Public Folders Using EWS

Bill Long [Exchange] — Tue, 14 Sep 2010 06:23:00 GMT

As a result of last week’s virus outbreak, I’ve been getting some questions about how to clean viruses out of public folders. Unfortunately, there’s no equivalent of Export-Mailbox to pull infected messages out of public folders. However, it’s pretty easy to write a script to accomplish this through Exchange Web Services or Outlook Object Model based on my previous post about accessing the Information Store from Powershell.

This script is an example that uses EWS through the EWS Managed API, so it can be used for public folders on Exchange 2007 or 2010. You can run it from a Powershell 2.0 command prompt on a machine where you’ve installed the EWS Managed API. The Exchange tools aren’t required.

Note that because this script accesses public folders via EWS, which is a client API, you must have the proper client permissions to the public folder in order to see and delete the items in question. Being an Exchange Administrator does not mean you will be able to delete items out of the public folders through a client. For more on admin rights versus client rights, see my post on that topic.

For an example that uses Outlook Object Model, which allows it to run against Exchange 2003 (or 2007 or 2010), see my next post.

# Delete-PublicFolderItems.ps1
# This script requires Powershell 2.0 and .NET Framework 3.5.
#
# Syntax:
#
# .\Delete-PublicFolderItems <email address for autodiscover> <path to folder> <recurse>
#
# Examples:
#
# .\Delete-PublicFolderItems administrator@contoso.com \TopLevelFolder\Subfolder $false
# This example runs against one specific folder and does not recurse.
#
# .\Delete-PublicFolderItems administrator@contoso.com "" $true
# This example recursively runs against every folder in the PF hierarchy.
#
# The email address you specify should typically be the email address of the admin
# you're logged in as. Otherwise you may see some strange autodiscover errors.

param([string]$autoDiscoverEmail, [string]$folderPath, [bool]$recurse)

# Remember to install the EWS managed API from here:
# http://www.microsoft.com/downloads/en/details.aspx?displaylang=en&FamilyID=c3342fb3-fbcc-4127-becf-872c746840e1
#
# Then point the following command to the location of the DLL:

Import-Module -Name "C:\Program Files\Microsoft\Exchange\Web Services\1.0\Microsoft.Exchange.WebServices.dll"

# If the following is set to $true, we run in READ ONLY mode. We report what we would have deleted,
# but we don't delete anything.
#
# Changing this to $false will make the script DELETE ITEMS OUT OF YOUR PUBLIC FOLDERS.
# Make sure you have tested this first and you have a backup.

$readOnly = $true

# Use AutoDiscover to get the ExchangeService object

$exchService = new-object Microsoft.Exchange.WebServices.Data.ExchangeService([Microsoft.Exchange.WebServices.Data.ExchangeVersion]::Exchange2007_SP1)
if ($exchService -eq $null)
{
    "Could not instantiate ExchangeService object."
    return
}
$exchService.UseDefaultCredentials = $true
$exchService.AutodiscoverUrl($autoDiscoverEmail)
if ($exchService.Url -eq $null)
{
    return
}

# Bind to the public folders

$pfs = [Microsoft.Exchange.WebServices.Data.Folder]::Bind($exchService, [Microsoft.Exchange.WebServices.Data.WellKnownFolderName]::PublicFoldersRoot)
if ($pfs -eq $null)
{
return
}

# Now let's define the function we'll call to process the folders.

function DoFolder([Microsoft.Exchange.WebServices.Data.Folder]$folder, [string]$path)
{
    ("Scanning folder: " + $path)

     try
     {
        # Scan the items in this folder
        $offset = 0;
        $view = new-object Microsoft.Exchange.WebServices.Data.ItemView(100, $offset)
        while (($results = $folder.FindItems($view)).Items.Count -gt 0)
        {
            foreach ($item in $results)
            {
                ############################################################
                # Here's the most important part. This is where we determine
                # whether to delete the item or not.
                #
                # By default this script will delete any item that contains
                # the phrase "Here you have" in the subject. If this is not
                # what you want, you need to edit the criteria here.
                if ($item.Subject.Contains("Here you have"))
                {
                    if ($readOnly)
                    {
                        ("    Would have deleted item: " + $item.Subject)
                    }
                    else
                    {
                        ("    Deleting item: " + $item.Subject)
                        $item.Delete([Microsoft.Exchange.WebServices.Data.DeleteMode]::HardDelete)
                    }
                }
                #
                ############################################################
            }

            $offset += $results.Items.Count
            $view = new-object Microsoft.Exchange.WebServices.Data.ItemView(100, $offset)
        }

        if ($recurse)
        {
            # Recursively do subfolders
            $folderView = new-object Microsoft.Exchange.WebServices.Data.FolderView(2147483647)
            $subfolders = $folder.FindFolders($folderView)
            foreach ($subfolder in $subfolders)
            {
                try
                {
                    DoFolder $subfolder ($path + "\" + $subfolder.DisplayName)
                }
                catch { "Error processing folder: " + $subfolder.DisplayName }
            }
        }
     }
     catch
     {
         throw
     }
}

# Call DoFolder on the top folder.
# If $recurse is true, it will recursively call itself.

if ($folderPath.Length -lt 1)
{
    DoFolder $pfs ""
}
else
{
    $folderPathTrim = $folderPath.Trim("\")
    $folderPathSplit = $folderPathTrim.Split("\")
    $tinyView = new-object Microsoft.Exchange.WebServices.Data.FolderView(2)
    $displayNameProperty = [Microsoft.Exchange.WebServices.Data.FolderSchema]::DisplayName
    $thisFolder = $pfs
    for ($x = 0; $x -lt $folderPathSplit.Length; $x++)
    {
        $filter = new-object Microsoft.Exchange.WebServices.Data.SearchFilter+IsEqualTo($displayNameProperty, $folderPathSplit[$x])
        $results = $thisFolder.FindFolders($filter, $tinyView)
        if ($results.TotalCount -gt 1)
        {
            "Ambiguous folder name."
            return
        }
        elseif ($results.TotalCount -lt 1)
        {
            "Folder not found."
            return
        }
        $thisFolder = $results.Folders[0]
    }

    DoFolder $thisFolder $folderPath
}

# When we return here, we're done!

"Done!"

ExFolders for Exchange 2010 Sp1

Bill Long [Exchange] — Mon, 13 Sep 2010 18:28:53 GMT

A few of you emailed me when you noticed an error running ExFolders on Exchange 2010 SP1, which was recently released. Attempting to navigate into a mailbox would throw an error stating, “Error: Method not found.” This was due to some changes in the DLLs that ExFolders relies on.

We’ve just posted a new version of ExFolders that works properly with Sp1. You can find it here.

The RTM version is still available for download as well. Be aware that the RTM version won’t work on Sp1, and the Sp1 version won’t work on RTM, as far as navigating into mailboxes.

How To Identify Bad Items In Public Folder Replication

Bill Long [Exchange] — Thu, 27 May 2010 19:24:56 GMT

As I’ve previously discussed here and here, there are various things that will prevent an item from replicating to an Exchange 2007 or Exchange 2010 public folder store – things like bad start/end dates on appointments, category names that contain certain characters, etc. However, once you’ve figured out that replication is failing because of one of these problems, it can still be tricky to identify which particular items are the ‘bad’ items. In this post, I want to share the approach we usually use to track these down.

Now, you may be thinking, “Well, I can just compare the source and the destination to see which items didn’t make it over, and those are my bad items, right?” Not quite.

When we replicate changes for a particular folder, we pack up as many changes as we can until we meet the Replication Message Size Limit configured on the public folder store. This applies to both the replication of new data and the backfill of existing data. As a result, when you observe a backfill response for a folder with a bunch of messages, you typically see a big list of items in the event:

Event Type:    Information
Event Source:    MSExchangeIS Public Store
Event Category:    Replication Outgoing Messages
Event ID:    3021
Description:
An outgoing replication message was issued.

Type: 0x80000004
Message ID: <003346D6BDE71245954899B0FC2F3702017DCF@bilongtimb1.bilong.test>
Folder: (1-16771) IPM_SUBTREE\FolderA\Subfolder1

Database "First Storage Group\Public Folder Store (BILONGTIMB1)".
CNSET: 1-1,1-1791E

CNSET (FAI): 1-1,1-1791E

Message IDs: 90
1: 1-1687B, 1-16C45
--- : Test 10 : 5/19/2010 4:16:12 PM
2: 1-1687C, 1-16C46
--- : Test 10 : 5/19/2010 4:16:12 PM
3: 1-16886, 1-16C47
--- : Test 10 : 5/19/2010 4:16:12 PM
4: 1-16890, 1-16C48
--- : Test 10 : 5/19/2010 4:16:12 PM
5: 1-1689A, 1-16C49
--- : Test 10 : 5/19/2010 4:16:12 PM
6: 1-168A4, 1-16C4A
--- : Test 10 : 5/19/2010 4:16:12 PM
7: 1-168AE, 1-16C4B
--- : Test 10 : 5/19/2010 4:16:12 PM
8: 1-168B8, 1-16C4C
--- : Test 10 : 5/19/2010 4:16:12 PM
9: 1-168C2, 1-16C4D
--- : Test 10 : 5/19/2010 4:16:12 PM
10: 1-1687A, 1-16C4E
--- : Test 9 : 5/19/2010 4:16:09 PM
11: 1-1687D, 1-16C4F
--- : Test 9 : 5/19/2010 4:16:09 PM
12: 1-16887, 1-16C50
--- : Test 9 : 5/19/2010 4:16:09 PM
13: 1-16891, 1-16C51
--- : Test 9 : 5/19/2010 4:16:09 PM
14: 1-1689B, 1-16C52
--- : Test 9 : 5/19/2010 4:16:09 PM
15: 1-168A5, 1-16C53
--- : Test 9 : 5/19/2010 4:16:09 PM
16: 1-168AF, 1-16C54
--- : Test 9 : 5/19/2010 4:16:09 PM
17: 1-168B9, 1-16C55
--- : Test 9 : 5/19/2010 4:16:09 PM
18: 1-168C3, 1-16C56
--- : Test 9 : 5/19/2010 4:16:09 PM
19: 1-16879, 1-16C57
--- : Test 8 : 5/19/2010 4:16:05 PM
20: 1-1687E, 1-16C58
--- : Test 8 : 5/19/2010 4:16:05 PM
21: 1-16888, 1-16C59
--- : Test 8 : 5/19/2010 4:16:05 PM
22: 1-16892, 1-16C5A
--- : Test 8 : 5/19/2010 4:16:05 PM
23: 1-1689C, 1-16C5B
--- : Test 8 : 5/19/2010 4:16:05 PM
24: 1-168A6, 1-16C5C
--- : Test 8 : 5/19/2010 4:16:05 PM
25: 1-168B0, 1-16C5D
--- : Test 8 : 5/19/2010 4:16:05 PM
26: 1-168BA, 1-16C5E
--- : Test 8 : 5/19/2010 4:16:05 PM
27: 1-168C4, 1-16C5F
--- : Test 8 : 5/19/2010 4:16:05 PM
28: 1-16878, 1-16C60
--- : Test 7 : 5/19/2010 4:15:59 PM
29: 1-1687F, 1-16C61
--- : Test 7 : 5/19/2010 4:15:59 PM
30: 1-16889, 1-16C62
--- : Test 7 : 5/19/2010 4:15:59 PM
31: 1-16893, 1-16C63
--- : Test 7 : 5/19/2010 4:15:59 PM
32: 1-1689D, 1-16C64
--- : Test 7 : 5/19/2010 4:15:59 PM
33: 1-168A7, 1-16C65
--- : Test 7 : 5/19/2010 4:15:59 PM
34: 1-168B1, 1-16C66
--- : Test 7 : 5/19/2010 4:15:59 PM
35: 1-168BB, 1-16C67
--- : Test 7 : 5...
MIDSET Deleted: 1-1,1-16871

Server: /O=FIRST ORGANIZATION/OU=EXCHANGE ADMINISTRATIVE GROUP (FYDIBOHF23SPDLT)/CN=CONFIGURATION/CN=SERVERS/CN=BILONGE12MB1/CN=MICROSOFT PUBLIC MDB

One thing to notice is that the end of the list of items you just have a “…”. In fact, it doesn’t even give the whole date for that last item – the date is truncated. This “…” at the end means there was more stuff there, but we didn’t log the rest. Because of this, you can’t really see the entire list of items that were included in this backfill response.

Anyway, if any one of these items fails in content conversion, all the other items in this message also fail to replicate. Because of this, a single problem item can prevent many other items from replicating, even when those other items are just fine.

Fortunately, there’s an easy way to find the problem items. You can let the replication engine narrow down the problem for you just by changing one simple setting.

On the store that’s sending the backfill response (the store that has the data), set the replication message size limit to 1KB. Here are some example screenshots of the size limit from various tools:

In the top two screenshots, I had already changed the limit to 1KB. In the bottom one I had not changed it yet, so it was still at its default of 300KB.

Once you’ve turned up logging, changed the limit, and re-mounted the public store, go to the server that is missing the data and force a backfill timeout on that folder. In Exchange 2007 and 2010, you do this with the Update-PublicFolder cmdlet, or by right-clicking on the folder and choosing “Update Content” in the PF management GUI. In Exchange 2003, you do this from ESM by going to the public folder tree, clicking on the folder in question in the left-hand pane, choosing the Status tab in the right-hand pane, right-clicking on the server that doesn’t have the data, and choosing “Synchronize Content”.

Because of the tiny replication size limit, instead of a few big replication messages, you’ll see a ton of tiny ones. Assuming you have logging turned up on Replication Incoming and Replication Outgoing, the application log on the server that has the data will look something like this:

That 3026 you see at the bottom is the backfill request from the other server. Then you have a huge series of 3021’s, one for each backfill response. Each of these responses will only contain one or two messages.

Once this process completes, most of the good messages will have successfully replicated. At that point, you can go and force a backfill timeout one more time. You’ll see another backfill request followed by some responses – but this time, since all the good stuff replicated, you’ll only see a few backfill responses. You can look at each of those response events to see the messages listed therein.

And with that information, you know which messages to focus on to determine what the problem is! You can force a backfill timeout over and over, and you should see the store repeatedly try to replicate those same problem items every time.

Once you’re done troubleshooting, you can change the replication message size limit back to 300KB.

Fixing Public Folder Directory Objects That Aren’t Linked To The Hierarchy

Bill Long [Exchange] — Wed, 12 May 2010 20:43:40 GMT

In my recent post on the Exchange Team Blog, I briefly mentioned the dangers of forcibly removing an administrative group from ADSI Edit. The most common unintended consequence of this is a deletion of the directory object that represents the Public Folder Hierarchy.

Deleting this object creates quite a mess, because all of your mail-enabled public folders and all of your public folder stores point to it. Even if you create a new hierarchy object and manually link the stores to it by setting the msExchOwningPFTree value, you still have a ton of public folder directory objects in Microsoft Exchange System Objects with no homeMDB.

I had a customer today in this situation, and I wrote a quick Powershell script to fix all the directory objects for the mail-enabled public folders. Here it is.

If you find yourself in this situation, you’ll still need to create a new hierarchy object and link your public folder stores to it. This script does not fix that part of the problem, although it wouldn’t be hard to adapt it to link the stores as well.

To keep things simple, this only runs against one domain at a time. You’ll need to change the container to point to the Microsoft Exchange System Objects container in each different domain.

# Link-PFProxies.ps1
#
# Change these two values to match your environment.
# $container should point to the MESO container you want to run against.
# $pfTreeDN should contain the distinguishedName of the public folder hierarchy object.

$container = [ADSI]("LDAP://CN=Microsoft Exchange System Objects,DC=contoso,DC=com")
$pfTreeDN = "CN=Public Folders,CN=Folder Hierarchies,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=First Organization,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=contoso,DC=com"

#################################################

$filter = "(!(homemdB=*))"
$propertyList = @("distinguishedName")
$scope = [System.DirectoryServices.SearchScope]::OneLevel

$finder = new-object System.DirectoryServices.DirectorySearcher($container, $filter, $propertyList, $scope)
$finder.PageSize = 100
$results = $finder.FindAll()

("Found " + $results.Count + " folder proxies with no homeMDB...")
foreach ($result in $results)
{
    ("Fixing object: " + $result.Path)
    $entry = $result.GetDirectoryEntry()
    $entry.Put("homeMDB", $pfTreeDN)
    $entry.SetInfo()
}

ExFolders Update

Bill Long [Exchange] — Fri, 07 May 2010 20:41:04 GMT

Due to a mix-up on my part, the April 13th update of ExFolders was actually missing some of the fixes from the previous release. Some of the bits included were from a preview build of ExFolders that was created prior to Exchange 2010 RTM. Oops!

The web site was just updated with the proper bits, so please grab the latest here.

Public Folder Replication And Legacy Admin Groups

Bill Long [Exchange] — Wed, 05 May 2010 19:49:57 GMT

Today, a new post from me went up on the Exchange Team Blog regarding a particular replication error in Exchange 2010. Be sure to check it out.

Ambiguous SID Error in PFDAVAdmin

Bill Long [Exchange] — Tue, 04 May 2010 15:09:37 GMT

May 13, 2010 Edit: The Download Center has been updated with the new build of PFDAVAdmin which contains the fix for this issue.

With the release of the new PFDAVAdmin a few weeks ago, some customers started running into an “Ambiguous SID” error when trying to add the Everyone group or Anonymous group in the permissions window. The error looks like this:

When you add a new entity to the permissions window, PFDAVAdmin looks up all objects with that SID to make sure it is unique. However, in some cases it’s actually normal to have more than one object with an ObjectSid that matches the Everyone or Anonymous SID. You should always have one in the configuration context under CN=WellKnown Security Principals, but many environments will also have one in the domain context under CN=Foreign Security Principals. This is when PFDAVAdmin finds two matches for these SIDs, and it chokes.

One customer worked with me directly to track down the problem (thanks Giuliano!), and it’s fixed in my internal build. The Download Center will be updated with the new build, hopefully next week.

In the meantime, if you have an urgent need for the fix, click the Email link to the left to contact me directly.

If you’re getting this error with something other than Everyone or Anonymous, then you can use the following Powershell script to figure out which objects match the SID in question.

# Find-Sid.ps1
#
# The purpose of this script is to find all objects that have a
# given SID in objectSid, sidHistory, or msExchMasterAccountSid.
#
# Syntax:
#
# .\Find-Sid <sid>
#
# Example:
#
# .\Find-Sid S-1-1-0

param([string]$sidString)

$gcRootDSE = [ADSI]"GC://RootDSE"
$gcRoot = [ADSI]("GC://" + $gcRootDSE.dnsHostName)

$sid = new-object System.Security.Principal.SecurityIdentifier($sidString)
[byte[]]$sidBytes = ,0 * $sid.BinaryLength
$sid.GetBinaryForm($sidBytes, 0)

$byteString = ""
for ($x = 0; $x -lt $sidBytes.Length; $x++)
{
$byteString = $byteString + "\" + $sidBytes[$x].ToString("X2")
}

$filter = "(|(objectSid=" + $byteString + ")(sidHistory=" + $byteString + ")(msExchMasterAccountSid=" + $byteString + "))"
$searcher = new-object System.DirectoryServices.DirectorySearcher($gcRoot, $filter, @("distinguishedName"), [System.DirectoryServices.SearchScope]::Subtree)
$results = $searcher.FindAll()

"Matching objects:"
foreach ($result in $results)
{
$result.Properties["distinguishedname"]
}

Public Folder Admin Permissions Versus Client Permissions

Bill Long [Exchange] — Thu, 29 Apr 2010 04:47:37 GMT

Most Exchange administrators have noticed at one time or another that there are folders they can see and access through admin tools such as Exchange System Manager or PFDAVAdmin, but they can’t see or access those same folders through Outlook or OWA. I see a lot of confusion over this issue, so I want to explain a bit about how this works.

Each public folder has two sets of permissions – client permissions and admin permissions. These are stored in two separate properties on the folder. Let’s start by taking a look at these two different ACLs.

Client Permissions

When you’re thinking about public folder permissions, you’re usually thinking about client permissions. Client permissions are the ones you see when you get properties on a folder in Outlook, or when you use the Get-PublicFolderClientPermission cmdlet, or when you look at Folder Permissions in the ExFolders or PFDAVAdmin tool. Here are some screenshots of looking at client permissions from different interfaces.

Client permissions in Exchange System Manager

Client permissions in Outlook 2007

Client permissions in PFDAVAdmin

Client permissions in Powershell

Since folders in the hierarchy have typically been created by many different users, the client permissions are often different on each folder. The creator of the folder will be the Owner, and will often change the other rights on the folder as well, so there’s no telling what the client permissions will look like throughout the hierarchy.

Admin Permissions

Each public folder has a separate ACL that contains admin permissions. Admin permissions can only be seen through admin tools such as ESM or the Get-PublicFolderAdministrativePermission cmdlet. Here are some screenshots of looking at admin permissions.

Admin permissions in ESM

Admin permissions in Powershell

Unlike client permissions, admin permissions are typically the same on every folder in the hierarchy. While it’s possible to change the admin permissions on individual folders, that’s a very rare scenario. This means that in most environments, Exchange admins do have admin permissions to every folder in the public folder hierarchy.

Am I using admin or client permissions?

When you access a public folder, we only honor one ACL or the other – either the client OR the admin permissions, but not both! The ACL we use to determine your access to a folder is determined entirely by the tool you are using to access it.

If you are using Outlook to access a public folder, we only look at the client permissions.
If you are using Outlook Web Access to access a public folder, we only look at the client permissions.
If you are using Exchange System Manager to access a public folder, we only look at the admin permissions.
If you are using the Exchange Powershell public folder cmdlets or the Exchange 2007 public folder management GUI, we only look at the admin permissions.
If you are using PFDAVAdmin or ExFolders to access a public folder, we only look at the admin permissions.

This means that while an administrator can see and access all the public folders through an admin tool, he does not have the same rights when using a client such as Outlook. The behavior is different between the two types of programs, because we only look at one ACL or the other.

This is actually controlled by the tool or client you are using. Any program has the option of asserting admin rights when connecting to the public store, but it is typically hard-coded and the user does not get a choice. The client programs never assert admin rights, while the admin programs always do. If you’re an admin using an admin tool, then you get to use the admin ACL instead of the client ACL to access the public folders through that tool, because the tool asserted admin rights when it connected to the public store.

One way to see this is to use the MFCMAPI tool. After logging on, if you go to the MDB menu and choose Open Public Store, you’ll see the following dialog box:

This shows a list of optional flags you can pass when opening the public store. Notice the very first one: 0x00000001 OPENSTORE_USE_ADMIN_PRIVILEGE. When a program passes this flag while opening the public store, it’s saying, “I’m an admin so let me use the admin permissions instead of the client permissions.” The store checks to see if the user has Exchange admin rights, and if so, he gets to use the admin ACL. MFCMAPI is the only tool I’m aware of that actually lets you choose whether to use the client or admin ACL by letting you modify the flags passed when opening the public store.

If you were writing your own tool, such as a migration tool that needs to access all the public folder content, then this is exactly what you would want to do – pass the OPENSTORE_USE_ADMIN_PRIVILEGE flag when opening the public folder store. If you assert admin rights when opening the store, then your tool automatically gets access to all the public folders (assuming it’s being run by an Exchange admin), and you don’t have to worry about the different client permissions on all the folders.

If you do need to access all public folders from a program that uses client permissions, you’ll need to use an admin tool to grant client permissions to the desired user on all the folders. This is easily accomplished using ESM, PFDAVAdmin, ExFolders, AddUsersToPFRecursive.ps1, etc.

Importing Public Folder Contacts From A CSV File

Bill Long [Exchange] — Fri, 23 Apr 2010 15:30:00 GMT

I just had a customer who needed to migrate contacts from an external database into a public folder. They could export the contacts from the database to a CSV, but they needed a way to get the CSV into the public folder. Last night, I whipped up this quick script.

This morning, I was looking at my RSS feeds, and found that Glen Scales had just solved the same problem in a new blog post. Figures! Here is my version, but I recommend taking a look at his work as well.

That said, here’s what an example CSV might look like for use with my script:

GivenName,Surname,EmailAddress1,EmailAddress2,HomePhone,MobilePhone,Business
Contact,1,contact1@contoso.com,contact1@northwindtraders.com,888-555-1212,888-555-1212,One Microsoft Way%Redmond%WA%US%98052
Contact,2,contact2@contoso.com,contact2@northwindtraders.com,888-555-1212,888-555-1212,One Microsoft Way%Redmond%WA%US%98052

Notice the weird physical address syntax where each part of the address is separated with a % character. This was my way of dealing with the need to parse out the individual address fields (Street, City, State, CountryOrRegion, PostalCode). Everything else is pretty straightforward. There are a lot of properties you can set that are not shown in this example CSV. I’ve listed most of them in the comments at the top of the script. Here it is.

# Import-PFContacts
#
# The purpose of this script is to import contacts into a public folder on
# Exchange 2007 or 2010 from a CSV file. The script reads from a CSV and then
# creates a contact with the appropriate fields using the Exchange Web
# Services Managed API.
#
# Requirements:
#
# This script requires Powershell 2.0. If you get the following error then you do not
# have Powershell 2.0 installed:
#
# The term 'Import-Module' is not recognized as a cmdlet, function, operable program,
# or script file. Verify the term and try again.
#
# The script also requires the EWS managed API, which can be downloaded here:
# http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=c3342fb3-fbcc-4127-becf-872c746840e1
#
# Make sure the Import-Module command below matches the DLL location of the API.
#
# Syntax:
# .\Import-PFContacts admin@contoso.com C:\SomeFile.csv "Top Level Folder\Subfolder\Contacts Folder"
#
# The first parameter is the email address that we'll use to run autodiscovery
# and logon. The second parameter is the import file, and the third is the folder
# where we want to create the contacts.
#
# Below you will find details on the columns that are supported in the import
# file. Note that column names are CASE SENSITIVE.
#
# The import file can contain the following email address columns:
#
# EmailAddress1, EmailAddress2, EmailAddress3
#
# The import file can contain the following phone number columns:
#
# AssistantPhone, BusinessFax, BusinessPhone, BusinessPhone2, Callback, CarPhone,
# CompanyMainPhone, HomeFax, HomePhone, HomePhone2, Isdn, MobilePhone, OtherFax,
# OtherTelephone, Pager, PrimaryPhone, RadioPhone, Telex, TtyTddPhone
#
# The import file can contain the following physical address columns. Note that
# the syntax for these is a little odd at the moment. The address has five fields,
# which must be in order and separated by % characters. They are street, city,
# state, country or region, and postal code. Some of the fields can be left
# blank if desired. An example address entry might look like this:
# One Microsoft Way%Redmond%WA%US%98052
# The physical address columns are:
#
# Home, Business, Other
#
# The imort file can contain the following other columns:
#
# AssistantName, Birthday, BusinessHomePage, CompanyName, CompleteName, Department,
# DisplayName, FileAs, Generation, GivenName, Initials, JobTitle, Manager, MiddleName,
# Mileage, NickName, OfficeLocation, Profession, SpouseName

param([string]$autoDiscoverAddress, [string]$importFile, [string]$folderPath)

##########
#
# This path must match the install location of the EWS managed API. Change it if needed.
#
Import-Module -Name "C:\Program Files\Microsoft\Exchange\Web Services\1.0\Microsoft.Exchange.WebServices.dll"
#
##########

$emailAddressColumns = new-object System.Collections.Specialized.StringCollection
$foo = $emailAddressColumns.AddRange(@("EmailAddress1", "EmailAddress2", "EmailAddress3"))
$phoneNumberColumns = new-object System.Collections.Specialized.StringCollection
$foo = $phoneNumberColumns.AddRange(@("AssistantPhone", "BusinessFax", "BusinessPhone", "BusinessPhone2", "Callback", "CarPhone"))
$foo = $phoneNumberColumns.AddRange(@("CompanyMainPhone", "HomeFax", "HomePhone", "HomePhone2", "Isdn", "MobilePhone", "OtherFax"))
$foo = $phoneNumberColumns.AddRange(@("OtherTelephone", "Pager", "PrimaryPhone", "RadioPhone", "Telex", "TtyTddPhone"))
$physicalAddressColumns = new-object System.Collections.Specialized.StringCollection
$foo = $physicalAddressColumns.AddRange(@("Business", "Home", "Other"))

$exchService = new-object Microsoft.Exchange.WebServices.Data.ExchangeService([Microsoft.Exchange.WebServices.Data.ExchangeVersion]::Exchange2007_SP1)
$exchService.UseDefaultCredentials = $true
$exchService.AutodiscoverUrl($autoDiscoverAddress)
$pfsRoot = [Microsoft.Exchange.WebServices.Data.Folder]::Bind($exchService, [Microsoft.Exchange.WebServices.Data.WellKnownFolderName]::PublicFoldersRoot)

$tinyView = new-object Microsoft.Exchange.WebServices.Data.FolderView(2)
$displayNameProperty = [Microsoft.Exchange.WebServices.Data.FolderSchema]::DisplayName
$folderPathSplits = $folderPath.Split(@('\'))
$folder = $pfsRoot
for ($x = 0; $x -lt $folderPathSplits.Length;$x++)
{
    $filter = new-object Microsoft.Exchange.WebServices.Data.SearchFilter+IsEqualTo($displayNameProperty, $folderPathSplits[$x])
    $results = $folder.FindFolders($filter, $tinyView)
    if ($results.TotalCount -gt 1)
    {
         ("Ambiguous name: " + $folderPathSplits[$x])
         return
    }
    elseif ($results.TotalCount -lt 1)
    {
         ("Folder not found: " + $folderPathSplits[$x])
         return
    }
    $folder = $results.Folders[0]
}

$importReader = new-object System.IO.StreamReader($importFile)
$headers = $importReader.ReadLine().Split(@(','))
$line = 1

while ($null -ne ($buffer = $importReader.ReadLine()))
{
    $line++
    if ($buffer.Length -gt 0)
    {
        $newContact = new-object Microsoft.Exchange.WebServices.Data.Contact($exchService)
        $columns = $buffer.Split(@(','))
        for ($x = 0; $x -lt $headers.Length; $x++)
        {
            if ($columns[$x].Length -lt 1)
            {
                continue
            }
            if ($emailAddressColumns.Contains($headers[$x]))
            {
                $emailAddressType = $headers[$x]
                $emailAddressType = [Microsoft.Exchange.WebServices.Data.EmailAddressKey]::$emailAddressType
                $newContact.EmailAddresses[$emailAddressType] = $columns[$x]
            }
            elseif ($phoneNumberColumns.Contains($headers[$x]))
            {
                $phoneType = $headers[$x]
                $phoneType = [Microsoft.Exchange.WebServices.Data.PhoneNumberKey]::$phoneType
                $newContact.PhoneNumbers[$phoneType] = $columns[$x]
            }
            elseif ($physicalAddressColumns.Contains($headers[$x]))
            {
                $addressFields = $columns[$x].Split(@('%'))
                $addressType = $headers[$x]
                $addressType = [Microsoft.Exchange.WebServices.Data.PhysicalAddressKey]::$addressType
                $address = new-object Microsoft.Exchange.WebServices.Data.PhysicalAddressEntry
                $address.Street = $addressFields[0]
                $address.City = $addressFields[1]
                $address.State = $addressFields[2]
                $address.CountryOrRegion = $addressFields[3]
                $address.PostalCode = $addressFields[4]
                $newContact.PhysicalAddresses[$addressType] = $address
            }
            elseif ($headers[$x] -eq "Birthday")
            {
                $newContact.Birthday = [System.DateTime]::Parse($columns[$x])
            }
            else
            {
                $attribute = $headers[$x]
                $newContact.$attribute = $columns[$x]
            }
        }
        $newContact.Save($folder.Id)
    }
}

"Done!"

Fixing Public Folder Replication Errors From Exchange 2003 to Exchange 2007 or 2010

Bill Long [Exchange] — Thu, 22 Apr 2010 19:18:00 GMT

In my post Accessing The Information Store From Powershell, I promised I would post a real-world Outlook Object Model example in the future, so here it is.

Back in 2008, I posted part 4 of my public folder replication troubleshooting series on the Exchange Team Blog, with part 4 focused on Exchange 2007. In that post, I described several content conversion issues that prevent items from replicating from Exchange 2003 to Exchange 2007. These same problems prevent replication to Exchange 2010.

We are looking at ways to possibly address this in the product, but for now, we often use a script I wrote to identify and correct some of these problems. The script uses Outlook Object Model from Powershell to dig through the public folder items on the Exchange 2003 side and examine them for known problems. It does not fix every possible problem, but it does fix a few of the common ones and can save you quite a bit of manual work.

If you’ve called in with one of these problems, you may have used an earlier version of this script already. This is the latest version. Enjoy!

# Fix-PFItems.ps1
#
# This script should be run on a workstation where Outlook is installed
# as well as Powershell. You do not need the Exchange Management Console
# installed in order to use this. The Outlook profile you are using should be
# one that will access public folders on the Exchange 2003 server, because
# the whole point is to fix the items on the 2003 side so they will replicate
# to Exchange 2007/2010. If you're not sure which replica of a folder your
# profile is accessing, launch MFCMapi, logon to the same profile, navigate
# to the public folder, and look at the PR_REPLICA_SERVER property. This will
# tell you which replica your client is looking at.
#
# Syntax info:
#
# Examples:
# .\Fix-PFItems -folderPath "Departments\HR" -splitOnBadchar $true -resetEmptyCategories $false -doAppointments $false -doInstanceKey $false -doSubfolders $false
# .\Fix-PFItems "Departments\HR" $true $false $false $false $false
#
# folderPath should be in the form: "TopLevelFolder\Subfolder\Subfolder 2"
# a folderPath of "" will run against all public folders
# splitOnBadChar determines whether we split the category into two names, or just replace badChar
# resetEmptyCategories will clear all categories on items that already appear to have no
#   categories. This ensures that the categories value is REALLY empty, and should fix
#   items that have an empty array in the categories. Be aware that since this changes ALL
#   items that appear to have no categories, it could cause a lot of replication.
# doAppointments determines whether appointment items are processed. Note that if this is $true
#    and $commitChanges is set to $true, ALL appointment items in the specified folders
#    will be modified by the script. This will change the last modified time and cause
#    replication.
# doInstanceKey determines whether we clear PR_INSTANCE_KEY on items. Note that if this is $true
#    and $commitChanges is set to $true, ALL items in ALL folders that you specify will be
#    be modified.
# doSubfolders determines whether we automatically traverse subfolders of the specified folder
#
# This script will identify 5 types of problems:
#
# 1. Categories that contain a bad character, typically a comma. This error is identified in the
#    content conversion tracing by output that states:
#
#    PropertyValidationException: Property validation failed. Property = [{GUID}:'Keywords']
#    Categories Error = Element 0 in the multivalue property is invalid..
#
#    This problem will always be fixed if $commitChanges is $true (see below). The
#    -splitOnBadChar parameter lets you choose whether to split the category into two separate
#    categories (A category such as "catone,cattwo" would become two separate categories
#    "catone" and "cattwo") or to simply get rid of the bad character ("catone,cattwo" becomes
#    "catonecattwo").
#
# 2. Categories that contain an emtpy array. This problem is identified by the same tracing
#    output as that shown for problem #1. This problem can be fixed if -resetEmptyCategories
#     is $true and $commitChanges is changed to $true (see below). Note that this will modify
#    all items that appear to have no categories, as the script can't tell the difference
#    between an empty array and an item that truly has no categories.
#
# 3. Appointment items with invalid start/end dates. This problem can be fixed if $doAppointments
#     is $true and $commitChanges is changed to $true. See below. Note that this does not fix
#    every possible problem with appointments.
#
# 4. Address type properties that are longer than 9 characters. This problem can be identified
#    by content conversion tracing that states:
#
#    Property validation failed. Property = [{GUID}:0x8nnn] Email2AddrType Error =
#    Email2AddrType is too long: maximum length is 9, actual length is nn..
#
#    This problem cannot be fixed by the script. You can try to fix it manually using
#    mfcmapi, but sometimes these properties cannot be changed and the item
#    must be deleted.
#
# 5. A bad PR_INSTANCE_KEY. This problem can be identified by an error in the
#    content conversion tracing that says MapiExceptionPropsDontMatch. This problem can
#    be fixed by the script if -doInstanceKey is $true and $commitChanges is $true (see
#    below).
#

param([string]$folderPath, [bool]$splitOnBadchar, [bool]$resetEmptyCategories, [bool]$doAppointments, [bool]$doInstanceKey, [bool]$doSubfolders)

#
# By default, the script does NOT change anything. You must
# change $commitChanges to $true in order for the script to actually
# do anything.
#
# This has the potential to change a lot of items in your public folders!
# Run it in READ ONLY mode first to see what will get changed, and have a
# good backup just in case!
#
$commitChanges = $false

# $badChar is the character we want to get rid of in the category names.
# Normally it's a comma, but it can be changed to anything you want.
$badChar = ","

# $reportAll will make the script report all categories and address types
# that were found. This is just for reporting purposes so you can see what values
# exist.
$reportAll = $false

#
##########################################################################
#
# Be careful changing anything below this point.
#
##########################################################################
#

$allCategories = new-object System.Collections.Specialized.StringCollection
$allAddressTypes = new-object System.Collections.Specialized.StringCollection

function GetNamedFromCollection($name, $collection)
{
    foreach ($item in $collection)
    {
        if ($item.Name -eq $name -or $item.DisplayName -eq $name)
        {
            return $item
        }
    }
    return $null
}

function RecordAddrType($type)
{
    if ($reportAll)
    {
        if (!($allAddressTypes.Contains($type)))
        {
            $temp = $allAddressTypes.Add($type)
        }
    }
}

function DoCategories($item)
{
    $categoryArray = $item.PropertyAccessor.GetProperty("urn:schemas-microsoft-com:office:office#Keywords")
    if ($categoryArray.Length -eq 0 -and $resetEmptyCategories -eq $true)
    {
        if ($commitChanges)
        {
            "    Resetting categories..."
            $item.PropertyAccessor.SetProperty("urn:schemas-microsoft-com:office:office#Keywords", $null)
            $item.Save()
        }
        else
        {
            "    Would have reset the categories, but we're in Read-Only mode."
        }
    }
    else
    {
        $fixedCategories = $false
        [string[]]$newCategoryArray = @()
        foreach ($cat in $categoryArray)
        {
            if ($reportAll)
            {
                if (!($allCategories.Contains($cat)))
                {
                    $temp = $allCategories.Add($cat)
                }
            }
            if ($cat.Contains($badChar))
            {
                if (!($splitOnBadchar))
                {
                    # In this case, we just replace the badChar with nothing
                    $newCategoryArray += $cat.Replace($badChar, "")
                }
                else
                {
                    # In this case, we split it into multiple separate categories
                    $categorySplit = $cat.Split($badChar)
                    foreach ($newCat in $categorySplit)
                    {
                        $newCat = $newCat.Trim()
                        if ($newCat.Length -gt 0)
                        {
                            $newCategoryArray += $newCat
                        }
                    }
                }
                $fixedCategories = $true
            }
            elseif ($cat.Trim() -eq "")
            {
                # this category should be deleted from the item
                $fixedCategories = $true
            }
            else
            {
                $newCategoryArray += $cat
            }
        }

        if ($fixedCategories)
        {
            "    Old category list for this item:"
            foreach ($cat in $categoryArray)
            {
                ("        " + $cat)
            }
            "    New category list for this item:"
            foreach ($cat in $newCategoryArray)
            {
                ("        " + $cat)
            }
            if ($commitChanges)
            {
                $item.PropertyAccessor.SetProperty("urn:schemas-microsoft-com:office:office#Keywords", $newCategoryArray)
                $item.Save()
                ("    Changes saved.")
            }
            else
            {
                ("    Changes not saved (READ ONLY mode).")
            }
        }
    }
}

function DoAppointmentProps($item)
{
    if ($item.Class -eq 26) # olAppointment in the olObjectClass enumeration is 26
    {
        if ($commitChanges)
        {
            ("    Updating appointment props: " + $item.Subject)
            if ($item.IsRecurring)
            {
                $recurPattern = $item.GetRecurrencePattern()
                $recurPattern.StartTime = $recurPattern.StartTime
                $recurPattern.EndTime = $recurPattern.EndTime
                $item.Save()
            }
            else
            {
                $item.Start = $item.Start
                $item.End = $item.End
                $item.Save()
            }
        }
        else
        {
            ("    Appointment props not updated (READ ONLY mode).")
        }
    }
}

function DoInstanceKey($item)
{
    if ($commitChanges)
    {
        ("    Clearing PR_INSTANCE_KEY: " + $item.Subject)
        $item.PropertyAccessor.DeleteProperty("http://schemas.microsoft.com/mapi/proptag/0x0FF60102")
        $item.Save()
        ("    Changes saved.")
    }
    else
    {
        ("    PR_INSTANCE_KEY not cleared (READ ONLY mode).")
    }
}

function DoAddrType($item)
{
    $senderAddrType = $item.PropertyAccessor.GetProperty("http://schemas.microsoft.com/mapi/proptag/0x0C1E001E")
    RecordAddrType $senderAddrType
    if ($senderAddrType.Length -gt 9)
    {
        ("    WARNING! PR_SENDER_ADDRTYPE is too long: " + $senderAddrType)
    }

    $sentRepresentingAddrType = $item.PropertyAccessor.GetProperty("http://schemas.microsoft.com/mapi/proptag/0x0064001E")
    RecordAddrType $sentRepresentingAddrType
    if ($sentRepresentingAddrType.Length -gt 9)
    {
        ("    WARNING! PR_SENT_REPRESENTING_ADDRTYPE is too long: " + $sentRepresentingAddrType)
    }

    $recipients = $item.Recipients
    if ($recipients -ne $null)
    {
        foreach ($recipient in $recipients)
        {
            $addrType = $recipient.PropertyAccessor.GetProperty("http://schemas.microsoft.com/mapi/proptag/0x3002001E")
            if ($addrType.Length -gt 9)
            {
                ("    WARNING! A recipient PR_ADDRTYPE is too long: " + $addrType)
            }
        }
    }

    if ($item.Email1AddressType.Length -gt 0)
    {
        RecordAddrType $item.Email1AddressType
    }
    if ($item.Email1AddressType.Length -gt 9)
    {
        ("    WARNING! Email1AddressType is too long: " + $item.Email1AddressType)
    }

    if ($item.Email2AddressType.Length -gt 0)
    {
        RecordAddrType $item.Email2AddressType
    }
    if ($item.Email2AddressType.Length -gt 9)
    {
        ("    WARNING! Email2AddressType is too long: " + $item.Email2AddressType)
    }

    if ($item.Email3AddressType.Length -gt 0)
    {
        RecordAddrType $item.Email3AddressType
    }
    if ($item.Email3AddressType.Length -gt 9)
    {
        ("    WARNING! Email3AddressType is too long: " + $item.Email3AddressType)
    }
}

function DoFolder($folder)
{
    $replicaServer = $folder.PropertyAccessor.GetProperty("http://schemas.microsoft.com/mapi/proptag/0x6644001E")
    $items = $folder.Items
    ($items.Count.ToString() + " items found in folder " + $folder.FolderPath)
    ("Accessing this folder on server: " + $replicaServer)
    foreach ($item in $items)
    {
        ("Checking item: " + $item.Subject)
        if ($item.PropertyAccessor -ne $null)
        {
            DoCategories($item)
            DoAddrType($item)
            if ($doAppointments)
            {
                DoAppointmentProps($item)
            }
            if ($doInstanceKey)
            {
                DoInstanceKey($item)
            }
        }
        else
        {
            "    No PropertyAccessor on this item. It may be in a conflict state."
        }
    }
    if ($doSubfolders)
    {
        foreach ($subfolder in $folder.Folders)
        {
            DoFolder($subfolder)
        }
    }
}

"Starting..."

if ($commitChanges)
{
"commitChanges has been set to TRUE. The script WILL save changes."
}
else
{
"commitChanges is set to FALSE. Running in READ ONLY mode. Changes will NOT be saved."
}

$outlook = new-object -com Outlook.Application
if (!($outlook.Version -like "12.*" -or $outlook.Version -like "14.*"))
{
("This script requires Outlook 2007 or 2010. Your version: " + $outlook.Version)
return
}
$mapi = $outlook.GetNamespace("MAPI")

#
# First, check the categories list and fix if necessary
#

"Checking Outlook categories list..."
$categories = $mapi.Categories
foreach ($category in $categories)
{
    if ($category.Name -ne $null)
    {
        if ($category.Name.Contains($badChar))
        {
            [string[]]$newNames = @()
            ("    Fixing category in Outlook category list: " + $category.Name)
            if (!($splitOnBadchar))
            {
                # In this case, we just replace the badChar with nothing
                $newName += $category.Name.Replace($badChar, "")
            }
            else
            {
                # In this case, we split it into multiple separate categories
                $categorySplit = $category.Name.Split($badChar)
                foreach ($newCat in $categorySplit)
                {
                    if ($newCat.Length -gt 0)
                    {
                        $newNames += $newCat
                    }
                }
            }
            if ($commitChanges)
            {
                $foo = $categories.Remove($category.Name)
                foreach ($newName in $newNames)
                {
                    $foo = $categories.Add($newName)
                }
            }
        }
    }
}

#
# Categories list should be in good shape now.
# Now find the specified folder.
#

"Finding the specified folder..."
$session = $mapi.Session
$pfStore = $null
foreach ($store in $mapi.Stores)
{
    if ($store.ExchangeStoreType -eq 2)
    {
        $pfStore = $store
    }
}
$pfRoot = $pfStore.GetRootFolder()
$allPFs = GetNamedFromCollection "All Public Folders" $pfRoot.Folders
if ($allPFs -eq $null)
{
    "Couldn't find All Public Folders folder."
    return
}

if ($pfStore -eq $null)
{
"Couldn't find public folder store."
return
}

$pfRoot = $pfStore.GetRootFolder()

$folderPath = $folderPath.Trim("\")
$folderPathSplit = $folderPath.Split("\")
$folder = $allPFs

if ($folderPath.Length -gt 0)
{
    "Traversing folder path..."
    for ($x = 0; $x -lt $folderPathSplit.Length; $x++)
    {
        $folder = GetNamedFromCollection $folderPathSplit[$x] $folder.Folders
    }
    if ($folder -eq $null)
    {
        ("Could not find folder: " + $folderPath)
        return
    }
    ("Found folder: " + $folder.FolderPath)
}

#
# Got the folder.
# Start processing the folder and subfolders.
#

DoFolder $folder

"Done!"

if ($reportAll)
{
    ""
    "Categories found:"
    $allCategories
    ""
    "Address types found:"
    $allAddressTypes
}

Checking For Protected ACLs In Active Directory

Bill Long [Exchange] — Tue, 13 Apr 2010 17:43:40 GMT

Exchange Server setup grants permissions to various groups at various places in the domain and configuration contexts of Active Directory. Since Exchange relies on these permissions in order for everything to work properly, from time to time we see cases where something is not working, and we eventually track it down to the fact that an object has been set to NOT inherit permissions from its parent. This is called a protected ACL, because it is protected from inheriting permissions from the parent. Here’s how this looks in Active Directory Users & Computers:

You can imagine what fun it is to go hunting around, clicking through multiple dialog boxes to try and find an object where inheritance has been unchecked in the GUI.

Fortunately, Powershell makes it easy to evaluate whether an ACL is protected by exposing the ObjectSecurity.AreAccessRulesProtected member on a DirectoryEntry object. This makes it relatively simple to write a script that scans the entire Active Directory for objects with protected ACLs, and I wrote just such a script a couple of months back.

Be aware that it’s normal for the ACLs on certain objects to be protected, such as certain containers in the domain’s System container, as well as any user objects affected by AdminSDHolder. Don’t panic when you see that inheritance is disabled on your administrator accounts, or on the various built-in groups, or the policies in the System container, etc. That’s totally normal. From an Exchange perspective, in the domain context we’d be looking for normal users or OUs that contain users. In the configuration context we’d be looking for anything under the Microsoft Exchange container.

In any case, here is the script. Read the top for usage details.

#################################################################################
#
# The sample scripts are not supported under any Microsoft standard support
# program or service. The sample scripts are provided AS IS without warranty
# of any kind. Microsoft further disclaims all implied warranties including, without
# limitation, any implied warranties of merchantability or of fitness for a particular
# purpose. The entire risk arising out of the use or performance of the sample scripts
# and documentation remains with you. In no event shall Microsoft, its authors, or
# anyone else involved in the creation, production, or delivery of the scripts be liable
# for any damages whatsoever (including, without limitation, damages for loss of business
# profits, business interruption, loss of business information, or other pecuniary loss)
# arising out of the use of or inability to use the sample scripts or documentation,
# even if Microsoft has been advised of the possibility of such damages
#
#################################################################################
#
# Check-Inheritance
#
# The purpose of this script is to check for Active Directory objects that are
# set to NOT inherit permissions from their parent. It takes four parameters,
# all of which are optional:
#
# $BaseDN - This is the root container where we start our search. If not specified
#      we use the root of the current domain. A DC can be specified or not.
# $Recurse - This determines whether we search only the root, or whether we
#      recursively go through its child containers. $false by default.
# $ContainersOnly - This determines if we look only at containers, or if we also
#      inspect every individual leaf object for inheritance. $false by default.
# $Verbose - This determines whether we log every single object we look at,
#      or whether we log only problem objects. $false by default.
#
# Examples:
#
# .\Check-Inheritance -BaseDN "LDAP://CN=Users,DC=contoso,DC=com" -Recurse $true -ContainersOnly $false -Verbose $false
# .\Check-Inheritance "LDAP://DC1/CN=Users,DC=contoso,DC=com" $true $false $false
# .\Check-Inheritance "LDAP://CN=Users,DC=contoso,DC=com"
# .\Check-Inheritance
#
# Output looks like this:
#
# Inheritance disabled: CN=Someobject,DC=contoso,DC=com
# Inheritance disabled: CN=Someotherobject,DC=contoso,DC=com
#
# If verbose is turned on, it will also report the DN of every object it looks at:
#
# CN=Object1,DC=contoso,DC=com
# CN=Object2,DC=contoso,DC=com
# Inheritance disabled: CN=Object3,DC=contoso,DC=com
# CN=Object4,DC=contoso,DC=com

param([string]$BaseDN, [bool]$Recurse, [bool]$ContainersOnly, [bool]$Verbose)

function DoContainer($base)
{
    $finder = new-object System.DirectoryServices.DirectorySearcher($base, $filter, $propertyList, $scope)
    $finder.PageSize = 100
    $results = $finder.FindAll()
    if ($results.Count -gt 0)
    {
        foreach ($result in $results)
        {
        $entry = $result.GetDirectoryEntry()
        if ($entry.ObjectSecurity.AreAccessRulesProtected)
        {
            ("Inheritance disabled: " + $entry.distinguishedName)
        }
        elseif ($Verbose)
        {
            $entry.distinguishedName
        }
        if ($Recurse)
        {
            DoContainer $entry
        }
    }
    }
}

if ($BaseDN -eq "")
{
$dcRoot = [ADSI]"LDAP://RootDSE"
$BaseDN = ("LDAP://" + $dcRoot.dnsHostName + "/" + $dcRoot.DefaultNamingContext)
}

$filter = "(objectClass=*)"
if ($ContainersOnly)
{
$filter = "(|(objectClass=organizationalUnit)(objectClass=container))"
}

$propertyList = @("distinguishedName")
$scope = [System.DirectoryServices.SearchScope]::OneLevel

("Base container: " + $BaseDN)
("Recurse: " + $Recurse)
("ContainersOnly: " + $ContainersOnly)
("Verbose: " + $Verbose)
"Here we go..."
""
$baseEntry = [ADSI]$BaseDN
DoContainer $baseEntry

"Done!"

PFDAVAdmin Update Released

Bill Long [Exchange] — Fri, 09 Apr 2010 17:47:00 GMT

As announced over on the Exchange Team Blog, an updated version of PFDAVAdmin was released to the download center a few days ago. See that post for details. This updated version has been used internally for a while, but if you notice any problems with it, let me know.