Case in the corner series: The crashing Add User Wizard

[Today’s post comes to us courtesy of Damian Leibaschoff from Commercial Technical Support ]

The symptoms for this case were that after a few seconds of launching the Add User Wizard, the wizard would crash, without any user input.

When we face a user mode crash like this we usually try to get our hands on the dump that is generated so we can review what triggered the crash. Depending on the crash, the associated wizard log might also contain valuable information (this was not the case here). Furthermore, we might also rely on tools such as Sysinternal’s Process Monitor.

For this case we launched the application within our Windows Debugger to see what was taking place, here is a summary of what we found and how we found it:

1. Launched Windbg (elevated rights)
2. Verified that the public symbols were configured properly.
3. Selected File, Open Executable, and picked C:\Program Files\Windows Small Business Server\Bin\adduser.exe
4. Once it loaded, proceeded to let the program run while connected to the debugger by issuing the GO ‘g’ command.
   
   
5. The Debugger will break on the second chance exception, on this case a CLR exception, pointing us to an issue while running .Net Framework code.
   
   
6. At this point, we need to load the proper extensions to handle .Net Framework debugging so we can take a peek into what is going on. You do that by loading the SOS extension:
   .loadby sos mscorwks
   
   To view the exception, you can now use:
   !printexception or !pe
   
   The output for this (key lines highlighted):
   0:006> .loadby sos mscorwks
   0:006> !pe
   Exception object: 00000000027f30a0
   Exception type: System.ArgumentException
   Message: An entry with the same key already exists.
   InnerException: <none>
   StackTrace (generated):
       SP IP Function
       000000001B2AEF80
   000007FEF69C4BD3System_ni!System.ThrowHelper.ThrowArgumentException(System.ExceptionResource)+0x2ad1d3
       000000001B2AEFC0 000007FEF62AC16DSystem_ni!System.Collections.Generic.SortedList`2[[System.Int32, mscorlib],[System.__Canon, mscorlib]].Add(Int32, System.__Canon)+0x5d
       000000001B2AF030 000007FF001AF6EF
   AdminCommon!Microsoft.WindowsServerSolutions.Admin.Common.Utils.GetPrimarySmtpWorker(System.Object)+0x89f
       000000001B2AF130 000007FEF6DC175C
   mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)+0x9c
       000000001B2AF180 000007FEF6E36112
   mscorlib_ni!System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(System.Threading._ThreadPoolWaitCallback)+0x62
   000000001B2AF1D0 000007FEF6E35F50
   mscorlib_ni!System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(System.Object)+0x50
   
   StackTraceString: <none>
   HResult: 80070057
   
7. We can easily tell by looking at the exception that we are getting a duplicate entry in a list of what should be unique items. By looking at the CLR call stack that led to the exception we can also see our code trying to find the Primary SMTP address from Exchange. From this, we rapidly found the lines of code that we were using when we hit the exception, confirmed what they were doing and quickly found the cause of the problem. Before we go into the details of what was wrong, let’s continue this scenario as if we did not have access to the source code, how would we have found what was wrong going the long way.
   
   When we look at a working Add User Wizard, we can see where we are using the SMTP address in plain sight:
   
   
   
   Once we see that, it is just a matter of understanding how Exchange 2007 manages SMTP addresses to find the place where the problem lies, in our case, the E-mail Address Policies <More information> . On normal Exchange server in SBS, they would look like this:
   
   
   
   Yet on the system where we had the crash, they looked like this:
   
   
   
   You can also get the output from Exchange Management Shell
   [PS] C:\>Get-EmailAddressPolicy
   
   Name                                                Priority                  RecipientFilter
   ----                                                    --------                 ---------------
   Default Policy                                  Lowest                  Alias -ne $null
   Windows SBS Email Address         Lowest                  Alias -ne $null
   Policy
   
   Can you spot what is wrong?
   ..
   ..
   ..
   
   Both policies have the SAME priority, not only that, they have the default LOWEST priority. This priority is a special case, where its actual value in active directory is 2147483647, this is the highest this value can be, thus the lowest priority. Our code queries for all AD objects that are of objectClass=msExchRecipientPolicy and the one with the lowest msExchPolicyOrder is the primary, we should not have duplicates.
   
8. To fix this we need to correct the priority of the “Windows SBS Email Address Policy” as that is clearly incorrect. Unfortunately we cannot change it using the Exchange native tools as we get an error:
   
   [PS] C:\>Set-EmailAddressPolicy "Windows SBS Email Address Policy" -Priority 1
   Set-EmailAddressPolicy : The operation cannot be completed on the default e-mail address policy.
   At line:1 char:23
   + Set-EmailAddressPolicy <<<< "Windows SBS Email Address Policy" -Priority 1
   
   To get this fixed we had to use ADSIEDIT.MSC and manually edit the AD object:
   
   
   
   
   
   
   Changing the attribute to the correct value was all that was needed to regain functionality.

This was a very strange situation, the most likely cause was that the original Default Policy had been manually renamed to the Windows SBS Email Address Policy and a subsequent service pack for Exchange tried to recreate the missing Default Policy or it could have been as simple as some incorrect manipulation of AD objects..