• Exchange 2010 DAG - NetworkManager has not yet been initialized

     

    Recently, in two separate occasions, I had to assist in resolving an issue where a member of an Exchange 2010 database availability group (DAG) failed to participate in the DAG's Cluster Communications and therefore were unable to bring any database on those servers online.  In both instances, this occurred after the server was rebooted.  While each issue had a slightly different resolution, I am fairly confident that they are related.  And since it took awhile to isolate and resolve these issues, I'd thought I would share this experience regarding these issues.

    Before I begin, in neither scenario did we lose quorum of the DAG.  Also, the symptoms of both scenarios were nearly identical. 

     

    SYMPTOMS

    • Viewing these servers from Failover Cluster Manager show them with a STATUS of DOWN.
    • Network Connections for these members are listed as UNAVAILABLE
    • Cluster Services Starts on these servers, however the following event is logged in the Event’s System Log
      Log Name:      System
      Source:        Microsoft-Windows-FailoverClustering
      Event ID:      1572
      Task Category: Cluster Virtual Adapter
      Level:         Critical
      Description:  Node 'SERVER' failed to join the cluster because it could not send and receive failure detection network messages with other cluster nodes. Please run the Validate a Configuration wizard to ensure network settings. Also verify the Windows Firewall 'Failover Clusters' rules.
    • Attempt to view Exchange DAG status or network returns error:
      A server-side administrative operation has failed. 'GetDagNetworkConfig' failed on the server. Error: The NetworkManager has not yet been initialized. Check the event logs to determine the cause. [Server: SERVER5.Contoso.inc]
          + CategoryInfo          : NotSpecified: (0:Int32) [Get-DatabaseAvailabilityGroup], DagNetworkRpcServerException
          + FullyQualifiedErrorId : A6AA817A,Microsoft.Exchange.Management.SystemConfigurationTasks.GetDatabaseAvailabilityGroup
    • Cluster Log Shows:
      WARN  [API] s_ApiOpenGroupEx: Group Cluster Group failed, status = 70
      DBG   [HM] Connection attempt to SERVER01 failed with error WSAETIMEDOUT(10060): Failed to connect to remote endpoint 1.2.3.45:~3343~.
      INFO  [JPM] Node 7: Selected partition 33910(1 2 3 4 5 6 9 10 11 12 13 14) as a target for join
      WARN  [JPM] Node 7: No connection to node(s) (10 12). Cannot join yet
    • Cluster Validation Report shows:
      Node SERVER01.Contoso.inc is reachable from Node SERVER5.Contoso.inc by only one pair of interfaces. It is possible that this network path is a single point of failure for communication within the cluster. Please verify that this single path is highly available or consider adding additional networks to the cluster.
      The following are all pings attempted from network interfaces on node SERVER5.Contoso.inc to network interfaces on node SERVER05.Contoso.inc.
    • Network Trace was showing that cluster communication was in fact going thru to all other nodes on port 3343 and responses were returned. 
    • There was no change in errors even after disabling Windows Firewall and removing file level antivirus and security products from the servers.
    • Removing NIC Teaming from the server did not work


    RESOLUTION #1
    In this scenario, this occurred within our lab running on Hyper-V.  Based on hyper-V's network summary output, I could see that the servers really were not communicating properly.  Yes, they could ping and they could authenticate with the domain, but cluster communication was failing. 
    The resolution was to consistently configure the network settings on all DAG members & to reset the hyper-v network properties.  This meant:

    • Confirm that the networks were identically configured between all DAG node members (i.e. REPL / MAPI Networks, TCP/IP settings, Binding Order, Driver versions, etc)
    • Disabled IPv6 from the servers [NOTE: It is recommended to leave IPv6 enabled, even if you do not have an IPv6-enabled network!  In most scenarios, disabling IPv6 on an Exchange 2010 should be a last option.]
    • Once rebooted, all was working fine.
    • Edit the Hyper-V Network Properties Page for this VM


    RESOLUTION #2
    In this scenario, this occurred in production.  Ultimately we decided to change the IP address of the 'broken' DAG member and reboot the server again.  This allowed the server to properly register its network connections with the cluster DB (ClusDB) and all other nodes were able to talk properly.  This allowed the DAG member to rejoin the DAG and then all databases were able to mount and/or replicate their copy successfully. 

    We found that not all of the production DAG members were identically configured with their network settings (i.e. 2 DAG members did not have a REPL network configured).  Per http://technet.microsoft.com/en-us/library/dd638104.aspx#NR, "each DAG member must have the same number of networks".  We fixed the networks and updated the servers to include the recommended hotfixes - http://blogs.technet.com/b/dblanch/archive/2012/02/27/a-few-hotfixes-to-consider.aspx


     

    Questions/Answers
    Why did changing the IP address of the DAG member work?   Well, not exactly sure but we believe that this was either a stale TCP route or something in the CLUSDB was preventing any server with that IP address from joining the cluster.
    Did you reboot all of the DAG member server before or after changing the IP address?  No, we did not want to risk losing another server within the DAG (had already lost 2 of the 12 members).  We did, however, reboot all of the servers in the lab scenario.
    Did you ever lose quorum of the DAG? Nope.
    Do you think that you could have prevented this?  Maybe, if we had applied all of the hotfixes outlined here & confirmed all network settings were identical on all DAG members, then maybe servers might not have caused this issue.   There may be other things causing this, but it is always recommended to resolve the known issues first.


    Good Luck.
    Doug

  • Managing Exchange Public Folder Permissions

    Over the years, there has been a request for finding various permissions on Public Folder objects within Exchange.  I figured that I would share how to do some of these tasks, specific to Exchange 2010 and 2013.

    NOTE: The following commands use the Exchange Management Shell

    Exchange 2010
    List All Top Level Public Folders Default Permissions
    Get-PublicFolder \ -GetChildren | Get-PublicFolderClientPermission | Where {$_.User.IsDefault -eq $True} | FT Identity, User, AccessRights -auto -wrap

    List All Top Level Public Folders Anonymous Permissions
    Get-PublicFolder \ -GetChildren | Get-PublicFolderClientPermission | ?{$_.User.IsAnonymous -eq $True} | FT Identity, User, AccessRights -auto -wrap

    List All Public Folders Where Anonymous is set to Owner
    Get-PublicFolder \ -Recurse | Get-PublicFolderClientPermission | ?{($_.User.IsAnonymous -eq $True) -AND ($_.AccessRights -eq 'Owner')} | FT Identity, User, AccessRights -auto -wrap

    List All Public Folders Where Default is NOT Author
    Get-PublicFolder \ -Recurse | Get-PublicFolderClientPermission | ?{($_.User.IsDefault -eq $True) -AND ($_.AccessRights -ne 'Author')} | FT Identity, User, AccessRights -auto -wrap

    List All Public Folders Where JoeUser is set to Owner
    Get-PublicFolder \ -Recurse | Get-PublicFolderClientPermission | ?{($_.User -like "*JoeUser*") -AND ($_.AccessRights -eq 'Owner')} | FT Identity, User, AccessRights -auto -wrap

    List All Public Folders Containing Old/Deleted Users with Permissions
    Get-PublicFolder \ -Recurse | Get-PublicFolderClientPermission | ?{$_.User -like "*NT User:*"} | FT Identity, User, AccessRights -auto -wrap

    Remove Old/Deleted Users from Public Folders (w/ WhatIf)
    Get-PublicFolder \ -Recurse | Get-PublicFolderClientPermission | ?{$_.User -like "*NT User:*"} | ForEach {Remove-PublicFolderClientPermission -Identity $_.Identity -User $_.User -AccessRights $_.AccessRights -WhatIf

     

    Modify/Add JoeUser to be an Owner of a Folder
    Add-PublicFolderClientPermission -Identity "\MyPublicFolder\Reports" -User JoeUser -AccessRights Owner

    Exchange 2013
    List All Top Level Public Folders Default Permissions
    Get-PublicFolder \ -GetChildren | Get-PublicFolderClientPermission | Where {$_.User.UserType -eq 'Default'} | FT Identity, User, AccessRights -auto -wrap

    List All Top Level Public Folders Anonymous Permissions
    Get-PublicFolder \ -GetChildren | Get-PublicFolderClientPermission | ?{$_.User.UserType -eq 'Anonymous'} | FT Identity, User, AccessRights -auto -wrap

    List All Public Folders Where Anonymous is set to Owner
    Get-PublicFolder \ -Recurse | Get-PublicFolderClientPermission | ? {($_.User.UserType -eq 'Anonymous') -AND ($_.AccessRights -eq 'Owner')} | FT Identity, User, AccessRights -auto -wrap

    List All Public Folders Where Default is NOT Author
    Get-PublicFolder \ -Recurse | Get-PublicFolderClientPermission | ?{($_.User.UserType -eq 'Default') -AND ($_.AccessRights -ne 'Author')} | FT Identity, User, AccessRights -auto -wrap

    List All Public Folders Where JoeUser is set to Owner
    Get-PublicFolder \ -Recurse | Get-PublicFolderClientPermission | ?{($_.User -like "*JoeUser*") -AND ($_.AccessRights -eq 'Owner')} | FT Identity, User, AccessRights -auto -wrap

    List All Public Folders Containing Old/Deleted Users with Permissions
    Get-PublicFolder \ -Recurse | Get-PublicFolderClientPermission | ?{$_.User.UserType -like "Unknown"} | FT Identity, User, AccessRights -auto -wrap

    Remove Old/Deleted Users from Public Folders (w/ WhatIf)
    Get-PublicFolder \ -Recurse | Get-PublicFolderClientPermission | ?{$_.User.UserType -like "Unknown"} | ForEach {Remove-PublicFolderClientPermission -Identity $_.Identity -User $_.User -AccessRights $_.AccessRights -WhatIf}


     

    Modify JoeUser to be an Owner of a Folder
    Add-PublicFolderClientPermission -Identity "\MyPublicFolder\Reports" -User JoeUser -AccessRights Owner

    More information on managing Public Folders can be found on TechNet for Exchange.

    Good Luck

    Da

  • Some Public Folder content is not replicating from Exchange 2003 to Exchange 2007

    I’ve recently experienced several different cases where replication between Exchange 2003 to 2007 was not working 100%. Much of the content had come over but some was not.  This blog talks about some of the steps that we took to isolate and resolve this.

    NOTE: Before you go any further, be sure to read the Public Folder Replication Troubleshooting blog: http://msexchangeteam.com/archive/2008/01/10/447843.aspx

    IS THE PROBLEM REAL

    Checking ESM may show you that the size and total items may be mismatched from a Exchange 2003 and 2007 server. If so, determine which folders are experiencing this and what type of content they hold. Log onto the Public Folder store using Outlook or MFCMapi to spot check that the content is in fact out-of-sync with each other.

    Know that Public Folder calendars may not be 100% equal from 1 server to another. This is because when you use OWA or any CDO application to access a calendar Public Folder, an instance of recurring items is created and stored within the Public Folder. While this instance is not visible in Outlook, it does add to the item count on the public folder, as seen in ESM. Additionally, those instances are not replicated between public folders so it is possible that a calendar PF on ServerX has many more items listed than ServerY. And they might have their content in-sync.

    TRY THE BASICS

    Once you have identified the Public Folder(s) that are not in sync, do some of the basic steps to force replication:

    NOTE: See KB 842273 How to troubleshoot public folder replication problems in Exchange 2000 Server and in Exchange Server 2003

    1. Turn up MSExchangeIS\Public Diagnostic Logging related to Replication on the source and target Exchange servers
    2. Enable Messaging Tracking on both source and target servers
    3. Within ESM, right click on the Organization object and make sure that Public Folder replication is not paused.
    4. Ensure that the Exchange 2007 Public Folder store is listed as a Public Folder replica and that the store is mounted
    5. Check the Exchange 2007 server Public Folder Administration tool (toolbox) and verify that the hierarchy has replicated to the server. Hierarchy before data!
    6. Check the configuration of the antivirus software – ensure that it does not include Public Folder messages (DoNotScanIPMReplicationMessages)
    7. Add new content to the source public folder and ensure that the new content has replicated over to the target server
    8. Use PFDavAdmin against the source server to remove item-level permissions
    9. Right click the problem folder and select Resend Changes. Send all changes over the past 1000 days or so.
    10. Review the application log on the source server to see if a replication message for that folder has left the source server.

    If so, check the target server’s application log to see if a replication message came in. If there is no inbound application log message for replication, message track that message from the source server.

    If not, search for warnings and errors related to PF Replication.

    NEXT STEPS

    At this point, you should have isolated which folders work and which do not. Also, you know if the message is leaving the source server or not and if the target server is accepting the message or not. Also, message tracking would tell you if an NDR was returned (ex: 554 5.6.0 STOREDRV.Deliver.Exception propertyValidationException) .

    An NDR may indicate that there is an incorrect property value set (or missing) on the Public Folder. To find that property, try:

    1. On the Hub Transport Server, enabled ContentConversionTracing & PiplineTracing
    2. Modify some items in the source Public Folder
    3. Search the content conversion file for Exception. Here are 3 different issues that we experienced:

    ERROR 1: Microsoft.Exchange.Data.Storage.PropertyValidationException: Property validation failed. Property = [{00020329-0000-0000-c000-000000000046}:'Keywords'] Categories 

    Error = Element 0 in the multivalue property is invalid..

    CAUSE: Content Conversion logging showed us that there was an issue with the Categories field on a Public Folder. So we opened Outlook against the source folder and viewed the content by Category (Views > Current Views). What we found was that some categories had an invalid character listed, such as a @, comma, or space. We edited those objects and removed the invalid character from the category field and then replication worked properly for those objects.

     

    ERROR 2: Microsoft.Exchange.Data.Storage.PropertyValidationException: Property validation failed. Property = [{00062004-0000-0000-c000-000000000046}:0x8092] Email2AddrType

    Error = Email2AddrType is too long: maximum length is 9, actual length is 25.

    CAUSE: Here we see that Email2AddrType is the problem. This particular folder was a Contacts folder. What we found was that some of the contacts had invalid addresses listed. We modified those addresses and replication completed properly for those objects.

     

    ERROR 3: FAIL 554 5.6.0 STOREDRV.Deliver.Exception.ObjectValidation Failed to process message due to a permanent exception

    Microsoft.Exchange.Data.Storage.ObjectValidationException: The object is invalid and cannot be saved.

    CAUSE: With this error, we were trying to replicate recurring appointments. Taking a closer look at those appointments, we realized that some did not have any end-date (EndTimeProperty) set. We edited the endtime of the meeting. In doing so, an update to the meeting was sent out to all of the attendees.

     

    WHAT ELSE

    If ContentConversion or Pipeline Tracing do not help, you could use the tracing features from the Exchange Troubleshooting Assistant (EXTRA) to potentially isolate the problem.

    1. Start > Run > EXTRA
    2. Select Trace Control from the Tasks
    3. Select Manual Trace Tags
    4. Select all types & select the following tags for the STORE component: tagPFReplHier, tagPFReplInbound, tagPFReplInit, TagAccessDenied, tagAccessDeniedDetails, tagDeliverMail, tagDispatchRpcCalls, tagDispatchRpcReturns, tagError, tagPFDLocalDelivery, & tagRpcIntError
    5. Replicate PF content again and then stop tracing
    6. Review the trace logs

    This should get you closer to the problem…

    Doug

  • Tracking Down Exchange 2007 Database Bloat

    I recently dealt with an issue of an Exchange 2007 database being physically larger than what was expected. So we took a few actions to find out more about the cause of the bloat. This outlines some of the work we did to isolate that bloat.

    GATHER DATA

    We started by getting more information about the database
    1. Check the most recent 1221 event in Application Log to ensure that Online Maintenance has completed
    2. From the Exchange 2007 Management Shell, run Get-mailboxstatistics
    3. PFDavAmin Item Content report
    4. Run ESEUTIL /MS against that database (ex: Eseutil /ms DBName.edb >C:\MSOutput.txt)
    5. Run ISINTEG –DUMP against that database

    DETERMINE HOW MUCH BLOAT

    Event 1221 showed us how much whitespace the DB reclaimed during online maintenance:
       Event ID : 1221
       Category : General
       Source : MSExchangeIS Mailbox Store
       Type : Information
       Message : The database "MyStorageGroup\MBXDB" has 5178 megabytes of free space after online defragmentation has terminated.
    Added up the Deleted Item Size & Item Size from the Get-mailboxstatistics output. This is rough version of how much “user data” that the database has.
    Noted the physical size of the database (ex: 50GB). Determine how much bloat may exist by adding the event 1221 whitespace (ex: 5GB) and the user data (24GB). In our example, we have a total of 29GB accounted for but 21GB unaccounted.

    NOTICE: Before you dig into the /MS output, you should read through the ESE Database Structure technet article. At a minimum, understand that pages in Exchange 2007 are divided into 8-KB pages, where Exchange 2000 and 2003 (ESE98) use 4KB pages.

    Do not expect to have a DB that is physically the same size as your whitespace and user data. There are many reasons why the database may require additional space. These might include database structure such as indexes, tables, and search folders as well as fragmented pages and unclaimed whitespace (i.e. changes since expiry and online maintenance).

    SAMPLE /MS OUTPUT

        ********************************** SPACE DUMP *************************************
        Name                  Type     ObjidFDP    PgnoFDP     PriExt     Owned      Available
        =====================================================================================
       Dbname.edb             Db          1           1        256-m      3187862      64000
       1-121                  Tbl        112        426          8-s         8             0
       ?B6708?T668f           Idx       1848        431          1-s         1             0
       MsgFolderIndex7        Idx        113        427          1-s         1             0
       MsgFolderIndexPtagDel  Idx        116        430          1-s         1             0
       MsgFolderIndexURLComp  Idx        115        429          1-s         1             0
       RuleMsgFolderIndex     Idx        114        428          1-s         1             0
       1-24                   Tbl         61        142          2-m       695104          3
       1-611BB71A86           Tbl        312        833          8-m        3014           5
       ?B6708?T668f+B67aa+S1  Idx       1850        476          1-s         1             0
       MsgFolderIndex7        Idx        313        834          1-s         1             0
       MsgFolderIndexPtagDel  Idx        316        837          1-s         1             0
       MsgFolderIndexURLComp  Idx        315        836          1-s         1             0
       RuleMsgFolderIndex     Idx        314        835          1-s         1             0
       S-1-28B913B0D4F        Tbl       1862        705          8-s         8             3
       MsgFolderIndexURLComp  Idx       1863        706          1-s         1             0
       ptagFIDIndex           Idx       1865        708          1-s         1             0
       ptagSearchedFIDIndex   Idx       1864        707          1-s         1             0
       - continued -
    ---------------------------------------------------------------------------------------------------------------------------
                                                                                      647540

    MS Output Field information

    • FDP is a special page in the database which indicates which B+tree this page belongs to. ObjidFDP is the Object ID of the FDP
    • PgnoFDP is the page number of the FDP
    • PriExt is the combination of a number and letter. The number before the dash is the initial number of pages when the object was first created in the B-Tree. The letter after the dash indicates whether the space for the B-Tree is currently represented using multiple pages ("m") or a single page ("s").
    • Owned number of pages that contain data and/or are in use
    • Available the number of free pages available
    • Type may include Table (TBL), Index (IDX), and Long-value (LV)
    • LV may be required because a column or a record in ESE cannot span pages in the data B+tree. There are values that break the 8KB boundary of a page; referred to as long-values (LV). A table's long-value B+tree is used to store these large values.

    READING THE /MS OUTPUT

    We decided to look at 4 things within the /MS output:

    Calculate Actual Whitespace: The number at the end of the dump (647540 in the above example) is the summation of the total number of pages that are available throughout all the tables. Take that number and multiply that by the page size value (8KB for Exchange 2007). In our example, we have 5,180MB of Whitespace.

    Attachment Table: Table 1-24 holds all attachments in the database. In our example, we have 695104 Owned Pages for this attachment table. We multiply that number with the page size (8KB) and the total is 5.5 GB of space is for attachments.

    Search Folders: Search folders are listed by the S- value. In the example above, S-1-28B913B0D4F is a search folder. Look for a many S- values in the output and follow DGoldmans blog to identify anyone users has a large number of Search Folders.

    Large Consumption Users: Look through the output and see if there is any object that has a large number of owned pages. In our example, we see that 1-611BB71A86 has 3014 pages.

    NOTE: All user mailbox folder tables are numbered, not named. In the example above, 1-611BB71A86 is a mailbox folder table. But also look at other tables, such as MailboxTombstone or Message Tombstone.

    If you find a numbered table that has a large number of owned pages, you can identify which mailbox that table belongs to by looking at the ISINTEG –DUMP output.

    To do this, copy the numbered value after the dash (611BB71A86) and then search the ISINTEG output file from the bottom up for that value.

    Example:

      [6] RootFID=0001-00611BB71A86
      Owner DN=???
      GUID=D12C30EC 4938E64D 89999899 906A78DA
      Display Name=Mailbox - John Doe
      Comment=
      Sentmail FID=0000-0000309F1E78
      Subtree=0001-00611BB71A87
      Inbox=0001-00611BB71A88
      Outbox=0001-00611BB71A89
      Sentmail=0001-00611BB71A8A
      Finder=0001-00611BB71A8C
      DAF=0001-00611BB71A8D
      Spooler Q=0001-00611BB71A8E
      Size=(ec:ecNotFound-MAPI_E_NOT_FOUND)
      Localized=TRUE
      Locale=0x409
      In some cases, the search results yield something like this:
      Folder FID=0001-00611BB71A86
      Parent FID=0001-00611BB71A92
      Root FID=0001-00611BB71A92
      Folder Type=1
      Msg Count=0
      Msgs Unread=0
      Msgs Submitted=0
      Rcv Count=1
      Subfolders=0
      Name=Shortcuts

    If your results do not show a mailbox name, then this folder may be a subfolder. You can then search the ISINTEG output for the Parent FID value (Ex: 00611BB71A92). You may have to do this several times until you locate the root mailbox name.

    BACK TO OUR ISSUE

    So what we found in our issue was that we had a very large number of Search Folders present in the /MS output. We decided to configure the RESET VIEWS registry key for that database, allow online maintenance to complete for several more times until more whitespace became available. We then perform an offline defrag of the database. This freed up some of the DB bloat.

    NOTE: If the database is continuing to grow in size, you may want to capture the data on a regular basis and see if there are any patterns for the growth (i.e. types of data or specific users). Then try to isolate why that bloat may be occurring.

     

    REFERENCES

     

    Doug

  • Using Sharepoint with your Outlook Client

    I find that many people have deployed Sharepoint but have not fully integrated it with their Outlook client. My advice would be to start integrating the various technologies to make the end-user productivity improve. Here are some links that might help educate on this topic.

    GENERAL
    These sites offer some general information about connecting Outlook to Sharepoint.

    CALENDAR

    Within your Outlook Calendar, you can view and update a SharePoint calendar, set it side-by-side, view an overlay of the calendars, and even copy events between the calendars.

    CONTACTS

    You can also add and remove Sharepoint contacts to your Outlook.

    TASKS

    If you spend a lot of time working with e-mail, you may find it easier to work with a tasks list from a SharePoint site directly in Office Outlook 2007, instead of switching to your site in a Web browser. Within Outlook, you can track, update or categorize a task and even drag or copy tasks back and forth between the folders for Outlook and the SharePoint site.

    SHAREPOINT FILES

    Once you connect your SharePoint library to Outlook, you can browse for and view the file just as you would an e-mail message, without leaving Outlook. Working with files from your SharePoint site in Outlook is best for browsing through and editing routine files that you store in a document library, such as documents, spreadsheets, and presentations. More complex data operations, such as working with custom lists or updating database applications, are better handled directly on the SharePoint site. You can also take the files offline to work on them. When you get back to the office, you can update the versions on your SharePoint site. Your changes to a file are not updated automatically on the server while you edit and save the file. This enables you to work more quickly with offline files, because your computer does not need to connect to the server while you are working. When you close a file, you are prompted to update the changes on the server. Only the changes that you made to specific files are updated, which means that Outlook does not have to synchronize the whole library again with the server.

    TROUBLESHOOTING

    If you find that your interaction with Outlook and Sharepoint is not working as you would expect it, you could try enabling logging on the Outlook client to see if you can identify an error. To enable logging in Outlook 2007, edit the following registry value:

    HKEYCurrentUser\Software\Microsoft\Office\12.0\Outlook\Options\Mail

    RegDWORD: EnableWSSSyncLogging – Set to a value of 1 to enable (0 to disable)

    Once enabled, restart Outlook and try the interaction activity that you are trying to complete between Sharepoint & Outlook. Then look for wss-sync-log.htm logs on the client (%temp%).

    As for understanding the logs, that might be another blog topic.

    Well, that pretty much sums up the basics for Outlook integration with Sharepoint.

    Doug