• Exchange 2013 – Performance counters and their thresholds

     

    Hi all,

     

    the official TechNet most important performance counters list to monitor for Exchange 2013 is now available !

     

    The good news is that since Exchange 2013 is now a single box (-ish as separable CAS role is merely just a protocol forwarder), the counters to monitor are now simplified down to 71 counters:

    Counter category Number of counters in the category
    Client Access Server Counters 12
    Database Counters 9
    HTTP Proxy Counters 7
    RPC Client Access Counters 6
    ASP.NET 6
    Processor and Process Counters 5
    Netlogon Counters 5
    Information Store Counters 5
    Exchange Domain Controller Connectivity Counters 4
    Network counters 4
    Workload Management Counters 3
    .NET Framework Counters 3
    Memory counters 2
    Grand Total 71

     

    Remember the process to analyze an Exchange performance concern (once it’s determined that the latency is server-side and not client side or network related):

    1- Check the RPC requests as well as the RPC latency counters

    and then:

    2- Determine which is likely the cause by checking

    • 2.1. the Database counters to see if some databases have read or write latency => then check the underlying disk latency counters to see if something with the disk subsystem is wrong (can be the queue depth, or simply not enough I/Os that a LUN can provide)
    • 2.2. the CPU and Memory counters to see if there are too much pressure on these two basic counters
    • 2.3. The RPC Operations/sec, active user count and active connection count counters to see if there is an unusual load that’s putting the server’s resources down (usually CPU and Memory suffer first if it’s the case – Database latency and disk also can be the cause if the sizing hasn’t been done properly)
    • 2.4. if none of the above 2.x counters shows unusual values, also check your Antivirus counters, which can also block the RPC processing sometimes (seen on a customer’s site: RPC requests growing and hardly decreasing, RPC latency quite high, but all other counters like CPU, Database latency, etc… were below the error thresholds…)

     

    Check out all the counters and their description below:

    Exchange 2013 Performance Counters

    https://technet.microsoft.com/en-us/library/dn904093(v=exchg.150).aspx

     

     

    For a summarized view, below is a sub-list of the above Technet Exchange counters to show only those which have thresholds (29 counters have thresholds out of the 71 TechNet counters) – the TechNet article above has the entire list and descriptions. You’ll see below a picture with colors for readability between categories (click on the small image to open the original one in a new window), the other table is a simple table to enable you to copy paste these to make it a bit easier to integrate the counters in SCOM or Perfmon custom alerts for example…

     

    E2013 Perf Counters Thresholds

     

     

     

    Type Counter full path
    (all instances)
    AVG MIN MAX
    Exchange Domain Controller Connectivity Counters MSExchange ADAccess Domain Controllers(*)\LDAP Read Time <= 50ms   <= 100ms
    Exchange Domain Controller Connectivity Counters MSExchange ADAccess Domain Controllers(*)\LDAP Search Time <= 50ms   <= 100ms
    Exchange Domain Controller Connectivity Counters MSExchange ADAccess Processes(*)\LDAP Read Time <= 50ms   <= 100ms
    Exchange Domain Controller Connectivity Counters MSExchange ADAccess Processes(*)\LDAP Search Time <= 50ms   <= 100ms
    Processor and Process Counters Processor(_Total)\% Processor Time <= 75%    
    Processor and Process Counters Processor(_Total)\% User Time <= 75%    
    Processor and Process Counters Processor(_Total)\% Privileged Time <= 75%    
    Processor and Process Counters System\Processor Queue Length (all instances)     <= 5 x Nb procs
    Memory counters Memory\Available Mbytes   >= 5% total RAM  
    Memory counters Memory\% Committed Bytes In Use     <= 80%
    .NET Framework Counters .NET CLR Memory(*)\% Time in GC <= 10%    
    .NET Framework Counters .NET CLR Exceptions(*)\# of Excepts Thrown / sec <= Web Service(_Total)\Connection attempts/sec x 0.5    
    Network counters Network Interface(*)\Packets Outbound Errors     = 0
    Network counters TCPv4\Connections Reset should never increase should never increase should never increase
    Network counters TCPv6\Connections Reset should never increase should never increase should never increase
    Database Counters MSExchange Database ==> Instances(*)\I/O Database Reads (Attached) Average Latency <= 20ms    
    Database Counters MSExchange Database ==> Instances(*)\I/O Database Writes (Attached) Average Latency <= 50ms    
    Database Counters MSExchange Database ==> Instances(*)\I/O Log Writes Average Latency <= 10ms    
    Database Counters MSExchange Database ==> Instances(*)\I/O Database Reads (Recovery) Average Latency <= 200ms    
    Database Counters MSExchange Database ==> Instances(*)\I/O Database Writes (Recovery) Average Latency <= 200ms    
    ASP.NET ASP.NET\Application Restarts     = 0
    ASP.NET ASP.NET\Worker Process Restarts     = 0
    ASP.NET ASP.NET\Request Wait Time     = 0
    ASP.NET ASP.NET Applications(*)\Requests In Application Queue     = 0
    RPC Client Access Counters MSExchange RpcClientAccess\RPC Averaged Latency     <= 250ms
    RPC Client Access Counters MSExchange RpcClientAccess\RPC Requests     <= 40
    Information Store Counters MSExchangeIS Client Type\RPC Requests     <= 70
    Information Store Counters MSExchangeIS Client Type(*)\RPC Average Latency <= 50ms    
    Information Store Counters MSExchangeIS Store(*)\RPC Average Latency <= 50ms   <= 100ms
  • Exchange 2010 – Powershell script to dump InternalURLs and ExternalURLs of basic Exchange services

     

     

    Hey all,

     

    Here is a script to dump the Internal and External URL properties from the main Exchange services.

    This script has been designed and tested to work out of the box on an Exchange 2010 environment, but it may work on Exchange 2007 and Exchange 2013 as well.

    Why these properties ? Because on many engagements, I found that these URLs were not properly set, leading to users or servers latencies, performance issues, proxy or redirection not working between Active Directory sites, …

     

    Look at the URL configuration tables on the below link, which are Microsoft’s recommendations to set correctly your URLs whether you have a Load Balancer or not:

    Understanding Proxying and Redirection

    https://technet.microsoft.com/en-us/library/bb310763(v=exchg.141).aspx

     

    Virtual directory /service

    InternalURL ExternalURL (Internet-facing Active Directory site) ExternalURL (non-Internet-facing Active Directory site)
    /OWA NLB FQDN NLB FQDN $null
    /ECP NLB FQDN NLB FQDN $null
    /Microsoft-Server-ActiveSync NLB FQDN NLB FQDN $null
    /OAB NLB FQDN NLB FQDN $null
    /EWS NLB FQDN NLB FQDN $null
    POP/IMAP (InternalConnectionsSettings)
    NLB FQDN
    Not applicable Not applicable

     

    Don’t forget to double check your certificates as well, they should contain all the above used URLs.

     

    The below script will dump the Internal and External URLs for the above services to that you’ll be able to check if your environment has been configured correctly (trust me, it’s worth to triple-check because for lots of my customers we saw surprises, that explained some user or server performance issues we had at that time).

     

    This type of script is pretty common among administrators, pretty straightforward, anyways I tried to comment the script so that it’s understandable by anyone, but leave me comments and suggestions if you don’t understand something…

     

    You can either copy-paste the below lines (but the formatting will be a bit weird), or download the script from the following link.

    Download the script

    https://gallery.technet.microsoft.com/Powershell-script-to-ccde9d5f

     

     

    $ErrorActionPreference='SilentlyContinue'

    #Loading Exchange 2010 snapins enabling script to be executed on a basic Powershell session
    #Note: you must have Exchange Admin tools installed on the machine where you run this. 

    Add-PSSnapin microsoft.exchange.management.powershell.admin -erroraction 'SilentlyContinue' | OUT-NULL      

    Add-PSsnapin Microsoft.Exchange.Management.PowerShell.E2010 -erroraction 'SilentlyContinue' | OUT-NULL      

    Add-PSsnapin Microsoft.Exchange.Management.PowerShell.Setup -erroraction 'SilentlyContinue'  | OUT-NULL      

    Add-PSsnapin Microsoft.Exchange.Management.PowerShell.Support -erroraction 'SilentlyContinue'  | OUT-NULL      

    #For Exchange 2007 and 2013, add the corresponding modules/snapins, or simply execute the script into an Exchange MAnagement Shell :-)      

     

    #Saving script path to use the same path to store the output file

    $ScriptPath = Split-Path -Parent -Path $MyInvocation.MyCommand.Definition      

     

    #Getting all Exchange servers in an array
    #Note: you can target only one server, or get servers list from a file,

    #just change the $Servers = @(Get-ClientAccessServer) line with $Servers = @(Get-content ServersList.txt) for example to get servers from a list...    

    $Servers = @(Get-ClientAccessServer)      

    #Initializing counters to setup a progress bar based on the number of servers browsed      

    # (more useful in an environment where you have dozen of servers - had 45 in mine)      

     

    $Counter=0

    $Total=$Servers.count      

     

    #Initializing the variable where I'll put all the results of my object browsing

    $report = @()      

     

    #For each server discovered in the "$Servers = Get-ClientAccessServer" line,
    # grab the Virtal Directories properties and store it in a custom Powershell object,
    # and then add this object in the $report array variable to eventually dump the whole result in a text (CSV) file.

    foreach( $Server in $Servers)      

    {

    #$Computername=$Server.Name   <- not needed for now       

    #This is to print the progress bar incrementing on each server (increment is later in the script $Counter++ it is...      

    $Pct=($Counter/$Total)*100           

    Write-Progress -Activity "Processing Server $Server" -status "Server $Counter of $Total" -percentcomplete $pct                       

    #For the current server, get the main vDir settings (including AutodiscoverServiceInternalURI which is important to determine       

    #whether the Autodiscover service will be hit using the Load Balancer (recommended).      

    $EAS = Get-ActiveSyncVirtualDirectory -Server $Server| Select Name, InternalURL,externalURL      

    $OAB = Get-OabVirtualDirectory -Server $Server| Select Name,internalURL,externalURL      

    $OWA = Get-OwaVirtualDirectory -Server $Server| Select Name,InternalURL,externalURL      

    $ECP = Get-EcpVirtualDirectory -Server $Server| Select Name,InternalURL,externalURL      

    $AutoDisc = get-ClientAccessServer $Server | Select name,identity,AutodiscoverServiceInternalUri      

    $EWS = Get-WebServicesVirtualDirectory -Server $Server| Select NAme,identity,externalURL      

    #If you want to dump more things, use the below line as a sample:      

    #$ServiceToDump = Get-Whatever -Server $Server | Select Property1, property2, ....  <- don't need the "Select property", you can omit this, it will just get all attributes...       

    #the below is a template if you need to dump more things into the final report      

    #just replace the "ServiceToDump" string with the service you with to dump - don't forget to     

    #Get something above like the $Service = Get-whatever -Server      

    #$Obj | Add-Member -MemberType NoteProperty -Name "ServiceToDump-vDirNAme" -Value $ServiceToDump.Name      

    #$Obj | Add-Member -MemberType NoteProperty -Name "ServiceToDump-InternalURL" -Value $ServiceToDump.InternalURL      

    #$Obj | Add-Member -MemberType NoteProperty -Name "ServiceToDump-ExernalURL" -Value $ServiceToDump.ExternalURL          

    #Initializing a new Powershell object to store our discovered properties      

    $Obj = New-Object PSObject      

    #the below is a template if you need to dump more things into the final report 

    #just replace the "ServiceToDump" string with the service you with to dump - don't forget to       

    #Get something above like the $Service = Get-whatever -Server      

    #$Obj | Add-Member -MemberType NoteProperty -Name "ServiceToDump-vDirNAme" -Value $ServiceToDump.Name      

    #$Obj | Add-Member -MemberType NoteProperty -Name "ServiceToDump-InternalURL" -Value $ServiceToDump.InternalURL      

    #$Obj | Add-Member -MemberType NoteProperty -Name "ServiceToDump-ExernalURL" -Value $ServiceToDump.ExternalURL          

    $Obj | Add-Member -MemberType NoteProperty -Name "ServerName" -Value $Server.Name      

    $Obj | Add-Member -MemberType NoteProperty -Name "EAS-vDirNAme" -Value $EAS.Name      

    $Obj | Add-Member -MemberType NoteProperty -Name "EAS-InternalURL" -Value $EAS.InternalURL      

    $Obj | Add-Member -MemberType NoteProperty -Name "EAS-ExternalURL" -Value $EAS.ExternalURL      

    $Obj | Add-Member -MemberType NoteProperty -Name "OAB-vDirNAme" -Value $OAB.Name      

    $Obj | Add-Member -MemberType NoteProperty -Name "OAB-InternalURL" -Value $OAB.InternalURL      

    $Obj | Add-Member -MemberType NoteProperty -Name "OAB-ExernalURL" -Value $OAB.ExternalURL      

    $Obj | Add-Member -MemberType NoteProperty -Name "OWA-vDirNAme" -Value $OWA.Name      

    $Obj | Add-Member -MemberType NoteProperty -Name "OWA-InternalURL" -Value $OWA.InternalURL      

    $Obj | Add-Member -MemberType NoteProperty -Name "OWA-ExernalURL" -Value $OWA.ExternalURL      

    $Obj | Add-Member -MemberType NoteProperty -Name "ECP-vDirNAme" -Value $ECP.Name      

    $Obj | Add-Member -MemberType NoteProperty -Name "ECP-InternalURL" -Value $ECP.InternalURL      

    $Obj | Add-Member -MemberType NoteProperty -Name "ECP-ExernalURL" -Value $ECP.ExternalURL         

    $Obj | Add-Member -MemberType NoteProperty -Name "AutoDisc-vDirNAme" -Value $AutoDisc.Name      

    $Obj | Add-Member -MemberType NoteProperty -Name "AutoDisc-URI" -Value $AutoDisc.AutodiscoverServiceInternalURI

    $Obj | Add-Member -MemberType NoteProperty -Name "EWS-vDirNAme" -Value $EWS.Name      

    $Obj | Add-Member -MemberType NoteProperty -Name "EWS-InternalURL" -Value $EWS.InternalURL      

    $Obj | Add-Member -MemberType NoteProperty -Name "EWS-ExernalURL" -Value $EWS.ExternalURL         

    #Appending the current object into the $report variable (it's an array, remember)      

    $report += $Obj      

    #Incrementing the Counter for the progress bar      

    $Counter++      

    }

    #Building the file name string using date, time, seconds ...      

    $DateAppend = Get-Date -Format "ddd-dd-MM-yyyy-\T\i\m\e-HH-mm-ss"      

    $CSVFilename=$ScriptPath+"\ExchvDirsInformation"+$DateAppend+".csv"      

    #Exporting the final result into the output file (see just above for the file string building...      

    $report | Export-csv -notypeinformation -encoding Unicode $CSVFilename    

     

    Download the script

    https://gallery.technet.microsoft.com/Powershell-script-to-ccde9d5f

  • Awareness update: Exchange 2010 SP3 RU8 – RU8 V2 is out !

     

    Exchange 2010 SP3 RollUp 8 Version 2 (V2) is now released ! Link below. 

     

    Update on 12th December from the Exchange Team Blog:

    Exchange Server 2010 SP3 Update Rollup 8 has been re-released to the Microsoft download center resolving a regression discovered in the initial release. The update RU8 package corrects the issue which impacted users connecting to Exchange from Outlook. The issue was insulated to the MAPI RPC layer and was able to be isolated to quickly deliver the updated RU8 package. The updated RU8 package is version number 14.03.0224.002 if you need to confirm you have the updated package. The updates for Exchange Server 2013 and 2007 were not impacted by this regression and have not been updated.

     

     

    Update Rollup 8 v2 For Exchange 2010 SP3 (KB2986475)

    http://www.microsoft.com/en-us/download/details.aspx?id=45225

     

    This update still include the security patch for Exchange MS14-075 (https://technet.microsoft.com/library/security/ms14-075)

     

    Other fixes included on this RollUp (also listed on the SP3 RU8’s page):

     

    • 3004235

      (http://support.microsoft.com/kb/3004235/ )

      Exchange Server meetings in Russian time zones as well as names of time zones are incorrect after October 26, 2014

    • 3009132

      (http://support.microsoft.com/kb/3009132/ )

      Hybrid mailbox moves to on-premises environment but finishes with CompletedWithWarnings status

    • 3008999

      (http://support.microsoft.com/kb/3008999/ )

      IRM restrictions are applied to incorrectly formatted .docx, .pptx, or .xlsx files in an Exchange Server 2010 environment

    • 3008370

      (http://support.microsoft.com/kb/3008370/ )

      Group members are not sorted by display name when HAB is used with OAB in Exchange Server 2010

    • 3008308

      (http://support.microsoft.com/kb/3008308/ )

      Public folder database migration issue in a mixed Exchange Server environment

    • 3007794

      (http://support.microsoft.com/kb/3007794/ )

      Hub Transport server cannot deliver messages when a database fails over to a cross-site DAG in Exchange Server 2010

    • 3004521

      (http://support.microsoft.com/kb/3004521/ )

      An Exchange server loses its connection to domain controllers if a public folder server is down in Exchange Server 2010

    • 2999016

      (http://support.microsoft.com/kb/2999016/ )

      Unreadable characters when you import ANSI .pst files of Russian language by using the New-MailboxImportRequest cmdlet

    • 2995148

      (http://support.microsoft.com/kb/2995148/ )

      Changing distribution group takes a long time in an Exchange Server 2010 environment

    • 2992692

      (http://support.microsoft.com/kb/2992692/ )

      Retention policy is not applied to Information Rights Management protected voice mail messages in Exchange Server 2010

    • 2987982

      (http://support.microsoft.com/kb/2987982/ )

      Issues caused by ANSI mode in Exchange Server 2010

    • 2987104

      (http://support.microsoft.com/kb/2987104/ )

      Email message is sent by using the "Send As" instead of "Send on Behalf" permission in Exchange Server 2010

    • 2982017

      (http://support.microsoft.com/kb/2982017/ )

      Incorrect voice mail message duration in Exchange Server 2013 and Exchange Server 2010

    • 2977279

      (http://support.microsoft.com/kb/2977279/ )

      You cannot disable journaling for protected voice mail in Exchange Server 2013 and Exchange Server 2010

     

     

    Sam.

  • Useful for your Lab and your production servers remote management - Remote Desktop Manager v2.7 is out !

     

    YaY, public version big jumping from v2.2 to v2.7 for many cool features and fixes, thanks Julian Burger, Microsoft Application Developper !

    Quote from the site:

    RDCMan manages multiple remote desktop connections. It is useful for managing server labs or large server farms where you need regular access to each machine such as automated checkin systems and data centers. It is similar to the built-in MMC Remote Desktops snap-in, but more flexible.

    The RDCMan 2.7 version is a major feature release.

    New features include:

    • Virtual machine connect-to-console support
    • Smart groups
    • Support for credential encryption with certificates
    • Windows 8 remote action support
    • Support for Windows 8, Windows 8.1 / Windows Server 2012, Windows Server 2012 R2

    Here is the download link:

    http://www.microsoft.com/en-us/download/details.aspx?id=44989

    And here are some more good information from my friend Rhoderick Milne:

    http://blogs.technet.com/b/rmilne/archive/2014/11/19/remote-desktop-connection-manager-download-rdcman-2-7.aspx

     

    Cheers,

    Sam.

  • Windows related Schannel vulnerability - since we are using HTTPS in many situations on Exchange Servers, I’m putting the information here as well.

     

     

    The security update MS014-066 was released last Tuesday with the normal patch Tuesday releases and very little mention was made of it.

      

    It addresses a vulnerability in Microsoft’s implementation of Secure Channel (SChannel) that is used to establish SSL connections.  If this vulnerability is exploited an attacker could run code remotely on servers and/or workstations that use the current SChannel algorithms. It can affect servers and clients as well, and the impact is the ability for remote code execution rather than just communication interception.

    The good news is that the patch is available.  We highly recommend testing the patch and deploy it once tested as soon as possible.

    So again, the patch is available and should be implemented as soon as possible.  The bad guys are already hard at work trying to reverse-engineer the patch (just as a comparison point, they only took 12 hours to figure-out and exploit Heartbleed)…

    More information is available here:

    https://technet.microsoft.com/library/security/MS14-066

    http://arstechnica.com/security/2014/11/potentially-catastrophic-bug-bites-all-versions-of-windows-patch-now/

     

    Pasting the FAQ from the TechNet link above for quick reference:

    What might an attacker use the vulnerability to do?
    An attacker who successfully exploited this vulnerability could run arbitrary code on a target server.

    How could an attacker exploit the vulnerability?
    An attacker could attempt to exploit this vulnerability by sending specially crafted packets to a Windows server.

    What systems are primarily at risk from the vulnerability?
    Server and workstation systems that are running an affected version of Schannel are primarily at risk.

     

    Many thanks to Mike MacGillivray for putting together all the above information !