Welcome to TechNet Blogs Sign in | Join | Help

Windows Desktop Search and the implications on WAN performance

Windows Desktop search (WDS) is a great tool to help you to search through the unwieldy plethora of documents or emails that you may have scattered across your desktop. With the addition of 3rd party IFilter add-ins, it makes it even easier to find what you are looking for.

As of version 3.01, Desktop search has disabled the indexing of online mailbox on a default installation due to performance implications on the Exchange server side. Companies sometimes have the need to still run Outlook in Online mode due to security requirements of not having local OST’s, or they need real-time email for business purposes. With some of those requirements, companies have the need to also have fast message/document retrieval which Windows Desktop Search can surely do without a problem.

WDS does have some group policy settings that will now allow online indexing of mailboxes and a listing of all the settings for WDS 4 is at http://technet.microsoft.com/en-us/library/cc732491.aspx. This setting that allows indexing of online mode Outlook profiles is "Enable Indexing uncached Exchange Folders". Once this is deployed via group policy, WDS will now start indexing online mode Outlook profiles. This of course could put a huge strain on the server as all of the users data is being indexed if deployed to a large user base. Recommended guidance states that you should deploy this policy to smaller subsets of users to prevent possible server performance problems. This is similar guidance to what Microsoft recommends for cached mode deployments.

With that said, I would like to now take us down a road where certain combinations of WDS policies can not only affect Exchange server side performance, but can also have serious implications on WAN performance. If you currently have a centralized Exchange deployment and users are accessing all of their email across WAN circuits, read on.

Let’s say you have an administrative assistant running in cached mode that needs to gain access to another users complete mailbox with a requirement that data in that mailbox is easily discoverable. This requirement can be easily met by using Windows Desktop Search and is very common in law firms. A default Outlook 2007 installation will have the "Download shared folders (exclude mail folders)" option selected for their user profile, so if this assistant had previously opened a another users non-email folder such as Contacts, Calendar and Tasks, WDS will index those items without any issue. This feature unfortunately does not meet the complete requirement as we need to index all items in the other users mailbox. After Full mailbox permissions is added for this assistant, they can now add this other users mailbox to their profile to view their data.  Once you do this, you will now see that WDS will still not find any email items unless you have selected the folder in the mailbox and then performing the search. Everything so far is the default behavior.

WDS has a feature which will allow you to index online delegate mailboxes and is deployed via the GPO setting "Enable Indexing of online delegate Mailboxes".  Once this setting is deployed, any user that has another users mail related folder in their profile will now get indexed. So that seems like a good thing, no? Well, we all know that indexing any type of mailbox in online mode will increase overall performance on the Exchange server and if users are doing this over a WAN, you will now see increased WAN utilization while WDS is indexing this data making direct RPC calls to the Exchange server. If this setting was deployed to a large user base while there are a number of profiles that have other mailboxes added to their profile, you could potentially saturate this network circuit. Your network administrator at this point would obviously not be too happy and your users would then start complaining that email access is really slow or Outlook may get disconnected due to this saturation. Our best practices dictate that this setting should be deployed to smaller user bases at a time to prevent increased client traffic to the Exchange Server.

Imagine deploying this policy to 1000 users all accessing Exchange across a WAN and all have an added mailbox to their profile. By default, WDS will only index 120 items per minute which should help keep the Indexing traffic under control. If all users workstations were indexing this amount of data at a time, we would be seeing about 120,000 items per minute of traffic.  Couple that with any attachments that WDS is configured to index for such as PDF or Word documents, and this will make for a very bad network day.

There are ways to change the amount of items that are indexed per minute by modifying the GPO setting "Enable Throttling Online Mailboxes". Setting this policy to a lower value will help reduce the amount of items that are indexed per minute per mailbox and should also help keep some of the network traffic down to a minimum. The caveat here is that it will take longer to index these mailboxes. Keep in mind that is still going to be direct RPC traffic to the Exchange server with minimal amount of throttling.

To help reduce some of this overhead, Outlook 2007 has a registry entry ( CacheOthersMail ) that will allow you to cache other users mail folders in an OST file. This was first introduced in KB955572 and requires that you disable the downloading of headers. This was then rolled up in to the Outlook 2007 post SP1 Sept. 24, 2008 hotfix package (957909) . If the indexing of delegate mailboxes policy has been deployed to these users and you add this Outlook registry key, you will now see a mixture of traffic being generated by WDS. One is direct RPC traffic to the Exchange server and the other is Outlook FxGetBuffer function calls or otherwise known as Outlook sync (ICS). The Outlook sync traffic will become more prevalent over time as the other users mailbox is cached locally in the OST file. FxGetBuffer calls are a lot less expensive than direct RPC calls to the Exchange server, so deploying the CacheOthersMail registry key may help with overall WAN utilization during initial indexing. You still need to plan on increased WAN traffic as synch traffic coming from many clients could also cause potential WAN degradation issues.

WDS Registry Reference

Registry data to index data in your mailbox if you have an Online mode profile
Key: HKLM\software\policies\Microsoft\windows\windows search
DWORD: PreventIndexingUncachedExchangeFolders
Value: 0

Registry data to index shared mailboxes:
Key: HKLM\software\policies\Microsoft\windows\windows search
DWORD: EnableIndexingDelegateMailboxes
Value: 1

Registry data to change the amount of mail items that are indexed per minute.
Key: HKLM\software\policies\Microsoft\windows\windows search
DWORD: EnableThrottlingOnlineMailboxes
Value: 120
Accepted Values (Default: 120, Min: 6, Max:600)

Outlook Registry Reference

Registry data to Cache others users mail data in an OST
One-off users
Key: HKCU\Software\Microsoft\Office\12.0\Outlook\Cached Mode
DWORD: CacheOthersMail
Value: 1

GPO deployed
Key: HKCU\Software\Policies\Microsoft\Office\12.0\Outlook\Cached Mode
DWORD: CacheOthersMail
Value: 1

One of the most taxing combinations of WDS settings with relationship to Exchange Server and WAN performance is deploying "Enable Indexing uncached Exchange Folders" and "Enable Indexing of online delegate Mailboxes" simultaneously. If you also index attachments which is the default behavior, this could put increased burden on network resources and could cause considerable downtime for your users. Deploying these settings needs to be carefully planned out especially in centralized Exchange installations to prevent the situations that I describe.

Outlook 2007 Performance Improvements Hotfix

If you haven't heard already, we have released a Pre-SP2 hotfix that help improve Outlook performance and responsiveness in a big way. Here is an excerpt from the article.

Performance improvements

Performance and responsiveness are key concerns for all our customers. That is why we made the large performance tuning and optimization changes that are included in Office suite Service Pack 2 (SP2).

Outlook 2007 SP2 delivers performance improvements in four major areas:

  • General Responsiveness
    SP2 reduces I/O disk usage and UI response time.
  • Startup
    SP2 removes long operations from initial startup.
  • Shutdown
    SP2 makes Outlook exit predictably despite pending activities.
  • Folder/View Switch
    SP2 improves view rendering and folder switching.

Before you go out applying this on your machine, you need to be warned of the first startup experience as we rebuild the tables and indexes in your OST. If you have a large OST, this is going to take some time, but I can tell you that the wait is well worth it. It is actually an entirely new experience at least for me anyway's since I have a good deal of email item counts in my folders. Switching between folders with large item counts is no longer painful and this hotfix provides immediate viewing of these folders.

Grab the hotfix from the following article:

Description of the Outlook 2007 hotfix package (Outlook.msp): February 24, 2009
http://support.microsoft.com/kb/961752

Check out the plethora of improvements in this release in the following article as there are many.

Outlook 2007 improvements in the February 2009 cumulative update
http://support.microsoft.com/kb/968009

Hope this helps tame some of the larger mailboxes that you have.

Mike

Posted by mikelag | 1 Comments
Filed under: ,

Client RPC Dialog box questionnaire for Administrators

There are times when you are troubleshooting an Exchange Server issue where it appears that the server is performing OK, but the users are still complaining of the dreaded RPC dialog box and hangs in their client. Most of the time an Exchange administrator or helpdesk personnel needs to speak directly with the end user to determine what actions they were taking at the time the RPC dialog box occurs. Since there are numerous ways which can promote this dialog box, an administrator needs to understand specific actions that users were taking at the time of the problem. A lot of the times, this has nothing to do with server side performance problems, but rather something that is installed on the client or something the user is doing.

I have created a simple document in which the users can answer to allow you to gain some insight in to a users actions and their habits that are aggravating this RPC dialog box.

The document is password protected so that the fields are checkable. The password currently is "Microsoft".

Please provide feedback regarding this document to help make this better.

Thanks,

Mike

Posted by mikelag | 1 Comments
Filed under: ,

Attachment(s): Client Questionnaire.doc

New Windows Dynamic Cache Service for 64-bit Windows 2003 servers

If you've ever had an issue where low memory conditions were causing working set trimming issues due to excessive use of the System File Cache, then we have just released a new service that can be used to help alleviate this issue called Microsoft Windows Dynamic Cache Service.

More information regarding this new service can be found here and a direct link to download this new service can be found here

With Exchange 2007 servers also running in to these issues which I blogged about here, this service could potentially allow other 3rd party services to play nice with Exchange 2007 which may be consuming more than it's fair share of the System File Cache.

So if you find that Exchange performance is suffering because of some other service taking up overall memory in the System File Cache, then this service may be just for you.

Hope this helps with some of your performance related issues.

Mike

Posted by mikelag | 0 Comments
Filed under: ,

Performance Counter Collection Tools for Exchange

I've created a Category called Perfwiz in which you can easily find all Perfwiz versions. Click on http://blogs.technet.com/mikelag/archive/tags/Perfwiz/default.aspx to get you there.

Have a good one!!

Posted by mikelag | 0 Comments
Filed under:

Updated Exchange 2003 Perfwiz

For all of you folks out there still hanging on to Exchange 2003 and dealing with performance issues, I have taken the liberty to update the Perfwiz data counter collection to the latest/greatest counters as the old Perfwiz tool on our download site is severely outdated.

 

Exchange 2003 Update Perfwiz

  1. Extract the contents of E2k3_Perfwiz.zip to your Exchange server.
  2. Open Performance Monitor
  3. Expand Performance Logs and Alerts and select Counter Logs.
  4. Right click Counter logs and select "New Log Settings From". Select the htm file that was extracted in Step 1. Click OK
  5. Select the Log Files tab and click the Configure button
  6. For the Log location, change this to a location of your choice. Click Ok
  7. Click OK to save the Performance Counter log.
  8. To start the Perfmon log, right click "Exchange_2003_Perfwiz" and then select Start.
  9. Let this run during the problem period where performance is affected
  10. Stop the perfmon log by right-clicking "Exchange_2003_Perfwiz" and selecting Stop
  11. Make arrangements with a CSS representative to get the files analyzed.

Counter Collection List

\Database ==> Instances(*)\*
\Database(*)\*
\Epoxy(*)\*
\Exchange Server HTTP Extensions(*)\*
\LogicalDisk(*)\*
\Memory\*
\Microsoft Exchange ActiveSync\*
\MSExchange Intelligent Message Filter(*)\*
\MSExchange Oledb Events(*)\*
\MSExchange Oledb Resource(*)\*
\MSExchange Sender ID(*)\*
\MSExchange Web Mail(*)\*
\MSExchangeActiveSyncNotify OmaPush\*
\MSExchangeAL(*)\*
\MSExchangeDSAccess Caches(*)\*
\MSExchangeDSAccess Domain Controllers(*)\*
\MSExchangeDSAccess Processes(*)\*
\MSExchangeIMAP4(*)\*
\MSExchangeIS Mailbox(*)\*
\MSExchangeIS Public(*)\*
\MSExchangeIS\*
\MSExchangeMTA Connections(*)\*
\MSExchangeMTA\*
\MSExchangeOMA\*
\MSExchangeSA - NSPI Proxy\*
\MSExchangeSRS\*
\MSExchangeTransport Filter Sink(*)\*
\MSExchangeTransport Store Driver(*)\*
\Network Interface(*)\*
\Paging File(*)\*
\PhysicalDisk(*)\*
\Process(*)\*
\Processor(*)\*
\Server\*
\SMTP NTFS Store Driver(*)\*
\SMTP Routing(*)\*
\SMTP Server(*)\*
\System\*
\Web Service(*)\*

Enjoy!!

Posted by mikelag | 2 Comments
Filed under: , ,

Getting started with Exchange 2007 Performance monitoring and Windows 2008

With all the moving bells and whistles of Exchange 2007 and the amount of dependencies that Exchange relies on, it could be a bit overwhelming on where to get started. For some, they may have their own monitoring tools such as MOM/SCOM to pull this data, but when you are in a real-time crisis or something that affecting critical business functions over time, finding out where to start can lead one in to frustration.

Well, never fear, I'll help guide you in to looking at the server from the 30,000 ft view and then delve deeper in to the plethora of pertinent counters to look at. So where do I get started?

We could start off with a once upon the time story or 10 ways to lick an Exchange performance problem in the bud, but lets start with the simple basics. First, you will need to start some monitoring on your server to start collecting the data. I've made this quite easy for you as I have created templates for you that are very easily importable in to the Reliability and Performance monitor in Windows 2008. I'll be concentrating on Windows 2008 analysis in this blog as that is the wave of the future. Go to http://blogs.technet.com/mikelag/archive/2008/05/02/perfwiz-replacement-for-exchange-2007.aspx to find the templates for your particular role installation. Once you have imported your favorite XML file, you will get a screen similar to the following. Simply right-click on your imported counter set and click start. Let this run through the problem time in which you are trying to troubleshoot. Note: I have imported all of the XML files here to get started.

image

Once you have collected enough data, you can stop the Performance log. From there, you can then expand reports in the menu and then click on the report associated with the log that you just ran. You will get a screen similar to the following. The really cool thing about Windows 2008 is that if you should double-click on a performance file (.blg) or open this report view, you will get all counters added automatically with all Totals listed for each counter. Trying to load all counters/all instances would take a substantial amount of time to load, so I feel this is the best place to begin your troubleshooting. This is what I call the 30,000 ft view. Loading this type of view is normally very fast to load and provides you a wealth of information right at your fingertips.

image

What you will want to do is to compare each of the counters with the threshold values that are listed in the Technet article Monitoring Without System Center Operations Manager. If any counters are deemed suspect, I would put them on your list of things to delve deeper in to. Once you go through all of these counters, you could see if there is anything that stands out which will give you something to concentrate on. Counters that you no longer want to see can be easily hidden as shown below or by simply un-checking the box in the Show column of each counter

image

Now, since we are only showing totals at this point, you may need to jump off the diving board and dive right in to specific instances of a counter in which a threshold is above normal. Simply click the Plus imagesign on the menu and then add the counters of your choice.

That is basically all there is too it in getting started, but there are other ways of getting to root cause of a complex issue. Read on...

If you have collected a .blg file already, you can parse all of the data automatically via the Performance Analyzer Log tool. See my previous blog on how to use this most helpful tool here

Another way to troubleshoot Exchange Performance can be had in the Exchange Management Console Toolbox as shown below via the Performance Troubleshooter. This can also be very helpful in troubleshooting these issues as this tool has some built in tracing mechanisms that allow you to peer in to some of the Exchange processes in which you cannot get to by any normal logging or tracing means without PSS involved.image

I'll be posting most of these type posts coming in the future, so keep a watch on my blog.

Mike

Posted by mikelag | 1 Comments
Filed under: ,

Updated Perfwiz for Windows 2008 servers

I updated my previous post that included only a Windows 2003 version of the Perfwiz replacement with a Windows 2008 XML version now. Since Performance counter imports are different in Windows 2008, this needed to be converted to a format that it understood. Check it out at http://blogs.technet.com/mikelag/archive/2008/05/02/perfwiz-replacement-for-exchange-2007.aspx

Enjoy!!

Mike

Posted by mikelag | 0 Comments

Steps to help mitigate Excessive Paging and Working Set trimming issues on Exchange 2007 servers

Previously, I wrote a blog about excessive paging on Exchange 2007 servers due to a system wide working set trimming problem at http://blogs.technet.com/mikelag/archive/2007/12/19/working-set-trimming.aspx. Over time, I have been asked to consult on many cases that had to do either with high memory usage or a server that was paging excessive causing performance problems.

I've moved the recommendations here to make it easier to reference. Find below a listing of some of the top reasons why you may run in to these issues

Top Reasons for these issues

  1. The amount of Available RAM on the server reaches 64MB causing the Operating System to issue a low memory notification if QueryMemoryResourceNotification fails. This in turn could causes a system wide working set trim across ALL applications on any given server
  2. Other Applications/Services consuming System Cache over time on the server could eventually lead to the low memory notification being triggered.
  3. Memory or handle leaks caused by other applications. This is not to say that Exchange is exempt from having its own memory leaks, but this is merely stating what I have found throughout troubleshooting these issues
  4. Improperly configured servers for sufficient RAM to support the amount of roles and current user load being generated
  5. Other applications being run directly on the Exchange Server leading to problems listed in issue #1.

Exchange Server Update Recommendations - Last updated (6-09-09) 
Find below a list of recommendations to help alleviate the working set trimming issues that you may be running in to on Exchange 2007 Mailbox servers

  1. Apply Windows 2003 SP2
  2. Apply the latest Exchange roll-ups to address any previously known memory issues.
  3. Apply latest Forefront roll-ups if applicable. See http://support.microsoft.com/kb/954941 for one such example
  4. Make sure that the paging file is set correctly to the recommended settings. See http://technet.microsoft.com/en-us/library/aa996719.aspx
    • If the server has 8GB of RAM or less, set the paging file to RAM times 1.5
    • If the server has more than 8GB of RAM, set the paging file to RAM+10MB.
  5. Apply .NET Framework 2.0 SP1 at http://www.microsoft.com/downloads/details.aspx?familyid=029196ed-04eb-471e-8a99-3c61d19a4c5a&displaylang=en
  6. If an HP ILO Driver (CPQCIDRV.sys) is being used, ensure that the latest driver is installed. See http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c00688313 for more information.
  7. Update all Network card drivers to the latest versions as there are some versions that are known to call MMAllocateContiguousMemory (Broadcom) excessively.
  8. Install http://support.microsoft.com/kb/954337 as a replacement for 938486 and 953600 to get the latest kernel OS fixes to help with the amount of pages that are trimmed, when they happen.
  9. Apply 905865 http://support.microsoft.com/kb/905865
    The sizes of the working sets of all the processes in a console session may be trimmed when you use Terminal Services (RDP) to log on to or log off from a computer that is running Windows Server 2003
  10. Apply 948496 http://support.microsoft.com/kb/948496 or disable advanced TCP features of the Scalable networking Pack and then apply the Scalable Networking pack rollup in http://support.microsoft.com/kb/950224
    An update to turn off default SNP features is available for Windows Server 2003-based and Small Business Server 2003-based computers
  11. Ensure Antivirus Exclusions are set appropriately per http://technet.microsoft.com/en-us/library/bb332342.aspx 
  12. Apply 940349 http://support.microsoft.com/kb/940349 is VSS backups are being taken to help address known memory leak issues.
  13. If Symantec Altiris Agents are installed, make sure you are on the latest bits as older version have known handle leaks. See https://kb.altiris.com/ Article ID: 35454
  14. Backup Exec Remote Agent (Beremote.exe) not releasing memory with GRT enabled after backups are completed. See http://seer.entsupport.symantec.com/docs/289695.htm
  15. Apply http://support.microsoft.com/kb/954903 to address known memory leak issues with SCOM and its agents. Also see http://support.microsoft.com/kb/891605 for a known issue with MOM and McAfee VirusScan Enterprise 8.0i.
  16. Quest ChangeAuditor for Exchange taking up excessive memory and processor time. For immediate relief, stop the Quest Compliance Agent service and then contact Quest to request update for Solution SOL53092 which will be rolled in to ChangeAuditor v4.9.

If the above updates do not resolve the issue, find below some recommendations on where to go at this point.

Outlook Client Recommendations

  • Disable unneeded Outlook Add-ins. Some Add-ins have been known to increase overall session usage and opening unnecessary messages/folders on the Exchange Server. This in turn causes increased memory and connection usage which could lead to 9646 events in the application log.

    1. One such example is the add-ins installed with Apple Itunes which allow calendar syncing with IPod and IPhone devices. If you have Itunes installed and do not have any of these devices, I highly recommend disabling both the "Outlook Change Notifier" and the "ITunes Outlook Addin" under the COM Add-ins settings in Outlook to prevent this problem. <Real World Scenario>, disabling these add-ins on Outlook clients in one organization prevented the Exchange server from having to be rebooted every 2-3 weeks </Real World Scenario>.

Other Recommendations

  • Monitor applications that perform buffered I/O on the server such as Content Indexing that make use of the file cache to ensure that this is not causing excessive memory growth within the system cache which is outside of the DBCache. Log shipping uses unbuffered I/O, so we will not make use of the system cache here.

  • Check for any scheduled tasks that might be running on a periodic basis to see if this might be causing this problem. There have been reports that scheduled ScanMail jobs cause working set trims to occur. We have confirmed that Scanmail uses memory mapped IO with the files they create to do their real time scanning, so this will affect overall system cache sizes.

  • Manage Exchange 2007 servers remotely and not locally on the server itself via an RDP session. Meaning, install the Management tools on your workstation and remotely administer the server. The reason for this is that the Management tools take a considerable amount of memory to load, sometimes 100-300MB, and in low memory conditions, this could trigger a system wide working set trim when the EMC is launched.

  • If VSS snapshots are being taken from the Active Node in a CCR configuration, try switching to backing up the passive node instead. Moving the backups to the passive node is a recommend best practice and will also conserve Memory/CPU on the server. As always, if you must run backups on the Active Node, ensure that the backup process does not overlap your Online Maintenance processes. There is one caveat with moving the backups to the passive node is that the active node DB's integrity is no longer checked. You can overcome this by setting a registry key (Online Maintenance Checksum) that is mentioned in http://technet.microsoft.com/en-us/library/bb676454(EXCHG.80).aspx.

  • Stagger online maintenance schedules for each mailbox store to limit the amount of overlapping. Online maintenance is aggressive in nature and touches a lot of pages in memory to perform it's work. This increases overall memory usage and you could see a 4GB swing upward in memory during this time. Once online maintenance finishes, the overall memory usage should return to normal. It is important to watch the Available Memory on an Exchange server because if we get close to the 64MB available memory mark, a system wide memory trim could occur causing excessive paging and huge performance hits. The operating system has a built-in low memory indicator to signal to the OS that we need to release memory from working sets of processes running on the server.

    One additional piece in this area is to check if your OLD window can be decreased from the default values if your Read:Freed ratio is 100:1 or greater. We don't need OLD to run to completion every night with SP1 installed and can be spread across multiple days or weeks as long as your Read:Freed ratio is within certain values. This ratio can be calculated by viewing MSExchangeDatabase -> Online Defrag Pages Freed/sec and MSExchangeDatabase -> Online Defrag Pages Read/sec in Performance Monitor. See http://msexchangeteam.com/archive/2007/12/06/447695.aspx for more information.

  • Monitor overall System I/O and Database I/O on the server with relationship to the amount of hard page faults (Transition pages repurposed/sec) that are occurring. If the amount of I/O that the databases is incurring is causing hard page faults, the DBCache size will grow. If the database is doing less I/O and not incurring hard page faults, then the size of the DBCache should decrease over time.

  • If System Cache is consuming all available memory on a server causing a system wide working set trim, there are 2 ways to overcome this:

  • Temporarily disable Online Maintenance Database Scanning (Checksumming or ESE-Zeroing) to see if the problem is corrected. See http://technet.microsoft.com/en-us/library/bb676537(EXCHG.80).aspx.

  • Upgrade the server to Windows 2008 as memory management has be redesigned to handle these type situations much better. Since there is no upgrade method, this would essentially be a migration to another Exchange 2007 server running on Windows 2008.

Setting DBCache – No longer recommended

You may ask why I no longer recommend setting DBCache on an Exchange Server. In almost 99.99% of the cases I have worked on, every issue was resolved by fixing another application or component on the server either leaking memory or causing excessive handle counts with no modification to the ESE Database Cache.

With that said, I now have a new motto, slogan, or saying for these type issues on an Exchange server. Here goes....

"Exchange is a victim of someone else behaving badly"

If you stand up and recite this a couple of times in the middle of troubleshooting one of these issues, then you will soon realize that this statement is 100% true. If you find differently, please let me know as I would love to hear your story on this.

Posted by mikelag | 3 Comments
Filed under:

Showing Process ID Information in Performance Monitor

In some situations while monitoring performance of a server, you may find a time when a particular process is restarting or crashing over time. By default, in Performance Monitor, you cannot view Process ID (PID) values to determine if a process was restarted for whatever reason. This is apparent for IIS Application pools as these processes could frequently crash/restart throughout the day. With each new restart of a process, a different PID is generated, so it would be nice if you could see this happening in Perfmon.

Luckily, there is a way to do just that to append the PID number for each process, so that if the PID should change, you can easily detect this in Performance Monitor.

To show ProcessID's along with process names, add the following registry entry on the server

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PerfProc\Performance]

"ProcessNameFormat"=dword:00000002

This needs to be implemented on the machine in which the perfmon is being taken from. For more information regarding this registry setting, see 281884

Posted by mikelag | 0 Comments
Filed under:

Slow Outlook Online Mode Performance

Switching to or running Outlook in Online mode against an Exchange 2003 server may exhibit slower than normal performance when clicking on particular folders in your mailbox that contain high item counts.

This is a known issue when the “Show in Groups” option is turned on in Outlook when many items exist in a particular folder. The fix is actually two parts. One for the server side and one for the client side. Note: The server side fix is not necessary for Exchange 2007.

What you might see in Performance Monitor is similar to the load that Client side search engines add. This counter in particular is “Slow Findrow Rate” located under MSExchangeIS Mailbox. One might think that this counter is only used for detecting client search engines, but this counter increases whenever an application is crawling through a users mailbox.

In this case, Outlook creates a search restriction (“Show in Groups” with a descending date sort order) initially using a Fast FindRow call, but the server immediately switches to a Slow FindRow call which causes a large delay in returning the view for that newly created restriction while items in that particular folder are being queried. This could lead up to the dreaded RPC dialog box or the balloon stating the Outlook is requesting data from the server. While we are creating this restriction on the Exchange Server, this may also cause an added CPU performance hit. Since these restrictions are not cached, these problems may occur each time the user clicks on these folders that may have high item counts.

To better optimize this search restriction behavior, an optimized search was added server side so that these restrictions created by Outlook will use the Fast FindRow call method instead of switching to the Slow Findrow method. This provides a huge performance gain and mitigates the Outlook delay problems.

To be able to handle these optimized searches efficiently, your server performance has to be within our recommended performance guidelines, so even though you have applied these fixes, there is still a possibility that these problems could occur due to underlying performance problems server side.

Here are the links to the articles and fixes for each.

Exchange 2003 Server Side fix
Error message when you enable the "Show in Groups" option for a folder that contains thousands of e-mail messages in Outlook: "Outlook is retrieving data from the Microsoft Exchange Server <ServerName>"
http://support.microsoft.com/kb/948828

Outlook fixes
Description of the Outlook 2003 hotfix package: February 21, 2008
http://support.microsoft.com/kb/949289

Description of the Outlook 2007 hotfix package: February 27, 2008
http://support.microsoft.com/kb/949401

 

Posted by mikelag | 0 Comments
Filed under: ,

Scalable Networking Pack Rollup Released

I am sure you are all are intimately familiar with the problems with the Scalable Networking Pack (SNP) including it's use of the TCP Chimney feature that I blogged about at http://msexchangeteam.com/archive/2007/07/18/446400.aspx.

The problems that surfaced due to these features being enabled by default in the Service Pack 2 release of Windows 2003 brought out the worst in some network card drivers causing all types of connectivity related problems and crippled some organizations.

After some time, a hotfix (http://support.microsoft.com/kb/948496) was created to disable these features by adding the appropriate registry keys to a server which then required a reboot to get the SNP features disabled.

As of 8/27/2008, a new hotfix (http://support.microsoft.com/kb/950224) has been released to help address some of the commonly reported problems with relationship to Chimney and RSS for Windows 2003 servers. As more companies deploy Windows 2008 and Vista, it is crucial, or in my opinion, critical that this hotfix be applied to all Windows 2003 servers that may communicate with these operating systems. One of the main reasons is a new feature called TCP auto-tuning which makes use of RSS to expand and shrink the sizes of your TCP window to increase/decrease throughput based on current network load. This feature greatly increases throughput on your network, but if there is an underlying problem with the network card driver or any of these features between disparate systems, you may experience slower than normal network performance. The good news is that the Chimney feature is disabled by default in Vista/Windows 2008.

 

Posted by mikelag | 6 Comments
Filed under:

Certain AMD processors might cause inaccurate counter data

If during your troubleshooting, you run in to some counters (Disk Read/Writes) that just don't seem correct based on the way the server is performing, then you might be running in to a drift calculation bug with certain AMD processors. You may also see negative PING response times as well.

Here is what it looks like in Performance Monitor as a steady rise in disk response times.

Avg. Disk sec/Read Counter

image

Avg. Disk sec/Write Counter 

image

Over time, this may look like a problem, but it really isn't, it is just that it is being reported incorrectly.

Clint Huffman, which is the creator of the PAL tool has a nice write-up on this problem at http://blogs.technet.com/clint_huffman/archive/2008/03/21/warning-perf-counter-data-might-be-inaccurate-on-some-amd-processor-computers.aspx

OEM References

HP - http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&taskId=110&prodSeriesId=428936&prodTypeId=15351&objectID=c01075682 

Citrix - http://support.citrix.com/article/CTX114782

IBM - http://www-01.ibm.com/support/docview.wss?rs=0&q1=opteron&q2=usepmtimer&uid=swg21267793&loc=en_US&cs=utf-8&cc=us&lang=en and ftp://ftp.software.ibm.com/systems/support/system_x/41y7884.rtf

Hope this helps in your travels.

 
Posted by mikelag | 0 Comments
Filed under:

Portable version of the Exchange 2007 ExBPA tool

Note: This is a read-only version of the tool and not the full fledged version that will scan Exchange Servers. Read-only meaning it will allow you to view reports from other servers

If you've ever had the need to make the version of ExBPA portable without having to install the Management Tools on a machine, here is how you can do this using the files installed on your production Exchange Server. Extracted, this is just over 6.5MB in size.

In this example, I am using the 64-bit binaries on a Vista machine and the English versions of the XMLs

To start, create a directory on your Vista Machine such as ExBPAx64 as shown below. You can use whatever directory name you choose, but mine is just an example.

Copy the below files from the Exchange Server to the following directories. These files by default are located in C:\Program Files\Microsoft\Exchange Server\Bin.

image

Here is what the sub-folder En looks like. This En sub-folder denotes the English language, but if you are using any other language, you could simply copy over the appropriate directory from the C:\Program Files\Microsoft\Exchange Server\Bin directory.

image

That is all there is to it. You can now launch ExBPA.exe and get the same functionality from anywhere.

For the most up to date rules, even for scanning Exchange 2003 servers, I would highly recommend to use the Exchange 2007 version of ExBPA to scan all of your servers instead of the publicly released version from our Analyzers site for the most accurate information.

Mike

Posted by mikelag | 1 Comments
Filed under:

How to enable Store PFD tracing upon Reboot

Tracing Store startup problems upon server reboot is not an easy task as ExTRA tracing does not survive the server being rebooted whereas in Exchange 2003, the “Enable Tracing” key did survive a reboot.

After enabling an ExTRA session for PFD store startup tracing and looking at the contents of what was enabled when doing this, you will see the following from a command prompt.

image

Notice the “Microsoft Exchange Server 2007” provider with its GUID of 79bb49e6-2a2c-46e4-9167-fa122525d540. This is the provider that Exchange 2007 uses for most of its tracing

So now we need a way to trace this out. First, we need to create a method to enable logging and another way to enable the logging upon reboot of the server

Enable Tracing

Method 1

Logman (Built in to the OS)

---------------

1. Create a directory called Tracing (ex. C:\tracing)

2. Create the trace log using the “Microsoft Exchange Server 2007” provider that Exchange 2007 tracing uses. The GUID or the name of the provider can be used as shown below.

logman create trace ExchangeDebugTraces -p {79bb49e6-2a2c-46e4-9167-fa122525d540} -nb 3 25 -bs 3 -o c:\Tracing\

logman create trace ExchangeDebugTraces -p " Microsoft Exchange Server 2007" -nb 3 25 -bs 3 -o c:\Tracing\

After performing this, you can see this using a logman query command. 

image

3. Create an EnabledTraces.Config file and add the following information to it and then save this file to the root of your C: drive. C:\EnabledTraces.Config

TraceLevels:Pfd
Store:tagPFDServerStart
FilteredTracing:No
InMemoryTracing:No

Note: You could also create this same file by using the ExTRA interface to enable this tracing as well, thus populating the same information as listed above.

4. Create a start_tracing.cmd file and add the following information to the file. Save this to the location created in Step 1.

Logman start ExchangeDebugTraces
Running logman query shows this trace running. 

image

5. Create a stop_tracing.cmd file and add the following information to the file. Save this to the location created in Step 2.

logman stop ExchangeDebugTraces

Method 2

Tracelog

6. Download and install tracelog from http://www.microsoft.com/downloads/details.aspx?FamilyID=55E51B3B-6C26-4CA0-ABF1-0E51D92B8298&displaylang=en

7. Create a directory called Tracing (ex. C:\tracing)

8. Copy tracelog.exe to this directory from the default install location of c:\program files(x86)\Resource Kit.

9. Create a start_tracing.cmd file and add the following information to the file. Save this to the location created in Step 2.

tracelog.exe -start ExchangeDebugTraces -f c:\Tracing\ExchangeDebugTraces.etl -seq 3500 -guid control.guid

10. Create a stop_tracing.cmd file and add the following information to the file. Save this to the location created in Step 2.

tracelog.exe -stop ExchangeDebugTraces

11. Create a control.guid file and then add the appropriate GUID for Microsoft Exchange Server 2007 tracing. Note: This GUID should be the only piece of information in this file. Save this to the c:\tracing directory.

79bb49e6-2a2c-46e4-9167-fa122525d540

12. Create an EnabledTraces.Config file and add the following information to it and then save this file to the root of your C: drive. C:\EnabledTraces.Config

TraceLevels:Pfd
Store:tagPFDServerStart
FilteredTracing:No
InMemoryTracing:No

13. To verify if these batch files work successfully, run start_tracing.cmd and then run tracelog –q ExchangeDebugTraces to check if we are now tracing 

image

This looks good. We are now happily tracing using PFD store startup tracing.

14. Run stop_tracing.cmd to stop the tracing.

Enable Logging upon bootup

On Windows 2003 and Windows 2008 servers, we can enable this tracing when the server is rebooted by using the built-in utility called Schtasks. More information regarding this tool can be found in the following locations

Windows 2008 schtasks help
Windows 2003 schtasks help

To create a new scheduled task, open a command prompt and type the following:

schtasks /create /tn "Start Store PFD Tracing Upon Reboot" /tr "C:\tracing\start_tracing.cmd" /sc onstart /ru ""

Note: Specifying "" for the /RU switch, tells the task to run as SYSTEM.

To verify that the scheduled task has been added properly, type schtasks from a command prompt. This should return the 'TaskName" and the "Next Run Time" of "At system start up"

You could also view this in the Control Panel under Scheduled Tasks
image

Reboot the server and the tracing will begin before the Information Store service starts.

Once the tracing has been collected, it is important to stop tracing and then remove this schtask to prevent this tracing from occurring in the future upon reboots.

To stop tracing, simply launch the stop_tracing.cmd file in the c:\tracing directory.

To remove the scheduled event, type the following command:

schtasks /delete /TN "Start Store PFD Tracing Upon Reboot" 

Unfortunately, the trace file cannot be read unless you are working with Microsoft Support representative, but at least this will give you some ways that will help catch problems as they occur and when they occur. These methods can be applied for any ExTRA tracing in general, so this is not just limited to tracing Store Startup issues.

Happy Troubleshooting!!

Mike

Posted by mikelag | 0 Comments
Filed under:
More Posts Next page »
 
Page view tracker