This tip pertains to all versions of SharePoint and no versions of SharePoint. It is really more of an IIS troubleshooting tip.
When strange things happen to IIS application pools I am reminded of the old public service announcement, “Do you know where your children are?” Well, do you know what your application pools are doing? When was the last time one restarted? What is the history of restarts? If you cannot answer these questions read ahead.
To find out when an application pool has started or how long it has been running use Process Explorer to view the time the application pool process started.
Now that you can find out the current start times of application pools how do you get a history of when application pools have been restarted.
Enable logging of application pool restarts to the Event Log; http://support.microsoft.com/kb/332088. To make a long KB simple; use the following command to enable the logging of all types of restarts for an application pool:
cscript adsutil.vbs Set "w3svc/AppPools/<App Pool Name>/LogEventOnRecycle" 255
To disable logging of recycle events for an application pool use the following:
cscript adsutil.vbs Set "w3svc/AppPools/<App Pool Name>/LogEventOnRecycle" 0
After the logging is set there is no need to IISRESET or reboot, the change is instant.
To view the recycle events open Event Viewer on the machine and look in the System Event Log for the following event ids: 1074, 1075, 1076, 1077, 1078, 1079, 1080 and 1177.
Now you can keep track of those application pools.
UPDATE 8:39P 9/21/2010 - Looks like the SharePoint Team blog post has been updated with new information. While you could do the steps below for WSS 3.0/MOSS 2007, they are not needed. Only SharePoint 2010 and WSS 2.0 are affected.
There have been some questions on whether or not WSS 3.0 and MOSS 2007 are affected by the Microsoft Security Advisory 2416728. Since the reported vulnerability deals with .Net Framework, specifically ASP.NET and error pages WSS 3.0 and MOSS 2007 may be affected (see above for update).
The SharePoint Team blog has some workaround steps for SharePoint 2010 that help to mitigate the attack. So what are the steps for WSS/MOSS? If you read the advisory, it becomes clear that there are 2 things that are needed for the workaround.
So what are the steps? Well, they are almost identical to the SharePoint 2010, with 2 minor differences.
<%@ Page Language="C#" AutoEventWireup="true" %> <%@ Import Namespace="System.Security.Cryptography" %> <%@ Import Namespace="System.Threading" %> <script runat="server"> void Page_Load() { byte[] delay = new byte[1]; RandomNumberGenerator prng = new RNGCryptoServiceProvider(); prng.GetBytes(delay); Thread.Sleep((int)delay[0]); IDisposable disposable = prng as IDisposable; if (disposable != null) { disposable.Dispose(); } } </script> <html> <head runat="server"> <title>Error</title> </head> <body> <div> An error occurred while processing your request. </div> </body> </html>
<customErrors mode="On" redirectMode="ResponseRewrite"defaultRedirect="/_layouts/error2.aspx" />
If .Net Framework 3.5 or less is installed change the line to;
<customErrors mode="On" defaultRedirect="/_layouts/error2.aspx" />
The reason that the web.config customErrors line is different for computers with .Net 3.5 or below is that the redirectMode property was not available before .Net 3.5 SP1.
This tip is for SharePoint Foundation 2010, SharePoint Server 2010 and Search Server 2010 Express.
So you have your heart set on the WSS 3.0/MOSS 2007 look; I know, how could you not just looooove it? But have just installed a fresh new SharePoint 2010 farm. Or, you have visual upgraded your sites after an upgrade from WSS 3.0/MOSS 2007 to SharePoint 2010 and want that 2007 look and feel. So how can you go back?
As it turns out, it’s easy with the power of PowerShell. The scriptlets below will show you how you can switch from version to version for every site in a site collection and to switch a single site.
To change the look and feel for every site in a site collection to WSS 3.0/MOSS 2007, use the following example code:
$SiteCollection=Get-SPsite http://SiteCollection foreach($SPWeb in $SiteCollection.AllWebs){$SPWeb.UIversion=3;$SPWeb.update();}
To change the look and feel for every site in a site collection to SharePoint 2010, use the following example code:
$SiteCollection=Get-SPsite http://SiteCollection foreach($SPWeb in $SiteCollection.AllWebs){$SPWeb.UIversion=4;$SPWeb.update();}
To change the look and feel for 1 site in a site collection to WSS 3.0/MOSS 2007, use the following example code:
$SPWeb=Get-SPWeb http://Site
$SPWeb.UIversion=3;$SPWeb.Update();
To change the look and feel for 1 site in a site collection to SharePoint 2010, use the following example code:
$SPWeb.UIversion=4;$SPWeb.Update();
So now you can go back and forth all day basking in your new found Visual Upgrade power! Next, SharePoint 2010 with laser beams!
When upgrading to SharePoint 2010 from WSS 3.0, MOSS 2007 or Search Search 2008 there are some choices on whether or not the administrator wants to upgrade sites to the new SharePoint 2010 look and feel.
If the sites are not upgraded to the new look and feel, there is a way to try out and then set the new look and feel through the site settings page->Title, description, and icon page.
If the sites are upgraded to the new look and feel, there is not a way to go back to the previous versions look through the site settings page->Title, description, and icon page.
There is hope though to go back to the previous version look and feel if it is needed, the SharePoint 2010 Management Shell.
To enable the look and feel choice back to enabled on every site within a site collection, use the script below. Replace SiteCollection with the actual URL for your site collection.
$SiteCollection=Get-SPsite http://SiteCollection foreach($SPWeb in $SiteCollection.AllWebs){$SPWeb.UIversionConfigurationEnabled=$true;$SPWeb.update();}
To enable the look and feel choice back to enabled on a single site use the following command. Replace Site with the actual URL for your site/web.
$Site=Get-SPWeb http://Site $Site.UIversionConfigurationEnabled=$true;$Site.update();
Next tip I will show you how you can go back and forth between look and feel versions even on a brand new 2010 site. So if you just love the WSS 3.0 site look you can get that too right in SharePoint 2010.
This tip is for SharePoint Server 2010 and Search Server 2010 Express.
In the olden days MOSS 2007 ;) you could set a dedicated web front end for crawling which is used to reduce the stress of crawling from other web front end servers that users use to browse.
This setting is for the entire search service, so all local SharePoint sites will use the dedicated web front end server.
In SharePoint Server 2010 you can still set a dedicated web front end for crawling but there are some changes. First the setting is not search service wide, it is a web application setting. Second, as of this time, there is no UI to change the web application settings for dedicated crawls. To set a web application to crawl with a dedicated web front end you will need to use PowerShell. The link above has some examples of PowerShell to add a dedicated web front end and to remove a web front end. To list if a web application is using dedicated web front end settings use the example below, if any values are returned, then the web application has dedicate web front end settings. Otherwise the output will be empty (SiteDataServers : {}).
$WebApplication=Get-SPWebApplication http://www.contoso.com
$WebApplication|fl SiteDataServers
This tip is for Microsoft Office SharePoint Server 2007, Search Server 2008, SharePoint Server 2010 and Search Server 2010 Express.
Need to return search results with a different URL than the URL that was crawled by SharePoint? You can do this using a Server Name Mapping in the search settings of SharePoint.
One caveat is that after the mapping is created, a full crawl has to be completed for the new URLs to show up in the results; http://support.microsoft.com/kb/974414.
If your SharePoint 2010 servers are running Automatic Updates you may see a new SharePoint Foundation 2010 update listed. Update 2032588 was released yesterday, July 13, 2010. For more information on what this update contains see the following KB; http://support.microsoft.com/kb/2032588. There is one thing to add to the KB though; the files list looks to be missing a file that is updated by this update. Microsoft.SharePoint.dll is updated to version 14.0.5050.5001 in addition to the other files listed in the KB.
The update can also be downloaded from a link in the KB article. Remember to run the SharePoint Products and Technologies Configuration Wizard when the update completes. The SharePoint server may need a reboot after the update installs.
Are you sick of waiting for a timer job to run? Don't want to use the "Run Now" button in the timer job definitions properties page in Central Administration->Monitoring->Review Job Definitions? Or do not know which exact timer job needs to run for a specific function?
Use the commands below in the SharePoint 2010 Management Console to run every timer job in the farm that is not disabled (FYI: disabled jobs throw an exception if you try to run them, but no harm is done.).
$timers=Get-SPTimerJob|? {$_.isDisabled -eq $false}
foreach ($timer in $timers) {$timer.RunNow()}
A good use of these commands is in development work, a lot of time can be wasted waiting for timer jobs to perform actions on data changed while developing and sometimes it's easier to just run all of the timer jobs if the specific job is unknown. That said, I do not recommend running this command on a production farm as you may see at the very least a performance hit while all of the jobs run and worse, errors while running the jobs.
This tip is for SharePoint Server 2010, Search Server 2010 and Search Server 2010 Express.
SharePoint Server 2010 and Search Server 2010 Express include a new feature for search called Search Suggestions. Search suggestions are listing below the search box that suggests search terms while you are typing your search query. They have been around for a few years in sites like www.bing.com and are now integrated into SharePoint 2010. Below is a screenshot of a SharePoint Enterprise Search Center site with a suggestion.
Suggestions are automatically built based on what search results are actually clicked on. It takes 6 clicks within a year for SharePoint to add a suggestion; if an administrator needs to add a suggestion manually this can be done using the SharePoint 2010 Management Shell (PowerShell). The commands below will add the suggestion, "Suggestion" to the first Search Service Application and run the Prepare Query Suggestions timer job now. The Prepare Query Suggestions timer job is set to run daily between 1AM and 11PM local server time so this will speed up the time before the suggestion will appear in the search query suggestions list.
$ssa=Get-SPEnterpriseSearchServiceApplication
New-SPEnterpriseSearchLanguageResourcePhrase -SearchApplication $ssa -Language en-US -Type QuerySuggestionAlwaysSuggest -Name "Suggestion"
$timer=Get-SPTimerJob|? {$_.Name -eq "Prepare Query Suggestions"}
$timer.RunNow()
After the above commands have been run and the timer job has been run, the suggestion and other suggestions in a Search Service Application can be listed by running the following command from the SharePoint 2010 Management Console.
Get-SPEnterpriseSearchQuerySuggestionCandidates -SearchApp $ssa
To remove the suggestion, run the following command.
Remove-SPEnterpriseSearchLanguageResourcePhrase -SearchApplication $ssa -Language En-Us -Type QuerySuggestionAlwaysSuggest -Identity "Suggestion"
Before even writing on how this can be done I have to state that I would not recommend this be done on a production system. Previous versions of any software is going to leave traces of itself even after an uninstall so, while everything may seem to be working fine at the time, issues may occur down the line from this type of workaround.
While trying to install SharePoint RTM version, the setup program may produce the following error if a previous version was installed; "The install in progress conflicts with a previously installed Microsoft Office 2010 Server product" Fig. 1. To find out how this error can be fixed, first let’s look at how a Microsoft Installer setup checks for previous versions of a program.
Fig. 1
When the setup for SharePoint 2010 runs the Windows Installer components it checks for previous versions. These previous versions will usually be listed in the Control Panel->Programs and Features in Windows 2008. If an update has been installed for a product though, it may be listed in the “View installed updates” link in Programs and Features. Fig. 2.
Fig. 2
When the View installed updates is clicked the Programs and Features will change to list only updates to the Programs installed. Updates listed here can be uninstalled. If any SharePoint updates are listed click them and uninstall. Fig. 3.
Fig. 3
After uninstalling all of the SharePoint related programs from both “Programs and Features” and “View installed updates” reboot the computer and try the install again. If the setup is still finding the previous versions check that there are no programs listed in the registry under the following key: HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall. Look first for any key with names similar to the ones in Fig. 4. See the following KB article on using the Windows Registry Editor for the warnings on usage; http://support.microsoft.com/kb/256986.
Fig. 4
For each GUID found ending in 0000000FF1CE in the HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall, find the UninstallString value and copy out the data by double-clicking or right-clicking and selecting Modify. In the “Edit String” window copy the “Value data”. Fig. 5. This is the command that will uninstall the package. Next, open a command prompt as Administrator by going to Start->All Programs->Accessories->Command Prompt; Right-Click Command Prompt and select “Run as administrator”. You may have to do this multiple times to remove all of the Office related registry entries in the uninstall key.
Fig. 5
After all of the entries are removed remember to reboot the computer, as the MSI will more than likely need the reboot to clean up all of the in use resources and may not let you spin up another install until a reboot is completed.
This tip is for SharePoint Foundation 2010 and SharePoint Server 2010 when running on Windows 2008 R2 and browsing SharePoint on Windows 7 and Windows 2008 R2 and up.
My internal training in the next version of SharePoint has sparked some intriguing thoughts around what will be possible in the near future to help our customers in situations where bandwidth constraints are making SharePoint performance less than optimal.
I have searched high and low, internally and our external sources (TechNet, MSDN), for any SharePoint specific information in regards to Branch Cache. There is a good white paper here; http://technet.microsoft.com/en-us/library/ff468721(WS.10).aspx on setting up a lab to test Branch Cache. The lab in this white paper is a good start on how to setup a lab where you can test the functionality of Branch Cache for SharePoint as it has steps on configuring IIS 7.5 for Branch Cache, which would be the same as setting up SharePoint for Branch Cache.
There are a few things I would like to add to the information given in the white paper though.
When testing the speed of downloading documents from the SharePoint server on client machines there is a very easy way to visualize the difference between a slower WAN type connection and the cached document download. Turn on bandwidth connection limits in IIS for the web application you are testing caching on (Figure 1). While testing, I set the Maximum Bandwidth (Bytes/second) to 57344; which is 50KB a second, or about the same as the old 56K MODEM days (Figure 2). For a multi megabyte file, you can see in the IE download window that without caching the file transfers at about 50KB (Figure 3).
Figure 2:
Figure 3:
The first time the document is downloaded from the SharePoint server, you will see the speed is about the same as the IIS web application bandwidth setting. Downloading the same file again, you should see that the file is downloaded much faster than the first time. This is the Branch Cache version of the file being downloaded. I could not snap a screenshot of the second download as the 3.5MB file completed too fast (under a second!).
To rerun the test on the same client run the following command to clear out the Branch Cache files on all of client computers.
NETSH branchcache flush
That’s it for now, the white paper above has some more information on how to use Performance Monitor to monitor Branch Cache usage and find out exactly how many bytes you are saving. So now when SharePoint 2010 is released you can rejoice in your new found SharePoint speed!
This tip is for Windows SharePoint Services 2.0, SharePoint Portal Server 2003, Windows SharePoint Services 3.0, Microsoft Office SharePoint Server 2007, SharePoint Foundation 2010 and SharePoint Server 2010 when running on Windows 2003 SP1 w/Scalable Networking Pack and versions of Windows since 2003 SP2.
There can be a lot of network traffic on a SharePoint server; incoming HTTP requests, incoming and outgoing SQL data and all of the rest of traffic that other applications may create. The networking updates can help with networking performance in these situations. That is, when the network adapter and driver support the updated networking features. If the network card or driver does not support or is having issues you may see errors on the server like; “General Network error”, “Communication link failure” and “A transport-level error has occurred” for more information check; http://support.microsoft.com/kb/945977.
To check whether or not a computer is using the TCP Offloading feature use Netstat –t at the command prompt and view the Offload Status column for each IP connection. More information is available in the following KB; http://support.microsoft.com/kb/951037.
This tip pertains to all versions of SharePoint and no versions of SharePoint. Again, it is really more of an IIS troubleshooting tip.
Since you have gone through Tip 30 and enabled all of the extended logging options in IIS you will not doubt have access to the “Protocol Status” and “Protocol Substatus” fields in all of your IIS logs. If not, go back and run through the tip and enable the options.
When a user cannot access a resource in SharePoint there can be many reasons for it. To track down the “why” of the failed access request a good place to start are the IIS logs. These logs are available in the location specified in the logging options in the web site properties. By default they are located at %SYSTEMROOT%\system32\Logfiles in IIS6 and c:\inetpub\Logfiles in IIS7.
The following list was taken from; The HTTP status codes in IIS 7.0.
I will get a little more in depth on these status codes as they relate to SharePoint. As there are many links in the chain of SharePoint security, it is best to have a starting point to begin troubleshooting and the HTTP status code will help in finding that starting point.
401.1
401.2
401.3
401.4
401.5
200.0
This tip pertains to all versions of SharePoint and no versions of SharePoint. It is really more of an IIS troubleshooting tip. But we’ll let it slide for today. :)
The extended logging options in IIS can help in troubleshooting issues in SharePoint. HTTP status code logging can help with unauthorized issues and time-taken logging can help in performance related issues.
To enable the extended logging options for all web sites in IIS 6.0, open Internet Information Services Manager and right-click the “Web Sites” node and select Properties. Under Enable Logging click the Properties button. In the Logging Properties window, click the Advanced tab. By default not all of the extended properties are selected, select all of the properties and click OK. For IIS 7 the wording a slightly different. See the images below for more details.
This tip is for Windows SharePoint Services 3.0, Microsoft Office SharePoint Server 2007 and may apply to SharePoint Foundation 2010 and SharePoint Server 2010.
Although it is a good practice to manage the RFC 2822 header fields, or X-Headers in your email system, SharePoint does use some X-Headers for alert and workflow emails that need to be allowed in messages for all functionality to work. For instance, the edit or open task button in Outlook client (see below in red) may not work if the X-Header to connect to the task list is not present.
When clicking on the open or edit buttons, you may receive an error like the following:
An error occurred while retrieving the workflow task details. This could be caused by not having connectivity to the server or because the task no longer exists.
Here is a list of some of the X-Headers used by SharePoint in various versions:
X-Sharing-WSSBaseUrl, X-AlertWebSoap, X-Sharing-ItemID, X-Sharing-Remote-Uid, X-Sharing-Config-Url, X-AlertServerType, X-AlertWebUrl, X-AlertId, X-AlertTitle, X-Sharing-Title
So if your email servers are using any kind of message filtering, like the Exchange Server 2010 Header Firewall and wanting to have full functionality in Outlook for SharePoint emails; make sure that the above X-Headers are not blocked.
This tip is for Windows SharePoint Services 2.0, SharePoint Portal Server 2003, Windows SharePoint Services 3.0, Microsoft Office SharePoint Server 2007, SharePoint Foundation 2010 and SharePoint Server 2010.
To reduce update surface and reduce conflicts, it is always a good idea to only load the specific versions of the .Net Framework needed for SharePoint to function. This follows with best practice to only load what is needed for the application(s) to function. Windows 2008 and IIS 7 are good examples of this type of methodology.
For WSS 2.0 and Portal Server 2003, this would be .Net Framework 1.1.
For WSS 3.0 and MOSS 2007, this would be .Net Framework 2.0 (x86, x64) and 3.0 (x86, x64).
For SharePoint Foundation 2010 and SharePoint Server 2010, this would be .Net Framework 3.5 SP1.
Note that some custom web parts and solutions may require a different .Net Framework version loaded on the SharePoint servers.
There is a hidden list in SharePoint called Workflow History. This list is hidden from the “View All Site Content” and the “Manage Content and Structure” pages but you can browse to it and view information on running and completed workflows. The list is located at http://site/Lists/workflow%20history. Whenever a new workflow is created using the default history list, the workflow writes information to this list. Since workflows can write many items to the list whenever a workflow is ran, you can imagine that it can grow fairly quickly. Since the workflow history is a list, it is also bound by capacity guidelines on list size. This means that after about 2000 items per level/view in the workflow history list, the list performance will suffer. The best way to mitigate this is to create a new workflow history list when a new workflow is created. This way the workflow information is spread out between more than one list. The way to create a new workflow history list for a workflow will very between the various workflow types, the Web GUI can be used to change the workflow history list when creating a new workflow instance based on one of the feature workflows like Approval, Three-State, Collect Signatures, Collect Feedback and Disposition Approval. To create a new workflow history list in the Web GUI go to the list/library settings->Workflow Settings->Add a workflow. Select “New history list” from the History List.
After running a workflow with a new history list, a new list will be created with the following format; <Workflow name> History. So if I created a workflow called Approval1, and selected new history list; then a list called Approval1 History will be created.
In workflows created with Visual Studio with VseWSS installed, if the “Auto Associate” property is set to False for the project, then you will use the same steps as above to create a new history list. You can also run through the “New Office SharePoint Workflow” wizard by selecting the History List property and click the … button.
New Office SharePoint Workflow wizard
Workflows created in SharePoint Designer will write to only the Workflow History list. There is not any way to change this. So in this case, the only thing that can be done to keep the item count of the Workflow History list down is to trim the list in datasheet view manually. Deleting history items that are older than 60 days would be a good start, or create a program that will delete the items that can be scheduled with Task Scheduler.
So the take away from all of this is that you want to keep each workflow history list under the 2000 item limit so that the performance of workflow and the Workflow Auto Cleanup timer job stay within acceptable levels.
Take the following example. You have a SharePoint web application with both NTLM and anonymous authentication enabled. You leave the default permissions on the root site collection for the web application. You create a new subweb under the root site collection and enable anonymous access. The site hierarchy would be similar to the one below:
· Web Application
·Site collection with NTLM and Anonymous
·Site with Anonymous
While it may be possible to shoehorn this configuration and get it working; I don’t advise doing so, a host of issues may occur depending on what type of site template and configuration the sites contain. One being, if the root site collection site is a publishing portal, the Anon subweb will not be able to search with the root site’s search center. Another being issues with the master page and style library files for the root site collection are not usually set to allow anonymous if the root site is NTLM. This can cause HTTP 401 Unauthorized issues on the anonymous site.
This tip is for Microsoft Office SharePoint Server 2007.
Just like a teenager, the Office Server Web Services need checking on from time to time as they both tend to get into trouble when left to their own devices.
So what are the Office Server Web Services? When would you want to check them and what can you do to make sure they are working correctly?
The Office Server Web Services at the most basic level is an IIS web application that contains the .asmx web service files used by SharePoint for some of the functions in the SSP like search and Excel Services. You will want to check the web services when your SharePoint servers have entries like the following in the ULS diagnostic logs or in an application Event Log ID #6482; “The underlying connection was closed: An unexpected error occurred on a send.”. Other effects that can occur when the web services are not working are profile information is not being updated in SharePoint web applications/sites and search gatherer not indexing.
So the quick way to check the health of the web services is to browse the following URLs in Internet Explorer from each SharePoint server in the farm. If you can see a list of web service methods like the images below the URLs, then the web services are working. Note that on the SSL URLs you will have to click continue on the IE window to ignore the untrusted certificate.
http://<APPSERVER>:56737/SearchAdmin.asmx
https://<APPSERVER>:56738/SearchAdmin.asmx
http://<APPSERVER>:56737/<SSPTITLE>/Search/SearchAdmin.asmx
https://<APPSERVER>:56738/<SSPTITLE>/Search/SearchAdmin.asmx
This tip is for Windows SharePoint Services 2.0/3.0, SharePoint Portal Server 2003 and Microsoft Office SharePoint Server 2007 and SharePoint Foundation 2010 and SharePoint Server 2010.
As a SharePoint administrator there will be many times that you will find youself running the SharePoint command line administration tool Stsadm.exe and other programs in the %CommonProgramFiles%\Microsoft Shared\Web Server Extensions\<version>\BIN folder. Please, save your hands from RSI and update your PATH environment varible to include the location above. This will allow you to type just the command of Stsadm.exe from any location in a command prompt instead of typing out the change directory each time.
To update the PATH variable, go to Start->Run. Type sysdm.cpl. In the System Properties dialog click the Environment Variables... button. Under System variables, click PATH and then the Edit... button. If there is not a semi-colon at the end of the path enter one and then enter the path to the SharePoint BIN folder for the version of SharePoint installed on the server. For MOSS 2007 the path would look like the following:
%CommonProgramFiles%\Microsoft Shared\Web Server Extensions\12\BIN
Now rejoice in your new found pain free SharePoint administration.
The Sysinternals web site was created in 1996 by Mark Russinovich and Bryce Cogswell to host their advanced system utilities and technical information. Microsoft acquired Sysinternals in July, 2006. These tools are not loaded on Windows operating systems by default. They can be downloaded from http://technet.microsoft.com/en-us/sysinternals/default.aspx or http://live.sysinternals.com/. The http://live.sysinternals.com site has the latest public builds of the tools and is more up to date than the TechNet site. All examples are based on at least Process Explorer version 11.31.0.0 and Process Monitor 2.3.0.0. There is no installer/uninstall for these tools. The first time the programs are run, the EULA will display, after accepting the EULA the first time, this screen should not reappear. The tools can also be run straight from the web using the following format: http://live.sysinternals.com/Procexp.exe and http://live.sysinternals.com/ProcMon.exe . You can also map a drive letter right to the public location by running SUBST drive: \\live.sysinternals.com\tools although this may not work when a proxy server is set.
Process Monitor (Process Monitor.exe)
Monitors File, Registry, network and process activity by process. Collects data when running and can be filtered to track down process issues. Process Monitor replaces FileMon and RegMon, except for back level operating systems.
Process Monitor runs on Windows 2000 SP4, XP SP2, Vista, 2003, 2008 and Windows 7 32 bit and 64 bit.
Process Monitor does not run on Windows 2000 pre SP4 and may not always be able to be used to troubleshoot SharePoint Portal Server 2001. You must use Filemon and Regmon to monitor Windows 2000 and SharePoint Portal Server 2001; if Process Monitor does not run on you server. These can be downloaded from the Sysinternals TechNet site, http://technet.microsoft.com/en-us/sysinternals/default.aspx.
Some of the command-line switches are below. Refer to the Procmon.chm file for a complete list.
/Openlog <saved PML log file>
Directs Process Monitor to open and load the specified log file.
/Noconnect
When this flag is present Process Monitor does not automatically start logging activity.
/AcceptEula
Automatically accepts the license and bypasses the EULA dialog.
/Quiet
Don't confirm filter settings on startup.
/Run32
Use this switch to run the 32-bit version of Process Monitor on 64-bit Windows to open logs generated on 32-bit systems
*Note: If you are going to open a .PML log file from a 32 bit computer on a 64 bit Windows computer you will need to enter the /Run32 switch to view the log, or you will get the following error when trying to open the log file.
Process Monitor User Interface
By default, Process Monitor uses virtual memory to store captured data. Use the Backing Files dialog, which you access from the File menu, to configure Process Monitor to store captured data in files on disk. Enabling this option has Process Monitor log data to the disk in its native PML format as it captures it. If running a long capture you can set the logging to a backing file, File->Backing Files...
Profiling
This event class can be enabled from the Options menu. When active, Process Monitor scans all the active threads in the system and generates a profiling even for each one that records the kernel and user CPU time consumed, as well as the number of context switches executed, by the thread since its previous profiling event. Note: the System process is not included in profiling.
Profiling Events
Use this menu entry to open the thread profiling configuration dialog, where you enable thread profiling and the rate at which thread profiling events generate. When thread profiling is enabled, Process Monitor captures thread stack traces and CPU utilization that you can use to identify the source of CPU-related performance issues.
Process Explorer (Process Explorer.exe)
Monitors data on running processes in real time, does not capture historical data, except for CPU, Disk, Page File and Network activity graphs. The data in the graphs cannot be saved.
Process Explorer can be used to view detailed information on the currently running process on a system; including which images or executable code is loaded in memory for a process and the handles to the registry, file system and other types of handles. You can also change the priority; suspend and kill processes and even certain threads of a process.
Some of the command-line switches are below. Refer to the Procexp.chm file for a complete list.
/e
Request UAC
/t
Start minimized
/p:[r|h|l]
Run at a different priority than normal
/ s:[PID]
Select the specified process
Troubleshooting SharePoint with Sysinternals Process Tools
These are some of the SharePoint processes that can be monitored.
· Windows SharePoint Services Timer (OWSTIMER.EXE)
· Internet Information Server application pools (w3wp.exe)
· Office SharePoint Server Search parent process (mssearch.exe)
· SharePoint Search crawl process (msdmn.exe)
Although there are other processes that may affect SharePoint, these are the major processes that should be monitored when using Process Monitor and Process Explorer Sysinternals tools. The first step in troubleshooting with Process Monitor or Process Explorer is to identify if the issue can be captured by monitoring process activity. Here is a list of some of the SharePoint processes that Process Monitor and Process Explorer can be useful to monitor.
· Badly performing and errors in application pools, especially when other web servers in the same farm are running normally.
· Security event login failures to local resources
· Timer job errors
· Crawling local file system content sources
· Installation errors
Here is a list of SharePoint issues that may not be best to monitor with Process Monitor or Process Explorer.
· Crawling SharePoint Sites or external sources
· SQL login failures
· SQL connection issues
Performance issues where detailed historical performance information is needed, use Perfmon.exe instead.
In this lab you will learn to view properties of the SharePoint web application processes, w3wp.exe. By viewing the properties of the running w3wp.exe application pool process you can determine possible issues with permissions, ASP.Net setup, performance and application pools hangs/leaks.
*Important - Do not follow the steps in any of these examples on production or live test systems as you will be running commands that can and will break functionality of the server. Only attempt these examples on computers that are expendable!
For all of the Process Explorer labs you will want to enable the ‘Command Line’ and ‘User Name’ columns in the main window view. To do this right-click the columns where it says Name, PID, CPU, Description and Company and select ‘Select Columns’. Check the columns you want to add to the main window.
On your SharePoint web front end server run IISRESET. After this command completes, start Process Explorer. Look for a process with the path of svchost.exe -k iissvcs. There should not be any child process under this svchost.exe process yet, Figure PE1-1.
Figure PE1-1: Application pool process after restarting IIS
Browse to your SharePoint site to produce some application pool events. After your site is fully rendered on your browser, look back to Process Explorer. You should see at least 1 w3wp.exe child process under the svchost.exe process. If you have the Command Line column in the view you will see which application pool process was started when you browsed to the site. In this case it was the Central Administration site, Figure PE1-2.
Figure PE1-2: Application pool process after browsing to site first time
You will notice that the svchost.exe parent process of our application pool process as seen with the blue line in Figure PE1-2. The application pool process identity can be determined if the ‘User Name’ column is enabled in the main window. Also notice the w3wp.exe is pink and that the w3wp.exe process is yellow. This process color coding can help in troubleshooting errors with SharePoint sites being unavailable as we will see in the next exercise.
On your SharePoint web front end server run Internet Information Services Manager, (%SystemRoot%\system32\inetsrv\iis.msc). Select the Web Sites node in the left-hand side TreeView, as seen in Figure PE1-3. Make a note of the web application identifier of your SharePoint web application.
Figure PE1-3: Finding the IIS site identifier
Next, open a command prompt and type the following command:
CD %SYSTEMROOT%\Microsoft.Net\Framework\v2.0.50727
On 64-bit systems use the following path:
CD %SYSTEMROOT%\Microsoft.Net\Framework64\v2.0.50727
Type the following command. Remember to change the web application ID number to your specific value from Figure PE1-3:
aspnet_regiis.exe -kn w3svc/221506137/ROOT
You should see similar output from the aspnet_regiis.exe command if ASP.Net was uninstalled successfully from the web application:
Start removing any version of ASP.NET DLL at w3svc/221506137/ROOT.
Finished removing any version of ASP.NET DLL at w3svc/221506137/ROOT.
Try to browse to your SharePoint site. You will receive an HTTP 404 error, the page cannot be found. Why is this? The web application and application pool are running in IIS Manager. The SharePoint database is running. To find out why you received the 404 response open the Process Explorer window and look for the w3wp.exe process with the application pool name in the Command Line column. You will see that the background color of the process is white, not yellow like previously. This is because the only code running in the w3wp.exe process of the application pool is unmanaged code; basically just the skeleton of the w3wp.exe process. Remember that in WSS 3.0 and Microsoft Office SharePoint Server 2007 one of the requirements is .Net 2.0 Framework. Since we uninstalled all versions of .Net for the web application in Figure PE1-3 SharePoint cannot run on that web application. See Figures PE1-4 and PE1-5 for examples of broken w3wp.exe application pool processes.
Figure PE1-4: Application pool managed code (.Net) process type
Figure PE1-5: Application pool unmanaged code (Win32) process type
To re-enable .Net code on the web application in Figure PE1-3, run the following command from the command prompt. Remember to change the web application ID number to your specific value from Figure PE1-3:
aspnet_regiis.exe -sn w3svc/221506137/ROOT
Browse to your SharePoint site. You should not see the 404 error and the site should be visible now. Go back to the Process Explorer window and you will see the w3wp.exe process background is yellow.
Remember that a lot of information can be gleaned from just the main Process Explorer window if the pertinent columns are enabled. You can verify application pool identities; verify .Net is running in your application pools, CPU usage, number of threads for each process, how many bytes were written and read for each process, even the time the process started.
Scenario
In this lab you will learn how to configure Process Monitor for effective SharePoint troubleshooting. You will also track down a generic Windows SharePoint Services 3 install error.
*Important - Do not follow the steps in any of these labs on production systems as you will be running commands that can and will break functionality of the lab server. Only attempt these labs on computers that are expendable!
Tasks
*Note - It is recommended that if you are running this lab exercise in a virtual environment that you take a snapshot before you begin as this exercise will leave your install unable to run SharePoint or reinstall without a significant amount work.
Login to your server using an administrative account that will be used to install WSS 3.0. Then run regedit.exe from Start->Run.
Navigate to registry key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Shared Tools. Right-click the Shared Tools and select ‘Permissions’. Add the currently logged in account and select Deny/Full Control. The WSS 3.0 installer needs access to this registry key to create the registry entries for Web Server Extensions, so the install will fail as seen in Figure PM1-1. The error you will receive is not very descriptive, so if you didn’t know that your account had deny access to the ‘Shared Tools’ registry key it would be difficult to troubleshoot this error without Process Monitor. With Process Monitor tracking down this error will be straightforward.
Start Process Monitor capturing all activity by enabling the buttons in the red box in Figure PM1-1. Make sure the filter is reset to the defaults before capturing.
Figure PM1-1: Enabling all activity tracing
Start the WSS 3.0 setup program. At the end of the install process, the installer will fail with the error in Figure PM1-2.
Figure PM1-2: WSS 3.0 Installation Error
Click ‘Don’t Send’ on the error; stop capturing activity in Process Monitor.
When capturing the WSS 3.0 install Process Monitor captured over 1.2 million events, Figure PM1-3 red box. While this may seem like a lot of events to search through to find the cause of the install error you will see that by using filtering we can quickly find the cause of the install error.
The capture in Figure PM1-3 was using the page file or virtual memory to write events to while running. This can be changed to write to a backing file so that virtual memory is not used up while taking a long capture. To change where the events are captured to go to the menu->File->Backing File; from there you can change the location to a file. This is useful if you know that virtual memory on the server is low or you want to take a long capture and do not want filling of the pagefile to impact the performance of virtual memory.
To start narrowing down the events to the event that caused the installation failure; go to the menu Filter->Filter. In the filter window add 2 filters. The first is Process Name is setup.exe then include. The second is Process Name is msiexec.exe then include. Click OK.
Figure PM1-3: Filtering
After these filters are loaded you will see only events for the WSS 3.0 install. There is another way to filter events also from the main window based on the information shown for an event. Right-click on one of the events where the Result column equals ‘SUCCESS’ and click Exclude ‘SUCCESS’. Since most programs do not error on successful actions, this will cut down the events even further. The right-click is column sensitive, so that if the PID column were right-clicked, then it will list include and exclude for the PID number and so on.
Figure PM1-4: Right-click filtering
Enabling these filters should have cut down the pertinent events, but you may still have thousands of events to look through. A faster way to track down a possible cause of the error is to add a filter for the following; Result is ACCESS DENIED then include, as seen in Figure PM1-5. Another good filter for finding possible causes of errors is; Result contains NOT then include. This will show events like Path Not Found and Name Not Found. Indicating there may be a problem finding a file or registry value that is needed.
Figure PM1-5: Filtering only ACCESS DENIED
After the ‘ACCESS DENIED’ filter is enabled, there should not be very many events listed. The events should be similar to the ones in Figure PM1-6. From these events you can clearly see that there was an access problem opening the registry key HKLM\Software\Microsoft\Shared Tools\Web Server Extensions.
Figure PM1-6: Cause of failed WSS 3.0 install
Review
One of the most powerful aspects of Process Monitor is the filter capabilities. By mastering filtering, you can quickly track down causes of errors related to failed file access, registry access and network activity. For an advanced troubleshooting video on Process Monitor please see Advanced Windows Troubleshooting with Sysinternalss Process Monitor.
In this lab you will learn how to track down long running threads in an application pool to determine performance related issues.
On your SharePoint web front end server run IISRESET. After this command completes, start Process Monitor capturing all activity by enabling the buttons in the red box in Figure PM2-1. Make sure the filter is reset to the defaults before capturing.
Figure PM2-1: Enabling all activity tracing
Browse to your SharePoint site to produce some application pool events. After your site is fully rendered on your browser, stop Process Monitor in the server. The browser page render should have taken more time than usual to render because the application pool was just restarted when you ran the IISRESET command.
In Process Monitor on the server you will see many events in the main window since the default filter was used to capture the events on the server. In this lab we are only going to concern ourselves with application pool performance, so we are going to filter out all other processes besides the application pool events.
Go to the Process Monitor menu; Filter->Filter to bring up the filter window in Figure PM2-2. Select Process Name, is, w3wp.exe then include and click Add. Click OK.
Figure PM2-2: Filtering only application pools
After filtering the events you will see only application pool events. Use the columns discussed in Process Monitor Lab1 so that you can view the command line column information and process ID. If you have multiple application pools running you may see different process IDs (PIDs) for the w3wp.exe processes. In the command line column you will see a reference to the application pool name, as seen in Figure PM2-3. Note that this Process Monitor capture was done on Windows 2008 Server, if using Windows 2003 you will see a different command line format, but the application pool name will be listed in the command. Make a note of the PID for the application pool you want to view more information on; in this case the PID is 5828 for the application pool ‘SharePoint - 80’.
Figure PM2-3: Events filter by application pools
To view more information on the application pool and what it was doing while the capture was taken go to the Process Monitor menu; Tools->Stack Summary… as seen in Figure PM2-4.
Note that viewing the stack works best when the Debugging Tools for Windows is loaded on the system and the symbol path is set via Options->Configure Symbols. The path to the DbgHelp.dll will need to be set to the file from the Debugging Tools. The public symbol server can be used for the symbol path; srv*http://msdl.microsoft.com/download/symbols. See http://www.microsoft.com/whdc/devtools/debugging for more information on downloading and installing the Debugging Tools.
Figure PM2-4: Viewing the thread stack of an application pool
In the Stack Summary screen you will see on the left-hand side a treeview with 1 node, All. This is the node for all processes that were captured when the trace was running along with other information on the total time and CPU time for all processes.
Expand the All node and you will see a list of w3wp.exe processes since this is what our current filter is set to view. Find the w3wp.exe node that has the PID number from Figure PM2-3 in the Module or Path column as seen in Figure PM2-5.
Figure PM2-5: Application pool thread information
After finding the w3wp.exe process for the application pool; expand the w3wp.exe process. You will see a list of the treads that were executed in the w3wp.exe process while the trace was running. To find the thread that was using the most time focus on the % Time column. You can also see how many Process Monitor events where captured for each process and tread, Figure PM2-6 red 1. Expand the thread listed under the w3wp.exe process with the highest % Time value. Keep expanding as seen in Figure PM2-6 red#2; if you expand a function and more than 1 function is listed below the current, expand the one with the highest % Time value as seen in Figure PM2-7. The function with address 0x5a5 is higher than the function with 0x374. While expanding the tread functions look for function names that may give a clue to what the function does. In this case I stopped expanding when I found a function called: VdbThemeManager::loadTheme. Just from the name you can hypothesize that this function looks up the selected theme for the site and loads the theme into the application pool memory to be used to construct the HTML page. In this case the % Time is not that great, Figure PM2-6 red 4&5, but if it was high you could focus troubleshooting on the theme and theme files. You can also see which executable file, Dll or exe the function was running from, Figure PM2-6 red 3. In this case it is stswel.dll, one of the main SharePoint Dlls.
Figure PM2-6: Unrolling the stack
Figure PM2-7: Selecting a high % Time fork
The main advantage of using Process Monitor to track down poor performing processes is that you can take a Process Monitor trace when the issue is happening live and save the file to view at any time.
These same troubleshooting steps can be taken for any poorly performing processes including the SharePoint timer service, OWSTIMER.EXE, SharePoint search service, mssearch.exe and any other process running on the computer that may be contributing to performance issues.
In this lab you will use both of the tools you have seen previously, plus some other Sysinternals tools to effectively troubleshoot an unexpected error in SharePoint.
Make sure you have at least 1 SharePoint web application created on your SharePoint farm/server. Make a note of the application pool identity on this web application. If you followed the Basic install, then the identity should be NETWORK SERVICE. On your SharePoint web front end server run IISRESET /STOP. After this command completes, navigate to %SYSTEMROOT%\Microsoft.Net\Framework\v2.0.50727. Right-click on the “Temporary ASP.NET Files” folder and select Properties. Select the Security tab. Click the Edit button to edit the permissions of this folder. Add the application pool account if it is not listed. Select the “Full control” “Deny” checkbox for this account. This will make the application pool account unable to write files to this location breaking any application pool running .Net 2.0 code under the user account, including SharePoint.
Run IISRESET /START and browse to the web application. You will see the following page.
Figure 5-1: SharePoint site error
Run Process Explorer to see if there is any information it provides that can help in identifying this error. After Process Explorer opens navigate to the w3wp.exe process for the application pool. Remember to enable the User Name and Command Line columns by right-clicking the column bar and selecting, Select Columns…
You should see something similar to Figure 5-2. Figure 5-2: W3wp.exe process information
From this we can see that the w3wp.exe process is yellow, which means it is running .Net code. To check the version of .Net code running make sure the lower pane is visible by selecting View->Show Lower Pane in the Process Explorer menu. Also make sure the lower pane is showing DLLs by selecting View->Lower Pane View->DLLs in the Process Explorer window. Sort the lower pane by Name and find aspnet_filter.dll make a note of the Version shown, if it starts with 2.00.50727, then the application pool is running .Net 2.0 code.
See Figure 5-3 for more information on selecting the DLL lower pane view and finding the DLL version information.
Figure 5-3: View DLL information for a process
In this case Process Explorer is telling us that things look to be working as far as the basic process is concerned.
Next, let’s open run Process Monitor using a filter on the w3wp.exe process and capture events while refreshing the SharePoint page. Right-click on one of the SUCCESS results and select Exclude ‘SUCCESS’. You will see many BUFFER OVERFLOW, NO SUCH FILE, NAME NOT FOUND and many more results displayed. Remember that the result that causes most issues is ‘ACCESS DENIED’, so create a filter to include ‘ACCESS DENIED’. You should see at least 1 access denied event as seen in Figure 5-4.
Figure 5-4: Access Denied Event
Since an access denied event occurred the next step will be to check the permissions on the folder structure specified in the Path column. Another Sysinternals tool, AccessEnum.exe can be used to view the permissions of a file or folders and the child files and folders that have different permissions.
The AccessEnum.exe tool can be downloaded from the same sites as Process Explorer and Process Monitor listed at the beginning of this article.
Run AccessEnum.exe on the SharePoint server and choose the folder from the Process Monitor Path column; C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files. The default view when the scan is run is to only show files ad folders that have more restrictive permissions than the selected folder. This can be changed from the menu-Options->File display options to include any differing permissions from the selected folder, Figure 5-5.
Figure 5-5: AccessEnum.exe, Changing Display Options
Click the scan button. You will see, depending on the display option and permissions, a few or many paths and their permissions. In the Deny column you should see the application pool user account listed on the selected folder as in Figure 5-6 in the red box.
Figure 5-6: AccessEnum.exe, Access Deny
After the scan is complete the permissions can be saved in text format to look for differences between the current permissions and known good permissions, as seen in Figure 5-6 green box.
The root cause of the SharePoint failure has been determined. Remove the Deny permission for the application pool and run IISRESET. When you browse to the web site it will compile and there should be no errors.
While Process Explorer and Process Monitor may not always be the best tool to use for every troubleshooting situation they can be an invaluable resource, especially when there are not clear Event Logs or ULS logs pointing to the cause, or when an issue only effects a certain server and not others in a farm. In SharePoint if you are relatively sure the issue is not database related and that there may be a problem with how the server is working locally on the file system or registry then it can’t hurt to load up Process Explorer and/or Process Monitor to see if the problem can be tracked down by these great tools.
SharePoint Tip of the Day will be back Monday January 4, 2010.
Some good tips on the way, so be on the lookout next year.
Happy Holidays!
Warning, this may create a space-time paradox, but the risk is worth it because of the payoff!
Also, this tip pertains to all versions of SharePoint and no versions of SharePoint.
I recently seen that the WSS 3.0 and MOSS 2007 Technical Libraries were updated and available in .chm format from the Microsoft downloads site, here are the links.
Windows SharePoint Services 3.0 Technical Library in Compiled Help format
http://www.microsoft.com/downloads/details.aspx?FamilyID=c9d6c8c5-8a62-4961-8c1b-df08b667b1c4&DisplayLang=en
Office SharePoint Server 2007 Technical Library in Compiled Help format
http://www.microsoft.com/downloads/details.aspx?FamilyID=ba006584-711d-4ce7-9e1f-181aedf6434a&DisplayLang=en
This reminded me of one of my tricks that I use to find SharePoint information quickly and keep things organized with the flood of SharePoint articles, white papers and books. I copy these documents to a file share on my home network and have a Search Server 2008 Express server index the documents. This way, when I need information on some particular feature of SharePoint, all I have to do is a search. Of course to index all of the documents in various formats the appropriate iFilter will need to be installed. A quick search with Bing can help track down some free, Shareware, open source and pay options for various iFilters.
This tip is for Windows SharePoint Services 3.0 and Microsoft Office SharePoint Server 2007.
It happens, server configurations change. If the default file or registry locations for SharePoint change permissions issues can occur on the farm. Luckily there is an easy way to set them back to defaults. Run a command prompt using the SharePoint farm administration account and change directory to the SharePoint binary location. Usually this is located at %CommonProgramFiles%\Microsoft Shared\Web Server Extensions\12\BIN. It is best to run a command to check whether the farm or SharePoint server has a pending upgrade as if your run the next command and there is a pending upgrade, Psconfig.exe will try to finish it, which you may not want to do for various reasons. So first run the Stsadm.exe -o localupgradestatus from tip #1. Then Run psconfig.exe -cmd secureresources to reset the SharePoint file system and registry security settings back to defaults.
Keep in mind if you have changed any of the location of logs/indexes/etc.., that these locations will not be reset. For a list of needed security permissions for WSS 3.0 and MOSS 2007 see the following; http://technet.microsoft.com/en-us/library/cc721662.aspx.