Stefan Goßner

How to programmatically retrieve the root site of a Variation

The Admin UI of the site collection provides easy access to the variation settings. But the object model does not provide easy access to these settings. There are many scenarios where it would be (e.g.) very interesting to know which site is the root site of the variation.

The solution for this problem is Reflection. Using Reflection it is possible to access the VariationSettings object which provides easy access to this information.

Here is some sample code:

Pimp My Content Deployment Job

As a follow up to my article series about the content deployment and migration API here are some tips on how you can "fine-tune" the out of the box content deployment jobs. This will include information about how adjust settings in Content Deployment Paths and Jobs and also about how to adjust content deployment global settings which are not available through the UI.

This article demonstrates

• How to configure the FileMaxSize property for a Content Deployment Job
• How to configure the Timeout and the Polling Interval for a Content Deployment Job
• How to configure a Content Deployment Job to keep the temporary Files to simplify troubleshooting
• How to disable Compression for a Content Deployment Job
• How to enable Event Receivers during Import if required
• How to configure a Content Deployment Job to automatically retry after a failure has happend
  

Content Deployment Settings

On the Operations Tab of the Central Administration you can find a link to the Content Deployment Settings page. Here you can configure the following options:

Accept Content Deployment Jobs

This option allows to decide whether this farm can be used as the target of an content deployment operations or not. Per default this option is disabled.

Import Server / Export Server

These options allows to decide which server in the current farm should act as the import and/or export server. As an import operation can impact the performance of a server significantly due - especially the memory consumption can be pretty high - you might want to configure a dedicated server to perform the import operation. Export is usually affecting the server performance less but that might depend on the specific content being exported.

Be aware that the server acting as the Import and Export server has to host an instance of the Central Administration Website. See section 1 of the following article for details.

Connection Security

Content Deployment first exports all content into one or more cab files and afterwards uploads the content to the destination server using http upload. In case that this upload goes over the internet or another somehow unsecure line you might want to use SSL to encrypt the uploaded content. This can be done using this option.

Temporary Files

Content Deployment requires a significant amount of disk space - as well on the exporting and on the importing server - as the content is first exported and then compressed. So as well the flat content and the cab files require space on your had drive. This option allows to configure the location for the compressed cab files. To see how to configure the location for the uncompressed content see section 2 of the following article for details.

Reporting

Per default MOSS keeps the last 20 reports for each content deployment job. In case you need to keep more or less you have the option to adjust the number of reports using this setting.

Ok, so far for the options that can be configured in the UI. But there are a couple more very intersting options which might be worth to look at which are NOT available through the UI. These can be configured using the object model using the ContentDeploymentConfiguration object. The documentation for this class can be found on MSDN.

The options above can also be configured using this class. But also a couple more which I will explain now.

Hidden Content Deployment Settings

MaxFileSize

This option allows to define the maximum size of each generated cab file. Per default this size is 10 MB. When creating your own applications to deploy content using the Content Deployment and Migration API you have full control over this parameter but the UI for the Content Deployment jobs does not allow you to change this parameter.

RemoteTimeout

During the import the exporting server that is hosting the content deployment job definition will poll the destination server in specific intervals to get the status of the import operation. The import will go through different phases like extracting and then performing the import. This timeout actually defines the maximum time that is allowed between changes to the status report. During the actual import the report changes quite often. But there are some phases - e.g. the decompression - which can take some time. If this time exceeds this remote timeout value then a timeout is reported back to the exporting machine. You might have seen this status in the content deployment jobs. Such a timeout does not mean that the import has failed! It just means that no status update did arrive for the configured time. The default here is 600 seconds which means 10 minutes.

In case that you have a large content deployment running with several GB of content it can happen that the decompression phase takes longer than 600 seconds - and then you will the a timeout. Adjusting this parameter allows you to avoid such timeout messages.

RemotePollingInterval

This setting is tightly bound to the previous. It defines how often the exporting server checks for status changes on the remote server. The default for this setting is 10 seconds. Usually it should not be required to change this settings as a SOAP call to the importing server every 10 seconds to get a status update should not add measurable load to this server.

As mentioned before to adjust these hidden properties it is required to write some custom code. Also be aware that these settings are - like the settings you can configure in the content deployment settings page - global settings and will affect all content deployment jobs.

Here is some sample code which allows you to configure the RemoteTimeout to one hour. It would be required to run this on the exporting farm if required:

using System;
using Microsoft.SharePoint.Publishing.Administration;  

namespace StefanG.Tools
{
    class AdjustContentDeploymentDeploymentSettings 
    {
        static void Main(string[] args)
        {
            ContentDeploymentConfiguration config = ContentDeploymentConfiguration.GetInstance();
            config.RemoteTimeout = 3600;
            config.Update();
        }
    }
}

Content Deployment Paths and Jobs

MOSS stores informations about content deployment paths and jobs in two hidden lists in the administration website. You can see all the details using the following URLs:

• http://url-to-centraladmin/Lists/Content%20Deployment%20Paths
• http://url-to-centraladmin/Lists/Content%20Deployment%20Jobs

When browsing to these urls you will find a list item for each configured path/job.

Attention: It is highly recommended not to modify any settings in this list directly! There are other supported methods available to modify most of the settings you can see here - even if they are not available in the UI where you configure the content deployment path and job.

I would like to highlight the following settings which are visible in the property pages but which are not available through the official UI.

Hidden Content Deployment Path properties

KeepTemporaryFiles

Possible Values: 0 = Never, 1 = Always, 2 = Failure, Default Value: Never.

When performing a content deployment operation the exporting server first exports the content into a temporary directory on the disk and then compresses the data. After the deployment is done the temporary files are cleaned up - more or less. Actually it will only remove the first cab file and not all of them, which makes the remaining content unusable but it requires that you still have to clean up the other files manually to preserve disk space. So why would it be interesting to change this setting? This is interesting when it comes to troubleshooting content deployment problems. Enabling this option allows you to keep and inspect the generated cab files to help you to identify what is being deployed. We at Microsoft Support often use this option when working on support cases.

To modify this option please use the following STSADM command:

STSADM -o editcontentdeploymentpath -pathname <pathname> -keeptemporaryfiles Never|Always|Failure

(in case your pathname contains blanks please specify it with double quotes)

EnableEventReceivers

Possible Values: 0 = No, 1 = Yes, Default Value: No.

This property allows you to control whether event receivers are allowed to fire during the import or not. The default is that event receivers are disabled during import. This configures the SPImportSettings.SuppressAfterEvents setting for the remote import job.

To modify this option please use the following STSADM command:

STSADM -o editcontentdeploymentpath -pathname <pathname> -enableeventreceivers yes | no

EnableCompression

Possible Values: 0 = No, 1 = Yes, Default Value: Yes

This property allows to control whether the data being deployed is being compressed during the deployment or if the uncompressed data should be sent to the destination server. Per default this setting is enabled to provide acceptable performance even when uploading the content to a server in a different subsidiary as in this situation the overhead to perform the compression is smaller than the additional transport time it would take to upload the uncompressed data to a remote server. But in an intranet or when deploying content between two site collections on the same server this might be different. Here the compress and uncompress time could cause a significant overhead in the performance of the content deployment jobs. So here it might make sense to disable the compression. This setting configures the SPDeploymentSettings.FileCompression property for export and import.

Attention: before disabling this option please check the version of the Microsoft.SharePoint.Publishing.dll on your disk. A known problem with disabling this options was fixed in build 12.0000.6315.5000. If your Microsoft.SharePoint.Publishing.dll has a lower build number you cannot disable this option as it would cause your content deployment job to fail! The first hotfixes containing the solution for this problem are 952698 (WSS fix) and 952704 (MOSS fix). You should ensure to install both fixes on your server.

To modify this option please use the following STSADM command:

STSADM -o editcontentdeploymentpath -pathname <pathname> -enablecompression yes | no

Hidden Content Deployment Job properties

Actually there are only two options which I would like to highlight here as these are really cool new additions being added in one of the latest hotfix packages. You will not find these options if your Microsoft.SharePoint.Publishing.dll has a build number lower than 12.0000.6315.5000. If your build number is lower and you need this feature, please request the following two hotfixes from Microsoft Support: 952698 (WSS fix) and 952704 (MOSS fix).

RetryOption

Possible Values: 0 = None, 1 = SkipFailedWebs, 2 = SimpleRetry, Default Value: None.

This option has been implemented to allow parallel execution of content deployment jobs. In the past we had a couple of cases where customers ran into problems when executing content deployment jobs in parallel. In such a situation the deployment job was likely to fail due to update conflicts caused by the parallel running content deployment jobs. To overcome this limitation the retry feature has been implemented which allows to redo a deployment if the deployment failed. This option can either retry without the failing sites (SPWeb objects) that caused the problem will not be exported on the next attempt or can try to export with all sites again. This allows to use content deployment in a limited manner in situations where problems in a specific sites persistently would cause content deployment jobs to fail.

ExportRetries

Possible Values: 1-999. A small number (3-5) is recommended.

This option defines how many times the deployment retry is attempted before the job finally ends with Failure. To ensure that the deployment jobs does not run forever on a persistent problem you should avoid configuring this to a number higher than 5.

As these options are pretty new and have been introduced through a hotfix there is no UI and no STSADM command to configure them. To get this option active you need to add a modification to the web.config file of the central administration website.

The following changes have to be applied:

Locate the following section group inside the web.config:

Inside the following sectionGroup

<configuration>
  <configSections>
    <sectionGroup name="SharePoint">

add the following element:

        <section name="ExportRetrySettings" 
                      type="Microsoft.SharePoint.Publishing.Administration.RetrySectionHandler, 
                                Microsoft.SharePoint.Publishing, Version=12.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" />

Inside the following element

<configuration>
  <SharePoint>

add the following element:

<ExportRetrySettings DefaultOption="<option>" DefaultRetries="<number-of-retries>">
      <Job Name="<Name-of-Deployment-Job1>" Option="<option>" Retries="<number-of-retries>"/> 
      <Job Name="<Name-of-Deployment-Job2>" Option="<option>" Retries="<number-of-retries>"/>
      ...
</ExportRetrySettings>

<option> can be one of the following: None, SkipFailedWebs, SimpleRetry
<number-of-retries> can be an integer between 1 and 999

If you don't add any <Job...> elements, then all new jobs will get the values configured as DefaultOption and DefaultRetries. Using the <Job...> element you can configure different jobs with different values.

After you have added the above listed configuration to the web.config of the Central Administration you have to open the content deployment job definition once in the central administration and save it again. During this save the configured options will be silently added to the content deployment job. New jobs created after you changed the web.config will automatically get the configured values.

You can verify this easily using the URL listed above which allows you to verify the configured settings for each of the content deployment jobs.

Troubleshooting SPSite/SPWeb leaks in WSS v3 and MOSS 2007

In an earlier article I have discussed that all SPSite and SPWeb (and potentially also PublishingWeb) objects need to be properly disposed to avoid memory pressure situations. Roger Lamb has a nice summary of coding patterns which will actually cause such leaking objects and also provides information about how such code would have to be adjusted to avoid such leaks.

In this article I will provide some details about how a sharepoint administrator is able to isolate if the site contains custom code which does not properly dispose these objects and also how to isolate the components that are leaking these objects.

Overview 

As discussed in my earlier article each SPWeb and SPSite object holds a reference to an SPRequest object which holds a reference to a SharePoint COM object that is responsible to communicate with the backend SQL server. The dispose of the SPWeb and SPSite object is necessary to ensure that the SPRequest object will release the COM object and also all the memory allocated by this COM object is released.

In SharePoint v2 the only way to identify such leaking objects was to analyze a memory dump - which usually required that a support case with Microsoft has to be opened to identify the code that is not correctly disposing the SPWeb or SPSite objects.

SharePoint v3 on the other hand contains code which monitors correct disposal of SPRequest objects (and therefore correct disposal of SPWeb and SPSite objects). This code has two different features:

1. It monitors how many SPRequest objects are open in parallel on all threads.
2. It checks if SPRequest objects still exist at the end of the thread that created it

I will now discuss how these build in features can be used to identify how expensive (in regards of SPRequest allocations) your site design is and also if the code contains any controls which don not properly dispose SPWeb and SPSite objects.

How many SPRequest objects are allocated by a single request to my site?

Usually you would think that if you don't add any custom code to your page that only one SPSite and one SPWeb object should be required (the site collection and the site the page lives in) but this is not correct. The reason is that all the master pages contain three navigation controls: The Top navigation, the Breadcrumb control and the Left Navigation control.

Per default these navigation controls are bound to one of the Site Map Providers shipped with SharePoint. To show the elements in the navigation control the control queries the site map provider which will now enumerate the sites in the sharepoint database to retrieve the title and the URL to be displayed on the navigation control. Each of the enumerated sites will be instantiated during the enumeration to retrieve these properties. That means if you have a navigation control configured to enumerate 3 levels of sites and on each level you have 20 sites that you will end up with more than 400 SPRequest objects. So you need to be very careful with how to configure your navigation controls and also how many sites to create on each level!

A valid question now could be: why does the site map provider not dispose each of the objects directly after it retrieved the URL and the Title? That should reduce the number of parallel existing SPRequest objects heavily. Indeed that would be correct. But remember: you have usually multiple navigation controls on a single page! Disposing these objects directly after the properties have been read would mean that for each navigation control on the page the same objects have to be retrieved from the database again and again - which would impact the performance of the site. To avoid this the items retrieved by the navigation controls are cached to ensure that other navigation controls needing the same items can leverage them without causing additional requests to the backend database. The caveat is that the overall memory consumption of the site will go up through this implementation.

As indicated in the overview section SharePoint v3 monitors how many SPRequest objects exist in parallel on the same thread. As soon as SharePoint identifies that with a new allocation the number of SPRequest objects exceeds a configurable threshold SharePoint will create the following entry into the ULS log:

12/12/2007 12:29:54.05 w3wp.exe (0x1380)                       0x126C  Windows SharePoint Services   General                                0                Medium               Potentially excessive number of SPRequest objects (14) currently unreleased on thread 5.  Ensure that this object or its parent (such as an SPWeb or SPSite) is being properly disposed.  Allocation Id for this object: {2DF92EAD-4959-4A21-8BA0-FBD91E8FDD48} Stack trace of current allocation: at Microsoft.SharePoint.SPRequestManager.Add(SPRequest request, Boolean shareable)

The default threshold to get the above warning is 8 - which is actually far to small for real live scenarios as we have seen that with navigation controls on your master pages you will for sure exceed this number. As you can see in the sample above the warning outlines that 14 SPRequest objects live in parallel on the same thread. As we have seen before with navigation controls a larger number of SPRequest objects is absolutly normal and as expected. So you can expect the ULS log to be filled up with many similar warnings.

You should inspect the ULS logs of your site and verify what are the highest numbers in these warnings to get an idea about how expensive your current site design currently is in regards to memory consumption through allocated SPRequest objects. If the number is higher than 150 you might need to consider to change the design of your site or the number of levels to show in your navigation controls to reduce the memory footprint.

Afterwards you can fine tune the threshold to avoid unnecessary warnings. This can be done by adjusting the threshold through the following a registry key (e.g. to 50):

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\HeapSettings

        LocalSPRequestWarnCount = 50

The actual number to configure here depends on your site structure. After setting this registry key you need to restart the sharepoint services and the application pools (or restart the server) to ensure that the settings get effective.

Also be aware that you will only see the above warning if you have configured diagnostic logging for the General category to at least Medium level - which is actually the default. So if you haven't changed the configuration of diagnostic logging you will see these warnings.

Do I have code on my site which does not correctly dispose SPSite/SPWeb objects?

One method (not completely reliable but a good indicator) is to monitor the ULS log message mentioned in the last section. If the number for some threads gets higher and higher during the livetime of a thread, without going down to the configured threshold again the chance is high that there is some code which does not properly dispose SPWeb and SPSite objects.

Be aware that you cannot just look at the callstack listed in these messages to identify the bad code as this warning will be written during allocation of the objects! This message does NOT indicate that the object is not properly disposed! So you can only use it as an indicator that some code might be wrong but not to identify the module causing the problem!

To identify the bad code we need to use a different approach.

As mentioned in the Overview section SharePoint not only monitors the number of SPRequest objects allocated by each thread but also checks if SPRequest objects still exist at the end of each thread and create the following message into the ULS log.

05/01/2007 12:58:47.31                w3wp.exe (0x105C)                                      0x09A8 Windows SharePoint Services                   General                               8l1n       High       An SPRequest object was not disposed before the end of this thread.  To avoid wasting system resources, dispose of this object or its parent (such as an SPSite or SPWeb) as soon as you are done using it.  This object will now be disposed.  Allocation Id: {5BFFCA4B-3B91-45BF-98CD-0BB508BE30EE} To determine where this object was allocated, create a registry key at HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\HeapSettings. Then create a new DWORD named SPRequestStackTrace with the value 1 under this key.

Be aware that the livetime of a thread can be much longer as a single request. As SharePoint only checks if undisposed SPRequest objects exist when the thread ends and not at the end of each request the number of entries of this type can be much smaller than you might expect. So you might have to search in multiple ULS logs and not just the latest for this warning (you can actually search for the string 8l1n as this is the unique identifier for this message).

In addition: if the object has been disposed by the garbage collector then you will see a slightly different message stating that the garbage collector reclaimed the object:

05/01/2007 14:48:43.32                w3wp.exe (0x105C)                                      0x09A8 Windows SharePoint Services                   General                               8l1n       High       An SPRequest object was reclaimed by the garbage collector instead of being explicitly freed.  To avoid wasting system resources, dispose of this object or its parent (such as an SPSite or SPWeb) as soon as you are done using it.  Allocation Id: {098092FF-E4D9-4BA3-B2B8-5C6CE9B96554}  To determine where this object was allocated, create a registry key at HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\HeapSettings.  Then create a new DWORD named SPRequestStackTrace with the value 1 under this key. 

To identify which code is responsible for the problem after you identified the problem you have two options:

1. You can search the earlier ULS logs for the allocation ID in the warning (if the allocation was done beyond the threshold you will find it)
2. You can follow the hints in the warning and set the following registry key as this will ensure that the stack trace of the allocation of the SPRequest object is preserved in memory and added to the 8l1n ULS log entry:

The second option is more reliable as it will ensure that you get a callstack for each of the 8l1n messages. Be aware that after setting this registry key you need to restart the sharepoint services and the application pools (or restart the server) to ensure that the settings get effective.

By analyzing the emitted callstack you should then be able to identify the component causing the problem.

SharePoint Administration Toolkit has been released

The Microsoft® SharePoint® Administration Toolkit contains functionality to help administrate and manage Microsoft® Office SharePoint® Server 2007 and Windows® SharePoint® Services version 3.0.

This toolkit contains two new functions - the ability to perform bulk operations on site collections and a Stsadm operation to update alert emails after a Web application has changed URLs.
More details about this toolkit can be found in the following Whitepaper.

The following issue can be resolved using the toolkit:

KB 936759 - Alerts do not fire after upgrade/migration 
KB 936760 - Alerts have incorrect URL in body of the alert when URL of web application changes

You can download the SharePoint Administration Toolkit at the following location:

• 32-bit Version
• 64-bit Version

Proxy Server, Cache-Control http header, ASP.NET and MOSS 2007

Today I got in contact with an interesting problem raised by a customer. He enabled output caching on his MOSS server machine and configured the output cache profile to use Cacheability "Public":

This setting should add a "Cache-Control: public" http header to the response sent to a browser. "Cache-Control: public" allows caching of the item as well inside proxy servers and in the client browser cache.

What the customer noticed was that event that he set the Cacheability to public the Cache-Control header sent to his browsers was "Cache-Control: private". "Cache-Control: private" allows caching of the item only in the Browser cache but not inside proxy servers.

Due to this Cache-Control header the MOSS server got hit by many more requests than expected.

What we found is that this behaviour was actually caused by a security feature of ASP.NET. Even that SharePoint added a "Cache-Control: public" header to the response ASP.NET changed this header to "Cache-Control: private" in the OnLeave method of the System.Web.Caching.OutputCacheModule.

So why did this OutputCacheModule change the Cache-Control header from public to private?

The answer is: MOSS was not configured for anonymous authentication. ASP.NET does not allow "Cache-Control: public" for authenticated requests. It enforces "Cache-Control: private".

On second thought this makes perfectly sense: if the server requires authentication to view specific content then it would not be nice if a proxy server would cache the items and show it to the next user trying to see the same item even if he is not authenticated against the MOSS server. The proxy server would not have a chance to verify if this second user has rights to see the item or not.

E.g. lets assume that one of the sharepoint pages contains a web part that shows each user private information about his bank account. If the page would be cached in a proxy server the next user would not see his account information but the account information of the user who previously browsed to the same URL.

To avoid this the ASP.NET output caching module intercepts the response and replaces Cache-Control: public with Cache-Control: private if the response is for an authenticated request.

Enabling anonymous access on the site collection provided the customer with the required functionality.

As an administrator you have to decide what is more important for you: optimum security or optimum performance - you cannot have both.

Training videos for enterprise search using MOSS are now available on Technet

The following link provide training resources that help to demonstrate enterprise search capabilities in Microsoft Office SharePoint Server 2007.

Building and implementing enterprise search solutions

The following 14 recorded presentations are based on training modules from three-day, in-person training sessions. The presentations provide details about key enterprise search capabilities in Microsoft Office SharePoint Server 2007. Additionally, the presentations provide guidance about how to combine SharePoint products and technologies in order to build, implement, and manage enterprise search solutions.

Module 1: Workshop Overview
Module 2: Enterprise Search Overview
Module 3: SharePoint Search 2007 Walkthrough
Module 4: Search Architecture and Deployment Scenarios
Module 5: Crawl and Query Processes
Module 6: Relevance Ranking
Module 7: Customizing the End-User Experience
Module 8: Developing Search Solutions
Module 9: Business Data Catalog Search
Module 10: Extensibility and Integration for Search
Module 11: Search Administration
Module 12: Security for Search
Module 13: Performance Scalability and Capacity Planning for Search
Module 14: Search Operations

MOSS - Common Issue - Content Deployment fails with "Failed to compare two elements in the array."

A common problem we see with content deployment and with STSADM -o export is the error message below:

[4/11/2008 9:25:01 AM]: FatalError: Failed to compare two elements in the array.
   at System.Collections.Generic.ArraySortHelper`1.QuickSort[TValue](T[] keys, TValue[] values, Int32 left, Int32 right, IComparer`1 comparer)
   at System.Collections.Generic.ArraySortHelper`1.QuickSort[TValue](T[] keys, TValue[] values, Int32 left, Int32 right, IComparer`1 comparer)
   at System.Collections.Generic.ArraySortHelper`1.QuickSort[TValue](T[] keys, TValue[] values, Int32 left, Int32 right, IComparer`1 comparer)
   at System.Collections.Generic.ArraySortHelper`1.Sort[TValue](T[] keys, TValue[] values, Int32 index, Int32 length, IComparer`1 comparer)
   at System.Collections.Generic.ArraySortHelper`1.Sort(T[] items, Int32 index, Int32 length, IComparer`1 comparer)
   at System.Array.Sort[T](T[] array, Int32 index, Int32 length, IComparer`1 comparer)
...
*** Inner exception:
Object reference not set to an instance of an object.
   at Microsoft.SharePoint.SPFeature.EnsureProperties()
   at Microsoft.SharePoint.SPFeature.get_TimeActivated()
   at Microsoft.SharePoint.Deployment.WebSerializer.ExportFeatureComparer.System.Collections. Generic.IComparer<Microsoft.SharePoint.Deployment.ExportObject>.Compare(ExportObject exportObject1, ExportObject exportObject2)
   at System.Collections.Generic.ArraySortHelper`1.QuickSort[TValue](T[] keys, TValue[] values, Int32 left, Int32 right, IComparer`1 comparer)

The usual reason for this problem is that some sites in the site collection have features assigned where the feature is not installed in the server farm. The main problem here is that you cannot easily identify which features are missing as the missing features are not reported in the error message.

To overcome this problem I have written a tool which allows to identify all features used in a site collection which are missing on the server: WssAnalyzeFeatures.

To resolve the problem you then have to install the identified missing features on the exporting server. In case that this is not possible you have to remove the features from the site colleciton or the affected sites. Usually this can be done using STSADM -o deactivatefeature but sometimes fails if the feaure definition is not installed on the server. In this case you can use WssRemoveFeatureFromSite.

Deep Dive into the SharePoint Content Deployment and Migration API - Part 6

[Part 1 - Part 2 - Part 3 - Part 4 - Part 5 - Part 6]

Requirements for a successful content deployment

Be aware that this article cannot provide a complete list of all requirements. I will update it from time to time if required. Today I will list all the requirements I have seen in the past which - when missed - can cause content deployment jobs to fail.
 

1) The servers configured as export and import server need to host an instance of the Central Administration website

When configuring the "Content Deployment Settings" for your farm you have the chance to select different servers in your farm to work as export and import servers for the content deployment. This allows to offload this task to a dedictated server to reduce the load on your web frontend servers.

The comments of this configuration options look as follows (here shown for the Import server):

Import Server
Specify the server you want to receive incoming content deployment jobs. This server must have enough available disk space to store the incoming jobs, and it must be running an administration Web application for the farm.

So this requirement is actually listed but a little bit hidden in the second half of a sentence.

In case you configure a server that does not host the Central Admin website then you will not get a warning here!

Impact: The effect you will see is that the content deployment export or import phase will not start.

How to resolve:
1) Provision the Central Administration website on the desired server
2) Change the configuration of the export and/or import server
 

2) Ensure that sufficient disk space is available

This sounds like a simple prerequesit but it isn't. Content deployment uses different places to store the extracted and compressed files. The compressed files are stored in the location you can configure on the "Content Deployment Settings" page in the "Temporary Files" row.

But before export creates the compressed files it first exports everything into a temporary directory. And this directory is placed inside the directory which is configured in the TMP environment variable of the user the Windows SharePoint Services Timer service (OWSTIMER.EXE) runs on (usually refered to as farm credentials). Per default this variable has the following value: "%USERPROFILE%\Local Settings\Temp" which is usually on your system drive.

Impact: So per default MOSS content deployment requires the disk space for the uncompressed exported files on your system drive!

How to resolve: The easiest way to resolve this is to logon to the machine with the farm credentials and adjust the TMP variable to point to a different location. Afterwards you would need to restart the OWSTIMER service.
 

3) Use an empty site collection as the destination of your content deployment job

As already discussed in Part 5 content deployment will fail if the destination database contains conflicting content. To avoid this it is required that the initial deployment is done into an empty site collection.

Be aware that the only way to create an empty site collection is to use the following STSADM command:

STSADM.EXE -o createsite -url <url-to-site-collection> -ownerlogin domain\user -owneremail <email-address>

Using the "Blank Site" template will NOT create an empty site collection! It will actually create a site collection with content. You can see the difference if you create a site collection using both methods and then inspect the content of the created sites using SharePoint designer.

Although serveral papers indicate that the "Blank Site" template can be used I would recommed to always use the STSADM command with the syntax above to ensure that you really have an empty site collection as destination

Impact: If the site collection has been created using a different method or already contains data the content deployment job will fail.

How to resolve: Deploy into an empty site collection
  

4) Install all required features for your site on the destination server

If your site requires custom features ensure that the features are installed on the destination server before running content deployment.

Impact: If the are missing the import phase will fail.

How to identify this: I have written the tool WssAnalyzeFeatures which allows you to identify such problems.

How to resolve: Copy the features to the destination server and install them using STSADM -o installfeature. 

5) Do not activate custom features for your site collection on the destination server manually

You should not activate custom features on the destination server if this activation creates content in the destination database as this can cause conflicts as outlined in "Problem 1" of Part 5 of this article series. Instead you should run content deployment and let the import process activate the features on your destination server as this will ensure that all items get created using the same ids as on the source server which is otherwise not guaranteed.

Impact: If the features have been activated and the content deployment import can fail with similar error messages in as "Problem 1" of Part 5.

How to resolve: Deactivate the feature in the destination site and ensure that all items created by the feature are removed. Alternatively do a full deployment into an empty site collection instead.
 

6) Do not expect that incremental deployment will deactivate features in the destination server site collection

The content deployment and migration API was not designed to deactivate features on the destination server. If a feature needs to be deactivated on the destination server you need to manually perform this deactivation.

7) Ensure that all feature definitions of features activated on the site collection exist on the source server

This is actually a high call generated for Microsoft Support Services: in the development a feature becomes obsolete and is removed or replaced with a different version with a different Guid but on some sites or site collection the old feature is still activated.

Impact: The affected sites can no longer be exported. You will get the errors listed in this article.

How to identify this: I have written the tool WssAnalyzeFeatures which allows you to identify such problems.

How to resolve: Either copy the missing feature files to the required location or uninstall the feature using STSADM -o deactivatefeature/uninstallfeature. In case that STSADM -o deactivatefeature fails to deactivate the feature you can use my tool WssRemoveFeatureFromSite.
 

8) Configure the retention period of the change log to be long enough for incremental deployment

See here for details.
 

9) Ensure that content deployment jobs do not run in parallel

The current implementation of the content deployment and migration API does not allow parallel execution. There are plans to change this behavior in the near future but as is you need to ensure that only one deployment is running at a time.

So if you have multiple deployment paths and jobs for the same site collection you need to ensure to schedule them in a way that they don't overlap.

But this is not the only place to look at! Sometimes it is nearly impossible to prevent parallel execution. Just think that the content deployment and migration API is not only used to deploy content between different webfarms. The same API is used in the copy/move implementation inside site manager and in the variation feature.

With other words: you can experience problems with content deployment if an author copies a page at the same time or creates a new page in the source variation label which is then replicated to the destination. And also vice versa a copy operation can fail because a quick deploy job was running at the same time.

Impact: parallel execution of deployment jobs can lead to failing content deployment.

How to resolve: you need to restart the failed deployment job

10) Avoid the problems listed in Part 5 of this article series.

Adjusting the MOSS ROBOTS meta tag for 3rd party search engines - using a Control Adapter (last update: April 11th)

When developing a public facing website using the publishing features of MOSS it might be required to emit a ROBOTS meta tag that prevents internet search engines from indexing specific pages.

The standard RobotsMetaTag control included in WSS generates the following tag:

<META NAME="ROBOTS" CONTENT="NOHTMLINDEX">

Unfortunatelly most search engines do not understand the NOHTMLINDEX content value but a value of NOINDEX.

I already posted a solution for this which replaced the RobotsMetaTag control. Here is now a more elegant solution based on ASP.NET Control Adapters. Control Adapters can be used to modify the rendering of a control based on different browsers. But by using the "Default" browser it also allows to modify the default rendering of a control. Thanks to Gael Duhamel for the idea!

Here is the source code of the control adapter which we will use to modify the behavior of the RobotsMetaTag control:

The code looks very similar to the code I used in my first approach.

To build the adapter you need to create a class library project, add this source code to it, configure the project to sign the assembly and add the resulting DLL to the global assembly cache (GAC).

To bind the control adapter to the RobotsMetaTag control we now need to add a new browser (e.g. SharePointAdapters.browser) file to the App_Browsers directory of your SharePoint web application.

The content of the browser file needs to look similar to this (you need to adjust the assembly name and strong name key based on your specific assembly):

That's it! After recycling your application pool the RobotsMetaTag control will now render it's content based on the custom control adapter. No modification to your master pages will be required.

WSS/MOSS - Common Issue - Error Message "Unknown compression type in a cabinet folder" during export

Another issue which I have seen a couple of times in the past when using STSADM -o  export or content deployment is a failure during the export phase with the following error message:

Failed to create package file
Unknown compression type in a cabinet folder

In all cases I have seen this problem was caused by insufficient disk space. Please monitor the disk space during the export/compression phase and ensure that sufficient disk space is available to perform this operation.

MOSS - Common Issue - events 7076, 6398, 6482 listed in the Event log and/or IIS MMC not responding

We had a big number of cases where customers reported the following pattern:

Event ID 7076 -

Office SharePoint Server "Attempted to read or write protected memory. This is 
often an indication that other memory is corrupt. " 
"System.AccessViolationException: Attempted to read or write protected memory. This 
is often an indication that other memory is corrupt.
...

Event ID 6398 - 

Windows SharePoint Services 3 NERA-WPWFE1 
"Microsoft.Office.Server.Administration.ApplicationServerAdministrationServiceJob 
""cddd50c5-bf98-4496-85b5-d1c755e4dca7 " 
"Exception from HRESULT: 0x80005006  
 
Event ID 6482 -
 
"Microsoft.Office.Excel.Server.ExcelServerSharedWebServiceInstance 
""3ff29932-af40-4296-b012-eb71dee2c40e ""Attempted to read or write protected 
memory. This is often an indication that other memory is corrupt. "" 
System.AccessViolationException: Attempted to read or write protected memory. This 
is often an indication that other memory is corrupt. 

Some customers also reported that the IIS MMC stops responding and a restart of IIS is required to resolve the problem.

After some research we found that all these problems were caused by a problem in the IIS ADSI providers. A hotfix to resolve this issue is meanwhile available:

946517 - FIX: You may be unable to manage IIS 6.0 by using Server Manager if two threads access IIS 6.0 at the same time

MOSS - Common Issue - Incremental deployment fails with "The changeToken refers to a time before the start of the current change log."

I have seens this problem a couple of times in the past: sometimes when running incremental content deployment the deployment job fails with the following error message

The changeToken refers to a time before the start of the current change log.

To make a long story short: to resolve the problem you should do a full deployment into an empty database. It is not recommended to run full deployment into the previously used database as the full deployment will not perform delete operations of items that have been deleted in the source database but still exist in the destination database. It will also not reliably work if items have been moved on the source database to other places after the last successful incremental deployment has been used.

Why does this problem occur? This can happen mainly for two different reasons:

A) The timespan since the last incremental deployment job is too long

MOSS stores the change token of the last successful incremental deployment inside the properties of the incremental content deployment job. When a new incremental deployment is run it compares the change token in theses settings with the entries in the change log.

Per default the change log is configured to keep changes for 15 days. If the timespan between two incremental deployment job exceeds this timespan then the change log does not contain entries from before the change token and incremental deployment will not start to prevent deploying only parts of the required content.

A solution for this would be to increase the timespan the change log should be preserved. This can done using the following steps:

Central Administration - Application Management - Web Application General Settings - (choose the desired website) - Change Log

B) The source database has been overwritten with a backup

When a database is restored through STSADM -o restore or using SQL backup and STSADM -o addcontentdb the change log is cleared.

Incremental deployment will not work in this case and you have to do a full deployment to synchronize the content with the destination database.

IIS 7 - How to send a custom "Server" http header

A question we have often seen in the past is to have a method to prevent IIS from sending the server identification header to a client which allows a client to identify which type of http server it is talking too. Usually this request comes from security concerns as knowing the server would allow a hacker to more easily be able to break into the system.

Although the above assumption from customers is very doubtable we still need to be able to provide a solution for this.

Out of the box all our IIS servers respond with a server header similar to the following (sample is for IIS 6.0):

Server: Microsoft-IIS/6.0

For IIS 5 and IIS 6 customers often used UrlScan which allows to remove the server header from the response.

On IIS 7 this tool cannot be installed - but due to the very modular structure of IIS 7 it is possible to remove or even replace the Server header in a much more convenient way: using a custom Module which is injected into the IIS 7 Pipeline. Such a module can be developed as well using managed or unmanaged code.

Here is a sample .Net module which replaces the server http header with a custom header:

That's it! When generating this module ensure to strong name it as it needs to be placed into the global assembly cache in order to allow IIS 7 to use it. To add the module to IIS 7 use the "Modules" configuration option on the server, choose "Add managed module" and select the module from the list of available modules.

Common error situation when using backup/restore to transfer a database to a new farm on MOSS 2007

In the past I have seen the following problem a couple of times: a customer creates a backup of a content database on one server farm (e.g. using STSADM -o backup or a DB backup in SQL) and restores the backup on a different farm and attaches the content database to a web application.

After this operation is done several operations (like variations and content deployment) fail to work with the following exception:

The reason for this problem is that backup/restore does not adjust the references from the publishing page objects in the Pages library to their Page Layouts. These URLs are sometimes stored as absolute URLs including the server name. And this server name is the server name of the old server farm which cannot be resolved on the new farm.

In case that you have run into the above problem you have two options:

1. Throw away the database and transfer it correctly using STSADM -o export/import or content deployment
2. Fix the incorrect links manually using the following steps
   
1. Open the web in SharePoint Designer
2. On the “task panes” window, pick hyperlinks.
3. For the “hyperlink” heading, click the arrow and pick (custom…)
4. In the dialog, ask to show rows where the hyperlink begins with a URLs which are not valid on the current server farm
5. For each of the files, right click and say “Edit hyperlink…” and Replace hyperlink with the hyper link that is valid on the current server farm.

In some situations it is also required to use the following STSADM extension to fix the PageLayout field: Fix Publishing Pages Page Layout URL