Microsoft Technology

SharePoint TeamSites Migration steps

2012-05-30T20:25:39Z

Introduction

This is to go through the steps involved in migrating SharePoint 2007 to SharePoint 2010 using database attach method. This article is very specific to the steps involved in the migration rather explaining how to build SharePoint 2010 farm or architecture of the migration.

Migrate SharePoint 2007 to SP2010

Preparation

Before migrating the data, you must configure a new server or server farm with SharePoint 2010, the following information about permissions, hardware requirements, and software requirements.

Ensure that you have met all hardware and software requirements. You must have a 64-bit version of Windows Server 2008 or Windows Server 2008 R2. For server farms, you must also have a 64-bit version of SQL Server 2005 or SQL Server 2008. For more information about these requirements (such as specific updates that you must install), http://technet.microsoft.com/hi-in/library/cc262485(en-us).aspx.
Ensure that you are prepared to set up the required accounts by using appropriate permissions. For detailed information, see Administrative and service accounts required for initial deployment (SharePoint Server 2010). http://technet.microsoft.com/hi-in/library/ee662513(en-us).aspx.
Run the pre-upgrade checker on your original environment. The pre-upgrade checker identifies potential upgrade issues in your environment so that you can address them before you upgrade. It can also help you identify settings that you need in your new environment. For more information, see Run the pre-upgrade checker (SharePoint Server 2010).http://technet.microsoft.com/hi-in/library/cc262231(en-us).aspx. It gives you a detailed report in the form of a html file and txt file which tells you about the warnings and errors if any which you have to take care before upgradation otherwise the upgrade may fail if all errors are not resolved.

In summary, you need to do the following

Install SharePoint Server 2010 on the server or servers.
Configure service applications.
Configure general farm settings.
Create and configure Web applications.
Reapply customizations.

Planning

The planning or assessment has been the first step in a five-step upgrade process. This section contains a summary of phases and a breakdown of activities and estimates.

As part of this assessment a high-level upgrade plan has been developed, which provides next steps and rough estimates for the complete process.

Learn

This section contains the goals and activities of this assessment project.

Goals

Perform an Assessment and get a clear picture of current state
Get action items for closing gaps to be ready for upgrade
Get high-level plan and estimate for 2010 upgrade

Activities

Task Name

Conduct interviews and collect information

Prepare Assessment report

Planning

Prepare

Goals

Select upgrade/migrate strategy
To close gap between SharePoint 2007 deployment and 2010 upgrade requirements
Re-factor and reorganize before migration

Activities

Task Name

Prepare upgrade plan

WinDiff 12 hive check

Preupgradecheck

Create Lab environments

Create upgrade Test environment

Prepare Customizations for upgrade

Prepare Content for upgrade

Testing changes

Create a Test plan

Training

Update planning and risks

Test

Goals

Find issues early
To validate upgrade planning
Perform test upgrades of content and customizations
Upgrade performance
Fix issues and planning

Activities

Task Name

Build Test environment

Test upgrade approaches

Finalize Upgrade Plan

Implement

Goals

Update documentation
Install and configure SharePoint 2010
Establish business continuity management
Perform upgrade

Activities

Task Name

Plan

Prepare dependencies

Build preproduction environment

Build production environment

Perform upgrade (attach upgrade strategy)

Post-upgrade

Test

Validate

Goals

Perform final validation of upgrade
Perform switch
Release and communicate

Activities

Task Name

Examine upgrade results

Examine system events

Execute UAT (Test Environment)

Identify and fix final issues

Switch to SharePoint 2010

Backup Site collections

Migration needs to happen in a batch. As part of the above planning session, you need to work with your project manager and team to identify the site collections to be included in each batch.

Site collection can be backup/restore or Export/import. If you are using lot of workflows and want to preserve the history, instances of it then Export/import will not work for you. Either way, before doing any backup/export, you need to change the content database in a "Read-only" mode.

To choose either Export or backup, please refer this http://blogs.msdn.com/b/yvan_duhamel/archive/2009/05/18/some-key-differences-between-stsadm-export-and-backup-operations.aspx

Note: You don't need to do the backup or export the site collection, but you can directly take the content database backup and attach it to the new SharePoint 2010 environment. However, I prefer to do this as I can have better governance (manage the quota and sites per db) on the new environment and as a cleaner approach.

Add Content database on your SP2007 test environment

Create content database on your SharePoint 2007 environment.

stsadm.exe -o addcontentdb -url "<http://teamsite.contosotest.com/>" -databasename "DatabaseName" -databaseserver "DbServer"

Using the restore command restore the site to the content db just added

stsadm –o restore –url http://teamsite.contosotest.com/site1 –filename "y:\backup\site1.bak"

Fixing any errors

Go to the SharePoint 2010 environment.

Open the SharePoint Powershell Command prompt and execute the following command

Test-SPContentDatabase - Name "DatabaseName" -WebApplication http://teamsiteSharePoint2010.contosotest.com/ > C:\migration\Log\result.log

Analyze the log file from Test-SPContentDatabase

Missing Feature

If you see "Missing Feature" and it is in your list, the site is probably in READ ONLY MODE.

You need to undo this.

Steps:

Copy the feature id from the log file
Run the query in the database (Content Db created in the above steps)
1. Select SiteId from Features where featureid = "xxxx-xxx-"
2. Copy the siteid guid from the result
3. Select * from webs where siteid = <siteidGuid>
4. Copy the FullURL from the result
5. Go to the SharePoint 2007 Test Environment
6. Check the site has been locked
  1. Stsadm –o getsitelock –url http://teamsite.contosotest.com/sites/IS test
7. If locked – unlock it
  1. STSADM -o setSiteLock –url http://teamsite.contosotest.com/sites/IS test -lock none
8. Run the test-SPContentdatabase command again

If still getting the missing feature error, you just want to clean things - remove the feature. First you can uninstall the feature (it has to be there to uninstall it.)

C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\BIN>stsadm -help uninstallfeature

stsadm.exe -o uninstallfeature
{-filename <relative path to Feature.xml> |
-name <feature folder> |
-id <feature Id>}
[-force]

C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\BIN>stsadm -o uninstallfeature -id aae8zef7-2xb9-c0d5-t960-jk156l837cde

You may get the following error

"Feature with Id 'aae8zef7-2xb9-c0d5-t960-jk156l837cde' is not installed in this farm. The feature was not uninstalled."

If the feature is not installed on the farm, and you want to force remove the feature you can run stsadm to deactivate the feature.

C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\BIN>stsadm -help deactivatefeature

stsadm.exe -o deactivatefeature
{-filename <relative path to Feature.xml> |
-name <feature folder> |
-id <feature Id>}
[-url <url>]
[-force]

C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\BIN>STSADM -o deactivatefeature -id aae8zef7-2xb9-c0d5-t960-jk156l837cde

You still may get the following error

The feature with Id ' aae8zef7-2xb9-c0d5-t960-jk156l837cde' is not currently installed. Use 'force' to deactivate it at this scope.

Adding "Force" parameter

C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\BIN>STSADM -o deactivatefeature -id aae8zef7-2xb9-c0d5-t960-jk156l837cde -force

Operation completed successfully.

Deactiviting this feature aae8zef7-2xb9-c0d5-t960-jk156l837cde across all the web apps cleaned it up:

Now run the Test-SPContentDatabase command again - Our missing feature has been removed/deactivated (Dependency removed!).

Missing setup file

This error as suggests is due to the missing feature or solution. This issue can occur in all cases as the error could be referring to the themes, files, workflows, web parts and more. We need to find the file that references the missing setup file and then deal with it accordingly.

There are several ways of finding the missing features, but I find the following worked well for me.

Run the below query against the database by replacing the file URLs

select DirName, LeafName from alldocs where SetupPath in('Features\<Copy and paste the missing setup file object which complaints>")

The result will produce an output with DirName, and LeafName

In case, if you are clear from the "DirName" reported from the above output, you can also use the PowerShell to get the URL of the actual file causing the issue. This can achieved by the GUID represented in the missing setup file error

$File = $web.GetFile ([Guid] "x5xc66e7-y920-5a55-6c3d-5b6bc936104e")

$file.ServerRelativeURl

Now you have to decide what to do with the file. Most of the times, I found that some of the features were not deactivated from the site prior to removing the solution from the farm. Depending on your situation, you can also delete that object from the site – if you feel that is not necessary for SharePoint 2010 environment going forward.

Missing web part

Run the below query

select AllDocs.SiteId,WebId,Webs.Title as 'Web Title', ListId, DirName,LeafName from AllDocs inner join WebParts on Alldocs.Id = Webparts.tp_PageUrlID inner join Webs on Alldocs.WebId = webs.Id where Webparts.tp_WebPartTypeId = '36f2680f-4855-f100-da5b-5dd1d07ae62b'

Copy the DirName from the result
Open IE à type the webapplication name (http://teamsite.contosotest.com)
Append the DirName(Sites/Regional office/xyz)
Copy LeafName from the result and append (/allitems.aspx)
Add ?Contents=1

In IE Url should be like àhttp://teamsite.contosotest.kraft.com/sites/regional office/xyz/allitems.aspx?contents=1

Delete the Error webpart.

Keep repeating the above steps with Test-SPContentdatabase until you see no errors.

Attach Content Database to SharePoint 2010

Detach the database from SharePoint 2007 Server from central administration.

Mount-SPContentDatabase -Name "DBNAME" -WebApplication "http://SharePoint2010Teamsite.contoso.com" -UpdateUserExperience

If you see the status of the database as "Database is upto date, but some sites are not completed upgraded"

Get-SPContentDatabase -Identity <<Dbname>>
Copy the ID(which is Guid)
Upgrade-SPContentdatabase <<GUID>
Confirm

On upgrade, if you get an error "Feature upgrade failed for Feature.." – it is because of the Orphan feature in SP2007. Again you need to make decision on those features either to fix it or delete it.

Also run the following command

Stsadm –o localupgradestatus

Make sure all Ok in that.

SharePoint 2010: Backup and Restore Best Practices

2012-05-29T21:59:53Z

Introduction

This whitepaper gives an operational area of SharePoint specifically on Backup and Restore.

Backup Solution

The following Table represents backup recommendation for SharePoint 2010 Collaborative environment.

Table 1 – Backup Solution

1.1.1.1. Core Content Recovery

For Core Content there are 2 levels of data recovery such as Content recovery and Site recovery. Each level addresses a different business issue and is often performed by persons in different roles within the organization.

Content recovery refers to recovering a document or list by using the Recycle Bin or versioning. Content recovery is a frequent and small-scale activity, and it can be performed by end users and site administrators.

Site recovery refers to using tools to recover from accidental deletion or data corruption of a site. Site recovery can be performed by site administrators.

Database recovery refers to the databases such as Content Database and Service application Databases.

1.1.1.2. Content Recovery

Content Recovery is one or the most frequently used features in SharePoint. SharePoint takes a layered approach to it and provides multiple options:

· Versioning
· Recycle Bin

What is Versioning

Versions allow keeping multiple copies of documents that vary in content over time. This mechanism provides self-service recovery for users and is most useful for incidents in which a document is overwritten or corrupted. Through this mechanism user can revert to a previous version of a document.

Versioning is configured by the site collection administrator on a per-site basis. By default it is turned off. There are 3 possible versioning policies:

· No Versioning
· Create Major Versions
· Create Major and Minor Versions

The policy should instruct administrators to proactively manage the versioning process because versions of documents are not represented as differentials. Therefore each version is a complete representation at that instant I time. This can drive database sizes to the quotas very fast and impact backup and restore performance. User and administrator education will be the most impactful effort can make.

Contoso’s recommended policy on Versioning for SharePoint 2010

Deciding on versioning is site collection administrator job. However, Contoso design team can advise the site collection administrator to limit the number of versioning in their site collection.

Restrict Versioning in the following ways:

1) Limit the number of major versions
2) limit a number of major version that will have minor versions
3) CANNOT limit a number of minor versions to keep for a major version

*Limit depends on business use cases, for example content publishing site requires more versioning than community sites.

What is Recycle bin

The Recycle Bin has a 2 stage model that is on by default and is configurable on a per-web application basis. The length of time that an item stays in the Recycle Bin is by default 30 days but is configurable. This gives a 60 day opportunity to recover deleted content. In the majority of cases this will be sufficient. Again this is a fundamental aspect of Content Recovery strategies. Users should be educated on the existence and use of this feature. Multiple versions of a document can exist in the Recycle Bin and cannot be restored over an existing document. Site collection admin would need to use versioning for that.

Stage 1 is the first stage Recycle Bin and is located at the site level. It is available to users with Contribute, Full Control, or Design permissions. When a document is deleted it is sent here and continues to impact the site quota. It remains for the 30 days configured by the administrator is reached.

Stage 2 is located at the site collection level and it contains items deleted from the Stage 1 Recycle Bin. It will remain here until the time period specified by the administrator. It does not affect the site quota but does impact the size of the site and its content databases. The administrator can place a quota on the Recycle Bin size.

This Recycle Bin times do affect the life of content and the policy should be consistent with Business Records Retention policy.

Contoso’s recommended policy on Recycle bin for SharePoint 2010

Default 2 stages bin will be turned on and configured for 30 days for an item to recover. However, Site collection admin have rights to change it.

Recovering a site directly is available in SharePoint 2010 SP1. SharePoint 2010 SP1 introduces “Site Recycle Bin” feature which will help Site collection administrator to restore the site. Hence, the recommendation is Contoso to use “Site Recycle Bin” for restoring a site.

1.1.1.3. Content Databases

All content databases can be backed up by using SQL server tools. In the event of a disaster, these databases backup with SQL server should be restored using the standard restore procedures.

1.1.1.4. Farm Backup and Configuration Database

The recommendation is Contoso should be using backup and recover a farm without the content databases attached. This method provides Contoso to backup farm settings and Web application settings, in addition to the settings for any service applications that have been selected.

Advantage of full farm backup is - On Recovery, Farm doesn’t need to recreated and reconfigured.

To copy configuration settings by using a farm backup, it is recommended that Contoso first detach the content databases from the farm.

Here are the steps to be performed

Get-SPWebApplication | %{$_.Name;$_.Url;%{$_.ContentDatabases|%{$_.Name};Write-Host ""}} Get-SPContentDatabase | Dismount-SPContentDatabase Backup-SPFarm -Directory \\servername\share -BackupMethod Full Mount-SPContentDatabase -Name <WSS_Content> -WebApplication <http://servername>l

1.1.1.5. Restoring Service Application

Using PowerShell, Service application will be restored.

Restore-SPFarm -Directory <BackupFolder> -Item Shared Services\Shared Services Applications\<ServiceApplicationName> -RecoveryMethod Overwrite [-BackupId <GUID>] [-Verbose]

Where:

<BackupFolder> is the path of the folder where the backups are stored.
<ServiceApplicationName> is the name of the service application.
<GUID> is the identifier of the backup to use in the restore process.

1.1.1.6. Configurations

Office SharePoint Server includes Internet Information Services (IIS) configurations and configurations stored in the configuration database and Central Administration content database. In the event of a full disaster recovery, the following information will be required to restore the farm to the same exact configuration.

IIS Configurations

Internet Information Services (IIS) configurations are set in Central Administration or IIS Manager on each front-end Web server.

IIS configurations are stored in the IIS metabase. The metabase is a plain-text XML file on each front-end Web server that can be modified by using IIS Manager, or directly by Office SharePoint Server. The metabase is susceptible to being corrupted or overwritten, and it should be included in backup strategy.

It is recommended to document all IIS configurations rather backup for each Web server by using a tool that provides the configuration monitoring, such as Microsoft System Center Configuration Manager.

IIS configurations documents should include the following:

1. Application pool settings, including service accounts

2. HTTP compression settings - default

3. Time-out settings - 180 Seconds

4. Custom Internet Server Application Programming Interface (ISAPI) filters

5. Computer domain membership - homeoffice

6. Internet Protocol security (IPsec) settings - default

7. Network Load Balancing settings

8. Host header entries

9. Secure Sockets Layer (SSL) certificates

10. Dedicated IP address settings.

1.1.1.7. Binary Files

In the event that a SharePoint server needs to be restored, it is recommended to reinstall Office SharePoint Server and all other installed programs using the original binaries. Binary files should be kept in a secure location that can be easily accessible during a recovery scenario. The following table can be used to identify the different components.

	Component
Basic Components	Operating System
	Office iFilter Pack
	SQL Server
	Office SharePoint Server 2010
	SharePoint 2010 Language Packs · Ja-jp (Japanese) · Es-es (Spanish) · Fr-fr (French) · Pt-br (Portuguese Brazil) · Pt-pt (Portuguese) · Zh-tw (Chinese Traditional)
	RMS Client
	URL Scan
Customizations	MSIT Site Delete Capture
	Foxit PDF IFilter
	Contoso SharePoint Branding
	Contoso Site Provisioning / Site Directory
	NewsGator 2010
	MS Application Templates
	JQuery
	SharePoint Administration Toolkit

Table 2: Contoso Binary Files

1.1.1.8. Customizations

This section lists all 3^rd party and in-house customizations that have been deployed to the environment and provides guidance for restoring them.

Custom code (WSP)

Customization includes

1. Assembly Development

Any custom code developed such as Web Parts, site or list definitions, custom columns, new content types, custom fields, custom actions, coded workflows, or workflow activities and conditions. It also includes 3^rd party tools, if any. Any Visual Studio development wrapped as WSP.

2. Artifact authored Development

Any code developed using SharePointdesigner or Internet Explorer.

For the Assembly development customization, it will be stored on the source control repository system (Team Foundation Server). In case of recovery WSP will be retrieved from Team Foundation server.

For the artifact authored development, objects will be stored in Content databases, which be backed up with SQL Server.

1.1.1.9. Read-Only Mode

Another capability in SharePoint 2010 is support for configuring content databases to be in a Read-only mode. In this mode SharePoint 2010 will seamlessly detect and respond to this change and disables any user interface options associated with write and edit scenarios. This allows user to continue using the system to retrieve data and work within SharePoint until the environment is again configured to be writable.

Recommendation

The above capability should be used in Disaster Recovery solutions with secondary environments during patching and upgrade, when Contoso decides to have DR farm. .

1.1.1.10. Backup scenarios

In addition to the configuration and content databases, it is necessary to back up the search databases, and other SharePoint services databases that have in the deployment. The backup should also include backup of any configuration settings on the Web front-end servers and the application servers.

The recommendation is to regularly backing up the complete farm by backing up both the configuration and content. Regularly backing up the farm reduces the possibility of data losses that might occur from hardware failures, power outages, or other problems. It is a simple process and helps to ensure that all the farm data and configurations are available for recovery, if that is required.

The following table lists components of a SharePoint environment that needs to protect, and the tools that can be used to back up and recover each component.

Component	Backup type	Frequency	Considerations
Farm	SharePoint	Once per week or after a customization.	· Performing a backup does not affect the state of the farm. · The farm backup process does not back up any certificates that used to form trust relationships. · Backing up the farm backs up the configuration and Central Administration content databases.
Web Applications	SharePoint	Once per month or after a customization.	· When back up a Web application, the Internet Information Services (IIS) settings and all content databases that are associated with the Web application are also backed up
Application Services	SharePoint	Full once per week or after a customization.	· Microsoft SharePoint Server 2010 backup backs up the Business Data Connectivity service external content type definitions but does not back up the data source itself
All SQL Server databases	SQL Server	Full once per week + several differentials per week or Full 2, 3 times a week + Transaction logs in between.
Transaction Logs	SQL Server	Every 10 min.
SQL Cluster Nodes	SQL Server	Weekly full system state backup and daily differential

Table 3: Contoso SharePoint 2010 Backup Scenario

Microsoft SQL Server 2008 R2 transaction logs record all changes that were made to a database since the last checkpoint or full backup. These logs contain required data for restoring the farm. The recommendation is backing up these logs every 5–10 minutes. Upon back up these logs, they are automatically truncated.

1.1.1.11. Restore Scenarios

What to restore	What to do
Web Front End	Replace hardware Restore ―available image‖ or Restore/reinstall Operating system.
Service application	Run power shell command Restore-SPFarm -Directory <BackupFolder> -Item <ServiceApplicationName> -RecoveryMethod Overwrite [-BackupId <GUID>] [-Verbose]
Single SQL Server Node	Wait for the resources to fail over to the passive node Replace hardware Restore ―available image‖ or Restore/reinstall Operating system. Restore latest backups using SMSQL
Farm	Replace hardware Restore available image or reinstall operating system. Run power shell command Restore-SPFarm -Directory <BackupFolder> -RestoreMethod Overwrite [-BackupId <GUID>] · View the backups for the farm by typing the following: Get-SPBackupHistory -Directory <Backup folder> -ShowBackup. · If no BackupId was provided, the most recent backup will be used.
Web Application	Run power shell command: Restore-SPFarm -Directory <BackupFolderName> -RestoreMethod Overwrite -Item <WebApplicationName> [-BackupId <GUID>] [-Verbose] Where: · <BackupFolderName> is the full path of the folder used for backup files. · <WebApplicationName> is the name of the Web application that was backed up. · <GUID> is the identifier of the backup to use for the restore operation.
Site	Site Recycle bin

Table 4: Contoso’s SharePoint 2010 Restore Scenario

SQL Backup Maintenance

SharePoint data protection is implemented through a Maintenance Plan on database servers.
The plan is designed to retain a 21 day recoverability of the databases in the event of a disaster.

The capacity of the Backup server has been planned based on assumption that total size of 21 days of backup with SQL compression enabled is less than 3x the size of the original databases.

Introducing a live maintenance plan on the secondary database server along with on-demand DFSR replication between secondary and primary backup servers allows for creation of two independent backup sets of SharePoint databases on local and remote backup servers, providing all prerequisites for retiring tape backup.

Maintenance Plan

The Microsoft maintenance plan includes the following scheduled jobs:

Back Up Database (Full)
Scope: All Databases
Backup set will expire: After 21 days
Destination: \\<BKServer>\BK
Backup Compression: On
Schedule: Occurs every week on Saturday at 6:00:00 PM
Back Up Database (Differential)
Scope: All Databases
Backup set will expire: After 21 days
Destination: \\<BKServer>\BK
Backup Compression: On
Schedule: Occurs every week on Monday, Tuesday, Wednesday, Thursday, Friday, Sunday at 6:00:00 PM
Back Up Database (Transaction Log)
Scope: All user databases not participating in log shipping
Backup set will expire: After 21 days
Destination: \\<BKServer>\BK
Backup Compression: On
Schedule: Occurs every day every 17 minute(s) between 12:00:00 AM and 11:59:59 PM
Check Database Integrity
Scope: All Databases
Include indexes
Schedule: Occurs every day at 3:00:00 AM
Shrink Database
Scope: All user databases
Limit: 102400 MB
Free space: 10%
Schedule: Occurs every week on Monday, Tuesday, Wednesday, Thursday, Friday, Sunday for 6 hour(s) between 8:00:00 PM and 1:49:59 AM
Reorganize Index
Scope: All databases
Object: Tables and views
Schedule: Occurs every week on Monday, Wednesday, Friday at 2:00:00 AM
Rebuild Index
Scope: All databases
Object Tables and views
Schedule: Occurs every week on Sunday at 2:00:00 AM
Update Statistics
Scope: All databases
Object: Tables and views
All existing statistics
Scan type: Full scan
Schedule: Occurs every week on Tuesday, Thursday at 2:00:00 AM
Clean Up History
History type: Backup, Job, Maintenance Plan
Age: Older than 1 week

This plan should be configured on all SharePoint database servers.

Monitoring

All scheduled jobs are monitored for success, warning, and error events.
Jobs 3-6 schedule could be adjusted based on monitoring results to ensure that there are no job overlapping and/or database lockups.

Backup

On the primary database server, jobs 1-3 will provide a complete backup of all databases. For all mirrored SharePoint databases the backup jobs will skip the mirror database and produce a backup of principle database only regardless of its location.

On the secondary database server, all log shipped databases are by default in standby read-only mode. The maintenance plan executed on secondary database server will create additional backup set of SharePoint databases on secondary backup server.

Restore

Restoring data from primary backup server has no specific plan, as each scenario could vary.

In event when data cannot be retrieved from database backups, or even log backups located on the primary backup server, a second set of backups could be retrieved from secondary backup server utilizing existing DFSR mechanism established between backup servers for log shipping.

Monitoring

Real-time and predictive monitoring is a key business requirement to ensure the SharePoint team is aware at all times of the health of their environments holistically. Monitoring provides the main basis for taking preventative or remedial action to ensure continued operation within accepted performance parameters.

Microsoft System Center Operations Manager (SCOM)

Microsoft uses SCOM to monitor the SharePoint environment. Error conditions and Failures regarding a particular server, service or feature, are captured and filtered into the Alert Stream and forwarded on to a System Center Operations Manager (SCOM) console.

The SCOM Management Packs (MPs) used are:

· IIS SCOM MP
· SQL SCOM MP
· SharePoint SCOM MP
· Hardware SCOM MP
· Operating System SCOM

The MP data is aggregated into configured monitors (reference: http://technet.microsoft.com/en-us/library/dd440880.aspx).

Event Based Monitoring

Events are used to capture the state of given elements of the SharePoint environment. Monitoring includes Desired Configuration Management allowing Operations to detect and monitor “drift” in the configuration from the baseline configuration deployed.

Poll Based Monitoring

Poll Based Monitoring tests performance of the SharePoint Online Service. The Operations team watching the console can be engaged real-time to return the SharePoint environment to a normal operating state. Without poll based monitoring in place, the service may experience significant performance issues or even failures without knowledge until customer complaints cause the engagement of Operations. Polling uses Microsoft internal utilities URLMon and SPMon.

· Poll based monitoring using a tool such as URLMon,
· Poll based monitoring using synthetic transactions simulated with a tool such as SPMon.
· Poll based monitoring using metrics for performance counters recorded by the Perfmon capability of Windows.

SharePoint 2010 adds new advanced health monitoring functionality and performance counters to the product; however, a complete availability and performance picture can only be seen through a set of synthetic transactions. Traditional “ping” type transactions are not effective for monitoring SharePoint. SPMon is used within Microsoft to generate the following SharePoint specific synthetic transactions:

· Get Home Page and Login
· List Creation
· List Delete
· List Item Creation / Edit
· Document library creation
· Document Library deletion
· Document Upload/Edit
· Do search query
· Verify search freshness

DPM (System Center Data Protection Manager)

There are several tools available for the backup and restore, For reference Microsoft Tool “System Center Data Protection Manager” DPM features are provided below.

Component	SharePoint backup	Microsoft SQL Server 2008 with Service Pack 1 (SP1) and Cumulative Update 2	System Center Data Protection Manager (DPM) 2010	File system backup
Environment	Yes		Yes⁶
Web application	Yes		Yes⁶
Content databases	Yes	Yes	Yes
Search and other services	Yes
Site collection	Yes^{1, 2}	Yes^{1, 2}	Yes^{1, 2}
Site	Yes²	Yes²	Yes
Document library or list	Yes²	Yes²	Yes
List item or document			Yes
Content stored in remote BLOB stores	Yes³	Yes³	Yes³
Customizations deployed as solution packages	Yes⁷	Yes⁷	Yes^{6, 7}
Changes to Web.config made by using Central Administration or an API	Yes	Yes	Yes⁴
Configuration settings (SharePoint)	Yes^{2, 8}	Yes^{2, 8}	Yes ^{2, 9}
Customizations not deployed as solution packages			Yes. Files can be recovered if protected as files.^{4, 5}	Yes
Changes to Web.config not made by using Central administration or an API			Yes⁴	Yes
IIS configurations not set through SharePoint			Yes⁵	Yes
SQL Server Reporting Services databases		Yes	Yes

Table 13: Summary of backup strategies in SharePoint Server 2010

1Environment-level and database-level backup and restore can be used for site collection recovery if a single site collection is stored in a database.

2Environment-level and database-level backups can be used with SharePoint Server unattached database recovery to restore site collections, sites, lists, and configurations.

3Content stored in remote BLOB stores is backed up and restored with other content, as long as the Remote BLOB Storage (RBS) provider in use has this capability.

4Changes to Web.config can be backed up by using file system backup from DPM 2010.

5IIS configurations can be recovered by using a bare metal backup from DPM 2010.

6 DPM 2010 can recover this item by using a combination of a bare metal backup and SharePoint Server backup. It cannot be backed up and recovered as an object.

7Fully-trusted solution packages are stored in the configuration database, and sandboxed solutions are stored in content databases. They can be recovered as part of Environment or content database recovery.

8Configuration settings can be recovered from Environment-level backups.

9The Central Administration content database and the configuration database for a SharePoint Server 2010 Environment can be recovered but only as part of a full-Environment recovery to the same Environment, with the same computers.

Search and Replace Retention tag on Microsoft Exchange 2010 (MRM)

2011-10-20T01:45:00Z

Messaging Records Management
development on Retention Policy Tag

Introduction

Microsoft introduces the messaging record management (MRM). In Exchange 2010, which helps the organizations to reduce legal risks associated with e-mail and other communications.

MRM makes it easier to keep messages needed to comply with company policy, government regulations, or legal needs, and to remove content that has no legal or business value.

To learn more about MRM and how to set it up, please visit http://technet.microsoft.com/en-us/library/dd297955.aspx

In this article, will go over what are retention policy tags? How it is being internally stored in MAPI objects? How programmatically change the retention policy tags in each item of the exchange.

What are retention policy tags?

MRM in Exchange 2010 is accomplished by using retention tags and retention policies. Before discussing the details about each of these retention features, it's important to learn how the features are used in the overall Exchange 2010 MRM strategy. This strategy is based on:

Assigning retention policy tags (RPTs) to default folders, such as the Inbox
Applying a default policy tag (DPT) to mailboxes to manage the retention of all untagged items
Allowing the user to assign personal tags to custom folders and individual items
Separating MRM functionality from users' Inbox management and filing habits.
- Users
  aren't required to file messages in managed folders based on retention
  requirements
- Individual
  messages can have a different retention tag than the one applied to the folder
  in which they're located

Figure 1 illustrates
the tasks involved in implementing this strategy

Figure 1

Ref: http://technet.microsoft.com/en-us/library/dd297955.aspx

Retention Tags Types (Reference: http://technet.microsoft.com/en-us/library/dd297955.aspx)

As illustrated in Figure 1, retention tags are used to apply retention settings to folders and individual items such as messages, notes, and contacts. These settings specify what policy a message should have, how long a message remains in a mailbox and the action to be taken when the message reaches the specified retention age. When a message reaches its retention age, it's moved to the personal archive, deleted, or flagged for user attention.

Unlike managed folders (the MRM feature introduced in Exchange 2007), retention tags allow users to tag mailbox folders and individual items for retention. Users no longer have to file items in managed folders based on message retention requirements.

Retention tag configuration objects are stored in Active Directory (AD) in the Retention Policy Tag Container, which is located under the organization container. There are three types of retention tags:

Retention Policy Tags
Default Policy Tags
Personal Tags

Retention Policy Tags (RPTs)

RPTs apply retention settings to default folders such as Inbox, Deleted Items, and Sent Items. Mailbox items in a default folder that have an RPT applied inherit the folder's tag. Although users can't apply a different tag to a default folder, they can apply a different tag to the items in a default folder.

You can create RPTs for the following default folders:

Calendar (requires Exchange 2010 SP1)	Notes (requires Exchange 2010 SP1)
Deleted Items	Outbox
Drafts	Sent Items
Inbox	RSS Feeds
Journal	Sync Issues
Junk E-mail	Conversation History

Important:

You can't include more than one RPT for the same
   default folder type in one retention policy. For example, if a retention
   policy has an Inbox tag, you can't add another RPT of type Inbox to that
   retention policy.

In Exchange 2010 RTM, RPTs are not supported for the Calendar and Notes default folders. In Exchange 2010 RTM and SP1, retention tags are not applied to Contacts.

Default Policy Tags (DPTs)

DPTs apply retention settings to untagged mailbox items. Untagged items are mailbox items that do not already have a retention tag applied, either by inheritance from the folder in which they are located or by the user. A retention policy cannot contain more than one DPT with the same action.

Personal Tags

Personal tags are available to Outlook 2010 and Outlook Web App users as part of their retention policy. Users can apply personal tags to folders they create or to individual items, even if those items already have a different tag applied.

Retention Policy Tag Attributes

MRM depends on the Active Directory (AD) directory to store configuration object settings used for driving Managed Folder assistant provisioning and retention actions. These configuration settings are retrieved and set using Exchange management tasks from EMS, EMC and ECP

Retention tag configuration objects are of objectClass msExchELCFolder and are located in the Retention Policy Tag Container directly under the Organization container. msExchgELCFlags, and msExchELCFolderTypes are the attributes used by MRM. For each retention tag there is a Content
Settings configuration object of objectClass msExchELCContentSettings that is a direct child object of the retention tag configuration object. Table 1 describes the attributes used by MRM.

Table 1: Content Setting Configuration Attributes

Attribute
msExchELCAutoCopyAddressLink
msExchELCExpiryAction
msExchELCExpiryAgeLimit
msExchELCFlags
msExchELCLabel
msExchELCMessageClass

Retention Policy Attributes

Retention policy configuration objects are of objectClass msExchRecipientTemplate and are located in the Retention Policies Container directly under the Organization container. Table 2 describes the attributes used by MRM.

Table 3: Retention Policy Configuration Attributes

Attribute
msExchELCFolderLink
msExchMailboxTemplateBL
msExchMinAdminVersion
msExchRecipientTemplateFlags

Mailbox Objects

User’s mailbox contains Information specific to MRM operation. This information is used by the Managed Folder Assistant for processing retention and archival actions, and by mailbox clients for displaying and managing relevant retention and archival elements in the user interface.

Configuration Message

The MRM configuration information for a mailbox to which a retention policy has been applied is stored in a hidden Folder Associated Item (FAI) message of type IPM.Configuration.MRM, located in the Associated Contents table of the Inbox folder. This message in the mailbox is not visible in Outlook or OWA to the end user. MRM uses the FAI item to communicate the Retention Policy to the mailbox client, and for processing retention tags that have been self-provisioned by the user.

The configuration information is stored as XML formatted data in property PR_ROAMING_XMLSTREAM (0x7C080102). This information can be examined using the MAPI Editor tool (MFCMAPI).

Important:

Once a
   retention tag is applied to a mailbox and the information for that tag is
   written to the FAI message, the information remains even if the tag is
   un-provisioned by the user or the tag is no longer applied by policy. When
   this occurs the IsVisible attribute is set to False so the user no longer
   sees the tag from the client.

However, the
   Managed Folder Assistant continues to process any messages that are tagged
   with the hidden retention tag as long as the configuration object for the
   tag is still available in AD. When the tag is removed from AD, the
   information for the tag is also removed from all FAI messages by the Managed
   Folder Assistant.

Updating the Configuration Message

The configuration information is updated by one of three
ways:

Managed Folder Assistant – The Managed Folder Assistant creates the configuration message when a retention policy is first applied to the mailbox, and updates the message anytime changes are made to the retention tags or retention policy applied to the mailbox.
Set-RetentionPolicyTag – This cmdlet makes it possible for the administrator to assign additional retention tags that are not already included in the retention policy applied to the mailbox. When the administrator adds or removes retention tags via this command, the configuration message is updated.
Self-Provisioning of Retention Policies via ECP – When a user adds or removes retention tags via the ECP, the configuration message is updated. The administrator can retrieve a list of all retention policies applied to a mailbox by policy assignment, administrative assignment or self-provisioning by using the Get-RetentionPolicyTag cmdlet with the Mailbox parameter. The command reads directly from the configuration message to retrieve the current settings. Because of this the mailbox must be available for this operation to succeed.

Folder Properties

MRM stores retention settings as MAPI property values on mailbox folders. Table 4 describes these property values.

Table 4: Retention Setting Properties on Mailbox Folders

Property Name	propTagID	Type	Stamping Details
StartDateEtc PR_START_DATE_ETC	0x30190102	GUID	The GUID of the retention tag applied to the folder. Stamped on all folders. Outlook changes this property when the user explicitly tags a folder. Outlook updates all the subfolders with this tag’s GUID as well.
RetentionPeriod PR_RETENTION_PERIOD	0x301A0003	Number Of Days	The retention period for keeping the item (if it’s a value like 0 or -1 it means never expire). Stamped on all folders. Outlook changes this property when the user explicitly tags a folder. Outlook updates all the subfolders with the retention period as well.
RetentionFlags PR_RETENTION_FLAGS	0x301D0003	Number	The property representing if the recipient tag is inherited from a parent folder. Outlook stamps the property in offline/cached mode, Exchange stamps for online mode. If the user tags a folder, RetentionFlags is 0 for each sub-folder and non-zero for the folder that is explicitly tagged. If the least significant bit is 0 or if the property is not present, then the tag is implicit. See RetentionFlags table for details.

Item Properties

MRM stores retention settings as MAPI property values on mailbox items. Table 5 describes these property values.

Table 5: Retention Setting Properties on Items

Property Name	propTagID	Type	Stamping Details
StartDateEtc PR_START_DATE_ETC	0x301B0102	DateTime	This is a composite prop containing the Start Time and Default Retention Period. The first 4 bytes is the default retention period and the next 8 bytes is the date stamped on every item. Exchange Event-Based MRM Assistant sets this property. Outlook changes if necessary (Calendar and Task Item) Add the policy length to the start time to calculate expiry date.
RetentionPeriod PR_RETENTION_PERIOD	0x301A0003	Number (In Days)	The time period for keeping the item (if it’s a value like 0 or -1 it means never expire). Stamped only when item is explicitly tagged. Outlook changes the value when explicitly tagged.
DefaultRetentionPeriod	<same as Start Time>	Number	The length to retain an item if it is under the default policy (varies based on message class). Stamped on every item by Exchange Event-Based/Time-Based Assistant. Never stamped by Outlook.
RetentionDate PR_RETENTION_DATE	0x301C0040	DateTime	DateTime for item expiration. This is a calculated prop when cached or offline and a property stamped by Exchange when online. Stamped on every item. Exchange stamps when online, Outlook calculates when cached or offline.
PolicyTag PR_POLICY_TAG	0x30190102	GUID	The retention policy an item is under (either implicit or explicit). Stamped on every item. Outlook calculates when cached or offline. Exchange stamps when online.
RetentionFlags PR_RETENTION_FLAGS	0x301D0003	Number	The property representing if the recipient tag is inherited from a parent folder. Outlook stamps the property in offline/cached mode, Exchange stamps for online mode. If the user tags an item, RetentionFlags is non-zero for the item that is explicitly tagged. If the least significant bit is 0 or if the property is not present, then the tag is implicit. See RetentionFlags table for details.
ArchiveTag PR_ARCHIVE_TAG	0x30180102	GUID	The archive policy an item is under (either implicit or explicit). Stamped on every item. Outlook calculates when cached or offline. Exchange stamps when online.
ArchivePeriod PR_ARCHIVE_PERIOD	0x301E0003	Number (In Days)	The time period for archiving the item (if it’s a value like 0 or -1 it means never expire). Stamped only when item is explicitly tagged. Outlook changes the value when explicitly tagged.
ArchiveDate PR_ARCHIVE_DATE	0x301F0040	DateTime	DateTime for item archival. This is a calculated prop when cached or offline and a property stamped by Exchange when online. Stamped on every item. Exchange stamps when online, Outlook calculates when cached or offline.

Table 5 describes the values used for the RetentionFlags property on mailbox folders and items.

Table 5: RetentionFlags Bitmask Values

RetentionFlags	Bitmask	Hex	Dec
None	00000000 00000000	0x00	0
ExplicitTag	00000000 00000001	0x01	1
UserOverride	00000000 00000010	0x02	2
Autotag	00000000 00000100	0x04	4
PersonalTag	00000000 00001000	0x08	8
AllRetentionFlags	00000000 00001111	0x0F	15
ExplictArchiveTag	00000000 00010000	0x10	16
KeepInPlace	00000000 00100000	0x20	32
AllArchiveFlags	00000000 00110000	0x30	48
SystemData	00000000 01000000	0x40	64
NeedsRescan	00000000 10000000	0x80	128
PendingRescan	00000001 00000000	0x100	256

Exchange Web Service

Microsoft Exchange Server 2010 provides Exchange Web Services as an extensibility point for clients that connect to the Exchange server and consume information about user availability, and the manipulation of items that are located in the Exchange data store.

To learn more about Exchange web service please click here http://msdn.microsoft.com/en-us/library/bb204119.aspx

Programmatically change the retention policy tag on the Exchange

Business Scenario

For my current customer, we had a scenario, using PowerShell the admin needs to

Search for all items which have a particular retention tag assigned AND belong to a specified policy
Specify the new retention tag in that command and have it assigned to the found mail items

We don’t have out-of-box functionality in Exchange 2010 SP1 for the above requirement. Hence, I have to do custom coding.

Solution

The custom script created as a framework. Using this script admin can search and replace the retention policy tag. Also, just by modifying the script, admin can search on how many users having certain policy tags, get the policy tag for the specific user, get the policy tag for the specific folder, and get the expiration date.

Below is the conceptual architecture, and code.

Pre-requisite

Only admin can run this utility as it needs impersonation (to search on other mailboxes)
PowerShell 2.0
Exchange 2010 SP2 Web Service SDK
Replaceable tag must be part of the same retention policy.

Conceptual Architecture

Figure 2

As shown in the Figure 2, PowerShell will read the users from the UserAccounts.txt file. Admin will be providing the description of the retention policy tag, but internally MAPI stores only the GUID of the tag. Hence, there will be a look-up function to match the tag description with the GUID, for both search and replace tags. Then
the Exchange Web Service (EWS) will be invoked to impersonate each user’s mailbox and search all the folders. Upon search, if match found the GUID will be replaced with the replace tag GUID provided by the admin. Report of the operations will be written in the SharePoint.

Code

#This Script is to replace the retention policyTag

#Parmamters

param($args1, $args2)

$SearchIdTag = $args1

$ReplaceTag = $args2

#Lookup Function to get the GUID of the retention policy tag description

Function ConvertIdentityToGuid($IdTag)

#Remote Session to get the exact GUID and description of the PolicyTag

   #$userCredential = get-Credential

   $username = "DOMAIN\SuperUser"

   $password = convertTo-SecureString "password" -AsPlainText -Force

   $cred = New-Object System.Management.Automation.PSCredential($username, $password)

   $session=New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri http://CasServer/PowerShell/ -Credential $cred

   Import-PSSession $session -AllowClobber

   $GuidTag = Get-RetentionPolicyTag -Identity $IdTag | Select GUID

   $Tag=$GuidTag.Guid

   $GuidPolicyTag = $Tag

   get-Pssession |remove-Pssession

   Write-output $GuidPolicyTag

#Replace the tag

Function ReplacePolicyTagOnItem($item)

  $GuidReplacePolicyTagArray = ConvertIdentityToGuid($ReplaceTag);

  $ReplaceGuidTag = $GuidReplacePolicyTagArray[1]

  #PR_POLICY_TAG 0x3019

  $PolicyTag = New-Object Microsoft.Exchange.WebServices.Data.ExtendedPropertyDefinition(0x3019,[Microsoft.Exchange.WebServices.Data.MapiPropertyType]::Binary)

  $PolicyTagGUID = New-Object GUID($ReplaceGuidTag)

  $enumAlwaysOverWrite = [Microsoft.Exchange.WebServices.Data.ConflictResolutionMode]::AlwaysOverWrite

  $item.SetExtendedProperty($PolicyTag,$PolicyTagGuid.ToByteArray())

  $item.Update($enumAlwaysOverWrite)

  #Write the report to SharePoint List

  $WriteToSP = $global:AuditLog.WriteAuditLog("RetentionTagReplaced",$MailboxName,$item.Subject ,"User" ,$global:FolderName , $args1, $args2, "Item", "")

  write-host "SharePoint: " $WriteToSP

#Search Function

Function SearchPolicyTagOnItem($SearchTag)

    $fvItemView = New-Object Microsoft.Exchange.WebServices.Data.ItemView(10000)

    $ItmpsPropertySet = new-object Microsoft.Exchange.WebServices.Data.PropertySet([Microsoft.Exchange.WebServices.Data.BasePropertySet]::FirstClassProperties)

    #PR_POLICY_TAG 0x3019

    $ItmPolicyTag = New-Object Microsoft.Exchange.WebServices.Data.ExtendedPropertyDefinition(0x3019,[Microsoft.Exchange.WebServices.Data.MapiPropertyType]::Binary)

    $ItmpsPropertySet.Add($ItmPolicyTag)

    $fvItemView.PropertySet = $ItmpsPropertySet;

    #Loop through  the folders

    while (($results = $ffFolder.FindItems($fvItemView)).Items.Count -gt 0)

        #Loop through each item in the folder

             foreach ($item in $results)

                $ItemPolicyGuid=$null;

                if ($item.TryGetProperty($ItmPolicyTag,[ref] $ItemPolicyGuid))

                  $Itmptag = [GUID]$ItemPolicyGuid

                  if ($SearchTag -eq ($Itmptag))

                      "Item Subject: " + $item.Subject

                      "Item  Retention Policy Tag:" +$Itmptag

                      ReplacePolicyTagOnItem($item)

            $offset += $results.Items.Count

            $fvItemView = new-object Microsoft.Exchange.WebServices.Data.ItemView(100, $offset)

# Script starts here

#Global declaration

$global:AuditLog = New-Object RetentionTagAuditLog.AuditLog

$global:FolderName = $null

#Get the GUID of the provided retention policy tag description

$GuidPolicyTagArray = ConvertIdentityToGuid($SearchIdTag);

#SharePoint assembly referenced for Reports

[System.Reflection.Assembly]::LoadFrom("G:\PowerShell\RetentionTagAuditLog.dll")

#Read data from the UserAccounts.txt.

#Make sure this file exists in the same location as the script or provide the path

import-csv AliasAccounts.txt | foreach-object

    $MailboxName = $_.EmailAddress.ToString()

    #Load the Exchange Web Service

    $dllpath = "C:\Program Files\Microsoft\Exchange\Web Services\1.1\Microsoft.Exchange.WebServices.dll"

    [void][Reflection.Assembly]::LoadFile($dllpath)

    $service = New-Object Microsoft.Exchange.WebServices.Data.ExchangeService([Microsoft.Exchange.WebServices.Data.ExchangeVersion]::Exchange2010_SP1)

    #imepersonation - need to have admin rights

    $service.ImpersonatedUserId = new-object Microsoft.Exchange.WebServices.Data.ImpersonatedUserId([Microsoft.Exchange.WebServices.Data.ConnectingIdType]::SmtpAddress,$MailboxName);

    #Get current user

    $windowsIdentity = [System.Security.Principal.WindowsIdentity]::GetCurrent()

    $sidbind = "LDAP://<SID=" + $windowsIdentity.user.Value.ToString() + ">"

    $aceuser = [ADSI]$sidbind

    $service.AutodiscoverUrl($aceuser.mail.ToString(),{$true})

    $GuidPolicyTag = $GuidPolicyTagArray[1]

    "Checking : " + $MailboxName

    $folderidcnt = new-object Microsoft.Exchange.WebServices.Data.FolderId([Microsoft.Exchange.WebServices.Data.WellKnownFolderName]::Root,$MailboxName)

    $fvFolderView = New-Object Microsoft.Exchange.WebServices.Data.FolderView(10000)

    $fvFolderView.Traversal = [Microsoft.Exchange.WebServices.Data.FolderTraversal]::Deep;

    $psPropertySet = new-object Microsoft.Exchange.WebServices.Data.PropertySet([Microsoft.Exchange.WebServices.Data.BasePropertySet]::FirstClassProperties)

#PR_POLICY_TAG 0x3019

    $PolicyTag = New-Object Microsoft.Exchange.WebServices.Data.ExtendedPropertyDefinition(0x3019,[Microsoft.Exchange.WebServices.Data.MapiPropertyType]::Binary)

    $psPropertySet.Add($PolicyTag)

    $fvFolderView.PropertySet = $psPropertySet;

    $fiResult = $Service.FindFolders($folderidcnt,$fvFolderView)

    #Loop through each folder in the mailbox

    foreach($ffFolder in $fiResult.Folders)

        $PolicyGuid=$null;

        if ($ffFolder.TryGetProperty($PolicyTag,[ref] $PolicyGuid))

          $ptag = [GUID]$PolicyGuid

          if ($ptag -eq $GuidPolicyTag)

              $global:FolderName = $ffFolder.DisplayName

              $WriteToSP = $global:AuditLog.WriteAuditLog("RetentionTagReplace",$MailboxName,"" ,"User" , $global:FolderName, $args1, $args2, "Folder", "")

        "Traversing folder: " + $ffFolder.DisplayName

        $global:FolderName = $ffFolder.DisplayName

        SearchPolicyTagOnItem($GuidPolicyTag)

    # catch excption

    trap [System.Exception]

     write-host ("Error: " +$_.Exception.Message)

     #write in log file

     continue

Screen Shots

1. Email with a retention policy tag

2. Mapi Properties of the message item. Please see the PR_Policy_Tag

3. Retention Policy Tag MAPI properties

4. Run the script

5. After running the script

6. Verify the MAPI Properties

7. The Retention Policy tag value has been changed

Reference

http://blogs.msdn.com/b/akashb/archive/2011/08/11/stamping-retention-policy-tag-using-ews-managed-api-1-1-from-powershell-exchange-2010.aspx

SharePoint 2010 Web Analytic Service Application and SQL Authentication

2010-03-22T21:41:00Z

Microsoft official guideline is to support SQL authentication for SharePoint 2010. Being said that on a RC release (i am using 4747),"WebAnalyticServiceApplication" does not have an option to configure the SQL authentication (both on UI and PowerShell cmdlets). However, other serivces (except WebAnalyticServiceApplication), seems to support SQL authentication.

SharePoint 2010 Configuration with PowerShell and Untrusted SQL domain (SQL Authentication)

2010-03-17T18:29:00Z

This blog will provide step by step instruction for configuring SharePoint 2010 with SQL Authentication using PowerShell. Eric Kraus has provided very good article about configuring SharePoint 2010 with PowerShell. This blog is just extension of it using SQL Authentication and adding servers to the existing farm.

I am using SharePoint 2010 RC (4747), SQL Server 2008 SP1 +CU, and Windows Server 2008 R2.

Before going into the configuration, we need setup accounts (farm accounts, db accounts) and some housekeeping.

My architecture is SharePoint WFEs (2 server) are in DMZ connected to my domain. However, SQL server is on untrusted domain.

1. Configure SQL Server to use mixed authentication. You can configure SQL Server to used mixed authentication when you install SQL Server. For information about how to change the authentication mode after you install SQL Server, visit the following Microsoft Web site: http://msdn2.microsoft.com/en-us/library/ms188670.aspx (http://msdn2.microsoft.com/en-us/library/ms188670.aspx)

2. Add a new SQL Server account in Microsoft SQL Server 2008 SP1+CU. Then, grant the roles of security administrator and database creator to the account. To do this, follow these steps:

a. Click Start, point to All Programs, point to Microsoft SQL Server 2005, and then click SQL Server Management Studio.

b. In SQL Server Management Studio, expand Security, right-click Logins, and then click New Login.

c. In the Login - New dialog box, type the name of the SQL Server account, click SQL Server Authentication, type the password, and then click Server Roles.

d. In the results pane, click to select the following check boxes, and then click OK:

§ dbcreator

§ securityadmin

3. Create a service domain account with a role “Needs to log on Batch job”.

Please refer to this blog http://www.cleverworkarounds.com/2008/09/16/sometimes-microsoft-bashing-is-justified/ for more on why we need log on batch job role in domain account.

I am NOT talking about configuring windows firewall for SQL in this blog.

If you are interested on configuring windows firewall please refer

Ø How to: Configure a Windows Firewall for Database Engine Access

o http://msdn.microsoft.com/en-us/library/ms175043.aspx

Ø How to: Configure a Server to Listen on a Specific TCP Port (SQL Server Configuration Manager)

o http://msdn.microsoft.com/en-us/library/ms177440.aspx

Ø SQL Server Browser Service

o http://msdn.microsoft.com/en-us/library/ms181087.aspx

Ø Configuring the Windows Firewall to Allow SQL Server Access

http://msdn.microsoft.com/en-us/library/cc646023.aspx

Now, we created accounts, will go to SharePoint installation and configuration.

After installing SharePoint 2010 pre-requisites, start SharePoint install.

I want to create a farm rather standalone so during the SharePoint Installation, i choose “Server Farm” and then “Complete” install.

After the install completes, the setup program will ask you if want to run the SharePoint Technologies configuration wizard – uncheck the box. We don’t want to run the wizard.

On StartàMicrosoft SharePoint 2010 ProductsàRight click on SharePoint 2010 Management Shell and choose “Run as administrator”

As we haven’t configure the farm yet, PowerShell will give the following error – it is ok and ignore it.

Figure 1

Now we need to run the script

· $dbcredential = New-Object –typename System.Management.Automation.PSCredential –argumentlist “Moss_SPAdmin”, (ConvertTo-secureString “password” –AsPlainText –Force)

· New-SPConfigurationDatabase –DatabaseName “SharePoint2010_Config” –DatabaseServer “<db server>” –AdministrationContentDatabaseName “SharePoint2010_Admin_Content” –Passphrase (ConvertTo-SecureString “pass@word1” –AsPlaintext –Force) –FarmCredentials (Get-Credential) –DatabaseCredentials $dbcredential

NOTE: Get-Credential will prompt for userid/password, where you can provide domain\userId and password. However, if you don’t provide domain name it will interrupt it as “\userid”, which will cause problem for my dbcredential as I don’t have domain. Hence, I created $dbcredential to hardcode userid/password.

Figure 2

Script may take couple of minutes as it has to create Dbs and stored procedure. After the process is run and you get prompt in PowerShell,

You can do either of the following option to verify farm creation

1. you can restart the PowerShell to verify the farm has been created (Now you should see any error or warning on the PowerShell)

2.    #verifying farm creation
        $spfarm = Get-SPFarm -ErrorAction SilentlyContinue -ErrorVariable err
        if ($spfarm -eq $null -or $err) {
            throw "Unable to verify farm creation."
        }

Figure 3

Install-SPHelpCollection –ALL

Intialize-SPResourceSecurity

Install-SPService

Figure 4

Install-SPFeature –AllExistingFeatures

Figure 5

Figure 6

New-SPCentralAdministration –Port 1234 –WindowProvider “NTLM”

Install-SPApplicationContent

Figure 7

I did NOT run “DisableLoopbackCheck” as I am pretending this is as my production server. To learn more about “DisableLoopbackCheck”, please refer http://www.harbar.net/archive/2009/07/02/disableloopbackcheck-amp-sharepoint-what-every-admin-and-developer-should-know.aspx

DONE!!. SharePoint farm created with Central admin

Now, I need to add another server for the farm

After installing Pre-requisite and SharePoint 2010 (in my case RC -4747)

Run the following command

· $dbcredential = New-Object –typename System.Management.Automation.PSCredential –argumentlist “SPAdmin”, (ConvertTo-secureString “password” –AsPlainText –Force)

· Connect-SPConfigurationDatabase –DatabaseName “SharePoint2010_Config” –DatabaseServer “<db server>”–Passphrase (ConvertTo-SecureString “pass@word1” –AsPlaintext –Force) –FarmCredentials (Get-Credential) –DatabaseCredentials $dbcredential

Intialize-SPResourceSecurity

Install-SPService

Install-SPFeature –AllExistingFeatures

Install-SPApplicationContent

NOTE: There is a SPModule http://sharepoint.microsoft.com/blogs/zach/Script%20Library/Modules/SPModule/SPModule.zip available which do all the above work with script (“Install-SharePoint, New-SharePointFarm, Join-SharePointFarm, etc.”). However, if anyone wants to use the manual way of configuration, hope this will help them out. Zack Rosenfield has provided very detailed description about SPModule here (http://sharepoint.microsoft.com/blogs/zach/Lists/Posts/Post.aspx?ID=54)

Kerberos Authentication Problem with Active Directory

2009-04-06T19:29:00Z

Recently I had to work with Kerberos and we faced the following problems.

1. Kerberos consistently NOT work for some user(s) throwing “400 Bad Request” error.

2. Kerberos works intermittently. In other words, user will not be authenticated on Kerberos (falls back to NTLM) for 5 minutes or so (no definite period) and then automatically Kerberos will start working for that user. However this won’t happen for all users at the same time.

3. “temp-id” always works for everybody.

Due to this problem, some of the applications heavily dependent on Kerberos such as Federated search, SAP Integration, and RSS Viewer Web Parts will fail; as Kerberos authentication falls back to NTLM.

After research, we found out the problem is in AD (Active Directory), as user belongs to many groups.

1. Joe Doe (for the purpose of the blog, will go with this id) is member of 123 groups in Active Directory.

2. Other users’ regular id’s such were member of 10 groups in Active directory.

a. Note: These 10 groups have nested groups – so it is not really 10 groups, it could be more.

3. “temp-id” which always works for everybody including Joe Doe is a member of 2 groups (Domain users and SharePoint developers)

Scenario 1: Kerberos consistently NOT working for some user(s) throwing “400 Bad Request” error

Why Joe Doe’s regular id did not work

Joe Doe’s regular id did not work because of his token size and header size. His token size was about 11K. However his “temp- id’ token size is 200 bytes. Hence, it is problem with token size and header size. In below section, I will try to explain what that it means and how it is related to Kerberos problem.

What does header size means

HTTP requests/responses contain two parts: the header and the body. The header contains most of time technical information exchanged between the client and the server, the body contains user-oriented information like the content of a webpage or a file to download, for example. The error message ere above is therefore generated by the server because the request sent by the client contains a header that is simply too large compared to what the server expects.

The factors that makes header section large will depends on how browser was configured (and the underlying OS as well in some case), but most of time, the culprits of larger header are cookies (header: Cookie) and authentication information (Header: Authorization).

When this error is experienced on an Internet site, there is not much we can do except cleaning up our cookies and hoping it will work afterwards while.
When this error shows up in an intranet environment when web servers are running IIS and possibly SharePoint, SQL Server Reporting Services or Exchange with OWA on top of it, this is caused by a combination of multiple factors which are the following:

· The web server (or web site) was configured to use Integrated Windows Authentication and Kerberos in particular.

· The client is able to authenticate using Kerberos (the client system is member of an AD forest, the user too)

· The size of user’s security token is large. By default, token size is 12000 bytes. However, Joe Doe’s token size was around 14k bytes . This can be caused by

o The user being member of many AD groups (hundreds of groups)

o The user’s object in AD contains SID (Security Identifier) history information as consequence of a domain migration/consolidation.

o The group the user is member of is also affected by SID history, just like the user.

o IIS is configured by default.

· The client sends Kerberos –based authentication AND authorization information. Unlike NTLM and Basic which only send authentication information, therefore smaller, Kerberos includes information such as group membership and SID history information in the request’s header.

Hence depending on the user’s group membership and SID history information, some users may be affected and other not.

How is it related to Kerberos Problem (specifically Joe Doe)

As mentioned above, Joe Doe is having more than 100 hundreds group, his regular id is not working for him.

Solution for Joe Doe’s problem

Hopefully, there are many solutions to work around this problem but all of them have their trade-offs:

a. Use IP address in the URL instead of host names
Since Kerberos solely works with host names, using UIP addresses will automatically force negotiation to NTLM instead. Of course, this does not only degrades security but it also involves hard coding IP’s, which is rarely practically and also means that this can work if you only host one and only one web site on their IIS, finally, if application uses “delegation” of credentials, which is a Kerberos feature, it will not work anymore (I am thinking about SQL SRS in particular in this case, or even Exchange OWA when used in FE-BE scenario’s)

Since most of the companies will have more than one web application, we cannot go with this approach.

b. Configure IIS to use NTLM only

As part of the integrated windows Authentication setup, you can simply configure IIS to use NTLM only; the following MS KB article will show you how to do so: http://support.microsoft.com/kb/215383.

Although this configuration is less impacting that the first one, it is still left with lower security as well as compatibility issue with Kerberos is required by its application.

Kerberos depended application such as Federated Search, SAP Integration, Rss Web part won’t work. Federated Search may work with NTLM by setting up with one ID (hard-coding Id) and will miss the security trimming, which is big.

c. Configure IIS to accept larger headers

You can do so by configuring IIS in registry. It is important to note that this configuration will apply to all web sites running on the system running IIS because this settings is used at kernel-component level (http.sys), it will therefore impact all ECM applications on that server.

MS KB article explains all those settings: http://support.microsoft.com/kb/820129

The 2 registry keys to fix this issue are:

· HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\HTTP\Parameters\

MaxFieldLength
Default Value: 16384
Min – Max Value to set: 64 - 65534 (64kb) bytes
Sets an upper limit for each header. See MaxRequestBytes. This limit translates to approximately 32k characters for a URL.

· HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\HTTP\Parameters\

MaxRequestBytes
Default value: 16384
Min-Max value to set: 256 - 16777216 (16MB) bytes
Determines the upper limit for the total size of the Request line and the headers.
Its default setting is 16KB.

If this value is lower than MaxFieldLength, the MaxFieldLength value is adjusted.

As KB article suggest, this has to be done after very careful design and thought, because this will increase the memory used by the system (kernel memory) to handle requests. On a “busy” (read: getting a lot of requests, not fewer large requests) 32-bit system, this can exhaust kernel memory. If the boot.ini switch /3GB is used (possibility combined to /USERVA), the situation can get worse since less memory is available to the kernel. On a 64-bit system, this configuration is harmless, even if the application is running in 32-bit mode, since this is handled in kernel mode. Of course, take care of what ECM application is doing with those headers too.

d. Cleanup AD users
by reducing (read optimizing) their group membership and removing SID history information from both user’s and group’s AD object attribute. Though this solution will be profitable in all scenario and not only web authentication (faster logon, less memory usage on application servers, Exchange mailbox servers…), you need to implement with a careful impact assessment.

MS KB article explains how to automate this task: http://support.microsoft.com/kb/295758

Token Size Problem

This is another problem with user having larger (more than 70) groups in AD.

The Kerberos token has a fixed size. If a user is a member of a group directly or through group nesting (which is mostly likely case here) the SID for that group is added to the user’s token. Once a SID is added to the users token it is passed via the Kerberos token during each authentication. If the required SID information exceeds the size of the token, authentication does not succeed.

How the Access Token Limitation Problem Can Occur

Any entity that can be authenticated by the security system in an Active Directory environment is referred to as a security principal. A user is an example of a security principal. A security context is information that describes the identity and capabilities of a security principal on a computer. In Windows Server 2003 all activities take place in a security context.

The security context of a security principal is represented by an access token. The access token includes a list of security identifiers (SIDs) and there is a limit (1,024) to the number of SIDs the token can contain. If this limit is exceeded, a denial of service, such as a user not being able to log on, can occur.

This section describes the following:

· How access tokens are created

· How the access token limit is reached.

· Symptoms that indicate that the access token limitation has been reached.

How Access Tokens Are Created

An access token is created whenever a user or any security principal logs on to a computer, or attempts to access a resource, as part of the authentication process. An access token contains information about the identity and privileges associated with the security principal (user, group, computer, or domain controller). Every process has a token that describes the security context of the principal's account associated with the process.

A security identifier (SID) is a unique value that identifies a security principal. A SID is issued to every security principal when it is created. Security groups are also security principals, and therefore are uniquely identified by SIDs. A user security principal can be a member of multiple security groups. Consequently, a user’s access token includes SIDs of all groups to which the user is a member.

In the following example, during Windows-based authentication, an access token is created when a user logs on in the following manner:

1. When a user logs on interactively or tries to make a network connection to a computer running Windows, the user’s logon credentials are authenticated.

2. If authentication is successful, the logon process returns a SID for the user and a list of SIDs for the user’s security group membership.

3. The Local Security Authority (LSA) on the computer uses this information to create an access token that includes the SIDs returned by the logon process. The token also includes a list of privileges assigned by local security policy to the user and to the user’s security groups. The LSA uses process called “Token evaluation” to determine which security groups to include in the token.

Note: Specific protocols like NTLM and Kerberos use different processes to create an access token.

This process of acquiring the SIDs for the user and user's group memberships is called the "token evaluation process."

Factors Affecting Token Evaluation

Several factors can affect the outcome of the token evaluation process, including the following:

· Whether the token is issued for logon purposes or for resource access.

· The groups that the principal is a member of, including direct and transitive memberships.

· The types of groups involved.

· There are two types of groups in Active Directory: distribution groups and security groups. Distribution groups are not included in the principal’s token, but all security groups are included. All group scopes (universal, global, domain local, machine local, and built-in) are included in the token evaluation.

· The functional level (for Windows server 2003)

The token evaluation process evaluates groups’ recursively. For example, if User A is a member of Group 1 and Group 1 is a member of Group 2, then a token generated for User A contains SIDs representing both Group 1 and Group 2. In native mode and higher domains, universal, global, and domain local groups are all evaluated recursively. Universal security groups do not exist in mixed mode domains.

How SIDs Are Added to a Token

The examples in this section show how SIDs are added to a user's token in two instances:

· When the user logs on

· When the user accesses a resource.

For each of these instances, the process is described for both NTLM and Kerberos authentication in the following sections.

How SIDs Are Added When the User Logs on to a Network

The following figure shows how SID(s) is added to a user's token when the user attempts to log on with NTLM authentication.

When the user attempts to log on to a network with NTLM authentication, the following process occurs:

1. The workstation collects the user’s credentials and passes them to a domain controller in the account domain.

2. The domain controller in the account domain adds global groups to the user’s token and passes the updated token list to the account domain global catalog server.

3. The workstation receives the list of SIDs and retrieves all of the local groups. The resulting union is the SIDs in the user token.

When the user attempts to log on in an environment with Kerberos authentication, the following process occurs:

1. The Kerberos client on the workstation uses the credentials from the user to request a Ticket Granting Ticket (TGT) from the Kerberos Key Distribution Center (KDC) in the user's domain.

2. The KDC obtains the list of the user's SIDs from a domain controller in the user's account domain. The KDC also queries the global catalog server and obtains any universal groups that include the user or the user's domain security groups. The KDC adds the user's SIDs and the SIDs from any applicable universal groups to the list in the TGT's authorization data field, and returns the TGT to the computer.

3. Once the TGT is received, the Kerberos Client requests a service ticket for access to the local workstation

4. The KDC copies the contents of the TGT's authorization data field to the service ticket's authorization data field. The service ticket is the token, and there can be no more than 1,024 SIDs in the token.

For more information about Kerberos, see the Kerberos Authentication Technical Reference at the Microsoft Web site(http://go.microsoft.com/fwlink/?LinkId=48839).

How the Access Token Limit Is Reached

When a user logs on and authentication is successful, the logon process returns a SID for the user and a list of SIDs for the user’s security groups and these comprise the access token. SID history can add additional SIDs to the token. The SIDs in an access token includes:

· The security principal's SID, including SIDs from the SID history of the principal.

· The SID from each domain local group that the principal is directly or transitively a member of, for the domain of the workstation or resource.

· The SID for each global group that the principal is directly or transitively a member of, including SIDs from the SID history of the group.

· The SID for each universal group that the principal is directly or transitively a member of, including SIDs from the SID history of the group.

· The SID for each built-in group the principal is directly or transitively a member of.

· The SID for each local group that the principal is directly or transitively a member of.

Due to a system limitation, the field that contains the SIDs of the principal's group memberships in the access token can contain a maximum of 1,024 SIDs. If there are more than 1,024 SIDs in the principal's access token, the Local Security Authority (LSA) cannot create an access token for the principal during the logon attempt. If this happens, the principal cannot log on or access resources.

In environments that use SID history, each security principal can have two or more SIDs. An additional SID is optionally added to the sIDHistory attribute when a security principal is migrated. Since groups, as well as users, can have SID history, the token of a migrated user with migrated groups can potentially have double the number of SIDs compared to a user that is not migrated.

Note: To reduce the token size of migrated users, ensure that your migration plans include security translation and retirement of the sIDHistory attribute, when possible.

There are two common ways in which the access token limit is exceeded:

· Large fan-out group structure, where a principal is directly a member of many groups, or is a member of a group that is directly a member of many groups.

· Deep nesting group structure, where a principal is a member of a group that results in a large number of transitive memberships.

Either of these structures is possible when an administrator creates groups to carry out legitimate authorization requirements of an organization.

Large Fan-out Group Structure

Large Fan-out Group structure scenario is applied to very few people (Example: Joe Doe). At this time, as we don’t know how much user(s) are affected with this structure, this has been described here.

The large fan-out group structure involves principals being members of many different account and resource groups. This can happen due to legitimate business needs. For example, consider the following characteristics:

· Operations in multiple regions.

· Activities that span multiple specialties.

· A large number of principals that access a large number of resources.

In order to address business requirements such as these, administrators (Joe Doe) might create hundreds of account and resource groups and use group nesting to facilitate required access for all principals in the organization. In this instance, taking into account group nesting, it is possible that a principal may end up being a member of more than 1,024 groups.

The following figure illustrates a large fan-out group structure.

Deep Nesting Group Structure

The deep nesting group structure involves creating groups that are nested within other groups. The following figure illustrates a deep nesting structure.

Since group membership is evaluated recursively, if a user is transitively a member of a group that is nested at 50 levels, that user is also a member of every other group in that hierarchy. The user is also a member of any groups that those groups are members of.

Who Can Cause the Problem

For Active Directory in Windows Server 2003, there are two types of administrative responsibilities:

· Service administrators are responsible for maintaining and delivering the directory service, including domain controller management and directory service configuration.

· Data administrators are responsible for maintaining the data that is stored in the directory service and on domain member servers and workstations.

Service administration accounts and groups have the most widespread power in a network environment and require the most protection. They are responsible for directory-wide settings, installation and maintenance of software, and application of operating system service packs and updates on domain controllers.

In a typical Active Directory environment, the following service administrator groups are capable of creating groups and potentially causing access token limitation problems:

· Default groups in the Builtin container:

a. Administrators

b. Server Operators

c. Backup Operators

d. Account Operators

e. Print Operators

· Default groups in the Users container:

a. Enterprise Admins

b. Schema Admins

c. Domain Admins

Administrators, Enterprise Admins, and Domain Admins, have the broadest range of permissions. Schema Admins can change the default security descriptor of the group class and thereby give write permissions to anyone in the forest. Account Operators have write permissions to any group in the domain and therefore can modify membership of any group.

In addition, delegated data administrators with the following permissions can create groups or modify memberships that can potentially result in users reaching the access token limitation:

· Any individual who has any of the following permissions in Active Directory on a container or OU or on the domain:

a. Full control

b. Modify owner

c. Modify permissions

d. Create containers

e. Create OUs

f. Create groups

· Any individual specifically delegated with any of the following permissions:

a. Create objects of type Group.

b. Write permissions to the member attribute of a security group.

c. Write permissions to the group-type attribute of a distribution group and write permissions to the member attribute of that group.

For recommendations regarding delegating Active Directory administration, see the topic Best Practices.

How to calculate token size

Following formula to determine whether it is necessary to modify the MaxTokenSize value or not

TokenSize = [12 X number of user rights] + [token overhead] + [40 X number of group memberships] + 8s

This formula uses the following values:

· d: The number of domain logical groups a user is a member of plus the number of universal groups outside the user’s account domain plus the number of groups represented in SID history.

· s: The number of security global groups that a user is a member of plus the number of universal groups in a user’s account domain.

· User rights include rights such as “Log on locally” or “Access this Computer from the network”. The only user rights that are added to an access token are those user rights that are configured on the server that hosts a secured resource. Most of the users are likely to have only two or three user rights on the Exchange server. Administrators may have dozens of user rights. Each user right requires 12 bytes to store it in the token.

· Token overhead includes multiple fields such as the token source, expiration time, and impersonation information. For example, a typical domain user has no special access or restrictions; token overhead is likely to be between 400 and 500 bytes.

· Estimated value for ticket overhead can vary depending on factors such as DNS domain name length, client name and other factors.

· Each group membership adds the group SID to the token together with an additional 16 bytes for associated attributes and information. The maximum possible size for SID is 68 bytes. Therefore, each security group to which a user belongs typically adds 44 bytes to the user’s token size.

In scenarios in which delegation is used (for example, when users authentication to a domain controller), Microsoft recommends to double the token size.

Default token size is 12000.

Reference

http://support.microsoft.com/kb/327825

Token Memory allocation

If a token is less than 4 KB, the amount of kernel memory that is allocated for it is exactly what is required to hold the token.

By using the formula this mentioned in the “How to calculate token size” section, my token will be about 2040 bytes .

But if a token is even slightly larger than 4 KB (4096 bytes) the amount of memory that is allocated per copy will jump to exactly 8 KB (8192 bytes). If a token is even slightly larger than 8 KB, the memory allocation will jump to exactly 12 KB. Therefore every time the token sizes crosses one of these critical 4-KB boundaries, there is a sudden jump in the use of paged pool memory and user will have intermitted results.

How to fix the token size problem (Solution)

A registry parameter is available to increase the Kerberos token size. For example, increasing the token size to 65 KB allows a user to be present in more than 900 groups. Because of the associated SID information, this number may vary.

To use this parameter:

1. Start Registry Editor (Regedt32.exe).

2. Locate and click the following key in the registry:

System\CurrentControlSet\Control\Lsa\Kerberos\Parameters

3. If this key is not present, create the key. To do so:

a. Click the following key in the registry:

System\CurrentControlSet\Control\Lsa\Kerberos

b. On the Edit menu, click Add Key.

c. Create a Parameters key.

d. Click the new Parameters key.

4. On the Edit menu, click Add Value, and then add the following registry value:

Value name: MaxTokenSize
Data type: REG_DWORD
Radix: Decimal
Value data: 65535

5. Quit Registry Editor.

The default value for MaxTokenSize is 12000 decimal. Microsoft recommendation is to set this value to 65535 decimal, FFFF hexadecimal. If the value set incorrectly to 65535 hexadecimal (an extremely large value) Kerberos authentication operations may fail, and programs may return errors.

MS KB article explains all those settings: http://support.microsoft.com/kb/263693.

Note: To test this scenario I changed registry setting on KFTUSOKTULSPS35 and it worked.

General information on large token (Reference only)

The way IIS handles headers and therefore authorization information is one thing; the way Windows system, in general, handle large token is another.

Make sure you keeps IIS and windows configuration consistent so that authentication is successful end-to-end.

Reference

http://www.microsoft.com/downloads/details.aspx?familyid=22DD9251-0781-42E6-9346-89D577A3E74A&displaylang=en

Scenario 2: Kerberos working inconsistently for users

Users in this scenario have intermitted problem on Kerberos authentication. After doing research with Microsoft Premier Support, with various network traces and analysis. We were able to identify that Client is not requesting Kerberos call, but it when this problem occurs, it is requesting for only NTLM.

Cause

After a period of inactivity on client workstation/laptop such as sleep, standby or successful unlocking of workstation and purging the Kerberos tickets OR after a client’s Kerberos token expires, the client will always start using NTLM authorization token while trying to access web application. Result of NTLM fallback, Kerberos depended applications such as RSS-Feed, Federated Search and SAP Integration are failing.

Steps to reproduce the problem

1. Purge all Kerberos tickets by Kerbtray or Klist (Available at c:\windows\System32).

a. This is part of Windows XP support tools (http://www.microsoft.com/downloads/details.aspx?familyid=9D467A69-57FF-4AE7-96EE-B18C4790CFFD&displaylang=en).

2. Do IISReset on SharePoint Server – as Rss Web part and Federated search caches for 2 hours.

3. Lock the workstation

4. Unlock the workstation

5. Purge all Kerberos tickets using Kerbtray or KList (Available at c:\windows\System32).

a. This is part of Windows XP support tools (http://www.microsoft.com/downloads/details.aspx?familyid=9D467A69-57FF-4AE7-96EE-B18C4790CFFD&displaylang=en)

6. Open IE and type “Kerberos Web application (say “RSS Web Part”).

7. You will see the error “Authentication Feed error” on Rss web part.

Solution

This has been identified as a bug in Windows XP – Service Pack 2 (Fixed on Windows Xp – Service Pack3). Hot fix has been identified and tested in my laptop and it worked.

KB Article: http://support.microsoft.com/kb/939850

Analyzing Tool

Ntdsutil.exe is a command-line tool that provides management facilities for Active Directory™, the Microsoft® Windows® 2000 and Microsoft® Windows® 2003 directory service.

You can use the Group Membership Evaluation task of the Ntdsutil.exe tool (By default, Ntdsutil is installed in the Winnt\System32 folder) to help recover from an access token limitation problem, such as a user not being able to log on. The purpose of this task is to generate data that will help you identify the source of the problem.

Note: The Group Membership Evaluation task does not directly identify the group that led to the problem for you. It produces a report that will help you with your analysis.

General Note

For simplicity’s sake, I use the word “Kerberos” in this document, when talking about authentication protocol between client and web server. The actual protocol is SPNEGO or “Negotiate”, which is a wrapper for multiple authentication protocols.

Refer to Wikipedia for the details: http://en.wikipedia.org/wiki/SPNEGO

1 Reference

http://kbalertz.com/912376/monitor-troubleshoot-paged-memory-Exchange-server-Exchange-server.aspx

http://support.microsoft.com/kb/263693

http://support.microsoft.com/kb/327825

http://www.microsoft.com/downloads/details.aspx?familyid=22DD9251-0781-42E6-9346-89D577A3E74A&displaylang=en

http://support.microsoft.com/kb/295758

http://support.microsoft.com/kb/215383.

http://technet.microsoft.com/en-us/library/cc781408.aspx

http://support.microsoft.com/kb/820129

http://www.microsoft.com/DownLoads/details.aspx?familyid=22DD9251-0781-42E6-9346-89D577A3E74A&displaylang=en

· For more information about Logon and Authentication Technologies, see the Windows Security Collection of the Windows Server 2003 Technical Reference on the Microsoft Web site (http://go.microsoft.com/fwlink/?LinkId=48827).

· For more information about Authorization and Access Control Technologies, see the Windows Security Collection in the Windows Server 2003 Technical Reference on the Microsoft Web site (http://go.microsoft.com/fwlink/?LinkId=48979).

· For more information about Active Directory users and groups, see Active Directory Users, Computers, and Groups on the Microsoft Web site (http://go.microsoft.com/fwlink/?LinkId=48829).

· For more information about Active Directory users and groups, see the Active Directory Collection in the Windows Server 2003 Technical Reference on the Microsoft Web site

(http://go.microsoft.com/fwlink/?LinkId=48983).

Search configuration - Failed to configure progation --> Query/Index Server

2008-08-05T00:40:00Z

Error : In multi server Moss 2007 farm environment, you might see error "Failed to Configure progation share"

Reason : This problem occurs mostly when your Office SharePoint Service Search is not working (or configured correctly). Follow the steps below,

1. Stop Office SharePoint Service on all servers ( StsAdm.exe -o osearch -action stop)

2. Start the office SharePoint Service Search service on index server first

3. Start the office SharePoint Service Search servicve on role query

4. on the command line type "StsAdm.exe -o osearch -propogationlocation "d:\Program files\Microsoft office Servers\12.0\data\Office Server\applications"

after couple of minutes - reconfigure your SSP and check the event log.

Architecture:

what is continuous propagation?

Instead of copying the entire index from the Index server to the search server, every time a change is made to that index, now we can find that as information is written to Content Store on the Search Server, it is continously propagated to the query server. In other words, it is the act of ensuring all the indexes on the query server are kept up to date by copying the indexes from the index servers. The faster you can update the indexes on the Query server, the faster you'll be able to give updated information to users in the result set.

Continuous propagation has the following characterstics:

Indexes are propagated to the query servers as they are updated within 30 seconds after the shadow index is written to the disk.
The update size must be at least 4 KB. No Maximum size limitation
Metadata is not propagated to the query servers. Instead, it is directly written to the SSP's Search SQL database.
There are no registry entries to manage, and these configurations are hard coded.

Note : Propagation uses the NetBIOS name of the query servers to connect. Hence, it is not a best practice to place a firewall between your Query server and Index server, due to number of ports you would need to open on the firewall.

Microsoft Technology

SharePoint TeamSites Migration steps

Introduction

Migrate SharePoint 2007 to SP2010

Preparation

Planning

Learn

Prepare

Test

Implement

Validate

Backup Site collections

Add Content database on your SP2007 test environment

Fixing any errors

Analyze the log file from Test-SPContentDatabase

Missing Feature

Missing setup file

Missing web part

Attach Content Database to SharePoint 2010

SharePoint 2010: Backup and Restore Best Practices

Introduction

Backup Solution

1.1.1.1. Core Content Recovery

1.1.1.2. Content Recovery

What is Versioning

Contoso’s recommended policy on Versioning for SharePoint 2010

What is Recycle bin

Contoso’s recommended policy on Recycle bin for SharePoint 2010

1.1.1.3. Content Databases

1.1.1.4. Farm Backup and Configuration Database

1.1.1.5. Restoring Service Application

1.1.1.6. Configurations

IIS Configurations

1.1.1.7. Binary Files

1.1.1.8. Customizations

Custom code (WSP)

1.1.1.9. Read-Only Mode

Recommendation

1.1.1.10. Backup scenarios

1.1.1.11. Restore Scenarios

SQL Backup Maintenance

Maintenance Plan

Monitoring

Backup

Restore

Monitoring

Microsoft System Center Operations Manager (SCOM)

Event Based Monitoring

Poll Based Monitoring

DPM (System Center Data Protection Manager)

Search and Replace Retention tag on Microsoft Exchange 2010 (MRM)

Messaging Records Managementdevelopment on Retention Policy Tag

Introduction

What are retention policy tags?

Retention Tags Types (Reference: http://technet.microsoft.com/en-us/library/dd297955.aspx)

Retention Policy Tags (RPTs)

Default Policy Tags (DPTs)

Personal Tags

Retention Policy Tag Attributes

Retention Policy Attributes

Mailbox Objects

Configuration Message

Updating the Configuration Message

Folder Properties

Item Properties

Exchange Web Service

Programmatically change the retention policy tag on the Exchange

Business Scenario

Solution

Pre-requisite

Conceptual Architecture

Code

Screen Shots

1. Email with a retention policy tag

2. Mapi Properties of the message item. Please see the PR_Policy_Tag

3. Retention Policy Tag MAPI properties

4. Run the script

5. After running the script

6. Verify the MAPI Properties

7. The Retention Policy tag value has been changed

Messaging Records Management
development on Retention Policy Tag