Vipul Shah's SQL Blogs

Failover Cluster Testing Methods

2009-06-18T01:19:51Z

During system testing, we will gather as much information as possible about the potential outcomes of system failures. We will not test, however, many component failures in the system such as a motherboard going out, losing a processor or having a cooling fan going out. These represent outages that have been planned for and many will be protected through Cluster Server, other failures will be covered through the fault-tolerance of the systems themselves (such as redundant power supply modules and cooling fans). We will test certain events that can cause failover within the cluster. These tests are listed below along with an explanation of the procedure along with the expected result of each test.

1.1.1 Disk Failure

Purpose: The purpose of testing a disk failure is to ensure that the RAID configuration will continue without interruption. We will also look at hot-spares and ensure that in the event of a disk failure the hot-spare will pickup for the failed drive.

Test Procedure: The procedure for this test is to pull out one of the hard drives in the SAN array while that drive is operational and is currently used by one of the nodes within cluster.

Expected Result: Uninterrupted service; Windows should not discover any problems at all; RAID management software should report loss of a drive and complete the procedure of substituting the hot spare and rebuilding the drive array. Disk performance might be significantly reduced during this time

1.1.2 Power Failure

Purpose: This test will verify that in the event of a server losing power, the opposite node in the cluster will bring all resources in the cluster online and resume operations.

Test Procedure: The procedure for this test is to simply pull all power plugs from one node while that node is operational and is hosting groups within cluster.

Expected Result: Cluster group hosted by the “failed” node should automatically fail over to a passive node. Service interruption should be in the range of 0-2 minutes.

1.1.3 Network Adapters

Purpose: Testing network adapters will serve dual purposes. We will test the functionality of the heartbeat and the ability for cluster heartbeat communications to be routed over the public network. We will also test the failover scenario in the case of both public network adapters (members of the network team) losing connections to the network.

Test Procedure: First we will test the heartbeat interconnect and ensure that cluster communications are carried through the public network without interrupting cluster communications by disconnecting the private network adapter. Secondly, we will test network adapters to ensure that when one out of the two network adapters are unplugged, the other network adapter will communicate with the network as usual. Then we will unplug the remaining network adapter, which at this time is carrying all network communication, including the heartbeat. After this series of tests is complete, we will bring the system back to a normal configuration and test the public network adapters by removing their connection to the network. However, in this last test we will leave the heartbeat interconnect in place.

Expected Result: For a disconnected private adapter, Windows is expected to switch internal cluster communications to a public adapter automatically. There should be no service interruption. For disconnecting one of the public network cards, network team driver is expected to switch to using another network card automatically. There should be no service interruption. For disconnecting all network adapters, cluster is expected to initiate the failover once it discovers that active node is unavailable. Service interruption should be in the range of 0-2 minutes.

1.1.4 Fiber Channel Components

Purpose: These tests will provide a level of understanding and documentation on expectations with regards to the redundancy of the fiber channel components within the HBA cards, the servers, and the CLARiiON SAN. We will be looking for results of unplugging certain components, simulating power losses, and other failures that will affect the cluster.

Test Procedure: During this test, we will disconnect the redundant fiber connections.

Expected Result: Depending on which fiber connection was disconnected, system should automatically switch to the reserved path. There should be no service interruption. In case of both fiber cables disconnected, so that cluster node completely loses communication to the SAN storage, cluster failover should be initiated. Service interruption should be in the range of 0-2 minutes.

1.2 Windows 2003 and SQL Server Failover Testing

Microsoft Cluster Server will ensure that application services continue running within the cluster in the event that either there are failures in Windows 2003 that prevent the application from operating properly, or if SQL Server itself ceases to function properly. The cluster can detect these failures and fail the application over to a passive node. During these tests, it is important to note that our single point of failure within Windows 2003 and SQL Server is the database(s). If database itself becomes corrupt or experiences some other catastrophic failure, the only solution is to restore this database from a backup copy.

Expected result in all tests is for a cluster to initiate the failover. Service interruption should be in the range of 0-2 minutes.

1.2.1 SQL Server Services

Purpose: The purpose of simulating service failures is to ensure that failover will occur and to monitor the activity that occurs during failover. We will be looking for the time required for failover, proper failover and ensuring dependencies are being brought online properly.

Procedure: The best approach to testing a clustered service is to stop this service from the Services snap-in within the Management MMC. A service that has become a clustered resource can only be managed through the cluster administrator. Performing services operations through the Services snap-in will appear to the cluster as a failure and therefore will simulate a service failing. Following are the services we will attempt to fail.

SQL Server service

SQL Server Agent service

MS DTC service

1.2.2 Windows 2003 Failure

Purpose: The purpose here is to simulate the failure of Windows 2003 to demonstrate the ability of Cluster Server to realize Windows 2003 is not functioning/running on one of the clustered nodes and to initiate failover.

Procedure: Testing Windows 2003 failure will be difficult to simulate by any other means than simply choosing Shut Down. This will stop all services on the node being shut down and this node will cease participation in the cluster. The cluster service will be notified on the opposite node and failover of the application will occur.

1.2.3 Cluster Service Failure

Purpose: Cluster Service is responsible for maintaining cluster membership, monitoring resources and managing the clustered node. If this service were to fail, all clustered groups of resources would be forced to move to another cluster node. We will simulate a Cluster Service failure in this test and monitor the failover activity.

Procedure: Stop the Cluster Service from the Services snap-in and record results.

1.2.4 Quorum Failure

Purpose: The Quorum serves as a log for changes that occur while one node of a cluster is offline and as a tiebreaker, in the event all heartbeat communications are lost. This test will observe the cluster behavior when the quorum is lost. To view the results of the quorum serving as a tiebreaker, view the results under “System Failover Testing” above.

Procedure: Utilize the SAN configuration utility and unpresent the quorum drive from the active node. This will provide us with the results that would be seen if the active node loses access to the quorum drive. Failover should be initiated.

Utilize the RAID configuration utility and mark the logical drive of the quorum as offline. This will provide us with the results that would be seen if the drive that contains the quorum were to fail.

Extracting Project Actuals information from Microsoft Project Server (MSPS)

2009-05-06T18:52:10Z

MSPS has four databases that it stores data from Enterprise Project Server – Draft, Published, Reporting, and Archive databases. It is recommended not to query Draft and Published databases directly. Reporting database as name indicates is the right database to query directly to extract MSPS data for further reporting of consumption in other applications. I had a requirements to extract Project Actuals data and push it to one of custom app. Following are the views that I touched to extract data from -

MSP_EpmProject_UserView – Project Information View (High level Project definition data)
MSP_EpmTask_UserView – Task Information View
MSP_EpmResource_UserView – Resource Information View – All the Resource related information (name, windows NT account, CostCenter, etc.)
MSP_EpmAssignment – Assignment Information Table
MSP_EpmAssignmentByDay – Assignment Information Table -- this is to extract Assignment Actual Work (TimeinHours) and Assignment Actual Cost (Invoiced Cost or any similar non-Labor cost)

Storage Consideration for SQL Server 2005 DW environment

2008-04-21T22:10:56Z

Storage design is a key component of data warehouse/BI environment. Optimum storage design provides significant performance. SQL Server performance depends heavily on the I/O subsystem. Unless your database fits into physical memory, SQL Server constantly brings database pages in and out of the buffer pool. This generates substantial I/O traffic. Similarly, the log records need to be flushed to the disk before a transaction can be declared committed. And finally, SQL Server uses tempdb for various purposes such as to store intermediate results, to sort, to keep row versions and so on. So a good I/O subsystem is critical to the performance of SQL Server.

Following key points should be considered while designing storage system for consolidated SQL Server environment.

RAID 1/0 (or RAID 10) gives excellent random read/write performance. RAID 10 is ideal for OLTP environment with lots of small random read/writes. This RAID type also provides good fault tolerance since it can survive the failure of up to half of the disks, provided one disk in each mirror image pair survives. Use RAID 10 if more than 30 percent of the IO is small random writes and if budget permits. RAID 10 is recommended for OLTP Data, Transaction Log, and Tempdb data volumes.
RAID 5 gives excellent read performance, especially large sequential I/O. But it provides lower random write performance. It also delivers flower fault tolerance than RAID 10 since it can tolerate only one drive failure per RAID 5 LUN. Also, in the event of a drive failure, the time for the storage system to rebuild the content of the failed drive is longer than RAID 10. RAID 5 costs less for the same storage capacity compared to RAID 10. RAID 5 is ideal for Backup volumes.
SQL Server log files are accessed sequentially and are write intensive. Since RAID 5 requires 4 I/O per write (write data, validate data, write parity, and validate parity), placing the log files on RAID 5 array greatly increases the possibility for an I/O bottleneck. Placing the data file and log files on the same drive spindles creates contention for the drive heads between the sequential movement required for the log writes versus the random drive head movement required to read and write data. Because of this, the log file should not be placed on the same set of spindles as the data files. The log files should be placed on a RAID 1+0 arrays.
SQL Server 2005 makes greater use of tempdb than SQL Server 2000. For high performance applications, tempdb should be placed on a dedicated RAID 1+0 array and sized large enough so it does not auto grow. TempDB has a large number of objects being created all the time. For an object to be created, space must be allocated to it. Space allocation is determined by looking at some of the internal system pages (the GAM, and SGAM). In the end, it is these pages that start to have significant contention (with just one file) in a VERY active TempDB. To minimize that contention you should create multiple files. The guideline is to create as many numbers of files as the number of processor cores on the system. So, for example if you have dual core 8 way processor, you should have 16 files for your tempdb database.
Backups should be made to drives other than the ones used for data and log not only to separate out I/Os, but physically separate drives means that even if the data or log disks fail, the backup drive will most likely still exist. Writing the backup file to the same set of disks means that the data file is being read from and the backup file is written to all using the same I/O pool. It also puts available space at risk since backups can fill up drives quickly without a proper retention policy to manage them.
All versions of SQL Server write in 8k pages and the read ahead is 64k. The recommendation is to format any disks which may be used for SQL Server data using a 64k block size. Using defaults for NTFS (which is not 64K) will hamper SQL Server performance. Even if there is the potential that a disk may be used for SQL Server data, format it with a 64k block size.
SQL Server on non-optimized and shared SAN where many different applications and I/O profiles access the same spindles can result in performance issues. Sections of the storage should be reserved for high performance SQL Server requirements and carved up at the time of deployment of the particular SQL Server solution. These storage allocations should not come from the general storage pool.
Sector alignment. Sector alignment can increase the I/O performance of SQL Server by up to 20%. If sector alignment is required, this will be done via the command line utility DISKPART in Windows Server 2003 prior to formatting the disks with NTFS.

Data Protection Manager - an ultimate solution for protecting SQL Server data

2008-02-28T07:17:00Z

I learned about new tool from Microsoft System Center family called Data Protection Manager 2007. Here is some details on how it works and how can it save big bucks when comparing with other backup compression tool in market such as LiteSpeed.

Below is its overall architecture:

How does SQL Server protection with DPM 2007 work?

DPM 2007 uses a combination of transaction log replication and block-level synchronization in conjunction with the SQL VSS Writer to help ensure your ability to recover SQL Server databases. After the initial baseline copy of data, two parallel processes enable continuous data protection with integrity:

Transaction logs are continuously synchronized to the DPM 2007 server, as often as every 15 minutes.
An “express full” uses the SQL Server VSS Writer to identify which blocks have changed in the entire production database, and send just the updated blocks or fragments. This provides a complete and consistent image of the data files on the DPM server or appliance. DPM 2007 maintains up to 512 shadow copies of the full SQL Server database(s) by storing only the differences between any two images.

Assuming one “express full” per week, stored as one of 512 shadow copy differentials between one week and the next, plus 7 days x 24 hours x 4 (every fifteen minutes), DPM 2007 can provide over 344,000 data consistent recovery points for SQL.

How does Restore work?

With only a few mouse clicks and DPM 2007, you can:

Restore a SQL database directly back to the original server

Databases are recovered back where they came from into the active SQL Server, with no additional work for the SQL administrator to do afterwards. This improves recovery time and reduces the number of people needed during a crisis recovery.

Restore to a “recovery database” on the original server

This provides an alternative where the active SQL database is untouched, but the older data can be restored onto the same SQL server but in an alternate database – allowing both versions of the data to be accessed independently but concurrently.

Copy database files to an alternate server,

This option enables disaster recovery, compliance auditing, or software testing – without affecting the production environment

Copy the database file to tape,

Use this option to create a long-term archive or portable media of the data at any recovery point, even after the fact. For example, if the accounting department closes the quarterly books at 10:45 AM on a Tuesday, you can “recover” the data from that point, directly to tape – without backing up anything else or impacting the production server that is already serving new data.

Comparison of DPM 2007 with other Compressed Backup Utility:

Customer Scenario – Protect 4 Databases totalling 250 GB for 14 days:

With good compression efficiency of 70%, like LiteSpeed:
30% additional disk space required on a production server’s local disk, for nightly compressed backups – which are then copied to the backup server at 75GB per night.
Total additional disk to protect 250 GB for 14 days with LiteSpeed = 1.12 TB (75 GB on local production server plust 1.05 TB for 2 weeks, 75 GB each day)
With DPM2007, you don’t need additional local disk space.
On DPM server, DPM replica “Express Full” which is exact replica of databases will consume 250 GB. In addition to that with DPM daily transaction logs at 10% change rate, it will require 25 GB on daily basis. So, to keep two weeks (14 days) of data on DPM server you need 575 GB (250 + (25 * 13)) as compared to 1.12 TB for LiteSpeed.

A Great SQL Resource - SQLCommunity.com

2008-02-08T08:13:00Z

I came across www.SQLCommunity.com - a great resource for SQL Server.

It has tones of information including Scripts and Tools, Tips, Tricks and Techniques, lots of info on SQL Events, etc. This site is run by Saleem Hakani - a Microsoft Database Engineer. I know Saleem very well, he is very passionate about bringing SQL awareness world wide. He organizes many internal SQL events such as SQLFest, SQLSummit. I have attended one his event, and I must say, in just one day I got lot out of that event. I highly recommend to put his site in your SQL tool box. It could be your one stop shop for all you need from SQL Server.

Finding Missing Indexes and Unused Indexes using DMVs

2007-12-05T00:25:00Z

Finding Missing Indexes:

Dynamic management object	Information returned
sys.dm_db_missing_index_group_stats	Returns summary information about missing index groups, for example, the performance improvements that could be gained by implementing a specific group of missing indexes.
sys.dm_db_missing_index_groups	Returns information about a specific group of missing indexes, such as the group identifier and the identifiers of all missing indexes that are contained in that group.
sys.dm_db_missing_index_details	Returns detailed information about a missing index; for example, it returns the name and identifier of the table where the index is missing, and the columns and column types that should make up the missing index.
sys.dm_db_missing_index_columns	Returns information about the database table columns that are missing an index.
Note:
When the metadata for a table changes, all missing index information about that table is deleted from these dynamic management objects. Table metadata changes can occur when columns are added or dropped from a table, for example, or when an index is created on a column of a table.

When SQL Server is restarted, all of the missing index information is dropped.

Example using DMVs:

Query:

SELECT mig.index_group_handle,mid.index_handle, migs.avg_total_user_cost as AvgTotalUserCostThatCouldbeReduced, migs.avg_user_impact as AvgPercentageBenefit,

'CREATE INDEX missing_index_' + CONVERT (varchar, mig.index_group_handle) + '_' + CONVERT (varchar, mid.index_handle)

+ ' ON ' + mid.statement

+ ' (' + ISNULL (mid.equality_columns,'')

+ CASE WHEN mid.equality_columns IS NOT NULL AND mid.inequality_columns IS NOT NULL THEN ',' ELSE '' END

+ ISNULL (mid.inequality_columns, '')

+ ')'

+ ISNULL (' INCLUDE (' + mid.included_columns + ')', '') AS create_index_statement

FROM sys.dm_db_missing_index_groups mig

INNER JOIN sys.dm_db_missing_index_group_stats migs ON migs.group_handle = mig.index_group_handle

INNER JOIN sys.dm_db_missing_index_details mid ON mig.index_handle = mid.index_handle

--where statement = '[<DBName>].[dbo].[<TableName>]'

Finding Unused Indexes:

DMVs used: sys.dm_db_index_usage_stats

Example to find Unused Indexes:

select object_name(i.object_id) as ObjectName, i.name as [Unused Index]

from sys.indexes i

left join sys.dm_db_index_usage_stats s on s.object_id = i.object_id

and i.index_id = s.index_id

and s.database_id = db_id()

where objectproperty(i.object_id, 'IsIndexable') = 1

AND objectproperty(i.object_id, 'IsIndexed') = 1

and s.index_id is null -- and dm_db_index_usage_stats has no reference to this index

or (s.user_updates > 0 and s.user_seeks = 0 and s.user_scans = 0 and s.user_lookups = 0) -- index is being updated, but not used by seeks/scans/lookups

order by object_name(i.object_id) asc

In the above query, user_seeks, user_scans, and user_lookups counters indicate the usage of the particular index. If the value of the counters is 0, it means that index has never been used for any Seek, or Scan or Lookup operations.

Every individual seek, scan, lookup, or update on the specified index by one query execution is counted as a use of that index and increments the corresponding counter in this view (DMV).

The user_updates counter indicates the level of maintenance on the index caused by insert, update, or delete operations on the underlying table or view.

The counters are initialized to empty whenever the SQL Server (MSSQLSERVER) service is started. In addition, whenever a database is detached or is shut down (for example, because AUTO_CLOSE is set to ON), all rows associated with the database are removed.

When an index is used, a row is added to sys.dm_db_index_usage_stats if a row does not already exist for the index. When the row is added, its counters are initially set to zero.

DDL Triggers and LOGON Triggers

2007-12-05T00:16:00Z

DDL Triggers:

DDL triggers can be used for administrative tasks such as auditing and regulating database operations. Use DDL triggers when you want to do the following:

¾ You want to prevent certain changes to your database schema.

¾ You want something to occur in the database in response to a change in your database schema.

¾ You want to record changes or events in the database schema.

DDL triggers fire only after the DDL statements that trigger them are run. DDL triggers cannot be used as INSTEAD OF triggers.

Example of DDL Trigger:

This example tracks the Login Events on a server level. It uses DDL_LOGIN_EVENTS event group from server scope events from above diagram. It fires against DDL Login activities such as CREATE, ALTER, DROP Login.

CREATE Trigger [Trg_TrackLoginManagement]

on ALL Server

for DDL_LOGIN_EVENTS

set nocount on

declare @data xml,

@EventType varchar(100),

@EventTime datetime,

@ServerName varchar(100),

@AffectedLoginName varchar(100),

@WhoDidIt varchar(100),

@EmailSubject varchar(500),

@EmailBody varchar(800),

@EmailRecipients varchar(300)

set @EmailRecipients = 'DBAS@microsoft.com'

set @data = eventdata()

set @EventType = @data.value('(/EVENT_INSTANCE/EventType)[1]', 'varchar(100)')

set @EventTime = @data.value('(/EVENT_INSTANCE/PostTime)[1]','datetime')

set @ServerName = @data.value('(/EVENT_INSTANCE/ServerName)[1]','varchar(100)')

set @AffectedLoginName = @data.value('(/EVENT_INSTANCE/ObjectName)[1]','varchar(100)')

set @WhoDidIt = @data.value('(/EVENT_INSTANCE/LoginName)[1]','varchar(100)')

insert into msdb..TrackLoginManagement values (@EventType,@EventTime,@ServerName,@AffectedLoginName,@WhoDidIt)

set @EmailSubject = 'ALERT: DDL_LOGIN_Event: ' + @EventType + ' occured by ' + @WhoDidIt + ' on ' + @ServerName

set @EmailBody = 'DDL_Login_Event: ' + @EventType + char(10) +

'Event Occured at: ' + convert(Varchar, @EventTime) + char(10) +

'ServerName: ' + @ServerName + char(10) +

'Affected Login Name: ' + @AffectedLoginName + char(10) +

'Event Done by: ' + @WhoDidIt

EXEC msdb.dbo.sp_send_dbmail

@recipients = @EmailRecipients,

@body = @EmailBody,

@subject = @EmailSubject ;

print @Eventtype + ' activity completed successfully.'

LOGON Triggers:

Similar to DDL Triggers, LOGON Triggers fire stored procedures or T-SQL statements in response to LOGON events. Logon triggers fire after the authentication phase of logging in finishes, but before the user session is actually established. Therefore, all messages originating inside the trigger that would typically reach the user, such as error messages and messages from the PRINT statement, are diverted to the SQL Server error log. Logon triggers do not fire if authentication fails.

You can use logon triggers to audit and control server sessions, such as by tracking logon activity, restricting logons to SQL Server, or limiting the number of sessions for a specific logon.

Example of LOGON Trigger:

In the following code, the logon trigger captures information from EventData() functions such as LoginName, HostName, LoginTime, and LoginType when any logon activity happened by a SQL Login which are connecting from hosts other than application servers. This is useful to monitor people connecting to SQL Server using application ids during peak hours and possibly causing performance issues by running inefficient queries outside of an application.

create table myTest

(LogonTime datetime,

LoginName varchar(50),

ClientHost varchar(50),

LoginType varchar(50)

)

create TRIGGER myTest_LogonTrigger

ON ALL SERVER WITH EXECUTE AS 'sa'

FOR LOGON

BEGIN

declare @LogonTriggerData xml,

@EventTime datetime,

@LoginName varchar(50),

@HostName varchar(50),

@LoginType varchar(50)

set @LogonTriggerData = eventdata()

set @EventTime = @LogonTriggerData.value('(/EVENT_INSTANCE/PostTime)[1]', 'datetime')

set @LoginName = @LogonTriggerData.value('(/EVENT_INSTANCE/LoginName)[1]', 'varchar(50)')

set @HostName = @LogonTriggerData.value('(/EVENT_INSTANCE/ClientHost)[1]', 'varchar(50)')

set @LoginType = @LogonTriggerData.value('(/EVENT_INSTANCE/LoginType)[1]', 'varchar(50)')

if @LoginType = 'SQL Login' and @HostName not in ('<comma separated list of hostname>')

and @LoginName not in ('<comma separated list of SQL logins>')

insert into master..myTest values (@EventTime, @LoginName, @HostName, @LoginType)

end

Plan Cache Analysis in SQL Server 2005

2007-09-25T07:16:00Z

Plan Cache: The portion of the memory pool allocated to store Query execution Plan

For how much memory is allocated for Plan Cache in different version of SQL server, please refer to my blog: - Understanding Query Plan Cache in SQL 2005 SP2 - why it's changed from SQL 2005 RTM/SP1?

A stored execution plan can be one of the following:

Compiled – the actual instructions describing how SQL Server will implement a query
Execution Context – run-time information unique to a user:
- Values of local variables
- Object ID’s of temporary objects
- Which statement currently being executed
Cursors – execution state of server-side cursors
Parse Tree – Aids in parsing the query. Usually not cached.

In SQL Server 2005 there are four major DMVs which can be very useful in Plan Cache Analysis:

1. sys.dm_exec_requests

shows information about all running processes (similar to sysprocesses in SQL 2000) along with sql_handle, and plan_handle which can be utilized as an input in following DMVs

example:

SELECT

    sder.session_id AS [SPID],

       sder.sql_handle as [SQL_Handle],

       sder.plan_handle as [PLAN_Handle],

    sdes.login_name AS [Login],

    sd.name AS [DBName],

    sder.start_time AS [Start Time],

    sder.status AS [Status],

    sder.command AS [Command],

    sdet.text AS [SQL Text],

    sder.percent_complete AS [Pct Cmplt],

    sder.estimated_completion_time AS [Est Cmplt Time],

    sder.wait_type AS [Wait],

    sder.wait_time AS [Wait Time],

    sder.last_wait_type AS [Last Wait],

    sder.cpu_time AS [CPU Time],

    sder.total_elapsed_time AS [Total Elpsd Time],

    sder.reads AS [Reads],

    sder.writes AS [Writes],

    sder.logical_reads AS [Logical Reads]

FROM

    sys.dm_exec_Requests sder

    CROSS APPLY sys.dm_exec_sql_text(sql_handle) AS sdet

    JOIN sys.dm_exec_sessions sdes on sder.session_id = sdes.session_id

    JOIN sys.databases sd on sder.database_id = sd.database_id

WHERE

    sder.session_id <> @@SPID and sder.session_id > 50

2. sys.dm_exec_query_plan (plan_handle)

returns the plan in XML format. You can save the XML format query plan with extension ".sqlplan" and when you open up that file it will show a nice graphical query plan.

SELECT * FROM sys.dm_exec_query_plan (<plan_handle>) --here <plan_handle> is supplied based on the results from Query in Example 1.

3. sys.dm_exec_query_stats

·         The most important DMV – sys.dm_exec_query_stats - returns aggregate performance statistics for cached query plans. The view contains one row per query statement within the cached plan, and the lifetime of the rows are tied to the plan itself. When a plan is removed from the cache, the corresponding rows are eliminated from this view.   Please note that the statement_start_offset and statement_end_offset columns are zero based, represents number of bytes and the end offset is -1 for last statement in the batch. In addition the text value returned by sys.dm_exec_sql_text is nvarchar(max) instead of text as documented incorrectly in Books Online. Combined with sys_dm_exec_sql_text, it could provide rich information with all the performance related aggregates - IO (logical reads/writes, physical reads/writes), CPU (worker time), Duration (elapsed time), #of executions (execution_count).

SELECT db_name(qt.dbid) as DatabaseName, object_name(qt.objectid,qt.dbid) as ObjectName,

Qt.text as 'statement',

substring(Qt.text, (qs.statement_start_offset/2)+1

                        , ((case qs.statement_end_offset

                              when -1 then datalength(Qt.text)

                              else qs.statement_end_offset

                           end - qs.statement_start_offset)/2) + 1) as 'Statement',

QS.Total_Logical_Reads,

QS.Total_Worker_Time,

QS.Total_Elapsed_Time,

QS.Execution_Count

FROM sys.dm_exec_query_stats AS QS

CROSS APPLY sys.dm_exec_sql_text (QS.sql_handle) AS QT

WHERE plan_handle = <plan_handle> -- <plan_handle> is supplied based on the results from query in example 1.

Useful resources:

SQL Programmability & API Development Team Blog - http://blogs.msdn.com/sqlprogrammability/archive/2007/01.aspx

SQL DMVStats – Performance Warehouse - http://www.codeplex.com/sqldmvstats

Understanding Query Plan Cache in SQL 2005 SP2 - why it's changed from SQL 2005 RTM/SP1?

2007-09-24T23:18:00Z

Query execution plan gets stored in Plan Cache - the portion of the memory pool allocated for storing execution plans. There is a very good blog written by SQL Programmability team on why Query Plan Cache behavior changed in SQL 2005 SP2 vs. SQL 2000 and SQL 2005 RTM/SP1 - http://blogs.msdn.com/sqlprogrammability/archive/2007/01/22/3-0-changes-in-caching-behavior-between-sql-server-2000-sql-server-2005-rtm-and-sql-server-2005-sp2.aspx

Based on this the plan cache size is different between SQL 2000, SQL 2005 SP1/RTM, and SQL 2005 SP2:

SQL Server Version	Maximum Limit on Plan Cache Size
SQL Server 2000	4GB upper cap on the plan cache
SQL Server 2005 RTM/SP1	75% of server memory from 0-8GB + 50% of server memory from 8Gb-64GB + 25% of server memory > 64GB
SQL Server 2005 SP2	75% of server memory from 0-4GB + 10% of server memory from 4Gb-64GB + 5% of server memory > 64GB

SQL Server 2000 4GB upper cap on the plan cache

Example:

For a SQL Server with 32Gb total SQL server memory, SQL Server 2005 RTM and SP1 cachestore limit will be 75% X 8 + 50% X (32 - 8) = 18GB

SQL Server 2005 SP2 cachestore limit will be 75% X 4 + 10% X (32-4) = 5.8GB

The reason for less maximum limit on Plan Cache in SQL 2005 SP2 is because Limiting the maximum size of the plan cache will ensure enough room for the database pages, and therefore improved throughput.

A large page cache is good and normal, and increasing the size of a large page cache usually improves performance.

A large plan cache is bad and abnormal, and increasing the size of a large plan cache rarely improves performance.

There are only a fixed number of pages in a database, so increasing the size of the page cache should always improve performance. However there is no limit to the number of different SQL statements that a poorly written application can generate. A very large plan cache usually indicates that you’re not getting much plan reuse, and caching more plans that don’t get reused isn’t a very good strategy. Better to cap the plan cache and use that memory for the page cache instead.

If you find that your plan cache is anywhere near the maximum that SQL will allow, you should fix the application to get more shared SQL and plan reuse through coding changes, forced parameterization, or plan guides.

Maximum Capacity Specification for SQL Server 2005 and 2008

2007-09-14T23:14:23Z

I came across this link - http://technet.microsoft.com/en-us/library/ms143432(SQL.100).aspx. Its amazing to see the max capacity specification for SQL Server. These are for SQL Server 2008, but they haven't changed from SQL Server 2005's limit. I would love to see systems with near capacity in below specifications:

SQL Server Database Engine object	Maximum sizes/numbers SQL Server (32-bit)	Maximum sizes/numbers SQL Server (64-bit)
Database size	1,048,516 terabytes	1,048,516 terabytes
Databases per instance of SQL Server	32,767	32,767
Filegroups per database	32,767	32,767
Files per database	32,767	32,767
File size (data)	16 terabytes	16 terabytes
File size (log)	2 terabytes	2 terabytes

This is for SQL Server 2005 - http://technet.microsoft.com/en-us/library/ms143432.aspx which is similar to SQL Server 2008.

Useful information about Guest Account in SQL Server

2007-08-16T00:26:00Z

guest user

When a database is created, the database includes a guest user by default. Permissions granted to the guest user are inherited by users that do not have a user account in the database.

The guest user cannot be dropped, but it can be disabled by revoking its CONNECT permission. The CONNECT permission can be revoked by executing REVOKE CONNECT FROM GUEST within any database other than master or tempdb.

http://msdn2.microsoft.com/en-us/library/aa905195(SQL.80).aspx

http://msdn2.microsoft.com/en-us/library/ms190928.aspx

Connecting to Remote Integration Services Server - info with missing things on msdn article

2007-07-12T20:57:00Z

I found it hard way how to connect to Integration Services server remotely. If you are part of sysadmin role on a server or if you are part of local administrators group on the server, there is no problem connecting to Integration Services server from your computer remotely. But for others they get "Access is denied" error. Below are some steps to resolve this issue:

Most of the steps are documented in msdn article - http://msdn2.microsoft.com/en-us/library/aa337083.aspx

But following is missing:

One above link, under "To configure rights for remote users on Windows Server 2003"...

replace step 9 with "Click OK to close the dialog box."
Add a step 9.1 with the following text: "On the same Security tab, under Access Permissions, select Customize, then click Edit to open the Access Permission dialog box."
Add a step 9.2 with the following text: "In the Access Permission dialog box, add or delete users, and assign the appropriate permissions to the appropriate users and groups. The available permissions are Local Access, and Remote Access. The easiest is to add the local DCOM Distributed Users group. "
Add a step 9.3 with the following text: "Click OK to close the dialog box. Close the MMC snap-in."
Step 10 stays as-is: "Restart the Integration Services service."

This should work.

Connecting to multiple Data Sources using single Report

2007-06-26T21:31:00Z

I have been looking for something like this for ages. This could be very useful to run a common report on a multiple servers.

http://technet.microsoft.com/en-us/library/ms156450.aspx

Data Source Expressions:
You can put an expression into a connection string to allow users to select the data source at run time. For example, suppose a multinational firm has data servers in several countries. With an expression-based connection string, a user who is running a sales report can select a data source for a particular country before running the report.

The following example illustrates the use of a data source expression in a SQL Server connection string. The example assumes you have created a report parameter named ServerName, and DatabaseName:

= "data source=" & Parameters!ServerName.Value & ";initial catalog=Parameters!DatabaseName.Value

See attached picture.

Data source expressions are processed at run time or when a report is previewed. The expression must be written in Visual Basic. Use the following guidelines when defining a data source expression:

Design the report using a static connection string. A static connection string refers to a connection string that is not set through an expression (for example, when you follow the steps for creating a report-specific or shared data source, you are defining a static connection string). Using a static connection string allows you to connect to the data source in Report Designer so that you can get the query results you need to create the report.
When defining the data source connection, do not use a shared data source. You cannot use a data source expression in a shared data source. You must define a report-specific data source for the report.
Specify credentials separately from the connection string. You can use stored credentials, prompted credentials, or integrated security.
Add a report parameter to specify a data source. For parameter values, you can either provide a static list of available values (in this case, the available values should be data sources you can use with the report) or define a query that retrieves a list of data sources at run time.
Be sure that the list of data sources shares the same database schema. All report design begins with schema information. If there is a mismatch between the schema used to define the report and the actual schema used by the report at run time, the report might not run.
Before publishing the report, replace the static connection string with an expression. Wait until you are finished designing the report before you replace the static connection string with an expression. Once you use an expression, you cannot execute the query in Report Designer. Furthermore, the field list in the Datasets window and the Parameters list will not update automatically.

/3GB /PAE and AWE on 32-bit Systems

2007-05-10T19:11:00Z

I was reading a blog post from Chad Boyd - http://blogs.msdn.com/chadboyd/archive/2007/03/24/pae-and-3gb-and-awe-oh-my.aspx. He has very detailed and excellent information about /3GB PAE and AWE. It made my understanding more clear. Here is the summarized version:

For 32-bit OS

Memory Addressability Limitation:

By default, the 32bit OS can only 'see' and use up to 4gb of memory, as it uses a 32bit range to map physical memory space (with a valid range of 0x00000000 up to 0xFFFFFFFF, or 0 thru 4,294,967,295 in decimal format, which correlates to a maximum high range pointer of 4GB. Yes, this means that on a default 32bit system no matter how much physical memory you install on the system, the system would only ever use at most 4 GB of it (kind of a waste if you have a 32bit server with 64GB of memory)

Virtual Address Space (VAS) Limitation:

When a process is created on a 32-bit Windows System, OS allocates Virtual Address Space (VAS) to that process. All user-mode process (such as SQL Server) has its own VAS. Each process in 32-bit system has 4 GB of VAS. VAS is further divided into two equal sections: 1st section which is available to the process and 2nd section which is reserved for kernel/system. So, out of 4 GB VAS, 2 GB is available to the process and 2 GB is reserved for kernel/system. So, though a VAS is 4GB in size, a user-mode process by default only has access to up to 2GB of that space - yes, this means that by default, a 32bit user-mode process can only ever access 2GB of memory at the most using standard API's.

/3GB can address the VAS limitations:

With /3GB you can change the VAS allocation from 2 GB for process to 3 GB availability for the process and reserving only 1 GB for kernel/system. Limitation of /3GB is that it can only be used for the system having memory upto 16 GB. This is because the limited 1GB of kernel space - that is very limited space to be able to store internal memory mapping structures.

/PAE can address the memory addressability limitation of 32-bit system:

With /PAE you can change the addressability of 32-bit system from 2^32 to 2^36 (from 4 GB to 64 GB).

AWE:

AWE can take the advantage of increased addressability by PAE. With AWE you can increase memory availability to SQL Server from 4 GB (for 32-bit) to up to 64 GB. So, although AWE allows you to access/use memory outside the processes' VAS, the process is still confined by the VAS in terms of being able to read/write data into memory mapped by AWE mechanisms. Only Data Cache can utilize AWE, no other memory consumer within SQL such as Hash cache, Sorting, plan cache, connection cache, etc. can use memory available through AWE.

Simple Formula:

If your system has < 4 GB - use only /3GB

If your system has > 4 GB and < 16 GB - use /3GB + /PAE + AWE

If your system has > 16 GB - use /PAE + AWE

Data Scripter Utility for SSMS

2007-05-04T20:15:00Z

You can generate a script for a data within a table - whole bunch of insert statement. There is an external plug-in available at http://www.codeproject.com/useritems/enisey.asp. I downloaded it and installed it on my machine, it works great. After installing it you will see an option in your SSMS within Right-click menu off of a selected table to Script out data within the table. Here is the introduction of this utility from this site:

"Enisey" is data scripter add-in to Microsoft SQL Server Management Studio (SSMS). One expands the functionality of SSMS and allows to script data from any tables into a choosen ".sql" file in the view of INSERT statements. It is one .NET 2.0 assembly, registered as COM-object and registered as SSMS addin. The sources contain also setup project for building the full-fledged setup file. Because the project is a plugin, the full built distributive is shipped instead of demo-project. It is prepared for the use. Just install it and you can see a new context menu item "Script Data" inside SSMS on each table.