Rob's SQL Server Blog : Database Maintenance

SQL Server Statistics

robcarrol — Fri, 16 May 2008 11:00:00 GMT

SQL Server 2005 collects statistical information about indexes and column data stored in the database. These statistics are used by the SQL Server query optimizer to choose the most efficient plan for retrieving or updating data. Good statistics allow the optimizer to accurately assess the cost of different query plans, and choose a high-quality plan. All information about a single statistics object is stored in several columns of a single row in the sysindexes table, and in a statistics binary large object (statblob) kept in an internal-only table. In addition, information about statistics can be found in the metadata views sys.stats and sys.indexes

Statistics Used by the Query Optimizer in Microsoft SQL Server 2005

SQL Server 2005 has the ability to automatically create and update statistics. This feature is on by default in SQL Server 2005 and SQL Server 2000. This allows the SQL Server 2005 query optimizer to produce good query plans consistently, while keeping development and administration costs low. If you need more control over statistics creation and update to get good query plans and manage the overhead of gathering statistics, you can use manual statistics creation and update capabilities.

SQL Server 2005 statistics features allow you to:

Implicitly create and update statistics with the default sampling rate (in the SELECT, INSERT, DELETE, and UPDATE commands, use of a column in a query condition such as a WHERE or JOIN clause causes statistics to be created or updated if necessary when automatic statistics creation and update is enabled)
Manually create and update statistics with any desired sampling rate, and drop statistics (CREATE STATISTICS, UPDATE STATISTICS, DROP STATISTICS, CREATE INDEX, DROP INDEX)
Manually create statistics in bulk for all columns of all tables in a database (sp_createstats)
Manually update all existing statistics in the database (sp_updatestats)
List statistics objects that exist for a table or database (sp_helpstats, catalog views sys.stats, sys.stats_columns)
Display descriptive information about statisticsobjects (DBCC SHOW_STATISTICS)
Enable and disable automatic creation and update of statistics database-wide or for a specific table or statistics object (ALTER DATABASE options AUTO_CREATE_STATISTICS and AUTO_UPDATE_STATISTICS, sp_autostats, and the NORECOMPUTE option on CREATE STATISTICS and UPDATE STATISTICS)
Enable and disable asynchronous automatic update ofstatistics (ALTER DATABASE option AUTO_UPDATE_STATISTICS_ASYNC)

SQL Server Management Studio allows you to graphically browse and control statistics objects within its Object Explorer view. Statistics are listed in Object Explorer in a folder underneath each table object.

SQL Server 2005 maintains the following information at the table level.

Number of rows in the table or index (rows column in sys.sysindexes).
Number of pages occupied by the table or index (dpages column in sys.sysindexes).

SQL Server 2005 collects the following statistics about table columns and stores them in a statistics object (statblob):

Time the statistics were collected.
Number of rows used to produce the histogram and density information
Average key length.
Single-column histogram, including the number of steps.
A string summary, if the column contains character data. DBCC SHOW_STATISTICS output contains a column "String Index" which has the value YES if a statistics object contains a string summary.

Use sp_helpindex and sp_helpstats to display the list of all statistics available for a given table. sp_helpindex lists all indexes on the table, and sp_helpstats lists all the statistics on the table. Each index also carries the statistical information for its columns. The statistical information created using the CREATE STATISTICS command is equivalent to the statistics built by a CREATE INDEX command on the same columns. The only difference is that the CREATE STATISTICS command uses sampling by default while the CREATE INDEX command gathers the statistics with fullscan since it has to process all rows for the index anyway.

Creating Statistics with SQL Server 2005

You can create statistics in SQL Server 2005 in a number of different ways, as described below.

• The optimizer automatically creates single-column statistics as needed as a side effect of optimizing SELECT, INSERT, UPDATE, and DELETE statements when AUTO_CREATE_STATISTICS is enabled, which is the default setting.

• CREATE INDEX generates the declared index in the first place, and also creates one set of statistics for the column combinations constituting the index keys (but not other included columns). CREATE STATISTICS only generates the statistics for a given column or combination of columns.

• Use sp_createstats to create statistics for all eligible columns (all except XML columns) for all user tables in the current database. A new statistics object will not be created for columns that already have a statistics object.

• Use dbcc dbreindex to rebuild one or more indexes for a table in the specified database.

• In Management Studio, expand the folder under a Table object, right click the Statistics folder, and choose New Statistics.

• Use the Database Tuning Advisor (DTA) to create indexes.

Here is an example of a CREATE STATISTICS command on the AdventureWorks.Person.Contact table:

CREATE STATISTICS FirstLast2 ON Person.Contact(FirstName,LastName)
WITH SAMPLE 50 PERCENT

Usually, statistics with default sampling are sufficient to allow good query plans to be produced. However, there may be cases when statistics with larger sample sizes may benefit query optimization, such as when the values in the given column sample are not random. Non-random samples may occur if the data is sorted or clustered. Sorting or clustering may be caused by the creation of indexes, or loading data into a heap structure that is already sorted or clustered. The most commonly used larger sample size is fullscan because it gives the most accurate statistics. The price for using statistics with larger sample sizes is the time required to create the statistics.

The above command creates a single two-column statistics object. In this case, the SAMPLE 50 PERCENT is ignored and a full scan is performed because the table is too small. Sampling is used primarily to avoid excessive scans of data and affects only tables and indices with 1,024 or more pages (8 MB).

In SQL Server 2005, statistics are created for all indices at the index creation time. SQL Server creates single-column statistics automatically when compiling queries. These statistics are created for columns where the optimizer would otherwise have to estimate the approximate density or distribution. There are the following exceptions to this rule. Statistics may not be created automatically when (1) the database is read-only, (2) there are too many outstanding compilations in progress, and (3) the column's data type is not supported for auto stats.

The automatic statistics creation function can be disabled at the database level by executing

ALTER DATABASE dbname SET AUTO_CREATE_STATISTICS OFF

It's recommended that you leave this setting ON. Only disable it if you need to do so to work around known performance issues, such as the need to specify other than the default sampling rate for some tables.

By default, statistics are created by sampling the data set when executing the CREATE STATISTICS command or when the statistics are automatically created. CREATE INDEX scans the whole data set anyway; therefore, the index statistics are initially created without sampling (the equivalent of fullscan). The CREATE STATISTICS command allows you to set the sample size in the WITH clause either by specifying fullscan or the percentage of data or number of rows to scan. It is also possible to inherit the previous sample size when specifying WITH RESAMPLE on the UPDATE STATISTICS command.

For small tables, a minimum of 8 MB of data is sampled. If a table starts small, and you sample with the default sampling rate, and thereafter use the resample option when updating the statistics, you will get the equivalent of fullscan even as the table grows beyond 8 MB. Avoid the use of resample if you want the default sampling rate as table size varies.

The dbcc show_statistics command displays the sample size under the Rows Sampled heading. Statistics created automatically, or updated automatically (as described in the next section) are always generated using default sampling. The default sampling rate is a slow-growing function of the table size, which allows statistics to be gathered relatively quickly even for very large tables.

SQL Server Profiler can monitor automatic statistics creation. The Auto Stats event is in the group of Performance trace events. Select also the IntegerData, Success and ObjectID columns for the Auto Stats event when defining the trace.

The DROP STATISTICS command is used to drop statistics, but it is not possible to drop statistics that are a byproduct of an index. Such statistics are removed only when the index is dropped.

Maintaining Statistics in SQL Server 2005

After a series of INSERTs, DELETEs, and/or UPDATEs are performed on a table, the statistics may not reflect the true data distribution in a given column or index. If the SQL Server query optimizer requires statistics for a particular column in a table that has undergone substantial update activity since the last time the statistics were created or updated, SQL Server automatically updates the statistics by sampling the column values (using auto update statistics). The statistics auto update is triggered by query optimization or by execution of a compiled plan, and involves only a subset of the columns referred to in the query. Statistics are updated before query compilation if AUTO_UPDATE_STATISTCS_ASYNC is OFF, and asynchronously if it is ON.

When a query is first compiled, if the optimizer needs a particular statistics object, and that statistics object exists, the statistics object is updated if it is out of date. When a query is executed and its plan is in the cache, the statistics the plan depends on are checked to see if they are out of date. If so, the plan is removed from the cache and during re-compilation of the query the statistics are updated. The plan also is removed from the cache if any of the statistics it depends on have changed.

SQL Server 2005 determines whether to update statistics based on changes to column modification counters (colmodctrs).

A statistics object is considered out of date in the following cases:

The table size has gone from 0 to >0 rows.
The number of rows in the table when the statistics were gathered was 500 or less, and the colmodctr of the leading column of the statistics object has changed by more than 500 since then
The table had more than 500 rows when the statistics were gathered, and the colmodctr of the leading column of the statistics object has changed by more than 500 + 20% of the number of rows in the table when the statistics were gathered

If the statistics object is defined on a temporary table, it is out of date as discussed above, except that there is an additional threshold for recomputation at 6 rows, with a test otherwise identical to test 2 in the previous list.

Table variables do not have statistics at all.

The auto update statistics feature described above may be turned off at different levels.

On the database level, disable auto update statistics using:

ALTER DATABASE dbname SET AUTO_UPDATE_STATISTICS OFF

At the table level, disable auto update statistics using the NORECOMPUTE option of the UPDATE STATISTICS command or CREATE STATISTICS command.
Use sp_autostats to display and change the auto update statistics setting for a table, index, or statistics object.

Re-enabling the automatic updating of statistics can be done similarly using ALTER DATABASE, UPDATE STATISTICS, or sp_autostats.

Auto statistics update is always performed by sampling the index or table using the default sampling rate. To set the sampling rate explicitly, run CREATE or UPDATE STATISTICS

Statistics update is covered by the same SQL Profiler event as statistics creation.

DBCC CHECKDB

robcarrol — Wed, 14 May 2008 13:15:00 GMT

DBCC CHECKDB checks the logical and physical integrity of all objects in the specified database by performing the following:

Runs DBCC CHECKALLOC on the database
Runs DBCC CHECKTABLE on every table and view in the database
Runs DBCC CHECKCATALOG on the database
Validates the contents of every indexed view in the database
Validates the service broker data in the database

DBCC CHECKALLOC:

Validates the allocation information maintained in the GAM, SGAM and IAM pages
Performs cross check to verify that every extent that the GAM or SGAM indicates has been allocated really has been allocated, and that any extents not allocated are indicated in the GAM and SGAM as not allocated
Verifies the IAM chain for each allocation unit, including the consistency on the links between the IAM pages in the chain
Verifies that all extents marked as allocated to the allocation unit really are allocated

DBCC CHECKTABLE:

Performs a comprehensive set of checks on the structure of a table, and by default these are logical and physical
With physical-only option, you can exclude the logical checks and only validate the physical structure of the page and record headers
PHYSICAL OPTION is a lightweight check of the physical consistency of the table and common hardware failures that can compromise data
Indexed views are verified by regenerating the view's rowset from the underlying select statement definition and comparing the results with the data stored in the indexed view
SQL performs 2 left anti-semi joins between the 2 rowsets

DBCC CHECKCATALOG:

Performs more than 50 cross-checks between various metadata tables
cannot fix errors it finds by running the DBCC operation with any of the REPAIR options

SERVICE BROKER Data is Verified:

Only way to check the service broker data as there is no specific DBCC command to perform the checks
DBCC CHECKFILEGROUP can also be considered to be a subset of CHECKDB because it performs DBCC CHECKTABLE on all tables and views in a specified filegroup

DBCC will run faster on a SQL 2005 database upgraded from 2000 but with no 2005 features or indexed views. On a new 2005 database, some of the logical checks added to compliment new features are necessarily complex and add to the runtime when invoked.

All the DBCC validation commands use database snapshots to keep validation operations from interfering with on-going database operations and to allow the validation operation to see a quiescent, consistent view of the the data. A snapshot is created at the beginning of the CHECK command, and no locks are acquired on any of the objects being checked. The actual check operation is performed against the snapshot. Unlike regular snapshots, the "snapshot file" that DBCC CHECKDB creates cannot be configured and is invisible to the end user. It always uses apace on the same volume as the database being checked. This capability is only available when your data directory is on an NTFS partition.

You can avoid creating a snapshot (to save disk space for example) by using the WITH TABLOCK option with the DBCC command
If you are using one of the repair options, a snapshot is not created as the database is in single-user mode, so no other transactions can be altering data
Without TABLOCK, DBCC is considered to be an online operation
DBCC validation checks require a significant amount of space as SQL Server needs to temporarily store data about pages and structures that have been observed during the check operation against pages and structures that are observed later during the DBCC scan
To determine the space tempdb needs in advance, run CHECKDB with the ESTIMATE ONLY option:

SET NOCOUNT ON;
DBCC CHECKDB ('AdventureWorks') WITH ESTIMATEONLY;
This is computed as a worst-case estimate and assumes there will be no room in memory for any of the sort operations required

VALIDATION CHECKS

SQL Server 2005 includes a set of logical validity checks to verify that data is appropriate for the columns data type. These can be expensive and affect server performance, so you can disable these, along with other non-core logical validations by using the PHYSICAL-ONLY option. All new databases in SQL Server 2005 have the DATA_PURITY logical validations enabled by default. For upgraded databases, you must run the DBCC CHECKDB with the DATA_PURITY option once, preferably immediately after the upgrade

DBCC CHECKDB ('dbname') WITH DATA_PURITY

After the purity check completes without any errors, performing the logical validations is the default behavior in all future executions of DBCC CHECKDB and there is no way to change this default. You can override the default by using the PHYSICAL_ONLY option. This skips the data purity check but also skips any checks that have to analyze the contents of individual rows of data and limits the checks that DBCC performs.

DBCC REPAIR OPTIONS

There are three DBCC repair options:

REPAIR_ALLOW_DATA_LOSS
REPAIR_REBUILD
REPAIR_FAST (legacy backward compatibility only)

Almost all errors can be repaired except:

DBCC CHECKCATALOG errors
Data purity errors found through DBCC CHECKTABLE

When you run DBCC CHECKTABLE with one of the repair options:

SQL runs DBCC CHECKALLOC and repairs what it can
Then runs DBCC CHECKTABLE on all tables and makes the appropriate repairs on all the tables
List of repairs are ranked to avoid duplication of effort (i.e. rebuilding an index, then removing a page from a table would duplicate effort)

REPAIR_ALLOW_DATA_LOSS option:

SQL Server tries to repair almost ALL detected errors.
For almost any severe error, some data will be lost when the repair is run.
Rows may be deleted if they are found to be inconsistent, such as when a computed column value is incorrect.
Whole pages can be deleted if checksum errors are discovered
During the repair, no attempt is made to maintain any constraints on the tables, or between tables
Some errors SQL Server won't even try to repair, particularly if the GAM or SGAM pages themselves are corrupt or unreadable.

REPAIR_REBUILD option:

Minor, relatively fast repair actions such as repairing extra keys in a non-clustered index
Time consuming repairs, such as rebuilding indexes
can be performed without risk of data loss
After successful completion, database is physically consistent and online but may not be in a logically consistent state in terms of constraints and business rules

Use repair options only as a last resort, restore the database from backup instead. If you are going to run repair_allow_data_loss, backup your database first. You can run the repair options for DBCC inside a user-defined transaction, which means you can perform a ROLLBACK to undo the repairs. Exception is when you are running the repair options on a database in EMERGENCY mode. If a repair in emergency mode fails, there are no other options except to restore the database from a backup.

Each individual repair in a DBCC operation runs in it's own system transaction, which means that if a repair is not possible, it will not affect any other repairs.

Create a snapshot before the repair is initiated, start a transaction, then run the REPAIR option. Before committing or rolling back, you can compare the repaired database with the original snapshot. If you are not satisfied with the changes made, rollback the repair operation.

PROGRESS REPORTING:

Use sys.dm_exec_requests
Command column indicates current DBCC command phase
Percent_completed represents %age completion of the DBCC command phase
Estimated_completion_time represents the time in ms of how long it will take to complete

DBCC BEST PRACTICES

Use CHECKDB with CHECKSUM option and a sound backup strategy to protect the integrity of the database from hardware failures

Perform DBCC CHECKDB WITH DATA_PURITY after upgrading a database to SQL Server 2005 to check for invalid data values

Make sure you have enough disk space available to accommodate the database snapshot

Make sure you have space available ion tempdb to allow the DBCC command to run. Use the ESTIMATEONLY command to find out how much space is required in tempdb

Use REPAIR_ALLOW_DATA_LOSS only as a last resort