Rob's SQL Server Blog : Tools

Scottish SQL Server User Group Meeting - Oct 8th

robcarrol — Thu, 24 Sep 2009 18:38:00 GMT

I have the great pleasure of speaking at the next Scottish SQL Server User Group meeting being held in Mocrosoft's Edinburgh ofice on Thursday the 8th of October. Full details can be found on the UK User Group site. My talk will focus on some of the great tools we have for collecting and analysing SQL Server performance data. Martin Bell MVP will also be talking about using Powershell with SQL Server, which should be excellent as always.

Hope to see you there !

SQL Server 2008 Upgrade Technical Reference Guide Released

robcarrol — Wed, 26 Nov 2008 13:11:00 GMT

The SQL Server 2008 Technical Reference Guide has just been released, covering the essential steps and best practices to upgrade existing 2000 and 2005 instances to 2008. There is also a list of other upgrade resources available at http://msdn.microsoft.com/en-us/library/cc936623.aspx

These guides are intended to suppliment the information already available in SQL Server 2008 Books Online.

SQL Server 2008 - Performance Studio

robcarrol — Fri, 20 Jun 2008 19:50:00 GMT

I've been looking at this new feature of SQL Server 2008 in order to demo it to customers, and I have to say it's pretty cool ! Along with Resource Governor and compression, it's one of my favourite new features and a good reason to consider an upgrade. Performance Studio builds on the concept of the Database Reports in SQL Server 2005 and the Performance Dashboard introduced in SQL Server 2005 SP2. Like it's predecessors, it's built on top of standard DMV's but with one crucial advantage... data is historical and is persisted across service restarts. Previously, you had to roll your own code in order to persist DMV data, now SQL Server 2008 gives you it straight out the box.

Setting it up is easy... expand the 'Management' folder and then right-click 'Data Collection' then 'Configure Management Data Warehouse'. This opens a wizard which guides you through the configuration of the Management Data Warehouse (MDW). Select the 'Create or Upgrade a Management Data Warehouse' option and enter your server details. You can either configure a new database or use an existing one in order to collect the performance data.

Once the MDW has been configured, run through the wizard again and select the 'Set up Data Collection' option. At this point you need to also ensure that SQL Server Agent is running or else the performance counters will not be uploaded to the database. Also, SQL Server Integration Services is required to manage collections. There are 3 new database roles in the MDW database: MDW Admin, MDW Writer, MDW Reader. It's a good idea to restrict access to this database, especially if you are collecting sensitive data. Performance Studio will only collect data against SQL Server 2008 databases, so unfortunately you can't use it to monitor older versions of SQL Server. MSDB is used to store the log entries, Agent jobs and SSIS packages.

When creating the MDW, plan for data growth of up to 250 - 500 MB a day, depending on your query plans and consider running an archive job to aggregate summary data before SQL Server runs it's purge job after 14 days. Regarding performance overhead, Microsoft detected approximately 3 - 4% increase in CPU performance on it's TPC-C tests, which is fairly low overhead.

Performance Studio comes with 3 built-in Collection Sets: Disk Usage Collection Set, Query Activity Collection Set and Server Activity Collection Set. The Disk Usage Collection Set collects data every 6 hours and retains it for a default of 730 days. It gathers data and log disk usage and plots them over time. This gives a nice visual view of data file growth over time.

The Query Activity Collection Set uploads query activity every 15 minutes and retains it for 14 days. It caches active sessions and requests from DMV's every 10 seconds. It uses dm_exec_query_stats and uploads the 3 most "interesting" queries and any queries where the query plan has changed. What constitutes an interesting query, I have no idea :-) These can then be viewed graphically based on CPU, Duration, Total I/O, Physical Reads or Logical Writes. You can drill-down into the reports to show the query text, query plan, showplan and any missing indexes identified that could improve the query execution.

The Server Activity Collection Set may turn out to be the most useful performance tuning weapon. Data is uploaded every 15 minutes to the MDW and is collected every 10/60 seconds depending on the particular counter. This data is retained for 14 days before being purged. It collects data on Server CPU usage, Memory, Waitstats, Disk I/O and Network Usage, amongst others. Again, you can click through these reports for detailed information.

In addition to the built-in functionality, you can also create your own collection sets, however be careful doing this or you may end up collecting huge amounts of data, particularly if you run a SQL Trace collection. If you do want to create a custom Profiler collection, set up a trace in Profiler then select the "Save as Trace Collection" option. This will then script out the XML trace definition for you which can then be executed against your SQL Server system... simple as that !

For more information, see the following Webcast. It's well worth taking a look !

https://msevents.microsoft.com/cui/WebCastEventDetails.aspx?culture=en-US&EventID=1032349947&CountryCode=US

Indexing Strategies

robcarrol — Tue, 17 Jun 2008 10:02:00 GMT

I attended a SQL Server User Group meeting earlier this year and heard a presentation about favourite DMV's. Mentioned in the discussion was sys.dm_db_index_usage_stats and that got me thinking about indexing strategies. I have been involved in performance troubleshooting databases that have used a variety of indexing stategies, ranging from none to lots of narrow indexes on practically every column ! However, there is no right and wrong indexing strategy, it depends entirely on your application and the type and frequency of queries being executed against the database.

So how do you know if you have a problem? For me, it's usually when users tell me that the application is "running slow" or they get timeouts. However, poor perfomance can be open to interpretation. There could be a whole host of factors to take into account when you are dealing with web applications, such as network connections and web servers problems. However, this shows that you need to have good benchmarks in place in order to compare performance over a period of time. Another sign could be high CPU, high memory usage or increased disk IO activity. The SQLCAT team has a post detailing the top OLTP performance issues on 2005 and some basic performance counters to monitor can be found here.

So once you've determined that there is a problem, what do you do? If you know the specific query that is causing the issue, then you can verify the query execution plan. You do this by chosing the 'Display Estimated Execution Plan' option under the 'Query' menu in Management Studio. This does not execute the query, but returns the estimated execution plan based on the statistics the server has. As a result, you need to ensure that the statistics are up to date, or you may get the wrong results. It's also a good idea to turn on statistics IO. Things to look out for in the execution plan are table or index scans and hash aggregates. Scans imply that there are no indexes for SQL Server to use or the indexes are not selective enough and SQL Server has decided it's less expensive to run a scan. Bear in mind that a clustered index scan is exactly the same as a table scan. Hash aggregates have to create a worktable i.e. a temp table in TempDB. Watch out for Hash aggregates as Statistics IO does not show the cost of the worktables, which can often be very expensive. As a rule of thumb, anytime you see "Hash" in your plan it means temp tables and this can be done better ! Another cool thing about showplan is that you can force the queries to use different indexes and run them side by side. This will show the execution plan for both queries and the cost of each relative to the batch. This is a quick way to see which index choices are most expensive.

However, on shared systems with multiple applications running against the SQL Server instance, chances are you will not know the queries that are causing the performance issues and you will need to do some digging. You can use DMV's in 2005, but as I work in a mixed environment (2000 and 2005) I prefer to use profiler to give me an idea what is going on. I run a server side trace and remotely log the results to a file as oppossed to a SQL Server table to minimise overhead on the system. I will typically run this for short periods (10 - 15 mins) just to get a feel for the queries executing against the server. This is a quick way to see if there are any expensive queries running and how frequently. This gives me some clues as to which database is causing the problems. Once I know this, I can really focus in on that database application. More information on running Profiler traces is available here.

Once I have identified the database, the hardest part is deciding which queries to index. Lots of frequently run queries can give you a far bigger overall performance gain than a large query run once a month. It is important to use Profiler and run traces over longer periods to get a feel for the mix of reads and writes. Never build an index in isolation, always consider the workload over the course of time. You can then combine these traces with DTA in order to fully analyse your workload and come up with the proper indexing strategy based on your application's specific needs. It is also worth restoring a backup of your database on a test system and use this for running your DTA analysis on. I cannot stress highly enough, do not create indexes for indexes sake. I often ask the question when interviewing, "when would droping an index actually help to increase performance"? A lot of people are conditioned to believe that you need to build lots of indexes to increase performance, but this is not true. If you have lots of inserts, updates and deletes then this is a significant overhead to keep all these indexes updated if they are not required in the first place.

Before going off and building indexes it is worth looking at alternative options. It is a good idea to update your statistics to ensure that SQL Server has most up to date information to work out the optimal execution plan. If you are working with stored procedures, you can consider recompiling them. You can also consider re-writting the code, especially if you are using cursors!

Generally, SQL Server does better with wider indexes (covering several columns) and fewer of them than it does with narrower ones. Narrow indexes will require a bookmark lookup to get the rest of the data if you don't cover the query. Bookmark lookups are expensive and SQL Server may even decide not to use the index at all and perform a table scan. In this situation, the narrow indexes will not be used and are an unecesary overhead. You can use the DMV sys.dm_db_missing_index_details to see which multi-column indexes could give better performance. Another word of warning here if you are using the 2005 DMV's, such as sys.dm_db_index_usage_stats. Remember that the data in these DMV's is not persisted across server shutdowns, or database restores. Do not assume that because the DMVs show that an index has not been used you can safely drop it. This only means that it hasn't been used since the cache was cleared out. If you are going to rely on the DMV's, you need to persist the data over a period of time and then analyse it. This is easy to do as you can select directly from the DMV's into tables, which can then persist the data. A good time period would be to persist the data every 30 mins. Paul Randal has a post here with information how to persist the index usage data from DMVs.

The final thing I'd like to say about indexing (for the time being !) is that you can't just create indexes and walk away. You need to regularly maintain them in order to reduce fragmentation. Fragmentation in indexes can be caused by inserts, updates and deletes and can be a problem if you have a busy system. By default, indexes are created with 100% fill factor, which means they are densely packed as soon as you create them. If you then need to insert rows or update data, SQL Server will need to carry out page splits in order to do so. To avoid this happening, it is advisable to create indexes with a fill factor of around 80 - 90%, however this may vary depending on the frequency of data updates. Not only will this increase the performance of inserts and updates, it will also keep fragmentation at a minimum. You can check for fragementation using DBCC SHOWCONTIG (2000) and sys.dm_db_index_physical_stats (2005) and remove fragmentation using DBCC INDEXDEFRAG or DBCC DBREINDEX in 2000, and ALTER INDEX REORGANIZE or REBUILD in 2005. There is further information and advce here for rebuilding indexes and updating statistics.

This is a huge topic and I could go on all day, but the importance of good indexing cannot be stressed highly enough ! In summary, you need to ensure that you understand your application's workload, you need to check that the indexes you have are actually being used and that you are not missing any indexes. Finally, you need to maintain your indexes so they are kept in optimal shape. Good indexing means good performance which means happy users !

[This was originally posted on http://sqlblogcasts.com/ in February 2008]

SQL Server 2005 Performance Dashboard

robcarrol — Thu, 12 Jun 2008 14:25:00 GMT

I've had one or two requests recently to set up the Performance Dashboard reports on SQL Server so I've created this blog post to step through the process. Firstly, the Performance Dashboard was created to allow customers and support engineers to monitor the general performance characteristics of a server and investigate the cause of any performance problems that may occur. The goal of the Performance Dashboard is to reduce the time spent discovering the problem so effort can be focused on actually resolving it.

The SQL Server 2005 Performance Dashboard Reports are available as a link off the SP2 Feature Pack (Feb 2007). The Performance Dashboard reports are built on the SQL Server 2005 dynamic management views, accessing data that is already captured by SQL Server 2005. Consequently there is no performance impact of using the dashboard except when you actually open/refresh a report. This allows engineers and customers to go much further on a performance problem while the problem is happening. The Performance Dashboard requires SQL Server 2005 SP2 or later.

Install the Dashboard by running the msi from the link above, which will install to a default location of Program Files\Microsoft SQL Server\90\Tools\PerformanceDashboard. After setup finishes, open Management Studio and connect to the server and run the SETUP.SQL script (once for each SQL instance you want to monitor). Then from Object Explorer select the server, right mouse click and choose Reports – Custom Reports and browse to find the PERFORMANCE_DASHBOARD_MAIN.RDL file. This report is the only report intended to be directly loaded from SSMS; all other reports are accessed as a drill through off of the main report. The help file, PERFDASH.CHM has details about setup and permission requirements, how each report is accessed, details about the general methodology used in the dashboard and how you can use/interpret the information on each individual report.

Sysinternals Live Beta Launched

robcarrol — Mon, 02 Jun 2008 02:10:00 GMT

Microsoft has launched the Sysinternals Live beta, allowing execution of Sysinternals tools directly from the web. Enter a tool’s Sysinternals Live path into Windows Explorer or a command prompt as \\live.sysinternals.com\tools\<toolname> or view the entire Sysinternals Live tools directory in a browser at http://live.sysinternals.com.

http://technet.microsoft.com/sysinternals