SQL Server SQLOS team blog

How to configure Soft-NUMA on a system with > 32 processors?

harshiam — Sat, 16 May 2009 01:10:00 GMT

I recently answered this customer question and was looking forward to post the solution here. So here you go!

There is already a lot of information about how Soft NUMA works and how to configure Soft NUMA on your system. So without delving into the basics, I am jumping right ahead to the topic.

Soft Affinity Setting

SQL Server 2000 and above supports the affinity option for greater than 32 processors (33 to 64) only on 64-bit operating system.

If you are running SQL Server 2000 or above on a system with greater than 32 processors, then you can use sp_configure ‘affinity mask’ (32 bit mask) option for the first 32 CPUs and sp_configure ‘affinity64 mask’ (32 bit mask) option for the CPUs 33 to 64.

Ex: Consider a machine with 33 CPUs. Setting the ‘affinity mask’ to 0x0000ffff and ‘affinity64 mask’ to 0x1 hard affinitizes CPU 1 to 16 and CPU 33 to ONLINE.

sp_configure 'show advanced options', 1;

RECONFIGURE;

GO

sp_configure 'affinity mask', 65535;

RECONFIGURE;

GO

sp_configure ‘affinity64 mask’, 1;

RECONFIGURE;

GO

Registry Settings

To map Soft NUMA nodes to greater than 32 CPUs, use a QWORD registry value to represent CPUs 1 to 64. Use the Registry Editor program (regedit.exe) to add the following registry keys to map soft-NUMA to CPUs as follows.

Ex: Mapping Soft NUMA nodes to CPUs as follows:

Node 0: CPUs 1 to 16

Node 1: CPUs 17 to 32

Node 2: CPUs 33 and 34

SQL Server 2005	Type	Value name	Value data
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\90\NodeConfiguration\Node0	QWORD	CPUMask	0x 0000 0000 0000 ffff
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\90\NodeConfiguration\Node1	QWORD	CPUMask	0x0000 0000 ffff 0000
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\90\NodeConfiguration\Node2	QWORD	CPUMask	0x0000 0002 0000 0000

SQL Server 2008	Type	Value name	Value data
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\100\NodeConfiguration\Node0	QWORD	CPUMask	0x 0000 0000 0000 ffff
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\100\NodeConfiguration\Node1	QWORD	CPUMask	0x0000 0000 ffff 0000
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\100\NodeConfiguration\Node2	QWORD	CPUMask	0x0000 0003 0000 0000

Rest of the details about using Soft NUMA and mapping the registry key remains the same.

If the registry is having faulty settings like overlapping NUMA node masks to CPU mapping or Soft NUMA mapping crossing hardware NUMA, then SQL Server ignores the Soft NUMA configuration.

Good news is that the new version of SQL Server (after SQL Server 2008) will be supporting greater than 64 logical processors and will be using a more convenient way for setting affinity!

-Harshitha Amit

SQLOS Test Team

Debugging slow response times in SQL Server 2008

IvanPe — Sat, 19 Jul 2008 01:56:00 GMT

Introduction

The other day I came across a nice article written by Mario Broodbakker called "SQL Server Wait Events: Taking the Guesswork out of Performance Profiling" (see references) on how to troubleshoot slow response times in SQL Server 2000 and 2005 using the YAPP Method. According to the author, one of the biggest challenges to apply this method in SQL Server is that fact that the wait statistics are collected and exposed at the server level (server wide):

... "This is simply not enough, unless you have the luxury of testing your application and SQL statements on an isolated server" ..." But, of course, it's often hard or impossible to isolate and replicate production problems on an isolated test server." ...

I decided to write this because in SQL Server 2008 the wait statistics are available at virtually any level, and (at the time of writing) I couldn't find any articles on the new sqlos.wait_info and sqlos.wait_info_external extended events.

YAPP Method

The YAPP Method: "Yet Another Performance Profiling Method" is based on a paper written by Anjo Kolk (see references). The basic idea of the method can be expressed with the following formula:

(1) Response time = service time + wait time

This method for performance troubleshooting has been around for some time and it is a fairly standard practice when debugging slow response times.

You can get the response time and the service time of a query by looking at sys.dm_exec_sessions, columns total_elapsed_time and cpu_time. Alternatively, you can use SQL trace or the more lightweight extended events in SQL Server 2008. Knowing response time and service time and applying formula (1), gives you wait time. So, we know how much we waited. However, in order to get the individual wait types we need the new extended events.

Per session wait statistics in SQL Server 2008

In SQL Server 2008 you can get wait statistics for a given session or statement by leveraging the new extended events. Optionally, you can limit the collection to a small set of wait types or collect only waits exceeding certain duration. If your event session is written carefully, you can use this technique on production systems with very little or no performance impact. In fact, SQL Server 2008 ships with a built-in event session, which is collecting wait statistics, and is running by default out of the box.

Here is how to create an event session to collect wait types for session 51. The events will go to the specified trace file:

create event session session_waits on server

add event sqlos.wait_info

(WHERE sqlserver.session_id=51 and duration>0)

, add event sqlos.wait_info_external

(WHERE sqlserver.session_id=51 and duration>0)

add target package0.asynchronous_file_target

(SET filename=N'c:\wait_stats.xel', metadatafile=N'c:\wait_stats.xem');

Here is how to start the event session, run your workload, and then stop the event session:

alter event session session_waits on server state = start;

exec sp_workload;

alter event session session_waits on server state = stop;

Here is how to read the trace file:

select * from from sys.fn_xe_file_target_read_file

('c:\wait_stats*.xel', 'c:\wait_stats*.xem', null, null)

The result is a list of wait events, representing individual waits, having wait type and duration column. In order to make better sense out of this we need to group the events by wait type and calculate the total duration per wait type. The following code can do this:

create view dbo.read_xe_file as

select object_name as event, CONVERT(xml, event_data) as data

from sys.fn_xe_file_target_read_file

('c:\wait_stats*.xel', 'c:\wait_stats*.xem', null, null)

create view dbo.xe_file_table as

select

event

, data.value('(/event/data/text)[1]','nvarchar(50)') as 'wait_type'

, data.value('(/event/data/value)[3]','int') as 'duration'

, data.value('(/event/data/value)[6]','int') as 'signal_duration'

from dbo.read_xe_file

select

wait_type

, sum(duration) as 'total_duration'

, sum(signal_duration) as 'total_signal_duration'

from dbo.xe_file_table

group by wait_type

order by sum(duration) desc

Example

Hopefully this example will demonstrate the power of sqlos.wait_info and sqlos.wait_info_external events. I did some measurements by executing the following workload from two connections:

create proc #sp_workload

begin

SET nocount ON

CREATE TABLE #t1000k (c2 char(1020));

BEGIN TRANSACTION

DECLARE @rows INT = 50000;

DECLARE @row INT = 0;

DECLARE @count INT = 0;

WHILE @row < @rows

BEGIN

INSERT INTO #t1000k (c2) VALUES (REPLICATE('a',1000));

SELECT @row = @row + 1

IF @count > 10

BEGIN

COMMIT

BEGIN TRANSACTION

SELECT @count=0

END

SELECT @count=@count+1

END

COMMIT

END

While running the workload I had IOMeter (see references) so that I can add some disk pressure.

Results for session 51: Apparently, most of the time was spent waiting for space in the log buffer to store log records - wait type LOGBUFFER. For complete list of all wait types with their descriptions search for sys.dm_os_wait_stats in MSDN or Books Online (see references).

Response time	Service time	Wait time
32619	1828	30791

wait_type	total_duration	total_signal_duration
LOGBUFFER	9662	66
PREEMPTIVE_OS_FILEOPS	6442	276
PREEMPTIVE_OS_FLUSHFILEBUFFERS	3833	150
PREEMPTIVE_OS_WRITEFILEGATHER	3015	66
SOS_SCHEDULER_YIELD	2481	2480
WRITE_COMPLETION	1971	0
IO_COMPLETION	1183	0
PAGEIOLATCH_UP	551	0
WRITELOG	417	0
PAGELATCH_UP	30	27
PAGELATCH_SH	10	10
PREEMPTIVE_XE_CALLBACKEXECUTE	4	200

Results for session 52: The result for this session is very similar. We can tell that most of the time was spent waiting. After accumulating the durations from all waits we get 30726, which is pretty close to the total wait time calculated using sys.dm_exec_sessions (31484).

Response time	Service time	Wait time
33890	2406	31484

wait_type	total_duration	total_signal_duration
LATCH_EX	16555	32
LOGBUFFER	10796	28
SOS_SCHEDULER_YIELD	2739	2729
WRITELOG	605	0
PAGELATCH_UP	26	25
PAGELATCH_SH	5	5

For the sake of completeness I also collected server-wide wait statistics, which are also available in previous version of SQL Server. The server-wide statistics clearly shows that we have IO bottleneck however you don't know exactly which session or query was affected and by how much:

wait_type	wait_time_ms	signal_wait_time_ms
LOGBUFFER	21666	127
WRITELOG	17118	4
LATCH_EX	16561	34
PAGEIOLATCH_UP	7891	0
SOS_SCHEDULER_YIELD	7058	7038
PREEMPTIVE_OS_FILEOPS	6443	0
WRITE_COMPLETION	5896	4
SLEEP_BPOOL_FLUSH	3916	278
PREEMPTIVE_OS_FLUSHFILEBUFFERS	3833	0
PREEMPTIVE_OS_WRITEFILEGATHER	3016	0
IO_COMPLETION	1680	0
PREEMPTIVE_OS_REPORTEVENT	141	0
PAGELATCH_UP	139	128
PREEMPTIVE_OS_PIPEOPS	128	0
PREEMPTIVE_XE_TARGETINIT	85	0
XE_BUFFERMGR_ALLPROCESSED_EVENT	70	0
PAGELATCH_SH	23	23
PREEMPTIVE_OS_QUERYREGISTRY	16	0
PREEMPTIVE_XE_CALLBACKEXECUTE	8	0
PAGELATCH_EX	7	7
PREEMPTIVE_XE_SESSIONCOMMIT	5	0
LATCH_SH	1	0
PREEMPTIVE_OS_GETDISKFREESPACE	1	0

Complete test code

Here is the complete source code of the test:

use tempdb

-- Enable xp_cmdshell to delete trace files

exec sp_configure 'show advanced options', 1

reconfigure

exec sp_configure 'xp_cmdshell', 1

reconfigure

BEGIN TRY

drop proc #sp_workload

END TRY BEGIN CATCH END CATCH

create proc #sp_workload

begin

SET nocount ON

CREATE TABLE #t1000k (c2 char(1020));

BEGIN TRANSACTION

DECLARE @rows INT = 50000;

DECLARE @row INT = 0;

DECLARE @count INT = 0;

WHILE @row < @rows

BEGIN

INSERT INTO #t1000k (c2) VALUES (REPLICATE('a',1000));

SELECT @row = @row + 1

IF @count > 10

BEGIN

COMMIT

BEGIN TRANSACTION

SELECT @count=0

END

SELECT @count=@count+1

END

COMMIT

END

BEGIN TRY

drop event session session_waits on server

END TRY BEGIN CATCH END CATCH

exec xp_cmdshell 'del c:\wait_stats*.xel', no_output;

exec xp_cmdshell 'del c:\wait_stats*.xem', no_output;

declare @cmd nvarchar(max);

set @cmd = N'create event session session_waits on server ' +

N'add event sqlos.wait_info (WHERE sqlserver.session_id=' + convert(nvarchar(10), @@spid) + N' and duration>0) ' +

N', add event sqlos.wait_info_external (WHERE sqlserver.session_id=' + convert(nvarchar(10), @@spid) + N' and duration>0) ' +

N'add target package0.asynchronous_file_target (SET filename=N''c:\wait_stats.xel'', metadatafile=N''c:\wait_stats.xem'')';

exec(@cmd);

BEGIN TRY

drop table #snapshot

END TRY BEGIN CATCH END CATCH

select total_elapsed_time, cpu_time into #snapshot from sys.dm_exec_sessions where session_id=@@SPID;

alter event session session_waits on server state = start;

exec #sp_workload;

alter event session session_waits on server state = stop;

select

(es.total_elapsed_time - ss.total_elapsed_time) as 'Response time',

(es.cpu_time - ss.cpu_time) as 'Service time',

((es.total_elapsed_time - ss.total_elapsed_time) - (es.cpu_time - ss.cpu_time)) as 'Wait time'

from sys.dm_exec_sessions es, #snapshot ss where session_id=@@SPID;

drop event session session_waits on server

BEGIN TRY

drop view dbo.read_xe_file

END TRY BEGIN CATCH END CATCH

create view dbo.read_xe_file as

select object_name as event, CONVERT(xml, event_data) as data

from sys.fn_xe_file_target_read_file('c:\wait_stats*.xel', 'c:\wait_stats*.xem', null, null)

BEGIN TRY

drop view dbo.xe_file_table

END TRY BEGIN CATCH END CATCH

create view dbo.xe_file_table as

select

event

, data.value('(/event/data/text)[1]','nvarchar(50)') as 'wait_type'

, data.value('(/event/data/value)[3]','int') as 'duration'

, data.value('(/event/data/value)[6]','int') as 'signal_duration'

from dbo.read_xe_file

select

wait_type

, sum(duration) as 'total_duration'

, sum(signal_duration) as 'total_signal_duration'

from dbo.xe_file_table

group by wait_type

order by sum(duration) desc

-- CLEANUP

exec xp_cmdshell 'del c:\wait_stats*.xel', no_output;

exec xp_cmdshell 'del c:\wait_stats*.xem', no_output;

drop view dbo.xe_file_table;

drop view dbo.read_xe_file;

drop proc #sp_workload;

References

Sys.dm_os_wait_stats

The following MSDN article shows all wait types with their description: http://msdn.microsoft.com/en-us/library/ms179984.aspx

IOMeter

IOMeter is an IO stress and measurement tool developed by Intel. It is now an Open Source tool, and freely downloadable from:

http://www.iometer.org

SQL Server Wait Events: Taking the Guesswork out of Performance Profiling

A very nice article by Mario Broodbakker on how to troubleshoot slow response times in SQL Server 2000 and SQL Server 2005:

http://www.simple-talk.com/sql/performance/sql-server-wait-events-taking-the-guesswork-out-of-performance-profiling/

Troubleshooting Performance Problems in SQL Server 2005

Here also has a very good presentation on the subject and describes a lot of SQL Server 2005 wait events here:

http://www.microsoft.com/technet/prodtechnol/sql/2005/tsprfprb.mspx

Part 2: Resource Governor CPU Demo on multiple CPUs

BorisB — Sat, 19 Jan 2008 07:00:00 GMT

In some cases CPU demo described in the previous post may behave not as expected.

One such case is doing the demo on multiple CPUs. Even though it appears as Resource Governor is "not working", it is "by design". Let's take a look at what happens in this case.

What is our promise about CPU bandwidth distribution?

When you specify percentage on the pool we will ensure average CPU bandwidth distribution on per scheduler basis for CPU bound workloads. Note, emphasis: average, per scheduler, CPU bound. What does this mean?

For example, you have a 2 CPU machine, and SQL Server runs on both of the CPUs. If you haven't set affinity mask as was indicated in the Setup step of the previous post,

select scheduler_id, cpu_id, status, is_online from sys.dm_os_schedulers

will get you something like:

scheduler_id cpu_id status is_online

------------ ----------- ------------------------------------------------------------ ---------

0 0 VISIBLE ONLINE 1

1 1 VISIBLE ONLINE 1

257 0 HIDDEN ONLINE 1

255 0 VISIBLE ONLINE (DAC) 1

Important item to note here is that we have 2 online schedulers to service normal user requests (scheduler_id < 255). You can compare it with the case when affinity mask is set to 1. You will still have 2 schedulers (with scheduler_id < 255), but one of them will be offline.

Now, for simplicity, let's configure 2 groups and 2 pools as follows:

CREATE RESOURCE POOL pSlow

WITH (MAX_CPU_PERCENT = 30)

CREATE RESOURCE POOL pFast

WITH (MAX_CPU_PERCENT = 70)

CREATE WORKLOAD GROUP gSlow

USING pSlow

CREATE WORKLOAD GROUP gFast

USING pFast

And corresponding classifier function as

CREATE FUNCTION f1()

RETURNS SYSNAME WITH SCHEMABINDING

BEGIN

DECLARE @val sysname

if 'UserSlow' = SUSER_SNAME()

SET @val = 'gSlow';

else if 'UserFast' = SUSER_SNAME()

SET @val = 'gFast';

return @val;

END

Create logins, assign function and make changes effective

CREATE LOGIN UserFast WITH PASSWORD = 'UserFastPwd', CHECK_POLICY = OFF

CREATE LOGIN UserSlow WITH PASSWORD = 'UserSlowPwd', CHECK_POLICY = OFF

ALTER RESOURCE GOVERNOR WITH (CLASSIFIER_FUNCTION = dbo.f1)

ALTER RESOURCE GOVERNOR RECONFIGURE

Add group counters for CPU Usage % and run 1 CPU intensive query UserSlow, then 1 with UserFast. You will observe a picture similar to this:

But how can explain that the resource governor does not obey 70:30 distribution we specified for the resource pools? The answer is quite simple: the queries landed on different schedulers (= different CPUs) and thus, there is nothing to share between them and each of them is taking a full CPU. How can you verify that?

First of all, verify the classification

-- verify the classification

select

sess.session_id, sess.group_id, CONVERT(NCHAR(20), grps.name) as group_name

from

sys.dm_exec_sessions as sess join

sys.dm_resource_governor_workload_groups as grps on sess.group_id = grps.group_id

where

session_id > 50

Where you will see something like

session_id group_id group_name

---------- ----------- --------------------

52 2 default

51 265 gSlow

54 266 gFast

Everything is normal here. Now, let's take a look at how currently executing requests landed on schedulers:

select

r.session_id,

CONVERT(NCHAR(20), wg.name) as group_name,

t.scheduler_id,

r.status

from sys.dm_exec_requests r

join sys.dm_os_tasks t on r.task_address = t.task_address

join sys.dm_resource_governor_workload_groups wg on r.group_id = wg.group_id

where

r.session_id > 50

Which will get

session_id group_name scheduler_id status

---------- -------------------- ------------ ------------------------------

54 gFast 0 runnable

52 default 0 running

51 gSlow 1 running

What you can see here is that gFast and gSlow is working on different schedulers, thus confirming our earlier statement. Thus Red and Blue line corresponds to utilization of 1 full CPU or 50%, since it is normalized to number of CPUs available on the box.

Is this a problem with Resource Governor in general? Certainly not if you have more executing requests than you have CPUs or if queries are running in parallel. But you should keep this in mind if you are trying to give 90% of CPU bandwidth to a single query running serially compared to a parallel query or number of serial queries in another group/pool.

A simple way of thinking about this is: is my workload capable of taking as much CPU as I told it to? Remember, when you set MAX_CPU_PERCENT on the pool, you allowing workloads belonging to the pool to take this much bandwidth on each scheduler. In case of the demo, you are working with a single query workload which produces this initially unexpected result.

Part 1: Anatomy of SQL Server 2008 Resource Governor CPU Demo

BorisB — Fri, 14 Dec 2007 23:19:00 GMT

This post is a step-by-step walkthrough of SQL Server 2008 Resource Governor CPU demo...(read more)