Margarita Naumova's life in SQL

The SHRINKDATABASE Evil

maginaumova — Sat, 23 May 2009 12:30:13 GMT

I mentioned in my previous Fragmentation post to not use the SRHINKDATABASE during the maintenance, and to switch off auto-shrink option. Some of you probably believe me and just don’t use it, but some of you wait an explanation and don’t accept advises based on promises :)

Well, let me prove you what the shrinkdatabase command can do for you to make your life.. everything else but not easier. The command actually sound very good – why not shrink the file; it has so much free space in the middle, when I delete some records or a whole table; I need the disk space, and I could make my database compact; etc. Those are the questions you could ask yourself and think that shrinkdatabase will be an answer, but neither of them actually are resolved using shrinkdatabase. Totally the opposite result is achieved. Actually I didn’t realize what it is doing until my MCM Training where we did some demos and Kimberly mentioned it in the class. I decided to use some of them and to add some more. So let’s do some testing:

1. Lets create a database ShrinkDemo

CREATE DATABASE ShrinkDemo;
GO
USE ShrinkDemo;
GO

SET NOCOUNT ON;
GO

2. Let’s create a table in the beginning of the database and fill with data

-- Create the 10MB first table in the begining of the data file

CREATE TABLE firstinfile(
c1 INT IDENTITY, c2 VARCHAR(8000))
GO

-- Fill up the firstinfile table
DECLARE @a INT;
SELECT @a = 1;
WHILE (@a < 1280) -- insert approx 10MB
BEGIN
    INSERT INTO firstinfile VALUES (
        REPLICATE ('a', 5000));
    SELECT @a = @a + 1;
END;
GO

3. Create the second table which will be after the first in the database file

CREATE TABLE secondinfile(
c1 INT IDENTITY, c2 VARCHAR(8000))
GO

--create an index

CREATE CLUSTERED INDEX second_cl
ON secondinfile(c1);
GO

-- Fill in the secondinfile table
DECLARE @a INT;
SELECT @a = 1;
WHILE (@a < 1280) -- insert approx 10MB
BEGIN
    INSERT INTO secondinfile VALUES (
        REPLICATE ('a', 5000));
    SELECT @a = @a + 1;
END;
GO

4. Now you can use the Allocation Info add-in () It is not very precise but you can still use it to take a look at your objects fragmentation inside the database.

I like this picture! Now you can recognize the placement inside the data file of both tables one after another, and see that the Avg Frag % is around 0, they are not fragmented in this moment.

We can check fragmentation using this code and the DMVs also

SELECT
    avg_fragmentation_in_percent,
    fragment_count
FROM sys.dm_db_index_physical_stats (
    DB_ID ('ShrinkDemo'),
    OBJECT_ID ('secondinfile'),
    1,
    NULL,
    'LIMITED');
GO

The result is the same

5. Now lets remove the firstinfile table creating a 10MB emtpy space in the beginning of the file

The secondinfile table is still 0% fragmented.

6. Now the next ‘logical’ action could be let’s shrink the database to get rid of the empty space.

DBCC SHRINKDATABASE (ShrinkDemo);
GO

Now what it the result of the srhinking?

Well, the secondinfile table is in the beginnig of the file, BUT note the Avg Frag % !! It’s 100%! Right after creating a Clustered index for this object!

Here is the result from the DMV query

1277 are total number of fragments (number of blocks with continuing pages in order). We have a total of 1279 rows, which means that almost every page is a separate fragment, they are not in continuous order.

Why is this fragmentation occurs after shrinking the database. It is actually a logical scan fragmentation when the order of pages in the data file does not confirm to the order in allocation map of the object. Which means the object pages are totally mixed and not in continuous order as they should be. This prevents from executing an optimized read ahead reading of pages, such destructing the reading performance of scans. The result is such because the shrinkdatabase do the following – it gets the last page of the object and move it at the beginning of the file or empty space, it then takes the page before the last one and move it after the first was placed, and so on. I tried to visualize it on the picture bellow:

So, we ended up with totally fragmented table just after creating the clustered index. To make the picture even more complicated I will continue with focusing on the size of the database. After shrinking the database is 11MB, which means it has been reduced in size and it doesn't have any empty space in it. But achieving the desired size reducing we gain fragmentation. So we have to fix this further. To eliminate the fragmentation I will reorganize the index, in this case I could use index reorganize, because it eliminates just the logical fragmentation. and orders object pages.

7. So I execute the following

alter index second_cl on secondinfile reorganize

As a result

But what happened to the database size? It actually increased to 13MB, which is not too much because of reorganizing the index. But if I choose further to rebuild the index the database size increases to 22MB, which is approx 1.5 times the table. So we are actually reached the initial size the database was before shrinking (containing 10MB table and 10MB empty space) which mean we did nothing but making unnecessary efforts and actions.

More over, the allocation map of the database object looks like in the picture bellow. It is because rebuilding index creates a parallel indexing Btree which it populates, after that it switches it as a new cluster index. That’s why we need a free space in the database, else the SQL server will increase the database allocating and initializing new chunks.

And the final curious think is what would happen with allocation objects map if I create again the firstinfile table and populate it with 799 rows. Back to the beginning :)

Well, you probably have some questions at this time

Why there is an auto-shrink option?

Honestly I don’t know. It is absolutely un-useful. Imagine this shrink action on every 30 min executing on your database. It could be a disaster for your database IO operations, fragmentation and respectively performance. It is probably one of those legacy options that remain trough versions. Anyway, don’t ever switch this option on.

When actually should I use dbcc shrinkdatabase?

Well, I really couldn’t find a use case for shrinkdatabase. But there is a scenario to use shrinkfile statement. For example: you have deleted a large amount of data in your data file and some small objects remain. Then you can create new filegroup with a new data file (.ndf) in it, you rebuild clustered index on your existing objects on the new filegroup. This rebuild actually moves objects to the new filegroup. Then you can use shrinkfile with emtpyfile option to truncate your old data file, and then remove it from the database.

How can I deal with empty space inside the database?

You don’t have to. The empty space in the database will be used in your further operations as showed above. Empty space is not an issue. If it is a large empty area as the example above it will be used for the next created object. If it is a small empty space as soon as you rebuild indexes it will be outside of the object pages. So don’t worry about empty spaces in your database, monitor the object fragmentation and fix it regularly. If you insist to remove those spaces use the shrinkfile as explained in the above question.

Fragmentation – the database performance killer

maginaumova — Fri, 22 May 2009 17:18:40 GMT

Fragmentation is something that we don’t usually pay enough attention but it causes a very large percents of cases with performance degradation. Sometimes it plays a main destructive role for the overall performance of our database.

Disk fragmentation

There are different kinds of fragmentation. One type of fragmentation is a disk fragmentation, where the disk the database files and log are placed is highly fragmented, causing chunks of db files to be separated on the disk, such causing IO operations for reading extents to increase and additional moving of the disk heads. This is especially true when the disk is not dedicated to the database files, the server is used as a file server and other files are written often to it.

The second scenario is when files of more than one database is placed on the db disk and databases have their default settings. The database default settings are performance killers, especially options for initial size and file growths. See bellow in the '”defaults” fragmentation.

How to deal with it

Use Windows Disk Tools to check disk fragmentation, the tool visualize how your files are fragmented 0n disk. If you realize that your disk is fragmented then you have to plan a downtime to defragment. You can use any tool, including windows out of the box defragmenter to deal with disk fragmentation. BUT first you have to:

Stop SQL Server service
Copy database files from the drive you intent to defragment (this is your file level backup in case your tool cause a damage in the database file)
Run the defragment tool
Start SQL Server
Check your databases, best tool to use for checking is DBCC CHECKDB, because it scans all pages inside the database and if there is an physical error it will found it.

The “defaults” fragmentation

When you create a database the default size and default growths settings are 1MB, 10%growth. For TLogs they are …This means every time the database file needs to increase it creates a chunk of .. MB, which is an operation of allocating and initializing disk space on the next disk blocks available which is not necessary to be continuous. Chunks of this size are absolutely not enough for any database operation, especially with large data loads. The result is an IO disaster, one way – in executing an expensive operation of allocating disk space and initializing for database usage at a time of executing transactions, and another - creating a fragmented database file.

According to the TLog, the default settings for initial size and log growth create TLog fragmentation too. In which way it happens? TLog file contains chunks called Virtual Log Files (VLFs). The number and size of VLFs depends on Tlog file size and growth settings. When they are not set correctly or left as defaults a huge number of VLFs are created which means that the TLog file is fragmented, causing delays in saving pages during transactions, when backing up and truncating the log file, such causing an overall degradation of the workload performance. You can check the number of VLFs in you database log file executing:

DBCC LogInfo(‘yourdbname’)

The number of rows in the result set shows the number of VLFs. When the number is more than 80-100 rows the TLog file is fragmented.

What are the rules SQL Server uses to create VLFs?

<64MB chunk – 4 VLFs

>64MB and <1GB – 8 VLFs

>1GB – 16 VLFs

Let me give you an example:

According to this rule if your log file has default settings for growth (1MB initial size and 10% growth) then 10% increase of 1MB file will create a chunk of 102KB, which is bellow 64MB, thus a 4 VLFs will be created with 25KB size each, next growth will be 110KB, again creating 4 VLFs, with 26KB each, etc.. Imagine if the file needs to growth from 1MB to 200MB, what about 2GB? it will create a hundred of VLFs.

Another example, imagine a Log file which has to increase from 100MB to 200MB:

With default settings of 10% it would have to increase approx 10 times to reach 200MB, and thus will create a total of 40VLFs

If we change the setting and set 64MB growth, then it would have to increase 2 times, thus creating 16VLFs.

It is easy to test this scenario, actually this was part of my MS Days 2009 Top 10 Presentation demo:

create a simple database and leave the default settings
create a simple table that will grow quickly and will generate huge log file. To reach this fast use one of the field for the table as char(8000) with a symbol as a default. If you want to calculate rows per second inserted you should include in the table a filed of date/time type and default(getdate())
backup the database to activate full recovery model (if you switch the db to full recovery it will not act as a full recovery model unless you do a full database backup)
In new query window start a loop doing an insert to the simple table with default values.
You can monitor Percent Log Used, Log Growth, Database Growth counters in PerfMon during the loop
Stop the loop after a while (8-10min)
Execute DBCC LogInfo(‘YourSimpleDBName’) and view the number of rows there
If you want to calculate how many rows are inserted into the table per second then you can use my script
Now you can execute Kimberly’s step 8 from 8 steps to better TLog troughput to reduce the Tlog fragmentation
Now truncate the simple table and do the test again, compare the rows per second inserted :) Note that it is gained only from changing one default setting.

How to deal with it

There is one main recommendation here – don’t leave default settings for initial size and growth of your database and TLog

What are the best settings or are there any? Well, those who know me probably already know the answer: it depends :) And Yes, it depends on the workload, backup schedule, etc, but in general

For the database file do some rough capacity planning to estimate initial size including tables, data and indexes, once defined you cannot make the settings lower. You can change the settings at any time.

Define the growth based on your regular operations, take into account the maintenance – rebuilding indexes require large amount of database space (approx 1.5 times your table size). Leave your database with 20-30% more free space and larger that data in it. You can monitor when the database file performs growths (there is a counter in PerfMon) and schedule the growths with alter database during non-working hours. This will eliminate the expensive disk allocations during the transactions.

Don’s use percentage for growth settings, use fixed size instead

If you use SQL Server 2005/2008 Enterprise Edition then don’t forget the Instant File Initialization setting. This will allow you the SQL Server to optimize and do a faster initialization when database file growth is needed. (By the way there are signals that the option will be included in standard edition too, which will be very cool! )

Turn OFF Auto-Shrink and don’t use SHRINKDATABASE nor SHRINKFILE as part of your maintenance step, never! I will do a separate post why the shrink is evil, but please trust me at this point.

For the log file

If your Tlog reaches 100GB or it is between 100GB and 300B, start at 8GB, and increase with 8GB chunks – 16, 24, 32, 40.. You will get 16VLFs on every 8GB, which is a 512MB VLF, such the log will clear on every 512MB on normal operations.

It is not recommended to set the growth more than 8GB, because you will end up in another direction of very large VLFs in size and clearing the log will be a challenge. Remember that the Inactive portion of the log is cleared.

What to do if you have your Log fragmented already. There are a perfect instructions how to deal with fragmented Tlog files on Kimberly’s blog 8 steps to better TLog troughput , look at step 8.

The fragmentation inside the database – index fragmentation

Well, this a huge topic and I will just mention thinks in general here and do a separate post on Index fragmentation and optimizations.

Index fragmentation is something you should monitor regularly and I will provide you with some scripts to do it in my next post. This kind of internal database fragmentation is not reached because of wrong database settings but as a result of regular database activities – inserts, deletes and updates in some cases. So it’s normal to have your indexes fragmented at some time, the point is to catch the time when further fragmentation will cause a degradation of performance. Actually based on type and usage of you indexes their fragmentation could reflect not only queries, but data modifications also.

How to deal with it

Regular monitoring and idx maintenance is the key here. The indexes should be monitored for physical and logical fragmentation periodically. Could be once daily for some intensive index/table usage, and most often once a week. During the monitoring you should check percentages of logical and physical fragmentation for particular index and the general rule is if the idx fragmentation (both logical and physical) is more than 30% you should deal with it either with ALTER INDEX REORGANIZE (for logical) or ALTER INDEX REBUILD (for physical).

There is one very important point here, and it is analysis and regular optimization of the indexing structure of your database, which is

How much of your indexes are not used and which are they. Those indexes are very important to be removed, beside consuming database space, they consume resources during maintenance increasing your maintenance time. Probably most important is that non-used indexes slow down the performance of your data modifications.

Which are you missing indexes. These are indexes that are not presented in your database but could be useful to satisfy you queries. You should deal with this recommendations carefully, testing is required. Sometimes it is more important to remove unused index rather that to create additional index.

Analyzing the indexing infrastructure is an exercise that should be done once monthly. This will guarantee that your database is in a good level of optimization.

SQL Server 2008 SP1

maginaumova — Sun, 10 May 2009 21:51:00 GMT

Almost a month ago the SQL Server 2008 SP1 was launched. I had a question or some kind of complain that it cannot be installed or it hangs. So I decided to test it. Well, installation was ok, it passed successfully no matter if the instance is stopped or started. So you could apply SP1 on working or not SQL Instance, it is not a problem. But I found out that setup documentation mentioned if you want to bypass the system restart you have to prepare somehow before running the SP1 installation. Along with backing up your user and system databases, which you definitely should do, you have the following options to prepare the setup:

Stop Services and Applications — To avoid a possible restart of the system, stop all applications and services that make connections to the instances of SQL Server that are being upgraded, before installing SQL Server 2008 updates. These include SQL Server Management Studio, Business Intelligence Development Studio, Control Panel, and Add or Remove Programs.

I passed this step attempting to avoid the restart. BUT it didn’t help. Finally the restart was required. Beside that the document says:

To eliminate the requirement to restart your computer after update installation, Setup will show a list of processes that are locking files. If the update Setup program must end a service during installation, it will restart the service after the installation finishes.

If Setup determines that files are locked during installation, you might have to restart your computer after the installation finishes. If it is required, Setup prompts you to restart your computer

Actually the setup didn’t say anything about locked files. It just passed and wanted a restart at the end. As for any SQL installation there was a log file generated during the setup. It is placed in <your installation folder>\Microsoft SQL Server\100\Setup Bootstrap\Log\<date and time of your installation>

SQL Server 2008 SP1 download link here

All SP1 fixes: http://support.microsoft.com/kb/968369

Probably the main improvement we have with SP1 is the Slipstream and Uninstall features. We actually experienced some kind of slipstream with SQL 2008 setup where we had to fix the installation before actually installing the product. It included the installation of CU1 first to install the fixed setup files without expecting some messages but just checking add/remove programs/control panel that they have been installed, and after that run the actual SQL Server 2008 Setup (a very good css posting on this). Creating a slipstream where you can pack installation of SQL 2008 with its SP consists of some similar steps. There is a very detailed post on creating a slipstream here.

A couple of days after releasing the SP1 the first CU1 for SQL Server 2008 SP1 was released too. It is actually a CU4 for those who want to have SP1 and CU4, they have to install SP1 and CU1 for SP1. SP1 contains CU1-CU3. There is a very important point – if you have already applied CU4, don’t install SP1, you will loose the fixes coming with CU4. There is a very good posting again from the CSS for what to do in this case:

http://blogs.msdn.com/psssql/archive/2009/04/09/sql-server-2008-sp1-and-cumulative-updates-explained.aspx

MS Days 2009

maginaumova — Wed, 29 Apr 2009 22:36:00 GMT

Няма как да пропусна да отбележа този форум, макар и със закъснение. Имах 3 презентации (добре че махнаха четвъртата, понякога не мога да спра устрема си да презентирам и ставам досадна, членовете на юзер групата най-добре знаят :)

Реших да експериментирам тази година: да говоря за нещо полезно (Top 10 things..); да направя нережисирана и забавна сесия (SQL Unplugged); и разбира се да има нещо съвсем ново (SQL Kilimandjaro). Май се получи, с изключение на това, че хич не бях предвидила, че демо сървъра на който бяха виртуалните машини съвсем ще издъхне при едновременното изпълнение на демота във всички зали. Получи се интересен ефект: тъкмо щях убедено да докажа че една елементарна промяна на default параметрите на TLog-a води до сериозна оптимизация и вмъкване на почти два пъти повече записи за единица време (което е факт, тествах го десетки пъти), хост машината така зависна че се получи точно обратното като резултат.

Unplugged сесията беше много забавна, толкова въпроси имаше. Обещавам ви най-интересните от тях да блогвам, както и още за нещата които не трябва да правите с базата си :)

Аз определено се забавлявах, силно се надявам да е било така и за хората в залата, както и да е било полезно.

Можете да свалите Top 10 презентацията: Link to download the Top 10 worst pptx file

От къде започна всичко..

maginaumova — Wed, 29 Apr 2009 18:09:00 GMT

Съвсем наскоро получих изненадващ подарък-покана, който съвпадна със създаването на този блог. Обичам съвпаденията! Точно когато се чудих от къде да започне ‘Magi’s life in SQL’ дойде една книжка заедно с покана за конференция от Икономическия Университет във Варна. Книжката се казва „Бази от данни” с автор Мария Кашева и страхотно посвещение: „На Маргарита за да си спомня началото..” Много много мило! Почти ме разплака и ме хвърли в спомени. Да, така започна моето приключение в света на базиите от данни, с доц. Мария Кашева и асистенството ми във Варненския ВИНС. Страхотни години, които ми дадоха много и ми оставиха незабравими приятели и мили спомени (както и скъсани на изпити студенти, които се надявам да не ме помнят с лошо :)

Поканата е за конференция „Информационни Технологии в Управлението на Бизнеса” организирана от катедрата през октомври (16-17). Участието не е под въпрос, за мен ще бъде удоволствие и по-скоро емоционално преживяване отколкото професионално.

Welcome to my new blog!

maginaumova — Mon, 27 Apr 2009 22:09:00 GMT

Here we go, I have just created a blog! I didn't know that it will take too long for me. My friends often say: What’s going on with you? Why don’t you publish?

Well, everything is just on time. I suppose now I know and I feel that there is something I can share with others. Actually there will be from now on :)

I promise to fill it up these days!

Enjoy