Mat Stephen's SQL Server WebLog : SQL Server Integration Services

SQL 2005 Metadata Samples Toolkit and whitepaper now available

Mat_Stephen — Tue, 03 Jan 2006 20:14:00 GMT

Reposted 10/1/06 to include in 'free tools' category

I'm constantly asked "what's Microsoft doing about metadata and do we have a solution."

In the old days (SQl 2000 and 7), Microsoft provided Meta Data Services, a platform designed to help tools and applications manage metadata and information models more effectively. It included the Open Information Model (OIM). A core model of sharable and reusable type descriptions. Microsoft worked with the Meta Data Coalition to publish the Open Information Model as an industry metadata model standard. Initially it was warmly received. However, soon after, a second set of companies in the Object Management Group proposed the Common Warehouse Metamodel (CWM) as a second standard for metadata. Metadata was everywhere!

But the excitement over cross-industry metadata models eventually wore off. For a variety of reasons, ISVs started to move away from industry-wide models and focused on point-to-point mappings between products (primarily focusing on XML-based exchange). The Meta Data Coalition merged with the OMG and cross-industry metadata modeling efforts have been stalled for several years. The world discovered Enterprise repositories and cross-product metadata models are very hard to integrate into existing computing environments (lots of custom applications and tools) and require lots of care and feeding for effective deployment and maintenance (i.e. lots of consulting).

So, Meta Data Services was deprecated in SQL 2005.

With SQL Server 2005 XML is every where, its used to describe all the really useful objects in the product stack. Consequently the possibilities to satisfy many of the needs, that metadata repositories are touted to solve, can now be easily solved by reporting off the XML that describes so many of the key components used in (Microsoft SQL Server) data warehousing (ie used by SSIS, SSRS and SSAS). The important aspects are data lineage, data dictionaries, auditing and impact analysis.

We are very pleased to announce the first incarnation of the "SQL Server 2005 Business Intelligence Metadata Samples Toolkit" http://www.microsoft.com/downloads/details.aspx?FamilyId=11DAA4D1-196D-4F2A-B18F-891579C364F4&displaylang=en. Along with the associated whitepaper http://www.microsoft.com/downloads/details.aspx?FamilyId=182BD330-0189-450C-A2FE-DF5C132D9DA9&displaylang=en it should now be possible to satisfy many of your metadata needs - simply and essentially for free!

The whitepaper also describes the importance of the new Extended Properties that can be applied to objects within the database - these can be used to store all sorts of business information including the all important data lineage and data definition.

New SQL 2005 code samples available

Mat_Stephen — Mon, 02 Jan 2006 20:15:00 GMT

I guess now the pressure is off to release the product we'll see more useful samples comming out of corp; here's a link to the latest December update:

http://msdn.microsoft.com/sql/downloads/samples/default.aspx

Microsoft .NET Data Provider for SAP

Mat_Stephen — Tue, 04 Oct 2005 15:07:00 GMT

After my recent SQL 2k5 BI session, here at TVP Reading, I had an enquiry asking for more info regarding the SAP provider I showed in the list of providers available to SQL Server Integration Services . The best public information I can find is @ http://msdn2.microsoft.com/en-us/library/ms216599. The provider isn't publicly available at the moment but when the product is released it will be made available for download, and will not be in the box. I don't know a lot about SAP R3 but I believe this provider will also allow access to BW - I'll check, if it doesn't I'll post an edit in the next few days.

SQL Server Integration Services Samples - Including Fuzzy Logic

Mat_Stephen — Thu, 15 Sep 2005 19:27:00 GMT

At a recent Technet presentation, someone came up to me and asked me if I could post my Fuzzy Logic sample package to my Blog. I confess I haven't got round to wrapping it all up, but now it looks like I'm off the hook as one has been supplied with the samples that come with the product.

Here's the complete list of samples (as it is today)

Installing Sample Integration Services Packages - Provides instructions on installing sample packages from the stand-alone Microsoft Windows installer.

Calculated Columns Package Sample - Demonstrates how to use the Aggregate and Derived Column transformations to compute values and write the results to a file.

Capture Data Lineage Package Sample - Demonstrates how to get lineage information about data, add it to the data flow, and write the source data and the lineage information to a table.

Process XML Data Package Sample - Demonstrates how to extract data from an XML data file by using an XPath operation, convert the data to a flat-file format, and insert the data into a text file.

Data Cleaning Package Sample - Demonstrates how to clean data by applying exact and fuzzy matching of new customers to existing customers, and how to identify duplicate customers by using fuzzy grouping.

Execute SQL Statements in a Loop Package Sample - Demonstrates how to run SQL CREATE TABLE statements in a loop, evaluate and clean data values, and insert data into tables.

Execute Process Package Sample - Demonstrates how to use the Execute Process task to run an executable that expands a CAB file, read the expanded data, and insert the data into a table.

SMO Tables DBCC Package Sample - Demonstrates how to enumerate the user tables in the AdventureWorks database by using SMO, use a Script task to retrieve the schema and table names, put the names into two variables, and use an Execute SQL task that executes a DBCC CHECKCONSTRAINTS command using the two variables.

AWDataWarehouseRefresh Package Sample - Demonstrates how to refresh the AdventureWorksDW data warehouse from the AdventureWorks OLTP database.

Synchronizing Adventure Works DW Partitions Package Sample - Demonstrates how to automate the synchronization of partitions between relational and Analysis Services databases by using an Integration Services package. The sample package uses the Script task and the Analysis Services Execute DDL task, and illustrates the use of the ASSL and AMO technologies."

Project REAL—Business Intelligence in Practice - great website!

Mat_Stephen — Tue, 30 Aug 2005 18:27:00 GMT

For sometime a group of Program Managers from the SQL BI team have been working with a customer, Barnes and Noble, to produce a SQL BI solution for their data warehouse and analysis needs using SQL 2005. The leasons learnt from all their work is being continually published to this website http://www.microsoft.com/sql/bi/projectreal/default.mspx in the form of webcasts and whitepapers. With so many new tools and functionality to play with in SQL 2005, I have found their work (what I've seen of it so far), to be invaluable.

SQL Server 2005 Integration Services - Top Reference Site

Mat_Stephen — Tue, 12 Jul 2005 18:51:00 GMT

SQL Server 2005 Integration Services - Top Reference Site

From the people that brought you www.SQLDTS.com we now have www.SQLIS.com . This is a top website for examples, information, news and links to MSDN, MSDNTV and blogs etc covering Integration Services. I know I'm going to find it very useful - thanks to the SQLIS team.

London roadshow follow up: Selective execution of SQL Server Integration Service Tasks and Transforms

Mat_Stephen — Tue, 14 Jun 2005 16:42:00 GMT

I had a question, at the London roadshow (8th June), which I wasn’t sure if I could answer reliably, so I promised to blog the answer.

The question: “Can I run selected (with mouse) tasks and transforms from inside the SQL Server Integration Services package designer?”

The answer I have is, for transforms, no you can’t. For tasks you can – or at least just about. You can select single or multiple tasks and then right mouse click and choose to 'disable' the selected task(s) – whereupon they appear greyed out. Then when the package runs, those disabled tasks are not included in the run.

Blogcast: How to add a multicast transform to a SQL Server Integration Services package

Mat_Stephen — Sat, 11 Jun 2005 18:52:00 GMT

Here is a 4min 8sec blogcast showing how to add a multicast transform to a SQL Server Integration Services package (the package created in the previous blogcast). A multicast transform can create two or more replica data flows from one original data flow.

Blogcast: How to create a simple SQL Server Integration Services package

Mat_Stephen — Fri, 10 Jun 2005 12:15:00 GMT

Here is an 8min 13sec blogcast showing how to create a simple SQL Server Integration Services package. The package aggregates 2 million records from a raw data file and passes the resulting 64 aggregated rows into a table in tempdb.

Transfering Data Oracle Data To And From SQL Server

Mat_Stephen — Mon, 23 May 2005 01:07:00 GMT

This post is for two purposes: firstly to point to a comprehensive blog post covering the transfer of data between Oracle and SQL Server using SQL Server 2005 Integration Services - which answers a recuring question from customers and partners with insight, accuracy and the most cherished of properties - honesty.

Secondly I want to sign post Donald Farmer's Blog. Donald is the Group Program Manager for SQL Server 2005 Integration Services (SSIS), thus making him an authrority on SSIS which alone would qualify his blog worthy of comment or linkage. However the catalyst that has motivated this blog, apart from the subject mastter, is the fact that Donald's blog is so well written - it deserves attention full stop. His blog is wonderfully humerous whilst remaining erstwhile. Life is too short to read dry scripts - so read Donalds blogs instead

http://sqljunkies.com/WebLog/donald%20farmer/archive/2005/03/13/8819.aspx

Another Blog with an interesting fuzzy logic example using SQL Server Integration Services to process web server logs

Mat_Stephen — Tue, 03 May 2005 09:00:00 GMT

For those of you interested in implementing fuzzy logic in SQL Server Integration Services Jamie Thomson's Blog has a very good blog post on the subject with an interesting real world example that overcomes a challenging performance problem. Thanks Jamie! http://blogs.conchango.com/jamiethomson/archive/2005/03/30/1203.aspx

SQL Server 2005 Integration Services - Migrating SQL Server 2000 DTS tasks

Mat_Stephen — Tue, 26 Apr 2005 07:03:00 GMT

I have extracted the following from a useful article:

SQL Server 2005 Integration Services: Lessons from Project REAL, http://msdn.microsoft.com/sql/default.aspx?pull=/library/en-us/dnsql90/html/SQL05InSrREAL.asp

SQL Server 2005 Integration Services is a completely new product. While it is able to continue running SQL Server 2000 DTS packages, SSIS has a new design surface, a new object model, and a new internal design. There isn't necessarily an equivalent or direct upgrade path available for all packages that could be designed in SQL Server 2000 DTS. The Migration Wizard represents a best-effort migration attempt.

SQL Server 2000 DTS tasks can be classified into three categories. Your migration experience will depend on which category your tasks fall into.

Category 1: Simple Tasks

These tasks are a straight port into SQL Server 2005 Integration Services. They include:

Execute SQL task
Bulk Insert task
File Transfer Protocol task
Execute Process task
Send Mail task
Copy Objects task
Execute Package task

If your package only includes these tasks you should have a good migration experience.

Category 2: Opaque Tasks

These tasks will migrate, but might not work once they have been migrated. This is especially true of the ActiveX and Dynamic Properties tasks. These tasks typically interact with the SQL Server 2000 DTS object model and the SSIS object model is not backwards-compatible with SQL Server 2000 DTS. Opaque tasks include:

ActiveX Script task
Dynamic Properties task
Analysis Services DTS Processing task

You will almost certainly find that migrating packages with these tasks requires the new development of at least these task components, and may entail a deeper review of your package design.

Category 3: Encapsulated Tasks

These tasks are not migrated. When the Migration Wizard is complete, it will have created a new DTS 2000 package that contains these tasks, and then use the Execute DTS 2000 Package task to call into the newly created packages to perform these SQL Server 2000 DTS tasks. They include:

Custom tasks
Data Pump tasks
Data Driven Query task
Transform Data task
Parallel Data Pump task
Copy Database Wizard tasks

For this category of tasks, you must either rewrite the components in SSIS if you want them migrated, or continue to call them using the Execute DTS 2000 Package task.

Microsoft SQL Server 2005 fuzzy logic in matching and grouping - how does it work in SQL Server Integration Services?

Mat_Stephen — Thu, 07 Apr 2005 17:11:00 GMT

Following on from a presentation 'First Look at SQL Server Integeration Services' to some MVPs here at TVP Reading, I am responding to the a question I was asked, 'how does SQL Server's fuzzy logic work'. I was meant to explain how the confidence and similarity values are calculated - I had no idea, so here's the results of some research I've done.

Thanks to Donald Farmer I can point you to:
Technical paper detailing the matching technology used by Fuzzy Lookup
Robust and Efficient Fuzzy Match for Online Data Cleaning. ACM SIGMOD 2003, San Diego. Surajit Chaudhuri, Kris Ganjam, Venkatesh Ganti, Rajeev Motwani.

ftp://ftp.research.microsoft.com/users/DataCleaning/sig03_FM.pdf

Thanks to Grant Dickinson, he adds:
I believe this is the algorithm used:

http://www.vldb.org/conf/2002/S17P01.pdf

Here's some info on the researcher:

http://research.microsoft.com/%7Esurajitc/

A lot of this looks incomprehensible, so to those without a degree in Maths I’ve also included a piece from Books On Line

Controlling Fuzzy Matching Behavior

The transformation provides a default set of delimiters used to tokenize the data, but you can add token delimiters to suit the needs of your data. The Delimiters property contains the default delimiters. Tokenization is important because it defines the units within the data that are compared against each other.

The Fuzzy Lookup transformation includes a property that specifies the maximum number of matches that the transformation can return. The transformation returns zero or more matches up to the number of matches specified. Specifying a maximum number of matches does not guarantee that the transformation returns the maximum number of matches; it only guarantees that the transformation returns at most that number of matches. If you set the maximum number of matches to return to a value greater than 1, the output of the transformation may include more than one row per lookup and the some of the rows may be duplicates.

Each match includes a similarity score and a confidence score. The similarity score is a mathematical measure of the difference between the input record and the record that the reference table returns. The confidence score is a measure of how likely it is that a given value is the best match among the matches found in the reference table. The confidence score assigned to a record depends on the other matching records returned. For example, matching St. and Saint returns a low similarity score regardless of other matches. If Saint is the only match returned, the confidence score is high. If both Saint and St. appear in the reference table, the confidence in St. is high and the confidence in Saint is low. However, high similarity may not mean high confidence. For example, if you are looking up the value Chapter 4, the returned results Chapter 1, Chapter 2, and Chapter 3 have a high similarity score but a low confidence score because it is unclear which of the results is the best match.

The similarity score is represented by a decimal value between 0 and 1, where a similarity score of 1 means an exact match between the value in the input and reference table columns. The confidence score, also a decimal value between 0 and 1, indicates the confidence in the match.. If no suitable match is found, similarity and confidence scores of 0 are assigned to the row and the output columns copied from the reference table will contain null values.

In some cases, Fuzzy Lookup may not locate appropriate matches in the reference table. This can occur if the input value that is used in a lookup is a single, short word. For example, the input value "helo" is not matched with the value "hello" in a reference table when no other tokens are present in that column or any other column in the row.

The transformation output columns contain the input columns that are marked as pass-through columns, the selected columns in the lookup table, and the following additional columns:

_Similarity, a column that describes the similarity between values in the input and reference columns.

_Confidence, a column that describes the quality of the match.

The transformation uses the connection to the SQL Server 2005 database to create the temporary tables that the fuzzy matching algorithm uses.