Mat Stephen's SQL Server WebLog : Analysis Services 2005

SQL 2005 Metadata Samples Toolkit and whitepaper now available

Mat_Stephen — Tue, 03 Jan 2006 20:14:00 GMT

Reposted 10/1/06 to include in 'free tools' category

I'm constantly asked "what's Microsoft doing about metadata and do we have a solution."

In the old days (SQl 2000 and 7), Microsoft provided Meta Data Services, a platform designed to help tools and applications manage metadata and information models more effectively. It included the Open Information Model (OIM). A core model of sharable and reusable type descriptions. Microsoft worked with the Meta Data Coalition to publish the Open Information Model as an industry metadata model standard. Initially it was warmly received. However, soon after, a second set of companies in the Object Management Group proposed the Common Warehouse Metamodel (CWM) as a second standard for metadata. Metadata was everywhere!

But the excitement over cross-industry metadata models eventually wore off. For a variety of reasons, ISVs started to move away from industry-wide models and focused on point-to-point mappings between products (primarily focusing on XML-based exchange). The Meta Data Coalition merged with the OMG and cross-industry metadata modeling efforts have been stalled for several years. The world discovered Enterprise repositories and cross-product metadata models are very hard to integrate into existing computing environments (lots of custom applications and tools) and require lots of care and feeding for effective deployment and maintenance (i.e. lots of consulting).

So, Meta Data Services was deprecated in SQL 2005.

With SQL Server 2005 XML is every where, its used to describe all the really useful objects in the product stack. Consequently the possibilities to satisfy many of the needs, that metadata repositories are touted to solve, can now be easily solved by reporting off the XML that describes so many of the key components used in (Microsoft SQL Server) data warehousing (ie used by SSIS, SSRS and SSAS). The important aspects are data lineage, data dictionaries, auditing and impact analysis.

We are very pleased to announce the first incarnation of the "SQL Server 2005 Business Intelligence Metadata Samples Toolkit" http://www.microsoft.com/downloads/details.aspx?FamilyId=11DAA4D1-196D-4F2A-B18F-891579C364F4&displaylang=en. Along with the associated whitepaper http://www.microsoft.com/downloads/details.aspx?FamilyId=182BD330-0189-450C-A2FE-DF5C132D9DA9&displaylang=en it should now be possible to satisfy many of your metadata needs - simply and essentially for free!

The whitepaper also describes the importance of the new Extended Properties that can be applied to objects within the database - these can be used to store all sorts of business information including the all important data lineage and data definition.

Feature Pack for Microsoft SQL Server 2005 - November 2005

Mat_Stephen — Mon, 28 Nov 2005 20:47:00 GMT

Overview

The Feature Pack is a collection of standalone install packages that provide additional value for SQL Server 2005. It includes:

Latest versions of redistributable components for SQL Server 2005
Latest versions of add-on providers for SQL Server 2005
Latest versions of backward compatibility components for SQL Server 2005

http://www.microsoft.com/downloads/details.aspx?familyid=d09c1d60-a13c-4479-9b91-9e8b9d835cdc&displaylang=en

TechNet Briefing: What's NEW in SQL 2005 - Analysis Services & Data Mining

Mat_Stephen — Tue, 01 Nov 2005 14:42:00 GMT

Here's a little self publicity for gig I'll be doing in December:

To register please go to https://msevents-eu.microsoft.com/cui/EventDetail.aspx?culture=en-GB&eventid=118761637

05 December 2005 18:30 - 05 December 2005 21:00 (GMT) GMT, London
Welcome Time: 18:00
Language: English

Microsoft Ltd

Chicago 1
Building 3
Microsoft Campus Thames Valley Park Reading Berkshire RG6 1WG
United Kingdom

General Event Information
Products: SQL Server.

Recommended Audience: Business Professional, IT Professional and Partner.

Session Abstract: Get Ready for SQL 2005! - This session will explain the benefits of the new Unified Dimension Model (UDM) that underpins the On-Line Analytical Processing (OLAP) component of SQL Server 2005 Analysis Services. The UDM represents a radically new approach to OLAP, seeking to combine the best aspects of OLAP with the best of aspects of On-Line Transaction Processing (OLTP). This session will compare and contrast the OLAP engine in SQL Server 2000 with the UDM and explore the new architectural possibilities it offers. Also covered will be an introduction to the new data mining algorithms which will show how they can be employed to gain competitive advantage.

Microsoft .NET Data Provider for SAP

Mat_Stephen — Tue, 04 Oct 2005 15:07:00 GMT

After my recent SQL 2k5 BI session, here at TVP Reading, I had an enquiry asking for more info regarding the SAP provider I showed in the list of providers available to SQL Server Integration Services . The best public information I can find is @ http://msdn2.microsoft.com/en-us/library/ms216599. The provider isn't publicly available at the moment but when the product is released it will be made available for download, and will not be in the box. I don't know a lot about SAP R3 but I believe this provider will also allow access to BW - I'll check, if it doesn't I'll post an edit in the next few days.

OLAP, Darwin and Evolution

Mat_Stephen — Tue, 20 Sep 2005 02:59:00 GMT

Edited 21st Sept 05 to include picture of herbarium sheet.

In my article 'Business Intelligence (BI): The way it is without the blah blah', I mention that I was aware of some exiting research using the SQL Server OLAP engine, the results of which where hopefully about to hit Nature magazine. I couldn't say anymore at the time for fear of jeopardising the conclusions' debut in Nature (If Nature isn't first to publish, it's not in Nature). I claimed this research would demonstrate OLAP, as a technology, can be used as a useful tool in fields well beyond those fenced in by the concept of Business Intelligence. Well this research has indeed made it to the hallowed pages of this erudite organ; if you are prepared to pay the fee, you can find it here 'What Henslow taught Darwin' @ http://www.nature.com/nature/journal/v436/n7051/index.html#Feature.

The article doesn't single out SQL Server per se, but I can assure you, as someone who is in touch with one of the authors, Analysis Services OLAP engine made a significant contribution to the research mentioned. Hopefully follow up material from the authors will document how OLAP was used. Apparently there's so much interest in this story there's even talk of a film!

Let me give you an abstract to show how SQL Server Analysis Services OLAP has contributed to mankind's understanding of its own history and the study of biology therein.

Darwin was a student (1829-31) of Professor John S. Henslow of Cambridge. Henslow is well known for arranging Charles Darwin's berth on the good ship HMS Beagle, the ship that took Darwin to the Galapagos islands. The Galapagos islands are where Darwin discovered 'Darwin's Finches', birds that had 'evolved', by appearance and behaviour, to such an extent that Darwin had to question whether they were different varieties of the same species, or different species altogether. In pursuit of the answer to this question, Darwin formed his seminal concepts that became the backbone to On the Origin of Species.

In the shadow of the afore mentioned article, we must now understand that it was Henslow who had first recognised variation in species, and that it was this recognition and the desire to seek its verification that put Darwin on the Beagle. Furthermore, it was the fidelity of Darwin to fulfil this ambition, with its incumbent rigours in procedure, that ultimately lead him to break free of the creationist shackles that had hindered his mentor, and from which, sadly, his mentor would never be freed.

The research in this article covers 10,172 plants collected by Henslow. By creating an OLAP cube to analyse the sheets of paper holding these plant samples, the researchers show that Henslow organised his documentation by none other than the variation he observed to be within the limits of species variation. Nobody is known to have attempted this before. It is of the stuff that makes a good film that we can now visit his herbarium at Cambridge and see, with newly enlightened eyes, how the plants within it have been carefully arranged to illuminate this most potent observation; a place where, hitherto, this brilliant enlightenment has been so shrouded in the dark silence of ignorance.

Below, courtesy of the authors, is an image of one of Henslow's herbarium sheets: Phleum arenarium. Eight numbered individuals are arranged in order of increasing height. Plants 1-5 were collected 3rd June 1829 at Mildenhall, Suffolk by J.S. Henslow. Plants 6-8 were collected in June 1822 at Liverpool by W. Wilson.

Project REAL—Business Intelligence in Practice - great website!

Mat_Stephen — Tue, 30 Aug 2005 18:27:00 GMT

For sometime a group of Program Managers from the SQL BI team have been working with a customer, Barnes and Noble, to produce a SQL BI solution for their data warehouse and analysis needs using SQL 2005. The leasons learnt from all their work is being continually published to this website http://www.microsoft.com/sql/bi/projectreal/default.mspx in the form of webcasts and whitepapers. With so many new tools and functionality to play with in SQL 2005, I have found their work (what I've seen of it so far), to be invaluable.

Serialising an ADOMD.net CellSet object

Mat_Stephen — Thu, 21 Jul 2005 16:38:00 GMT

I'm grateful to T.K. Anand for this useful code snippet which is useful if you're writing a web services client that consumes OLAP data.

The following Adomd.net code can be used for saving and loading cellsets…

Save:
            XmlReader xmlReader = command.ExecuteXmlReader();
            string xml = xmlReader.ReadOuterXml();
            StreamWriter streamWriter = File.CreateText("CellSet.xml");
            streamWriter.WriteLine(xml);
            streamWriter.Close();

Load:
XmlTextReader xmlTextReader = new XmlTextReader("CellSet.xml");
CellSet cellSet = CellSet.LoadXml(xmlTextReader);

Using SQL against the Unified Dimension Model in Microsoft SQL Server 2005 Analysis Services

Mat_Stephen — Mon, 21 Mar 2005 22:48:00 GMT

Following Deepak's question, asked in his comment on my previous UDM post, I quizzed our internal Yukon alias and received the following useful info regarding using SQL against a UDM; thanks to Akshai Mirchandani.

AS2005 supports a limited set of SQL features – I can’t say for sure what the level of SQL compatibility is. Basically this is what it looks like:

- A cube is a schema (like “dbo”)

- A dimension in the cube is a table

- A measure group in the cube is a table

- Dimensions and measure groups can be joined together using the NATURAL JOIN clause

- You can only aggregate measures and group by attributes

- You can only aggregate measures by their aggregation type (e.g. SUM a measure with aggregation type SUM, MIN a measure with aggregation type MIN, etc.) – its simplest to just use the Aggregate function

- You can apply functions to attributes like NAME/UNIQUENAME/KEY/MEMBERVALUE/etc.

An example would look something like this:

SELECT

Aggregate( [Sales].[Sales MG].[Unit Sales] ) AS [Unit Sales],

Name( [Sales].[Customers].[Country] ) AS [Customer Country Name],

Key( [Sales].[Customers].[Country], 0 ) AS [Customer Country Key 0]

FROM

[Sales].[Sales MG] NATURAL JOIN [Sales].[Customers]

GROUP BY

Name( [Sales].[Customers].[Country] ),

Key( [Sales].[Customers].[Country], 0 )

- [Sales] is the schema (name of the cube)

- [Sales MG] is the name of a measure group

- [Customers] is the name of a cube dimension

Note that although you can specify Name/Key etc in the GROUP BY, they get ignored – the GROUP BY is only performed based on the uniqueness of the attribute.

We also support SELECT * without GROUP BY, but it can be very slow without a good slice.

We don’t support HAVING but do support a simple WHERE clause like:

WHERE

UniqueName( [Sales].[Customers].[Country] ) = ‘[Customers].[Country].[USA]’

AND Name( [Sales].[Customers].[State] ) = ‘[Customers].[State].[WA]’

Microsoft SQL Server 2005 Analysis Services – Unified Dimensional Model (UDM). Can you get your head round it? If you do BI, you’ll need to!

Mat_Stephen — Thu, 10 Mar 2005 14:40:00 GMT

There are very many, and from my own professional perspective, too many (I’ve got to try and learn all this stuff) new features in SQL 2005. Some of them can be described as ‘catch up’ features and some of them might be described as ‘tick in the box’ features. There is one that is wholly revolutionary and which shouldn’t be described as a feature and that is the Unified Dimensional Model (UDM).

I have read and re-read articles that talk about the UDM, some internal, some public, and I’m still trying to get my head all the way round it. It is beautiful, to be sure, but it is a major shift in paradigm. This means you have to chuck out some of the old mind set before you can let the new stuff gel; all of which just adds to the overall learning effort.

To help you on this journey I’m going to point you towards some articles I’ve read and some I haven’t. Firstly however, I’m going to give you some facts you should keep in mind while you travel on this voyage of discovery, facts that should help break the old mind set and lubricate the transition to this new enlightenment. Then I’ll outline some of the key benefits of the UDM to wet your appetite and to keep you motivated through the morphing process.

The big fact: The “UDM- Server” is Microsoft SQL Server Analysis Services.

A UDM provides a single dimensional model for all OLAP analysis and relational reporting needs. So you can use either MDX or SQL
Perspectives are the new data marts
Cubes are largely transparent concepts downgraded to the status of caches
Commonly you’ll only have 1 cube with multiple measure groups and multiple perspectives.
Its better to think of measure groups instead of cubes; partitions now apply to measure groups.
Virtual cubes are gone
Whilst a UDM can gather data from numerous data sources, the need to cleanse data still requires a data warehouse.
A cube is structured around dimensional attributes (previously known as member properties) rather than dimensional hierarchies. Hence the virtual dimension, as a term, is now gone and concept converted to a real, first class, dimension.
UDM has five new dimension types, Role Playing, Fact, Reference, Data Mining and Many to many.

So what are the benefits over AS 2000?

The best of both the OLAP world and the best of the Relational world whilst shedding the worst of both – this is the revolution!
- Flexible schema
- Real time data access
- Single data store
- Simple management
- Detail reporting
- High performance
- End-user oriented
- Ease of navigation and exploration
- Rich analytics
Data from multiple data sources can be made to behave as if they’ve come form one – so you can do table joins across those sources as if the tables were in same database.
Apart from the data warehouse, no data staging is required.
Real-time OLAP (proactive cache as opposed to, and in preference to ROLAP) – a platform for the real-time enterprise and Business Activity Monitoring.
Currency conversion.
Language translation.
No need for dimensional schemas, star or snow flake.
MDX scripting to replace calculated members and to poke calculations into any corner of a cube.
Storing KPIs.
Stored procedures.
Server side conditional formatting.
Categorising of dimensions and other objects.
Display folders - for meaningful grouping objects (KPIs, hierarchies etc).
Controlled default ordering.
Discretisation of continuous values into brackets – e.g. salary bands.
Highly flexibletime dimensions that don’t require underlying tables.
Integration with data mining by way of data mining dimensions.
Scalable, flexible, fine grained server based Security.

There’s more – but I can’t keep my eyes open, its late, I’ve had too many sherbets and wine gums …. I’m off to bed

"UDM - the best of both worlds" - by Paul Sanders http://www.winnetmag.com/Article/ArticleID/41240/41240.html
Microsoft Technet http://www.microsoft.com/technet/prodtechnol/sql/2005/evaluate/dwsqlsy.mspx
Introduction to the Unified Dimensional Model (UDM) - by Paul Sanders http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnsql90/html/IntrotoUDM.asp
The UDM End-User Mode - by Paul Sanders http://www.winnetmag.com/SQLServer/Article/ArticleID/41241/SQLServer_41241.html