Some things just really work well together. Peanut butter and chocolate. Abbot and Costello. Coffee and...well...anything. You get the idea. But there are some real advantages in using SQL Server 2008 (and of course the upcoming SQL Server 2008 R2 release) with Windows Server 2008 and higher. It's not just that they are both better than their predecessors, SQL Server actually takes advantage of the improvements in Windows Server 2008.
One practical example is in how Windows Server 2008 handles the infamous "drive offset". This is a small block size movement from the first part of the hard drive sectors - it's an internal thing - but it causes real issues with software that exercises the I/O subsystem, and makes its own calls there. Like SQL Server. In the past, the data professional had to follow a process called "Partition Alignment", and this had to be done when the system was set up. That's all now a thing of the past - with SQL Server 2008 and Windows 2008 Server, this just happens.
Another example is in how Windows 2008 Server deals with the "sliding TCP/IP window". This enhancement directly affects how fast SQL Server can send large frames of data - especially with Replication and large binary objects. At Microsoft we noticed tremendous speed gains just by moving to Windows 2008 Server.
There are lots of other examples - from new virtualization and consolidation changes in both products to clustering enhancements, and now in SQL Server 2008 R2 the ability to run the "sysprep" utility after SQL Server has been installed.
You can read more about this "pairing effect" in this White Paper.
And be sure to check out John Kelbley's Post on the Windows Server blog where he also talks about ways that SQL Server and Windows Server work "Better Together".
There has been a lot of buzz lately about "the cloud" and what it means for IT, and the organizations they serve. I just finished watching Steve Ballmer deliver Microsoft's vision for the cloud, wondering if it would be different than the offerings of other firms. It feels like it is different. So what is that strategy, and more importantly, what does that mean to the data professional?
First, let's recap what Steve mentioned for Microsoft's strategy. The word "cloud" can mean a lot of things, from hosting data (often called "somebody else's hard drives) to running software where the user logs into an application hosted on a web server somewhere (called Software as a Service, or SaaS). Microsoft actually has both. In fact, we've had things like Hotmail (which is a service) and the various Live offerings for quite some time. And we've had XBox Live, which is a hosted environment that is kind of a hybrid - there's a "fat" or hardware client that talks to the main server out on the web. There's also an offering of a complete Microsoft Office, SharePoint services and even LiveMeeting in the cloud - nothing to install, runs on lots of platforms, and more (details here).
But there's something new. Microsoft has two new offerings, called Azure, and SQL Azure. Azure is more of a programming platform that can host data, and SQL Azure is SQL Server in the cloud. But SQL Azure isn't just a server you rent - we actually maintain the systems, handle the optimizations and so on. You create databases and database objects. You can take the data from SQL Azure and send it to a local SQL Server, and vice-versa. There's also the ability (through something called Sync Services) to replicate data between the two.
So how does the Data Professional meld in the "cloud" to our day-to-day systems? How do we use Microsoft's strategy in our own strategies? I've seen a couple of interesting uses so far from those that are trying it out:
Front-End Start, Back-End Archive: In this mode, companies spin up an application quickly (with no server build) into Azure, backed by SQL Azure. They are able to quickly deploy the app, and if it grows they can bring that data in-house or even bring a subset of that data locally for reporting, keeping a smaller data-set up in SQL Azure.
Start There, Stay There: Some of the companies I've seen are taking the application ideas and starting them in SQL Azure or Azure (or both). No capital expense, no hardware purchases, no installs, nothing to deploy. When the app is developed, you just point your clients there and off they go. It's a pay-as-you-use system, so the costs mimic the profit.
Start There, Come Here: In other cases, organizations want to start projects quicker than they can get hardware and software installed. So they follow the previous process, and then bring the code and the database in-house when they are ready.
There are other strategies, such as using the cloud as part of High Availability/Disaster Recovery or a remote-office access system and so on, but whatever your reasons, you need to spend a little time getting familiar with the cloud - and Azure and SQL Azure should be high on your research list. Here's some places to get started:
Azure Overview: http://tinyurl.com/y8s52vh Windows Azure Training Kit: http://tinyurl.com/5vrt7q Writing a Azure Program in 5 steps: http://tinyurl.com/ye6chog Azure Data Sync: http://tinyurl.com/ylsykfb Future of programming with Azure Video: http://tinyurl.com/ykqsbsf Reference list for Windows Azure: http://tinyurl.com/yze9azr
A friend insists that we'll only know the recession is over when software vendors no longer start every whitepaper with the phrase "In these tough economic times ..." It may be as reliable an indicator as any.
Meanwhile, in these tough economic times, I often read of factories suffering so badly that they are "operating at only 50% of capacity." For a manufacturing plant, such low utilization is a disaster. So, gentle reader, what do you think would be the average utilization of your data center's capacity? Nothing like 50%, that's for sure. Typical enterprise servers run at about 10% utilization according to a recent McKinsey report. They may, just may, be able to reach as high as 35% with a concerted effort.
There are many good excuses for this situation, with both business and technical justifications. Enterprise applications on the same server do not always play together nicely. One will demand all the memory it can get, sulking unresponsively in a corner if it can't get it; another will push over less aggressive applications in order to grab more CPU. In the SQL Server world, we're working on that continuously, with every version adding better resource governance and management. (See http://bit.ly/ss2008rg for specific information about SQL Server 2008.) Then again, these same applications are often mission-critical and it is business requirements which force us to isolate them: from the risk of downtime, or other disruptions. Approaching our problems in this way, it's quite easy to add a new server for this app, and another server for that one, and sure enough, the result is soon 10% utilization.
It won't do. There's a capital cost, and fixed running costs, for every server we add, not to mention the environmental considerations of wasted energy and resources that weigh heavily on many of us, recession or not. I have visited datacenters in emerging economies from Egypt to China where simply having enough power available is a problem and resource management is imperative.
In the database world, we have traditionally approached these problems by running multiple native instances of servers on the same box. This can indeed consolidate hardware and reduce costs. Nevertheless, IT managers and DBAs are increasingly looking to virtualization. Why? There are numerous advantages. For example, with virtualization each application can have a dedicated, rather than shared, Windows instance: especially useful for mixed workloads; and with virtualization, instances are limited only by the capacity of the machine, rather than the native 50-instance limit.
SQL Server 2008 works exceptionally well with Windows Server 2008 R2 and Hyper-V to deliver effective virtualization. In SQL Server 2008 R2 (shipping in the first half of 2010) we will support up to 256 logical processors on that platform to scale those solutions even further. There are some great scenarios for this. Business Intelligence applications such as Analysis Services and Reporting Services are prime candidates, especially when mixed BI and operational workloads peak at different times. Virtualization has other benefits for the database user: for example, the lifecycle from development to test to production becomes easier to manage with a consistent, virtualized, environment.
It's really worth considering virtualization, and building up your understanding of the technology and requirements. There's a great whitepaper at http://bit.ly/sqlcatvirtual with sound advice and background for any SQL Server 2008 DBA considering this technology. Good material to have to hand, in these tough economic times.
I think I may have set myself an all-but-impossible task: to choose ten bloggers who write about SQL Server, and who have been outstanding in the last year. Nearly impossible, not because I can't find ten, but because there are so many more worthy of recognition. In addition, many of those I will not be including are friends and colleagues, so the task may be as thankless as it is difficult.
Nevertheless, having set myself the goal, I may as well get on with it. My method was simple enough. I started with those blogs I subscribe to, and, of those, found the ones I bookmark most often. These were neatly objective measures, but I was still left with about 20 blogs to consider. Then I had to find some more subjective criteria: are the blogs helpful, insightful, original, well written, newsworthy, and so on. I excluded official Microsoft blogs, focusing instead on the community blogs, so there is only one Microsoft team member on the list.
Here then are My Top 10 for this last year. To be fair to the others who so narrowly missed out, I'll publish a longer blogroll later of those who I consider to be essential reading. For now, let me know what you think of my top ten, in strictly alphabetical order.
Bob Beaucheminhttp://www.sqlskills.com/BLOGS/BOBB/Like most of the bloggers in this top ten, Bob is an active and excellent speaker and writer. Bob is notably excellent when writing about data access and programmability, areas which require both sound understanding of the database technology and the ability to work with, and explain, the latest programming models. If you're an application developer working with SQL Server, then Bob is essential reading ... and don't miss his conference sessions either!
Rob Colliehttp://powerpivotpro.com/Rob is the only Microsoft employee on My Top 10 list because his blog is really very independent and hosted with a quite separate presence and identity. Rob has set out to create a compelling blog for the new PowerPivot product and he does a great job synthesizing his years of experience in the Excel world with his detailed knowledge of the PowerPivot technology. Even better, Rob presents compelling, easy-to-understand scenarios with a great sense of humor. If you're interested in PowerPivot, you need to follow this blog.
Kasper de Jongehttp://business-intelligence.kdejonge.net/This blog has been a revelation to me this year. Kasper works in the Netherlands and blogs on BI topics. One outstanding feature of his blog is his use of copious screenshots. Often, with a new product just out in public like PowerPivot or the new Report Builder, Kasper sedulously records his experience with setup and first impressions, all captured with useful screens and comments. Even I learn stuff about setting up our BI products here! It's not just about installation either: Kasper explores many new features with the same careful approach.
Andy Leonardhttp://sqlblog.com/blogs/andy_leonard/default.aspxI really enjoy Andy's blog, not just for the technical posts (especially about SSIS), but for the way he writes with a perceptiveness and passion about the community of SQL users. Andy persuades, cajoles and encourages SQL Server users to get out and be part of something bigger: whether blogging, or simply attending a conference or event. Even better, Andy is always very clear about how community support fits in to an often challenging and difficult career path.
Sean McCownhttp://www.infoworld.com/blogs/sean-mccownNow this is a kick-ass blog. In fact, often times you get the impression that Sean's key motivation in sitting down to blog for the day is just to kick some ass. But he chooses his victims well! Whether it is Microsoft's product teams, officious auditors, or even himself (for delivering a bad presentation), Sean is typically forthright and on target. Sean is also, like Andy Leonard, excellent at supporting DBAs in their career and personal development, with advice in the last year on technical skills, interview techniques and even office politics!
Adam Machanichttp://sqlblog.com/blogs/adam_machanic/default.aspxThe sheer breadth and depth of Adam's posts are testaments to his knowledge of SQL Server. I have only two things to say: read this blog, and try every code sample Adam posts. You'll be better for it.
Paul Randalhttp://www.sqlskills.com/BLOGS/paul/Paul is a former Microsoftie who often draws on his detailed understanding of the relational engine's internals to give unique insights on his blog. As an expert on DBCC, this blog is simply essential reading if you are interested in recovery or repair - it's title is, in fact,"In Recovery." Even better, if you want to avoid recovery and repair, you need to read this. Paul also writes very entertainingly, which really helps with the often deeply technical matter.
Jamie Thomsonhttp://sqlblog.com/blogs/jamie_thomson/Jamie's previous blog used to be called "SSIS Junkie." I don't think he has quite kicked the habit, as his technical posts about SSIS are always excellent, but there is certainly a wider range of interests on display here from data warehousing to SQL Azure.
Kimberly Tripphttp://www.sqlskills.com/blogs/kimberly/Kimberly is inimitable, both on stage and in her blog. I wouldn't know where to start recommending her work - and if I started I could hardly stop. Let me take one example. Want to know about indexing? Read this blog - for the examples, the technical detail, the good humour, and the sheer practicality of the advice. And that's only one topic. Read the blog, every post.
Chris Webbhttp://cwebbbi.spaces.live.com/Chris is an OLAP guy, and if you know OLAP (whether in the form of SQL Server Analysis Services or any other vendor) you really should subscribe to Chris's blog for its breadth. For those specifically in the SQL Server sphere, Chris's posts on the MDX query language, and more recently on the use of PowerPivot DAX, are not only practical and perceptive, but help to stretch your skills and cover challenging scenarios.
So that's the list. What do you think? Anyone I missed out that you feel really needs to be there? And if so, who would you remove? I'd be fascinated to hear from you.
First, the reason for the release schedule was to properly align Microsoft's flagship database product with Microsoft Office, and with Microsoft's "cloud" strategy. One of the strengths of the SQL Server platform is that it works well with our other products, and in Microsoft Office 2010 and the latest release of SharePoint we have included amazing array of Business Intelligence features for the "non-IT" worker. This means your business users can get at the data they need and want, and the IT department can still control and protect the data the way it should be. It's the best of all worlds.
But it doesn't stop there. As you may have heard, Microsoft is "all in", with a comprehensive cloud strategy. We have not only a complete cloud development platform (Azure) but also a relational database offering (SQL Azure) that goes beyond just hosting a SQL Server Instance in a rack somewhere. SQL Server 2008 R2 allows you to connect to SQL Azure like you're connecting to a local server. You now have capacity on demand, without losing any of your local systems or control.
And there's more - this release also includes the "Datacenter" edition, with support for up to 256 logical processors, data and backup compression (from SQL Server 2008) and the ability to use SQL Server with "Live Migration" - a virtualization technology that lets you move virtualized servers without downtime. These features, along with rapid adoption in the most mission-critical, enterprise-class environments means that you should consider SQL Server as a "Tier 1" application platform.
These are indeed exciting times for the data professional. Make sure you hit these links to learn more - your organization is counting on you as the data professional to know what's new and useful in the data world. You can also post any questions you have on this post - I'll try and make sure someone gets back with you:
SQL Server 2008 R2 Launch Site: http://www.sqlserverlaunch.com/
Official Microsoft Site for SQL Server 2008 R2: http://www.microsoft.com/sqlserver/2008/en/us/R2.aspx
In early 2001 I worked @ a start-up company with about 150 people. I call it a start-up because it had no revenue but did have free breakfast, lunch and dinner. @ the company there were five people named Michelle and six named Dan (including yours truly). One day the VP of operations got confused about which Michelle and Dan he was talking about; he followed it with the following comment – you can’t swing a dead cat around here without hitting a Dan or a Michelle. Ok, I apologize to all cat lovers – I’m one myself and I for sure don’t condone the swinging of dead animals, or live ones for that matter. But I feel the same way about virtualization and consolidation. Hardly a days goes past when I’m not in a conversation about these topics. It’s important to remember that while virtualization has a coolness factor I recommend you approach it as an enabler to meeting business objectives and not the silver bullet that’ll solve all your woes. Also, while virtualization and consolidation are often mentioned in the same breath they’re not the same thing. A couple of guys on my team put together the following graphic to discuss SQL Server consolidation options:
As you move left to right across the picture you move from Higher Isolation (which equates to higher cost) to Higher Density (which equates to lower costs). High isolation refers to resource and security isolation. Obviously you could take this to the extreme and have each application (instance of SQL Server) reside on its own hardware sitting in its own data center residing in its own building on its own power grid. But that’s crazy, right? Higher density means the greatest sharing of resources.
Here’s a brief explanation of each lane:
The main point here is there is no one size fits all solution or technology and virtualization is just one of the ways to meet consolidation needs. You will likely employ multiple solutions within your environment with the key being to chose the right technology to meet the business requirements of the application. As an IT professional it’s your job to understand the technology and how to apply that technology to solve business problems.
There are several different models out there for approaching consolidation. The high-level steps that resonate with me are:
Finally, I couldn’t end this article without mentioning my favorite virtualization related features in Win7 and Win2K8 R2:
Virtualization and consolidation are concepts that have been around for at least 30 years. With the ever increasing pressure on IT to do more with less they are realities that can no longer be ignored. As I mentioned in my opening blog post, one of the challenging aspects of my job is I’m always thinking one to two releases out and I’m talking these days a lot about consolidation and virtualization… The first of this wave of capabilities sees the light of day with SQL Server 2008 R2 Application and Multi-Server Management.
Cheers,Dan
Bar Bosch, in Palma de Mallorca, is one of my favourite places to hang out. It's hectic and noisy, but still leaves space and time for endless discussions or reflection. It is the perfect place to pretentiously pore over Žižek or Derrida. Yet in the autumn of '98 I found myself sitting there, ordering carajillas and reading up as much as I could about Oracle's Business Intelligence tools.
It was before I joined Microsoft, and I had just started a new job with a BI team focused on Oracle products. I had to get up to speed quickly, but, fortunately, I had some vacation already booked. So, one week after I joined, my wife and son headed to the beaches of Cala Ferrera, and I headed to Bar Bosch with my Oracle books under my arm. It was a long week of reading, and mostly in vain. When we returned home, my CEO had news for me. "All that Oracle stuff," he said. "Forget it. We're focussing 100% on SQL Server now." The rest, as far as my personal story is concerned, is history. I've never looked back.
What happened? Some of the C-level team had attended the SQL Server 7 technical preview at Microsoft's invitation in Redmond. On day one, during a break, they phoned back to our office from the lobby and told the development team to stop Oracle development. They found the Microsoft BI story, told then by Bill Baker and Amir Netz, so compelling that they immediately changed direction. Microsoft presented three irresistible messages: a vision of what BI is about, a commitment to BI customers and partners, and a compelling pricing model.
I could wish all Oracle migrations were so easy. Some are, but I always maintain, as in a recent discussion with Ted Cuzillo, that migrating technologies is easier than migrating people. When one has established a way of working, it's difficult to adapt to a new way, even if it is technically equivalent or superior. It's just easier to carry on with one's current practices. Software manufacturers know this, of course. We all want our products to be, as the marketers say, "sticky" - and all of us in the software business like features and methodologies that build loyalty. That loyalty, once gained, is naturally a most valuable asset.
Yet, recently, Oracle, appear to have taken a different approach: one that customers increasingly resent. They are tying companies into the Oracle ecosystem financially, especially by acquiring other strategic suppliers: writing checks, not code, as Larry Ellison himself once said. Their customers are complaining in stark terms, as they find themselves not only tied in, but subject to ever-increasing prices and intransigent demands. Business Week headlined that "Oracle has Customers Over a Barrel" quoting one customer as saying "Once you've made a deal with the devil, it's hard to get away."
This approach from Oracle is surely a deeply alienating move, and it tastes to me of a certain desperation. Personally, I find it baffling. Here is a company with a truly great database product, a broad range of applications and excellent engineers, but they are driving their own customers to despair with licensing practices that feel like extortion according to some. So, while there have always been Oracle customers willing to make the effort to change, at Microsoft we are expecting to see that number increase.
Naturally, we're encouraging them, and why not? SQL Server is growing rapidly: over 11% in 2008 according to IDC. In the past, much of our growth came from net new customers: we have always been aware that we win many businesses choosing their first enterprise database or BI implementation. Now, we are seeing a large number of switchers: customers willing to make that migration from another technology. To win more of these customers, we need to show that they need not be tied into the Oracle universe, even when their existing commitments are substantial. This case study of the Turkish appliance manufacturer Arçelik describes them effectively moving a 5Tb SAP implementation from Oracle to SQL Server, and gaining substantially in performance and lower costs. We also have a neat little tool that TCO calculator that has proved very popular. To be sure, you don't need an animated tool to find significant TCO savings persuasive, but it's fascinating and fun to play with the various combinations of servers and staff and requirements.
Of course, I would not deny that Oracle are still growing. There's a lot to admire in the efficiency with which they target and execute on acquisitions. Frankly, they sometimes take the rest of us in the industry by surprise; which is partly a sign that they are imaginative and bold, yet partly suggests that they are chasing growth by acquisition at all costs, often seemingly at random. This week they acquired Hyperroll. That gives Oracle four OLAP products. It's little surprise that their customers, and from what I hear, their internal teams, are puzzled and often anxious about their direction. On a larger scale, their acquisition of Sun sent many of the same confusing signals.
How does all this affect someone like me, making suggestions and decisions about our own BI futures in Microsoft? In some ways, it affects my work remarkably little. When Oracle buy a company like Hyperion, it may mean that there's a new flag on the island, but it takes a much longer time to build bridges and integration. As a result, we in Microsoft, especially in the product teams (rather than, say, in marketing) may look at the Oracle applications and see relatively few feature-for-feature challenges. In other cases, such as Oracle's own in-database-OLAP features, we largely had to ignore their feature set when considering their future direction because it seems so unclear.
Oracle integration remains an important priority for us, for several reasons. For one thing, it helps our customers who are coming off Oracle to do so in a planned, progressive manner. Supporting Oracle systems in our tools - such as Analysis Services, Reporting Services and Integration Services also helps us to provide BI features to those customers who do find themselves on the wrong side of the devil's bargain and unable to extract themselves for it. For example, we see many customers using Analysis Services, or Reporting Services or Integration Services with Oracle systems. In fact, demand for Oracle support was so great, that in SQL Server 2008 we introduced an Integration Services high-performance data loader for Oracle. Yes, a "data loader" not an extractor: we released a feature that makes it easier to get data into an Oracle database!
Seeing these customer satisfaction issues at another major and successful vendor, and at the risk of sounding too much like a motivational call, I am convinced that we, in the SQL Server team, must keep to our own core value propositions: a persuasive vision, a commitment to our customers' success, and a compelling pricing model. After all, these are the same key points that were so signifcant to my old team over 10 years ago at that technical preview in Redmond. For their part, our friends at Redwood Shores appear to be doing their best to send customers our way. Is that what they call synergy? I'll ponder that, although this autumn it will be over a latte in Third Place Books rather than a carajilla at the Bar Bosch.
In the coming year, you'll see a new release, SQL Server 2008 R2, which is, we hope, full of goodies for your DBAs. As ever, there will some features immediately relevant to your business, some that will enable you to do new things over time, and some that you may not plan to use, but which may yet be of interest. So, in general I recommend using that extra training time to do three things: extend your current skills; expand your range with new skills; and explore and incubate some experimental projects.
Extend your current skills
I have rarely met a DBA with time on their hands, and I know that in your business they now manage more physical servers than ever, and with virtualization and consolidation, more instances and more databases, with more data, than ever. So, if I may suggest one feature that you need to learn thoroughly in SQL Server 2008 R2 it is our multi-server and application management improvements. There is a great whitepaper here, from our team, on this topic: http://bit.ly/6yVmOL You'll find this really is an essential feature to save money and to manage a healthy environment in the coming years.
Expand with new skills
When we first talked about SQL Server together, back in 2005, we remarked then on how the scope of the DBAs role was changing. Not only did they manage databases, but with SQL Server your DBAs were also managing reporting systems, OLAP servers, and the ETL process. I know that your DBAs thought this was, and is, a good thing. They not only "owned the data" but all the surrounding services that integrated, enhanced and gave meaning to the data. In SQL Server this was relatively easy, as the development and management environments for all these services are highly integrated. However, there is another area which I suggest your DBAs should delve into : SharePoint. That is a new administration experience, so there is more to learn. But it will be a worthwhile investment of time. Here's why ...
Not only is SharePoint our fastest growing server product, it is also the heart of our collaboration platform; and, as such, SharePoint is fast becoming critical to Business Intelligence. You know about PowerPivot, of course. (See www.powerpivot.com for more on that, especially you can try the hands-on lab.) I expect that in your organization, your adoption of PowerPivot will be departmental - I don't think you can hold the marketing guys back! In this case, I can see your DBAs getting very involved, not only provisioning data, but managing the infrastructure. There's are a couple of great blogs out there already exploring PowerPivot for SharePoint: www.powerpivotgeek.com and www.powerpivottwins.com - and if you want to give your DBAs a head-start on SharePoint there are excellent training links here on Arpan Shah's blog: http://bit.ly/5Ez7xT
Explore and incubate
Finally, I always think it is good to experiment. Even if you have no immediate plans to use a technology, learning more about it can often uncover useful cases, and prepares the team well for the day when the CEO, fresh from reading his latest business magazine asks "Shouldn't we be doing this?" This year, he'll be asking about the cloud. I can just about guarantee it. Fortunately, SQL Azure, the first significant relational database technology for the cloud, is easy to experiment with - in fact, the development and admin tools are basically the same as you are used to. See the team site for more information: http://bit.ly/7zdfAJ I'm not suggesting yet that you port any applications to the cloud - but we'll help all we can if you want to, just let me know. However, I am sure you and your team will find plenty of opportunities to host experimental applications and incubations. We'll be pleased to help with that too.
So, in short, those are my recommendations for those extra training hours in 2010. It is going to be a good year for SQL Server, and it's great to have you aboard.
This past week we released SQL Server 2008 R2 to manufacturing. This is a huge accomplishment for the team and our customers are anxious to get their hands on it. I came across one blog post that expressed disappointment that the only thing they could download was the evaluation edition – they couldn’t wait to get their hands on a fully licensed edition, which will be available shortly.
Rather than go into a laundry list of what’s in the release here are links to a few of the RTM stories:
Even though I’m a Manageability Guy and there are some terrific manageability features in R2 the most important feature, in my opinion, is PowerPivot. PowerPivot is going to change everything about business intelligence for IT and information workers. Early in my career as an IT Pro I designed a system that used Excel Pivot Tables that were loaded with massive volumes of sales data. Unfortunately I had to have tens of Pivot Tables spread out across an almost equal number of Excel Workbooks. Since there were so many files and tables I had to build a monthly process for refreshing the data. Plus if one of the users wanted a new view of the data I had to craft it by hand for them. It’s an understatement to say this was a pain. If I had PowerPivot back then it would have greatly simplified my life and better supported the needs of my users. As you read up on PowerPivot you’re going to think it’s too good to be true, take it for a test drive to convince yourself how truly remarkable this technology is and how it’ll transform the way you think about BI.
Nineteenth century London was famous, or perhaps notorious, for the jokey catchphrases heard in its streets. They sprang from who-knows-where and spread around the city, amongst urchins and gentry alike, in a matter of hours. Charles MacKay, in his classic Extraordinary Popular Delusions and the Madness of Crowds, gives numerous examples, including, as you may have guessed: What a shocking bad hat! and (a particular favourite of mine) Has your mother sold her mangle? These were the viral memes of the day; a Victorian equivalent of Rick-Rolling or videos like Hamster on a Piano. Naturally enough, they have all but vanished from use, except for one or two: the phrase to flare up, which quickly became a cliché after the Reform Act riots; and, curiously, the cry Tom and Jerry! which may have originated from a line in a play.
A couple of things strike me about this phenomenon: the enthusiasm and speed with which a catchy phrase spread; and, the difficulty of predicting, or, on backward reflection, of understanding, which phrases would persist and why.
I am thinking about this just now, because I see a similar, if less amusing, phenomenon at work in many of my customers' businesses. There's a banking customer who discovered - the hard way - that a summer intern's programming project had become widely relied on in their foreign exchange department. It came to their attention, because it was hosted on the desktop machine of an administrative assistant. Whenever she stressed her bandwidth sending a fax or, perhaps more likely, viewing a viral video over the net, the application ground to a halt for its many users who soon complained to IT about the performance of their application. You can imagine that IT were mightily confused - they did not know the application even existed - until they tracked down problem. It's a good example of how business solutions can spread virally and persist in your operations when found to be useful. It also shows how difficult it may be for IT to understand where such initiatives might spring from, and which of them are likely to become mission-critical.
This is an important issue for us in SQL Server, as we prepare, with our 2008 R2 release, to give business users ever more analytic power, and with that also, tools for readily sharing their analyses. We call this Self-Service Business Intelligence; but I must qualify that further. We call these techniques and technologies Managed Self-Service Business Intelligence, and the difference is significant.
With a self-service application such as the Project Gemini add-in for Excel, we give business users unprecedented computational power within their familiar tools. With such power, most anyone has the potential to build a compelling, and attractive, BI solution: a solution they will be happy to share, and that others may find answers their business needs unequivocally, as it comes directly from another business user. When those others find the solution useful, they will pass that knowledge along too.
How does these practices help the IT department? Are they not an invitation to yet more problems? Not when the full picture is seen.
First of all, self-service business intelligence unburdens IT from responding to numerous ad-hoc requests for reports and analyses. They can manage their resources more effectively, by giving users the means to help themselves with the Gemini add-in for Excel. Secondofly, managed self-service does not cut IT out of the loop: it involves them deeply, for IT provision the required services for collaboration, with Sharepoint and the Gemini Add-in for Sharepoint. IT will also still provide much of the data for analysis, especially the authoritative master data (with SQL Server Master Data Services) or the traditionally warehoused historical data of the enterprise at any scale (with Project Madison.)
By managing the infrastructure for collaboration, IT have unique oversight of, and insight to, the sharing and spreading of successful solutions. Microsoft will provide the tools for administrators to discover which solutions are flaring up. When IT discover a new application growing to unexpected responsibility beyond its original desktop environment, they can ask, and act on, the equivalent of another viral Victorian catch-phrase: Does your mother know you're out?
Donald Farmertwitter: @donalddotfarmer