By Michael Sullivan, SQL Product Manager, Microsoft UK
We all know some pretty bad SQL jokes (the language that is). Well I do anyway. Like, a DBA walks up to two tables in a bar and says 'may I join you'? Enough. But imagine that same DBA walking into a restaurant and finding no tables or chairs! Only cloud tags, long arrays of text strings, angry looking web site logs, a zillion tweets munged together with neighbouring RFID tag streams, and a bunch of unemployed maps with geospatial attitude! His* mission? To chat with all of them, and get them to yield business insights which us mere mortals can consume? Maybe that takes a rocket scientist. Or does it?
Well a Microsoft DBA can approach it as follows.
First, he* thinks parallel. That means he is enlisting his company's new Microsoft SQL Server 2012 Parallel Data Warehouse (PDW) to solve the challenge. No need to try and invite all the data back to your place - even though you have incredibly efficient seating (in-memory column stores, for example) at your disposal. Nope. Leave those weird and wonderful data types where they are, on their comfortable HDFS sofas, and just get their phone numbers for now.
OK, it's time to ask this universe of guests about their favourite 'data nibbles': in parallel. A Microsoft SQL Server DBA you can do that easily, avoiding getting bogged down in dietary requirements, long queues or complicated seating plans. How? With a new (not so) secret weapon called POLYBASE! Think of Polybase as an amazing 'data butler' that speaks your language (SQL) and your guests' language too (in this case MapReduce). In no time you have queried all this structured and un-structured data (your complete guest population). In tech-speak: you issued a standard T-SQL query that joins tables containing a relational source with tables in a Hadoop cluster without needing to learn MapReduce queries. And you got compliments back on your near-fluent Hadoopsch accent too.
It's the next day (the day after the party). The Heads of Business Intelligence and Enterprise Applications in your company are very happy. And the CFO too. Using nothing more complicated than Microsoft Excel, she* has everything she needs on her laptop, at the board meeting, to show amazing insights into her customers, invoices, SKUs, cash on hand, days outstanding, the shelf-life of everything that left her factory last week, not to mention deep insight into customer sentiment about a product recall triggered by a quality audit last month. Wow! And all because you are a SQL Server DBA.
Book your ticket to a parallel universe of insights at the upcoming SQL Bits event, or by drop us a note: sqluk@microsoft.com to learn more.
* my fictional DBA is a 'he' and my CFO is a 'she'.
Over the past couple weeks we’ve been talking Systems Center and this week we are focusing in on Virtual Machine Manager. There have been a few articles that I thought would be worth sharing more broadly on this topic.
Upgrading our TFS 2012 Lab Management to use SC-VMM 2012 SP1
Richard Fennell runs through his experience of upgrading Black Marble’s TDL Lab Management System. This article provides some great learning experiences from doing that and is a great real life example of upgrading to the latest version of System Center.
Things to remember when building virtual machines for a lab manager environment
Rick Hepworth takes us on his journey of building lab manager environments. He shows the issues and solutions to the issues he faced when upgrading his environment. If you have come across an issue then he’s probably go the solution right here for you, he even has a checklist to guide you through things.
Useful Links
If you have an article with top tips or how to’s on System Center Virtual Machine Manager remember to leave a link with the details of what is covered in the comments area for this blog post!
Unilever has been hugely successful in selling personal care, homecare, and food products to billions of
customers throughout the world—so successful that it expects to double in size in 10 years. To ensure that its IT organization could support this growth, Unilever worked with Avanade to migrate from VMware to Hyper-V technology in Windows Server 2008 R2. It then used Hyper-V and Microsoft System Center 2012 to create a private cloud environment that contains 3,175 virtual machines. With its private cloud, Unilever will deliver IT services 40 percent faster and be more agile in the marketplace. It also expects to achieve its growth goals with no increase in IT costs. By eliminating hundreds of servers, Unilever realized significant savings and became a better environmental citizen. It is upgrading to Windows Server 2012 to gain even more IT efficiency.
Situation On any given day, 2 billion people use Unilever products to look good, feel good, and get more out of life. Unilever is one of the world’s largest consumer goods companies, with more than 400 brands that include foods, soaps, shampoos, and household care products. Some of its well-known international brands include Lipton, Knorr, Dove, Lifebuoy, Hellmann’s, and Omo, alongside trusted local names such as Blue Band, Pureit, and Suave. Unilever was established in the 1890s, and today has 171,000 employees selling products in more than 190 countries.
Unilever has enjoyed significant growth in recent years, with more than half of its revenues coming from emerging markets. Management intends to capitalize on the company’s success in these markets and set an ambitious goal to double the company’s size in a decade.
Every business unit in the company has responded to the growth goal by fostering a culture of continuous improvement, driving for small changes every day to increase speed, improve quality, and reduce costs. The IT organization wanted to devise a way to support the company’s growth goal without doubling the company’s IT “footprint”—the cost of servers, data center power and cooling costs, and IT staff.
Actually, long before the company announced this growth goal, Mike Royle, Director of Enterprise Computing at Unilever, had made good progress toward trimming server counts and costs. In 2008, Unilever had more than 5,000 servers in its global data centers and in hundreds of remote locations. Royle and his team began virtualizing servers by using VMware ESX software and reduced its physical servers by 65 percent. However, VMware was too expensive to proceed with.
Meanwhile, Royle and Roland Meier, Strategy and Technology Director for Unilever, closely tracked the progress of Microsoft virtualization software, and in late 2009 engaged Avanade to create a proof of concept around migrating to the Hyper-V technology in the Windows Server 2008 R2 operating system. The business case for moving to Hyper-V focused on increasing data center consolidation, extending virtualization to remote office locations, and reducing management costs. “By migrating our VMware estate to Hyper-V, we estimated a six-figure operational cost savings from lower licensing and support costs, power and cooling savings from fewer physical servers, and IT management savings,” Meier says.
The proof of concept was a success, and between 2009 and 2011, Avanade used its Next Generation Datacenter model and Migration Factory methodology to migrate most Unilever VMware virtual machines to Hyper-V and converted hundreds of physical servers to Hyper-V virtual machines. In total, Avanade helped Unilever create 2,000 Hyper-V virtual machines.
“Avanade’s people are first rate; they tell it how it is,” says Royle. “They listened and responded to our requirements with a strong balance of ‘skin in the game’ and empathy.”
Even after reducing its physical server count by two-fifths, Unilever wanted to do more. IT costs were still higher than the company wanted and would only increase as the company grew. Additionally, while the IT staff had greatly reduced the time it took to deploy servers, there were still too many manual steps involved in reacting to business needs.
In developing markets, in particular, where the company’s growth was focused, the IT staff wanted to meet business demands for IT services more efficiently. “Doing business in emerging markets requires customized approaches, more solution diversity, and often shorter lead times,” Meier says. “Virtualization helped to increase IT efficiency and reduce costs, which are critical to sustainability. But we needed to accelerate overall service provisioning times, which required new system management tools and a revamping of our operational processes.”
To learn more about the products they used have a trial of:
Find out how they managed to meet their requirements by viewing the case study here
After my latest round of camps across the UK I noticed that a lot of you are new to System Center 2012 and given the great response to Hyper-V I thought a gentle introduction to the part of System Center that manages virtual machines (VMs), Virtual Machine Manager, (VMM) would be worth a read.
Virtual Machine Manager (VMM) does what it says on the tin, it manages virtual machines and Hyper-V is the hypervisor that provides the server virtualisation capabilities in Windows Server. As a bonus VMM 2012 SP1 has had significant changes which make the most of the new features in Windows Server 2012 Hyper-V, and has also been updated to manage Vmware VSphere 5.1 so if you are using both this can manage VMs across both of those and the rarer Citrix XenServer.
Of course tools like Server Manager in Windows Server 2012 allow you to easily manage multiple servers in one go to a basic level, and if you have less than 70 VMs running only on Hyper-V you could probably stop reading this now and use that and PowerShell. However if your working in a larger environment with multiple hypervisors please read on!
The easiest way to manage large numbers of things is to group and catalogue them and in VMM VMs are grouped in two distinct but complimentary ways:
Services: VMs belong to services that you deliver to the business e.g. applications. Generally applications have tiers of components for example web front ends, middle tiers and databases and to keep the service running you need to understand these dependencies especially if you are making the application highly available. Some services can be scaled up and down e.g. those web front ends. You may want to deploy copies of a service rather than just spinning up new VMs and gluing them together and this can be done from service templates which understand Web Deploy, SQL Server databases and can even virtualise a Windows Service in the same way that applications can be virtualised on a desktop using App-V here’s a service I prepared earlier.. note the scalability setting at the bottom where you can set how big and small each tier of the service can be. Clouds: The business are paying us to provide services for them and they own the data centre while we are the curators who look after it on their behalf. The Projects , Divisions business units, call them what you will, then own parts of it but rather than put stickers marked finance on a rack of servers VMM allows your data centre to be logically split up into clouds that can be owned by one of these business units/divisions etc. So clouds comprise compute storage and networking but scattered across hypervisors, servers, SANs etc. You would then delegate control of that cloud downwards to an IT guy how actually works for that business unit and you would offer them a list of service and virtual machine templates that they could use within that cloud, however they can’t go beyond the limits you set on the compute storage etc. it’s your cloud and you can specify how big it is
Services: VMs belong to services that you deliver to the business e.g. applications. Generally applications have tiers of components for example web front ends, middle tiers and databases and to keep the service running you need to understand these dependencies especially if you are making the application highly available. Some services can be scaled up and down e.g. those web front ends. You may want to deploy copies of a service rather than just spinning up new VMs and gluing them together and this can be done from service templates which understand Web Deploy, SQL Server databases and can even virtualise a Windows Service in the same way that applications can be virtualised on a desktop using App-V
here’s a service I prepared earlier..
note the scalability setting at the bottom where you can set how big and small each tier of the service can be.
Clouds:
The business are paying us to provide services for them and they own the data centre while we are the curators who look after it on their behalf. The Projects , Divisions business units, call them what you will, then own parts of it but rather than put stickers marked finance on a rack of servers VMM allows your data centre to be logically split up into clouds that can be owned by one of these business units/divisions etc. So clouds comprise compute storage and networking but scattered across hypervisors, servers, SANs etc. You would then delegate control of that cloud downwards to an IT guy how actually works for that business unit and you would offer them a list of service and virtual machine templates that they could use within that cloud, however they can’t go beyond the limits you set on the compute storage etc.
it’s your cloud and you can specify how big it is
Another key part of VMM is it’s library. This is simply a share where you can store and create all the objects you need to create VM. This might be nothing more than a standard blank hard disk and an iso with an OS on it. More advanced users will create VM and Service templates as I mentioned earlier, but one other thing you can do is to create profiles which are essentially templates which can be used in templates. For example a hardware profile gives you a place to store all the CPU, storage and network information that you use across all your templates and an OS profile details how you configure the OS. You can then mix and match the OS and hardware profiles to quickly create a set of templates you need without filling in all of the details every time you create a new template. By the a way a top tip is to turn on deduplication in Windows Server 2012 for the Library volume you create it should save you about70% of the space it would otherwise occupy as you can see below..
VMM also allows the VM administrator to maintain the hosts by allowing them to have a desired configuration and to patch them by automatically sweeping VMs off and on them during the process and getting appropriate updates from your update server (WSUS). In a similar fashion you can manage your servers to save power by shunting VMs on to a smaller set of hosts allowing the rest to be shut down when not in use
VMM also a deep understanding of your fabric for example :
Storage. VMM understands Storage Management Initiative Specification (SMI-S) and classified based on performance which can then be apportioned to clouds. SAN copies can be initiated and offloaded from VMM via this as well. Note most of the major storage players (Dell, EMC, NetApp, Vmware, etc.) have signed up to SMI-S. Networking. If you are using Hyper-V you can see the virtual networking and extensible virtual switch exposed to allow virtual machines to belong to a network that moves around with them independently of the setting of the hosts, here I have shown the networking of a few of my hosts and VMs
Storage. VMM understands Storage Management Initiative Specification (SMI-S) and classified based on performance which can then be apportioned to clouds. SAN copies can be initiated and offloaded from VMM via this as well. Note most of the major storage players (Dell, EMC, NetApp, Vmware, etc.) have signed up to SMI-S.
Networking. If you are using Hyper-V you can see the virtual networking and extensible virtual switch exposed to allow virtual machines to belong to a network that moves around with them independently of the setting of the hosts, here I have shown the networking of a few of my hosts and VMs
VMM is part of the System Center suite and really only works when it is crossed wired into the rest of the suite and you might start with:
Orchestrator allows you to automate processes into VMM by using simple scripts. the process (runbook) can be initiated from other bits of System Center or from any of the leading management tools from third parties BMC, etc. Then you could create a self service system based on service requests to spin up services and VM as required, although this would be easier if you used System Center Service Manager as this integrates into both VMM and Orchestrator and there is a Cloud Service Process Pack to make this easier still. Orchestrator has extensive tools to automate process in VMM Operations Manager can not only peer into the health of VMs but the applications running on those VMs. If the application is not running as planned then an alert initiated here could then fire an orchestrator runbook to scale up a service on VMM. Operations Manager can also check the health of VMM and the hosts it is managing.
Orchestrator allows you to automate processes into VMM by using simple scripts. the process (runbook) can be initiated from other bits of System Center or from any of the leading management tools from third parties BMC, etc. Then you could create a self service system based on service requests to spin up services and VM as required, although this would be easier if you used System Center Service Manager as this integrates into both VMM and Orchestrator and there is a Cloud Service Process Pack to make this easier still.
Orchestrator has extensive tools to automate process in VMM
Operations Manager can not only peer into the health of VMs but the applications running on those VMs. If the application is not running as planned then an alert initiated here could then fire an orchestrator runbook to scale up a service on VMM. Operations Manager can also check the health of VMM and the hosts it is managing.
So that’s a quick lap around VMM 2012 sp1. However to really understand you are need to do three things
1. Check Out the Microsoft Virtual Academy content on VMM
2. Try it out in your sandbox. You can get System Center 2012 sp1 evaluation edition here
3. Think about getting the MCSE Private Cloud certification. This is hard so you will need to study hard, but then the qualification is in demand so you will stand out form the crowd
The TechNet team are running a competition from 22nd March to the 31st May this year to win a fully funded ticket to TechEd Europe.
Trial one or all of the following products:
And write us a product review to be in with a chance of winning our top prize.
TechEd Europe is the premier annual Microsoft event for IT professionals from the 25th to 28th June 2013. Winners will be able to improve their knowledge, talk to leading experts face to face and gain valuable information that’ll put them streets ahead of others in your area. Plus on top of all this they will do it in sunny Madrid!
To find out more and enter, click here
Sideloading is the process of installing a Line of Business (LoB) app onto a Windows RT or Windows 8 device without the need to publish the app to the Windows Store. System Center Configuration Manager 2012 SP1 is able to publish apps for sideloading on Windows 8 Enterprise devices natively and can use Windows Intune to publish apps for sideloading to Windows RT devices. In the following video I show you how to sideload in just this way.
This video is broken down into just two parts:
If you want to give this a try, download the Windows Server 2012 Evaluation and System Center Evaluation, then give part 003 a view to help you connect Configuration Manager to Windows Intune. If you find the video helpful please “like” it on YouTube.
The Azure Customer Advisory Team and Product Group will deliver an intensive all day seminar at SQLBITS on Thursday May 2nd. This is a deep dive technical event for experienced architects and developers who want to understand and learn from the people who build the platform and the largest systems that run on it.
Places are filling fast, you can find out more information and book your place here!
Despite common misconceptions Microsoft now has extensive interoperability with open source technologies for example you can run a php application on Azure, get support from us to run RedHat, SUSE or CentOs on Hyper-V and manage your applications from System Center. , So extending this approach to the world of big data with Hadoop is a logical step given the pervasiveness of Hadoop in this space.
Hopefully your reading this because you have some idea of what big data is. If not it is basically an order of magnitude bigger than you can store, it changes very quickly and is typically made up of different kinds of data that you can’t handle with the technologies you already have. For example web logs, tweets, photos, and sounds. Traditionally we have discarded this information as having little or no value compared with the investment needed to process it, especially as it often not clear what value is contained in this information. For this reason big data has been filed in the too difficult drawer, unless you are megacorp or a government.
However after some research by Google, an approach to attacking this problem called map reduce was born. Map is where the structure for the data is declared for example pulling out the actual tweet from a twitter massage, the hashtags and other useful fields such as whether this is a retweet. The Reduce phase then pulls out meaning from these structures such as digrams ( the key 2 word phrases) sentiment, and so on.
Hadoop uses map reduce but the key to its power is that it applies the map reduce concept on large clusters of servers by getting each node to run the functions locally, thus taking the code to the data to minimise IO and network traffic using its own file system – HDFS. There are lots of tools in the Hadoop armoury built on top of this, notably Hive which presents HDFS as a data warehouse that you can run SQL against and the PIG (latin) language where you load data and work with your functions.
Here a Map function defines what a word is in a string of character and the reduce function then counts the words. Obviously this a bit sledgehammer/nut, but hopefully you get the idea. Also the clever bit is that each node has part of the data and the algorithm to process and then reports back when it’s done with the answers to a controlling node a bit like High Performance Computing and the SQL Server Parallel Data warehouse.
So where does Microsoft fit into this?
The answer is HDInsight which is now in public beta. This is a toolkit developed in conjunction with Horton Works to add integration to Hadoop to make it more enterprise friendly:
Big Data is definitely happening, for example there was even a special meeting at the last G8 meeting on this as it is such a significant technology. However it cannot be solved in one formulaic way by one technology; rather it’s an approach and in the case of Microsoft a set of rich tools to consolidate, store, analyse and consume: The point being to integrate Big Data into your business intelligence project using familiar tools, the only rocket science being the map reduce bit, and that is the specialism of a data scientist. Some of their work is published by academics so you might find the algorithm you need is already out there - for example the map function to interpret a tweet and pull out the bits you need is on twitter.
However research is going all the time to crack such problems as earthquake prediction, emotion recognition from photographs, ,edical research and so on. If you are interested in that sort of thing world then you might want to go along to the Big Data Hackathon 13/14th April in Haymarket, London, and see what other like minded individuals can do with this stuff.
In this video I explain the process of making a Windows Store app available to Windows Intune clients using System Center Configuration Manager 2012 SP1 to publish the apps through the Windows Intune company portal. This process is known as deeplinking and allows an enterprise to publish apps from the Windows Store to Windows RT clients. The video is broken into segments:
This is the fourth video in this series, the other videos can be found on The Deployment Sessions mini-site.
If you want to give this a try you’ll need to get the Windows Server 2012 Evaluation and the System Center 2012 SP1 Evaluation. Also take a look at part 003 to connect Config Manager to Windows Intune.
Business intelligence sounds like something every company would want, but what does it mean in practice? We spoke to Paul Marten, Senior Finance Controller for Microsoft’s UK Business and Support unit to find out how he is using Microsoft’s own BI software to transform the role of finance.
Over the last eight years, finance’s role has changed from ‘process and control’ to ‘business partner’ . My team and I have been empowered by business intelligence technology to think more broadly, present information more effectively and provide more insight to our business partners.
As a finance team, one of our biggest challenges is turning all our data into valuable insights for the managers who run the different business units that make up Microsoft UK. Business intelligence allows us to standardise our reports and speed up their production. The technology works like an efficient filter:
As a controller, it really helps to have everything in one place. SQL Server is doing all the heavy lifting so I can concentrate on analysing the figures and not get distracted by the chore of creating reports.
General managers have access to a dashboard in SharePoint or even through a regular daily mail that lands in their inboxes that shows them a daily revenue report, scorecards for individual business units and key performance indicators. They can use the standardised reports to compare their performance with other business units and spot cross-selling opportunities.
We can also provide our business partners with real-time insights on budgets and expenses at a high level with the ability to drill down into details. In particular, managers were having trouble keeping track of travel and entertainment budgets, but now they can quickly assess individual requests in the context of previous spending and behaviour.
PowerView is a particularly useful tool: it allows you to interact with the data without having to go back to the raw data or an Excel spreadsheet. Managers are able to understand more about certain aspects of reports simply by clicking, giving them the ability to do their own analysis and reporting.
BI lets us close our books every month in just two and a half days. This gives the business timely, reliable information in a consistent format. We used to have 15 people running 15 different reports but no more.
Having such powerful and interactive reporting technology also allowed us to do a paperless mid-year review this year for the first time. We presented using Microsoft OneNote, where 100 people could share a document as the presenter made live updates and added comments. It worked really, really well.
Personally, I can log into SharePoint and see share price performance, revenue for the UK and comparable information about my peers in other parts of the world. It's my one pulse-check for what's going on. I can then spend more time talking about the business, which helps me stay ahead of the wave and stay conscious of what decisions are on the horizon.
As a result of using Microsoft’s own Business Intelligence software, finance has become a more valuable part of the organisation that can make a real contribution to important business decisions. .