Insufficient data from Andrew Fryer

The place where I page to when my brain is full up of stuff about the Microsoft platform

February, 2012

  • Data Centre Resource Planning and System Center 2012

    Most of us like have the proper tools for the job; trying to undo Phillips screws with a penknife, using your gold card to get the ice off the windscreen or using notepad to write F# is no fun. I would argue that the proper tool for running a modern Data Centre is System Center not just to save money as this report from last year’s War on Cost event by Inframon suggests..

    but so that IT keeps pace with what the business demands of it.

    That’s easy to say but how do you justify the expense of implementing it, I don’t mean the license costs rather the time and effort  taken to adapt its potential to the needs of your business.  I would liken this to investing in an ERP (Enterprise Resource Planning):

    • You could mash together a series of tools and utilities that you have in place already or that cam bundled with some of your infrastructure.  However the ability of each part of an ERP system to understand the resources, people and assets and have one mechanism for creating business process automation does offer economies of scale and the ability to adapt process far more quickly than would be the case otherwise
    • ERP systems are about managing all of your resources, while System Center this is just limited to your IT infrastructure, however the similarities are striking
      • You define process and and standards with the business.  Service Manager and the
      • the approval processes for getting resources automatically spawns the task to create those services 
      • The use of resources needs to be accounted for
      • You use reporting and analytics to monitor activity and predict demand
    • As any ERP consultant will tell you, there’s a great career to be had in implementing these systems and I am sure that individuals who embrace this vision as well as companies like Inframon will find plenty of projects to work on. Not only that there is a whole ecosystem around this implementation from, audit & compliance to security and business intelligence.

    I mentioned the War on Cost event earlier and General Nathan Bedford Forrest was once misquoted as saying  winning a war is about getting there “fustest with the mostest”. Applying that to your career in these increasingly cloudy times I would advise you to get there fustest by learning about System Center 2012 on the Microsoft Virtual Academy  or at our rapidly filling IT Camps before your peers and getting their mostest by looking out for the upcoming System Center 2012 certifications

  • Managing a hybrid cloud with System Center 2012

    Hybrid cloud is the term applied to the increasingly common scenario where a business runs some of its services in the public cloud but retains some on premises, although it is also possible to use assets from both to provide a service as well.  For some business this is just a transitional step as more and more services are moved to the public cloud, but for many organisations this might be the long term situation as some services can only be run internally.

    For services/applications running in your own data centre you have a large degree of control over everything that application runs on, and you can tune and tweak that as you see fit. You get slightly less control as you outsource use hosting or run on someone else’s "Infrastructure as a Service" cloud platform.  However with "Platform as a Service" like Microsoft Azure there is less to manage; you loose control of the operating system and only have limited ability to set storage, and compute resources.  This is a good thing as it increase agility and reduces management overheads, but it does still need management.

    This loss of control reminds of the early days of banking, you lost control of where your money actually was in fact you couldn’t point to a fiver in a bank vault and say that it was yours. However you still had control of which types of accounts to put your money in and you would be foolish not to request and make decisions on the bank statements you were sent. In the electronic world of the cloud you don’t have bank statements you have something like the Azure Management Portal..

    image

    which shows you how your services are deployed and allows you to change them as you see fit for example to deploy more web front ends, change the schema of your SQL Azure database..

    sql azure overview

    and so on..

    That’s fine if you run all of your services in the public cloud, however if you also have a data centre you’ll probably want to see an overview of how everything is working and be able to report to management on how all your services are performing no matter where they are.  Not only that you’ll want one set of tools to make whatever changes you need bearing in mind the limitations of what can be changed in a cloud service.  There are several tools in System Center 2012 to directly interact with Azure and whatever you have running in your data centre to provide the monitoring and control you need;

    System Center Operations Manager has an Azure management pack to report on your Azure performance and setup in addition to a raft of other management packs to manage databases, switches, hardware, various operating systems, and numerous applications.  The Azure management pack has to be configured with certificates and then uses Azure storage as an intermediate cache for storing and then transmitting the required telemetry down to the local Operations Manger database.

    Note this can all be done in Operations Manager 2007 R2 (CU3 or later)

    System Center Application Controller is a new tool for System Center 2012 that provides a web portal showing what is running in your private and public clouds.  The private cloud works by pulling in data from System Center Virtual Machine Manager 2012.  This latest release of Virtual Machine Manager now allows private clouds to be created and managed on different hypervisor platforms;  Citrix Microsoft Hyper-V and Vmware.  Application Controller also connects to your Azure services using a management certificate (as Operations Manager does) to secure this ..

    image

    Delegated users can create or enhance their own services from here. 

    However for more control you may wish to create more rigorous but still automated process using your help desk (e.g. System Center Service Manager, Remedy etc.) hooked up to these tools.  For that there is System Center Orchestrator which allows run books to be setup with approval process, to capture additional data such as cost centre department etc. and carry out additional configuration tasks.  

    One thing to note that is this is all IT Professional stuff, System Center 2012 is designed for us, not for bank managers, marketeers or accountants. With my business intelligence background I would also recommend that for large and complex organisations you overlay System Center with a BI solution like SharePoint, coupled with web parts for them to be able to initiate service requests. A lot of the tooling for this is already in System Center 2012 e.g. there are optional analysis services cubes in Virtual machine Manager, and extensive reporting services reports in Operations Manager.

    So plenty to think about and keep us busy in 2012, and if you want to see some of this in action before trying it out yourselves (System 2012 betas can be downloaded here), you can come to one of our IT Camps on System Center.

  • Microsoft System Center and BMC Management Tools Integration and Interaction

    In his second guest post Greg Charman from Kelverion has a look at BMC integration.

    Many customers have made large investments in both Microsoft System Center and BMC Datacenter Management Tools. Historically integrating these two vendors product sets has been challenging and time consuming.With the purchase of Opalis (an IT Process Automation Solution) in December 2009 and the release of Opalis 6.3, but particularly with its replacement System Center 2012 Orchestrator, Microsoft has laid the foundations for a much easier and speedier interaction between the two product suites.

    The challenge with any integration and automation product is keeping the interfaces to vendor products current and when a vendor has many management products, like BMC does, this only becomes more complex. In the newer versions of their Enterprise Management products BMC have updated the product APIs (Application Programming Interfaces) to web services APIs. This now means you can now build your own interaction to a BMC application in Orchestrator if it has a web service API using the Standard Activities within Orchestrator particularly the Invoke Web Services Object. To create a DIY interaction with a web service API in Orchestrator you need to have completed the following tasks in your runbooks:

    1. Understand in the Published API so that you know which methods to use interact with the API and know how the results will be returned to you from the API.

    2. Create a Pre-processing Runbook to generate the Security Key which has to be passed to the Insert Into or Extract From API Interaction Object

    3. Create the Insert Into API Interaction Object which formats data from the published data bus into the right XML format for the API method

    4. Create the Extract From API Interaction Object which submits the correct XML format for the API method so that the data you want is extracted.

    5. Create a Post Processing Runbook to manipulate the output of the API extract call into a format which is easily useable within Orchestrator

    This DIY solution often takes 3+ days to create, per interaction you want to achieve with the target system. This is fine, in principal, if you only want to do one or two simple interactions with the target system but if you are looking at say a Service Desk application as a target you won’t be looking at just one or two interactions, you will want many interactions; create incident, update incident, close incident, create change, update change, close change etc.

    However if building it yourself all sounds too complicated then there is an alternative, Kelverion announced recently the planned availability of 4 new Integration Packs (IPs) for both Microsoft System Center 2012 Orchestrator and Opalis 6.3. Kelverion already have a close partner relationship with Microsoft, including the development of third party Integration Packs, and have partnered with BMC to bring the new IPs to market for the benefit of joint Microsoft and BMC customers.

    The new IPs announced target integration with:

    • BMC Atrium CMDB 7.6 - Federates data from across IT into a single, logical data store.

    • BMC Bladelogic Server Automation 8.1 - Management and enforcement of server configuration changes

    • BMC Remedy Action Request System 7.5 and 7.6 - Platform for managing Service Management business processes

    • BMC Event Manager 7.4 - Proactively correlates and prioritizes events within a real-time business context

    Beta Release Schedule Update

    • BMC Atrium for Opalis 6.3 Released

    • BMC Bladelogic - Released (Opalis and Orchestrator versions)

    • BMC Remedy February 2012 (Opalis and Orchestrator versions)

    • BMC Event Manager February 2012 (Opalis and Orchestrator versions)

    • BMC Atrium for Orchestrator March 2012.

    Summary

    If these integration packs are of interest to you and you want to know more updates will be posted at http://www.kelverion.com/new-integrations/ or you can contact David Wooster for more information.

    Editors note Kelverion have also written quite a few of the new Orchestrator Integration Packs for Microsoft, and you’ll see these being released as we gat closer to the final release of of System Center 2012 itself

  • Exploiting Orchestrator - Tips to avoid the hidden bear traps

    Following on from my last post about getting started with System Center 2012 Orchestrator, I asked a good friend of mine Greg Charman from Kelverion, to share some of his experience.

     

    System Center 2012 Orchestrator and its predecessor Opalis 6.3 are a very new subject to many IT Professionals and using IT Process Automation tools is not just as easy as install them and then learn on the job. Each ITPA tool is fully customizable to support the business process you wish to implement so they largely start as a blank sheet of paper.

    It is very easy to install Orchestrator or Opalis and create simple workflows but very quickly people find themselves hitting problems when they want to do more complex activities like branching, looping, persisting data and monitoring process completion. There are pitfalls to be avoided when implementing the solution and although some information exists, the reasons why you should avoid these and the correct approaches are not so well documented.

    After more than 4 years of using and implementing Opalis and Orchestrator I have learn what to do and not do to make the implementation successful. Some of the key points for success are as follows.

    Things to do before you start:

    1. Configure the Designer Client – in Orchestrator and Opalis you can define the behaviour of the Runbook/Policy Designer Client. There are a number of settings you can adjust but the key ones you want to set are;

    • Turn on Show Link Labels – Then you will be able to see the Labels that have been added to the Links in the runbooks
    • Turn on Automatic check out – The benefit with Auto Checkout is that it is easy in Orchestrator to make modifications to a Runbook and then find you haven’t checked it out which means all your changes are lost. Orchestrator does not prompt you to Checkout if you try to make a change.

    Things to be avoided:

    1. Counters - Never use Counters in Orchestrator as they are Global Counters and can thus be modified by any Runbook at any time. You cannot therefore rely on their value at anyone point unless you are running in a Single Thread. This defeats the fundamental benefit of Orchestrator - parallel execution and multi-threading.

    2. Variables - Variables in Orchestrator are also Global. When you export a Runbook all the variables in Orchestrator are exported not just the ones in use in the Runbook, it is therefore very easy to pollute your installation with variables gained from imported Runbooks. Therefore use them sparingly; they are ideal for Database Server Names, Table Names etc.; which will be used in many runbooks.

    3. Text Files for data storage or processing – using text files for persisting data or data manipulation in a Runbook is a recipe for disaster. Text files can only be accessed by one process at a time, the file is locked whilst interaction takes place, you therefore force Orchestrator to become Single Thread. This defeats the fundamental benefit of Orchestrator - parallel execution and multi-threading. You will also find you will do a huge amount of post processing manipulation when you try and extract the data back out of the text file and the post processing is not a simple or pretty activity to implement in Orchestrator.

    Best Practices:

    1. Implement Orchestrator with a Runtime Database – if you want to persist data, have a run time variable in a Runbook, dynamic input data, complex look up logic, audit History, you are struggling to parse data or have got XML/CSV as an input or have got to manipulate the data, then put it in a Microsoft SQL Server Database and use the Database Activities to process it.

    Why put it in a Microsoft SQL Server Database? Because all the database objects are much more powerful against Microsoft SQL than any other database type.
    You design this database to meet the needs of the IT Processes you are automating in Orchestrator and the needs of the Runbooks you are creating.

    2. Keep Monitor Runbooks very short - Got a Runbook which starts with Monitor some enterprise tool and when new/update event do something?

    Keep the workflow very short for maximum performance. Grab the event and write it to a Microsoft SQL Server Database and use a Database Activity to process it. This way there is less chance of Orchestrator missing a new event because it is still busy processing the last set of events.

    3. If you are not already a database guru then Kelverion also offers a Database Integration Pack for Orchestrator and Opalis, which gives you easy script free interaction with databases from within Orchestrator or Opalis and formats the columns selected from a Database Table into individual items of published data – no more need for the dreaded [Field(…..)] statement.

    Summary

    It is difficult in a blog to emphasise how powerful Orchestrator becomes when implemented with its own run time database. There has been nothing I haven’t been able to achieve using Orchestrator and a runtime database but there has been many instances where things could not be achieved if a database was not in use.

    There is a good Microsoft MVP book by SAMS press called System Center Opalis Integration Server 6.3 Unleashed which provides information and guidance on Opalis implementation. Although written for Opalis all the principals apply to Orchestrator.

    If you don’t fancy learning everything from a book or you would like more hints and tips then Kelverion provide a range of training courses and offer bespoke training offering based around specific customer requirements. Kelverion is an established systems integration and software development organisation specializing in IT Process Automation solutions, founded in April 2010 by previous employees of Opalis.

    For more information on Kelverion Training and our other offerings got to www.kelverion.com/itpa

  • Superglue now available for System Center 2012

    One of my frustrations to date when showing off System Center 2012 is that it doesn’t quite fit together yet; each of the individual products is in beta and works pretty well, however joining them up is not there yet.  This is not because there’s a flaw in the line up or missing functionality, it’s just that the bit that really makes System Center an integrated suite, namely System Center Orchestrator hasn’t had the integration packs available for it to work with the other new versions of the suite.  Until now that is..

    The new 2012 integration packs are available here, and in time will be incorporated into the System Center 2012 unified installer when the suite is released to manufacture (rtm).

    The integration packs then need to be applied to Orchestrator (this can be downloaded individually or as part of the whole suite here). This is process is broadly similar to how you setup management packs in Operations Manager if you are familiar with that, but I thought it would be useful if I walked you through it, and I have put a slide deck together of the steps to get you started including setting up Virtual Machine manger 2012 inside the Runbook designer once the IP is configured..

    One thing I find a bit counter intuitive when designing run books is the way you pick up variables in each step, so I wanted to put that in the deck as well.  The trick is to remember to right click on any filter or text in any of the details properties in an Orchestrator activity to pick up variables that are flowing along as each activity in a Runbook is processed. For example in this screenshot I want to stop a VM and to do that I need the VM ID.  Upstream from this step I have entered a VM name and then used an intermediate step to get the properties of that VM including the ID.

    29 RB configure stop vm activity

    and from there you can select any variable on the Orchestrator data bus to match against or use

    30 RB configure stop vm activity 2

    If I then test the Runbook it will run for real and I can see that it’s calling Virtual machine Manager to stop my VM by opening the Virtual Machine Manager console and looking at the most recent job..

    34 RB tester 3 in scvmm

    I can go on from here and maybe reach out to  Operations Manager and put the server into maintenance mode or hook the Runbook up to Service Manager and hook it up to a service request but you get the idea.

    To conclude this is one of the key ingredients in Microsoft’s Private Cloud and now the IPs are there to make calls to the other parts of the System Center suite.  I’ll be showing this off on our IT Camps as we tour the UK this spring/summer and there will be more in depth content on line at the Microsoft Virtual Academy

  • Application Platform Knowledge Hub

    Albert Einstein once remarked that you don’t need to remember everything, you just needed to remember where to look.  When it comes to learning about Microsoft products this can sometimes be harder than it ought to be and what resources there are can be fragmented so you do have to remember where to look.  That’s why I have been trying to get you to remember look at the Microsoft Virtual Academy as that is a great learning resource for all things cloud including, Azure, Hyper-V and more recently System Center.  

    However that resource is never going to have anything on the other technical love of my life, SQL Server, and getting up to speed on that is hard if only because it has been around for longer and you will want to learn about the version you have and possibly the version you would like to have. Until now that is because the some of my colleagues in the UK have  decided to do something about that and created the Application Platform Knowledge Hub.  A quick glance at the opening screen gives you the idea:

    image

    As you can see each topic (in my cases High Availability) has content for SQL Server 2005 through to SQL Server 2012, and is skill ranked as well so you can get as deep into a topic as you need to an also find that whitepaper to waggle under your managers nose that he’ll understand.

    Like the Microsoft Virtual Academy there is also a section on Azure and because this is hub is about development there also some good resources on there about Application Lifecycle Management (ALM).

    So now you’ll only have remember where this portal is and all I’ll have to do is to remember I wrote this post!

  • What’s good in SQL Server 2012 Integration Services

    This is a guest post by Allan Mitchell MVP, and one of the top speakers at SQL Bits

     

    The launch of SQL Server 2012 is not too far away and the SQL Server team have been hard at work adding new functionality and also improving upon existing features.  I am lucky enough to have been working with SQL Server since version 6.5 and have seen it gradually become the great product that it is today and will be in the future.
    My passion has always been around data quality and the movement of data.  Microsoft in SQL Server 7 introduced a new tool called Data Transformation Services (DTS) for which I still have a special place in my heart.  In SQL Server 2005 they completely rewrote this tool and gave it a new name "SQL Server Integration Services (SSIS).  SSIS was a paradigm shift from DTS and took some getting used to.  SSIS in SQL Server 2012 is certainly a release that improves on the existing release and this article will put forwards my two favourite things in SQL Server 2012 SSIS.  Your choices may vary but this is my article and I get to choose my two things ;)

    The SSIS Server and Catalog.

    At first I did not like this at all.  When I spoke to people like Matt Masson on the development team I was never very positive about it.  Looking back though I think I was being stubborn and not wanting to accept change.  I am quite happy to admit I was wrong.  It is my opinion that if you want to use SSIS in SQL Server 2012 properly and get the most from it then you are going to want to look at the new Project Deployment mode and the SSIS Server.


    Using the SSIS Server and catalog gives you a huge amount of functionality including but not limited to;

    • Data Taps: The ability at runtime to "tap" into a part of the data flow and extract the data flowing through.  Incredibly useful for debugging.

     

    • Environments:  Think configurations but better.  You create an environment in which your package will run.  You specify values for parameters and properties in your package and map in them from the environment and off you go. 

     

    • Reporting:  Reporting in a version of SSIS prior to SQL Server 2012 was pretty basic, frustrating and even bordering on pointless.  Not any more.  The team have obviously thought about what we want (and they have been told through Connect).  Now you can get proper SSRS reports showing you phases in your package along with durations and trending of performance over executions. 

    WARNING.  If you are an SSRS pro then the reports are functionally very good, aesthetically not so much.  You may also want to look at Jamie Thomson's reporting pack to compliment what comes out of the box

    The User Interface.

    For a long time now I have been telling anybody who will listen that the weakest part of SSIS is the Business Intelligence Development Studio environment.  Once SSIS is into runtime it is pretty stable and does exactly what it was asked (not necessarily what you wanted, but what you asked).
    The SSIS toolbox for example is currently mashed in with the Visual Studio toolbox meaning a whole load of things have to be loaded that have no relevance to the package you are creating
    Precedence constraints just disappear from the UI.  They are there underneath the covers you just can't see them.  To this day I have no idea why this happens occasionally but what I do know is that it is almost impossible to develop a package when this happens and you have to restart the package for the designer to redraw the lines.
    The UI for developing SSIS packages in SQL Server 2012 has been rewritten in the Windows Presentation Framework.  The edges on the tasks and data flow components have been rounded off resulting in a 10% improvement in speed due to less resistance (only kidding about the speed improvement).  The UI has the look and feel of a proper Visual Studio UI.  The SSIS toolbox lives by itself.  No more loading of not-needed components.  I am very happy with the new UI, very happy indeed.

    Summary

    This article has covered two of my favourite things coming in SSIS for SQL Server 2012.  By no means are these the only changes.  Go take a look at parameters for instance.  The SSIS team have been hard at work for this release.  There are some things I don't like but there probably always will.  That's why we have Microsoft Connect so we can tell the team about these "opportunities" for improvement.


    Go on, give it try and download SQL Server 2012 RC0

  • Event: IT Camp Leeds Feb 28th

    Back in December Simon and I invited a few friends along to help us beta test a different kind of event, an IT Camp.  The idea is simply to do smaller, more interactive events where discussions and questions are encouraged.  The presenters, (Simon and I) being a bit more in the background.  The first of these events went down very well, but in this spirit of interaction we were keen to understand what we could do to make the camp even better.   Having made some changes based on the feedback from that test event we are now ready to go on the road and our events team are looking out for regional venues so we can come and see you rather than you travelling down to London or Reading.

    There are actually four types of camps designed to complement the online resources we have on the Microsoft Virtual Academy site:

    Camp 1. Server Virtualisation. This is for those unfamiliar with the latest version of Hyper-V in Windows Server 2008 R2.  We will explore all the basics including networking and clustering and run through a short introduction to using System Center Virtual Machine Manager 2012

    Camp 2. An Introduction to Microsoft’s Private Cloud.  This builds on the first camp and explores all the  key features in System Center 2012 that enable cloud like processes and capabilities in your own data centre.

    Camp 3. Server Migration.  Specifically the tricks and tools from migrating your various server workloads from Windows Server 2003 to 2008 R2,

    Camp 4.  Consumerisation of IT.  This explores the issues and benefits of integrating your users’ own devices with your infrastructure, such as slates, smart phones and laptops.

    You’ll see more coming out about this in the TechNet newsletter and the various dates and venues are on the IT Camps page on the UK TechDays site, shortly, but I wanted to specifically mention that the first of these camps will take place in Leeds on 28th Feb.  This will be a  Camp 1 server virtualisation event. Our plan on the day is to work together to build a Hyper-V cluster using your machines as well as our own kit. You don’t have to use your own machine if you don’t want to but if you have a laptop with 20Gb of free space on and 4Gb of RAM and it can run Hyper-V then we’ll take you thought the steps to set up Hyper-V without affecting what you have on there already.

    If you follow the links you’ll see that this event is by invitation and this is because we want to try and ensure the right people attend as it is limited to 70 delegates.  So who are the right people?  For this event it will be Techie IT professional types, who are only now becoming interested in what Microsoft is doing with server virtualisation, perhaps because they are using something else or their business for whatever reason is only now considering a move to use this technology, i.e. it’s not really for Hyper-V experts.  If this sounds like you or a colleague then drop me an e-mail and I’ll send you the code, and you’re in.