...building hybrid clouds that can support any device from anywhere
If you ever doubted how much heterogeneity you can manage using System Center, this post should help change your mind!
I’m Bruno Saille, Program Manager within the Customer Architecture & Technologies (CAT) team here at Microsoft and this is my first post in the Building Clouds team blog.
This blog post will cover a specific scenario spanning technologies and protocols like VMware, Linux, SFTP, SSH, all automated through System Center 2012 Orchestrator and integrated with System Center 2012 Virtual Machine Manager. It will also highlight a few tips/tricks implemented in the Orchestrator runbooks, and that you may find useful in other situations as well.
Let’s say you are running a virtualized environment on VMware today. Regardless if you are also looking at Hyper-V at this stage (and you should, with the latest and upcoming enhancements!), you may be interested in augmenting VMware capabilities through System Center management. We’re talking about features like cloud application-centric management, deep application insight, rich automation, creation and delegation, service templates, etc. To that extent, a great feature from System Center 2012 is the ability for the Virtual Machine Manager (VMM) component to manage your ESX hosts and deploy/customize Windows virtual machines and services. However, while VMM can manage Linux VMs running on ESX, its Linux "OS specialization" capabilities (changing host name, etc,) are only available today for Hyper-V hosts. In this post, we will explain how you can use System Center to deploy Linux virtual machines on VMware, customize the VMs and add applications.
This sample solution consists of runbooks that achieve the following set of actions:
There is a main process runbook handling the end to end execution, and calling subroutines for #1 and #2.
Main process runbook
Input parameters for main process runbook
Result in a Red Hat virtual machine
Resulting VM properties in VMM (when using IP addressing using VMM pools)
When it comes to surfacing these runbooks in self-service portals, there are multiple options. One of them being System Center 2012 Service Manager. That way, you can provide a single pane of glass to provision Windows and Linux VMs. The Windows VM provisioning process would be calling the Cloud Services Process Pack (CSPP) runbooks. All these VMs will be managed in System Center 2012 Virtual Machine Manager and in System Center 2012 App Controller.
You could also “just” use these to learn how to work with protocols like SCP and SSH with Orchestrator. This is why the solution is called the Linux Orchestration and Customization Kit (LOCK)
The next section explains in more detail how the sample solution works. How to import/configure the runbooks is explained further down in this post.
This runbook takes the following parameters:
It calls the “Deploy and Customize OS” subroutine with these parameters
Note: The network name for the VM is in a variable in this example – in production this would probably be an input parameter when creating the VM
This first subroutine returns the IP address of the VM that was created:
The main process runbook then calls the “Deploy Apps” subroutine with the right parameters based on the application chosen (this example is with the System Center 2012 Operations Manager RPM package – see below for more details on application installations)
OS deployment and customization subroutine
This runbook creates the VM through PowerCLI. The OS Customization template is created on the fly in all three modes (Static IP, VMM pool).
In all three branches, the runbook then waits for the VM to be up before exiting. This serves two purposes: 1/ Make sure the VM is fully up and running and ready to receive file copies, and SSH commands 2/ Retrieve the IP address through PowerCLI. This is especially useful in the DHCP scenario, where we do not know which IP was assigned…!
<Tip #1> The “Wait for VM” activity has a few interesting features.
The script within is just querying PowerCLI in a loop, to try to retrieve the IP address:
But the exit condition uses a pattern to ensure that not only an IP was returned, but an IPv4 address:
This was added because there seems to be a small time window where a Linux VM could receive the IPv6 first. Adding that pattern in the loop ensure that we wait for an actual IPv4 address before proceeding to applications install. If you use IPv6, you might want to tune this differently.
Notice the last branch at the bottom: The runbook enables you to use VMM pools as a single pane of glass for IP addressing (Remember, this is done out of the box with Windows VMs on ESX, and Windows/Linux VMs on Hyper-V). The trick is that IP addresses cannot be “reserved” in the pool, they have to be assigned to an existing object. To work around this, we’re assigning a new IP to a “dummy VM” to mimic IP reservation. And then, after VM creation in PowerCLI, the runbook just waits for VMM to discover the new VM from vCenter, and transfers the IP address assignment from the dummy VM to the actual new VM.
Note : To work with VMM, the scripts use PowerShell remoting. You could update this subroutine to use the “Run VMM script” activity from Powershell if you prefer. Also, if you adapt the scripts to your environment and end up having to work with multiple output from PowerShell, make sure you look at Charles Joy’s post here.
As part of this “OS Deployment and Customization” subroutine, you could also run LVM scripts if needed, since it is just a matter of running SSH scripts. For example, adding a drive in a VM would consist of adding a virtual disk through PowerCLI, and then scanning the disk and configuring them via SSH.
At this stage when entering the second subroutine, the VM is up and running, and we want to deploy an application. The sample runbooks described in this post deploy only one application on each VM, using a keyword. One keyword leads to an installation success, and another usually leads to a failure based on missing package dependencies (the goal being to showcase how the runbooks surface both situations)
Note : It would be easy to change the application installation process to rely on a list of applications or, even better, query a CMDB like System Center 2012 Service Manager to retrieve list of applications and their installation properties (source location, installation command line, etc.). You could also Leverage System Center 2012 Configuration Manager as the technology to push applications after the Linux machine has been deployed and customized. Configuration Manager supports several UNIX and Linux flavors.
Application deployment consists of copying the files to the remote box, executing the installation command line, and validating installation was successful.
Applications copy and installation subroutine
The file copy uses WinSCP and the SFTP protocol, and happens in the “Upload Files” activities in the previous screenshot.
Note: This file copy phase could also be modified to mount a CIFS share, but WinSCP being a familiar tool on Linux, made sense in this example.
The tricky part is connecting with WinSCP. It requires a “hostkey” specific to the target host, for security purposes. The “Check ability to connect” activity is there to try a WinSCP connection, and the runbook analyzes the standard output to see if the host key is already in the cache or not, or if the host does not reply.
<Tip #2> The “Check ability to connect” activity shows how to automate a command line application and pass the standard output to the databus.
You might wonder why the two options below were not used to achieve this, and here is why:
The rest of this subroutine uses SSH to actually install and validate installation of the chosen package.
<Tip #3> Even without using a .SH file, the “Run SSH Command” standard activity is able to run multiple commands in a single activity. Like many fields in the Orchestrator runbook designer, you can expand the command window and add multiple lines
You should then be ready to test the runbooks using the System Center 2012 Orchestrator web console, and once you are satisfied with them you could expose them into a portal of your choice, like System Center 2012 Service Manager.
In a classic production situation where a self-service portal is being used to request virtual environments, this solution helps provide:
Note : As an alternative, you could also rely on the capability to deploy a Linux VMDK through VMM (without OS specialization), and have custom Runbooks to change IP, name, etc. This is probably more of an option if there is only one specific Linux distribution, otherwise the approach from this blog post is more generic (it could apply to multiple distributions, as it offloads the OS specialization to vCenter). Both options are possible depending on your requirements.
As always, this is a sample and there are variations you could make to adapt and optimize it for your own environment. In production, this would also require more error handling, and further componentization as the solution expands.
Thank you for reading this post! Hopefully you learned a few tips and tricks along the way, as well as how to achieve a specific scenario that is not fully covered out of the box with System Center.
How would you do this if you don't allow root access through SSH? How would you use sudo to elevate after the connection is made?
The "Run SSH" activity does not need to run under the root account. The "Advanced" tab lets you use any identity to connect with, and you could use any multi-line SSH script you may be using today, pasting it into the activity.
If you need to use sudo, and depending on your configuration and security requirements, you could use an account that does not require interactive password prompt, or you could use an SSH key file instead of a password (also an option in the "Advanced" tab).