MAT (powered by Project Shift)

MAT (powered by Project Shift)

  • Comments 15
  • Likes

Can’t we convert faster?

The most common complaint I hear when working with customers who are converting VMs from a VMware platform is that it’s slow. Now for many customers technologies like MVMC or SCV2V are great. They are easy to use and provide relatively quick results. These customers have some flexibility in their migration windows and they can tolerate a bit of downtime.

Of course, not everyone has that much flexibility. Large enterprises and hosters often have smaller service windows to work within. For many of them just the time it takes to copy the VM from the VMware datastore over to the target is too long and risks their SLAs on downtime. In fact I’ve worked with a few who have had maintenance windows as small 15 minutes. That’s crazy small and the fact is, tools like MVMC just can’t convert most VMs within a timeframe that small. The original MAT allows us to convert multiple machines at the same time but it doesn’t alter the speed of the underlying conversion engine.

The simple fact is that most conversion technologies are fighting physics. The VM needs to move from VMware’s storage onto a platform that can convert it, then it needs to get to the Hyper-V destination – so basically it has to be copied twice. Of course, there are limits to how fast a customer can move bits across their network. That speed limit multiplied by the number of bytes is the best conversion time they can manage. Even warm conversions like Double-Take Move have this liability; the difference is that with a warm conversion tool the copy happens while the VM is running and without downtime. Cold migrations perform their copies when the VM is off.

 BC-DLButton

The opportunity

Not long ago I was watching NetApp show off their conversion technology, dubbed internally at NetApp Project Shift. I even mentioned it in my previous blogs.

What Project Shift does is convert VMs located on a NetApp controller between formats at amazing speeds. It can convert between several formats but I was obviously most interested in the VMDK to VHD. For example, this Project Shift can take a 40GB VMDK file and convert it to a VHDx using their cmdlet “ConvertTo-NaVhd” in about 5 seconds. Crazy right?

The trick here is that they don’t have to move the data, it stays on the controller. More than that, they don’t even need to copy the data on the controller. Their cmdlets use NetApp FlexClone technology to create a virtual copy of the VMDK that consists of a minimal amount of metadata (pointers to existing data blocks). The NetApp cmdlet simply Clone the Data from the VMDK into a VHD/VHDX writing the appropriate metadata as it goes. The resulting file is a VHD or VHDx file that takes up practically no extra space on disk. It’s like magic made out of chocolate.

In terms of a full conversion, the NetApp cmdlet just converts the disk format from VMware to Hyper-V. It doesn’t actually create a new VM. For a complete conversion, you need to collect details about the VM, remove VMware Tools and of course create a Hyper-V VM to which uses that new VHDx file. That sounds like a lot of work for an average customer to go off and script. Sigh.

So I went back to blogging about the MAT (Migration Automation Toolkit) with management and workflow features, each as rich and fantastic as peanut butter and I wondered wistfully about Project Shift and the chocolaty magic it performs.

And then it happened, we had our peanut butter cup moment. What if we took specific portions of the conversion capabilities from NetApp’s Project Shift technology and mixed it with MAT? Two great technologies that work better together.

Much goodness followed.

Results

What we ended up with, we’re calling MAT (powered by Project Shift), a cold conversion technology that operates at warm conversion speed. We can convert most VMs with less than 5 minutes of downtime (and most of that it waiting for VMware tools removal).

The technology preview of MAT (powered by Project Shift) is available today!

Now this only works on NetApp FAS controllers (running Data ONTAP 8.2 or later) so you can’t just run this in any environment.

How it works:

I’ll post more in the coming weeks but we certainly need to start out with some basics around how all this works.

What you need:

As you can see from the architecture diagram below this overall design is very similar to MAT. Much like with MAT you simply need to configure the Variables.xml file with your details and you can run the script (fair warning there are quite a few more variables now).

The technology preview also assumes that your VMs are already stored on the NetApp FAS controller (if not, a simple command like “Get-VM $VMName | Move-VM –Datastore $NetAppController” can perform a Storage vMotion and get the VMs where they need to be).

MAT4Shift

To get started, after you have your variables set, you go through a few steps:

  • ./Convert-VM.ps1 Configure-NetApp
    • This will configure your NetApp controller so it is talking to both Hyper-V and VMware and the configuration is correct for the conversions
  • Move your VMs to the new container you just created
  • ./Convert-VM.ps1 Collect
    • This will collect your VMware information (like in the MAT)
  • Update the database with the VMs using the VMList.txt (like in MAT)
  • ./Convert-VM.ps1 Convert
    • Starts the actual conversion (like in MAT)

The version we have available today is a technology preview and any NetApp customer is free to test it.

The Dream Team

The MAT (powered by Project Shift) is the result of my collaboration with two guys with a can-do attitude and I want to thank them here:

Glenn Sizemore (NetApp). Glenn has been preaching the awesomeness of NetApp’s Project Shift and was critical in helping get the disk conversion logic into the toolkit. He’s also more than a little handy with PowerShell, I think he may be gunning for Jeffrey Snover’s job. A number of general improvements to the layout and design of the MAT as a whole are all thanks to his suggestions.

I also want to especially thank NetApp for being as generous with Glenn’s time as they have been. Sharing a resource as important as Glenn is a rare thing.

Michael Greene (Microsoft). Michael Greene has forgotten more about networking than I know. His primary focus was enabling the toolkit to collect information about the networking details for each VM and making sure they were recreated on the other end. The dude is smart. He would suggest up solutions quicker than we could test them. When he hands you a chunk of encoded PowerShell (which looks like gobbledy goop), just run it. Watch what happens – you will be amazed.

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment
  • Hi,

    I would like to understand if the tool support conversion of VMs from FC datastore to Hyper-v FC mounted volume.

    Thanks

  • @steve Your connection type is abstracted from the MAT process. As long as NetApp controller can access your VMware environment which in turn can access your fibre channel datastore, that's all you need.

  • Quick question on the NetApp requirements that I am hoping you can answer. Does this require that the NetApp be running in 'C-mode' or will it work with Data ONTAP 8.2P4 running in '7-mode'? Just wondering as the specific unit we run has some resource caveats when running in 'C-Mode'. I am in contact with NetApp as well, but the specific resources were not familiar with this project. Any insight you can offer is appreciated!

  • Hi.

    Does this tool allow for mbr realignment if within a Hyper-V 2012 environment.  VM's have already been converted across from VMware to Hyper-V but need to go back and re-align disks.

    Thanks

  • Shift, the underlying technology that powers rapid container shifting, requires NetApp's Data ONTAP.  ONTAP is a multiprotocol storage operating system that transcends artificial boundaries like FC or NAS.  All long as the virtual disk file resides on ONTAP, you can convert it in seconds.  This is not a copy process.  Shift leverages the same core ONTAP technology used for NetApp Dedupe and rapid cloning.  The only actual data written is the metadata for the target format.  The bulk of the data, the "blob" within the virtual disk file representing a LUN, is cloned by referencing the blocks in situ.

  • Hi Mark,

    I am getting an error during the conversion that states:

    "Error: Migrating VM You cannot call a method on a null-valued expression"

    Have you seen this before and something simple I may have mis-configured?

  • @Merlus

    We had that problem with some older versions when we would get back DisplayName instead of the FQDN. Make sure you have the latest version, which was just updated today. If the problem persists, set your log level to 7 and collect a new set of logs. Send those to mat-questions@microsoft.com (along with any other details about your environment) and we can have a look.

  • Stuck here, my conversion machine, hyperv machine, esx have all access to that path. Not sure why it's not able to access the path. Any ideas?

    Get-Content : Access to the path '\\172.16.112.55\m_shift\qshift\test-vm\test-vm-000001.vmdk' is denied.

    At C:\MAT\ConvertVM-Functions.ps1:711 char:9

    +         Get-Content ('{0}\{1}' -F $NFSPath, $HardDisk.replace('/','\')) | ?{$_ - ...

    +         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

       + CategoryInfo          : PermissionDenied: (\\172.16.112.55...-vm-000001.vmdk:String) [Get-Content], U

      nauthorizedAccessException

       + FullyQualifiedErrorId : GetContentReaderUnauthorizedAccessError,Microsoft.PowerShell.Commands.GetCont

      entCommand

  • Ignore the last one had to fix the name mapping on the netapp. I am now stuck on this error: WARNING - Failed to stage Credentials in Guest - aborting Regedit section when it is saving VM network Configuration. My domain name is in the Administrator group of the VM. I can log on remotely, so not sure what the issue is. Regards Paul (paul dot pa at gmail dot com

  • Ignore the last one had to fix the name mapping on the netapp. I am now stuck on this error: WARNING - Failed to stage Credentials in Guest - aborting Regedit section when it is saving VM network Configuration. My domain name is in the Administrator group of the VM. I can log on remotely, so not sure what the issue is. Regards Paul (paul dot pa at gmail dot com

  • @Paul - At that stage we are accessing the VM through the VMware guest, not remotely. Please set your log level to 7 and collect a new set of logs. Send those to mat-questions@microsoft.com along with any other details about your environment and we can have a look.