How to upgrade an Azure VM Scale Set without shutting it down

Note: This article was later published in the official Azure documentation. You can find it here: Upgrade a virtual machine scale set.

This article describes how you can roll out an OS update to an Azure VM Scale Set without any downtime. In this context an OS update is either changing the version/sku of the OS, or changing the URI of a custom image. Updating without downtime means updating VMs one at a time, or in groups (such as one fault domain at a time), rather than all at once, so any VMs which are not being upgraded can keep running.

Note: Azure scale set rolling upgrade and automated OS upgrade are now in preview. With these features you’ll be able to roll out upgrades to OS images without needing the scripts or steps described below. To find out more information about rolling and automated OS upgrade go here: Azure VM scale set automatic upgrade and rolling upgrade preview.

To avoid ambiguity let’s distinguish 3 types of OS update you might want to do:
1. Changing the version or sku of a platform image. E.g. changing Ubuntu 14.04.2-LTS version from 14.04.201506100 to 14.04.201507060, or changing the Ubuntu 15.10/latest sku to 16.04.0-LTS/latest etc.. Covered in this article.
2. You built a new version of a custom image and want to change the URI which points to the image (properties->virtualMachineProfile->storageProfile->osDisk->image->uri). Covered in this article.
3. Patching the OS from within a VM e.g. installing a security patch, using Windows Update etc. Supported but not covered in this article.

The first 2 are supported requirements. For the third one, at least for now, you’d need to create a new scale set to do that. This article covers options 1. and 2.
Note: VM Scale Sets which are deployed as part of an Azure Service Fabric cluster are not covered here.

The basic sequence for changing the OS version/sku of a platform image or the URI of a custom image looks like this:
– Get the VMSS model.
– Change the version, sku or URI value in the model.
– Update the model.
– Do a manualUpgrade call on the VMs in the scale set. This is only relevant if the upgradePolicy property of your Scale Set is set to “Manual”. If it is set to “Automatic”, all the VMs will upgraded at once and there will be downtime.

With this all this in mind, let’s review how you could update the version of a scale set in PowerShell, and using the REST API. These examples cover the case of a platform image, but hopefully I’ve provided enough information for you to adapt this to a custom image..

PowerShell

This example updates a Windows VM Scale Set to a new version “4.0.20160229”. After updating the model, it does an update one VM instance at a time.

$rgname = "myrg"
$vmssname = "myvmss"
$newversion = "4.0.20160229"
$instanceid = "1"

# get the VMSS model
$vmss = Get-AzureRmVmss -ResourceGroupName $rgname -VMScaleSetName $vmssname

# set the new version in the model data
$vmss.virtualMachineProfile.storageProfile.imageReference.version = $newversion

# update the VMSS model
Update-AzureRmVmss -ResourceGroupName $rgname -Name $vmssname -VirtualMachineScaleSet $vmss

# now start updating instances
Update-AzureRmVmssInstance -ResourceGroupName $rgname -VMScaleSetName $vmssname -InstanceId $instanceId

If you were updating the URI for a custom image instead of changing a platform image version, you’d replace the “set the new version” line with something like this:

# set the new version in the model data
$vmss.virtualMachineProfile.storageProfile.osDisk.image.uri= $newURI

Using the REST API

Here are a couple of Python examples which use the Azure REST API to roll out an OS version update. In both cases they make use of the lightweight azurerm library of Azure REST API wrapper functions to do a GET on the scale set to get the model, and then a PUT with an updated model. They also look at VM instances views to identify the VMs by update domain.

vmssupgrade

vmssupgrade is Python script to roll out an OS upgrade to a running VM Scale Set, one update domain at a time. You can find it here: https://github.com/gbowerman/vmsstools

This script lets you choose specific VMs to update, or specify an update domain, and supports changing a platform image version OR changing the URI of a custom image.

vmsseditor

This is a general purpose editor for VM Scale Sets, which shows VM status as a heatmap where one row represents one UD. Among other things you can update the model for a VMSS with a new version, sku or custom image URI, and then pick Fault Domains to upgrade (i.e. all the VMs in that UD are then upgraded to the new model), or a rolling upgrade based ont he batch size of your choice. vmsseditor can be found in the following github repo: https://github.com/gbowerman/vmssdashboard

E.g. here I’ve just updated the model of a scale set to Ubuntu 14.04-2LTS version 14.04.201507060 (note this is an old screenshot, many more options have since been added to this tool)..

image

After clicking Upgrade and then Get Details again, VMs in UD 0 are starting to update..

image

CLI?

I haven’t included a CLI example yet. Will try and get to that soon. I’d probably do it by deploying an empty template just consisting of an updated SKU packet and version to an existing template.

Advertisements
This entry was posted in Cloud, Python, VM Scale Sets and tagged , , . Bookmark the permalink.

10 Responses to How to upgrade an Azure VM Scale Set without shutting it down

  1. Homer Cavazos says:

    Excellent Post, Thanks You!

  2. Pingback: VIP Swap – blue-green deployment in Azure Resource Manager | MSFT Stack

  3. Sijo says:

    Hi Thanks for this post.
    When using azurerm how can I use 2-factor authentication. Is it possible? If so can you please provide the details.

    • sendmarsh says:

      Hi, unfortunately it only works with “service principal” authentication at the moment. It’s a good suggestion, supporting 2FA would be a good addition to azurerm.

      • sendmarsh says:

        Note, azurerm has added get_token_from_cli() which gets an authentication from the CLI cache if you run it in an environment which has recently logged on using CLI, for example from Azure Cloud Shell. This is one way to authenticate without needing a service principal.

  4. Hi Guy,
    Thanks for this tool! I liked it so much I have created a version in PowerShell.
    https://github.com/fbinotto/AzureScaleSetManagement

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s