Deploying Applications in Azure VM Scale Sets


How are Applications deployed on VM Scale Sets?

An application running on a VM Scale Set is typically deployed in one of three ways:

1. Installing new software on a Platform image at deployment time. A platform image in this context is a operating system image from the Azure Marketplace, like Ubuntu 16.04, Windows Server 2012 R2 etc.

You can install new software on a platform image using a VM Extension. A VM extension is software that runs when a VM is deployed. You can run any code you like at deployment time using a custom script extension. Here’s an example Azure Resource Manager Template with two VM extensions, a custom script extension to install Apache and PHP, and a diagnostic extension to emit performance data which can be used by Azure autoscaling: Autoscale a VM Scale Set running an Ubuntu/Apache/PHP app.

An advantage of this approach is you have a level of separation between your application code and the OS, and can maintain your application separately. Of course that means there are also more moving parts, and depending on how much needs to download and configure when the extension runs, it could add to the VM deployment time.

2. Create a custom VM image which includes both the OS and the application in a single VHD. Here the scale set consists of a set of VMs copied from an image created by you, which you have to maintain. This means no extra configuration is required at VM deployment time, but there are some limitations with custom images in the current version of VM Scale Sets – you are limited to a single storage account, and hence a maximum of 40 VMs in a scale set (as opposed to 100 VMs in a scale set which uses platform images).

3. Deploy a platform or a custom image which is basically a container host, and install your application as one or more containers which you can manage with an orchestrator or config management tool. This nice thing about this approach is that you have completely abstracted your cloud infrastructure from the application layer and can maintain them separately.

What happens when a VM Scale Set Scales Out? 

When you add one or more VMs to a scale set by increasing the capacity – whether manually or through autoscale – the application is automatically installed. For example if the scale set has extensions defined, they run on a new VM each time it is created. If the scale set is based on a custom image, any new VM will be a copy of the source custom image. If the scale set VMs are container hosts, then you might have startup code to load the containers in a custom script extension, or an extension might install an agent which registers with a cluster orchestrator (e.g. Azure Container Service).

How do you manage application updates in VM Scale Sets?

For application updates in VM Scale Sets, three main approaches follow from the three application deployment methods outlined above:

1. Updating with Extensions. Any VM extensions which are defined for a VM Scale Set are executed each time a new VM is deployed, an existing VM is reimaged, or a VM extension is updated. If you need to update your application, Directly updating an application through extensions is a viable approach – you can update the extension definition, or the extension code can point to a location which contains updateable software.

The hard problems there are:

– Security – how to maintain certificates/shared access signatures.

– Scaling – how the application updates, how long it takes, when you scale out.

2. The immutable approach. When you bake the application (or app components) into a VM image you can focus on building a reliable pipeline to automate build, test, deployment of the images (e.g. Jenkins based). You can design infrastructure architecture to facilitate rapid swapping of a stage scale set into production. A good example of this approach is the Azure Spinnaker driver work:

Packer and Terraform support Azure Resource Manager, so you can also define your images “as code” and build them in Azure, then use the VHD in your scale set. Where this would become problematic is for Marketplace images, where extensions/custom scripts become more important as you don’t directly manipulate bits from Marketplace.

3. Update Containers. Abstract the application lifecycle management to a level above the cloud infrastructure, e.g. by encapsulating applications, and app components into containers and manage these through container orchestrators and app managers like chef/puppet.

The scale set VMs then become a stable substrate for the containers and only require occasional security and OS related updates. As mentioned, the Azure Container Service is a good example of taking this approach and building a service around it.

How do you roll out an OS update across update domains?

Suppose you want to update your OS image while keeping the VM Scale Set running. One way to do this is to update the VM images one VM at a time. You can do this with PowerShell or Azure CLI. There are separate commands to update the VM Scale Set model (how its configuration is defined), and to issue “manual upgrade” calls on individual VMs.

Here’s an example Python script which automates this to update a VM Scale Set one update domain at a time: (Caveat: it’s more of a proof of concept than a hardened production-ready solution – you might want to add some error checking etc.).

This entry was posted in Cloud, Computers and Internet, Containers, VM Scale Sets and tagged , . Bookmark the permalink.

9 Responses to Deploying Applications in Azure VM Scale Sets

  1. Hermano Leite says:

    Hi, thank you for your post, helped me to understand how the deployment works with custom image!
    Is there a way that each VM automatically logs in when created?
    I was trying with a custom image that was configured to autologon, however it is erased at generalizing process.
    Thank you very much

  2. sendmarsh says:

    Firstly I think this is a question is about VMs as much as VM scale sets right? I.e. your problem is that the generalizing process when creating a VM image is re-setting something you need. I’m curious why you need autologon. Can you use a VM extension to run something or change any setting when the VM starts, or is there something you need this VM, or VM scale set, to do which only autologon can provide?

  3. Soulemane Moumie says:

    HI NeilaiVijay, Thank you for this wonderful post. It is something I was eager looking for. Please can you confirm which of the following template match with type 2 (App + OS as image).

    Thank you.

  4. sendmarsh says:

    Take a look at this template: – it creates a VM Scale Set from a custom image, and defines a load balancer with 2 load balancer rules. One is to load balance incoming requests to port 25565 to different backends, the other is an inbound NAT rule to map port 50000 and higher to port 22 on each VM in the scale set. In your example, the template you use to create the scale set should also include a load balancer, with an inbound NAT rule, which would map a port range (e.g. 50000-50099) to port 3389 on each VM.
    This should work, the only additional thing you need to keep in mind is that if you have the default scale set ‘overprovision’ property set to true, it’s possible that your first NAT rule might be 50001 instead of 50000 (you can query the load balancer to see these rules, check in the portal, or just try both).

  5. mwweeb says:

    This post is GREAT! It is the only one that finally explains in a clear way “How do you manage application updates in VM Scale Sets?” …. I found no other documents about this question.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s