Extension sequencing in Azure VM Scale Sets

VM extensions are a good way to customize an Azure VM at deployment time. You can deploy a platform image from the Azure Marketplace and then customize it with one or more extensions. Examples of extensions include diagnostics extensions to emit performance data, antivirus extensions, custom script extensions (where you can run your shell script or PowerShell at VM startup).

Templates, and Azure Resource Manager in general, will do everything they can in parallel. This can be a problem when you have multiple VM extensions defined, and both of them try to use an OS resource which is locked. For example, on Ubuntu, only one process at a time can successfully run apt-get to install software. If Azure Resource Manager ran these two extensions at the same time on the same VM, one would fail.

Extension sequencing is not a problem for regular Azure virtual machines, because you can define the extension resource with a dependsOn clause. You can use this to make one extension depend on another, so it will only run after the other extension has completed.

Extension sequencing is a problem for Azure VM Scale Sets however, because scale set extensions are not defined externally. Extensions are just another property under the Microsoft.Compute/virtualMachineScaleSets resource and don’t have a dependsOn clause. If you define a list of extensions, they could all potentially run at the same time.

Autoscale

A common VM Scale Set scenario where lack of extension sequencing can be a problem is with autoscale. To set up autoscale on a scale set, you need to define a diagnostic extension to emit performance data to a storage account. This performance data is then evaluated by the Insights engine to determine when to emit scale events.

On Linux, and let’s take the example of Ubuntu, the Linux Diagnostic Extension uses the OS installer while it’s setting up. If you also want to to install something on your VMs at deployment time, and use a custom script extension, the custom script extension can fail with errors like this in the extension.log file: Could not get lock /var/lib/dpkg/lock

Ways to sequence extensions in VM Scale Sets

Here are a couple of Ubuntu examples of how to make a custom script extension wait for the Linux Diagnostic extension to finish executing:

1. Put a loop around apt-get until it returns success

Here’s an example from the custom script used by the Ubuntu Apache PHP autoscale example in Azure Quickstart templates. The idea is put a loop around your apt-get update calls to keep trying until they work:

until apt-get -y update && apt-get -y install apache2 php5 
do
 echo "Try again"
 sleep 2
done

This usually works. The only potential issue is that there could be a timing problem where the script temporally gets a lock on the apt process and then loses it, causing the extension to fail.

2. Wait for the Linux Diagnostic extension to complete

Another approach is to put a loop at the beginning of your custom script which waits for the Linux Diagnostic extension to write a record to the extension log saying it’s done installing stuff. This check can’t make any assumptions about what version of the diagnostic extension is running, so it involves using the find command. Here’s an example from another autoscale example which installs Python bottle:

while ( ! (find /var/log/azure/Microsoft.OSTCExtensions.LinuxDiagnostic/*/extension.log | xargs grep "Start mdsd"));
do
  sleep 5 
done 
apt-get -y update 
apt-get -y install python3-bottle

Overall this approach is safer, because you know the diagnostic extension is done at this point. Though if a future updated version of the diagnostic extension changed the way it wrote to the extension log, you might need to maintain this code.

How you implement extension sequencing is going to depend on which extensions are running and how they compete with one another if at all.

The good news is that a built-in way to do extension sequencing will be added to VM Scale Sets in the near future, so there will be an easier way to manage the order in which extensions are installed.

Advertisements
This entry was posted in Cloud, Computers and Internet, Linux, Ubuntu, VM Scale Sets and tagged , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s