Azure Planned Maintenance Experience

The Azure Team recently announced the availability of the new Planned Maintenance experience. The goal is to provide customers a new experience that enables them to initiate a proactive redeploy of their VMs for planned maintenance events that require a reboot. Maintenance will be communicated via the Azure portal, Azure Service Health dashboard, CLI and PowerShell. Subscription owners and admins will also continue to get maintenance notifications via e-mail.

Customers can now test the Planned Maintenance experience in the US West Central region. To do so, you need to use a special link for accessing the Azure Portal. In the overview screen for a VM which will be impacted by scheduled maintenance, you’ll see a screen similar to the one below.

azureportalmaintenance

Choosing to initiate maintenance now will redeploy the VM onto a node which has already been upgraded, allowing you to control when the VM is restarted. In addition, the VM blade in the preview portal will contain new columns which can be used to determine the maintenance status of the VM:

azureportalmaintenance2

Maintenance windows which involve VM reboots follow the same update patterns Azure would use in any other situation. This means that VMs that are part of Availability Sets or VM Scale Sets will be part of separate update domains and Azure will never update hosts in more than one update domain at a time. Each Azure region also has a region pair (like US East and US West) and updates are never performed across region pairs at the same time. However, for VMs which are not in an availability set, this new feature gives you the ability to control when the updates are done and schedule those within maintenance windows. Also, if you need an exact window during which maintenance is performed even for VMs in an availability set, this is the way to accomplish that. Lastly, if you need to coordinate the impact of VM redeployment or applications don’t support HA, the planned maintenance support experience would help you deal with those situations as well.

Full control of this is also available via PowerShell as you would expect. You can query the planned maintenance status of a VM using Get-AzureRmVM: Get-AzureRmVM -ResourceGroupName “PrevMaintenance” -Name “mytestvm2” -Status. You can also perform (kick off) the maintenance task using Restart-AzureRmVM and the “PerformMaintenance” parameter: Restart-AzureRmVM -PerformMaintenance -name “mytestvm2” -ResourceGroupName “PrevMaintenance”. PowerShell makes it easy to get the status of all of the VMs across a subscription very easily. Here’s a sample of how you would get a list of all VMs and their planned maintenance status in the current subscription:

listvms

Some other useful resources:

Overview Video on Channel 9 (15min)
Linux Planned Maintenance Documentation
Windows Planned Maintenance Documentation
Monitoring Service Notifications using Service Health

 

Azure Application Gateway SSL Policies

A client recently ran Qualys SSL Server Test against their web applications published through the Azure Application Gateway. The test graded the SSL security on the site as a “B” mainly because the server supported weak Diffie-Hellman (DH) key exchange parameters.

Diffie-Hellman key exchange is a popular cryptographic algorithm that allows Internet protocols to agree on a shared key and negotiate a secure connection. SSL sites that support export cipher suites and don’t use 2048-bit or stronger Diffie-Hellman groups with “safe” primes are susceptible to attacks like LogJam. Luckily, a feature known as SSL Policy in the Azure Application Gateway allows you to reduce the potential for these types of attacks.

The SSL handling in Azure Application Gateway (used for things such as SSL offloading and centralized SSL handling) allows you to specify a central SSL policy that’s suited to your organizational security requirements. The SSL policy includes control of the SSL protocol version as well as the cipher suites and the order in which ciphers are used during an SSL handshake. Application Gateway offers two mechanisms for controlling SSL policy: either a predefined policy or a custom policy. Here’s a link to the documentation for SSL policy with Azure Application Gateway. Changing the SSL policy for a new Application Gateway deployment can be accomplished using PowerShell and changing an existing deployment’s SSL policy is also easily done via the cmdlets. Below is an example of how to do this with a few lines of PowerShell.

One “gotcha” is that the predefined SSL policy which disables the weaker cipher suites also sets a minimum TLS version of v1.2 and breaks most older browsers. If that’s not a concern, use the latest predefined SSL policy – otherwise you’ll have to use a custom policy and specify a lower minimum TLS version to support older IE browsers running on Windows 7, for example.

# Get Configuration of AppGW
$appgw = Get-AzureRmApplicationGateway -Name $GWName -ResourceGroupName $GWResourceGroupName
# Set SSL Policy on AppGW to Custom Policy based on Most Recent Security Policy w/TLSv1.0 Support. FYI: Will work on any version of IE > 8.0 running on Windows 7. No Windows XP support!
Set-AzureRmApplicationGatewaySslPolicy -ApplicationGateway $appgw -PolicyType Custom -MinProtocolVersion TLSv1_0 -CipherSuite “TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256″,”TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384″,”TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA”,”TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA”,”TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256″,”TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384″,”TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384″,”TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256″,”TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA”,”TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA”,”TLS_RSA_WITH_AES_256_GCM_SHA384″,”TLS_RSA_WITH_AES_128_GCM_SHA256″,”TLS_RSA_WITH_AES_256_CBC_SHA256″,”TLS_RSA_WITH_AES_128_CBC_SHA256″,”TLS_RSA_WITH_AES_256_CBC_SHA”,”TLS_RSA_WITH_AES_128_CBC_SHA”
# Set SSL Policy on AppGW to Most Recent Policy w/TLSv1.2 Minimum Support. FYI: Becuase TLS v1.0 is not supported, this will break any browser earlier that IE 11!
Set-AzureRmApplicationGatewaySslPolicy -ApplicationGateway $appgw -PolicyType Predefined -PolicyName “AppGwSslPolicy20170401S”
# Update the gateway with validated SSL policy
Set-AzureRmApplicationGateway -ApplicationGateway $appgw

PowerShell Language Modes

Language Modes in Windows PowerShell essentially define which elements of the PowerShell language can be used in a PowerShell session. As you’ve already discovered, the different language modes are documented publicly here. Language modes are intended to be specified as part of a session configuration file when a new PowerShell session is launched on a system.

Another way of setting the language mode for a PowerShell session would be to use either DeviceGuard or AppLocker in enforced mode with PowerShell 5+ and WMF 5+ or (more easily) to simply make use of a system environment variable called “__PSLockDownPolicy” to configure it. The system environment variable could be controlled on managed PCs in a variety of ways including via Group Policy.

There are two things to keep in mind regarding the use of language modes and the “__PSLockDownPolicy” system variable.

The first is that both the restricted language mode and constrained language mode could very likely have an impact on any SCCM scripts that use PowerShell or with any remote PowerShell management that you may be performing against systems. If scripts contain any scripting elements or code elements from other languages, they will most likely be prevented from running as expected. At its most restrictive setting (restricted and no language), script blocks aren’t allowed and/or the only thing that can be executed are cmdlets without any language elements at all.

The second thing is that when setting the language mode via the system variable, it will apply to all sessions started on that system and can’t be easily overridden on a user-by-user basis or a session-by-session basis. This means that it will affect built-in security principals as well as local users and would certainly have an effect on not only SCCM scripts and remote PowerShell management but other things like setting task scheduler to run as System and kick off a script at a pre-defined time.

The moral of the story is that while enforcing the PowerShell language mode on systems in an environment can reduce the potential for PowerShell to be used as an attack vector in malware or other exploits, it can have a pretty big impact on managing those systems via PowerShell for the purposes of good, not evil.

Copying Exported Azure Managed Disks

Of the many benefits of Azure Managed Disks, one of the more important ones is the move away from addressing virtual disks as regular blob URIs in standard and premium storage accounts. Azure storage accounts rely on the use of primary and secondary access keys to secure access to the contents of a storage account and there’s no ability to assign granular access permissions at the blob level. If someone has either the primary or secondary access key for a storage account, they essentially have full access to the contents of that storage account including any virtual disks (VHDs) contained within it. The only way to secure the contents of these VHDs is by implementing Azure Disk Encryption (ADE) so that the virtual disks are encrypted at the OS level and copying them through the use of an access key and the Azure Storage REST APIs won’t allow someone to access the contents unless they also have the encryption key used to secure the data.

Enter Azure Managed Disks. With this feature, there is no longer a way to access the VHD blobs via a URI and access to a storage account. Placement of the VHD blobs on Azure storage is managed by the service and Azure Role-Based Access Control can be used to assign specific permissions for a managed disk to one or more users. Managed Disks exposes a variety of operations including read, write, delete and retrieval (export) of a disk via a shared access signature (SAS) URI. For this last part, an administrator can export the managed disk via the Azure Management Portal and they’re receive a URL to use for copying the managed disk to either an Azure storage account or to a location on their local PC.

Copying the VHD via the SAS URI is a bit tricky if you’re trying to use standard tools like Azcopy. To download the VHD to a local PC you can use the hyperlink provided directly in the portal. To copy the VHD to a storage account, though, you can’t use Azcopy. Azcopy requires both READ and LIST SAS permissions to copy a blob. However, the Azure REST SAS API currently only supports read URIs (https://docs.microsoft.com/en-us/rest/api/manageddisks/disks/disks-grant-access). The easiest way I’ve found to complete the copy is to use the Start-AzureStorageBlobCopy cmdlet. This cmdlet starts an asynchronous copy job using the storage REST APIs and works just fine with the read only SAS URI provided by the Azure Portal. Here’s how to accomplish the task:

First, make sure you’re authenticated to Azure and have selected the proper subscription containing the destination storage account for the VHD:

Login-AzureRmAccount

Select-AzureRmSubscription -SubscriptionName <Subscription Name>

Then, get the access key for the destination storage group and create a context to use for the copy job:

$dstkey = Get-AzureRmStorageAccountKey -StorageAccountName <Destination Storage Account Name> -ResourceGroupName <Destination Storage Account Resource Group>

$dstcontext = New-AzureStorageContext -StorageAccountName <Destination Storage Account Name> -StorageAccountKey $dstkey.Value[0]

Lastly, create the copy job and then run the Get-AzureStorageBlobCopyState with the “WaitforComplete” parameter to provide a visual indication of the progress of the copy:

$copyblob = Start-AzureStorageBlobCopy -AbsoluteUri <Export managed disk URL from Portal – Full URL> -DestContext $dstcontext -DestContainer <Destination Container Name> -DestBlob <Destination VHD Filename>

$copyblob | Get-AzureStorageBlobCopyState –WaitForComplete

Azure VM Recreation Script

There are many tasks associated with virtual machines in Azure which can only be performed by deleting the VM configuration while preserving the underlying virtual hard disks and then recreating the VM configuration with the proper settings. Some of the things that come to mind are:

  • Renaming a VM resource
  • Moving a VM from one VNET to another
  • Moving a VM from one Azure region to another
  • Moving a VM into or out of an availability set
  • Changing a single NIC VM to a multi NIC VM and vice versa

There are likely others, I’m sure. However, these are actually pretty basic needs for most folks and it’s a shame it isn’t easier to do it. I’ve gotten these types of requests so many times that I finally decided to build a script as a basis for enabling them in a semi-automated fashion. Attached is sample script which includes the methodology and was designed to move a VM in one region to another region assuming that someone copied the underlying VHDs to the destination using either AzCopy or Start-AzureStorageBlobCopy.

At a high-level, here’s what the script does:

  1. Authenticates to the specified Azure subscription if necessary
  2. Reads a list of disk names and VHD URIs for the underlying disks
  3. Gets the source VM definition using Get-AzureRmVM
  4. Creates a new VM definition in the destination location
  5. Adds OS Disks and Data Disks using the provided disk mappings
  6. Creates NIC(s) for destination VM – prompts for new IPs for static addresses
  7. Deploys new VM

It goes without saying that the source VM should be stopped when the VHDs are copied and the script is run. There’s no need to delete the source VM until the new one is successfully built. This handles availability sets if the original VM was in one and works for VMs with either managed disks or unmanaged disks. There are some limitations as of now:

  • Doesn’t work with VMs encrypted using Azure Disk Encryption (ADE)
  • Doesn’t move Public IP resources
  • Doesn’t move NSGs if those were assigned to the VM’s NIC(s) directly
  • Will only move the first IP configuration for each NIC
  • Doesn’t recreate any tags associated with the source VM
  • Doesn’t redeploy any extensions present on the source VM

The real idea behind this was that it would serve as a starting point. This same script could easily be used to migrate a VM from standard to premium storage with a few tweaks or to redeploy a VM to a different VNET or availability set if desired. Hopefully folks find it useful as a starting point for their own adventures.

Create VM from VHD v3 (PowerShell script)

Azure AppGW and IP Address Usage

I had to do some digging recently to answer a question about the Azure Application Gateway and its consumption of IP addresses from the Application Gateway subnet address space. It’s not well documented and without an understanding of it you might end up painting yourself in a corner when designing the layout of your subnets and address spaces in an Azure Virtual Network.

If you’re not familiar with it, the Azure Application Gateway is a dedicated virtual appliance which provides various HTTP/HTTP (layer 7) load balancing applications and is provided as a multi-instance, fully managed service. It also has some web application firewall (WAF) capabilities and can be configured as an internet facing gateway, internal gateway or a combination of both.

scenario

The first important point to understand is that the Azure Application Gateway must be deployed in its own subnet. The subnet created or used for application gateway cannot contain any other types of resources (VMs, load balancers, etc.) but *can* contain other application gateways. If the subnet in question doesn’t meet this criteria, it won’t appear in the Azure portal as one of the possible subnets for selection in the deployment wizard. Because of this, one might be tempted to create the smallest address space possible for the subnet (a /29 with 3 usable IPs once Azure takes the first three from the range) especially if there’s no plan to deploy Azure Application Gateways with internal addresses. That would be a big mistake and here’s why: each instance of an application gateway consumes an address from the subnet’s address pool.

gwinstances

When you create an application gateway, an endpoint (public VIP or internal ILB IP) is associated and used for ingress network traffic. This VIP or ILB IP is provided by Azure and works at the transport layer (TCP/UDP). The application gateway then routes the HTTP/HTTPS traffic based on its configuration to a backend pool comprised of a virtual machine, cloud service, or an internal or an external IP address.

frontendconfigs

Regardless of whether an application gateway is deployed with a Public IP, Private IP or both each instance of a gateway consumes an address from the subnet’s address pool. If the gateway is then configured with a private ILB IP, it consumes one additional IP address from the subnet’s address pool for that ILB IP. The only IP that will show up in the subnet as being consumed, though, is the ILB IP. You won’t see any of the IPs in use by the gateway instances. The only way you’ll know if you hit the limit is that your deployment (or instance increase) will fail with an error that there’s not enough address space. This is definitely not ideal and hopeful will be improved at some point.

Some examples:

2 instance GW with a Public IP Only = 2 IPs (2 + 0 = 2)

2 instance GW with a Private IP Only = 3 IPs (2 + 1 = 3)

4 instance GW with a Public & Private IP = 5 IPs (4 + 1 = 5)

So keep this in mind when designing the layout of your Azure VNET and its subnets and address spaces. Not having enough addresses in the subnet to expand the number of instances for your production Application Gateway can be a real pain and certainly will require downtime to resolve.