Restricting Managed Disk Export

One of the big benefits of Managed Disks in Azure is that a VM’s VHD(s) are stored in a storage account which is managed by Azure and no longer has a public endpoint like page blobs did in the unmanaged disk model. One potential downside to this old approach was that anyone with the primary or secondary access key to the storage account where the VM’s VHDs were stored could simply connect to the storage account and download the VHD. Without encryption technologies like ADE or DMCrypt, this meant that person had a working copy of your VM’s disks and full access to the data on them.

With Managed Disks, VHDs no longer have a public blob endpoint but the console does provide the ability to export the managed disk which creates a temporary SAS URI and the ability to connect to the blob and download via your browser or via the Azure Storage REST API.

snip_20171208113940

The concern that some might have is that any administrator with access to the Managed Disk resource(s) can simply export the VHDs and run off with the data. First, you should be always auditing actions such as these via the Activity Log. Second, you can block the ability to do this in the first place by using RBAC in Azure. Here is the full list of Azure Role-Based Access Control operators. The two operations specific to Managed Disk export are:

  • Microsoft.Compute/disks/beginGetAccess/action (Get Disk SAS URI)
  • Microsoft.Compute/disks/endGetAccess/action (Revoke Disk SAS URI)

To block these operations, you need to add them to the NotActions for a custom role and then assign that custom role to your administrators. For example, this is the JSON definition for a custom role that is equivalent to a contributor but blocks managed disk export and snapshot operations specifically. Full documentation for creating custom roles here: https://docs.microsoft.com/en-us/azure/active-directory/role-based-access-control-custom-roles.

snip_20171208114701

Here’s a simple PowerShell script which will list all of the RBAC roles in the current selected subscription and identify whether or not these operations are listed either in the Action or NotAction. You can run this against a specific subscription to see if there’s a role defined to block these operations and then look at that role in the Web Portal to see who is assigned that role.

snip_20171208114812.png

 

ASR Post-Failover VM Customizations

I had a customer ask whether it was possible to perform scripted customizations against a recovery VM after an Azure Site Recovery (ASR) fail-over. The catch was that they didn’t necessarily know whether Remote PowerShell ports (5985/5986) would be open on recovery VM nor could they guarantee they’d have valid credentials for accessing the VM with administrative privileges. In this case, the Azure Custom Script Extension fits the requirements perfectly.

The Custom Script Extension downloads and executes scripts on Azure virtual machines. This extension is useful for post deployment configuration, software installation, or any other configuration / management task. Scripts can be downloaded from Azure storage or GitHub, or provided to the Azure portal at extension run time. This means you can create a script to perform the customizations, store that script in Azure Blob Storage and then use an Azure Automation Runbook as part of the ASR Recovery Plan to install the extension in the VM post-failover.

High-level steps and resources:

  1. Create the script which will perform the VM changes and upload to Azure Blob Storage.
  2. Setup an Azure Automation account – https://docs.microsoft.com/en-us/azure/automation/automation-offering-get-started. The Automation account can be in any Azure region. The Automation account must be in the same subscription as the Azure Site Recovery vault.
  3. Create a runbook in Azure Automation which should do the following:
    • Receive the Recovery Plan context from ASR as a parameter.
    • Authenticate to the Azure subscription using a RunAsConnection or RunAsCertificate
    • Using the VM information supplied by ASR in the Recovery Plan Context, use the Set-AzureRmVMCustomExtension cmdlet to install the Custom Script Extension on the VM using the script from Step #1.
    • You can use runbook variables to pass in details about the storage account to use, the name of the script, etc. Here is a sample runbook to use as a starting point: CSEASRRunbook.
  4. Add the runbook to the ASR Recovery Plan as a post-failover step as documented here: https://docs.microsoft.com/en-us/azure/site-recovery/site-recovery-runbook-automation.

In researching this, I also found a post on the Azure blog which provides a runbook that takes the script as an input and builds the script file, uploads it to Azure and configures the Custom Script Extension. It’s a bit more involved than this simple example as it saves you the effort of building the script and uploading to Azure Blob Storage ahead of time. You can check it out at https://azure.microsoft.com/en-in/blog/azure-automation-run-tasks-on-azure-virtual-machines-without-opening-ports/.

Azure Planned Maintenance Experience

The Azure Team recently announced the availability of the new Planned Maintenance experience. The goal is to provide customers a new experience that enables them to initiate a proactive redeploy of their VMs for planned maintenance events that require a reboot. Maintenance will be communicated via the Azure portal, Azure Service Health dashboard, CLI and PowerShell. Subscription owners and admins will also continue to get maintenance notifications via e-mail.

Customers can now test the Planned Maintenance experience in the US West Central region. To do so, you need to use a special link for accessing the Azure Portal. In the overview screen for a VM which will be impacted by scheduled maintenance, you’ll see a screen similar to the one below.

azureportalmaintenance

Choosing to initiate maintenance now will redeploy the VM onto a node which has already been upgraded, allowing you to control when the VM is restarted. In addition, the VM blade in the preview portal will contain new columns which can be used to determine the maintenance status of the VM:

azureportalmaintenance2

Maintenance windows which involve VM reboots follow the same update patterns Azure would use in any other situation. This means that VMs that are part of Availability Sets or VM Scale Sets will be part of separate update domains and Azure will never update hosts in more than one update domain at a time. Each Azure region also has a region pair (like US East and US West) and updates are never performed across region pairs at the same time. However, for VMs which are not in an availability set, this new feature gives you the ability to control when the updates are done and schedule those within maintenance windows. Also, if you need an exact window during which maintenance is performed even for VMs in an availability set, this is the way to accomplish that. Lastly, if you need to coordinate the impact of VM redeployment or applications don’t support HA, the planned maintenance support experience would help you deal with those situations as well.

Full control of this is also available via PowerShell as you would expect. You can query the planned maintenance status of a VM using Get-AzureRmVM: Get-AzureRmVM -ResourceGroupName “PrevMaintenance” -Name “mytestvm2” -Status. You can also perform (kick off) the maintenance task using Restart-AzureRmVM and the “PerformMaintenance” parameter: Restart-AzureRmVM -PerformMaintenance -name “mytestvm2” -ResourceGroupName “PrevMaintenance”. PowerShell makes it easy to get the status of all of the VMs across a subscription very easily. Here’s a sample of how you would get a list of all VMs and their planned maintenance status in the current subscription:

listvms

Some other useful resources:

Overview Video on Channel 9 (15min)
Linux Planned Maintenance Documentation
Windows Planned Maintenance Documentation
Monitoring Service Notifications using Service Health

 

Azure Application Gateway SSL Policies

A client recently ran Qualys SSL Server Test against their web applications published through the Azure Application Gateway. The test graded the SSL security on the site as a “B” mainly because the server supported weak Diffie-Hellman (DH) key exchange parameters.

Diffie-Hellman key exchange is a popular cryptographic algorithm that allows Internet protocols to agree on a shared key and negotiate a secure connection. SSL sites that support export cipher suites and don’t use 2048-bit or stronger Diffie-Hellman groups with “safe” primes are susceptible to attacks like LogJam. Luckily, a feature known as SSL Policy in the Azure Application Gateway allows you to reduce the potential for these types of attacks.

The SSL handling in Azure Application Gateway (used for things such as SSL offloading and centralized SSL handling) allows you to specify a central SSL policy that’s suited to your organizational security requirements. The SSL policy includes control of the SSL protocol version as well as the cipher suites and the order in which ciphers are used during an SSL handshake. Application Gateway offers two mechanisms for controlling SSL policy: either a predefined policy or a custom policy. Here’s a link to the documentation for SSL policy with Azure Application Gateway. Changing the SSL policy for a new Application Gateway deployment can be accomplished using PowerShell and changing an existing deployment’s SSL policy is also easily done via the cmdlets. Below is an example of how to do this with a few lines of PowerShell.

One “gotcha” is that the predefined SSL policy which disables the weaker cipher suites also sets a minimum TLS version of v1.2 and breaks most older browsers. If that’s not a concern, use the latest predefined SSL policy – otherwise you’ll have to use a custom policy and specify a lower minimum TLS version to support older IE browsers running on Windows 7, for example.

# Get Configuration of AppGW
$appgw = Get-AzureRmApplicationGateway -Name $GWName -ResourceGroupName $GWResourceGroupName
# Set SSL Policy on AppGW to Custom Policy based on Most Recent Security Policy w/TLSv1.0 Support. FYI: Will work on any version of IE > 8.0 running on Windows 7. No Windows XP support!
Set-AzureRmApplicationGatewaySslPolicy -ApplicationGateway $appgw -PolicyType Custom -MinProtocolVersion TLSv1_0 -CipherSuite “TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256″,”TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384″,”TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA”,”TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA”,”TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256″,”TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384″,”TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384″,”TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256″,”TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA”,”TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA”,”TLS_RSA_WITH_AES_256_GCM_SHA384″,”TLS_RSA_WITH_AES_128_GCM_SHA256″,”TLS_RSA_WITH_AES_256_CBC_SHA256″,”TLS_RSA_WITH_AES_128_CBC_SHA256″,”TLS_RSA_WITH_AES_256_CBC_SHA”,”TLS_RSA_WITH_AES_128_CBC_SHA”
# Set SSL Policy on AppGW to Most Recent Policy w/TLSv1.2 Minimum Support. FYI: Becuase TLS v1.0 is not supported, this will break any browser earlier that IE 11!
Set-AzureRmApplicationGatewaySslPolicy -ApplicationGateway $appgw -PolicyType Predefined -PolicyName “AppGwSslPolicy20170401S”
# Update the gateway with validated SSL policy
Set-AzureRmApplicationGateway -ApplicationGateway $appgw

Copying Exported Azure Managed Disks

Of the many benefits of Azure Managed Disks, one of the more important ones is the move away from addressing virtual disks as regular blob URIs in standard and premium storage accounts. Azure storage accounts rely on the use of primary and secondary access keys to secure access to the contents of a storage account and there’s no ability to assign granular access permissions at the blob level. If someone has either the primary or secondary access key for a storage account, they essentially have full access to the contents of that storage account including any virtual disks (VHDs) contained within it. The only way to secure the contents of these VHDs is by implementing Azure Disk Encryption (ADE) so that the virtual disks are encrypted at the OS level and copying them through the use of an access key and the Azure Storage REST APIs won’t allow someone to access the contents unless they also have the encryption key used to secure the data.

Enter Azure Managed Disks. With this feature, there is no longer a way to access the VHD blobs via a URI and access to a storage account. Placement of the VHD blobs on Azure storage is managed by the service and Azure Role-Based Access Control can be used to assign specific permissions for a managed disk to one or more users. Managed Disks exposes a variety of operations including read, write, delete and retrieval (export) of a disk via a shared access signature (SAS) URI. For this last part, an administrator can export the managed disk via the Azure Management Portal and they’re receive a URL to use for copying the managed disk to either an Azure storage account or to a location on their local PC.

Copying the VHD via the SAS URI is a bit tricky if you’re trying to use standard tools like Azcopy. To download the VHD to a local PC you can use the hyperlink provided directly in the portal. To copy the VHD to a storage account, though, you can’t use Azcopy. Azcopy requires both READ and LIST SAS permissions to copy a blob. However, the Azure REST SAS API currently only supports read URIs (https://docs.microsoft.com/en-us/rest/api/manageddisks/disks/disks-grant-access). The easiest way I’ve found to complete the copy is to use the Start-AzureStorageBlobCopy cmdlet. This cmdlet starts an asynchronous copy job using the storage REST APIs and works just fine with the read only SAS URI provided by the Azure Portal. Here’s how to accomplish the task:

First, make sure you’re authenticated to Azure and have selected the proper subscription containing the destination storage account for the VHD:

Login-AzureRmAccount

Select-AzureRmSubscription -SubscriptionName <Subscription Name>

Then, get the access key for the destination storage group and create a context to use for the copy job:

$dstkey = Get-AzureRmStorageAccountKey -StorageAccountName <Destination Storage Account Name> -ResourceGroupName <Destination Storage Account Resource Group>

$dstcontext = New-AzureStorageContext -StorageAccountName <Destination Storage Account Name> -StorageAccountKey $dstkey.Value[0]

Lastly, create the copy job and then run the Get-AzureStorageBlobCopyState with the “WaitforComplete” parameter to provide a visual indication of the progress of the copy:

$copyblob = Start-AzureStorageBlobCopy -AbsoluteUri <Export managed disk URL from Portal – Full URL> -DestContext $dstcontext -DestContainer <Destination Container Name> -DestBlob <Destination VHD Filename>

$copyblob | Get-AzureStorageBlobCopyState –WaitForComplete

Azure VM Recreation Script

There are many tasks associated with virtual machines in Azure which can only be performed by deleting the VM configuration while preserving the underlying virtual hard disks and then recreating the VM configuration with the proper settings. Some of the things that come to mind are:

  • Renaming a VM resource
  • Moving a VM from one VNET to another
  • Moving a VM from one Azure region to another
  • Moving a VM into or out of an availability set
  • Changing a single NIC VM to a multi NIC VM and vice versa

There are likely others, I’m sure. However, these are actually pretty basic needs for most folks and it’s a shame it isn’t easier to do it. I’ve gotten these types of requests so many times that I finally decided to build a script as a basis for enabling them in a semi-automated fashion. Attached is sample script which includes the methodology and was designed to move a VM in one region to another region assuming that someone copied the underlying VHDs to the destination using either AzCopy or Start-AzureStorageBlobCopy.

At a high-level, here’s what the script does:

  1. Authenticates to the specified Azure subscription if necessary
  2. Reads a list of disk names and VHD URIs for the underlying disks
  3. Gets the source VM definition using Get-AzureRmVM
  4. Creates a new VM definition in the destination location
  5. Adds OS Disks and Data Disks using the provided disk mappings
  6. Creates NIC(s) for destination VM – prompts for new IPs for static addresses
  7. Deploys new VM

It goes without saying that the source VM should be stopped when the VHDs are copied and the script is run. There’s no need to delete the source VM until the new one is successfully built. This handles availability sets if the original VM was in one and works for VMs with either managed disks or unmanaged disks. There are some limitations as of now:

  • Doesn’t work with VMs encrypted using Azure Disk Encryption (ADE)
  • Doesn’t move Public IP resources
  • Doesn’t move NSGs if those were assigned to the VM’s NIC(s) directly
  • Will only move the first IP configuration for each NIC
  • Doesn’t recreate any tags associated with the source VM
  • Doesn’t redeploy any extensions present on the source VM

The real idea behind this was that it would serve as a starting point. This same script could easily be used to migrate a VM from standard to premium storage with a few tweaks or to redeploy a VM to a different VNET or availability set if desired. Hopefully folks find it useful as a starting point for their own adventures.

Create VM from VHD v3 (PowerShell script)

Azure AppGW and IP Address Usage

I had to do some digging recently to answer a question about the Azure Application Gateway and its consumption of IP addresses from the Application Gateway subnet address space. It’s not well documented and without an understanding of it you might end up painting yourself in a corner when designing the layout of your subnets and address spaces in an Azure Virtual Network.

If you’re not familiar with it, the Azure Application Gateway is a dedicated virtual appliance which provides various HTTP/HTTP (layer 7) load balancing applications and is provided as a multi-instance, fully managed service. It also has some web application firewall (WAF) capabilities and can be configured as an internet facing gateway, internal gateway or a combination of both.

scenario

The first important point to understand is that the Azure Application Gateway must be deployed in its own subnet. The subnet created or used for application gateway cannot contain any other types of resources (VMs, load balancers, etc.) but *can* contain other application gateways. If the subnet in question doesn’t meet this criteria, it won’t appear in the Azure portal as one of the possible subnets for selection in the deployment wizard. Because of this, one might be tempted to create the smallest address space possible for the subnet (a /29 with 3 usable IPs once Azure takes the first three from the range) especially if there’s no plan to deploy Azure Application Gateways with internal addresses. That would be a big mistake and here’s why: each instance of an application gateway consumes an address from the subnet’s address pool.

gwinstances

When you create an application gateway, an endpoint (public VIP or internal ILB IP) is associated and used for ingress network traffic. This VIP or ILB IP is provided by Azure and works at the transport layer (TCP/UDP). The application gateway then routes the HTTP/HTTPS traffic based on its configuration to a backend pool comprised of a virtual machine, cloud service, or an internal or an external IP address.

frontendconfigs

Regardless of whether an application gateway is deployed with a Public IP, Private IP or both each instance of a gateway consumes an address from the subnet’s address pool. If the gateway is then configured with a private ILB IP, it consumes one additional IP address from the subnet’s address pool for that ILB IP. The only IP that will show up in the subnet as being consumed, though, is the ILB IP. You won’t see any of the IPs in use by the gateway instances. The only way you’ll know if you hit the limit is that your deployment (or instance increase) will fail with an error that there’s not enough address space. This is definitely not ideal and hopeful will be improved at some point.

Some examples:

2 instance GW with a Public IP Only = 2 IPs (2 + 0 = 2)

2 instance GW with a Private IP Only = 3 IPs (2 + 1 = 3)

4 instance GW with a Public & Private IP = 5 IPs (4 + 1 = 5)

So keep this in mind when designing the layout of your Azure VNET and its subnets and address spaces. Not having enough addresses in the subnet to expand the number of instances for your production Application Gateway can be a real pain and certainly will require downtime to resolve.