Google Cloud’s Second Region in Australia

Google Cloud Platform (GCP) has extended its reach in Australia and New Zealand (ANZ) with a second region in Melbourne.

Why does this matter?

Having two regions inside Australia allows for customers to extend their architecture for highly available or disaster recoverable solutions. Google now join Azure (who has 3) as having multiple regions inside Australia, no doubt we will keep a close eye on AWS whom were the first public cloud provider to enter Sydney many moons ago.

What’s different about Google’s Regions?

The distinguishing network feature that sets GCP apart from its rivals is how they allow customers to design their Virtual Private Cloud (VPC). GCP allow for subnets in the single VPC to span across as many regions as you’d like. You have the ability to create a single globally distributed VPC with subnets in the Americas, Asia and Europe. Build a logical DMZ zone that has subnets in each region for your globally distributed web services. Unlike the other cloud providers whom their software defined networks are region specific and peering must be setup that incurs bandwidth usage and connection charges. Thoughts go through my head as to how you would build a globally distributed VPC with local on-ramp for your regions on-premises networks. None the less, allowing for your traffic from Asia to Europe to traverse Google’s backhaul could makes life easier. The devil is in the detail, as what is also unique to Google is the VM-VM egress charges with cross region. Which is kind of saying the same thing as peering, just putting the price on a different object. All things to carefully think about when planning your cloud deployments. Maybe in some circumstances based on the pricing model, Google outweighs the other heavy weight hitters.

What will be interesting to see is whether with Sydney and Melbourne onboarding soon, will the VM-VM egress pricing update to support Egress between Google Cloud regions within Australia. Currently if I look at what is written on the tin I’d assume that it falls under Egress between Google Cloud regions within Oceania (per GB) at $0.08, where Oceania includes Australia, New Zealand, and surrounding Pacific Ocean islands such as Papua New Guinea and Fiji. This region excludes Hawaii.

Google have a nice inter-Region Google Cloud Inter-region latency and throughput pivot table that gets metrics form a Perfkit test. The current lowest latent packet is to asia-southeast1 Singapore at ~92ms, we will definitely knock the socks off of that in ANZ with Melbourne to Sydney.

Google Cloud Inter-Region Latency and Throughput › Inter-region latency and throughput

Googles VPC network Example (here)


Anatomy of a Successful Cloud Migration

Free eBook Download

Download Now

“Through 2024, 80% of companies that are unaware of the mistakes made in their cloud adoption will overspend by 20 to 50%.” (4 Lessons Learned From Cloud Infrastructure Adopters – gartner.com)

It’s 2021 and you’re ready to move to the cloud. But first, you’ll want to do your research.

This eBook will walk you through:

  • Challenges faced by individuals within an organisation when migrating to the cloud
  • How to solve these challenges efficiently and effectively with powerful tools
  • How to best align business and IT goals to make sure everything runs smoothly

With years of experience working with Government and Enterprise organisations at a State and Local level, we want to share the lessons we’ve learnt. From procurement, to funding to security challenges, we’ve covered everything you need to know to migrate to the cloud successfully.

Complete the form for instant access to this eBook.


SQL Database Backup on IaaS using Azure Automation

I had a need to take a full SQL Database backup from a virtual machine with SQL Server hosted on Azure. This is done via an Azure Automation account, executing a runbook on a hybrid worker. This is a great way to take a offline copy of your production SQL and store it someplace safe.

To accomplish this we will use the PowerShell module ‘sqlps‘ that should be installed with SQL Server and run the command Backup-SqlDatabase.

Backup-SqlDatabase (SqlServer) | Microsoft Docs

Store SQL Storage Account Credentials

Before we can run the Backup-SqlDatabase command we must have a saved credential stored in SQL for the Storage Account using New-SqlCredential.

New-SqlCredential (SqlServer) | Microsoft Docs

Import-Module sqlps
# set parameters
$sqlPath = "sqlserver:\sql\$($env:COMPUTERNAME)"
$storageAccount = "<storageAccountName>"  
$storageKey = "<storageAccountKey>"  
$secureString = ConvertTo-SecureString $storageKey -AsPlainText -Force  
$credentialName = "azureCredential-"+$storageAccount

Write-Host "Generate credential: " $credentialName
  
#cd to sql server and get instances  
cd $sqlPath
$instances = Get-ChildItem

#loop through instances and create a SQL credential, output any errors
foreach ($instance in $instances)  {
    try {
        $path = "$($sqlPath)\$($instance.DisplayName)\credentials"
        New-SqlCredential -Name $credentialName -Identity $storageAccount -Secret $secureString -Path $path -ea Stop | Out-Null
        Write-Host "...generated credential $($path)\$($credentialName)."  }
    catch { Write-Host $_.Exception.Message } }

Backup SQL Databases with an Azure Runbook

The runbook below works on the DEFAULT instance and excludes both tempdb and model from backup.

Import-Module sqlps
$sqlPath = "sqlserver:\sql\$($env:COMPUTERNAME)"
$storageAccount = "<storageAccount>"  
$blobContainer = "<containerName>"  
$backupUrlContainer = "https://$storageAccount.blob.core.windows.net/$blobContainer/"  
$credentialName = "azureCredential-"+$storageAccount
$prefix = Get-Date -Format yyyyMMdd

Write-Host "Generate credential: " $credentialName

Write-Host "Backup database: " $backupUrlContainer
  
cd $sqlPath
$instances = Get-ChildItem

#loop through instances and backup all databases (excluding tempdb and model)
foreach ($instance in $instances)  {
    $path = "$($sqlPath)\$($instance.DisplayName)\databases"
    $databases = Get-ChildItem -Force -Path $path | Where-object {$_.name -ne "tempdb" -and $_.name -ne "model"}

    foreach ($database in $databases) {
        try {
            $databasePath = "$($path)\$($database.Name)"
            Write-Host "...starting backup: " $databasePath
            $fileName = $prefix+"_"+$($database.Name)+".bak"
            $destinationBakFileName = $fileName
            $backupFileURL = $backupUrlContainer+$destinationBakFileName
            Write-Host "...backup URL: " $backupFileURL
            Backup-SqlDatabase -Database $database.Name -Path $path -BackupFile $backupFileURL -SqlCredential $credentialName -Compression On 
            Write-Host "...backup complete."  }
        catch { Write-Host $_.Exception.Message } } }


NOTE: You will notice a performance hit on the SQL Server so schedule this runbook in a maintanence window.


Deploy Craft CMS with Azure App Service for Linux Containers

Here is some key points to deploy a Craft CMS installation on Azure Web App using container images. In this blog we will step you through some of the modifications needed to make the container image run in Azure and the deployment steps to run in an Azure DevOps Pipeline.

CraftCMS have reference material for their docker deployments found here:
GitHub – craftcms/docker: Craft CMS Docker images

Components

The components required are:

  • Azure Web App for Linux Containers
  • Azure Database for MySQL
  • Azure Storage Account
  • Azure Front Door with WAF
  • Azure Container Registry

Custom Docker Image

To make this work in an Azure Web App we have to do the following additional steps:

  • Install OpenSSH & Enable SSH daemon on 2222 at startup
  • Set the password for root to “Docker!”
  • Install the Azure Database for MySQL root certificates for SSL connections from the Container

We do this in the Dockerfile. We are customizing the NGINX implementation of CraftCMS to allow for the front end to service the HTTP/HTTPS requests from the App Service.

# composer dependencies
FROM composer:1 as vendor
COPY composer.json composer.json
COPY composer.lock composer.lock
RUN composer install --ignore-platform-reqs --no-interaction --prefer-dist

FROM craftcms/nginx:7.4
# Install OpenSSH and set the password for root to "Docker!". In this example, "apk add" is the install instruction for an Alpine Linux-based image.
USER root
RUN apk add openssh sudo \
     && echo "root:Docker!" | chpasswd 
# Copy the sshd_config file to the /etc/ directory
COPY sshd_config /etc/ssh/
COPY start.sh /etc/start.sh
COPY BaltimoreCyberTrustRoot.crt.pem /etc/BaltimoreCyberTrustRoot.crt.pem 
RUN ssh-keygen -A
RUN addgroup sudo
RUN adduser www-data sudo
RUN echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers

# the user is `www-data`, so we copy the files using the user and group
USER www-data
COPY --chown=www-data:www-data --from=vendor /app/vendor/ /app/vendor/
COPY --chown=www-data:www-data . .

EXPOSE 8080 2222
ENTRYPOINT ["sh", "/etc/start.sh"]

The corresponding ‘start.sh’

#!/bin/bash
sudo /usr/sbin/sshd &
/usr/bin/supervisord -c /etc/supervisor/conf.d/supervisor.conf

Build the Web App

The Azure Web App resource is deployed using a ARM template. Here is a snippet of the template, the key is to have your environment variables defined:

{
            "comments": "This is the docker web app running craftcms/custom Docker image",
            "type": "Microsoft.Web/sites",
            "name": "[parameters('siteName')]",
            "apiVersion": "2020-06-01",
            "location": "[parameters('location')]",
            "tags": "[parameters('tags')]",
            "dependsOn": [
                "[variables('hostingPlanName')]",
                "[variables('databaseName')]"
            ],
            "properties": {
                "siteConfig": {
                    "appSettings": [
                        {
                            "name": "DOCKER_REGISTRY_SERVER_URL",
                            "value": "[reference(variables('registryResourceId'), '2019-05-01').loginServer]"
                        },
                        {
                            "name": "DOCKER_REGISTRY_SERVER_USERNAME",
                            "value": "[listCredentials(variables('registryResourceId'), '2019-05-01').username]"
                        },
                        {
                            "name": "DOCKER_REGISTRY_SERVER_PASSWORD",
                            "value": "[listCredentials(variables('registryResourceId'), '2019-05-01').passwords[0].value]"
                        },
                        {
                            "name": "WEBSITES_ENABLE_APP_SERVICE_STORAGE",
                            "value": "false"
                        },
                        {
                            "name": "DB_DRIVER",
                            "value": "mysql"
                        },
                        {
                            "name": "DB_SERVER",
                            "value": "[reference(resourceId('Microsoft.DBforMySQL/servers',variables('serverName'))).fullyQualifiedDomainName]"
                        },
                        {
                            "name": "DB_PORT",
                            "value": "3306"
                        },
                        {
                            "name": "DB_DATABASE",
                            "value": "[variables('databaseName')]"
                        },
                        {
                            "name": "DB_USER",
                            "value": "[variables('databaseUserName')]"
                        },
                        {
                            "name": "DB_PASSWORD",
                            "value": "[parameters('administratorLoginPassword')]"
                        },
                        {
                            "name": "DB_SCHEMA",
                            "value": "public"
                        },
                        {
                            "name": "DB_TABLE_PREFIX",
                            "value": ""
                        },
                        {
                            "name": "SECURITY_KEY",
                            "value": "[parameters('cmsSecurityKey')]"
                        },
                        {
                            "name": "WEB_IMAGE",
                            "value": "[parameters('containerImage')]"
                        },
                        {
                            "name": "WEB_IMAGE_PORTS",
                            "value": "80:8080"
                        }

                    ],
                    "linuxFxVersion": "[variables('linuxFxVersion')]",
                    "scmIpSecurityRestrictions": [
                        
                    ],
                    "scmIpSecurityRestrictionsUseMain": false,
                    "minTlsVersion": "1.2",
                    "scmMinTlsVersion": "1.0"
                },
                "name": "[parameters('siteName')]",
                "serverFarmId": "[variables('hostingPlanName')]",
                "httpsOnly": true      
            },
            "resources": [
                {
                    "apiVersion": "2020-06-01",
                    "name": "connectionstrings",
                    "type": "config",
                    "dependsOn": [
                        "[resourceId('Microsoft.Web/sites/', parameters('siteName'))]"
                    ],
                    "tags": "[parameters('tags')]",
                    "properties": {
                        "dbstring": {
                            "value": "[concat('Database=', variables('databaseName'), ';Data Source=', reference(resourceId('Microsoft.DBforMySQL/servers',variables('serverName'))).fullyQualifiedDomainName, ';User Id=', parameters('administratorLogin'),'@', variables('serverName'),';Password=', parameters('administratorLoginPassword'))]",
                            "type": "MySQL"
                        }
                    }
                }
            ]
        },

All other resources should be ARM defaults. No customisation required. Either put them all in a single ARM template or seperate them out on their own. Your choice to be creative.

Build Pipeline

The infrastructure build pipeline looks something like the below:

# Infrastructure pipeline
trigger: none

pool:
  vmImage: 'windows-2019'
variables:
  TEMPLATEURI: 'https://storageAccountName.blob.core.windows.net/templates/portal/'
  CMSSINGLE: 'singleCraftCMSTemplate.json'
  CMSSINGLEPARAM: 'singleCraftCMSTemplate.parameters.json'
  CMSFILEREG: 'ContainerRegistry.json'
  CMSFRONTDOOR: 'frontDoor.json'
  CMSFILEREGPARAM: 'ContainerRegistry.parameters.json'
  CMSFRONTDOORPARAM: 'frontDoor.parameters.json'
  LOCATION: 'Australia East'
  SUBSCRIPTIONID: ''
  AZURECLISPID: ''
  TENANTID: ''
  RGNAME: ''
  TOKEN: ''
  ACS : 'registryName.azurecr.io'
resources:
  repositories:
    - repository: coderepo
      type: git
      name: Project/craftcms
stages:
- stage: BuildContainerRegistry
  displayName: BuildRegistry
  jobs:
  - job: BuildContainerRegistry
    displayName: Azure Git Repository
    pool:
      vmImage: 'windows-latest'
    steps:
    - task: CopyFiles@2
      name: copyToBuildHost
      displayName: 'Copy files to the build host for execution'
      inputs:
        Contents: '**'
        TargetFolder: '$(Build.ArtifactStagingDirectory)'
    - task: AzureFileCopy@4
      inputs:
        SourcePath: '$(Build.Repository.LocalPath)\CMS\template\*'
        azureSubscription: ''
        Destination: 'AzureBlob'
        storage: ''
        ContainerName: 'templates'
        BlobPrefix: portal
        AdditionalArgumentsForBlobCopy: --recursive=true
    - task: AzureResourceManagerTemplateDeployment@3
      displayName: "Deploy Azure ARM template for Azure Container Registry"
      inputs:
        deploymentScope: 'Resource Group'
        azureResourceManagerConnection: 'azureDeployCLI-SP'
        subscriptionId: '$(SUBSCRIPTIONID)'
        action: 'Create Or Update Resource Group'
        resourceGroupName: '$(RGNAME)'
        location: '$(LOCATION)'
        templateLocation: 'URL of the file'
        csmFileLink: '$(TEMPLATEURI)$(CMSFILEREG)$(TOKEN)' 
        csmParametersFileLink: '$(TEMPLATEURI)$(CMSFILEREGPARAM)$(TOKEN)' 
        deploymentMode: 'Incremental'
    - task: AzurePowerShell@5
      displayName: 'Import the public docker images to the Azure Container Repository'
      inputs:
        azureSubscription: 'azureDeployCLI-SP'
        ScriptType: 'FilePath'
        ScriptPath: '$(Build.ArtifactStagingDirectory)\CMS\template\dockerImages.ps1'
        errorActionPreference: 'silentlyContinue'
        azurePowerShellVersion: 'LatestVersion'

- stage: BuildGeneralImg
  dependsOn: BuildContainerRegistry
  displayName: BuildImages
  jobs:
  - job: BuildCraftCMSImage
    displayName: General Docker Image
    pool:
      vmImage: 'ubuntu-18.04'
    steps:
    - checkout: self
    - checkout: coderepo
    - task: CopyFiles@2
      name: copyToBuildHost
      displayName: 'Copy files to the build host for execution'
      inputs:
        Contents: '**'
        TargetFolder: '$(Build.ArtifactStagingDirectory)'
    - task: Docker@2
      displayName: Build and push
      inputs:
        containerRegistry: ''
        repository: craftcms
        command: buildAndPush
        dockerfile: 'craftcms/Dockerfile'
        tags: |
          craftcms
          latest

- stage: Deploy 
  dependsOn: BuildGeneralImg
  displayName: DeployWebService
  jobs:
  - job:
    displayName: ARM Templates
    pool:
      vmImage: 'windows-latest'
    steps:
    - checkout: self
    - checkout: coderepo
    - task: CopyFiles@2
      name: copyToBuildHost
      displayName: 'Copy files to the build host for execution'
      inputs:
        Contents: '**'
        TargetFolder: '$(Build.ArtifactStagingDirectory)'
    
    - task: AzureResourceManagerTemplateDeployment@3
      displayName: "Deploy Azure ARM single template for remaining assets"
      inputs:
        deploymentScope: 'Resource Group'
        azureResourceManagerConnection: ''
        subscriptionId: '$(SUBSCRIPTIONID)'
        action: 'Create Or Update Resource Group'
        resourceGroupName: '$(RGNAME)'
        location: '$(LOCATION)'
        templateLocation: 'URL of the file'
        csmFileLink: '$(TEMPLATEURI)$(CMSSINGLE)$(TOKEN)' 
        csmParametersFileLink: '$(TEMPLATEURI)$(CMSSINGLEPARAM)$(TOKEN)' 
        deploymentMode: 'Incremental'

- stage: Secure 
  dependsOn: Deploy
  displayName: DeployFrontDoor
  jobs:
  - job:
    displayName: ARM Templates
    pool:
      vmImage: 'windows-latest'
    steps:
    - task: CopyFiles@2
      name: copyToBuildHost
      displayName: 'Copy files to the build host for execution'
      inputs:
        Contents: '**'
        TargetFolder: '$(Build.ArtifactStagingDirectory)'
    
    - task: AzureResourceManagerTemplateDeployment@3
      displayName: "Deploy Azure ARM single template for Front Door"
      inputs:
        deploymentScope: 'Resource Group'
        azureResourceManagerConnection: ''
        subscriptionId: '$(SUBSCRIPTIONID)'
        action: 'Create Or Update Resource Group'
        resourceGroupName: '$(RGNAME)'
        location: '$(LOCATION)'
        templateLocation: 'URL of the file'
        csmFileLink: '$(TEMPLATEURI)$(CMSFRONTDOOR)$(TOKEN)' 
        csmParametersFileLink: '$(TEMPLATEURI)$(CMSFRONTDOORPARAM)$(TOKEN)' 
        deploymentMode: 'Incremental'
    - task: AzurePowerShell@5
      displayName: 'Apply Front Door service tags to Web App ACLs'
      inputs:
        azureSubscription: 'azureDeployCLI-SP'
        ScriptType: 'FilePath'
        ScriptPath: '$(Build.ArtifactStagingDirectory)\CMS\template\enableFrontDoorOnWebApp.ps1'
        errorActionPreference: 'silentlyContinue'
        azurePowerShellVersion: 'LatestVersion'    

Enable Front Door with WAF

The pipeline stage DeployFrontDoor has an enableFrontDoorOnWebApp.ps1

$azFrontDoorName = ""
$webAppName = ""
$resourceGroup = ""

Write-Host "INFO: Restrict access to a specific Azure Front Door instance"
try{
    $afd = Get-AzFrontDoor -Name $azFrontDoorName -ResourceGroupName $resourceGroup
}
catch{
    Write-Host "ERROR: $($_.Exception.Message)"
}

Write-Host "INFO: Setting the IP ranges defined in the AzureFrontDoor.Backend service tag to the Web App"
try{
    Add-AzWebAppAccessRestrictionRule -ResourceGroupName $resourceGroup -WebAppName $webAppName -Name "Front Door Restrictions" -Priority 100 -Action Allow -ServiceTag AzureFrontDoor.Backend -HttpHeader @{'x-azure-fdid' = $afd.FrontDoorId}}
catch{
    Write-Host "ERROR: $($_.Exception.Message)"
}


You should now have a CraftCMS web app that is only available through the FrontDoor URL.

Continuous Deployment

There are many ways to deploy updates to your website, an Azure Web App has a beautiful thing called slots that can be used.

# Trigger on commit
# Build and push an image to Azure Container Registry
# Update Web App Slot

trigger:
  branches:
    include:
      - main
  paths:
    exclude:
      - pipelines
      - README.md
  batch: true

resources:
- repo: self

pool:
  vmImage: 'windows-2019'
variables:
  TEMPLATEURI: 'https://storageAccountName.blob.core.windows.net/templates/portal/'
  LOCATION: 'Australia East'
  SUBSCRIPTIONID: ''
  RGNAME: ''
  TOKEN: ''
  SASTOKEN: ''
  TAG: '$(Build.BuildId)'
  CONTAINERREGISTRY: 'registryName.azurecr.io'
  IMAGEREPOSITORY: 'craftcms'
  APPNAME: ''

stages:
- stage: BuildImg
  displayName: BuildLatestImage
  jobs:
  - job: BuildCraftCMSImage
    displayName: General Docker Image
    pool:
      vmImage: 'ubuntu-18.04'
    steps:
    - checkout: self
    - task: CopyFiles@2
      name: copyToBuildHost
      displayName: 'Copy files to the build host for execution'
      inputs:
        Contents: '**'
        TargetFolder: '$(Build.ArtifactStagingDirectory)'
    - task: Docker@2
      displayName: Build and push
      inputs:
        containerRegistry: ''
        repository: $(IMAGEREPOSITORY)
        command: buildAndPush
        dockerfile: 'Dockerfile'
        tags: |
          $(IMAGEREPOSITORY)
          $(TAG)


- stage: UpdateApp 
  dependsOn: BuildImg
  displayName: UpdateTestSlot
  jobs:
  - job:
    displayName: 'Update Web App Slot'
    pool:
      vmImage: 'windows-latest'
    steps:
    - task: AzureWebAppContainer@1
      displayName: 'Update Web App Container Image Reference' 
      inputs:
        azureSubscription: ''
        appName: $(APPNAME)
        containers: $(CONTAINERREGISTRY)/$(IMAGEREPOSITORY):$(TAG)
        deployToSlotOrASE: true
        resourceGroupName: $(RGNAME)
        slotName: test




Fault Tolerant Multi AZ EC2, On a beer budget – live from AWS Meetup

Filmed on 18th of March 2021 at the Adelaide AWS User Group, where Arran Peterson presented on how to put together best practice (and cheap!) cloud architecture for business continuity. The title:

“Enterprise grade fault tolerant multi-AZ server deployment on a beer budget”

Recording

RATED ‘PG’ – Mild course language.

Presenter

Arran Peterson
Arran Peterson

Arran is an Infrastructure Consultant with a passion for Microsoft Unified Communications and the true flexibility and scalability of cloud-based solutions.
As a Senior Consultant, Arran brings his expertise in enterprise environments to work with clients around Microsoft Unified Communications product portfolio of Office 365, Exchange and Skype/Teams, along with expertise around transitioning to the cloud-based platforms including AWS, Azure and Google.

More Reading

Amazon Elastic Block Store

https://aws.amazon.com/ebs/

AWS Sydney outage prompts architecture rethink

https://www.itnews.com.au/news/aws-sydney-outage-prompts-architecture-rethink-420506

Chalice Framework

https://aws.github.io/chalice/

Adelaide AWS User Group

https://www.meetup.com/en-AU/Amazon-Web-Services-User-Group-Adelaide/events/276728885/


Azure Migrate – Additional Firewall Rules

When deploying Azure Migrate Appliances to discovery servers, the appliance needs outbound internet access. In many IT environments, servers are disallowed internet access unless prescribed to certain URL sets. Gratefully Microsoft have given us a list of what they think is the list of URLs that the appliance will need to have whitelisted. This can be found here:

https://docs.microsoft.com/en-us/azure/migrate/migrate-appliance#public-cloud-urls

Issue

Once the appliance has booted up and your onto the GUI, you must ener your Azure Migrate Project key from your subscription and then authenticate to your subscription. We entailed the following error when attempting to resolve the initial key:

Azure Migrate Error

Failed to connect to the Azure Migrate project. Check the errors details, follow the remediation steps and click on ‘Retry’ button

The Azure Migrate Key doesn’t have an expiration on it so this wasn’t the issue. We had whitelisted the URL‘s but on the firewall we were seeing dropped packets:

13:40:41Default DROPTCP10.0.0.10:50860204.79.197.219:443
13:40:41Default DROPTCP10.0.0.10:50861204.79.197.219:80
13:40:41Default DROPTCP10.0.0.10:50857152.199.39.242:443
13:40:42Default DROPTCP10.0.0.10:50862204.79.197.219:80
13:40:42Default DROPTCP10.0.0.10:50858104.74.50.201:80
13:40:43Default DROPTCP10.0.0.10:5086352.152.110.14:443
13:40:44Default DROPTCP10.0.0.10:50860204.79.197.219:443
13:40:44Default DROPTCP10.0.0.10:50861204.79.197.219:80
13:40:45Default DROPTCP10.0.0.10:50862204.79.197.219:80
13:40:46Default DROPTCP10.0.0.10:5086352.152.110.14:443
13:40:46Default DROPTCP10.0.0.10:50859204.79.197.219:443
13:40:47Default DROPTCP10.0.0.10:5086440.90.189.152:443
13:40:47Default DROPTCP10.0.0.10:5086552.114.36.3:443
13:40:49Default DROPTCP10.0.0.10:5086440.90.189.152:443
13:40:50Default DROPTCP10.0.0.10:5086552.114.36.3:443
13:40:50Default DROPTCP10.0.0.10:50860204.79.197.219:443
13:40:50Default DROPTCP10.0.0.10:50861204.79.197.219:80
13:40:51Default DROPTCP10.0.0.10:50862204.79.197.219:80
13:40:52Default DROPTCP10.0.0.10:5086352.152.110.14:443
Subset of the dropped packets based on IP destination during connection failure

Reviewing the SSL certificates on these IP addresses, they are all Microsoft services with multiple SAN entries. We also had a look at the traffic from the developer tools in the browser:

We can see that the browser is trying to start a AAD workflow for device login, which is articulated in the onboarding documentation. Our issue was that the JavaScript for inside the browser session wasn’t located in the whitelist URLs. Reviewing the SAN entries in the certificates presented in the IP destination table we looked for ‘CDN’ or ‘Edge’ URLs.

The fix

The following URLs were added to the whitelist group for the appliance and problems went away.

204.79.197.219*.azureedge.net
152.199.39.242*.azureedge.net
152.199.39.242*.wpc.azureedge.net
152.199.39.242*.wac.azureedge.net
152.199.39.242*.adn.azureedge.net
152.199.39.242*.fms.azureedge.net
152.199.39.242*.azureedge-test.net
152.199.39.242*.ec.azureedge.net
152.199.39.242*.wpc.ec.azureedge.net
152.199.39.242*.wac.ec.azureedge.net
152.199.39.242*.adn.ec.azureedge.net
152.199.39.242*.fms.ec.azureedge.net
152.199.39.242*.aspnetcdn.com
152.199.39.242*.azurecomcdn.net
152.199.39.242cdnads.msads.net