Tagging EC2 EBS Volumes in Auto Scaling Groups

Tagging becomes a huge part of your life when in the public cloud. Metadata is thrown around like hotcakes, and why not. At cloudstep.io we preach the ways of the DevOps gods and especially infrastructure as code for repeatable and standardised deployments. This way everything is uniform and everything gets a TAG!

I ran into an issue recently where I would build an EC2 instance and capture the operating system into an AMI as part of a CloudFormation stack. This AMI would then be used as part of a launch configuration and subsequent auto scaling group. The original EC2 instance had every tag needed across all parts that make up the virtual machine including:

  • EBS root volume
  • EBS data volumes
  • Elastic Network Interfaces (ENI)
  • EC2 Instance itself

When deploying my auto scaling group all the user level tags I’d applied had been removed from the volumes and ENI. This caused a few issues:

  1. EBS volumes couldn’t be tagged for billing.
  2. EBS volumes couldn’t be snapped based on tag level policies in Lifecycle Manager.
  3. Objects didn’t have a ‘Name’ tag which made it hard in the console to understand which virtual machine instance the object belonged too.

There are two methods I derived to add my tags back that I’ll share with you. The tags needed to be added upon launch of the instance when the auto scaling group added a server. The methods I used were:

  1. The auto scaling group has a Launch Configuration where the ‘User data’ field runs a script block at startup.
  2. Initiate a Lambda whenever CloudTrail logged an API reference of a launch event of an instance using CloudWatch.

Tagging with the User Data property and PowerShell

User data is simply:

When you launch an instance in Amazon EC2, you have the option of passing user data to the instance that can be used to perform common automated configuration tasks and even run scripts after the instance starts. You can pass two types of user data to Amazon EC2: shell scripts and cloud-init directives.

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html

Try {
 # Use the metadata service to discover which instance the script is running on
 $InstanceId = (Invoke-WebRequest '169.254.169.254/latest/meta-data/instance-id').Content
 $AvailabilityZone = (Invoke-WebRequest '169.254.169.254/latest/meta-data/placement/availability-zone').Content
 $Region = $AvailabilityZone.Substring(0, $AvailabilityZone.Length -1)
 $mac = (Invoke-WebRequest '169.254.169.254/latest/meta-data/network/interfaces/macs/').content
 $URL = "169.254.169.254/latest/meta-data/network/interfaces/macs/"+$mac+"/interface-id"
 $eni = (Invoke-WebRequest $URL).content
# Get the list of volumes attached to this instance
 $BlockDeviceMappings = (Get-EC2Instance -Region $Region -Instance $InstanceId).Instances.BlockDeviceMappings
 $Tags = (Get-EC2Instance -Region $Region -Instance $InstanceId).Instances.tag

 }
Catch{Write-Host "Could not access the AWS API, are your credentials loaded?" -ForegroundColor Yellow}
$BlockDeviceMappings | ForEach-Object -Process {
        $volumeid = $_.ebs.volumeid # Retrieve current volume id for this BDM in the current instance
        # Set the current volume's tags
        $Tags | ForEach-Object -Process {
        If($_.Key -notlike "aws:*"){
        New-EC2Tag -Resources $volumeid -Tags @{ Key = $_.Key ; Value = $_.Value } # Add tag to volume
        }
        }
}
# Set the current nics tag
$Tags | ForEach-Object -Process {
  If($_.Key -notlike "aws:*"){
        New-EC2Tag -Resources $eni -Tags @{ Key = $_.Key ; Value = $_.Value } # Add tag to eni
  }
}


This script block is great and works a treat with newly created instances from an Amazon Marketplace AMI’s e.g. a vanilla Windows Server 2019 template. The launch configuration would apply the script as a part of the cfn-init function at startup. Unfortunately I’d already used the cfn-init function as part of the original image customisation and capture, the cfn-init would not re-run and didn’t execute this script block. So back to the drawing board in my scenario.

Tagging with CloudWatch and Lambda Function

The second solution was to create a Lambda function and trigger it using an Amazon CloudWatch Events rule. The Instance ID is parsed from the CloudWatch event in JSON to the Lambda function.

Here is the Lambda function that is written in python2.7 and leverages the boto3 and JSON modules.



from __future__ import print_function
import json
import boto3
def lambda_handler(event, context):
  print('Received event: ' + json.dumps(event, indent=2))
  ids = []
  try:
      ec2 = boto3.resource('ec2')
      items = event['detail']['responseElements']['instancesSet']['items']
      for item in items:
        ids.append(item['instanceId'])
      base = ec2.instances.filter(InstanceIds=ids)
      for instance in base:
        ec2tags = instance.tags
        tags = [n for n in ec2tags if not n["Key"].startswith("aws:") ]
        print('   original tags:', ec2tags)
        print('   applying tags:', tags)
        for volume in instance.volumes.all():
          print('    volume:', volume)
          if volume.tags != ec2tags:
            volume.create_tags(DryRun=False, Tags=tags)
        for eni in instance.network_interfaces:
          print('    eni:', eni)
          eni.create_tags(DryRun=False, Tags=tags)
      return True
  except Exception as e:
    print('Something went wrong: ' + str(e))
    return False   



AWS ECS CloudFormation Fails – Unable to assume the service linked role.

I ran into an interesting issue when building a new ECS Cluster using CloudFormation. The CloudFormation stack would fail on Type: AWS::ECS::Service with error:

Unable to assume the service linked role. Please verify that the ECS service linked role exists. (Service: AmazonECS; Status Code: 400; Error Code: InvalidParameterException; Request ID: beadf3d5-3406-11e9-828d-b16cd52796ef)

Okay google, what’s this service linked role thingy?

A service-linked role is a unique type of IAM role that is linked directly to Amazon ECS. Service-linked roles are predefined by Amazon ECS and include all the permissions that the service requires to call other AWS services on your behalf.

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using-service-linked-roles.html

The first few times I ran my stack I assumed that this was for an IAM role that I was needing to assign to the AWS::ECS::Service to perform tasks much like a IamInstanceProfile of Type: AWS::EC2::Instance. When reviewing the available properties for Type: AWS::ECS::Service there was a Role definition:

  • Cluster
  • DeploymentConfiguration
  • DesiredCount
  • HealthCheckGracePeriodSeconds
  • LaunchType
  • LoadBalancers
  • NetworkConfiguration
  • PlacementConstraints
  • PlacementStrategies
  • PlatformVersion
  • Role
  • SchedulingStrategy
  • ServiceName
  • ServiceRegistries
  • TaskDefinition
Role - The name or ARN of an AWS Identity and Access Management (IAM) role that allows your Amazon ECS container agent to make calls to your load balancer.

I had some well defined Type: AWS::IAM::Role objects in my YAML for ECS execution and task roles but none of them were helping me with service linked account issue no matter how far I took the IAM policies.

Solution

To cut a long story and much googling short, the issue was nothing to do with my IAM policies but rather that the very first ECS cluster you create in the console using the getting started wizard creates the linked account in the backend. If your unlike me and read the full article about service linked roles you would have read:

when you create a new cluster (for example, with the Amazon ECS first run, the cluster creation wizard, or the AWS CLI or SDKs), or create or update a service in the AWS Management Console, Amazon ECS creates the service-linked role for you, if it does not already exist.

No mention in the above statement about CloudFormation. As per usual I jumped straight into a CloudFormation template without a test drive of the service and this time my attempt at being clever had given me a few moments of madness.

The easiest fix is to open up AWS CLI and run the following against your account once, then jump back into CloudFormation for YAML fun:

aws iam create-service-linked-role --aws-service-name ecs.amazonaws.com  

Resulting output:

{
    "Role": {
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17", 
            "Statement": [
                {
                    "Action": [
                        "sts:AssumeRole"
                    ], 
                    "Effect": "Allow", 
                    "Principal": {
                        "Service": [
                            "ecs.amazonaws.com"
                        ]
                    }
                }
            ]
        }, 
        "RoleId": "AROAIXGB2WBYGCXSPXY4O", 
        "CreateDate": "2019-02-19T05:55:58Z", 
        "RoleName": "AWSServiceRoleForECS", 
        "Path": "/aws-service-role/ecs.amazonaws.com/", 
        "Arn": "arn:aws:iam::112233445566:role/aws-service-role/ecs.amazonaws.com/AWSServiceRoleForECS"
    }
}

Job done. It all seemed so simple in retrospect.


Add VC Accounts to Microsoft Teams Channels with Azure Automation

At cloudstep.io® HQ Microsoft Teams is a big part of how we organise digital asset structure in the business. We are a consulting firm by trade, as new prospects become paying customers we add them as a team. The default General channel is used for administration and accounts, additional channels are created per project name or scope of works. We find ourselves no longer needing to going into dark corners of SharePoint administration (commonly referred to in the office as ‘SwearPoint!’). We have adopted Microsoft Teams as our preferred web, audio and video conferencing platform for internal and external meetings. Our board room video conferencing unit runs a full version of Windows 10 and Microsoft Teams that we setup as a ‘do it yourself‘ versus the off the shelf room systems. The VC unit requirements we had were:

  • cloudstep.io®, our web application uses a full desktop browser experience.
  • Mouse and keyboard are preferred for web navigation inside the app.
  • VC to have full OS is preferred to eliminate employees having to BYOD and connect either physically or wirelessly for screen presentation.
  • We can connect to third party conferencing platforms by installing the addons for guest access (zoom, webex, gotomeeting, chime etc) with our partner lead meetings direct onto the machine.
  • Wirelessly present our Macs, iPads, iPhones, Androids and Windows laptops.
  • We are all ‘power users‘ and can handle the meeting join experience in Microsoft Teams client without the need for a single ‘click-to-join’ button on the table which the Microsoft Teams Room (MTR) system provides via a touch device.

We have a boardroom account that has a 365 license to be able to leverage the desktop tools. Windows 10 automatically logs in each morning and the Microsoft Teams client is started automatically. The bill of materials is notably:

  • Intel NUC
  • Windows 10
    • Teams Client
    • Office 365 Pro Plus (Word, Excel, PowerPoint, OneNote)
    • Windows 10 Calendar (Connect to Office 365 Mailbox)
    • AirServer client (ChromeCast, MiraCast, AirPlay)
    • Chrome Browser
  • Office 365 user license
  • Logitech Meetup camera
  • Biggest screen we could fit in the room
  • Microsoft Bluetooth keyboard and mouse

The VC mailbox type is set to ‘room‘ with the following to enhance the experience for scheduled meetings in the board room:


#Add tips when searching in Outlook
Set-Mailbox -Identity $VC -MailTip "This room is equipped to support native Teams & Skype for Business Meetings remember to add meeting invite details via the Teams outlook button in the ribbon." 

#Auto Accept
Set-CalendarProcessing -Identity $VC -AutomateProcessing AutoAccept -AddOrganizerToSubject $false -RemovePrivateProperty $false -DeleteComments $false -DeleteSubject $false –AddAdditionalResponse $true –AdditionalResponse "Your meeting is now scheduled and if it was enabled as a Teams Meeting will be available from the conference room client."

This has worked well in the last 12 months, the only user experience problem we have had is when running a meeting from the VC unit, the account isn’t a member of the team where the data attempting to be presented is stored and therefor cannot see/open the content. A simple solution for this is automation. We looked to investigated two automation solutions available in the Microsoft services offering we have available.

  1. Flow (Office 365 Suite)
  2. Azure Automation (Azure Subscription)

Unfortunately option 1 didn’t have any native integration for triggers based on Office 365 groups or teams creation. So we resorted to a quick Azure Powershell Runbook that executes on a simple schedule. The steps needed to run were:

  1. Get a list of all the teams.
  2. Query them against the UnifiedGroup properties to get…
    1. AccessType equals ‘Public
    2. CreationDate within 2 days
  3. Check the newly created teams group membership for the VC unit username.
  4. If it doesn’t exist add the VC unit as the role ‘member‘.
Write-verbose "Getting Credentials ..." -Verbose
$Credentials = Get-AutomationPSCredential -Name 'Admin-365'
Write-verbose  "Credential Imported : $($Credentials.UserName)" -Verbose

$o365Cred = New-Object System.Management.Automation.PSCredential ($Credentials.UserName, $Credentials.Password)
Write-verbose  "Credential Loaded : $($o365Cred.UserName)" -Verbose
Write-verbose 'Connecting to 365 ...' -Verbose
$Session = New-PSSession –ConfigurationName Microsoft.Exchange -ConnectionUri https://outlook.office365.com/powershell-liveid/ -Credential $Credentials -Authentication Basic -AllowRedirection
Write-verbose 'Importing UnifiedGroups PowerShell Commands ...' -Verbose
Import-PSSession -Session $Session -DisableNameChecking:$true -AllowClobber:$true | Out-Null
Write-verbose 'Connecting to Teams ...' -Verbose
Connect-MicrosoftTeams -Credential $Credentials

$creationdate = ((Get-Date).AddDays(-2))
$teams = get-team
#$groups = Get-UnifiedGroup |Where-Object {$_.WelcomeMessageEnabled -like "False" -and $_.AccessType -like "Public" -and $_.WhenCreated -ge $creationdate}
$TeamsOutput = @()
foreach ($Team in $Teams){
$UnifiedGroup = Get-UnifiedGroup -Identity $Team.GroupId
    if($UnifiedGroup.AccessType -like "Public" -and $UnifiedGroup.WhenCreated -ge $creationdate){
    Write-verbose "Processing team named: $($UnifiedGroup.DisplayName)" -Verbose
        $VC = Get-TeamUser -GroupId $Team.GroupId | Where-Object {$_.User -like "user@domain.com"} 
        If($VC.count -eq 0){
            Write-verbose "VC not member, adding..." -Verbose
            Add-TeamUser -GroupId $Team.GroupId -User "user@domain.com" -Role Member
        }else{Write-verbose "VC is member already" -Verbose}
    }

$TeamsOutput+=$UnifiedGroup
}
Write-verbose "Total teams processed for selection: $($TeamsOutput.Count)" -Verbose 

The result is simple

Additional member added via PowerShell

Next day the board room account is logged in, the Microsoft Teams client will have access to all the teams channels, files, OneNote and apps. This is great for native Teams meetings, but also when we have customers in the board room without the need for an online meeting. The VC account has access to see the required teams and channel data to present to the physical display.

This solution doesn’t have to be for a video conferencing units, you may have some standardised members you want on all groups, or it could be certain owner enforcement or member list.

Hello Microsoft Teams! Bye bye SwearPoint, may you remain in the background forever.


Warm AWS WorkSpaces On a Schedule

AWS WorkSpaces VDI solution has two pricing options that you need to choose between for your implementation.

  1. Monthly
  2. Hourly (On demand)

In my opinion it is always worth attempting to run your WorkSpaces VDI deployment in on-demand where there is chance of cost savings when the virtual desktops can be turned off and you will not be charged.

With hourly billing you pay a small fixed monthly fee per WorkSpace to cover infrastructure costs and storage, and a low hourly rate for each hour the WorkSpace is used during the month. Hourly billing works best when Amazon WorkSpaces are used, on average, for less than a full working day or for just a few days a month, making it ideal for part-time workers, job sharing, road warriors, short-term projects, corporate training, and education.

https://aws.amazon.com/workspaces/pricing/

Turning off the VDI is done by AWS using a setting called Running Mode per VDI:

Always on – Billed monthly. Instant access to an always running WorkSpaces

AutoStop – Billed by the hour. WorkSpaces start automatically when you login, and stop when no longer being used (1-48hrs).

In my opinion a turn off period of 1 hour is too short, it doesn’t cover a user who has a long lunch or meeting that runs slightly over. 2 hours cool down period seems to be perfect for cost optimisation. With this in mind, all your VDI’s will be off at the beginning of your working day. To eliminate the need for the 60-90 second boot up time imposed by AWS for cold starts we can pre-warm the VDI’s using Lambda function on a schedule. The process will be as follows:

  1. CloudWatch Event that runs based on a CRON schedule.
  2. Event triggers the execution of a Lambda function
  3. The Lambda function runs python that starts WorkSpaces based on a set of conditions using the boto3 library to interact with the service.

The python code block below wakes all VDI’s in a region that are in a ‘STOPPED’ state but there is no reason why you couldn’t be more granular with tagging per VDI.


import boto3

def lambda_handler(event, context):
    directory_id= ''
    region = 'ap-southeast-2'
    running_mode = 'AVAILABLE'
    # Event
    session = boto3.session.Session(
        aws_access_key_id='',
        aws_secret_access_key=''
    )
    
    ws = session.client('workspaces')
    workspaces = []
    
    resp = ws.describe_workspaces(DirectoryId=directory_id)
    
    while resp:
      workspaces += resp['Workspaces']
      resp = ws.describe_workspaces(DirectoryId=directory_id, NextToken=resp['NextToken']) if 'NextToken' in resp else None
    
    for workspace in workspaces:
    
      if workspace['State'] == running_mode:
        continue
    
      if workspace['State'] in ['STOPPED']:
    
        ws.start_workspaces(
          StartWorkspaceRequests=[
            {
                'WorkspaceId': workspace['WorkspaceId'],
            },
          ]
        )
      
        print 'Starting WorkSpace for user: ' + workspace['UserName']
    
      else:
        print 'Could not start workspace for user: ' + workspace['UserName']


To start the Workspaces on a schedule, Lambda can invoke using a CRON expression:

cron(0 22 ? * SUN-THU *)

The cron schedule runs in GMT, so in this case 10:00 PM in GMT is 8:00 AM in AEST for following day (GMT +10:00).

The end result is the WorkSpaces you have chosen to wake up would start at 8am and shutdown again at 10am if not used. If you had departments or user groups that are heavy users versus sometimes users this might be where your code looks at some tags you’ve set per VDI.


Backup Palo Alto VM Series Config with Azure Automation

If you have implemented a VM-Series firewall in Azure, AWS or on-premises but don’t have a Panorama Server for your configuration backups. Here is a solutions for getting the firewall configuration into an Azure Blob Storage, this could be done similarly with Lambda and S3 using python and the boto3 library.

Why Do This?

If there are multiple administrators of the firewall and configuration changes are happening frequently you may want a daily/hourly backup of the configuration to restore in the event that a recent commit has caused unwanted disruption to your network.

Azure Automation is a great place to start, we will have to interact with the API interface of the firewall to ask for a copy of the XML. Generally speaking we don’t want to expose the API interface to the internet, nor is it easy to allow on the Azure Automation public IPs, so in this case a Hybrid Worker (VM inside your trusted network) can execute the code against the internal trusted interface that has the API listening.

Depending on your version of PowerShell and Invoke-WebRequest you may not be able to ignore a certificate error coming from the API interface. Which is why I’m updating system .Net class for X509 certificates policies.

The steps are pretty simple

  1. Create a directory on the file system (I’m using the Azure VM with temporary D drive local storage)
  2. Request the XML from the URL
  3. Login to Azure with service credentials
  4. Map to the cold storage account i’m putting the files in
  5. Copy the file

add-type @"     using System.Net;     using System.Security.Cryptography.X509Certificates;     public class TrustAllCertsPolicy : ICertificatePolicy {         public bool CheckValidationResult(             ServicePoint srvPoint, X509Certificate certificate,             WebRequest request, int certificateProblem) {             return true;         }     } "@ [System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Ssl3, [Net.SecurityProtocolType]::Tls, [Net.SecurityProtocolType]::Tls11, [Net.SecurityProtocolType]::Tls12 
$todaydate = Get-Date -Format yy-MM-dd
$File = "PaloConfig-"+$todaydate+".xml"  
$FilePath = "D:\Palo\"+$File 
#Create Directory 
New-Item -ItemType directory -Path D:\Palo -Force 
#Download Config
Invoke-WebRequest -Uri "https://PaloIPAddress/api/?type=export&amp;category=configuration&amp;key=<ONETIMEKEY&gt;=" -OutFile $FilePath 
#Login with service principal account 
$TenantId = 'AzureTenantID' 
$ApplicationId = 'ServiceID' 
$Thumbprint = (Get-ChildItem cert:\LocalMachine\My\ | Where-Object {$_.Subject -match "CN=AzureHybridWorker" }).Thumbprint 
Connect-AzureRMAccount -ServicePrincipal -TenantId $TenantId -ApplicationId $ApplicationId -CertificateThumbprint $Thumbprint 
#Get key to storage account 
$acctKey = (Get-AzureRmStorageAccountKey -Name StorageAccountName -ResourceGroupName ResourceGroupName).Value[0] 
#Map to the backup BLOB context 
$storageContext = New-AzureStorageContext -StorageAccountName StorageAccountName -StorageAccountKey $acctKey 
#Copy the file to the storage account 
Get-ChildItem -LiteralPath $FilePath | Set-AzureStorageBlobContent -Container "paloconfigbackup" -BlobType "Block" -Context $storageContext -Verbose 

If you are not currently using a Hybrid Worker in your subscription, create one from the below link:

https://docs.microsoft.com/en-us/azure/automation/automation-hybrid-runbook-worker

Paste the code into an Azure PowerShell Runbook and create a re-occuring schedule.

You’ll have backups saved in cold storage for as long as you would like to retain the data. Create a storage policy can help you