SD-WAN made easy

I’ll start by asking you two questions:

Are you paying too much for your Wide Area Network (WAN)?

And, is it the best method of connecting to the public Cloud?

At cloudstep.io we are constantly looking for ways to improve our customers connectivity to the public cloud. We consider cloud network connectivity a foundation service that must be implemented at the beginning of a successful cloud journey. Getting it right at the start is imperative to allow any cloud service adoption to truely reach its potential and not suffer from underserved network issues like latency, bandwidth and round-trip.

If the public cloud is going to become your new datacenter then why not structure your network around it.

What if I could solve your cloud connectivity and WAN connectivity in a single solution. Azure WAN is a service that offers you a centralised software defined managed network. Connect all your sites via VPN or ExpressRoute to Azure WAN and let Microsoft become your network layer 3 cloud that traditional telco providers are probably charging you hand over fist for. Who better to become your network service provider for your software defined network (SDN) then one of the largest software companies in the world! Microsoft.

Commodity business grade internet services are becoming cheaper now thanks to things like the NBN where it is truely a race to the bottom in regards to price in my opinion, which is great for the consumer… finally! Procuring NBN business grade connections for each of your office locations and then use Azure WAN to quickly deploy a secure network for site-to-site and site-to-Azure connectivity.

I believe that a service like this is really here to disrupt traditional network service providers and add great value to existing or new Microsoft Azure customers.

We are always looking to save money in a move to the cloud, potentially your network cost could be your biggest reduction. Get in contact with us at cloudstep.io to see if we can help you reform your network.


Tagging EC2 EBS Volumes in Auto Scaling Groups

Tagging becomes a huge part of your life when in the public cloud. Metadata is thrown around like hotcakes, and why not. At cloudstep.io we preach the ways of the DevOps gods and especially infrastructure as code for repeatable and standardised deployments. This way everything is uniform and everything gets a TAG!

I ran into an issue recently where I would build an EC2 instance and capture the operating system into an AMI as part of a CloudFormation stack. This AMI would then be used as part of a launch configuration and subsequent auto scaling group. The original EC2 instance had every tag needed across all parts that make up the virtual machine including:

  • EBS root volume
  • EBS data volumes
  • Elastic Network Interfaces (ENI)
  • EC2 Instance itself

When deploying my auto scaling group all the user level tags I’d applied had been removed from the volumes and ENI. This caused a few issues:

  1. EBS volumes couldn’t be tagged for billing.
  2. EBS volumes couldn’t be snapped based on tag level policies in Lifecycle Manager.
  3. Objects didn’t have a ‘Name’ tag which made it hard in the console to understand which virtual machine instance the object belonged too.

There are two methods I derived to add my tags back that I’ll share with you. The tags needed to be added upon launch of the instance when the auto scaling group added a server. The methods I used were:

  1. The auto scaling group has a Launch Configuration where the ‘User data’ field runs a script block at startup.
  2. Initiate a Lambda whenever CloudTrail logged an API reference of a launch event of an instance using CloudWatch.

Tagging with the User Data property and PowerShell

User data is simply:

When you launch an instance in Amazon EC2, you have the option of passing user data to the instance that can be used to perform common automated configuration tasks and even run scripts after the instance starts. You can pass two types of user data to Amazon EC2: shell scripts and cloud-init directives.

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html

Try {
 # Use the metadata service to discover which instance the script is running on
 $InstanceId = (Invoke-WebRequest '169.254.169.254/latest/meta-data/instance-id').Content
 $AvailabilityZone = (Invoke-WebRequest '169.254.169.254/latest/meta-data/placement/availability-zone').Content
 $Region = $AvailabilityZone.Substring(0, $AvailabilityZone.Length -1)
 $mac = (Invoke-WebRequest '169.254.169.254/latest/meta-data/network/interfaces/macs/').content
 $URL = "169.254.169.254/latest/meta-data/network/interfaces/macs/"+$mac+"/interface-id"
 $eni = (Invoke-WebRequest $URL).content
# Get the list of volumes attached to this instance
 $BlockDeviceMappings = (Get-EC2Instance -Region $Region -Instance $InstanceId).Instances.BlockDeviceMappings
 $Tags = (Get-EC2Instance -Region $Region -Instance $InstanceId).Instances.tag

 }
Catch{Write-Host "Could not access the AWS API, are your credentials loaded?" -ForegroundColor Yellow}
$BlockDeviceMappings | ForEach-Object -Process {
        $volumeid = $_.ebs.volumeid # Retrieve current volume id for this BDM in the current instance
        # Set the current volume's tags
        $Tags | ForEach-Object -Process {
        If($_.Key -notlike "aws:*"){
        New-EC2Tag -Resources $volumeid -Tags @{ Key = $_.Key ; Value = $_.Value } # Add tag to volume
        }
        }
}
# Set the current nics tag
$Tags | ForEach-Object -Process {
  If($_.Key -notlike "aws:*"){
        New-EC2Tag -Resources $eni -Tags @{ Key = $_.Key ; Value = $_.Value } # Add tag to eni
  }
}


This script block is great and works a treat with newly created instances from an Amazon Marketplace AMI’s e.g. a vanilla Windows Server 2019 template. The launch configuration would apply the script as a part of the cfn-init function at startup. Unfortunately I’d already used the cfn-init function as part of the original image customisation and capture, the cfn-init would not re-run and didn’t execute this script block. So back to the drawing board in my scenario.

Tagging with CloudWatch and Lambda Function

The second solution was to create a Lambda function and trigger it using an Amazon CloudWatch Events rule. The Instance ID is parsed from the CloudWatch event in JSON to the Lambda function.

Here is the Lambda function that is written in python2.7 and leverages the boto3 and JSON modules.



from __future__ import print_function
import json
import boto3
def lambda_handler(event, context):
  print('Received event: ' + json.dumps(event, indent=2))
  ids = []
  try:
      ec2 = boto3.resource('ec2')
      items = event['detail']['responseElements']['instancesSet']['items']
      for item in items:
        ids.append(item['instanceId'])
      base = ec2.instances.filter(InstanceIds=ids)
      for instance in base:
        ec2tags = instance.tags
        tags = [n for n in ec2tags if not n["Key"].startswith("aws:") ]
        print('   original tags:', ec2tags)
        print('   applying tags:', tags)
        for volume in instance.volumes.all():
          print('    volume:', volume)
          if volume.tags != ec2tags:
            volume.create_tags(DryRun=False, Tags=tags)
        for eni in instance.network_interfaces:
          print('    eni:', eni)
          eni.create_tags(DryRun=False, Tags=tags)
      return True
  except Exception as e:
    print('Something went wrong: ' + str(e))
    return False   



AWS ECS CloudFormation Fails – Unable to assume the service linked role.

I ran into an interesting issue when building a new ECS Cluster using CloudFormation. The CloudFormation stack would fail on Type: AWS::ECS::Service with error:

Unable to assume the service linked role. Please verify that the ECS service linked role exists. (Service: AmazonECS; Status Code: 400; Error Code: InvalidParameterException; Request ID: beadf3d5-3406-11e9-828d-b16cd52796ef)

Okay google, what’s this service linked role thingy?

A service-linked role is a unique type of IAM role that is linked directly to Amazon ECS. Service-linked roles are predefined by Amazon ECS and include all the permissions that the service requires to call other AWS services on your behalf.

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using-service-linked-roles.html

The first few times I ran my stack I assumed that this was for an IAM role that I was needing to assign to the AWS::ECS::Service to perform tasks much like a IamInstanceProfile of Type: AWS::EC2::Instance. When reviewing the available properties for Type: AWS::ECS::Service there was a Role definition:

  • Cluster
  • DeploymentConfiguration
  • DesiredCount
  • HealthCheckGracePeriodSeconds
  • LaunchType
  • LoadBalancers
  • NetworkConfiguration
  • PlacementConstraints
  • PlacementStrategies
  • PlatformVersion
  • Role
  • SchedulingStrategy
  • ServiceName
  • ServiceRegistries
  • TaskDefinition
Role - The name or ARN of an AWS Identity and Access Management (IAM) role that allows your Amazon ECS container agent to make calls to your load balancer.

I had some well defined Type: AWS::IAM::Role objects in my YAML for ECS execution and task roles but none of them were helping me with service linked account issue no matter how far I took the IAM policies.

Solution

To cut a long story and much googling short, the issue was nothing to do with my IAM policies but rather that the very first ECS cluster you create in the console using the getting started wizard creates the linked account in the backend. If your unlike me and read the full article about service linked roles you would have read:

when you create a new cluster (for example, with the Amazon ECS first run, the cluster creation wizard, or the AWS CLI or SDKs), or create or update a service in the AWS Management Console, Amazon ECS creates the service-linked role for you, if it does not already exist.

No mention in the above statement about CloudFormation. As per usual I jumped straight into a CloudFormation template without a test drive of the service and this time my attempt at being clever had given me a few moments of madness.

The easiest fix is to open up AWS CLI and run the following against your account once, then jump back into CloudFormation for YAML fun:

aws iam create-service-linked-role --aws-service-name ecs.amazonaws.com  

Resulting output:

{
    "Role": {
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17", 
            "Statement": [
                {
                    "Action": [
                        "sts:AssumeRole"
                    ], 
                    "Effect": "Allow", 
                    "Principal": {
                        "Service": [
                            "ecs.amazonaws.com"
                        ]
                    }
                }
            ]
        }, 
        "RoleId": "AROAIXGB2WBYGCXSPXY4O", 
        "CreateDate": "2019-02-19T05:55:58Z", 
        "RoleName": "AWSServiceRoleForECS", 
        "Path": "/aws-service-role/ecs.amazonaws.com/", 
        "Arn": "arn:aws:iam::112233445566:role/aws-service-role/ecs.amazonaws.com/AWSServiceRoleForECS"
    }
}

Job done. It all seemed so simple in retrospect.


Add VC Accounts to Microsoft Teams Channels with Azure Automation

At cloudstep.io® HQ Microsoft Teams is a big part of how we organise digital asset structure in the business. We are a consulting firm by trade, as new prospects become paying customers we add them as a team. The default General channel is used for administration and accounts, additional channels are created per project name or scope of works. We find ourselves no longer needing to going into dark corners of SharePoint administration (commonly referred to in the office as ‘SwearPoint!’). We have adopted Microsoft Teams as our preferred web, audio and video conferencing platform for internal and external meetings. Our board room video conferencing unit runs a full version of Windows 10 and Microsoft Teams that we setup as a ‘do it yourself‘ versus the off the shelf room systems. The VC unit requirements we had were:

  • cloudstep.io®, our web application uses a full desktop browser experience.
  • Mouse and keyboard are preferred for web navigation inside the app.
  • VC to have full OS is preferred to eliminate employees having to BYOD and connect either physically or wirelessly for screen presentation.
  • We can connect to third party conferencing platforms by installing the addons for guest access (zoom, webex, gotomeeting, chime etc) with our partner lead meetings direct onto the machine.
  • Wirelessly present our Macs, iPads, iPhones, Androids and Windows laptops.
  • We are all ‘power users‘ and can handle the meeting join experience in Microsoft Teams client without the need for a single ‘click-to-join’ button on the table which the Microsoft Teams Room (MTR) system provides via a touch device.

We have a boardroom account that has a 365 license to be able to leverage the desktop tools. Windows 10 automatically logs in each morning and the Microsoft Teams client is started automatically. The bill of materials is notably:

  • Intel NUC
  • Windows 10
    • Teams Client
    • Office 365 Pro Plus (Word, Excel, PowerPoint, OneNote)
    • Windows 10 Calendar (Connect to Office 365 Mailbox)
    • AirServer client (ChromeCast, MiraCast, AirPlay)
    • Chrome Browser
  • Office 365 user license
  • Logitech Meetup camera
  • Biggest screen we could fit in the room
  • Microsoft Bluetooth keyboard and mouse

The VC mailbox type is set to ‘room‘ with the following to enhance the experience for scheduled meetings in the board room:


#Add tips when searching in Outlook
Set-Mailbox -Identity $VC -MailTip "This room is equipped to support native Teams & Skype for Business Meetings remember to add meeting invite details via the Teams outlook button in the ribbon." 

#Auto Accept
Set-CalendarProcessing -Identity $VC -AutomateProcessing AutoAccept -AddOrganizerToSubject $false -RemovePrivateProperty $false -DeleteComments $false -DeleteSubject $false –AddAdditionalResponse $true –AdditionalResponse "Your meeting is now scheduled and if it was enabled as a Teams Meeting will be available from the conference room client."

This has worked well in the last 12 months, the only user experience problem we have had is when running a meeting from the VC unit, the account isn’t a member of the team where the data attempting to be presented is stored and therefor cannot see/open the content. A simple solution for this is automation. We looked to investigated two automation solutions available in the Microsoft services offering we have available.

  1. Flow (Office 365 Suite)
  2. Azure Automation (Azure Subscription)

Unfortunately option 1 didn’t have any native integration for triggers based on Office 365 groups or teams creation. So we resorted to a quick Azure Powershell Runbook that executes on a simple schedule. The steps needed to run were:

  1. Get a list of all the teams.
  2. Query them against the UnifiedGroup properties to get…
    1. AccessType equals ‘Public
    2. CreationDate within 2 days
  3. Check the newly created teams group membership for the VC unit username.
  4. If it doesn’t exist add the VC unit as the role ‘member‘.
Write-verbose "Getting Credentials ..." -Verbose
$Credentials = Get-AutomationPSCredential -Name 'Admin-365'
Write-verbose  "Credential Imported : $($Credentials.UserName)" -Verbose

$o365Cred = New-Object System.Management.Automation.PSCredential ($Credentials.UserName, $Credentials.Password)
Write-verbose  "Credential Loaded : $($o365Cred.UserName)" -Verbose
Write-verbose 'Connecting to 365 ...' -Verbose
$Session = New-PSSession –ConfigurationName Microsoft.Exchange -ConnectionUri https://outlook.office365.com/powershell-liveid/ -Credential $Credentials -Authentication Basic -AllowRedirection
Write-verbose 'Importing UnifiedGroups PowerShell Commands ...' -Verbose
Import-PSSession -Session $Session -DisableNameChecking:$true -AllowClobber:$true | Out-Null
Write-verbose 'Connecting to Teams ...' -Verbose
Connect-MicrosoftTeams -Credential $Credentials

$creationdate = ((Get-Date).AddDays(-2))
$teams = get-team
#$groups = Get-UnifiedGroup |Where-Object {$_.WelcomeMessageEnabled -like "False" -and $_.AccessType -like "Public" -and $_.WhenCreated -ge $creationdate}
$TeamsOutput = @()
foreach ($Team in $Teams){
$UnifiedGroup = Get-UnifiedGroup -Identity $Team.GroupId
    if($UnifiedGroup.AccessType -like "Public" -and $UnifiedGroup.WhenCreated -ge $creationdate){
    Write-verbose "Processing team named: $($UnifiedGroup.DisplayName)" -Verbose
        $VC = Get-TeamUser -GroupId $Team.GroupId | Where-Object {$_.User -like "user@domain.com"} 
        If($VC.count -eq 0){
            Write-verbose "VC not member, adding..." -Verbose
            Add-TeamUser -GroupId $Team.GroupId -User "user@domain.com" -Role Member
        }else{Write-verbose "VC is member already" -Verbose}
    }

$TeamsOutput+=$UnifiedGroup
}
Write-verbose "Total teams processed for selection: $($TeamsOutput.Count)" -Verbose 

The result is simple

Additional member added via PowerShell

Next day the board room account is logged in, the Microsoft Teams client will have access to all the teams channels, files, OneNote and apps. This is great for native Teams meetings, but also when we have customers in the board room without the need for an online meeting. The VC account has access to see the required teams and channel data to present to the physical display.

This solution doesn’t have to be for a video conferencing units, you may have some standardised members you want on all groups, or it could be certain owner enforcement or member list.

Hello Microsoft Teams! Bye bye SwearPoint, may you remain in the background forever.


Warm AWS WorkSpaces On a Schedule

AWS WorkSpaces VDI solution has two pricing options that you need to choose between for your implementation.

  1. Monthly
  2. Hourly (On demand)

In my opinion it is always worth attempting to run your WorkSpaces VDI deployment in on-demand where there is chance of cost savings when the virtual desktops can be turned off and you will not be charged.

With hourly billing you pay a small fixed monthly fee per WorkSpace to cover infrastructure costs and storage, and a low hourly rate for each hour the WorkSpace is used during the month. Hourly billing works best when Amazon WorkSpaces are used, on average, for less than a full working day or for just a few days a month, making it ideal for part-time workers, job sharing, road warriors, short-term projects, corporate training, and education.

https://aws.amazon.com/workspaces/pricing/

Turning off the VDI is done by AWS using a setting called Running Mode per VDI:

Always on – Billed monthly. Instant access to an always running WorkSpaces

AutoStop – Billed by the hour. WorkSpaces start automatically when you login, and stop when no longer being used (1-48hrs).

In my opinion a turn off period of 1 hour is too short, it doesn’t cover a user who has a long lunch or meeting that runs slightly over. 2 hours cool down period seems to be perfect for cost optimisation. With this in mind, all your VDI’s will be off at the beginning of your working day. To eliminate the need for the 60-90 second boot up time imposed by AWS for cold starts we can pre-warm the VDI’s using Lambda function on a schedule. The process will be as follows:

  1. CloudWatch Event that runs based on a CRON schedule.
  2. Event triggers the execution of a Lambda function
  3. The Lambda function runs python that starts WorkSpaces based on a set of conditions using the boto3 library to interact with the service.

The python code block below wakes all VDI’s in a region that are in a ‘STOPPED’ state but there is no reason why you couldn’t be more granular with tagging per VDI.


import boto3

def lambda_handler(event, context):
    directory_id= ''
    region = 'ap-southeast-2'
    running_mode = 'AVAILABLE'
    # Event
    session = boto3.session.Session(
        aws_access_key_id='',
        aws_secret_access_key=''
    )
    
    ws = session.client('workspaces')
    workspaces = []
    
    resp = ws.describe_workspaces(DirectoryId=directory_id)
    
    while resp:
      workspaces += resp['Workspaces']
      resp = ws.describe_workspaces(DirectoryId=directory_id, NextToken=resp['NextToken']) if 'NextToken' in resp else None
    
    for workspace in workspaces:
    
      if workspace['State'] == running_mode:
        continue
    
      if workspace['State'] in ['STOPPED']:
    
        ws.start_workspaces(
          StartWorkspaceRequests=[
            {
                'WorkspaceId': workspace['WorkspaceId'],
            },
          ]
        )
      
        print 'Starting WorkSpace for user: ' + workspace['UserName']
    
      else:
        print 'Could not start workspace for user: ' + workspace['UserName']


To start the Workspaces on a schedule, Lambda can invoke using a CRON expression:

cron(0 22 ? * SUN-THU *)

The cron schedule runs in GMT, so in this case 10:00 PM in GMT is 8:00 AM in AEST for following day (GMT +10:00).

The end result is the WorkSpaces you have chosen to wake up would start at 8am and shutdown again at 10am if not used. If you had departments or user groups that are heavy users versus sometimes users this might be where your code looks at some tags you’ve set per VDI.