YAML it Rhymes with Camel

I’ve blogged before about my passion for automation and the use of ARM templating in the Azure world to eradicate the burden of dull and mundane tasks from the daily routine of system administrators for whom I do consulting for.

I loath repetitive tasks, its in this space where subtle differences and inconsistency love to live. Recently I was asked to help out with a simple task, provisioning a couple of EC2 Windows servers in AWS. So in the spirit of infrastructure as code, I thought, there is no better time to try out AWS CloudFormation to describe my EC2 instances . I’ve actually used CloudFormation before in the past, but always describing my stack in JSON. CloudFormation also supports YAML, so challenge accepted and away I went. . .

So what is YAML anyway. . .Yet Another Mark-up Language. Interestingly its described at the official YAML website (https://yaml.org) as a “YAML Ain’t Markup Language” rather,  “human friendly data serialisation standard for all programming languages”.

What attracted me to YAML is its simplicity, there are no curly braces {} just indenting. Its also super easy to read. So if JSON looks a bit to cody for your liking, YAML may be a more palatable alternative.

So how would you get started? As you’d expect AWS have extensive CloudFormation documentation. The AWS::EC2::Instance resource is described here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-instance.html#cfn-ec2-instance-volumes. You’ll notice that there is a Syntax description for JSON and YAML. The YAML looks like this:


Type: AWS::EC2::Instance
Properties: 
  Affinity: String
  AvailabilityZone: String
  BlockDeviceMappings: 
    - EC2 Block Device Mapping
  CreditSpecification: CreditSpecification
  DisableApiTermination: Boolean
  EbsOptimized: Boolean
  ElasticGpuSpecifications: [ ElasticGpuSpecification, ... ]
  ElasticInferenceAccelerators: 
    - ElasticInferenceAccelerator
  HostId: String
  IamInstanceProfile: String
  ImageId: String
  InstanceInitiatedShutdownBehavior: String
  InstanceType: String
  Ipv6AddressCount: Integer
  Ipv6Addresses:
    - IPv6 Address Type
  KernelId: String
  KeyName: String
  LaunchTemplate: LaunchTemplateSpecification
  LicenseSpecifications: 
    - LicenseSpecification
  Monitoring: Boolean
  NetworkInterfaces: 
    - EC2 Network Interface
  PlacementGroupName: String
  PrivateIpAddress: String
  RamdiskId: String
  SecurityGroupIds: 
    - String
  SecurityGroups: 
    - String
  SourceDestCheck: Boolean
  SsmAssociations: 
    - SSMAssociation
  SubnetId: String
  Tags: 
    - Resource Tag
  Tenancy: String
  UserData: String
  Volumes: 
    - EC2 MountPoint
  AdditionalInfo: String

With this as a starting point I was quickly able to build a EC2 instance and customise my YAML so as to do some extra things.

If you’ve got this far and YAML is starting to look like it might be the ticket for you, its worth familiarising yourself with the CloudFormation built-in functions. You can use these to do things like assign values to properties that are not available until runtime.

Fn::Base64
Fn::Cidr
Condition Functions
Fn::FindInMap
Fn::GetAtt
Fn::GetAZs
Fn::Join
Fn::Select
Fn::Split
Fn::Sub
Fn::Transform
Ref

The link to the complete Intrinsic Function Reference can be found here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference.html

With a learning curve of a couple of hours including a bit of googling and messing around I was able to achieve my goal. I built an EC2 instance, applied tagging, installed some Windows features post build via a PowerShell script (downloaded from S3 and launched with AWS::CloudFormation::Init cfn-init.exe), all without having to logon to the server or touch the console. Here is a copy of my YAML. . .


AWSTemplateFormatVersion: "2010-09-09"
Description: CloudFormation Template to deploy an EC2 instance
Parameters: 
  Hostname: 
    Type: String
    Description: Hostname - maximum 15 characters
    MaxLength: '15'    
  LatestAmiId :
    Type: 'AWS::SSM::Parameter::Value'
    Default: /aws/service/ami-windows-latest/Windows_Server-2019-English-Full-Base
  InstanceSize: 
    Type: String
    Description: Instance Size
    Default: t2.micro
    AllowedValues:
      - "t2.micro" 
      - "t2.small"
      - "t2.medium"
  AvailabilityZone:
    Type: String
    Description: Default AZ
    AllowedValues: 
      - ap-southeast-2a
      - ap-southeast-2b
      - ap-southeast-2c
    Default: ap-southeast-2a
  KeyPair: 
    Type: String
    Description: KeyPair Name
    Default: jtwo
  S3BucketName:
    Default: NotARealBucket
    Description: S3 bucket containing boot artefacts
    Type: String
  
  # tag values
  awPurpose: 
    Type: String
    Description: A plain English description of what the object is for.
    Default: WindowsServer2019 Domain Controller
  awChargeTo: 
    Type: String
    Description: Billing Code for charge back of resource.
    Default: IT-123
  awRegion: 
    Type: String
    Description: Accolade Wines Region not AWS. 
    Default: Australia
  awExpiry: 
    Type: String
    Description: The date when the resource(s) can be considered for decommissioning.
    Default: 01-01-2022
  awBusinessSegment: 
    Type: String
    Description: Agency code.
    Default: ICT
  awEnvironment: 
    Type: String
    Description: Specific environment for resource.
    AllowedValues: 
      - prod
      - prodServices
      - nonprod
      - uat
      - dev
      - test 
  awApplication: 
    Type: String
    Description: A single or multiple word with the name of the application that the infrastructure supports. "JDE", "AD", "Apache", "Utility", "INFOR", "PKI".
    Default: AD

Mappings:
  SubnetMap: 
    ap-southeast-2a:
      prodServices: "subnet-idGoesHere"
    ap-southeast-2b:
      prodServices: "subnet-idGoesHere"
    ap-southeast-2c:
      prodServices: "subnet-idGoesHere"
      
# Resources
Resources:
  # IAM Instance Profile
  Profile:
    Type: 'AWS::IAM::InstanceProfile'
    Properties:
      Roles:
        - !Ref HostRole
      Path: /
      InstanceProfileName: !Join
        - ''
        - - 'instance-profile-'
          - !Ref S3BucketName
  HostRole:
    Type: 'AWS::IAM::Role'
    Properties:
      RoleName: !Join
        - ''
        - - 'role-s3-read-'
          - !Ref S3BucketName
      Policies:
        - PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Action:
                  - 's3:GetObject'
                Resource: !Join
                  - ''
                  - - 'arn:aws:s3:::'
                    - !Ref S3BucketName
                    - '/*'
                Effect: Allow
          PolicyName: s3-policy-read
      Path: /
      AssumeRolePolicyDocument:
        Statement:
          - Action:
              - 'sts:AssumeRole'
            Principal:
              Service:
                - ec2.amazonaws.com
            Effect: Allow
        Version: 2012-10-17  

  # ENI
  NIC1:
    Type: AWS::EC2::NetworkInterface
    Properties: 
      Description: !Sub 'ENI for EC2 instance: ${Hostname}-${awEnvironment}'
      GroupSet:
          - sg-050cadbf0e159b0ac
      SubnetId: !FindInMap [SubnetMap, !Ref AvailabilityZone, !Ref awEnvironment]
      Tags:
        - Key: Name
          Value: !Sub '${Hostname}-eni'
  
  # EC2 Instance
  Instance:
    Type: 'AWS::EC2::Instance'
    Metadata:
      'AWS::CloudFormation::Authentication':
        S3AccessCreds:
          type: S3
          buckets:
            - !Ref S3BucketName
          roleName: !Ref HostRole
      'AWS::CloudFormation::Init':
        configSets: 
          config:
            - get-files 
            - configure-instance
        get-files:
          files:
            'c:\s3-downloads\scripts\Add-WindowsFeature.ps1':
              source: https://NotARealBucket.s3.amazonaws.com/scripts/Add-WindowsFeature.ps1
              authentication: S3AccessCreds
        configure-instance:
          commands:
            1-set-powershell-execution-policy:
              command: >-
                powershell.exe -Command "Set-ExecutionPolicy UnRestricted -Force"
              waitAfterCompletion: '0'
            2-rename-computer:
              command: !Join
                - ''
                - - >-
                  -  powershell.exe -Command "Rename-Computer -Restart -NewName "
                  -  !Ref Hostname
              waitAfterCompletion: forever  
            3-install-windows-components:
              command: >-
                powershell.exe -Command "c:\s3-downloads\scripts\Add-WindowsFeature.ps1"
              waitAfterCompletion: '0'


    Properties:
      DisableApiTermination: 'false'
      AvailabilityZone: !Sub "${AvailabilityZone}"
      InstanceInitiatedShutdownBehavior: stop
      IamInstanceProfile: !Ref Profile
      ImageId: !Ref LatestAmiId
      InstanceType: !Sub "${InstanceSize}"
      KeyName: !Sub "${KeyPair}"
      UserData: !Base64
        'Fn::Join': 
          - ''
          - - "\n"
            - "cfn-init.exe "
            - " --stack "
            - "Ref": "AWS::StackId"
            - " --resource Instance"
            - " --region "
            - "Ref": "AWS::Region"
            - " --configsets config"
            - " -v \n"
            - "cfn-signal.exe  "
            - " ---exit-code 0"
            - " --region "
            - "Ref": "AWS::Region"
            - " --resource Instance" 
            - " --stack "
            - "Ref": "AWS::StackName"
            - "\n"           
            - "\n"
      Tags:
        - Key: Name
          Value: !Sub "${Hostname}"
        - Key: awPurpose
          Value: !Sub "${awPurpose}"
        - Key: awChargeTo
          Value: !Sub "${awChargeTo}"
        - Key: awRegion
          Value: !Sub "${awRegion}"
        - Key: awExpiry
          Value: !Sub "${awExpiry}"
        - Key: awBusinessSegment
          Value: !Sub "${awBusinessSegment}"
        - Key: awEnvironment
          Value: !Sub "${awEnvironment}"
        - Key: awApplication
          Value: !Sub "${awApplication}"

      NetworkInterfaces:
        - NetworkInterfaceId: !Ref NIC1
          DeviceIndex: 0

Outputs:
  InstanceId:
    Description: 'InstanceId'
    Value: !Ref Instance
    Export:
      Name: !Sub '${Hostname}-${awEnvironment}-InstanceId'
  InstancePrivateIP:
    Description: 'InstancePrivateIP'
    Value: !GetAtt Instance.PrivateIp
    Export:
      Name: !Sub '${Hostname}-${awEnvironment}-InstancePrivateIP'

So my question now is, why doesn’t Azure also support YAML?


Invest one hour in learning about AWS

Getting educated about cloud services early will make it easier to transition and will burnish your CV nicely too!

Amazon Web Services (AWS) offers a free virtual machine instance on AWS Elastic Compute Cloud (EC2). All you have to do is register with an email address and a creditcard. Don’t worry, they really don’t charge your card unless the machine is still running a year later or if you exceed the data allowances. 
A year ago I wanted a USA VPN so I created a free EC2 instance and installed a prepackaged machine image with OpenVPN. I logged in, run a couple of commands and it all worked. I still use it today and the first bill showed up only at the end of the year. Internet data in the USA is very very cheap. 

My starting point was a conference session very like this video: Your first hour on AWS which walks you through how to sign up and set up your first instance on AWS.

Go on! It’s only going to take an hour of your life. At the end you’re one of the cool kids who has an AWS instance in the cloud and no one can ever try to convince you the cloud is “hard” ever again. 

You can experiment with 5 gigabytes of Simple Storage Service (S3) storage too and learn how to make any piece of data a web URI accessible object.

You can learn about AWS Lamda which is computing on demand as I always imagined it would be. Your code is run in response to a trigger like uploading a file to a directory, a URI being accessed or an address receiving an email. You pay nothing until the trigger event. You don’t need to maintain a virtual machine instance either, the code just runs in its own private environment and does whatever you need on demand. You can even build high availability high performance interactive web sites with little more than S3 and Lamda. 

When you’re done with the tutorial you can reflect on how easy all of this is and ask yourself the question “how much faster could my organisation respond to change if we used these tools?” 


AWS obtain PROTECTED level certification for Australian Region

Earlier this week Amazon Web Services made a statement, indicating that the battle of tier-one public cloud providers is still heating up. Yesterday Matthew Graham (AWS Head of Security Assurance for Australia and New Zealand) announced that The Australian Cyber Security Centre (ACSC) had awarded PROTECTED certification to AWS for 42 of their cloud services. 

In what appears to be a tactical move that has been executed hot off the trail of Microsoft announcing their PROTECTED accredited Azure Central Regions in the back half of last year. This clearly demonstrates that AWS aren’t prepared to reduce the boil to a gentle simmer any time soon.

Graham announced “You will find AWS on the ACSC’s Certified Cloud Services List (CCSL) at PROTECTED for AWS services, including Amazon Elastic Compute Cloud (Amazon EC2), Amazon Simple Storage Service (Amazon S3), AWS Lambda, AWS Key Management Service (AWS KMS), and Amazon GuardDuty.”

He continued to state “We worked with the ACSC to develop a solution that meets Australian government security requirements while also offering a breadth of services so you can run highly sensitive workloads on AWS at scale. These certified AWS services are available within our existing AWS Asia-Pacific (Sydney) Region and cover service categories such as compute, storage, network, database, security, analytics, application integration, management and governance. “

Finally, delivering a seemingly well orchestrated jab “Importantly, all certified services are available at current public prices, which ensures that you are able to use them without paying a premium for security.”

It is no secret that the blue team currently charges a premium for entry into their PROTECTED level facility (upon completion of a lengthy eligibility assessment process) due to a finite amount of capacity available.

Both vendors state that consumers must configure services in line with the guidance in the respective ACSC certification report and consumer guidelines. This highlights that additional security controls must be implemented to ensure workloads are secured head to toe whilst storing protected level data. Ergo, certification is not implicit by nature of consuming accredited services.

AWS have released the IRAP assessment reports under NDA within their Artefact repository. For more information, review the official press release here.


Using the AWS CLI for Process Automation

Amazon Web Services is a well established cloud provider. In this blog, I am going to explore how we can interface with the orange cloud titan programmatically. First of all, lets explore why we may want to do this. You might be thinking “But hey, the folks at AWS have built a slick web interface which offers all the capability I could ever need.”Whilst this is true, repetitive tasks quickly become onerous. Additionally, manual repetition introduces the opportunity to introduce human error. That sounds like something we should avoid, right? After all, many of the core tenets of the DevOps movement is built on these principles (“To increase the speed, efficiency and quality of software delivery”– amongst others.)

From a technology perspective, we achieve this by establishing automated services. This presents a significant speed advantage as automated processes are much faster than their manual counterparts. The quality of the entire release process improves because steps in the pipeline become standardised, thus creating predictable outcomes.

Here at cloudstep, this is one of our core beliefs when operating a cloud infrastructure platform. Simply put, the portal is a great place to look around and check reporting metrics. However, any services should be provisioned as code. Once again, to realise efficiency and improve overall quality.

How do we go about this and what are some example use cases?”

AWS provide an open source CLI bundle which enables you to interface directly with their public API’s. Typically speaking, this is done using a terminal of your choice (Linux shells, Windows Command Line, PowerShell, Puty, Remotely.. You name it, its there.) Additionally, they also offer SDK’s which provide a great starting point for developing applications on-top of their services in many different languages (PowerShell, Java, .NET, JavaScript, Ruby, Python, PHP and GO.)   

So lets get into it… The first thing you’ll want to do is walk through the process of aligning your operating environment with any mandatory prerequisites, then you can get install the AWS CLI tools in a flavour of your choice. The process is well documented, so I wont cover it off here.

Link – https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html

Once you have the tools installed, you will need to provide the CLI tools with a base level of configuration which is stored in a profile of your choice. Running “AWS Configure” from a terminal of your choice is the fastest way to do this. Here you will provide IAM credentials to interface with your tenant, a default region and an output format. For the purpose of this example I’ve set my region to “ap-southeast-2” and my output format to “JSON.”

aws configure example

From here I could run “aws ec2 describe-instances” to validate that my profile had been defined correctly within the AWS CLI tools. The expected return would be a list of EC2 instances hosted within my AWS subscription as shown below.

aws ec2 describe-instances example

This shouldn’t take more than 5 minutes to get you up and running. However, don’t stop here. The AWS CLI supports almost all of the capability which can be found within the management portal. Therefore, if you’re in an operations role and your company is investing in AWS in 2019. You should be spending some time to learn about how to interface with services such as DynamoDB, EC2, S3/Glacier, IAM, SNS and SWF using the AWS CLI.

Lets have a look at a more practical example whereby automating a simple task can potentially save you hours of time each year. As a Mac user (you’ve probably already picked up on that) I often need to fire up a windows PC for Visual Studio or Visio. AWS is a great use case for this. I simply fire up my machine when I need it and shut it down when I’m done. I pay them a couple of bucks a month for some storage costs and some compute hours and I’m a happy camper. Simple right?

Lets unpack it further. I am not only a happy camper. I’m also a lazy camper. Firing up my VM to do my day job means:

  • Opening my browser and navigating to the AWS management console
  • Authenticating to the console
  • Navigating to the EC2 service
  • Scrolling through a long list of instances looking for my jumpbox
  • Starting my VM
  • Waiting for the network interface to refresh so I can get the public IP for RDP purposes.

This is all getting too hard right? All of this has to happen before I can even do my job and sometimes I have to do this a few times each day. Maybe its time to practice what I preach? I could automate all of this using the AWS tools for PowerShell, which would allow me to automate this process by running a script which saves me hours each year (employers love that.) Whilst this example wont necessarily increase the overall quality of my work, it does provide me with a predictable outcome every single time.

For a measly 20 lines of PowerShell I was able to define an executable script which authenticates to the AWS EC2 service, checks the power state of my VM in question. If the VM is already running it will return the connectivity details for my RDP client. If the VMis not running, it will fire up my instance, wait for the NIC to refresh and then return the connectivity details for my RDP client. I then have a script based on the same logic to shutdown my VM to save money when I’m not using the service. All of this takes less than 5 seconds to execute.

PowerShell Automation Example

The AWS CLI tools provide an interface to interact with the cloud provider programmatically. In this simple example we looked at automating a manual process which has the potential to save hours of time each year whilst also ensuring a predictable outcome for each execution. Each of the serious public cloud players offer similar capability. If you are looking to increase your overall efficiency, improve the quality of your work whilst automating monotonous tasks, consider investing some effort into learning a how to interface with your cloud provider of choice programmatically. You will be surprised how many repetitive tasks you can bowl over when you maximise the usage of the tools you have available to you. 



Warm AWS WorkSpaces On a Schedule

AWS WorkSpaces VDI solution has two pricing options that you need to choose between for your implementation.

  1. Monthly
  2. Hourly (On demand)

In my opinion it is always worth attempting to run your WorkSpaces VDI deployment in on-demand where there is chance of cost savings when the virtual desktops can be turned off and you will not be charged.

With hourly billing you pay a small fixed monthly fee per WorkSpace to cover infrastructure costs and storage, and a low hourly rate for each hour the WorkSpace is used during the month. Hourly billing works best when Amazon WorkSpaces are used, on average, for less than a full working day or for just a few days a month, making it ideal for part-time workers, job sharing, road warriors, short-term projects, corporate training, and education.

https://aws.amazon.com/workspaces/pricing/

Turning off the VDI is done by AWS using a setting called Running Mode per VDI:

Always on – Billed monthly. Instant access to an always running WorkSpaces

AutoStop – Billed by the hour. WorkSpaces start automatically when you login, and stop when no longer being used (1-48hrs).

In my opinion a turn off period of 1 hour is too short, it doesn’t cover a user who has a long lunch or meeting that runs slightly over. 2 hours cool down period seems to be perfect for cost optimisation. With this in mind, all your VDI’s will be off at the beginning of your working day. To eliminate the need for the 60-90 second boot up time imposed by AWS for cold starts we can pre-warm the VDI’s using Lambda function on a schedule. The process will be as follows:

  1. CloudWatch Event that runs based on a CRON schedule.
  2. Event triggers the execution of a Lambda function
  3. The Lambda function runs python that starts WorkSpaces based on a set of conditions using the boto3 library to interact with the service.

The python code block below wakes all VDI’s in a region that are in a ‘STOPPED’ state but there is no reason why you couldn’t be more granular with tagging per VDI.


import boto3

def lambda_handler(event, context):
    directory_id= ''
    region = 'ap-southeast-2'
    running_mode = 'AVAILABLE'
    # Event
    session = boto3.session.Session(
        aws_access_key_id='',
        aws_secret_access_key=''
    )
    
    ws = session.client('workspaces')
    workspaces = []
    
    resp = ws.describe_workspaces(DirectoryId=directory_id)
    
    while resp:
      workspaces += resp['Workspaces']
      resp = ws.describe_workspaces(DirectoryId=directory_id, NextToken=resp['NextToken']) if 'NextToken' in resp else None
    
    for workspace in workspaces:
    
      if workspace['State'] == running_mode:
        continue
    
      if workspace['State'] in ['STOPPED']:
    
        ws.start_workspaces(
          StartWorkspaceRequests=[
            {
                'WorkspaceId': workspace['WorkspaceId'],
            },
          ]
        )
      
        print 'Starting WorkSpace for user: ' + workspace['UserName']
    
      else:
        print 'Could not start workspace for user: ' + workspace['UserName']


To start the Workspaces on a schedule, Lambda can invoke using a CRON expression:

cron(0 22 ? * SUN-THU *)

The cron schedule runs in GMT, so in this case 10:00 PM in GMT is 8:00 AM in AEST for following day (GMT +10:00).

The end result is the WorkSpaces you have chosen to wake up would start at 8am and shutdown again at 10am if not used. If you had departments or user groups that are heavy users versus sometimes users this might be where your code looks at some tags you’ve set per VDI.