Diving into the AWS CloudFormation update hook: cfn-hup

Provisioning scripts for EC2 instances in Auto-Scaling Group

AWS CloudFormation allows us to define MetaData (AWS::CloudFormation::Init) that gets normally executed as part of an EC2 instance UserData. This is particularly helpful when launching EC2 instances as part of an auto-scaling group, as it allows us to run the same provisioning scripts on every instance in a more organized fashion.

Problem: Updating a running EC2 instance in an Auto-Scaling Group

When we modify the UserData that’s part of an AutoScaling LaunchConfiguration, CloudFormation needs to start new instances and tear down the old ones, as the UserData cannot be changed after the instance start. Any configuration sets that are part of the AWS::CloudFormation::Init also won’t get run again, as they are executed in the UserData.

In order to update data on running instances, we need to use the cfn-hup CloudFormation helper script.

What is cfn-hup

AWS describes the cfn-hup helper as following:

The cfn-hup helper is a daemon that detects changes in resource metadata and runs user-specified actions when a change is detected. This allows you to make configuration updates on your running Amazon EC2 instances through the UpdateStack API action.

The full documentation can be found here: cfn-hup – AWS CloudFormation.

Example: Update the host file in any EC2 instances started by an AutoScaling Group

Let’s assume we have an AutoScaling LaunchConfiguration resource called EcsInstanceLaunchConfiguration with this (abbreviated) stack definition:

Resources:
  EcsInstanceLaunchConfiguration:
    Type: AWS::AutoScaling::LaunchConfiguration
    Properties:
      IamInstanceProfile: ecsInstanceRole
      ImageId: !FindInMap [EcsOptimizedAmi, !Ref "AWS::Region", AmiId]
      InstanceType: !Ref EcsInstanceType
      KeyName: !Ref KeyName
      UserData:
        Fn::Base64:
          !Sub |
            yum update -y
            yum install -y aws-cfn-bootstrap
            # Install the files and packages from the metadata.
            /opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --region ${AWS::Region} --resource EcsInstanceLaunchConfiguration --configsets SetupEnvironment,UpdateEnvironment
    Metadata:
      AWS::CloudFormation::Init:
        configSets:
          SetupEnvironment:
            - setupCfnHup
            - installAwsCli
          UpdateEnvironment:
            - updateHostsFile
        setupCfnHup:
           files:
             '/etc/cfn/cfn-hup.conf':
               content: !Sub |
                 [main]
                 stack=${AWS::StackId}
                 region=${AWS::Region}
                 interval=1
               mode: '000400'
               owner: root
               group: root
             '/etc/cfn/hooks.d/cfn-auto-reloader.conf':
               content: !Sub |
                 [cfn-auto-reloader-hook]
                 triggers=post.update
                 path=Resources.EcsInstanceLaunchConfiguration.Metadata.AWS::CloudFormation::Init
                 action=/opt/aws/bin/cfn-init --verbose --stack=${AWS::StackName} --region=${AWS::Region} --resource=EcsInstanceLaunchConfiguration --configsets UpdateEnvironment
                 runas=root
               mode: '000400'
               owner: root
               group: root
           services:
             sysvinit:
               cfn-hup:
                 enabled: true
                 ensureRunning: true
                 files:
                 - '/etc/cfn/cfn-hup.conf'
                 - '/etc/cfn/hooks.d/cfn-auto-reloader.conf'
        installAwsCli:
          packages:
            yum:
              unzip: []
          commands:
            01_download_bundled_installer:
              command: curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
            02_unzip_package:
              command: unzip awscli-bundle.zip
            03_run_install_executable:
              command: sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws
        updateHostsFile:
          commands:
            01_remove_local_graylog_entries:
              command: sed -i '/local.graylog/d' /etc/hosts
            02_add_graylog_entry:
              command: echo "$(/usr/local/bin/aws ec2 describe-instances --region us-east-1 --filters "Name=tag:aws:cloudformation:stack-name,Values=${GRAYLOG_CLUSTER_STACK_NAME}" --query 'Reservations[*].Instances[*].NetworkInterfaces[*].PrivateIpAddresses[*].PrivateIpAddress' --output text) local.graylog" >> /etc/hosts
              env:
                GRAYLOG_CLUSTER_STACK_NAME: "GraylogCluster"

Goal of this stack

We want to run a single Graylog EC2 instance for logging in a cluster called “GraylogCluster”. We don’t want to hardcode the IP address for that Graylog instance, but instead use the custom hostname local.graylog, which will point to the internal IP address of the Graylog instance.

Requirements

We need to install the AWS CLI on our cluster in order to be able to query for the private IP address of the Graylog instance. This example also assumes that the ecsInstanceRole has the permissions to DescribeInstances. We install the AWS CLI in the installAwsCli configuration of the metadata.

Run the cfn-hup service to monitor resources

This service is defined in the setupCfnHup configuration of the metadata. We can define several files that contain hooks monitoring several resources. Let’s examine the hook defined in /etc/cfn/hooks.d/cfn-auto-reloader.conf.

This hook monitors Resources.EcsInstanceLaunchConfiguration.Metadata.AWS::CloudFormation::Init, so any changes to the metadata of this launch configuration. When anything changes, it executes the command /opt/aws/bin/cfn-init --verbose --stack=${AWS::StackName} --region=${AWS::Region} --resource=EcsInstanceLaunchConfiguration --configsets UpdateEnvironment, which updates the host file with the private IP address of the Graylog instance.

Caveat : Loading configsets from different stacks

The documentation for what exactly happens when we specify the stack as --stack=${AWS::StackName} wasn’t documented quite well enough for me. My assumption was that we are executing a configuration set in a specific stack, but that is incorrect. Actually, we are executing a configuration set from a specific stack.

This means: the /opt/aws/bin/cfn-init command is able to pull configuration sets from anywhere and run them on the current instance. However, this assumes that all required dependencies are available. This also allows us to define a single stack with MetaData that can be shared with other stacks.