Introduction Link to heading

Recently I was tasked with spinning up an ec2 instance with some attached ebs “disks”. This isn’t generally how I roll. It’s old-school, the opposite of the modern, serverless / on-demand way. It drove me to write some particularly clever bash code to manage /etc/fstab. I hate writing clever bash code. It never turns out well. I can imagine this being useful to someone else who gets stuck solving this problem. I sincerely hope it won’t ever be useful to me again.

The Easy Part Link to heading

Getting the box deployed using cdk was, as expected, pretty easy.

import * as cdk from '@aws-cdk/core';
import * as ec2 from '@aws-cdk/aws-ec2';
import * as iam from '@aws-cdk/aws-iam';

export class Postgresql94 extends cdk.Construct {
  readonly box: ec2.Instance;
  readonly sg: ec2.SecurityGroup;

  // Note that we're requiring this construct to be a direct part of a Stack.
  constructor(scope: cdk.Stack, id: string) {
    super(scope, id);

    this.sg = new ec2.SecurityGroup(this, 'sg', {
      vpc: scope.vpc, // I bet your Stack doesn't support this.
      // I'm working on a blog post about extending core.Stack and will update here when it's done.
      description: `Security group for PostgreSQL 9.4`,
    });

    this.box = new ec2.Instance(this, 'box', {
      vpc: scope.vpc, // You'd need to super-call the aws-cdk/core scop for this to work.
      vpcSubnets: scope.vpc.selectSubnets({ subnetType: ec2.SubnetType.PUBLIC }),
      securityGroup: this.sg,
      instanceType: new ec2.InstanceType('m5.large'),
      machineImage: new ec2.LookupMachineImage({
        name: 'CentOS 7*',
        filters: {
          architecture: ['x86_64'],
          state: ['available'],
          'owner-id': ['125523088429'],
        },
      }),
      blockDevices: [
        {
          deviceName: '/dev/sdf', // mounted on /three
          // SEE BELOW we depend on the size, of all things, to identify the device.
          volume: ec2.BlockDeviceVolume.ebs(1008, {
            encrypted: true,
            deleteOnTermination: true,
            volumeType: ec2.EbsDeviceVolumeType.GP2,
          }),
        },
        {
          deviceName: '/dev/sdg', // mounted on /one
          volume: ec2.BlockDeviceVolume.ebs(10, {
            encrypted: true,
            deleteOnTermination: true,
            volumeType: ec2.EbsDeviceVolumeType.GP2,
          }),
        },
        {
          deviceName: '/dev/sdh', // mounted on /two
          volume: ec2.BlockDeviceVolume.ebs(100, {
            encrypted: true,
            deleteOnTermination: true,
            volumeType: ec2.EbsDeviceVolumeType.GP2,
          }),
        },
      ],
    });

    this.box.role.addManagedPolicy(iam.ManagedPolicy.fromAwsManagedPolicyName('AmazonSSMManagedInstanceCore'));

    const instanceResource = this.box.node.findChild('Resource') as cdk.CfnResource;
    this.box.addUserData(`
set -x

yum -y update
yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm
systemctl enable amazon-ssm-agent
systemctl start amazon-ssm-agent
yum install -y epel-release
yum install -y awscli
yum install -y xfsprogs
/usr/bin/easy_install --script-dir /opt/aws/bin https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz
cp -v /usr/lib/python2*/site-packages/aws_cfn_bootstrap*/init/redhat/cfn-hup /etc/init.d
chmod +x /etc/init.d/cfn-hup

##############################
# Clever stuff will go here. #
##############################

/opt/aws/bin/cfn-signal -e $? --stack ${scope.stackName}  --resource ${instanceResource.logicalId} --region ${scope.region}
`);
    instanceResource.cfnOptions.creationPolicy = {
      ...this.box.instance.cfnOptions.creationPolicy,
      resourceSignal: {
        count: 1,
        timeout: cdk.Duration.minutes(30).toIsoString(),
      },
    };
  }
}

This gets us started with some basics. Importantly, the ssm agent is now installed and running.

The Clever Stuff Link to heading

Now we have to solve the problem of getting our ebs volumes mounted. And convincing them to stay mounted through a reboot. Note we have (thankfully only) one layer of escaping involved because we are inside a typescript back-tick interpolation block.

`
for block in nvme1n1 nvme2n1 nvme3n1; do
  mkfs -t xfs "/dev/$block"
done

# We know where we'll be mounting these
mkdir -p /one
mkdir -p /two
mkdir -p /three

# This disturbs me. There ought to be a better way than using the size to identify the EBS volumes.
lsblk | while read name _ _ size _; do
    case "$size" in
      10G)
        mount "/dev/$name" /one
        ;;
      100G)
        mount "/dev/$name" /two
        ;;
      1008G)
        mount "/dev/$name" /three
        ;;
      *)
        echo "skip"
        ;;
    esac
done

# example output of the join command below
# nvme1n1 259:2 0 10G 0 disk /one 545378bc-be57-48b3-97ba-58cef8e04fde xfs
# nvme2n1 259:3 0 100G 0 disk /two cfe3cb17-2bce-4996-aef5-6697a1e27683 xfs
# nvme3n1 259:4 0 1008G 0 disk /three faa43069-a001-47ee-9730-4f08e028d7d0 xfs
join -1 1 -2 1 \
<(lsblk | sort) \
<(blkid \
  | while read name uuid fstype; do \
    cleaned_dev=$(echo "\${name#/dev/}" | tr -d :); \
    cleaned_uuid=$(echo "\${uuid#UUID=\\"}" | tr -d \\"); \
    cleaned_fstype=$(echo "\${fstype#TYPE=\\"}" | tr -d \\"); \
    echo "$cleaned_dev $cleaned_uuid $cleaned_fstype"; \
  done | sort) \
| while read _ _ _ _ _ _ mountpoint uuid fstype; do
    echo "UUID=$uuid $mountpoint $fstype defaults 1 1" >> /etc/fstab
done
`

How It Works Link to heading

We start off by making the filesystems and getting them mounted to the correct location. I can’t believe we are using the size of the filesystem to figure out what goes where, but I couldn’t find a better solution for the problem.

The clever bit is around updating the /etc/fstab. Neither lsblk nor blkid provide the output we need. Perversely, they don’t even agree on the formatting of their output. At least they both talk about the nvme* device names. And… that’s what I’ve used to join the two outputs.

The join command takes two parameters: the column number for the first file and the column number for the second file. These columns are generated by simply splitting on whitespace. The inputs need to be sorted, and the inputs from blkid need a lot of work to get them usable. We handle this using bash process substitutions. I’m going to feed these into cat so you can copy-paste to see what they actually do.

cat <(lsblk | sort)

This one is pretty straight forward. We’re just sorting the output. The next one is a bit of a beast. I’ve stripped out the layer of escaping required by typescript below, for readability.

cat <(blkid \
  | while read dev uuid fstype; do \
    cleaned_dev=$(echo "${dev#/dev/}" | tr -d :); \
    cleaned_uuid=$(echo "${uuid#UUID=\"}" | tr -d \"); \
    cleaned_fstype=$(echo "${fstype#TYPE=\"}" | tr -d \"); \
    echo "$cleaned_dev $cleaned_uuid $cleaned_fstype"; \
  done | sort)

Example output:

nvme0n1p1 6cd50e51-cfc6-40b9-9ec5-f32fa2e4ff02 xfs
nvme0n1 PTTYPE=dos
nvme1n1 552214f7-d4d1-49b0-a4c6-8435d4473d08 xfs
nvme2n1 23846396-44cc-4798-a77e-fc9fb71f5c74 xfs
nvme3n1 c3e508a6-3917-4145-826d-5102753e0a14 xfs

This takes the output of blkid, splits it on whitespace, cleans the junk off the front of each of the device name, uuid, and fstype using a bash parameter substitution then trims the junk off the back using tr. The last step of the while loop echos out the reformatted data, with whitespace separation. Finally this is piped through sort.

Join takes these two inputs and joins them based on the device name. This is immediately piped into yet another while read. The only step in the while loop is to append a line for the /etc/fstab.

Conclusion Link to heading

I’m glad I still know how to bash stuff. I’m sad I still NEED to know how to bash stuff. And… blkid -o list would have made this substantially easier.