Introduction Link to heading

So far in this series, we have set up the basics of a Hugo blog and push the code to GitHub and bootstrapped a cdk application and used it to deployed a lambda. Then your faithless narrator wandered off for a month, and when he came back nagged you about upgrading cdk. Our next step is to deploy the rest of what was described in serving static sites with AWS. That’s going to require us to add a few more things to our cdk app, so now is a good time to talk about apps, stacks and constructs.

Caveat Lector Link to heading

I have only been working with CDK for about a year now. As a result, I have developed some opinions, but I want to temper this discussion with some humility. I could be really really wrong here. That said, this is what I think makes sense in terms of how to compose in CDK.

Apps and Stacks and Constructs Oh My! Link to heading

When writing an app, there are three major building blocks involved. The first, and easiest to think about is the app. You will probably only have one app. The app will collect all the stacks that you need to deploy and should come pretty close to matching the project you’re trying to build in both name and scope. For example, if you want to build the notes application from the amazing serverless-stack tutorial, your app would probably be named notes, and would be responsible for handling all the infrastructure involved in the notes project.

The second building block is the stack. If you have a CloudFormation background, you will be happy to hear that this maps exactly to Stacks in Cfn. Stacks are the unit of deployment. Whatever is in the same stack is deployed, updated or destroyed together. If you build large and complicated stacks, you will do a great job of hiding complexity. Until you don’t. And then you will have a blast radius at least as large as your stack.

Stacks consist of constructs, the third and final building block we will be considering. There are three flavors of constructs. The first to flavors are typically a single deployable thing in AWS. At the lowest level are the Cfn constructs. These all have the Cfn prefix. They are auto-generated and map directly to CloudFormation. Right immediately above that layer are the “high level” constructs. These constructs build on top of the lower level Cfn constructs and add the first layer of abstraction. For example, a Cfn construct might require the ARN of a KMS key. A high level construct might accept a KMS key passed into it, but by default just create create a KMS key.

The third and final layer of constructs are the pattern constructs. A pattern construct stitches together a number of other constructs to implement an architectural pattern. These represent an even higher level of abstraction and are accordingly more opinionated.

You can find a much deeper introduction in the AWS developer Blog. But this is enough for now.

What we are doing here is building a couple of pattern level constructs, and then stitching them together to build a stack.

Start With Constructs Link to heading

I generally start with constructs. For hugo-cdk, we are going to build two pattern level constructs: a static-site and a builder. Then we will pull them together into a stack. That stack can then be deployed once for every Hugo site.

The idea here is that we might want to use different implementations. So, we’ll try to keep the two constructs somewhat decoupled.

My First CodeBuild Link to heading

The first thing in our diagram is a CodeBuild. We’re going to build a construct that contains a CodeBuild and demonstrates how to pass in a number of other components. Most importantly, we’ll take a look at how easy it is to manage IAM in cdk.

Getting Started Link to heading

Well, we’re obviously going to need to know a couple of things. First off, we need to know what to build. I’ve decided to pull from GitHub, because, duh. To pull from GitHub, we need to know the name of the repo org / owner and the name of the repo. In the case of my blog, that’s ahammond and blog respectively. We’ll also need to know where to publish it. I’m going to cheat a little and ask you to just allow the following interface. I’ll explain it later. Probably. For now, it’s going to encapsulate our publishing target.

// From lib/static-site.ts
export interface IStaticSite {
  readonly bucket: s3.IBucket;
  readonly distribution: cloudfront.CloudFrontWebDistribution;
}

Ok, let’s specify some properties and the skeleton of the class.

const hugoDeb = 'https://github.com/gohugoio/hugo/releases/download/v0.74.2/hugo_0.74.2_Linux-64bit.deb';

export interface SiteBuilderProps extends cdk.StackProps {
  /**
   * The name of the repo.
   */
  githubRepo: string;
  /**
   * The name of the github owner / organization
   */
  githubOrg: string;
  /**
   * Where should master builds be published?
   */
  production: staticSite.IStaticSite;
}

export class SiteBuilder extends cdk.Construct {
  public readonly project: codebuild.Project;

  constructor(scope: cdk.Construct, id: string, props: SiteBuilderProps) {
    super(scope, id);

    const buildSpec = {
      version: '0.2',
      environment: {
        vars: {
          HUGO_ENV: 'production',
        },
      },
      phases: {
        install: {
          'runtime-versions': {
            python: 3.8,
          },
          commands: [
            // Install hugo
            `curl -L -o hugo.deb "${hugoDeb}"`,
            'dpkg -i hugo.deb',
          ],
        },
        build: {
          commands: [
            // Render the site
            'hugo -v',
          ],
        },
        post_build: {
          commands: [
            // https://docs.aws.amazon.com/codebuild/latest/userguide/build-env-ref-env-vars.html
            `if [ 'branch/master' = "$CODEBUILD_WEBHOOK_TRIGGER" ]; then
              cd public;
              aws s3 sync --delete --no-progress --sse AES256 --storage-class INTELLIGENT_TIERING . "s3://${props.production.bucket.bucketName}";
              aws cloudfront create-invalidation --distribution-id "${props.production.distribution.distributionId}" --paths '/*';
            else
              echo "not master: do not publish";
            fi`,
          ],
        },
      },
    };

    this.project = new codebuild.Project(this, 'Project', {
      projectName: props.githubRepo,
      buildSpec: codebuild.BuildSpec.fromObject(buildSpec),
      source: codebuild.Source.gitHub({
        // Credentials from CLI, search ImportSourceCredentials at https://docs.aws.amazon.com/cdk/api/latest/docs/aws-codebuild-readme.html
        owner: props.githubOrg,
        repo: props.githubRepo,
        webhookFilters: [
          // Build all the PRs to master (CI)
          codebuild.FilterGroup.inEventOf(
            codebuild.EventAction.PULL_REQUEST_CREATED,
            codebuild.EventAction.PULL_REQUEST_UPDATED,
            codebuild.EventAction.PULL_REQUEST_REOPENED,
          ).andBaseBranchIs('master'),
          // And any change to master. (CD)
          codebuild.FilterGroup.inEventOf(
            codebuild.EventAction.PUSH
          ).andBranchIs('master'),
        ],
      }),
    });
    new cdk.CfnOutput(this, 'TheProject', { value: this.project.projectArn });

Boom. There it is. Let’s break it down since that’s almost 100 LoC. We start off with a const. This is the location of the current latest Hugo release in .deb packaging. Next up is the properties interface. We require a githubOrg, a githubRepo and an instance of the IStaticSite interface we mentioned above.

Then we define the SiteBuilder class which is just another construct. The constructor is tweaked to require a SiteBuilderProps. We define an enormous blob that is the buildSpec. Let’s walk through the details of that blob. First up we have the version of buildSpec that we’re writing to. This has been0.2 for a while, and no, it’s not a version for you to increment to track versioning of your build.

Next up, we have the environment variables. You could simply manage this below in the commands, but this is a tiny bit cleaner. This is also where you would fetch secrets from either SecretsManager or SSM Parameter Store.

And the last chunk of the buildSpec is the phases. With the exception of the runtime-versions stanza in install, these consist of a name and then an array of commands which will be executed by a single bash instance. The exact same bash instance. So, state persists between phases. The only thing that splitting things into different phases gives you is some blue text dividing up the output.

I’m going to skip down to the post_build phase because it’s really obvious what the others are doing. In post_build, we look at the CODEBUILD_WEBHOOK_TRIGGER env var to decide if we are executing because of a change to the master branch, or, well… not. It’s worth noting that just manually triggering a build of the master branch from the console will NOT meet this criteria.

When we are on master, we execute our CD steps. This is simply to first run an aws s3 sync to upload the build files to the S3 bucket, and second, run an aws cloudfront create-invalidation to trigger a cache invalidation in CloudFront.

Once we have a buildSpec, it’s time to declare the actual project. You can see the typical three parameter instantiation of a cdk construct here: this, 'Project', {properties}. The properties of interest are buildSpec and the source.

We’re using a GitHub source for this project. Configuration is pretty straight forward. We need to feed it the owner (aka org) and the repo name. The only cleverness is the webhookFilters. We want to build every change to every PR which targets the master branch. This will provide our CI. We also want to build any change to the master branch, and then deploy it. This will provide our CD.

Bootstrapping CodeBuild’s GitHub OAuth Token Link to heading

This was a huge pain point for me. If you decided to run ahead and deploy stuff and see the following error message:

Failed to call CreateWebhook, reason: Could not find access token for server type github

then… you need to bootstrap. Start out by creating a Personal Access Token (PAT) at GitHub

Next, head over to the CodeBuild console and create a new project. Name it junk, because you’re never even going to save it.

Select GitHub as your source provider.

Choose OAuth and add the PAT you just generated.

Connect, and clicketty clicketty click until you get all the way through OAuth pairing your account.

You may now abandon your “junk” project.

I’m a pretty hardcore IaC guy. Writing this section pained me.

Submodules and Escape Hatches Link to heading

Git submodules suck. That said, there are times when they are the right solution. For example, submodules are the only reasonable way to use a Hugo themes. For those of you who are about to neckbeard about git subtrees, I don’t even. The good news here is that this provides us an opportunity to demo cdk’s “escape hatch”. Take a look through the high level construct for Project and you will note the absence of support for submodules. However, if you dig into the underlying CfnProject low level construct, you can find the property.

// Submodules, please.
const cfnProject = this.project.node.defaultChild as codebuild.CfnProject;
cfnProject.addPropertyOverride('Source.GitSubmodulesConfig.FetchSubmodules', 'True');

What we’re doing here is digging into the object we instantiated when we declared the high level Project construct. We take the defaultChild node and cast it as a CfnProject. Remember the Cfn low level constructs we discussed above? Here we are leveraging the Cfn construct to manipulate the rendering of the CloudFormation. In this case we are setting a somewhat deeply nested property. As cdk matures, I expect that there will be less and less need to leverage the escape hatch.

Writing Less IAM Link to heading

The great thing about IAM is that it largely works. That said, it sure isn’t fun to write. cdk provides a couple of ways to manage this. You can write policy statements directly.

this.project.addToRolePolicy(
  new iam.PolicyStatement({
    resources: [
      props.production.bucket.bucketArn,
      `${props.production.bucket.bucketArn}/*`,
    ],
    actions: [
      's3:ListBucket',
      's3:GetObject',
      's3:PutObject',
      's3:DeleteObject',
    ],
  }),
);

This works, but… is still involves writing a lot of IAM. The following achieves the same effect as the above 14 LoC in a single LoC.

// eslint-disable-next-line @typescript-eslint/no-non-null-assertion
props.production.bucket.grantReadWrite(this.project.role!);

The Project will implicitly declare it’s role and configure it correctly. You can then access it via this.project.role… except that the type there is readonly role?: iam.IRole. Typescript allows us to make a non-null assertion using !. This isn’t ideal because it is effectively telling Typescript “trust me, this will not be null”. This is why we have the eslint entry above it. This is something that I’d love to fix in cdk: providing a getter which always returns the role.

The final permission we’ll need to grant is to the CloudFront distribution. We need the builder to be able to trigger a cache invalidation.

props.production.distribution.grant(
  this.project.role!,
  'cloudfront:CreateInvalidation',
);

This follows the pattern seen in other high-level constructs, but… nope! There are no grant methods in CloudFrontWebDistributions. We’ll have to do it the less elegant way:

this.project.addToRolePolicy(
  new iam.PolicyStatement({
    resources: [
      `arn:aws:cloudfront::${props.env?.account}:distribution/${props.production.distribution.distributionId}`,
    ],
    actions: ['cloudfront:CreateInvalidation'],
  }),
);

Conclusion Link to heading

That’s it, we’ve built the builder. The full code can be found at hugo-cdk builder. In the process, we discovered a number of minor inconsistencies and limitations in cdk. Remember, cdk is new and still under active development. More importantly, we addressed some of the workarounds for these challenges.

Tune in next week. Same bat time, same bat channel.