When I architected my SaaS’ deployment platform and delivery pipeline, I specifically sought to:

1. maximize Aspects of a High Performance Software Delivery Process
2. minimize components and platforms and offload as much undifferentiated, heavy lifting to focus my team’s very limited capacity on product development versus say, operations

The Serverless Framework is the foundation of the product delivery pipeline:

The Serverless Framework bills itself as ‘the complete solution for building & operating serverless applications’ covering development, deployment, and monitoring use cases.

My take is the Serverless Framework delivers on this promise, at least for the development and deployment use cases. I have not used the monitoring yet.

Serverless is a true application development framework that:

• promotes specific development patterns
• uses enabling constraints for how those functions are developed
• provides a suite of commands for development and testing — both local and remote
• integrates with the Cloud providers native operations and observability services

Serverless makes a ton of decisions for you and in return you get tooling that will let you describe a ‘service’ implemented as a set of Lambda functions. Events emitted by integration resources such as SNS topics or API Gateways are straightforward to connect to your Lambda function definitions.

Here’s a snippet from my serverless yaml showing the publishing_dispatcher function is subscribed to an SNS topic:

functions:
  resource_access_inventory:
    handler: handler.resource_access_inventory
    timeout: 600
  publishing_dispatcher:
    handler: handler.publishing_dispatcher
    timeout: 45
    events:
      - sns: report-generated-${self:provider.stage}
... snip ...

And that SNS topic is not any old topic, it’s one whose name is customized via Serverless’ (basic) variable interpolation system. In my setup, stage is a value like: prod, dev, or dev-$commithash. There’s a significant amount of work underneath the covers to make that look simple.

Every resource deployed in this application is namespaced by this stage variable so that we are able to deploy a full, independent set of resources for each commit and test them in isolation. Resources include: Lambda functions, SNS topics, and (especially) IAM roles & policies. Serverless describes and deploys these resources using CloudFormation, and you can embed additional ‘custom’ resources in your application’s deployment descriptor.

Here’s how this experiment is shaking out.

Spoiler: I love it.

The Good

We have a fully-automated continuous delivery setup with isolated unit, integration, and functional testing of our codebase. I think the delivery performance results are great, let’s break this down in terms of DORA’s model:

Deployment frequency

Our team has the option to deploy changes to production that pass automated testing on the task branch and our release branch (aka master).

Over the past two weeks, we mostly deployed one or zero changes per day as stories or small improvements completed. However, there was one day where we deployed 9 changes in a single day.

Hot

Performance level: Elite

Lead time for changes

The lead time for a commit to the release branch to its release in production is about 15 minutes.

HOT

I’ve never had it this good, especially for the level of safety that we have baked into the delivery pipeline.

Performance level: Elite

Time to restore service

We haven’t had to enough of this yet to offer a measurement. However, I think this indicates the Serverless framework is doing a good job generating and managing the CloudFormation stack.

Performance level: TBD

Change failure rate

So far our data indicates less than a 10% change failure rate. The Serverless framework contributes to this through its:

• robust infra-as-code support
• support for deploying isolated resources and testing functions via serverless invoke --function

The deployment jobs in the pipeline each include a functional test that perform a quick sanity test of the deployment. No matter what the stage ( prod, dev, or dev-$commithash), we get immediate feedback on whether things are working. The primary reason we don’t have a lot of prod change failure rate data is that each change passes this hurdle multiple times in development.

Performance level: Elite

The Bad

There’s nothing in this setup that I would call bad for our small team and its rapid product development goals. I really think Serverless and CircleCI have hit the mark.

I can spot a few things that orgs with multiple product teams and existing delivery processes might be conflicted with:

• what should we standardize and share across teams, if anything?
• how do we integrate our security principals and processes with this?
• what should we manage with our existing Terraform/CloudFormation/$IaCTool approach versus a serverless-focused tool?

My generic answers to those questions are to:

• Develop and publish a security and governance standards first and let teams meet those standards using with their choice of tools.
• One of the best places to share tooling is in the ‘Build Tools’ component at the top of the dependency stack; we are doing this primarily via a docker image and this could be shared across teams. Serverless and similar tools are extensible via plugins, so this is also an option.
• Teams developing serverless applications should default to using a serverless-focused tool or recognize they will not reach elite delivery performance without investing a significant amount of time reimplementing what those tools do out of the box.

If you’re interested in discussing this with me and your context, hit reply and let’s chat.

The YAML

The weakest parts of this setup that I would change if I had a good alternative are:

1. eliminate use of yaml in general
2. replace CloudFormation’s YAML/JSON lang

YAML

This delivery pipeline uses yaml in two (three?) places: the CircleCI build and Serverless deployment definition, which permits embedded CloudFormation.

I’ve burned a significant amount of time on indentation issues and trying to use ‘advanced’ yaml features like aliases and anchors to DRY-out the code (yes, this is build and infrastructure code, not configuration). Heads-up: yaml anchors and aliases are parser-specific features and aliases lists (like say a series of deployment steps) is not supported — at least not with CircleCI’s parser.

So here’s our validate-build Makefile target:

.PHONY: validate-build
validate-build:
	yamllint -c .yamllint-conf.yml serverless.yml
	serverless print > /dev/null
	yamllint -c .yamllint-conf.yml .circleci/config.yml
	circleci config validate

My solutions at this point aren’t that satisfying:

First, use yamllint to validate yaml documents. This is fine, though I find it a bit ironic that this is the first time I’ve had to bring a third-party linter into my delivery ecosystem. Context: I’m one of the people who liked validating XML docs using DTDs and or XSD.

Second, invoke a serverless or circleci command that will trigger validation against the program’s internal types. Everyone wants to use yaml, but no one wants to publish a schema.

I sure would be interested in a build and deployment system that used Hashicorp’s HCL. I was really excited to try GitHub Actions until they removed support for defining actions in HCL. I guess I’ve been spoiled for the past several years with HCL 1.0’s basic but useful modeling and expression language.

CloudFormation, the Language

This product has been my first “deep” foray into CloudFormation, AWS’ infrastructure as code language and service. CloudFormation the service is great, however…

Coming from Terraform, I feel like CloudFormation is a pretty weak language that is missing features like a reasonable approach to control flow, standard library of functions, and local variables/datasets.

I have settled on outlining my CloudFormation code in yaml, but specify anything complicated by embedding the json form. I would love to see an HCL or maybe Python CDK option.

That said, “it’s fine.” Maybe it’ll grow on me.

Takeaways

With this continuous delivery setup k9 is ready to Go Fast, Safely.

• The Serverless Framework and CircleCI tools, along with Make and Docker, have enabled us to implement the principles of continuous delivery easily.
• This continuous delivery pipeline lets us achieve Elite levels of Software Delivery Performance.
• It would be challenging to replicate this level of serverless delivery capability with a custom framework, even built around a best-in-class infrastructure as code tool like Terraform. But there is nuance to minimizing delivery pipelines and platforms.

Stephen

#NoDrama

KEEP LEARNING

 

 

 

Receive #NoDrama articles in your inbox whenever they are published. Reply to Stephen and the QualiMente team when you want to dig deeper into a topic.

SUBSCRIBE!

You have Successfully Subscribed!