CI/CD with Bitbucket Pipelines and AWS ElasticBeanstalk Dockerized Deployment with docker-compose.yml + AWS Secret Manager (2023)

I’m going to share a very simple yet effective setup to create a CI/CD pipeline using Bitbucket pipelines and AWS ElasticBeanstalk for auto scaling. The whole setup consists of:

bitbucket-pipelines.yml
A python deployment script.
Part of a Terraform config to create the ElasticBeanstalk environment.
docker-compose.yml and some details about ElasticBeanstalk configuration files.

bitbucket-pipelines.yml

image: python:3.7.2

pipelines:
  custom:
    deployment-to-staging:
      - step:
          name: Package application
          image: kramos/alpine-zip
          script:
            - zip -r artifact.zip * .platform .ebextensions
          artifacts:
            - artifact.zip
      - step:
          name: Deployment
          deployment: STAGING
          trigger: automatic
          caches:
            - pip
          script:
            - curl -O https://bootstrap.pypa.io/pip/3.4/get-pip.py
            - python get-pip.py
            - python -m pip install --upgrade "pip < 21.0"
            - pip install boto3==1.14.26 jira==2.0.0 postmarker==0.13.0 urllib3==1.24.1
            - python deployment/beanstalk_deploy.py
            - deploy_env=STAGING
            - new_tag=$deploy_env$(date +_%Y%m%d%H%M%S)
            - git tag -a "$new_tag" -m "$new_tag"
            - rm -f .git/hooks/pre-push
            - git push origin --tags

For this pipeline to work there are some prerequisites. First, you need to setup deployments on Bitbucket. This is usually found in this path under your project:

/admin/addon/admin/pipelines/deployment-settings

There, you need to enable deployments and set up your staging deployment with a minimum of two environment variables within the deployment.

APPLICATION_NAME // Which is your AWS ElasticBeanstalk application name
APPLICATION_ENVIRONMENT // Which is your AWS ElasticBeanstalk environment name.

We need three other environment variables in the “Repository Variables” section of the repo:

S3_BUCKET // This is where the zipped application code is uploaded then to be deployed to Beanstalk.
AWS_SECRET_ACCESS_KEY // AWS secret access key for authentication to AWS services.
AWS_ACCESS_KEY_ID // AWS access key id for authentication to AWS services.

Every Beanstalk application can have many environments. I usually set the applications as different environments as in “Production-Application” or “Staging-Application” and then in these applications I have “Web-Environment” and “Worker-Environment” and so on.

Deployment Script

# Copyright 2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file
# except in compliance with the License. A copy of the License is located at
#
#     http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is distributed on an "AS IS"
# BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations under the License.
"""
A Bitbucket Builds template for deploying
an application to AWS Elastic Beanstalk
[email protected]
v1.0.0
"""
from __future__ import print_function
import os
import sys
from time import strftime, sleep
import boto3
from botocore.exceptions import ClientError, WaiterError

VERSION_LABEL = strftime("%Y%m%d%H%M%S")
BUCKET_KEY = os.getenv('APPLICATION_NAME') + '/' + VERSION_LABEL + \
             '-bitbucket_builds.zip'


def upload_to_s3(artifact):
    """
    Uploads an artifact to Amazon S3
    """
    try:
        client = boto3.client('s3', region_name='eu-central-1')
    except ClientError as err:
        print("Failed to create boto3 client.\n" + str(err))
        return False

    try:
        client.put_object(
            Body=open(artifact, 'rb'),
            Bucket=os.getenv('S3_BUCKET'),
            Key=BUCKET_KEY
        )
    except ClientError as err:
        print("Failed to upload artifact to S3.\n" + str(err))
        return False
    except IOError as err:
        print("Failed to access artifact.zip in this directory.\n" + str(err))
        return False

    return True


def create_new_version():
    """
    Creates a new application version in AWS Elastic Beanstalk
    """
    try:
        client = boto3.client('elasticbeanstalk', region_name='eu-central-1')
    except ClientError as err:
        print("Failed to create boto3 client.\n" + str(err))
        return False

    try:
        response = client.create_application_version(
            ApplicationName=os.getenv('APPLICATION_NAME'),
            VersionLabel=VERSION_LABEL,
            Description='New build from Bitbucket',
            SourceBundle={
                'S3Bucket': os.getenv('S3_BUCKET'),
                'S3Key': BUCKET_KEY
            },
            Process=True
        )
    except ClientError as err:
        print("Failed to create application version.\n" + str(err))
        return False

    try:
        if response['ResponseMetadata']['HTTPStatusCode'] is 200:
            return True
        else:
            print(response)
            return False
    except (KeyError, TypeError) as err:
        print(str(err))
        return False


def deploy_new_version(environment):
    """
    Deploy a new version to AWS Elastic Beanstalk
    """
    try:
        client = boto3.client('elasticbeanstalk', region_name='eu-central-1')
    except ClientError as err:
        print("Failed to create boto3 client.\n" + str(err))
        return False

    try:
        client.update_environment(
            ApplicationName=os.getenv('APPLICATION_NAME'),
            EnvironmentName=os.getenv(environment),
            VersionLabel=VERSION_LABEL,
        )
    except ClientError as err:
        print("Failed to update environment.\n" + str(err))
        return False

    waiter = client.get_waiter('environment_updated')
    try:
        waiter.wait(
            ApplicationName=os.getenv('APPLICATION_NAME'),
            EnvironmentNames=[os.getenv(environment)],
            IncludeDeleted=False,
            WaiterConfig={
                'Delay': 20,
                'MaxAttempts': 30
            }
        )
        return True
    except WaiterError:
        print('Deployment might be failed or it might be a false positive. Please check beanstalk.')
        return False


def main():
    " Your favorite wrapper's favorite wrapper "
    if not upload_to_s3('artifact.zip'):
        sys.exit(1)
    if not create_new_version():
        sys.exit(1)
    # Wait for the new version to be consistent before deploying
    sleep(5)
    if not deploy_new_version('APPLICATION_ENVIRONMENT'):
        sys.exit(1)

if __name__ == "__main__":
    main()

The deployment script is pretty much self explanatory. It gets the zipped code of the application which was created in the first step of the pipeline and creates a new beanstalk application version. Then it deploys the new version to beanstalk and waits for the status to change to environment_updated and that’s it.

You can see the line in the pipeline yml file where it calls:

python deployment/beanstalk_deploy.py

This means your script is under a folder named deployment within your source code.

Terraform for Elasticbeanstalk

The following is a section of a terraform file that I’m using to create resources on AWS. This file won’t work by itself but it can give you a general idea.

Specifically, you need to set some data and some resources before this section of the file. For example, you need to set your vpc data like this:

data "aws_vpc" "my_vpc" {
  id = "your-vpc-id"
}

So that this section can use the variable “data.aws_vpc.my_vpc”.

resource "aws_elastic_beanstalk_application" "examplewebapp_staging" {
  name        = "examplewebapp-staging-application"
  description = "Examplewebapp Staging Application"

  appversion_lifecycle {
    service_role          = "arn:aws:iam::502026590581:role/aws-elasticbeanstalk-service-role"
    max_count             = 128
    delete_source_from_s3 = true
}

resource "aws_elastic_beanstalk_configuration_template" "examplewebapp_staging_template" {
  name                = "examplewebapp-staging-template-config"
  application         = aws_elastic_beanstalk_application.examplewebapp_staging.name
  solution_stack_name = "64bit Amazon Linux 2 v3.5.5 running Docker"
  setting {
    namespace = "aws:autoscaling:launchconfiguration"
    name      = "InstanceType"
    value     = "c5.2xlarge"
  }
  setting {
    namespace = "aws:autoscaling:launchconfiguration"
    name      = "IamInstanceProfile"
    value     = "aws-elasticbeanstalk-ec2-role"
  }
  setting {
    namespace = "aws:autoscaling:launchconfiguration"
    name      = "EC2KeyName"
    value     = "examplewebapp-beanstalk-staging"
  }
  setting {
    namespace = "aws:autoscaling:launchconfiguration"
    name      = "SecurityGroups"
    value     = data.aws_security_group.default.id
  }
  setting {
    namespace = "aws:ec2:vpc"
    name      = "Subnets"
    value     = aws_subnet.private_subnet.id
  }
  setting {
    namespace = "aws:ec2:vpc"
    name      = "VPCId"
    value     = data.aws_vpc.my_vpc.id
  }
  setting {
    namespace = "aws:ec2:vpc"
    name      = "ELBSubnets"
    value     = data.aws_subnet.public_subnet.id
  }
}

resource "aws_elastic_beanstalk_environment" "examplewebapp_staging_environment" {
  name                = "examplewebapp-staging-environment"
  application         = aws_elastic_beanstalk_application.examplewebapp_staging.name
  template_name = aws_elastic_beanstalk_configuration_template.examplewebapp_staging_template.name
  tier                = "WebServer"
}

With this section of the terraform we created an Elasticbeanstalk application called examplewebapp-staging-application and an Elasticbeanstalk environment called examplewebapp-staging-environment.

docker-compose.yml and ElasticBeanstalk Configuration Files and Folders

docker-compose.yml

version: '3.8'
services:
  app:
    build:
      context: .
      dockerfile: Dockerfile
    restart: always
    env_file:
      - .env
    ports:
      - 80:80

This is a very simplified docker-compose.yml file. When ElasticBeanstalk fetches your application version, it unzips it and checks for several things.

If it finds a docker-compose.yml file it uses the file to create the containers. Here’s there’s a trick. You might have a docker-compose.yml file for your development environment and another one for the staging/production environments. Let’s say you have these files:

docker-compose.yml // for development.
docker-compose.beanstalk.yml // for staging/production deployments.

You don’t want to use the default docker-compose.yml file to deploy to elasticbeanstalk environments so you need rename your docker-compose.beanstalk.yml file to docker-compose.yml file so that beanstalk creates the actual containers.

You can achieve this with beanstalk platform hooks. All you need to do is creating a folder called .platform/hooks/prebuild in your source code and put some scripts there to be run before the docker container creation.

.platform/hooks/prebuild/01prepare.sh

cp docker-compose.beanstalk.yml docker-compose.yml

Another thing we are missing now is the .env file within the docker-compose.beanstalk.yml. Let’s say we have created our environment variables in AWS Secret Manager. Now, we need to fetch those and put them in a file called .env so that our docker compose does not break. We can update out 01prepare.sh file to do this:

cp docker-compose.beanstalk.yml docker-compose.yml
aws --region "eu-central-1" secretsmanager get-secret-value --secret-id "examplewebapp_staging" | \
  jq -r '.SecretString' | \
  jq -r "to_entries|map(\"\(.key)=\\\"\(.value|tostring)\\\"\")|.[]" > .env

This will create the .env file with the proper format and that’s it.

Debugging Problems

There’s only one certainty: Deployments will fail! There can be many reasons, some missing configuration, wrong instance type that can’t handle your app, other errors in your AWS setup and so on. Anything is possible.

When your deployment fails, ElasticBeanstalk will give you some information but that is usually not enough by itself. You’ll need to get into the EC2 instance that Beanstalk created and check the logs under /var/log.

The main file that I check to debug problems is eb-engine.log. I usually do a tail -f to that file when trying to deploy new versions at the very first setup. Once the deployments are successful you won’t need to check these logs ever again unless there’s a change in your deployment setup.

Conclusion

This is probably 95% of what you need to do to create an automated CI/CD pipeline on Bitbucket with AWS Elastic Beanstalk and AWS Secret Manager. The remaining 5% are small adjustments here and there which you’ll need to do depending on the information you get from failed deployments. I tried to give all the information but it’s possible that I forgot some minor details.

There are a lot more configuration details for Elastic Beanstalk so I suggest you deep dive into AWS docs.

Thank you for reading.

Computer Science