Gotchas: Elastic Beanstalk and Cross-Account CodePipeline with CloudFormation
Building a cross-account continuous delivery pipeline for a simple Spring Boot Elastic Beanstalk app using CodePipeline and CloudFormation.
This post will not provide step-by-step instructions for provisioning the AWS infrastructure—hopefully the source code is reasonably self-explanatory. Instead, this post hopes to identify and document some common gotchas.
I have opted for a solution that deploys infrastructure then application code in a single pipeline.
Prerequisites for using recreating the example are documented in the GitHub project.
The trickiest thing about doing cross-account deployments with AWS is granting enough permissions to the relevant services while still following the principle of least privilege. For me, in practice, this sometimes meant over-allocating permissions while developing and gradually removing while refactoring, often by trial and error in both scenarios.
Order of Resource Creation
The order in which I have chose to create resources can be understood from the
go script. The
must be created last and the
dummyPreReqs stack must be created before
dummyIam. Reasons for this include but are not
- The S3 bucket that acts as the artifact store for the pipeline and its associated customer master key need to exist before the pipeline can reference and create roles with permissions to access them
- The creation of
dummyPreReqsinvolves adding dynamically adding values to the parameters of
- The creation of
dummyElasticBeanstalkinvolves adding dynamically adding values to the parameters of
dummyPipeline—the pipeline needs to know the application and environment to which it will deploy
The same rules are reversed in the
Artifact S3 Bucket (tools account)
Storage to back the pipeline.
Customer Master Key (CMK) in Key Management Service (KMS) (tools)
When using CodePipeline in a single environment, CodePipeline can implicitly use the account’s default KMS key for encryption and decryption. However, whenever sharing artifacts between accounts, we need to define our own key in our tools account and permit our test account to use it.
Initial Creation of Elastic Beanstalk Application and Environment
To deploy your own code to Elastic Beanstalk via a CI/CD pipeline, CodePipeline needs to know the name of the application and environment. This is achieved by initially creating an Elastic Beanstalk stack from a template without an application version SourceBundle. This results in the default app being deployed the first time your Elastic Beanstalk resources are created:
./go script uses the application and environment names created in the initial creation of the Elastic Beanstalk
stack to connect the CodePipeline deployment stage with the correct app, meaning subsequent deployments will use
artifacts built by the CodeBuild project (and ultimately sourced from your GitHub repository).
Implicit Stack Creation
CloudFormation and Elastic Beanstalk work together to create an auxiliary stack with all the scaffolding infrastructure
that Elastic Beanstalk needs. I have defined close to the bare minimum for creating an Elastic Beanstalk app and
The auxiliary stack will create a default load balancer security group allowing ingress access from anywhere and This security group is seemingly deleted upon the first actual deployment to Elastic Beanstalk (when the default app is replaced by my own, sourced from GitHub). At this point, the EC2 instances associated with the Elastic Beanstalk app have their security group replaced with a self-referential security group that allows traffic from and to any other resource within the account’s default VPC.
I haven’t been able to find definitive reason for why this happens. I am assuming that it is a safeguarding measure to prevent users from accidentally exposing their own application to whole internet.
I found this Stack Overflow answer very helpful.
Reasoning for Pipeline Stages
Elastic Beanstalk by default expects the application to be listening on port 5000. Opting for convention rather
than any additional configuration,
server.port=5000 is configured in
Pipeline Fails First Time
As soon as the pipeline has been successfully provisioned, an execution will be kicked off. This first run will fail at
the Source stage because the CodePipeline and CodeBuild roles will not yet have been added as principals to the CMK
(this happens as the very last step in the
./go script) and so will not be able to encrypt and store the
artifact from GitHub. Clicking “Release change” or pushing a change to GitHub once the
dummyPreReqs stack has been
updated should cause Source to succeed.
./destroy to automate the process of tearing everything down while ensuring order. If you want to delete
things manually via the AWS console, be aware of the following:
- Remove the
dummyIamstack last in the test environment (it contains IAM resources that are needed to delete the
dummyElasticBeanstalkstack and associated automatically-created stack)
- Empty the artifact bucket (tools account) before trying to delete the
No serious thoughts around security, VPC or custom domain configuration. Or proper logging. Or lots of other important stuff.