Infrastructure Automation using AWS CloudFormation
About Kauffman Foundation
The Ewing Marion Kauffman Foundation (sometimes simply called Kauffman Foundation) is a registered non-profit, private foundation based in Kansas City, Missouri. The foundation was founded in 1966 by Ewing Marion Kauffman, who had previously founded the drug company Marion Laboratories. The Kauffman Foundation focuses on projects that encourage entrepreneurship, support education and contribute to Kansas City civic life.
The Founder’s Education Platform (FEP) is a responsive, web-based application providing the ability for its users to engage with the educational assets of the Kauffman Foundation across a number of different systems. The content and support available to a user of the FEP will take at least four distinct forms – educational courseware, peer-to-peer discussions, blog content, and community experts. The architectural goal of the FEP project is to help users of the system find an appropriate mixture of content from these sources through an aggregated search facility. Under the hood, the infrastructure consists of multiple backend applications that serve user’s HTTP web requests, Database, Elasticsearch domain, and document storage. Applications run in the Docker environment.
As the customer base started increasing, development and release teams had a tough time releasing frequent new features and patches to production. Deployments on QA, staging, and production were manual and there was no automation process on AWS to setup a new environment to test the new features. Developers & infrastructure engineers had to spend 2-3 days of effort to setup the environment and deploy the code to test and move to production systems. This is significant overhead from a project execution perspective. Another point which always comes to mind was the disaster recovery, the possibility to lose whole or even some part of infrastructure considered as a huge problem. The last but not least issue was the control and monitor changes in the infrastructure, currently, it was hard to evaluate who did an infrastructure change when and why.
Based on all of these requirements Kauffman Foundation was looking for a solution to manage the current infrastructure in an easy way and has the following features:
- Automatic resource provisioning
- Fast infrastructure recovery in case of emergency
- Quick deploy, repeatable infrastructure for development and test needs
- Monitor external changes in AWS resources for the production environment (drift management)
- Control and manage resources ownership
- Secure, monitor and control access to the resources
- Infrastructure must include CI/CD features for application changes integration.
- Architecture must be resilient depending on application traffic (number of HTTP requests)
AWS CloudFormation lets developers and system administrators use code to provision, update, and manage a collection of related AWS resources — called a stack — in a consistent and controlled manner. There is a complete list of storage services for the project needs and Amazon CloudFront for fast content delivery. The requirements connected with continuous deployment are also covered with the services AWS CodeBuild, AWS CodePipeline.
The proposed and implemented solution consists of numbers of dependent AWS CloudFormation templates responsible for managing an appropriate part of the customer’s infrastructure:
- global resources such as Route53 hosted zone, IAM policies
- resources for data management, especially RDS database instance, Elasticsearch domain, S3 bucket
- network: resources such as VPC, IGV, route tables. subnets, etc.
- resources for continuous deployment.
- Amazon ECS cluster resources – cluster itself, Amazon ECR repository, appropriate IAM policies, log groups, Application Load Balancer with listeners, target groups and so on, Launch Configuration for AutoScalingGroup, and CloudWatch alarms
These templates are used to create production, development and test environment. Also, a blue-green deployment approach was used to create parallel production environments on AWS, enabling continuous deployment and faster time-to-market.
In terms of security, a number of policies and deployment procedures were developed, for instance, the production stack protection with update policies, IAM policies for managing access to the stack resources and stack creation/update actions. AWS CloudWatch in connection with AWS Config services is used for monitoring changes in the production environment.
The proposed and implemented solution resulted in several benefits for UC Berkeley department, for instance:
- spend less time managing resources and more time focusing on applications
After solution implementation, the development and test engineers can create the infrastructure in the “push button” manner to create the appropriate infrastructure.
- easily control and track changes in the infrastructure
Using AWS CloudFormation Change Sets allow infrastructure engineers to preview the changes in infrastructure and how these changes might impact currently running resources. Drift detection feature is used for detecting changes made outside from the CloudFormation template and infrastructure engineers take appropriate actions if necessary. In addition, AWS config is used to record configuration changes to all AWS CloudFormation stacks
- repeatable infrastructure
The development and test engineers are able to create “the same” infrastructure for their needs, it helps to obtain stable and robust test results and product with better quality.
- simplify infrastructure management
AWS CloudFormation manages dependencies between resources and allows make a change in infrastructure without keeping in mind how the set of resources needs to be configured to work together and what actions need to be done if one or more resources are changed.
pi5.cloud is a global technology consulting company at the forefront of cloud computing. Through collaboration with Amazon Web Services, we help customers embrace a broad spectrum of innovative solutions. From a migration strategy to operational excellence, cloud-native development, and immersive transformation, pi5.cloud is a full spectrum integrator.
Tell us about your project. Get a free consultation and estimate.