DevOps: Slashing Deployment Cost by 90% using IaC

One of our clients (a company handling geo-spatial data) has built a Web Data Application that contains components such as API, Databases, Authentication System and Frontend — the works. The application had grown its size gradually and hence what started as a simple application with a HTML page was now a full-blown web data application. With size came scaling and deployment challenges — deployment of the app into a setup took several hours followed by several days of debugging.

We at Pevatrons fixed this nightmare for our client with scalable and maintainable DevOps solution with Infrastructure as Code. The solution resulted in deployment effort reducing to one-tenth of that of the legacy system. The rest of the post describes the problem and the solution.

System Architecture — before

The legacy system had the following components

Backend Server: It is an EC2 instance where a Python Flask Server is running along with a MySQL database server.
Authentication System: The clients use Auth0 for their authentication and user management
Frontend: This is a Single Page Application (SPA) built on React js and Node js. The assets of the single-page application such as logos, Javascript files, CSS and an index.html is all stored in AWS S3.
Amazon CloudFront: This connects the users to the application, for the front-end this helps in serving the assets stored in AWS S3, for the backend API this serves as a proxy and passes the request to the backend API hosted in an EC2 Instance.

Limitations of the legacy architecture

The backend components especially the API and the MySQL are not easily scalable as to make it scalable they would need to start a new EC2 instance install Python, Flask and the other API dependencies, figure out how to scale MySQL etc. Even if all this is achieved deployability becomes a nightmare as they would need to manually sync the two APIs for code changes.
Setting up another environment say for Staging or Testing means that all this infrastructure needs to be set up all over again which might take a long time to configure and human error.

It is not all that bad from a scalability point of view the following components are highly scalable and designed well

Authentication System: Since they use Auth0 the scale is automatically managed by Auth0.
The frontend app: Since the assets are in AWS S3 and served by AWS CloudFront no need to worry about the scale. They also have a CI/CD pipeline that automatically builds and pushes the latest front-end code changes to AWS S3.

The solution

Pevatrons’ team of expert data engineers after evaluating the architecture and after getting a thorough understanding of client needs decided to improve on the limitations of the current architecture.

Make the Backend scalable automatically.
Make the system easily deployable so new environments can be deployed and ready in a few hours.

Making the Backend Scalable

Decouple the services in the Backend EC2 Instance: we decided to have a separate infrastructure for the API Server and the Database Server and scale them independently based on their needs.
Use AWS ECS to scale the API: Since the API service was custom-built using Python Flask we decided to use AWS ECS and the first step to achieve this was to containerise the API using Docker and upload the image to AWS ECR.
Use AWS Aurora Serverless for the DB: We were looking for an alternative to MySQL DB that is fully compatible with it but easily scalable and since our clients were already on AWS using Aurora Serverless became a no-brainer.

Deploy all the infrastructure using AWS CloudFormation

Setting up the entire application manually would need a lot of time and effort, and it made sense to automate this tedious task by using Infrastructure as a Code (IaC) service. The client was on AWS and hence the natural choice for IaC was CloudFormation.

Deep Dive into deploying infrastructure using AWS CloudFormation

AWS CloudFormation or any Infrastructure as a code service allows you to set infrastructure by defining the infrastructure in some kind of a file, in the case of AWS CloudFormation we have defined the infrastructure using a YAML file.

Before we begin to write the file, it is necessary to design the entire architecture of the system and its needs

We wanted to answer the following questions before implementing our architecture.

What components can we not configure or deploy using AWS CloudFormation?

While it was clear that we could deploy all the AWS components using cloud formation, how do we configure something like Auth0 where we would need to create a new application for a new environment in Auth0?

We will configure the Auth0 service manually.We decided to make a template that clearly defines steps and instructions to perform in Auth0, this made it very clear for the Ops team what to do and to do it in a reliable error-free manner.

When the AWS Aurora service starts for the first time, how do we populate the data?

We will store the database and table structures in an SQL dump in AWS S3, and populate the Aurora DB manually for the first time. We would also fill in the necessary data as needed.

How to backup and restore data when something goes wrong in AWS Aurora?

We decided to use AWS Provided DB snapshot and recovery. We will periodically take the backup and restore it when something goes wrong.

When a disaster strikes it is necessary to recover as soon as possible. So we decided to document and explain the process to stakeholders so that the process is well understood and the team can recover the database as soon as possible.

How to deal with network components such as VPC and Subnet?

In our case, we decided that rather than creating a new VPC and Subnet every time, we would use an already-created VPC and subnet.

When to Scale up and Scale down the backend services?

Our API service was CPU intensive so we decided to scale up and scale down based on CPU usage of the ECS containers.

It is not easy to arrive at these numbers so we did a load test and then after various iterations we decided on a certain percentage.

How to securely store the Auth0 and DB credentials and share them with the API Service?

We decided to use AWS Secrets Manager to do this as it can do all the necessary things we need.

What kind of load balancer to use when there are multiple instances of API nodes running?

We decided to use a network load balancer as our application is stateless, and we do not need any additional features that the Application Load Balancer provides such as routing to a particular node based on resource path etc.

What are the resource requirements for a single API node running in ECS?

We put our API service under high load, low load and average load situations and after a few iterations, we decided on the necessary CPU cores and memory requirements. We used this to setup the task definition in AWS ECS

How to deploy the code changes to these newly configured services?

For the front end, the team already had an existing CI/CD setup using Jenkin’s which would still work with a few changes we had to make a change so that for different branches of code (environment) it would upload the build to a different S3 location.

For the back end, we did not have an existing CI/CD setup so we started building one from scratch, below is the flow for the newly created CI/CD setup for the backend

How do we monitor the cost of all the resources being deployed?

We decided to attach the following tags to all the resources deployed using cloud formations

Environment: The deployment environment such as staging, testing, QA and Production.
Owner: The team that is responsible for managing the deployment.
Application: The name of the application being deployed, used to differentiate this application from various applications running in the organisation.

After this, you can use AWS Cost Explorer to find the cost by tags.

Putting it all together

Based on all this we arrived at things that need to be configurable while deploying infrastructure using Cloud formation, they are as follows

VPC ID
Subnet ID
Access Key for Auth0
Secret Key for Auth0
Aurora DB Username
Aurora DB password
API Scale UP CPU Percentage
API Scale Down CPU Percentage
Maximum and Minimum nodes of API Server’s to run.
ECR URI to the container of the Backend API
Tags for Resource monitoring such as Environment, Owner, Application
The Site CNAME to link it to Cloudfront.
The S3 path to the directory containing front-end assets.

All the above we put it in Cloudformation as a parameter so that the Operations team can fill it as needed for deployment and for later as configuration.

Implementing all the above led to architecture down below

The new scalable architecture with CloudFormation

After implementing all the solutions mentioned above comes the above new architecture.

The deployment process of the entire application

Create a new application and set the necessary things in Auth0 to get the Access key and Secret Key. (The process instruction is already in the document so easy to implement)
Set the CI/CD for the front end and upload a build to an S3 path.
Set the CI/CD for the back end and build and upload the container to AWS ECR.
Fill the necessary items in the Cloudformation template and then deploy the infrastructure to AWS.
Load the necessary data to AWS Aurora DB.

That’s it, with these few steps that are well documented the deployment takes at max a few hours compared to a few days before and the application is fully scalable and robust as well.

Advantages of the new Architecture

Every single component of the system is now scalable unlike before when the backend API and the database server were not.
We have built our AWS CloudFormation template in such a way that all resources created by it are automatically tagged, so cost monitoring is now taken care of. With this implementation, we can view the costs by component and environment.
Secrets such as the Auth0 access key and secret key and credentials to connect to Amazon Aurora are securely stored in AWS Secrets Manager. When the backend API container is started, the container pulls the necessary secrets by accessing the secret manager.
All the AWS Infrastructure and configuration can be managed in a single place using AWS CloudFormation.
Setting up a new environment for the application which used to take days now takes only a few hours (with all non-AWS setup). Setting up only AWS components now takes minutes.
Deployment of code with new changes is now easy with the CI/CD systems available for both front-end and back-end.

Conclusion

Initially, there would be a feeling that the deployment process is not that difficult but as you add components to your system and scalability as well there is a very good chance that deployment quickly becomes a scary process.

But as we showed in this post, if you ask the right questions, understand the requirements of the system and know how to design and implement Infrastructure as a Code service for your application the entire process can become a far smoother experience. If you have any such projects where we can help feel free to contact us by visiting our website pevatrons.net. Also we would be happy to hear from you regarding any such experiences you had.