When we use Kubernetes deployments to deploy our pod workloads, it is simple to scale the number of replicas used by our applications up and down using the kubectl scale command. However, if we want our applications to automatically respond to changes in their workloads and scale to meet demand, then Kubernetes provides us with Horizontal Pod Autoscaling.
This article is an excerpt taken from the book Kubernetes on AWS written by Ed Robinson. In this book, you will start by learning about Kubernetes’ powerful abstractions – Pods and Services – that make managing container deployments easy.
Horizontal Pod Autoscaling allows us to define rules that will scale the numbers of replicas up or down in our deployments based on CPU utilization and optionally other custom metrics. Before we are able to use Horizontal Pod Autoscaling in our cluster, we need to deploy the Kubernetes metrics server; this server provides endpoints that are used to discover CPU utilization and other metrics generated by our applications.
In this article, you will learn how to use the horizontal pod autoscaling method to automatically scale your applications and to automatically provision and terminate EC2 instances.
Before we can make use of Horizontal Pod Autoscaling, we need to deploy the Kubernetes metrics server to our cluster. This is because the Horizontal Pod Autoscaling controller makes use of the metrics provided by the metrics.k8s.io API, which is provided by the metrics server.
While some installations of Kubernetes may install this add-on by default, in our EKS cluster we will need to deploy it ourselves.
There are a number of ways to deploy add-on components to your cluster:
Start by cloning the configuration from GitHub. The metrics server began supporting the authentication methods provided by EKS in version 0.0.3 so make sure the manifests you have use at least that version.
You will find a number of manifests in the deploy/1.8+ directory. The auth-reader.yaml and auth-delegator.yaml files configure the integration of the metrics server with the Kubernetes authorization infrastructure. The resource-reader.yaml file configures a role to give the metrics server the permissions to read resources from the API server, in order to discover the nodes that pods are running on. Basically, metrics-server-deployment.yaml and metrics-server-service.yaml define the deployment used to run the service itself and a service to be able to access it. Finally, the metrics-apiservice.yaml file defines an APIService resource that registers the metrics.k8s.io API group with the Kubernetes API server aggregation layer; this means that requests to the API server for the metrics.k8s.io group will be proxied to the metrics server service.
Deploying these manifests with kubectl is simple, just submit all of the manifests to the cluster with kubectl apply:
$ kubectl apply -f deploy/1.8+
You should see a message about each of the resources being created on the cluster.
If you are using a tool like Terraform to provision your cluster, you might use it to submit the manifests for the metrics server when you create your cluster.
Before we continue, we should take a moment to check that our cluster and the metrics server are correctly configured to work together.
After the metrics server is running on your cluster and has had a chance to collect metrics from the cluster (give it a minute or so), you should be able to use the kubectl top command to see the resource usage of the pods and nodes in your cluster.
Start by running kubectl top nodes. If you see output like this, then the metrics server is configured correctly and is collecting metrics from your nodes:
$ kubectl top nodes NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% ip-10-3-29-209 20m 1% 717Mi 19% ip-10-3-61-119 24m 1% 1011Mi 28%
If you see an error message, then there are a number of troubleshooting steps you can follow.
You should start by describing the metrics server deployment and checking that one replica is available:
kubectl -n kube-system describe deployment metrics-server
If it is not, you should debug the created pod by running kubectl -n kube-system describe pod. Look at the events to see why the server is not available. Make sure that you are running at least version 0.0.3 of the metrics server.
If the metrics server is running correctly and you still see errors when running kubectl top, the issue is that the APIservice registered with the aggregation layer is not configured correctly. Check the events output at the bottom of the information returned when you run kubectl describe apiservice v1beta1.metrics.k8s.io.
One common issue is that the EKS control plane cannot connect to the metrics server service on port 443.
Once the metrics server has been installed into our cluster, we will be able to use the metrics API to retrieve information about CPU and memory usage of the pods and nodes in our cluster. Using the kubectl top command is a simple example of this.
The Horizontal Pod Autoscaler can also use this same metrics API to gather information about the current resource usage of the pods that make up a deployment.
Let’s look at an example of this; we are going to deploy a sample application that uses a lot of CPU under load, then configure a Horizontal Pod Autoscaler to scale up extra replicas of this pod to provide extra capacity when CPU utilization exceeds a target level.
The application we will be deploying as an example is a simple Ruby web application that can calculate the nth number in the Fibonacci sequence, this application uses a simple recursive algorithm, and is not very efficient (perfect for us to experiment with autoscaling). The deployment for this application is very simple. It is important to set resource limits for CPU because the target CPU utilization is based on a percentage of this limit:
deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: fib
labels:
app: fib
spec:
selector:
matchLabels:
app: fib
template:
metadata:
labels:
app: fib
spec:
containers:
- name: fib
image: errm/fib
ports:
- containerPort: 9292
resources:
limits:
cpu: 250m
memory: 32Mi We are not specifying a number of replicas in the deployment spec; when we first submit this deployment to the cluster, the number of replicas will therefore default to 1. This is good practice when creating a deployment where we intend the replicas to be adjusted by a Horizontal Pod Autoscaler, because it means that if we use kubectl apply to update the deployment later, we won’t override the replica value the Horizonal Pod Autoscaler has set (inadvertently scaling the deployment down or up).
Let’s deploy this application to the cluster:
kubectl apply -f deployment.yaml
You could run kubectl get pods -l app=fib to check that the application started up correctly.
We will create a service, so we are able to access the pods in our deployment, requests will be proxied to each of the replicas, spreading the load:
service.yaml
kind: Service
apiVersion: v1
metadata:
name: fib
spec:
selector:
app: fib
ports:
- protocol: TCP
port: 80
targetPort: 9292 Submit the service manifest to the cluster with kubectl:
kubectl apply -f service.yaml
We are going to configure a Horizonal Pod Autoscaler to control the number of replicas in our deployment. The spec defines how we want the autoscaler to behave; we have defined here that we want the autoscaler to maintain between 1 and 10 replicas of our application and achieve a target average CPU utilization of 60, across those replicas.
When CPU utilization falls below 60%, then the autoscaler will adjust the replica count of the targeted deployment down; when it goes above 60%, replicas will be added:
hpa.yaml
kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta1
metadata:
name: fib
spec:
maxReplicas: 10
minReplicas: 1
scaleTargetRef:
apiVersion: app/v1
kind: Deployment
name: fib
metrics:
- type: Resource
resource:
name: cpu
targetAverageUtilization: 60 Create the autoscaler with kubectl:
kubectl apply -f hpa.yaml
The kubectl autoscale command is a shortcut to create a HorizontalPodAutoscaler. Running kubectl autoscale deployment fib --min=1 --max=10 --cpu-percent=60 would create an equivalent autoscaler.
Once you have created the Horizontal Pod Autoscaler, you can see a lot of interesting information about its current state with kubectl describe:
$ kubectl describe hpa fib Name: fib Namespace: default CreationTimestamp: Sat, 15 Sep 2018 14:32:46 +0100 Reference: Deployment/fib Metrics: ( current / target ) resource cpu: 0% (1m) / 60% Min replicas: 1 Max replicas: 10 Deployment pods: 1 current / 1 desired
Now we have set up our Horizontal Pod Autoscaler, we should generate some load on the pods in our deployment to illustrate how it works. In this case, we are going to use the ab (Apache benchmark) tool to repeatedly ask our application to compute the thirtieth Fibonacci number:
load.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: fib-load
labels:
app: fib
component: load
spec:
template:
spec:
containers:
- name: fib-load
image: errm/ab
args: ["-n1000", "-c4", "fib/30"]
restartPolicy: OnFailure This job uses ab to make 1,000 requests to the endpoint (with a concurrency of 4). Submit the job to the cluster, then observe the state of the Horizontal Pod Autoscaler:
kubectl apply -f load.yaml watch kubectl describe hpa fib
Once the load job has started to make requests, the autoscaler will scale up the deployment in order to handle the load:
Name: fib Namespace: default CreationTimestamp: Sat, 15 Sep 2018 14:32:46 +0100 Reference: Deployment/fib Metrics: ( current / target ) resource cpu: 100% (251m) / 60% Min replicas: 1 Max replicas: 10 Deployment pods: 2 current / 2 desired
The metrics server provides APIs that the Horizontal Pod Autoscaler can use to gain information about the CPU and memory utilization of pods in the cluster.
It is possible to target a utilization percentage like we did for the CPU metric, or to target the absolute value as we have here for the memory metric:
hpa.yaml
kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta1
metadata:
name: fib
spec:
maxReplicas: 10
minReplicas: 1
scaleTargetRef:
apiVersion: app/v1
kind: Deployment
name: fib
metrics:
- type: Resource
resource:
name: memory
targetAverageValue: 20M The Horizonal Pod Autoscaler also allows us to scale on other metrics provided by more comprehensive metrics systems. Kubernetes allows for metrics APIs to be aggregated for custom and external metrics.
Custom metrics are metrics other than CPU and memory that are associated with a pod. You might for example use an adapter that allows you to use metrics that a system like Prometheus has collected from your pods.
This can be very beneficial if you have more detailed metrics available about the utilization of your application, for example, a forking web server that exposes a count of busy worker processes, or a queue processing application that exposes metrics about the number of items currently enqueued.
External metrics adapters provide information about resources that are not associated with any object within Kubernetes, for example, if you were using an external queuing system, such as the AWS SQS service.
On the whole, it is simpler if your applications can expose metrics about resources that they depend on that use an external metrics adapter, as it can be hard to limit access to particular metrics, whereas custom metrics are tied to a particular Pod, so Kubernetes can limit access to only those users and processes that need to use them.
The capabilities of Kubernetes Horizontal Pod Autoscaler allow us to add and remove pod replicas from our applications as their resource usage changes over time. However, this makes no difference to the capacity of our cluster. If our pod autoscaler is adding pods to handle an increase in load, then eventually we might run out of space in our cluster, and additional pods would fail to be scheduled. If there is a decrease in the load on our application and the pod autoscaler removes pods, then we are paying AWS for EC2 instances that will sit idle.
When we created our cluster in Chapter 7, A Production-Ready Cluster, we deployed the cluster nodes using an autoscaling group, so we should be able to use this to grow and shrink the cluster as the needs of the applications deployed to it change over time.
Autoscaling groups have built-in support for scaling the size of the cluster, based on the average CPU utilization of the instances. This, however, is not really suitable when dealing with a Kubernetes cluster because the workloads running on each node of our cluster might be quite different, so the average CPU utilization is not really a very good proxy for the free capacity of the cluster.
Thankfully, in order to schedule pods to nodes effectively, Kubernetes keeps track of the capacity of each node and the resources requested by each pod. By utilizing this information, we can automate scaling the cluster to match the size of the workload.
The Kubernetes autoscaler project provides a cluster autoscaler component for some of the main cloud providers, including AWS. The cluster autoscaler can be deployed to our cluster quite simply. As well as being able to add instances to our cluster, the cluster autoscaler is also able to drain the pods from and then terminate instances when the capacity of the cluster can be reduced.
Deploying the cluster autoscaler to our cluster is quite simple as it just requires a simple pod to be running. All we need for this is a simple Kubernetes deployment.
In order for the cluster autoscaler to update the desired capacity of our autoscaling group, we need to give it permissions via an IAM role. If you are using kube2iam, we will be able to specify this role for the cluster autoscaler pod via an appropriate annotation:
cluster_autoscaler.tf
data "aws_iam_policy_document" "eks_node_assume_role_policy" {
statement {
actions = ["sts:AssumeRole"]
principals {
type = "AWS"
identifiers = ["${aws_iam_role.node.arn}"]
}
}
}
resource "aws_iam_role" "cluster-autoscaler" {
name = "EKSClusterAutoscaler"
assume_role_policy = "${data.aws_iam_policy_document.eks_node_assume_role_policy.json}"
}
data "aws_iam_policy_document" "autoscaler" {
statement {
actions = [
"autoscaling:DescribeAutoScalingGroups",
"autoscaling:DescribeAutoScalingInstances",
"autoscaling:DescribeTags",
"autoscaling:SetDesiredCapacity",
"autoscaling:TerminateInstanceInAutoScalingGroup"
]
resources = ["*"]
}
}
resource "aws_iam_role_policy" "cluster_autoscaler" {
name = "cluster-autoscaler"
role = "${aws_iam_role.cluster_autoscaler.id}"
policy = "${data.aws_iam_policy_document.autoscaler.json}"
}
In order to deploy the cluster autoscaler to our cluster, we will submit a deployment manifest using kubectl. We will use Terraform’s templating system to produce the manifest.
We create a service account that is used by the autoscaler to connect to the Kubernetes API:
cluster_autoscaler.tpl
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-addon: cluster-autoscaler.addons.k8s.io
k8s-app: cluster-autoscaler
name: cluster-autoscaler
namespace: kube-system The cluster autoscaler needs to read information about the current resource usage of the cluster, and needs to be able to evict pods from nodes that need to be removed from the cluster and terminated. Basically, cluster-autoscalerClusterRole provides the required permissions for these actions. The following is the code continuation for cluster_autoscaler.tpl:
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: cluster-autoscaler
labels:
k8s-addon: cluster-autoscaler.addons.k8s.io
k8s-app: cluster-autoscaler
rules:
- apiGroups: [""]
resources: ["events","endpoints"]
verbs: ["create", "patch"]
- apiGroups: [""]
resources: ["pods/eviction"]
verbs: ["create"]
- apiGroups: [""]
resources: ["pods/status"]
verbs: ["update"]
- apiGroups: [""]
resources: ["endpoints"]
resourceNames: ["cluster-autoscaler"]
verbs: ["get","update"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["watch","list","get","update"]
- apiGroups: [""]
resources: ["pods","services","replicationcontrollers","persistentvolumeclaims","persistentvolumes"]
verbs: ["watch","list","get"]
- apiGroups: ["extensions"]
resources: ["replicasets","daemonsets"]
verbs: ["watch","list","get"]
- apiGroups: ["policy"]
resources: ["poddisruptionbudgets"]
verbs: ["watch","list"]
- apiGroups: ["apps"]
resources: ["statefulsets"]
verbs: ["watch","list","get"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["watch","list","get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: cluster-autoscaler
labels:
k8s-addon: cluster-autoscaler.addons.k8s.io
k8s-app: cluster-autoscaler
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-autoscaler
subjects:
- kind: ServiceAccount
name: cluster-autoscaler
namespace: kube-system Note that cluster-autoscaler stores state information in a config map, so needs permissions to be able to read and write from it. This role allows that. The following is the code continuation for cluster_autoscaler.tpl:
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
name: cluster-autoscaler
namespace: kube-system
labels:
k8s-addon: cluster-autoscaler.addons.k8s.io
k8s-app: cluster-autoscaler
rules:
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["create"]
- apiGroups: [""]
resources: ["configmaps"]
resourceNames: ["cluster-autoscaler-status"]
verbs: ["delete","get","update"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: cluster-autoscaler
namespace: kube-system
labels:
k8s-addon: cluster-autoscaler.addons.k8s.io
k8s-app: cluster-autoscaler
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: cluster-autoscaler
subjects:
- kind: ServiceAccount
name: cluster-autoscaler
namespace: kube-system Finally, let’s consider the manifest for the cluster autoscaler deployment itself. The cluster autoscaler pod contains a single container running the cluster autoscaler control loop. You will notice that we are passing some configuration to the cluster autoscaler as command-line arguments. Most importantly, the --node-group-auto-discovery flag allows the autoscaler to operate on autoscaling groups with the kubernetes.io/cluster/<cluster_name> tag. This is convenient because we don’t have to explicitly configure the autoscaler with our cluster autoscaling group.
If your Kubernetes cluster has nodes in more than one availability zone and you are running pods that rely on being scheduled to a particular zone (for example, pods that are making use of EBS volumes), it is recommended to create an autoscaling group for each availability zone that you plan to use. If you use one autoscaling group that spans several zones, then the cluster autoscaler will be unable to specify the availability zone of the instances that it launches.
Here is the code continuation for cluster_autoscaler.tpl:
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: cluster-autoscaler
namespace: kube-system
labels:
app: cluster-autoscaler
spec:
replicas: 1
selector:
matchLabels:
app: cluster-autoscaler
template:
metadata:
annotations:
iam.amazonaws.com/role: ${iam_role}
labels:
app: cluster-autoscaler
spec:
serviceAccountName: cluster-autoscaler
containers:
- image: k8s.gcr.io/cluster-autoscaler:v1.3.3
name: cluster-autoscaler
resources:
limits:
cpu: 100m
memory: 300Mi
requests:
cpu: 100m
memory: 300Mi
command:
- ./cluster-autoscaler
- --v=4
- --stderrthreshold=info
- --cloud-provider=aws
- --skip-nodes-with-local-storage=false
- --expander=least-waste
- --node-group-auto-discovery=asg:tag=kubernetes.io/cluster/${cluster_name}
env:
- name: AWS_REGION
value: ${aws_region}
volumeMounts:
- name: ssl-certs
mountPath: /etc/ssl/certs/ca-certificates.crt
readOnly: true
imagePullPolicy: "Always"
volumes:
- name: ssl-certs
hostPath:
path: "/etc/ssl/certs/ca-certificates.crt" Finally, we render the templated manifest by passing in the variables for the AWS region, cluster name and IAM role, and submitting the file to Kubernetes using kubectl:
Here is the code continuation for cluster_autoscaler.tpl:
data "aws_region" "current" {}
data "template_file" " cluster_autoscaler " {
template = "${file("${path.module}/cluster_autoscaler.tpl")}"
vars {
aws_region = "${data.aws_region.current.name}"
cluster_name = "${aws_eks_cluster.control_plane.name}"
iam_role = "${aws_iam_role.cluster_autoscaler.name}"
}
}
resource "null_resource" "cluster_autoscaler" {
trigers = {
manifest_sha1 = "${sha1("${data.template_file.cluster_autoscaler.rendered}")}"
}
provisioner "local-exec" {
command = "kubectl
--kubeconfig=${local_file.kubeconfig.filename} apply -f -<<EOF\n${data.template_file.cluster_autoscaler.rendered}\nEOF"
}
}
Thus, by understanding how Kubernetes assigns Quality of Service classes to your pods based on the resource requests and limits that you assign them, you can have precisely control how your pods are managed. By ensuring your critical applications, such as web servers and databases, run with the Guaranteed class, you can ensure that they will perform consistently and suffer minimal disruption when pods need to be rescheduled.
If you have enjoyed reading this post, head over to our book, Kubernetes on AWS, for tips on deploying and managing applications, keeping your cluster and applications secure, and ensuring that your whole system is reliable and resilient to failure
Low Carbon Kubernetes Scheduler: A demand side management solution that consumes electricity in low grid carbon intensity areas
A vulnerability discovered in Kubernetes kubectl cp command can allow malicious directory traversal attack on targeted system
Kubernetes 1.15 releases with extensibility around core Kubernetes APIs, cluster lifecycle stability, and more!
I remember deciding to pursue my first IT certification, the CompTIA A+. I had signed…
Key takeaways The transformer architecture has proved to be revolutionary in outperforming the classical RNN…
Once we learn how to deploy an Ubuntu server, how to manage users, and how…
Key-takeaways: Clean code isn’t just a nice thing to have or a luxury in software projects; it's a necessity. If we…
While developing a web application, or setting dynamic pages and meta tags we need to deal with…
Software architecture is one of the most discussed topics in the software industry today, and…