Setting up a Kubernetes cluster on AWS EKS with eksctl and Deploying an App
It might come as a surprise, but not too long ago, configuring a Kubernetes cluster on Amazon EKS was quite challenging since you had to create all the components manually by yourself. This changed with the introduction of eksctl
, a tool written in Go by the community and sponsored by weaveworks that acts as the official AWS CLI for EKS management. eksctl
allows for the creation and administration of clusters through YAML files, while leveraging CloudFormation stacks behind the scenes to handle the required resources. In this blog post you'll see that using eksctl
simplifies the creation and deployment of Kubernetes clusters on Amazon EKS.
Installation
Like other binaries written in Go, installation is super simple:
curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
sudo mv /tmp/eksctl /usr/local/bin
eksctl version
Here's a breakdown of what the code snippet does:
- Downloads and extracts the
eksctl binary
:- The
curl
command fetches the latesteksctl
binary from its GitHub releases page. This binary is compressed in a .tar.gz file for efficient transfer. --silent --location
: These options ensure that the download process is not only quiet (no progress output) but also follows any redirects, which is crucial for reaching the final download link.- The URL contains a dynamic part,
$(uname -s)_amd64
, which automatically adjusts the download link based on your operating system (uname -s outputs your OS name, such as Linux or Darwin) and assumes an amd64 architecture. - The downloaded file is piped (
|
) into thetar
command, which extracts the binary into the/tmp
directory. This is a temporary location, safe for staging files before moving them to a more permanent location.
- The
Moves the binary to
/usr/local/bin
:- The
sudo mv /tmp/eksctl /usr/local/bin
command moves the extractedeksctl
binary from/tmp
to/usr/local/bin
. This directory is commonly used for manually installed binaries and is typically included in the system's PATH. Placingeksctl
here allows you to run it from any terminal without specifying its full path. - Using
sudo
is necessary because/usr/local/bin
is a system directory, and modifying its contents requires administrative privileges.
- The
Verifies the installation
- Finally, running
eksctl version
checks the installed version ofeksctl
. This not only confirms thateksctl
is correctly installed and accessible but also lets you know the exact version you have. It's a good practice to verify installations to ensure everything is set up as expected.
- Finally, running
Configuring an AWS account
Do not use your root account for cluster creation. In fact, avoid using your root account for anything other than creating other accounts or managing very specific AWS resources.
If it's your personal account, create another user. Now, if it's a corporate account, I suggest enabling AWS Organizations.
Create a new user in IAM with the following policies attached to it:
- AmazonEC2FullAccess
- AWSCloudFormationFullAccess
- EksFullAccess
- IamLimitedAccess
The first two are AWS-managed policies, so you don't need to create them. The last two will have the following content:
EksAllAccess
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "eks:*",
"Resource": "*"
},
{
"Action": [
"ssm:GetParameter",
"ssm:GetParameters"
],
"Resource": [
"arn:aws:ssm:*:<account_id>:parameter/aws/*",
"arn:aws:ssm:*::parameter/aws/*"
],
"Effect": "Allow"
},
{
"Action": [
"kms:CreateGrant",
"kms:DescribeKey"
],
"Resource": "*",
"Effect": "Allow"
},
{
"Action": [
"logs:PutRetentionPolicy"
],
"Resource": "*",
"Effect": "Allow"
}
]
}
IamLimitedAccess
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"iam:CreateInstanceProfile",
"iam:DeleteInstanceProfile",
"iam:GetInstanceProfile",
"iam:RemoveRoleFromInstanceProfile",
"iam:GetRole",
"iam:CreateRole",
"iam:DeleteRole",
"iam:AttachRolePolicy",
"iam:PutRolePolicy",
"iam:ListInstanceProfiles",
"iam:AddRoleToInstanceProfile",
"iam:ListInstanceProfilesForRole",
"iam:PassRole",
"iam:DetachRolePolicy",
"iam:DeleteRolePolicy",
"iam:GetRolePolicy",
"iam:GetOpenIDConnectProvider",
"iam:CreateOpenIDConnectProvider",
"iam:DeleteOpenIDConnectProvider",
"iam:TagOpenIDConnectProvider",
"iam:ListAttachedRolePolicies",
"iam:TagRole",
"iam:GetPolicy",
"iam:CreatePolicy",
"iam:DeletePolicy",
"iam:ListPolicyVersions"
],
"Resource": [
"arn:aws:iam::<account_id>:instance-profile/eksctl-*",
"arn:aws:iam::<account_id>:role/eksctl-*",
"arn:aws:iam::<account_id>:policy/eksctl-*",
"arn:aws:iam::<account_id>:oidc-provider/*",
"arn:aws:iam::<account_id>:role/aws-service-role/eks-nodegroup.amazonaws.com/AWSServiceRoleForAmazonEKSNodegroup",
"arn:aws:iam::<account_id>:role/eksctl-managed-*"
]
},
{
"Effect": "Allow",
"Action": [
"iam:GetRole"
],
"Resource": [
"arn:aws:iam::<account_id>:role/*"
]
},
{
"Effect": "Allow",
"Action": [
"iam:CreateServiceLinkedRole"
],
"Resource": "*",
"Condition": {
"StringEquals": {
"iam:AWSServiceName": [
"eks.amazonaws.com",
"eks-nodegroup.amazonaws.com",
"eks-fargate.amazonaws.com"
]
}
}
}
]
}
Replace with your AWS account ID.
Now generate credentials for the user and keep the access key ID and secret key. eksctl
requires the AWS CLI to be installed. If you don't have it installed, run:
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
aws configure
You'll be prompted for details like the default AWS region (e.g., us-east-1), access key ID, and secret key to create the default profile. Now, if you already have the AWS CLI installed and configured, you can manually create the new profile by editing the ~/.aws/credentials
file.
Attention: when you have multiple profiles within your ~/.aws/credentials
file, specify which one you want to use by setting the value of the AWS_PROFILE
variable with the command export AWS_PROFILE=my_profile
and replacing my_profile
with the name of the AWS profile you want to use. This ensures that any subsequent AWS CLI commands will use the credentials and configuration associated with the specified profile.
Creating the cluster
By default, eksctl will create the cluster in a separate VPC using m5.large instances. This has some implications:
- By default quota, you can have a maximum of five VPCs per account;
- If you're using a study, lab, or test environment, this type of instance can be considered somewhat expensive;
- If you need the applications in the cluster to communicate with AWS resources - an instance in RDS, for example - located in another VPC, VPC Peering configuration and routes are necessary for this to happen.
Our cluster will have the following characteristics:
- It will be called demo-eks and will be located in us-east-1 (Northern Virginia);
- Four subnets will be created in the new VPC, two public and two private. By default, eksctl allocates the cluster nodes in the private subnets;
- A nodegroup called ng-01 will be created with an auto scaling group, with a minimum of one instance and a maximum of 5 of type t3.medium;
- IAM policies for external-dns, auto-scaling, cert-manager, ebs, and efs. We'll talk more about them shortly.
Save the file below as demo-eks.yaml
and then execute eksctl create cluster -f demo-eks.yaml --kubeconfig=demo-eks-kubeconfig
and let the magic happen. eksctl will generate the stack (or stacks) within CloudFormation and create all the necessary resources, which will take about fifteen to twenty minutes (yes, it's a bit slow) and voilà, you'll have a Kubernetes cluster for use in AWS EKS, with credentials for access to it in the demo-eks-kubeconfig.yaml
file.
---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: demo-eks
region: us-east-1
managedNodeGroups:
- name: ng-01
instanceType: t3.medium
desiredCapacity: 2
minSize: 1
maxSize: 5
iam:
withAddonPolicies:
autoScaler: true
externalDNS: true
certManager: true
ebs: true
efs: true
Important: If the
--kubeconfig
option is omitted, eksctl will generate the access file in~/.kube/config
, which can be a problem if you already have other clusters configured. If that's the case, you can create a directory called~/.kube/configs
to store the files and switch between them by setting theKUBECONFIG
variable pointing to the desired file path (e.g.,export KUBECONFIG=${HOME}/.kube/configs/demo-eks-kubeconfig
). This step is essential forkubectl
to be able to connect to the cluster.If you're still getting an error suggesting that
kubectl
is trying to connect to a Kubernetes cluster on your local machine (localhost, IP address 127.0.0.1) instead of connecting to your Amazon EKS cluster, run the commandexport KUBECONFIG=demo-eks-kubeconfig
.
Helm Installation
You've probably heard of Helm, but if not, don't worry. Helm is, in essence, a package manager for Kubernetes. These packages are provided in the form of charts, and with Helm, we can create manifests for our applications in a more dynamic, organized, parameterizable way, and then host them in repositories for deployment.
You can quickly install it using curl
and bash
:
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
Here's what the command does:
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
: This part of the command usescurl
, a command-line tool for transferring data using various network protocols. Here, it's used to download the installation script for Helm 3 from its official GitHub repository. The URL points to the raw version of theget-helm-3
script, which is intended to be executed directly.| bash
: This pipe (|) takes the output of thecurl
command (which is the Helm 3 installation script) and feeds it directly intobash
, the Bourne Again SHell. Running a script in this manner executes the script's contents.
Here's what happens when you run the command:
- The
get-helm-3
script is fetched from the Helm GitHub repository usingcurl
. - The script is then immediately executed in your shell (
bash
) without needing to be saved to a file on your disk. - The script performs several actions to install Helm 3:
- It checks your system to ensure it meets the prerequisites for installing Helm.
- It determines the latest version of Helm 3 if not specified.
- It downloads the correct Helm 3 binary for your operating system and architecture.
- It places the Helm binary in a location on your system where executable files are typically stored, making it accessible from anywhere in your terminal.
- It might also perform some additional setup tasks, such as setting up autocomplete features for Helm commands.
Or, if you're using a Debian-like distribution and want to use a package you can install it with the following commands:
curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null
sudo apt-get install apt-transport-https --yes
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update
sudo apt-get install helm
Add-ons Installation
Metrics Server
The Metrics Server is a necessary resource for using HPAs (Horizontal Pod Autoscalers). HPAs are extremely useful when we want to scale our applications horizontally when they need more resources.
helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/
helm upgrade --install --create-namespace -n metrics-server metrics-server metrics-server/metrics-server
Prometheus
Prometheus is a monitoring solution. If you use Lens or another dashboard for monitoring your clusters, it will provide CPU, memory, network, and disk usage graphs for the cluster, nodes, controllers, and pods. Prometheus is commonly used alongside Grafana.
helm repo add bitnami https://charts.bitnami.com/bitnami
helm install --create-namespace -n kube-prometheus prometheus bitnami/kube-prometheus
ExternalDNS
ExternalDNS synchronizes Kubernetes resources with external DNS records hosted on providers like AWS Route 53, Google Cloud DNS, AzureDNS, etc. With it, we don't need to manually create DNS entries every time we deploy a new application to our cluster. Cool, right?
helm install external-dns stable/external-dns \
--create-namespace \
--namespace external-dns \
--set provider=aws \
--set aws.zoneType=public \
--set txtOwnerId=<ZONE_ID>
--set policy=sync
--set registry=txt
--set interval=20s
--set domainFilters[0]=<DOMAIN>
Replace <ZONE_ID>
with your AWS Route 53 zone ID and <DOMAIN>
with your domain in the format domain.com
.
cert-manager
cert-manager is a CRD (Custom Resource Definition) that dynamically generates TLS/SSL certificates for our applications using Let's Encrypt (although it also supports other issuers).
helm repo add bitnami https://charts.bitnami.com/bitnami
helm install --create-namespace -n cert-manager cert-manager bitnami/cert-manager \
--create-namespace \
--namespace cert-manager \
--set installCRDs=true
With the chart installed, it's necessary to create two CRDs, each representing a different issuer from Let's Encrypt, one for production and one for staging. The difference between them is the request limit we can make. In a test environment, prefer to use the staging environment.
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: <meu_email>
privateKeySecretRef:
name: letsencrypt-prod
solvers:
- http01:
ingress:
class: nginx
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
server: https://acme-staging-v02.api.letsencrypt.org/directory
email: <meu_email>
privateKeySecretRef:
name: letsencrypt-prod
solvers:
- http01:
ingress:
class: nginx
Note: Replace
<my_email>
with your email address. If you're using an Ingress other than Nginx, you need to change the manifests above by setting the appropriate class.
Run the command kubectl apply -f cert-manager-staging.yaml -f cert-manager-prod.yaml
to create them in the cluster.
NGINX Ingress Controller
We allow access to our applications through the NGINX Ingress Controller. You can think of an Ingress as a reverse proxy.
helm upgrade --install ingress-nginx ingress-nginx \
--repo https://kubernetes.github.io/ingress-nginx \
--namespace ingress-nginx \
--create-namespace
Auto Scaling Group
One of the coolest features of cloud computing is elasticity. Imagine that throughout the year, we have a relatively constant demand, but at certain times - like Black Friday - it can increase considerably, and soon you'll need more nodes within your nodegroups to meet the needs of your pods. So that we don't have to do this manually, let's add automatic scaling functionality to our nodegroup.
helm repo add autoscaler https://kubernetes.github.io/autoscaler
helm install --create-namespace -n cluster-autoscaler autoscaler/cluster-autoscaler \
--set 'autoDiscovery.clusterName=<cluster_name>'
Note: Replace
<cluster_name>
with the name of your cluster (in this case,demo-eks
).
To test, after deploying any application, increase the number of replicas to a number not met by the resources available in your nodegroup. The cluster-autoscaler will provide new nodes up to the maximum number set in the nodegroup. When the number of pods consumes less than 50% of the resources of a node for more than 10 minutes, this node will be removed from the nodegroup. Cool, right?
Deploying an Application
With our cluster set up and the add-ons installed, it's time to launch a test application. Below are the manifests within a single file named hello-kubernetes.yaml
:
apiVersion: v1
kind: Namespace
metadata:
name: hello-kubernetes
---
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: hello-kubernetes
name: hello-kubernetes
spec:
replicas: 2
selector:
matchLabels:
app: hello-kubernetes
template:
metadata:
labels:
app: hello-kubernetes
spec:
containers:
- name: hello-kubernetes
image: paulbouwer/hello-kubernetes:1.5
ports:
- containerPort: 8080
resources:
requests:
memory: "128Mi"
cpu: "250m"
limits:
memory: "256Mi"
cpu: "500m"
---
apiVersion: v1
kind: Service
metadata:
namespace: hello-kubernetes
name: hello-kubernetes
spec:
type: ClusterIP
selector:
app: hello-kubernetes
ports:
- port: 80
targetPort: 8080
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
namespace: hello-kubernetes
name: hello-kubernetes-ingress
annotations:
nginx.ingress.kubernetes.io/ingress-class: nginx
cert-manager.io/cluster-issuer: letsencrypt-prod
external-dns.alpha.kubernetes.io/hostname: <app.DOMAIN>
spec:
tls:
- hosts:
- <app.DOMAIN>
secretName: <app.DOMAIN>
rules:
- host: <app.DOMAIN>
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: hello-kubernetes
port:
number: 80
Note: Replace with the domain you want to use for your application, something like app.juliaK8s.net.
Once you've created the manifest and saved it, deploy the YAML file using kubectl apply
:
kubectl apply -f hello-kubernetes.yaml
This command will create the resources defined in the YAML file (Namespace, Deployment, Service, Ingress) in your Kubernetes cluster.
After applying the YAML file, you can verify that the resources have been created successfully by running the command kubectl get all -n hello-kubernetes
.
If everything goes well, here's what should happen:
- Entries will be created in your DNS zone inside AWS Route 53 according to what was configured in your ingress and the settings of your ExternalDNS, and within a few minutes, you will be able to access your application by that name. You can monitor the propagation using the dig tool;
- A certificate will be automatically generated for your application. This can take some time for new applications;
- We allocate two replicas for the application, and here's a very important caveat: Kubernetes will by default perform round-robin load balancing between the pods. In most of today's web applications, we work with sessions and, as a result, there may be scenarios where a user authenticates on one pod, and the next request, is redirected to another where the session does not exist. As a consequence, strange behaviors are presented to the user, such as redirection to the login screen, "Unauthorized" messages, intermittent access, etc. To prevent this, we usually use some service like Redis, memcached (which may or may not be on AWS ElastiCache), or the sticky-sessions feature, where a user is consistently sent to the same pod.
Deleting Cluster
If you want to delete the cluster and resources you've just created, run the following command:
eksctl delete cluster --name demo-eks --region us-east-1
Conclusion
Kubernetes is an extremely extensive and complex subject with a myriad of possibilities. I presented some of them here that can be used not only in AWS EKS but also in on-premises installations (except for the cluster-autoscaler, unless you are using OpenStack in your company). There is also a series of security improvements that can be made to the cluster setup in this article. I hope you enjoyed it!
References
If you liked this article, follow me on Twitter (where I share my tech journey daily), connect with me on LinkedIn, check out my IG, and make sure to subscribe to my Youtube channel for more amazing content!!