Cross-Cloud Orchestration: Architecting a Multi-Master, Multi-Node Kubernetes Cluster Spanning AWS and Azure
Why is the Kubernetes orchestration tool needed ?
Kubernetes orchestrates and automates the deployment, scaling, and management of containerized applications. It provides benefits like efficient resource utilization, automatic scaling, and high availability, making it essential for modern cloud-native application development and deployment.
To learn about the deployment journey from bare metal to Kubernetes orchestration, check out the blog below.
Why do we need a Multi-node Kubernetes cluster?
If we run a Kubernetes cluster on a single node, it will lead to a single point of failure, risking the loss of all our applications running on that node. To address this challenge, it is recommended to transition to a multi-node Kubernetes cluster, providing high availability. Each of the multiple worker nodes will have its respective container engine.
Why do we need a Multi Master Multi Node Kubernetes Cluster in a Multi-Cloud environment?
Multi Master Multi Node Kubernetes Cluster in a Multi-Cloud environment involves deploying and managing a Kubernetes cluster with multiple master nodes and worker nodes across different cloud providers. This setup offers several benefits, addressing various requirements and challenges associated with scalability, high availability, and redundancy.
Architecture Diagram:
In a multi-master setup, there are multiple control plane nodes (master nodes) that manages the cluster. Having multiple masters improves fault tolerance and ensures high availability. If one master node fails, the others can take over, reducing the risk of a single point of failure.
A multi-node cluster includes multiple worker nodes that run application workloads in containers. Distributing nodes across different cloud providers enhances reliability, performance, and geographical redundancy.
Deploying across multiple cloud providers (multi-cloud) adds an extra layer of redundancy and mitigates risks associated with a single cloud provider’s outages. It allows organizations to take advantage of different cloud services and pricing models.
Prerequisites:
Prerequisites for setting up a Multi Master Multi Node Kubernetes Cluster in a Multi-Cloud environment include having an Azure account with the necessary permissions to create Virtual Networks, Virtual Network Gateway, Local Gateway, and Virtual Machines, along with an AWS account with the required permissions to create VPC, VPN Gateway, Customer Gateway, and EC2 instances.
This blog will cover the steps to set up a multi-master, multi-node Kubernetes cluster across multiple clouds, end to end.
The deployment and configuration of a Multi-Cloud Kubernetes Cluster, involves both AWS and Azure environments. The process includes certificate generation, container runtime and Kubernetes component installations, initialization of master nodes, and the creation of deployments and services for cross-cloud testing.
I. Setup steps for establishing private connectivity between AWS and Azure
To learn about the establishing private connectivity between AWS and Azure, check out the blog below.
II. Deploying and Configuring Multi-Cloud Kubernetes Cluster
Step 1: Instance Setup
1.1 Launch 3 EC2 instances in AWS
Launch 3 EC2 instances: one as the Kubernetes master node, one as the Kubernetes worker node, and one as HAProxy, all within the same VPC that I created while establishing private connectivity between AWS and Azure, as described in the above blog.
AWS Kubernetes Master Node Private IP: 10.10.1.255
AWS Kubernetes Worker Node Private IP: 10.10.1.210
Kubernetes Load balancer instance Private IP: 10.10.1.149
1.2 Launch a 2 Virtual Machines in Azure
Launch a Virtual Machine in the Azure for the Kubernetes Master node and Kubernetes Worker node in the same Virtual Network that I created while establishing private connectivity between AWS and Azure, as described in the above blog.
Azure Kubernetes Master Node Private IP: 172.10.1.4
Azure Kubernetes Worker Node Private IP : 172.10.1.5
1.3 Checking the connectivity between Azure VM and AWS instance
This three-step process involves setting up the necessary networking components in both Azure and AWS, including virtual networks, subnets, gateways, and VPN connections. The final step ensures the establishment of private connectivity between the two cloud environments.
I connected to the Kubernetes master node EC2 instance in AWS to verify connectivity with the Azure Kubernetes virtual machine using its private IP address.
Similarly, I connected to the Kubernetes virtual machine in Azure to confirm connectivity with the Kubernetes master node in AWS using its private IP address.
Step 2: Certificate and Key Generation
2.1 Install the cfssl package on the HAProxy instance
CFSSL provides tools for generating certificate authorities (CAs), certificates, and keys, as well as managing certificate revocation lists (CRLs) and certificate signing requests (CSRs). It’s a popular choice for managing the security aspects of Kubernetes clusters. I downloaded the package using the command line below.
$ wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
$ wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64After downloading, I changed the permissions of the cfssl command to executable, moved the command to the /usr/local/bin/ folder, and checked its version.
$ chmod +x cfssl*
$ sudo mv cfssl_linux-amd64 /usr/local/bin/cfssl
$ sudo mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
$ sudo cfssl version2.2 Generate certificates and private keys using cfssl
- Create the certificate authority configuration file
{
"signing": {
"default": {
"expiry": "8760h"
},
"profiles": {
"kubernetes": {
"usages": ["signing", "key encipherment", "server auth", "client auth"],
"expiry": "8760h"
}
}
}
}2. Create the certificate authority signing request configuration file.
{
"CN": "Kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "IE",
"L": "Cork",
"O": "Kubernetes",
"OU": "CA",
"ST": "Cork Co."
}
]
}3. Generate the certificate authority certificate and private key.
$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca4. Create the certificate signing request configuration file
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "IE",
"L": "Cork",
"O": "Kubernetes",
"OU": "Kubernetes",
"ST": "Cork Co."
}
]
}5. Generate the certificate and private key
$ cfssl gencert \
-ca=ca.pem \
-ca-key=ca-key.pem \
-config=ca-config.json \
-hostname=10.10.1.149,10.10.1.225,172.10.1.4,127.0.0.1,kubernetes.default \
-profile=kubernetes kubernetes-csr.json | \
cfssljson -bare kubernetes2.3 Copy the generated certificates and private keys to the respective master nodes.
Copying the generated certificates and private keys to the respective master nodes using the scp command
$ chmod 400 K8s.pem
$ scp -i K8s.pem ca.pem kubernetes.pem kubernetes-key.pem ubuntu@10.10.1.225:~;Switch back to the AWS and Azure master node instance to place those files in the system configuration of the etcd , these files will be referenced while initializing the master node.
$ sudo mkdir /etc/etcd/
$ sudo cp ca.pem kubernetes.pem kubernetes-key.pem /etc/etcd/Step 3: External ETCD and HAProxy Configuration
3.1 Configure external ETCD on the HAProxy instance
After generating the certificates and private keys, the next step is to set up an external ETCD for the multi-node cluster. ETCD serves as the key-value pair database in Kubernetes. While initializing the Kubernetes cluster with Kubeadm, an internal ETCD is automatically created. However, I require an external database to store data from both the AWS Kubernetes master and Azure Kubernetes master.
I downloaded the ETCD tar file using the command below. This file will help in configuring the external ETCD cluster.
$ wget https://github.com/etcd-io/etcd/releases/download/v3.5.11/etcd-v3.5.11-linux-amd64.tar.gzAfter downloading the etcd package, we need to extract the files from it.
$ tar xvzf etcd-*Moving the etcd executable file to one of the executable folders in the system path, such as the /usr/local/bin folder.
$ sudo mv etcd-*/etcd* /usr/local/bin/Creating a service for the etcd executable file with the following content will enable us to start/stop the etcd service using the systemctl or service command.
[Unit]
Description=etcd
Documentation=https://github.com/coreos
[Service]
ExecStart=/usr/local/bin/etcd \
--name 10.10.1.149 \
--cert-file=/etc/etcd/kubernetes.pem \
--key-file=/etc/etcd/kubernetes-key.pem \
--peer-cert-file=/etc/etcd/kubernetes.pem \
--peer-key-file=/etc/etcd/kubernetes-key.pem \
--trusted-ca-file=/etc/etcd/ca.pem \
--peer-trusted-ca-file=/etc/etcd/ca.pem \
--peer-client-cert-auth \
--client-cert-auth \
--initial-advertise-peer-urls https://10.10.1.149:2380 \
--listen-peer-urls https://10.10.1.149:2380 \
--listen-client-urls https://10.10.1.149:2379,http://127.0.0.1:2379 \
--advertise-client-urls https://10.10.1.149:2379 \
--initial-cluster-token etcd-cluster-0 \
--initial-cluster 10.10.1.149=https://10.10.1.149:2380 \
--initial-cluster-state new \
--data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.targetReload the system services with the command below to reflect the changes.
After that, I enabled the etcd service to ensure it runs the daemon after reboot. Finally, I started the etcd service.
$ sudo systemctl daemon-reload
$ sudo systemctl enable etcd
$ sudo systemctl start etcd
$ ETCDCTL_API=3 etcdctl member list3.2 Set up HAProxy to manage traffic to the Kubernetes master nodes.
In this architecture, we have two master nodes: one in AWS and another in Azure. To distribute the traffic between the two master nodes, we need a load balancer. Therefore, the HaProxy package will help us manage the traffic between the two Kubernetes worker nodes.
To install the haproxy package I am using the below command in the Kubernetes Load balancer instance
$ sudo apt-get install haproxy -yOnce it get installed, to manage the traffic between the two master nodes we have to configure it in the configuration file of haproxy so it will redirect the traffic from haproxy load balancer instance to kuberntes master nodes.
$ cat <<EOL | sudo tee -a /etc/haproxy/haproxy.cfg
frontend kubernetes
bind 10.10.1.149:6443
option tcplog
mode tcp
default_backend kubernetes-master-nodes
backend kubernetes-master-nodes
mode tcp
balance roundrobin
option tcp-check
server k8s-master-aws-1 10.10.1.225:6443 check fall 3 rise 2
server k8s-master-azure-2 172.10.1.4:6443 check fall 3 rise 2
EOL
To reflect the configuration changes we have to restart the haproxy service.
$ sudo systemctl restart haproxyStep 4: Container Runtime Installation
Get Kalla kruparaju’s stories in your inbox
Join Medium for free to get updates from this writer.
4.1 Install CRIO on all master and worker nodes in both AWS and Azure.
To launch the pod we need container runtime for that I am installing CRIO on all master and worker nodes in both AWS and Azure. The CRIO package won’t come with the default package so for that I have to add external repository to install the CRIO package.
I am running the update command To reflect the external repository which we added before after installing the CRIO package and it’s dependency packages.
$ sudo apt-get update -y
$ sudo apt-get install cri-o cri-o-runc cri-tools -yReloading the system daemon and enabling the CRIO service permanently to get service up even after the reboot.
$ sudo systemctl daemon-reload
$ sudo systemctl enable crio --now
$ sudo systemctl start crio.serviceStep 5: Kubernetes Components Installation
An overlay network is a virtual network that is built on top of one or more existing networks (underlay networks). One popular overlay network technology used in container orchestration is the “Overlay Network” driver in Docker, and similar concepts are employed in container orchestration platforms like Kubernetes with the help of container networking plugins. These technologies use techniques like VXLAN (Virtual Extensible LAN) or other tunneling protocols to create the overlay network.
br_netfilter refers to the bridge netfilter module in the Linux kernel. It is a module that provides the necessary functionality for filtering and manipulating network packets in the context of Linux bridging. In the context of container orchestration systems like Kubernetes, `br_netfilter` is often used to enable network communication between containers running on different hosts within a cluster. It allows the application of firewall rules and network policies at the bridge level, contributing to the overall security and network management of the containerized environment.
The combination of overlay and br_netfilter command is often seen in Kubernetes setups to ensure that the necessary kernel modules are loaded for networking features, especially when using overlay networks for container communication across different hosts in a cluster.
$ cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
$ sudo modprobe overlay
$ sudo modprobe br_netfiltermodprobe is a command in Unix and Linux operating systems that is used to add or remove a loadable kernel module (LKM) from the Linux kernel.
$ cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
$ sudo sysctl --systemiptables is a user-space utility program that allows a system administrator to configure the IP packet filter rules of the Linux kernel firewall, implemented as different Netfilter modules. The filters are organized in different tables, which contain chain of rules for how to treat network traffic.
$ sudo swapoff -a
$ (crontab -l 2>/dev/null; echo "@reboot /sbin/swapoff -a") | crontab - || trueswap is a mechanism used by operating systems to handle situations where physical RAM is fully utilized. While it provides additional memory, excessive swapping can impact performance, and systems may be configured to manage swap space based on specific requirements.
On Linux systems, the swapon and swapoff commands are used to enable and disable swap devices, respectively. The /etc/fstab file or other configuration files are used to define swap partitions or files. In certain situations, administrators may choose to disable swap temporarily or permanently. This can be done using the swapoff command.
For kubeadm to work properly, The below commands (sudo swapoff -a and the related crontab entry) disable swap immediately and ensure it stays disabled after a system reboot.
5.1.2 Install kubeadm on all master and worker nodes in both AWS and Azure.
Kubernetes components packages also won’t be available by default configuration , we have to download repositories and configure it.
Once repositories get configured kubelet, kubectl and kubeadm packages will be available to install in the server. Installing the Kubernetes components
$ sudo apt-get install -y kubelet=1.29.0-1.1 kubectl=1.29.0-1.1 kubeadm=1.29.0-1.1I don’t want to upgrade version of these packages so I marked it hold.
$ sudo apt-mark hold kubelet kubeadm kubectlStep 6: Initialize Master Node
6.1 Initialize the AWS master node using kubeadm
One of the ways to initialize the Kubernetes master is using kubeadm. We installed it in the last step. However, during the initialization of the master node, it assumes that all configurations are within this system. But, since we have an external etcd cluster, Kubernetes key paths of etcd, and different server endpoints for proxying between the master nodes and the pod network, we are consolidating all configurations in one place.
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: stable
apiServerCertSANs:
- 10.10.1.149
controlPlaneEndpoint: "10.10.1.149:6443"
etcd:
external:
endpoints:
- https://10.10.1.149:2379
caFile: /etc/etcd/ca.pem
certFile: /etc/etcd/kubernetes.pem
keyFile: /etc/etcd/kubernetes-key.pem
networking:
podSubnet: 192.168.0.0/16
apiServerExtraArgs:
apiserver-count: "3"Initializing the Kubernetes master node using the configuration file.
$ sudo kubeadm init --config=config.yamlThe Kubernetes control-plane has been successfully configured.
$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/configThe Kubernetes cluster has a single configuration file containing the endpoint of the control plane and authentication settings. This file is typically located in the root Kubernetes configuration folder. To make it accessible to general users, I created a `.kube` directory in the user’s home folder and copied the configuration file into it. Then, I adjusted the file permissions accordingly to ensure that the user has the necessary access.
$ kubectl get nodes
$ kubectl get pods --all-namespacesTo validate whether the AWS master node has been initialized, I checked the node status. If the node status shows as “Ready” and all Kubernetes components’ pods are running without issues, it indicates that the initialization process was successful.
$ kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/tigera-operator.yaml
$ curl https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/custom-resources.yaml -O
$ kubectl apply -f custom-resourcesInstalling Calico in Kubernetes is crucial because it manages pod communication, ensuring it’s secure. Calico assigns unique IP addresses to pods and enforces traffic rules, making the Kubernetes cluster efficient and safe.
$ kubectl get nodes
$ kubectl get pods --all-namespacesAll Calico pods are running as expected, indicating that the Calico networking and security solution is functioning properly within the Kubernetes cluster.
In a Kubernetes cluster, you can add any number of control plane nodes by copying the certificate authorities and service account keys to each control plane node. Similarly, you can add any number of worker nodes to the cluster following the same process. This scalability allows you to expand your cluster’s capacity and redundancy according to your needs.
6.2 Copy certificate authorities and service account keys from the AWS master to Azure master node
Copying certificate authorities and service account keys from the AWS master node to the Azure master node to join it with the AWS master node.
Once the files are copied to the Azure master node, switch to the Azure master node to move them from the home folder to the Kubernetes configuration folder.
6.3 Join secondary Azure master node with the primary AWS master node
Run the kubeadm join command to join the Azure master node with the AWS master node.
$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
$ kubectl get nodesCopy the config file from the Kubernetes configuration folder to the user’s home folder to make it available to the general user.
To validate whether the Azure master node has been initialized, I checked the node status. If the node status shows as “Ready” and all Kubernetes components’ pods are running without issues, it indicates that the initialization process was successful.
6.6 Join the worker nodes with the master nodes
Run the kubeadm join command on the AWS worker node to join it with the master node.
To validate whether the AWS worker node has been initialized, I checked the node status. If the node status shows as “Ready” and all Kubernetes components’ pods are running without issues, it indicates that the initialization process was successful.
Run the kubeadm join command on the Azure worker node to join it with the master node.
To validate whether the Azure worker node has been initialized, I checked the node status. If the node status shows as “Ready” and all Kubernetes components’ pods are running without issues, it indicates that the initialization process was successful.
Step 7: Deployments and Services
7.1 Create Kubernetes deployments and services from the Azure master node to AWS worker node
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: nginx
replicas: 2
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:latest
ports:
- containerPort: 80
EOFCreated a Kubernetes deployment with the Nginx image to launch two identical pods on the Kubernetes worker nodes. These pods expose port 80.
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
name: nginx-service
spec:
selector:
app: nginx
type: NodePort
ports:
- port: 80
targetPort: 80
nodePort: 32000
EOFCreated a Kubernetes service to expose the Nginx deployment to the outside world. Two pods are launched in the AWS worker node, and the service ensures that external traffic can reach these pods.
Successfully accessed the pods using the Cluster IP.
7.2 Create Kubernetes pod from the AWS master node to Azure worker node
First, I’ll taint the Azure worker node to prevent pods from being scheduled on it. Then, I’ll launch a pod from the AWS master node, explicitly specifying that it should be deployed on the tainted Azure worker node.
Similarly, I want to launch a pod from the AWS master node, specifically on an Azure worker node. To do this, I’ll apply a taint to the Azure worker node, preventing pod launches on that node. However, since I want to launch the pod on the Azure worker node, I’ll use the Tolerations concept, which lets the scheduler schedule pods despite matching taints.
The example-pod is successfully running on the Kubernetes Azure worker node.
First, I established private connectivity between AWS and Azure clouds. Then, I launched master and worker nodes in both clouds, configuring them accordingly. Finally, I concluded the entire demo by launching pods from the Azure master node to the AWS worker node, and vice versa, and accessed the pods.
Thank you for reading the blog. To stay updated with more informative content, please follow me on Medium and LinkedIn.
