VMware Tanzu Kubernetes Grid on AWS using the CLI, Part II.

Lubomir Tobek

7 min read

·

Jan 6, 2024

--

Listen

Share

In this section, I continue to install the necessary components for deploying TKG on AWS using commands.

Log in to the instance via SSH, download the necessary tools, and install them. I’ll test-run them.

Install Tanzu CLI, kubectl:

#Download Tanzu CLI bundle and kubectl
wget https://download3.vmware.com/software/TKG-161/tanzu-cli-bundle-linux-amd64.tar.gz

wget https://download3.vmware.com/software/TKG-161/kubectl-linux-v1.23.10+vmware.1.gz

#Unpack the bundle
tar -xvf tanzu-cli-bundle-linux-amd64.tar.gz
gzip -d kubectl-linux-v1.23.10+vmware.1.gz

#Make the file executable
chmod +x tanzu-cli-bundle-linux-amd64.tar.gz.1
chmod +x kubectl-linux-v1.23.10+vmware.1

#Rename the bundle
mv tanzu-cli-bundle-linux-amd64.tar.gz.1 tanzu
mv kubectl-linux-v1.23.10+vmware.1 kubectl

#Install kubectl/tanzu
~/cli/core/v0.25.4$ sudo install tanzu-core-linux_amd64 /usr/local/bin/tanzu
sudo install kubectl /usr/local/bin/kubectl

#Verify the version
kubectl version
Client Version: version.Info{Major:"1", Minor:"23"

tanzu version
version: v0.25.4
buildDate: 2022-12-22

tanzu plugin sync

or it can help:

Install jq, AWS CLI:

#Install jq
sudo apt-get install jq -y
sudo apt-get update -y

#Download AWS CLI
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"

#Download unzip/Upack AWS CLI archive
sudo apt install unzip
unzip awscliv2.zip

#Install AWS CLI
sudo ./aws/install
You can now run: /usr/local/bin/aws --version

#Check AWS CLI version
aws --version
aws-cli/2.15.1 Python/3.11.6 Linux/6.2.0-1014-aws exe/x86_64.ubuntu.22 prompt/off

or it can help:

Install clusterctl:

#Download for AMD64:
curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.6.0/clusterctl-linux-amd64 -o clusterctl

#Make it executable
chmod +x clusterctl

#Install clusterctl
sudo install -o root -g root -m 0755 clusterctl /usr/local/bin/clusterctl

#Test to ensure the version you installed is up-to-date
clusterctl version
GitVersion:"v1.6.0"

or it can help:

Install clusterawsadm:

#Download the latest release; on Linux, type
curl -L https://github.com/kubernetes-sigs/cluster-api-provider-aws/releases/download/v2.3.0/clusterawsadm-linux-amd64 -o clusterawsadm

#Make it executable
chmod +x clusterawsadm

#Move the binary to a directory present in your PATH
sudo mv clusterawsadm /usr/local/bin/clusterawsadm

#Check version to confirm installation
clusterawsadm version
GitVersion:"v2.3.0"

or it can help:

Once I have everything I need installed, I will not forget to check the services on the jumbox machine, such as docker and so on.

I will define the values:

export AWS_ACCESS_KEY_ID=AK*********************JB
export AWS_SECRET_ACCESS_KEY=JS***************************************B/
export AWS_REGION=us-east-1

I will start with the creation of cluster management.

tanzu management-cluster permissions aws set

I click through the AWS console to see how the Stack is created. After I’m done, I’ll check to ensure everything is fine.

Press enter or click to view image in full size

Press enter or click to view image in full size

I will prepare a yaml configuration file to create the definition and values to create the cluster management.

sudo nano mgmt-cl-config.yaml

I will use the following in the config:

#! ------------------------------------------------------------------
#! Basic cluster creation configuration
#! ------------------------------------------------------------------
CLUSTER_NAME: mgmt-cl
CLUSTER_PLAN: dev
INFRASTRUCTURE_PROVIDER: aws
ENABLE_CEIP_PARTICIPATION: true
# TMC_REGISTRATION_URL:
ENABLE_AUDIT_LOGGING: true
CLUSTER_CIDR: 100.96.0.0/11
SERVICE_CIDR: 100.64.0.0/13
#! ------------------------------------------------------------------
#! Image repository configuration
#! ------------------------------------------------------------------
# TKG_CUSTOM_IMAGE_REPOSITORY: ""
# TKG_CUSTOM_IMAGE_REPOSITORY_CA_CERTIFICATE: ""
#! ------------------------------------------------------------------
#! Proxy configuration
#! ------------------------------------------------------------------
# TKG_HTTP_PROXY: ""
# TKG_HTTPS_PROXY: ""
# TKG_NO_PROXY: ""
#! ------------------------------------------------------------------
#! Node configuration
#! ------------------------------------------------------------------
CONTROL_PLANE_MACHINE_TYPE: "t3.large"
NODE_MACHINE_TYPE: "m5.large"
# SIZE:
# CONTROLPLANE_SIZE:
# WORKER_SIZE:
# CONTROL_PLANE_MACHINE_TYPE: t3.large
# NODE_MACHINE_TYPE: m5.large
# OS_NAME: ""
# OS_VERSION: ""
# OS_ARCH: ""
#! ------------------------------------------------------------------
#! AWS configuration
#! ------------------------------------------------------------------
AWS_REGION: us-east-1
AWS_NODE_AZ: "us-east-1a"
AWS_ACCESS_KEY_ID: AK********************JB
AWS_SECRET_ACCESS_KEY: JS*****************************************B/
AWS_SSH_KEY_NAME: tkg-key-pair
BASTION_HOST_ENABLED: false
# AWS_NODE_AZ_1: ""
# AWS_NODE_AZ_2: ""
# AWS_VPC_ID: vpc-*************
# AWS_PRIVATE_SUBNET_ID: ""
# AWS_PUBLIC_SUBNET_ID: ""
# AWS_PUBLIC_SUBNET_ID_1: ""
# AWS_PRIVATE_SUBNET_ID_1: ""
# AWS_PUBLIC_SUBNET_ID_2: ""
# AWS_PRIVATE_SUBNET_ID_2: ""
# AWS_VPC_CIDR: 10.0.0.0/16
# AWS_PRIVATE_NODE_CIDR: 10.0.0.0/24
# AWS_PUBLIC_NODE_CIDR: 10.0.1.0/24
# AWS_PRIVATE_NODE_CIDR_1: 10.0.2.0/24
# AWS_PUBLIC_NODE_CIDR_1: 10.0.3.0/24
# AWS_PRIVATE_NODE_CIDR_2: 10.0.4.0/24
# AWS_PUBLIC_NODE_CIDR_2: 10.0.5.0/24
#! ------------------------------------------------------------------
#! Machine Health Check configuration
#! ------------------------------------------------------------------
ENABLE_MHC: true
MHC_UNKNOWN_STATUS_TIMEOUT: 5m
MHC_FALSE_STATUS_TIMEOUT: 12m
#! ------------------------------------------------------------------
#! Identity management configuration
#! ------------------------------------------------------------------
#IDENTITY_MANAGEMENT_TYPE: "oidc"
#! Settings for OIDC
# CERT_DURATION: 2160h
# CERT_RENEW_BEFORE: 360h
# OIDC_IDENTITY_PROVIDER_CLIENT_ID:
# OIDC_IDENTITY_PROVIDER_CLIENT_SECRET:
# OIDC_IDENTITY_PROVIDER_GROUPS_CLAIM: groups
# OIDC_IDENTITY_PROVIDER_ISSUER_URL:
# OIDC_IDENTITY_PROVIDER_SCOPES: email
# OIDC_IDENTITY_PROVIDER_USERNAME_CLAIM: email
#! The following two variables are used to configure Pinniped JWTAuth
# SUPERVISOR_ISSUER_URL:
# SUPERVISOR_ISSUER_CA_BUNDLE_DATA:
#! Settings for LDAP
# LDAP_BIND_DN:
# LDAP_BIND_PASSWORD:
# LDAP_HOST:
# LDAP_USER_SEARCH_BASE_DN:
# LDAP_USER_SEARCH_FILTER:
# LDAP_USER_SEARCH_USERNAME: userPrincipalName
# LDAP_USER_SEARCH_ID_ATTRIBUTE: DN
# LDAP_USER_SEARCH_EMAIL_ATTRIBUTE: DN
# LDAP_USER_SEARCH_NAME_ATTRIBUTE:
# LDAP_GROUP_SEARCH_BASE_DN:
# LDAP_GROUP_SEARCH_FILTER:
# LDAP_GROUP_SEARCH_USER_ATTRIBUTE: DN
# LDAP_GROUP_SEARCH_GROUP_ATTRIBUTE:
# LDAP_GROUP_SEARCH_NAME_ATTRIBUTE: cn
# LDAP_ROOT_CA_DATA_B64:
#! ------------------------------------------------------------------
#! Antrea CNI configuration
#! ------------------------------------------------------------------
# ANTREA_NO_SNAT: false
# ANTREA_TRAFFIC_ENCAP_MODE: "encap"
# ANTREA_PROXY: false
# ANTREA_POLICY: true
# ANTREA_TRACEFLOW: false

I will start the deployment for the management cluster.

tanzu management-cluster create --file mgmt-cl-config.yaml

Management cluster config file has been generated and stored at: ‘/root/.config/tanzu/tkg/clusterconfigs/mgmt-cl.yaml’.

This is what the deployment process looked like:

Validating the pre-requisites...
Identity Provider not configured. Some authentication features won't work.
Using default value for CONTROL_PLANE_MACHINE_COUNT = 1. Reason: CONTROL_PLANE_MACHINE_COUNT variable is not set
Using default value for WORKER_MACHINE_COUNT = 1. Reason: WORKER_MACHINE_COUNT variable is not set
NODE_MACHINE_TYPE_1 not set, using the default NODE_MACHINE_TYPE instead
NODE_MACHINE_TYPE_2 not set, using the default NODE_MACHINE_TYPE instead

Setting up management cluster...
Validating configuration...
Using infrastructure provider aws:v1.2.0
Generating cluster configuration...
Setting up bootstrapper...
Bootstrapper created. Kubeconfig: /root/.kube-tkg/tmp/config_DyOYMKLo
Installing providers on bootstrapper...
Fetching providers
Installing cert-manager Version="v1.7.2"
Waiting for cert-manager to be available...
Installing Provider="cluster-api" Version="v1.1.5" TargetNamespace="capi-system"
Installing Provider="bootstrap-kubeadm" Version="v1.1.5" TargetNamespace="capi-kubeadm-bootstrap-system"
Installing Provider="control-plane-kubeadm" Version="v1.1.5" TargetNamespace="capi-kubeadm-control-plane-system"
Installing Provider="infrastructure-aws" Version="v1.2.0" TargetNamespace="capa-system"
Using default value for CONTROL_PLANE_MACHINE_COUNT = 1. Reason: CONTROL_PLANE_MACHINE_COUNT variable is not set
Management cluster config file has been generated and stored at: '/root/.config/tanzu/tkg/clusterconfigs/mgmt-cl.yaml'
Start creating management cluster...
[cluster control plane is still being initialized: WaitingForControlPlane, cluster infrastructure is still being provisioned: WaitingForControlPlane]
cluster control plane is still being initialized: WaitingForControlPlane
cluster control plane is still being initialized: WaitingForKubeadmInit
cluster control plane is still being initialized: InstanceProvisionStarted @ Machine/mgmt-cl-control-plane-4pxmm
cluster control plane is still being initialized: WaitingForKubeadmInit
Saving management cluster kubeconfig into /root/.kube/config
Installing providers on management cluster...
Fetching providers
Installing cert-manager Version="v1.7.2"
Waiting for cert-manager to be available...
Installing Provider="cluster-api" Version="v1.1.5" TargetNamespace="capi-system"
Installing Provider="bootstrap-kubeadm" Version="v1.1.5" TargetNamespace="capi-kubeadm-bootstrap-system"
I1220 14:12:28.123069    9372 request.go:665] Waited for 1.029656558s due to client-side throttling, not priority and fairness, request: GET:https://mgmt-cl-apiserver-1681321353.us-east-1.elb.amazonaws.com:6443/apis/clusterinformation.antrea.tanzu.vmware.com/v1beta1?timeout=30s
Installing Provider="control-plane-kubeadm" Version="v1.1.5" TargetNamespace="capi-kubeadm-control-plane-system"
I1220 14:12:38.133547    9372 request.go:665] Waited for 1.393804086s due to client-side throttling, not priority and fairness, request: GET:https://mgmt-cl-apiserver-1681321353.us-east-1.elb.amazonaws.com:6443/apis/acme.cert-manager.io/v1?timeout=30s
Installing Provider="infrastructure-aws" Version="v1.2.0" TargetNamespace="capa-system"
Waiting for the management cluster to get ready for move...
Waiting for addons installation...
Moving all Cluster API objects from bootstrap cluster to management cluster...
Performing move...
Discovering Cluster API objects
Moving Cluster API objects Clusters=1
Moving Cluster API objects ClusterClasses=0
Creating objects in the target cluster
Deleting objects from the source cluster
Waiting for additional components to be up and running...
Waiting for packages to be up and running...
You can now access the management cluster mgmt-cl by running 'kubectl config use-context mgmt-cl-admin@mgmt-cl'

Management cluster created!

You can now create your first workload cluster by running the following:

  tanzu cluster create [name] -f [file]

Some addons might be getting installed! Check their status by running the following:

  kubectl get apps -A

Checking for required plugins...
Installing plugin 'cluster:v0.25.4'
Installing plugin 'feature:v0.25.4'
Installing plugin 'kubernetes-release:v0.25.4'
Successfully installed all required plugins

I will check the cluster creation.

tanzu management-cluster get

  NAME     NAMESPACE   STATUS   CONTROLPLANE  WORKERS  KUBERNETES         
  mgmt-cl  tkg-system  running  1/1           1/1      v1.23.10+vmware.1  

Details:

NAME                                                        READY  
/mgmt-cl                                                    True                    
├─ClusterInfrastructure - AWSCluster/mgmt-cl                True                     
├─ControlPlane - KubeadmControlPlane/mgmt-cl-control-plane  True                     
│ └─Machine/mgmt-cl-control-plane-4pxmm                     True                     
└─Workers
  └─MachineDeployment/mgmt-cl-md-0                          True                     
    └─Machine/mgmt-cl-md-0-6b47c495d6-svc82                 True                     

Providers:

  NAMESPACE                          NAME                   TYPE                    
  capa-system                        infrastructure-aws     Infrastructure
  capi-kubeadm-bootstrap-system      bootstrap-kubeadm      Bootstrap
  capi-kubeadm-control-plane-system  control-plane-kubeadm  ControlPlane
  capi-system                        cluster-api            CoreProvider            

Press enter or click to view image in full size

I will simply review the basics:

kubectl config get-contexts

CURRENT   NAME                    CLUSTER   AUTHINFO        
*         mgmt-cl-admin@mgmt-cl   mgmt-cl   mgmt-cl-admin

tanzu login

? Select a server mgmt-cl             ()
✔  successfully logged in to management cluster using the kubeconfig mgmt-cl
Checking for required plugins...
All required plugins are already installed and up-to-date

kubectl get nodes

NAME                         STATUS   ROLES                  AGE   
ip-10-0-0-132.ec2.internal   Ready    control-plane,master   16m   
ip-10-0-0-175.ec2.internal   Ready    <none>                 15m 

kubectl get pods --all-namespaces

Press enter or click to view image in full size

Everything seems okay; in the next part, I will look at creating a Workload cluster. I will already put this part in the section:

VMware Tanzu Kubernetes Grid on AWS using the CLI, Part III.