After install, setup and configuration of Kubernetes Cluster in my bare metal servers, I wanted to do the POC (proof of concept) of Ingress resource for one of the production environment.
Kubernetes Ingress is an API entity that lets entry to your Kubernetes services from outside network. It provides routing table rules to administer access to the services within a Kubernetes cluster. This normally utilizes HTTPS and HTTP protocols to enable the routing. Ingress is the ideal choice for a production environment (Ingress is also called overlay network which is virtual network that operates on top of different networks and spans across all the nodes in the cluster).
Below is the Kubernetes cluster configuration setup in my lab:
Configure Nginx Load Balancer for the Kubernetes API Server - Part 1
Install and configure Kubernetes cluster master nodes using kubeadm - Part 2
Install and configure Kubernetes cluster worker nodes using kubeadm - Part 3
Download this complete ingress install and setup script here or it is also available on github.com/janviudapi.
Before starting configuration of ingress, I will do the health checkup of my Kubernetes Cluster in the Lab by getting the list of nodes and status with kubectl command. I also need IP addresses list of nodes. All looks good.
In this POC I have a website domain name web.example.com hosted on the K8s deployment pods, which I will try to access it this from outside network. As I don't have domain name registered in domain registrar, I will simulate it using DNS record entries in my local /etc/hosts files. Check curl request (text based and API browser) for domain name but connection is refused and not working.
root@k8smaster01:~# root@k8smaster01:~# kubectl get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME k8smaster01 Ready control-plane 14d v1.25.2 192.168.34.61 Ubuntu 20.04.5 LTS 5.15.0-52-generic containerd://1.6.8 k8smaster02 Ready control-plane 14d v1.25.2 192.168.34.62 Ubuntu 20.04.5 LTS 5.15.0-50-generic containerd://1.6.8 k8smaster03 Ready control-plane 14d v1.25.2 192.168.34.63 Ubuntu 20.04.5 LTS 5.15.0-50-generic containerd://1.6.8 k8sworker01 Ready 13d v1.25.2 192.168.34.66 Ubuntu 20.04.5 LTS 5.15.0-50-generic containerd://1.6.8 k8sworker02 Ready 13d v1.25.2 192.168.34.67 Ubuntu 20.04.5 LTS 5.15.0-50-generic containerd://1.6.8 k8sworker03 Ready 13d v1.25.2 192.168.34.68 Ubuntu 20.04.5 LTS 5.15.0-50-generic containerd://1.6.8 root@k8smaster01:~# root@k8smaster01:~# vim /etc/hosts root@k8smaster01:~# root@k8smaster01:~# cat /etc/hosts 127.0.0.1 localhost 127.0.1.1 k8smaster01.vcloud-lab.com k8smaster01 # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters 192.168.34.61 k8smaster01.vcloud-lab.com k8smaster01 192.168.34.66 web.example.com 192.168.34.67 web.example.com 192.168.34.68 web.example.com root@k8smaster01:~# root@k8smaster01:~# curl web.example.com curl: (7) Failed to connect to web.example.com port 80: Connection refused root@k8smaster01:~#
Verify the component status of Kubernetes Cluster control-plane, No error here. Git clone the Kubernetes ingress project from github/nginxinc.
root@k8smaster01:~# root@k8smaster01:~# kubectl get cs -o wide Warning: v1 ComponentStatus is deprecated in v1.19+ NAME STATUS MESSAGE ERROR controller-manager Healthy ok scheduler Healthy ok etcd-0 Healthy {"health":"true","reason":""} root@k8smaster01:~# root@k8smaster01:~# git clone https://github.com/nginxinc/kubernetes-ingress.git Cloning into 'kubernetes-ingress'... remote: Enumerating objects: 45922, done. remote: Counting objects: 100% (59/59), done. remote: Compressing objects: 100% (46/46), done. remote: Total 45922 (delta 26), reused 32 (delta 13), pack-reused 45863 Receiving objects: 100% (45922/45922), 60.82 MiB | 16.68 MiB/s, done. Resolving deltas: 100% (27340/27340), done. root@k8smaster01:~#
Reference to official document to deploy Nginx Ingress controller:- https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-manifests/
Once folder is downloaded from github, Change the directory kubernetes-ingress/deployments and deploy/create below resources from K8S yaml manifests as shown below in the screenshot.
common/ns-and-sa.yaml: Creates two resources, namespace and service account for the Kubernetes cluster ingress controller
rbac/rbac.yaml: In the Kubernetes cluster create rbac with cluster role resource and cluster role binding resource for service account.
common/default-server-secret.yaml: This is a part of creating common resources, it creates secret resource with a TLS certificate and key for the default server in NGINX.
common/nginx-config.yaml: This creates a configmap resource for customizing NGINX configuration.
common/ingress-class.yaml: It creates an IngressClass resource. If you would wish to configure the Ingress Controller as the default one, uncomment the annotation ingressclass.kubernetes.io/is-default-class. With this annotation set to true all the new Ingresses without an ingressClassName field specified will be assigned this IngressClass.
By default, it is needed to apply custom resource definitions (CRDS) for VirtualServer, VirtualServerRoute, TransportServer and Policy. If you don't, the Ingress Controller pods will not become Ready.
common/crds/k8s.nginx.org_virtualservers.yaml
common/crds/k8s.nginx.org_virtualserverroutes.yaml
common/crds/k8s.nginx.org_transportservers.yaml
common/crds/k8s.nginx.org_policies.yaml
common/crds/k8s.nginx.org_globalconfigurations.yaml: To utilize the TCP and UDP load balancing aspects of the Ingress Controller, create the this object.
daemon-set/nginx-ingress.yaml: This will deploy and run ingress controller in kubernetes cluster. I am creating it as daemon-set so every worker node in the cluster will have one ingress controller pod deployed. My lab size is small, but in bigger environment use the deployment/nginx-ingress.yaml, It will only deploy one pod, you can adjust the replicas number in the yaml file by editing it.
root@k8smaster01:~# root@k8smaster01:~# cd kubernetes-ingress/deployments root@k8smaster01:~/kubernetes-ingress/deployments# root@k8smaster01:~/kubernetes-ingress/deployments# kubectl apply -f common/ns-and-sa.yaml namespace/nginx-ingress created serviceaccount/nginx-ingress created root@k8smaster01:~/kubernetes-ingress/deployments# root@k8smaster01:~/kubernetes-ingress/deployments# kubectl apply -f rbac/rbac.yaml clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress created root@k8smaster01:~/kubernetes-ingress/deployments# root@k8smaster01:~/kubernetes-ingress/deployments# kubectl apply -f common/default-server-secret.yaml secret/default-server-secret created root@k8smaster01:~/kubernetes-ingress/deployments# root@k8smaster01:~/kubernetes-ingress/deployments# kubectl apply -f common/nginx-config.yaml configmap/nginx-config created root@k8smaster01:~/kubernetes-ingress/deployments# root@k8smaster01:~/kubernetes-ingress/deployments# #edit file to uncomment annotations root@k8smaster01:~/kubernetes-ingress/deployments# kubectl apply -f common/ingress-class.yaml ingressclass.networking.k8s.io/nginx unchanged root@k8smaster01:~/kubernetes-ingress/deployments# root@k8smaster01:~/kubernetes-ingress/deployments# kubectl apply -f common/crds/k8s.nginx.org_virtualservers.yaml customresourcedefinition.apiextensions.k8s.io/virtualservers.k8s.nginx.org created root@k8smaster01:~/kubernetes-ingress/deployments# kubectl apply -f common/crds/k8s.nginx.org_virtualserverroutes.yaml customresourcedefinition.apiextensions.k8s.io/virtualserverroutes.k8s.nginx.org created root@k8smaster01:~/kubernetes-ingress/deployments# kubectl apply -f common/crds/k8s.nginx.org_transportservers.yaml customresourcedefinition.apiextensions.k8s.io/transportservers.k8s.nginx.org created root@k8smaster01:~/kubernetes-ingress/deployments# kubectl apply -f common/crds/k8s.nginx.org_policies.yaml customresourcedefinition.apiextensions.k8s.io/policies.k8s.nginx.org created root@k8smaster01:~/kubernetes-ingress/deployments# root@k8smaster01:~/kubernetes-ingress/deployments# kubectl apply -f common/crds/k8s.nginx.org_globalconfigurations.yaml customresourcedefinition.apiextensions.k8s.io/globalconfigurations.k8s.nginx.org created root@k8smaster01:~/kubernetes-ingress/deployments# root@k8smaster01:~/kubernetes-ingress/deployments# #kubectl apply -f deployment/nginx-ingress.yaml root@k8smaster01:~/kubernetes-ingress/deployments# root@k8smaster01:~/kubernetes-ingress/deployments# kubectl apply -f daemon-set/nginx-ingress.yaml daemonset.apps/nginx-ingress created root@k8smaster01:~/kubernetes-ingress/deployments# root@k8smaster01:~/kubernetes-ingress/deployments# kubectl create -f service/nodeport.yaml service/nginx-ingress created
After deploying various yamls configuration in K8S cluster, verify the status of Ingress controller pods are running and there are no errors. If all looks good curl to the worker node IPs, now I am getting some response 404 Not found instead of connection refused.
root@k8smaster01:~/kubernetes-ingress/deployments# root@k8smaster01:~/kubernetes-ingress/deployments# kubectl get ns NAME STATUS AGE calico-apiserver Active 14d calico-system Active 14d default Active 15d kube-node-lease Active 15d kube-public Active 15d kube-system Active 15d nginx-ingress Active 51m tigera-operator Active 14d root@k8smaster01:~/kubernetes-ingress/deployments# root@k8smaster01:~/kubernetes-ingress/deployments# kubectl get all --namespace=nginx-ingress NAME READY STATUS RESTARTS AGE pod/nginx-ingress-cdxh2 1/1 Running 0 4m31s pod/nginx-ingress-j2t6d 1/1 Running 0 4m31s pod/nginx-ingress-v7jbb 1/1 Running 0 4m31s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/nginx-ingress NodePort 10.111.41.11 <none> 80:31102/TCP,443:30748/TCP 4m28s NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/nginx-ingress 3 3 3 3 3 <none> 4m31s root@k8smaster01:~/kubernetes-ingress/deployments# root@k8smaster01:~/kubernetes-ingress/deployments# curl 192.168.34.66 <html> <head><title>404 Not Found</title></head> <body> <center><h1>404 Not Found</h1></center> <hr><center>nginx/1.23.1</center> </body> </html> root@k8smaster01:~/kubernetes-ingress/deployments# root@k8smaster01:~/kubernetes-ingress/deployments# curl 192.168.34.67 <html> <head><title>404 Not Found</title></head> <body> <center><h1>404 Not Found</h1></center> <hr><center>nginx/1.23.1</center> </body> </html> root@k8smaster01:~/kubernetes-ingress/deployments#
Verify web.example.com dns name using curl, I am getting 404 Not Found response.
root@k8smaster01:~/kubernetes-ingress/deployments# curl web.example.com <html> <head><title>404 Not Found</title></head> <body> <center><h1>404 Not Found</h1></center> <hr><center>nginx/1.23.1</center> </body> </html> root@k8smaster01:~/kubernetes-ingress/deployments#
Ingress controller is setup well till this point, I will now deploy my applications in the cluster and test ingress now. Create a new namespace and deployment with 3 replicas with NGINX image under it. Under the same namespace, create new ClusterIP service by exposing the deployment. Use port and target ports parameter, use selector label of deployment (start with app=deployment_name). In the last verify the status of all the resources deployed in the project namespace.
root@k8smaster01:~/kubernetes-ingress/deployments# root@k8smaster01:~/kubernetes-ingress/deployments# kubectl create namespace project namespace/project created root@k8smaster01:~/kubernetes-ingress/deployments# root@k8smaster01:~/kubernetes-ingress/deployments# kubectl get ns project NAME STATUS AGE project Active 15s root@k8smaster01:~/kubernetes-ingress/deployments# root@k8smaster01:~/kubernetes-ingress/deployments# kubectl create deployment nginx-deploy --image=nginx --port=80 --replicas=3 -n project deployment.apps/nginx-deploy created root@k8smaster01:~/kubernetes-ingress/deployments# root@k8smaster01:~/kubernetes-ingress/deployments# kubectl get all -n project NAME READY STATUS RESTARTS AGE pod/nginx-deploy-968984648-df4vp 1/1 Running 0 10s pod/nginx-deploy-968984648-fsrxm 1/1 Running 0 10s pod/nginx-deploy-968984648-lmdhm 1/1 Running 0 10s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/nginx-deploy 3/3 3 3 10s NAME DESIRED CURRENT READY AGE replicaset.apps/nginx-deploy-968984648 3 3 3 10s root@k8smaster01:~/kubernetes-ingress/deployments# root@k8smaster01:~/kubernetes-ingress/deployments# kubectl expose deployment nginx-deploy --name=nginx-svc --port=80 --target-port=80 --selector=app=nginx-deploy -n project service/nginx-svc exposed root@k8smaster01:~/kubernetes-ingress/deployments# root@k8smaster01:~/kubernetes-ingress/deployments# kubectl get all -n project NAME READY STATUS RESTARTS AGE pod/nginx-deploy-968984648-df4vp 1/1 Running 0 10s pod/nginx-deploy-968984648-fsrxm 1/1 Running 0 10s pod/nginx-deploy-968984648-lmdhm 1/1 Running 0 10s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/nginx-svc ClusterIP 10.101.250.212 80/TCP 12s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/nginx-deploy 3/3 3 3 2m1s NAME DESIRED CURRENT READY AGE replicaset.apps/nginx-deploy-968984648 3 3 3 2m1s root@k8smaster01:~/kubernetes-ingress/deployments# root@k8smaster01:~/kubernetes-ingress/deployments# cd ~/
Go to home directory. Create a new Ingress resource in the yaml file with the below content. I have provided an annotations to instruct this ingress resource to use IngressClass named nginx (This is NGINX ingress controller name). Mention the DNS name details in yaml. Apply the configuration. Describe the resource and verify details.
root@k8smaster01:~/kubernetes-ingress/deployments# cd ~/ root@k8smaster01:~# root@k8smaster01:~# vim ingress-resource.yaml root@k8smaster01:~# root@k8smaster01:~# cat ingress-resource.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: nginx namespace: project annotations: kubernetes.io/ingress.class: "nginx" spec: rules: - host: web.example.com http: paths: - path: / pathType: Prefix backend: service: name: nginx-svc port: number: 80 root@k8smaster01:~# root@k8smaster01:~# kubectl apply -f ingress-resource.yaml ingress.networking.k8s.io/nginx created root@k8smaster01:~# root@k8smaster01:~# kubectl describe ingress nginx -n project Name: nginx Labels: <none> Namespace: project Address: Ingress Class: <none> Default backend: <default> Rules: Host Path Backends ---- ---- -------- web.example.com / nginx-svc:80 (10.244.145.22:80,10.244.218.29:80,10.244.53.158:80) Annotations: kubernetes.io/ingress.class: nginx Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal AddedOrUpdated 19s nginx-ingress-controller Configuration for project/nginx was added or updated Normal AddedOrUpdated 19s nginx-ingress-controller Configuration for project/nginx was added or updated Normal AddedOrUpdated 19s nginx-ingress-controller Configuration for project/nginx was added or updated root@k8smaster01:~#
Everything is setup correctly now. Curl web.example.com website, I am getting successful result welcome to nginx!. Ingress is working fine. Test the worker nodes IP in curl, It is still showing 404 Not Found.
root@k8smaster01:~# root@k8smaster01:~# curl web.example.com <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> <style> html { color-scheme: light dark; } body { width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; } </style> </head> <body> <h1>Welcome to nginx!</h1> <p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p> <p>For online documentation and support please refer to <a href="http://nginx.org/">nginx.org</a>.<br/> Commercial support is available at <a href="http://nginx.com/">nginx.com</a>.</p> <p><em>Thank you for using nginx.</em></p> </body> </html> root@k8smaster01:~# root@k8smaster01:~# curl 192.168.34.66 <html> <head><title>404 Not Found</title></head> <body> <center><h1>404 Not Found</h1></center> <hr><center>nginx/1.23.1</center> </body> </html> root@k8smaster01:~# curl 192.168.34.67 <html> <head><title>404 Not Found</title></head> <body> <center><h1>404 Not Found</h1></center> <hr><center>nginx/1.23.1</center> </body> </html> root@k8smaster01:~#
This is a trick, as I have setup the Ingress Controller in the bare metal kubernetes setup with NodePort service. I can access it using the node ip and service nodeport port number mentioning Host as DNS.
root@k8smaster01:~# kubectl get services -n nginx-ingress NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE nginx-ingress NodePort 10.111.104.325 <none> 80:31313/TCP,443:30203/TCP 5h52m root@k8smaster01:~# root@k8smaster01:~# curl http://192.168.34.66:31313 -H 'Host:web.example.com' <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> <style> html { color-scheme: light dark; } body { width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; } </style> </head> <body> <h1>Welcome to nginx!</h1> <p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p> <p>For online documentation and support please refer to <a href="http://nginx.org/">nginx.org</a>.<br/> Commercial support is available at <a href="http://nginx.com/">nginx.com</a>.</p> <p><em>Thank you for using nginx.</em></p> </body> </html> root@k8smaster01:~#
This is a last part Incase if you want to delete the ingress controller setup use below commands.
kubectl delete namespace nginx-ingress kubectl delete clusterrole nginx-ingress kubectl delete clusterrolebinding nginx-ingress cd kubernetes-ingress/deployments kubectl delete -f common/crds/ cd ~/ rm -rf ~/kubernetes-ingress
Useful Articles
Kubernetes kubeadm join could not find a jws signature in the cluster-info ConfigMap for token ID
Kubernetes kubeadm join couldn't validate the identity of the API server connection refused
How to install kubernetes master control-plane on ubuntu Part 1
How to install kubernetes worker node on ubuntu Part 2
ansible create an array with set_fact
Ansible get information from esxi advanced settings nested dictionary with unique keynames
Install Ansible AWX Tower on Ubuntu Linux
Ansible AWX installation error Cannot have both the docker-py and docker python modules
Ansible AWX installation error docker-compose run --rm --service-ports task awx-manage migrate --no-input
