Menu

Virtual Geek

Tales from real IT system administrators world and non-production environment

Setup and deploy Ingress controller for Kubernetes on Bare Metal servers

After install, setup and configuration of Kubernetes Cluster in my bare metal servers, I wanted to do the POC (proof of concept) of Ingress resource for one of the production environment. 

Kubernetes Ingress is an API entity that lets entry to your Kubernetes services from outside network. It provides routing table rules to administer access to the services within a Kubernetes cluster. This normally utilizes HTTPS and HTTP protocols to enable the routing. Ingress is the ideal choice for a production environment (Ingress is also called overlay network which is virtual network that operates on top of different networks and spans across all the nodes in the cluster).

Below is the Kubernetes cluster configuration setup in my lab: 
Configure Nginx Load Balancer for the Kubernetes API Server - Part 1
Install and configure Kubernetes cluster master nodes using kubeadm - Part 2
Install and configure Kubernetes cluster worker nodes using kubeadm - Part 3

Kubernetes cluster Infrastructure ingress controller setup configuration in my home lab.jpg

Download this complete ingress install and setup script here or it is also available on github.com/janviudapi.

Before starting configuration of ingress, I will do the health checkup of my Kubernetes Cluster in the Lab by getting the list of  nodes and status with kubectl command. I also need IP addresses list of nodes. All looks good.

In this POC I have a website domain name web.example.com hosted on the K8s deployment pods, which I will try to access it this from outside network. As I don't have domain name registered in domain registrar, I will simulate it using DNS record entries in my local /etc/hosts files. Check curl request (text based and API browser) for domain name but connection is refused and not working.

Kubernetes kubectl get nodes -o wide vim etc hosts web.example.com curl wget kubenretes master networking control-plane kube-proxy ipv4 esxi Virtual Machine vm internal-ip ingress controller nginx.jpg

root@k8smaster01:~#
root@k8smaster01:~# kubectl get node -o wide
NAME          STATUS   ROLES           AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
k8smaster01   Ready    control-plane   14d   v1.25.2   192.168.34.61           Ubuntu 20.04.5 LTS   5.15.0-52-generic   containerd://1.6.8
k8smaster02   Ready    control-plane   14d   v1.25.2   192.168.34.62           Ubuntu 20.04.5 LTS   5.15.0-50-generic   containerd://1.6.8
k8smaster03   Ready    control-plane   14d   v1.25.2   192.168.34.63           Ubuntu 20.04.5 LTS   5.15.0-50-generic   containerd://1.6.8
k8sworker01   Ready                    13d   v1.25.2   192.168.34.66           Ubuntu 20.04.5 LTS   5.15.0-50-generic   containerd://1.6.8
k8sworker02   Ready                    13d   v1.25.2   192.168.34.67           Ubuntu 20.04.5 LTS   5.15.0-50-generic   containerd://1.6.8
k8sworker03   Ready                    13d   v1.25.2   192.168.34.68           Ubuntu 20.04.5 LTS   5.15.0-50-generic   containerd://1.6.8
root@k8smaster01:~#
root@k8smaster01:~# vim /etc/hosts
root@k8smaster01:~#
root@k8smaster01:~# cat /etc/hosts
127.0.0.1       localhost
127.0.1.1       k8smaster01.vcloud-lab.com k8smaster01

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

192.168.34.61   k8smaster01.vcloud-lab.com k8smaster01

192.168.34.66   web.example.com
192.168.34.67   web.example.com
192.168.34.68   web.example.com
root@k8smaster01:~#
root@k8smaster01:~# curl web.example.com
curl: (7) Failed to connect to web.example.com port 80: Connection refused
root@k8smaster01:~#

Verify the component status of Kubernetes Cluster control-plane, No error here. Git clone the Kubernetes ingress project from github/nginxinc.

kubernetes cluster kubectl get cs git clone github nginxinc kubernetes-ingress controller kube-proxy networking clone controller-manager scheduler etcd k8s.jpg

root@k8smaster01:~#
root@k8smaster01:~# kubectl get cs -o wide
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE                         ERROR
controller-manager   Healthy   ok
scheduler            Healthy   ok
etcd-0               Healthy   {"health":"true","reason":""}
root@k8smaster01:~#
root@k8smaster01:~# git clone https://github.com/nginxinc/kubernetes-ingress.git
Cloning into 'kubernetes-ingress'...
remote: Enumerating objects: 45922, done.
remote: Counting objects: 100% (59/59), done.
remote: Compressing objects: 100% (46/46), done.
remote: Total 45922 (delta 26), reused 32 (delta 13), pack-reused 45863
Receiving objects: 100% (45922/45922), 60.82 MiB | 16.68 MiB/s, done.
Resolving deltas: 100% (27340/27340), done.
root@k8smaster01:~#

Reference to official document to deploy Nginx Ingress controller:- https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-manifests/

Once folder is downloaded from github, Change the directory kubernetes-ingress/deployments and deploy/create below resources from K8S yaml manifests as shown below in the screenshot. 

common/ns-and-sa.yaml: Creates two resources, namespace and service account for the Kubernetes cluster ingress controller
rbac/rbac.yaml: In the Kubernetes cluster create rbac with cluster role resource and cluster role binding resource for service account.
common/default-server-secret.yaml: This is a part of creating common resources, it creates secret resource with a TLS certificate and key for the default server in NGINX.
common/nginx-config.yaml: This creates a configmap resource for customizing NGINX configuration.
common/ingress-class.yaml: It creates an IngressClass resource. If you would wish to configure the Ingress Controller as the default one, uncomment the annotation ingressclass.kubernetes.io/is-default-class. With this annotation set to true all the new Ingresses without an ingressClassName field specified will be assigned this IngressClass.

By default, it is needed to apply custom resource definitions (CRDS) for VirtualServer, VirtualServerRoute, TransportServer and Policy. If you don't, the Ingress Controller pods will not become Ready
common/crds/k8s.nginx.org_virtualservers.yaml
common/crds/k8s.nginx.org_virtualserverroutes.yaml
common/crds/k8s.nginx.org_transportservers.yaml
common/crds/k8s.nginx.org_policies.yaml


common/crds/k8s.nginx.org_globalconfigurations.yaml: To utilize the TCP and UDP load balancing aspects of the Ingress Controller, create the this object.
daemon-set/nginx-ingress.yaml: This will deploy and run ingress controller in kubernetes cluster. I am creating it as daemon-set so every worker node in the cluster will have one ingress controller pod deployed. My lab size is small, but in bigger environment use the deployment/nginx-ingress.yaml, It will only deploy one pod, you can adjust the replicas number in the yaml file by editing it.

kubernetes cluster kubernetes-ingress kubctl apply nginx-ingress namespace serviceaccount clusterrolebinding rbac k8s.io configmap ingressclass ingress controller customresourcedefinition daemonset service.jpg

root@k8smaster01:~#
root@k8smaster01:~# cd kubernetes-ingress/deployments
root@k8smaster01:~/kubernetes-ingress/deployments#
root@k8smaster01:~/kubernetes-ingress/deployments# kubectl apply -f common/ns-and-sa.yaml
namespace/nginx-ingress created
serviceaccount/nginx-ingress created
root@k8smaster01:~/kubernetes-ingress/deployments#
root@k8smaster01:~/kubernetes-ingress/deployments# kubectl apply -f rbac/rbac.yaml
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress created
root@k8smaster01:~/kubernetes-ingress/deployments#
root@k8smaster01:~/kubernetes-ingress/deployments# kubectl apply -f common/default-server-secret.yaml
secret/default-server-secret created
root@k8smaster01:~/kubernetes-ingress/deployments#
root@k8smaster01:~/kubernetes-ingress/deployments# kubectl apply -f common/nginx-config.yaml
configmap/nginx-config created
root@k8smaster01:~/kubernetes-ingress/deployments#
root@k8smaster01:~/kubernetes-ingress/deployments# #edit file to uncomment annotations
root@k8smaster01:~/kubernetes-ingress/deployments# kubectl apply -f common/ingress-class.yaml
ingressclass.networking.k8s.io/nginx unchanged
root@k8smaster01:~/kubernetes-ingress/deployments#
root@k8smaster01:~/kubernetes-ingress/deployments# kubectl apply -f common/crds/k8s.nginx.org_virtualservers.yaml
customresourcedefinition.apiextensions.k8s.io/virtualservers.k8s.nginx.org created
root@k8smaster01:~/kubernetes-ingress/deployments# kubectl apply -f common/crds/k8s.nginx.org_virtualserverroutes.yaml
customresourcedefinition.apiextensions.k8s.io/virtualserverroutes.k8s.nginx.org created
root@k8smaster01:~/kubernetes-ingress/deployments# kubectl apply -f common/crds/k8s.nginx.org_transportservers.yaml
customresourcedefinition.apiextensions.k8s.io/transportservers.k8s.nginx.org created
root@k8smaster01:~/kubernetes-ingress/deployments# kubectl apply -f common/crds/k8s.nginx.org_policies.yaml
customresourcedefinition.apiextensions.k8s.io/policies.k8s.nginx.org created
root@k8smaster01:~/kubernetes-ingress/deployments#
root@k8smaster01:~/kubernetes-ingress/deployments# kubectl apply -f common/crds/k8s.nginx.org_globalconfigurations.yaml
customresourcedefinition.apiextensions.k8s.io/globalconfigurations.k8s.nginx.org created
root@k8smaster01:~/kubernetes-ingress/deployments#
root@k8smaster01:~/kubernetes-ingress/deployments# #kubectl apply -f deployment/nginx-ingress.yaml
root@k8smaster01:~/kubernetes-ingress/deployments#
root@k8smaster01:~/kubernetes-ingress/deployments# kubectl apply -f daemon-set/nginx-ingress.yaml
daemonset.apps/nginx-ingress created
root@k8smaster01:~/kubernetes-ingress/deployments#
root@k8smaster01:~/kubernetes-ingress/deployments# kubectl create -f service/nodeport.yaml
service/nginx-ingress created

After deploying various yamls configuration in K8S cluster, verify the status of Ingress controller pods are running and there are no errors. If all looks good curl to the worker node IPs, now I am getting some response 404 Not found instead of connection refused.

kubernetes cluster kubectl pod nginx-ingress kube-public curl nodeport daemon 404 not found nginx deployment controller haproxy traffic master nodes worker nodes control plane networking dns.jpg

root@k8smaster01:~/kubernetes-ingress/deployments#
root@k8smaster01:~/kubernetes-ingress/deployments# kubectl get ns
NAME               STATUS   AGE
calico-apiserver   Active   14d
calico-system      Active   14d
default            Active   15d
kube-node-lease    Active   15d
kube-public        Active   15d
kube-system        Active   15d
nginx-ingress      Active   51m
tigera-operator    Active   14d
root@k8smaster01:~/kubernetes-ingress/deployments#
root@k8smaster01:~/kubernetes-ingress/deployments# kubectl get all --namespace=nginx-ingress
NAME                      READY   STATUS    RESTARTS   AGE
pod/nginx-ingress-cdxh2   1/1     Running   0          4m31s
pod/nginx-ingress-j2t6d   1/1     Running   0          4m31s
pod/nginx-ingress-v7jbb   1/1     Running   0          4m31s

NAME                    TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
service/nginx-ingress   NodePort   10.111.41.11   <none>        80:31102/TCP,443:30748/TCP   4m28s

NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/nginx-ingress   3         3         3       3            3           <none>          4m31s
root@k8smaster01:~/kubernetes-ingress/deployments#
root@k8smaster01:~/kubernetes-ingress/deployments# curl 192.168.34.66
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.23.1</center>
</body>
</html>
root@k8smaster01:~/kubernetes-ingress/deployments#
root@k8smaster01:~/kubernetes-ingress/deployments# curl 192.168.34.67
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.23.1</center>
</body>
</html>
root@k8smaster01:~/kubernetes-ingress/deployments#

Verify web.example.com dns name using curl, I am getting 404 Not Found response.

Kubernetes controller cluster esxi tkg ingress controller deployment setup configuration esxi 404 not found curl web.example.com haproxy reverse proxy.jpg

root@k8smaster01:~/kubernetes-ingress/deployments# curl web.example.com
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.23.1</center>
</body>
</html>
root@k8smaster01:~/kubernetes-ingress/deployments#

Ingress controller is setup well till this point, I will now deploy my applications in the cluster and test ingress now. Create a new namespace and deployment with 3 replicas with NGINX image under it.  Under the same namespace, create new ClusterIP service by exposing the deployment. Use port and target ports parameter, use selector label of deployment (start with app=deployment_name). In the last verify the status of all the resources deployed in the project namespace.

kubernetes cluster docker kubectl create namespace ns deployment create image port replicas ingress controller setup config expose service selector tags label svc external-IP clusterip loadbalancer nodeport external.jpg

root@k8smaster01:~/kubernetes-ingress/deployments#
root@k8smaster01:~/kubernetes-ingress/deployments# kubectl create namespace project
namespace/project created
root@k8smaster01:~/kubernetes-ingress/deployments#
root@k8smaster01:~/kubernetes-ingress/deployments# kubectl get ns project
NAME      STATUS   AGE
project   Active   15s
root@k8smaster01:~/kubernetes-ingress/deployments#
root@k8smaster01:~/kubernetes-ingress/deployments# kubectl create deployment nginx-deploy --image=nginx --port=80 --replicas=3 -n project
deployment.apps/nginx-deploy created
root@k8smaster01:~/kubernetes-ingress/deployments#
root@k8smaster01:~/kubernetes-ingress/deployments# kubectl get all -n project
NAME                               READY   STATUS    RESTARTS   AGE
pod/nginx-deploy-968984648-df4vp   1/1     Running   0          10s
pod/nginx-deploy-968984648-fsrxm   1/1     Running   0          10s
pod/nginx-deploy-968984648-lmdhm   1/1     Running   0          10s

NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-deploy   3/3     3            3           10s

NAME                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-deploy-968984648   3         3         3       10s
root@k8smaster01:~/kubernetes-ingress/deployments#
root@k8smaster01:~/kubernetes-ingress/deployments# kubectl expose deployment nginx-deploy --name=nginx-svc --port=80 --target-port=80 --selector=app=nginx-deploy -n project
service/nginx-svc exposed
root@k8smaster01:~/kubernetes-ingress/deployments#
root@k8smaster01:~/kubernetes-ingress/deployments# kubectl get all -n project
NAME                               READY   STATUS    RESTARTS   AGE
pod/nginx-deploy-968984648-df4vp   1/1     Running   0          10s
pod/nginx-deploy-968984648-fsrxm   1/1     Running   0          10s
pod/nginx-deploy-968984648-lmdhm   1/1     Running   0          10s

NAME                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/nginx-svc   ClusterIP   10.101.250.212           80/TCP    12s

NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-deploy   3/3     3            3           2m1s

NAME                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-deploy-968984648   3         3         3       2m1s
root@k8smaster01:~/kubernetes-ingress/deployments#
root@k8smaster01:~/kubernetes-ingress/deployments# cd ~/

Go to home directory. Create a new Ingress resource in the yaml file with the below content. I have provided an annotations to instruct this ingress resource to use IngressClass named nginx (This is NGINX ingress controller name). Mention the DNS name details in yaml. Apply the configuration. Describe the resource and verify details.

vim cat ingress resource yaml yml networking.k8s.io apiversion kind metadata sepc rule host pathtype prefix backend service name controller kubernetes cluster svc networking.jpg

root@k8smaster01:~/kubernetes-ingress/deployments# cd ~/
root@k8smaster01:~#
root@k8smaster01:~# vim ingress-resource.yaml
root@k8smaster01:~#
root@k8smaster01:~# cat ingress-resource.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx
  namespace: project
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: web.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-svc
            port:
              number: 80
root@k8smaster01:~#
root@k8smaster01:~# kubectl apply -f ingress-resource.yaml
ingress.networking.k8s.io/nginx created
root@k8smaster01:~#
root@k8smaster01:~# kubectl describe ingress nginx -n project
Name:             nginx
Labels:           <none>
Namespace:        project
Address:
Ingress Class:    <none>
Default backend:  <default>
Rules:
  Host             Path  Backends
  ----             ----  --------
  web.example.com
                   /   nginx-svc:80 (10.244.145.22:80,10.244.218.29:80,10.244.53.158:80)
Annotations:       kubernetes.io/ingress.class: nginx
Events:
  Type    Reason          Age   From                      Message
  ----    ------          ----  ----                      -------
  Normal  AddedOrUpdated  19s   nginx-ingress-controller  Configuration for project/nginx was added or updated
  Normal  AddedOrUpdated  19s   nginx-ingress-controller  Configuration for project/nginx was added or updated
  Normal  AddedOrUpdated  19s   nginx-ingress-controller  Configuration for project/nginx was added or updated
root@k8smaster01:~#

Everything is setup correctly now. Curl web.example.com website, I am getting successful result welcome to nginx!. Ingress is working fine. Test the worker nodes IP in curl, It is still showing 404 Not Found.

Microsoft Kubernetes cluster curl web.example.com welcome to nginx 404 not found networking configuration ingress controller cluster pod service class setup nginx ingress ubuntu container docker.jpg

root@k8smaster01:~#
root@k8smaster01:~# curl web.example.com
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
root@k8smaster01:~#
root@k8smaster01:~# curl 192.168.34.66
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.23.1</center>
</body>
</html>
root@k8smaster01:~# curl 192.168.34.67
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.23.1</center>
</body>
</html>
root@k8smaster01:~#

This is a trick, as I have setup the Ingress Controller in the bare metal kubernetes setup with NodePort service. I can access it using the node ip and service nodeport port number mentioning Host as DNS.

VMware kubernetes get service cluster-ip internal-ip external-ip nodeport loadbalancer ip clusterip ports curl host kubernetes cluster welcome to nginx documentation ingress controller.jpg

root@k8smaster01:~# kubectl get services -n nginx-ingress
NAME            TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
nginx-ingress   NodePort   10.111.104.325   <none>        80:31313/TCP,443:30203/TCP   5h52m
root@k8smaster01:~#
root@k8smaster01:~# curl http://192.168.34.66:31313 -H 'Host:web.example.com'
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
root@k8smaster01:~#

This is a last part Incase if you want to delete the ingress controller setup use below commands.

kubectl delete namespace nginx-ingress
kubectl delete clusterrole nginx-ingress
kubectl delete clusterrolebinding nginx-ingress
cd kubernetes-ingress/deployments
kubectl delete -f common/crds/
cd ~/
rm -rf ~/kubernetes-ingress

Useful Articles
Kubernetes kubeadm join could not find a jws signature in the cluster-info ConfigMap for token ID
Kubernetes kubeadm join couldn't validate the identity of the API server connection refused
How to install kubernetes master control-plane on ubuntu Part 1
How to install kubernetes worker node on ubuntu Part 2
ansible create an array with set_fact
Ansible get information from esxi advanced settings nested dictionary with unique keynames
Install Ansible AWX Tower on Ubuntu Linux
Ansible AWX installation error Cannot have both the docker-py and docker python modules
Ansible AWX installation error docker-compose run --rm --service-ports task awx-manage migrate --no-input

Go Back

Comment

Blog Search

Page Views

8559542

Follow me on Blogarama