Although I have written two articles for installation and configuration Kubernetes cluster, it was built with very basic configuration and tried to keep it simple. Here in this article I am configuring full fledged Kubernetes cluster requiring intermediate skills on linux.
Check my articles on simple & basic kubernetes cluster:
How to install kubernetes master control-plane on ubuntu Part 1
How to install kubernetes worker node on ubuntu Part 2
Below is my architectural diagram of Kubernetes cluster in the Home Lab, I have 3 master nodes as control-plane, 3 worker nodes. Before configuring k8s cluster I am installing and configuring NGINX web server, it will work as load balancer for master nodes API server and make it redundant. Incase any of two master nodes are down or absent, my lab control plane of Kubernetes cluster is still working and functioning, while I am going to do experiments.
Another use of load balancer - Worker nodes (Worker nodes components kube-proxy and kubelet) in k8s cluster will talk to master nodes control plane via NGINX load balancer. Users will also use tools such as kubectl to connect to same NGNIX LB as shown in below diagram.
Complete articles:
Configure Nginx Load Balancer for the Kubernetes API Server - Part 1
Install and configure Kubernetes cluster master nodes using kubeadm - Part 2
Install and configure Kubernetes cluster worker nodes using kubeadm - Part 3
I have installed Ubuntu server in my lab, It has assigned IP address and hostname. First task is get root access using command sudo su -. Update and upgrade Ubuntu os system using apt-get update -y && apt-get upgrade -y. Next Install NGINX package with command apt-get install nginx -y.
ubuntu@k8slb:~$ sudo su - root@k8slb:~# root@k8slb:~# apt-get update -y && apt-get upgrade -y Hit:1 http://us.archive.ubuntu.com/ubuntu focal InRelease Get:2 http://us.archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] Get:4 http://us.archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] Fetched 336 kB in 12s (28.2 kB/s) Reading package lists... Done Reading package lists... Done Building dependency tree Reading state information... Done Calculating upgrade... Done Try Ubuntu Pro beta with a free personal subscription on up to 5 machines. Learn more at https://ubuntu.com/pro 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. root@k8slb:~# root@k8slb:~# apt-get install nginx -y Reading package lists... Done Building dependency tree Reading state information... Done The following additional packages will be installed: libnginx-mod-http-image-filter libnginx-mod-http-xslt-filter libnginx-mod-mail libnginx-mod-stream nginx-common nginx-core Suggested packages: fcgiwrap nginx-doc The following NEW packages will be installed: libnginx-mod-http-image-filter libnginx-mod-http-xslt-filter libnginx-mod-mail libnginx-mod-stream nginx nginx-common nginx-core 0 upgraded, 7 newly installed, 0 to remove and 0 not upgraded. Need to get 605 kB of archives. After this operation, 2,134 kB of additional disk space will be used. Do you want to continue? [Y/n] Y Get:1 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 nginx-common all 1.18.0-0ubuntu1.3 [37.7 kB] Get:2 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 libnginx-mod-http-image-filter amd64 1.18.0-0ubuntu1.3 [14.8 kB] Get:3 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 libnginx-mod-http-xslt-filter amd64 1.18.0-0ubuntu1.3 [13.0 kB] Get:4 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 libnginx-mod-mail amd64 1.18.0-0ubuntu1.3 [42.8 kB] Get:5 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 libnginx-mod-stream amd64 1.18.0-0ubuntu1.3 [67.3 kB] Get:6 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 nginx-core amd64 1.18.0-0ubuntu1.3 [425 kB] Get:7 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 nginx all 1.18.0-0ubuntu1.3 [3,620 B] Fetched 605 kB in 10s (58.5 kB/s) Preconfiguring packages ... Selecting previously unselected package nginx-common. (Reading database ... 182290 files and directories currently installed.) Preparing to unpack .../0-nginx-common_1.18.0-0ubuntu1.3_all.deb ... Unpacking nginx-common (1.18.0-0ubuntu1.3) ... Selecting previously unselected package libnginx-mod-http-image-filter. Preparing to unpack .../1-libnginx-mod-http-image-filter_1.18.0-0ubuntu1.3_amd64.deb ... Unpacking libnginx-mod-http-image-filter (1.18.0-0ubuntu1.3) ... Selecting previously unselected package libnginx-mod-http-xslt-filter. Preparing to unpack .../2-libnginx-mod-http-xslt-filter_1.18.0-0ubuntu1.3_amd64.deb ... Unpacking libnginx-mod-http-xslt-filter (1.18.0-0ubuntu1.3) ... Selecting previously unselected package libnginx-mod-mail. Preparing to unpack .../3-libnginx-mod-mail_1.18.0-0ubuntu1.3_amd64.deb ... Unpacking libnginx-mod-mail (1.18.0-0ubuntu1.3) ... Selecting previously unselected package libnginx-mod-stream. Preparing to unpack .../4-libnginx-mod-stream_1.18.0-0ubuntu1.3_amd64.deb ... Unpacking libnginx-mod-stream (1.18.0-0ubuntu1.3) ... Selecting previously unselected package nginx-core. Preparing to unpack .../5-nginx-core_1.18.0-0ubuntu1.3_amd64.deb ... Unpacking nginx-core (1.18.0-0ubuntu1.3) ... Selecting previously unselected package nginx. Preparing to unpack .../6-nginx_1.18.0-0ubuntu1.3_all.deb ... Unpacking nginx (1.18.0-0ubuntu1.3) ... Setting up nginx-common (1.18.0-0ubuntu1.3) ... Created symlink /etc/systemd/system/multi-user.target.wants/nginx.service → /lib/systemd/system/nginx.service. Setting up libnginx-mod-http-xslt-filter (1.18.0-0ubuntu1.3) ... Setting up libnginx-mod-mail (1.18.0-0ubuntu1.3) ... Setting up libnginx-mod-http-image-filter (1.18.0-0ubuntu1.3) ... Setting up libnginx-mod-stream (1.18.0-0ubuntu1.3) ... Setting up nginx-core (1.18.0-0ubuntu1.3) ... Setting up nginx (1.18.0-0ubuntu1.3) ... Processing triggers for systemd (245.4-4ubuntu3.18) ... Processing triggers for man-db (2.9.1-1) ... Processing triggers for ufw (0.36-6ubuntu1) ... root@k8slb:~# root@k8slb:~# curl localhost
Once NGINX web server is installed. Update firewall table to allow NGINX ports and traffic. Below are the commands ufw app list and ufw allow 'Nginx Full'. Check the status of Nginx daemon service with systemctl status nginx. Service is running and enabled.
You can additionally allow port 6443 of kubernetes api server with sudo ufw allow 6443 and sudo ufw allow 6443/tcp.
root@k8slb:~# ufw app list Available applications: CUPS Nginx Full Nginx HTTP Nginx HTTPS OpenSSH root@k8slb:~# root@k8slb:~# ufw allow 'Nginx Full' Rules updated Rules updated (v6) root@k8slb:~# root@k8slb:~# ufw allow 'Nginx HTTP' Rules updated Rules updated (v6) root@k8slb:~# root@k8slb:~# ufw allow 'Nginx HTTPS' Rules updated Rules updated (v6) root@k8slb:~# root@k8slb:~# ufw status Status: inactive root@k8slb:~# root@k8slb:~# systemctl status nginx ● nginx.service - A high performance web server and a reverse proxy server Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2022-10-07 06:33:57 EDT; 13min ago Docs: man:nginx(8) Main PID: 37306 (nginx) Tasks: 3 (limit: 4572) Memory: 3.8M CGroup: /system.slice/nginx.service ├─37306 nginx: master process /usr/sbin/nginx -g daemon on; master_process on; ├─37307 nginx: worker process └─37308 nginx: worker process Oct 07 06:33:56 k8slb systemd[1]: Starting A high performance web server and a reverse proxy server... Oct 07 06:33:57 k8slb systemd[1]: Started A high performance web server and a reverse proxy server. root@k8slb:~#
NGINX package is installed. I need few more packages to modify NGINX configuration file and test kubernetes cluster API url later. Install packages using command apt-get install vim curl wget -y.
root@k8slb:~# apt-get install vim curl wget -y
Reading package lists... Done
Building dependency tree
Reading state information... Done
curl is already the newest version (7.68.0-1ubuntu2.13).
wget is already the newest version (1.20.3-1ubuntu2).
The following additional packages will be installed:
vim-runtime
Suggested packages:
ctags vim-doc vim-scripts
The following NEW packages will be installed:
vim vim-runtime
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 7,111 kB of archives.
After this operation, 34.6 MB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 vim-runtime all 2:8.1.2269-1ubuntu5.9 [5,873 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 vim amd64 2:8.1.2269-1ubuntu5.9 [1,238 kB]
Fetched 7,111 kB in 1min 1s (116 kB/s)
Selecting previously unselected package vim-runtime.
(Reading database ... 182368 files and directories currently installed.)
Preparing to unpack .../vim-runtime_2%3a8.1.2269-1ubuntu5.9_all.deb ...
Adding 'diversion of /usr/share/vim/vim81/doc/help.txt to /usr/share/vim/vim81/doc/help.txt.vim-tiny by vim-runtime'
Adding 'diversion of /usr/share/vim/vim81/doc/tags to /usr/share/vim/vim81/doc/tags.vim-tiny by vim-runtime'
Unpacking vim-runtime (2:8.1.2269-1ubuntu5.9) ...
Selecting previously unselected package vim.
Preparing to unpack .../vim_2%3a8.1.2269-1ubuntu5.9_amd64.deb ...
Unpacking vim (2:8.1.2269-1ubuntu5.9) ...
Setting up vim-runtime (2:8.1.2269-1ubuntu5.9) ...
Setting up vim (2:8.1.2269-1ubuntu5.9) ...
update-alternatives: using /usr/bin/vim.basic to provide /usr/bin/vim (vim) in auto mode
update-alternatives: using /usr/bin/vim.basic to provide /usr/bin/vimdiff (vimdiff) in auto mode
update-alternatives: using /usr/bin/vim.basic to provide /usr/bin/rvim (rvim) in auto mode
update-alternatives: using /usr/bin/vim.basic to provide /usr/bin/rview (rview) in auto mode
update-alternatives: using /usr/bin/vim.basic to provide /usr/bin/vi (vi) in auto mode
update-alternatives: using /usr/bin/vim.basic to provide /usr/bin/view (view) in auto mode
update-alternatives: using /usr/bin/vim.basic to provide /usr/bin/ex (ex) in auto mode
Processing triggers for man-db (2.9.1-1) ...
root@k8slb:~#
Enable nginx service systemctl enable nginx, so it will auto start after OS reboot. Create new folder/directory with mkdir -p /etc/nginx/tcpconf.d. Add Kubernetes cluster master nodes configuration to kubernetes.conf file as shown below. I am creating TCP stream, defining upstream which has name of kubernetes. There are 3 master node servers mentioned in the upstream. On these 3 servers I am going to load balance the traffic. Below are IPs belongs to my lab kubernetes cluster control-plane master nodes as shown in the above design. I am using port 6443, which is used by Kubernetes api server and it listens on the port.
root@k8slb:~# systemctl enable nginx Synchronizing state of nginx.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install enable nginx root@k8slb:~# root@k8slb:~# mkdir -p /etc/nginx/tcpconf.d root@k8slb:~# root@k8slb:~# cat << EOF | sudo tee /etc/nginx/tcpconf.d/kubernetes.conf > stream { > upstream kubernetes { > server 192.168.34.61:6443; > server 192.168.34.62:6443; > server 192.168.34.63:6443; > } > server { > listen 6443; > listen 443; > proxy_pass kubernetes; > } > } > EOF stream { upstream kubernetes { server 192.168.34.61:6443; server 192.168.34.62:6443; server 192.168.34.63:6443; } server { listen 6443; listen 443; proxy_pass kubernetes; } } root@k8slb:~#
Configuration file is configured, Add the configuration file pointer in the main nginx.conf file under located folder /etc/nginx. Go to the bottom of the file and add statement include /etc/nginx/tcpconf.d/*;. It will make sure it is pointing to all the file under tcpconf.d directory.
Instead of using text editor, with this one liner command you can add statement to the nginx.conf file - echo 'include /etc/nginx/tcpconf.d/*;' >> /etc/nginx/nginx.conf. Verify the changes using command cat /etc/nginx/nginx.conf.
root@k8slb:~# echo 'include /etc/nginx/tcpconf.d/*;' >> /etc/nginx/nginx.conf root@k8slb:~# root@k8slb:~# cat /etc/nginx/nginx.conf user www-data; worker_processes auto; pid /run/nginx.pid; include /etc/nginx/modules-enabled/*.conf; events { worker_connections 768; # multi_accept on; } http { ## # Basic Settings ## sendfile on; tcp_nopush on; tcp_nodelay on; keepalive_timeout 65; types_hash_max_size 2048; # server_tokens off; # server_names_hash_bucket_size 64; # server_name_in_redirect off; include /etc/nginx/mime.types; default_type application/octet-stream; ## # SSL Settings ## ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE ssl_prefer_server_ciphers on; ## # Logging Settings ## access_log /var/log/nginx/access.log; error_log /var/log/nginx/error.log; ## # Gzip Settings ## gzip on; # gzip_vary on; # gzip_proxied any; # gzip_comp_level 6; # gzip_buffers 16 8k; # gzip_http_version 1.1; # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript; ## # Virtual Host Configs ## include /etc/nginx/conf.d/*.conf; include /etc/nginx/sites-enabled/*; } #mail { # # See sample authentication script at: # # http://wiki.nginx.org/ImapAuthenticateWithApachePhpScript # # # auth_http localhost/auth.php; # # pop3_capabilities "TOP" "USER"; # # imap_capabilities "IMAP4rev1" "UIDPLUS"; # # server { # listen localhost:110; # protocol pop3; # proxy on; # } # # server { # listen localhost:143; # protocol imap; # proxy on; # } #} include /etc/nginx/tcpconf.d/*; root@k8slb:~#
Next perform the very crucial step of reloading the NGINX configuration. If you don't reload the configuration, NGINX will not see the changes made in the configuration file. Use command nginx -s reload or restart daemon using systemctl restart nginx.
root@k8slb:~# nginx -s reload
root@k8slb:~#
In next article I will use this NGINX load balancer to configure in Kubernetes master nodes in control-plane.
Useful Articles
ansible create an array with set_fact
Ansible get information from esxi advanced settings nested dictionary with unique keynames
Install Ansible AWX Tower on Ubuntu Linux
Ansible AWX installation error Cannot have both the docker-py and docker python modules
Ansible AWX installation error docker-compose run --rm --service-ports task awx-manage migrate --no-input
Ansible AWX Tower create Manual SCM (Source Control Credential Type) project
Reset Ansible AWX Tower admin password
Install Ansible AWX on Microsoft Windows OS
Step by Step Install Ansible on Ubuntu OS
Install Ansible AWX Tower on Ubuntu Linux OS
Ansible AWX Tower Github inventory integration | Github inventory source
docker: Got permission denied while trying to connect to the Docker daemon socket
