一、安装要求

在开始之前，部署Kubernetes集群机器需要满足以下几个条件：

• 一台或多台机器，操作系统欧拉系统
• 硬件配置：2GB或更多RAM，2个CPU或更多CPU，硬盘30GB或更多
• 集群中所有机器之间网络互通
• 可以访问外网，需要拉取镜像
• 禁止swap分区

1.2 主机硬件配置说明

CPU	内存	角色	主机名
4C	4G	Master	k8s-master
4C	4G	Node01	k8s-node01
4C	4G	Node02	k8s-node02

二、主机准备

2.1 主机名配置

由于本次使用3台主机完成kubernetes集群部署，其中1台为master节点,名称为k8s-master01;其中2台为node节点，名称分别为：k8s-node01及k8s-node02

master节点

hostnamectl set-hostname k8s-master01

node01节点

hostnamectl set-hostname k8s-node01

node02节点

hostnamectl set-hostname k8s-node02

在master添加hosts：

cat >> /etc/hosts << EOF
192.168.10.1 k8s-master
192.168.10.2  k8s-node1
192.168.10.3  k8s-node2
EOF

2.2 主机IP地址配置

k8s-master01节点IP地址为：192.168.10.1/24

vim /etc/sysconfig/network-scripts/ifcfg-ens33
TYPE="Ethernet"
PROXY_METHOD="none"
BROWSER_ONLY="no"
BOOTPROTO="none"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
NAME="ens33"
DEVICE="ens33"
ONBOOT="yes"
IPADDR="192.168.10.1"
PREFIX="24"
GATEWAY="192.168.10.254"
DNS1="8.8.8.8"

k8s-node01节点IP地址为：192.168.10.2/24

vim /etc/sysconfig/network-scripts/ifcfg-ens33
TYPE="Ethernet"
PROXY_METHOD="none"
BROWSER_ONLY="no"
BOOTPROTO="none"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
NAME="ens33"
DEVICE="ens33"
ONBOOT="yes"
IPADDR="192.168.10.2"
PREFIX="24"
GATEWAY="192.168.10.254"
DNS1="8.8.8.8"

k8s-node02节点IP地址为：192.168.10.3/24

vim /etc/sysconfig/network-scripts/ifcfg-ens33
TYPE="Ethernet"
PROXY_METHOD="none"
BROWSER_ONLY="no"
BOOTPROTO="none"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
NAME="ens33"
DEVICE="ens33"
ONBOOT="yes"
IPADDR="192.168.10.3"
PREFIX="24"
GATEWAY="192.168.10.254"
DNS1="8.8.8.8"

2.3 主机名与IP地址解析

所有集群主机均需要进行配置。

cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.10.1 k8s-master01
192.168.10.2 k8s-node01
192.168.10.3 k8s-node02

2.4 防火墙配置

所有主机均需要操作。

关闭现有防火墙firewalld

systemctl disable firewalld
systemctl stop firewalld
firewall-cmd --state
not running

2.5 SELINUX配置

所有主机均需要操作。修改SELinux配置需要重启操作系统。

sed -ri 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config

2.6 时间同步配置

所有主机均需要操作。最小化安装系统需要安装ntpdate软件。

yum install ntpdate -y 
ntpdate time1.aliyun.com

修改时区

ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

修改语言

sudo echo 'LANG="en_US.UTF-8"' >> /etc/profile;source /etc/profile

2.7 配置内核转发及网桥过滤

所有主机均需要操作。

将桥接的IPv4流量传递到iptables的链：

cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
net.ipv4.tcp_tw_recycle=0
vm.swappiness=0
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_instances=8192
fs.inotify.max_user_watches=1048576
fs.file-max=52706963
fs.nr_open=52706963
net.ipv6.conf.all.disable_ipv6=1
net.netfilter.nf_conntrack_max=2310720         
EOF
sysctl --system

开启内核路由转发

vi /etc/sysctl.conf
net.ipv4.ip_forward=1

添加网桥过滤及内核转发配置文件

cat /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
vm.swappiness = 0

加载br_netfilter模块

modprobe br_netfilter

查看是否加载

lsmod | grep br_netfilter
br_netfilter           22256  0
bridge                151336  1 br_netfilter

使用默认配置文件生效

sysctl -p

使用新添加配置文件生效

sysctl -p /etc/sysctl.d/k8s.conf

2.8 安装ipset及ipvsadm

所有主机均需要操作。

安装ipset及ipvsadm

yum -y install ipset ipvsadm

配置ipvsadm模块加载方式

添加需要加载的模块

cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
EOF

授权、运行、检查是否加载

chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack

2.9 关闭SWAP分区

修改完成后需要重启操作系统，如不重启，可临时关闭，命令为swapoff -a

临时关闭

swapoff -a

永远关闭swap分区，需要重启操作系统

cat /etc/fstab
......

# /dev/mapper/openeuler-swap none                    swap    defaults        0 0
在上一行中行首添加#

三、容器运行时工具安装及运行

查看是否存在docker软件

yum list | grep docker
pcp-pmda-docker.x86_64                                  5.3.7-2.oe2203sp1
docker-client-java.noarch                               8.11.7-2.oe2203sp1
docker-client-java.src                                  8.11.7-2.oe2203sp1
docker-compose.noarch                                   1.22.0-4.oe2203sp1
docker-compose.src                                      1.22.0-4.oe2203sp1
docker-engine.src                                       2:18.09.0-316.oe220
docker-engine.x86_64                                    2:18.09.0-316.oe220
docker-engine.x86_64                                    2:18.09.0-316.oe220
docker-engine-debuginfo.x86_64                          2:18.09.0-316.oe220
docker-engine-debugsource.x86_64                        2:18.09.0-316.oe220
docker-runc.src                                         1.1.3-9.oe2203sp1
docker-runc.x86_64                                      1.1.3-9.oe2203sp1
podman-docker.noarch                                    1:0.10.1-12.oe2203s
python-docker.src                                       5.0.3-1.oe2203sp1
python-docker-help.noarch                               5.0.3-1.oe2203sp1
python-docker-pycreds.src                               0.4.0-2.oe2203sp1
python-dockerpty.src                                    0.4.1-3.oe2203sp1
python-dockerpty-help.noarch                            0.4.1-3.oe2203sp1
python3-docker.noarch                                   5.0.3-1.oe2203sp1
python3-docker-pycreds.noarch                           0.4.0-2.oe2203sp1
python3-dockerpty.noarch                                0.4.1-3.oe2203sp1

安装docker

dnf install docker
Last metadata expiration check: 0:53:18 ago on 
Dependencies resolved.
===============================================================================
Package                    Architecture               Version                 Repository            Size
===============================================================================
Installing:
docker-engine                      x86_64            2:18.09.0-316.oe2203sp1          OS               38 M
Installing dependencies:
libcgroup                          x86_64            0.42.2-3.oe2203sp1               OS         Transaction Summary
===============================================================================
Install  2 Packages
Total download size: 39 M
Installed size: 160 M
Is this ok [y/N]: y
Downloading Packages:
(1/2): libcgroup-0.42.2-3.oe2203sp1.x86_64.rpm                           396 kB/s |  96 kB     00:00
(2/2): docker-engine-18.09.0-316.oe2203sp1.x86_64.rpm                    10 MB/s |  38 MB     00:03
--------------------------------------------------------------------------------
Total                                                                 10 MB/s |  39 MB     00:03
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing        :                                                               1/1
Running scriptlet: libcgroup-0.42.2-3.oe2203sp1.x86_64                             1/2
Installing       : libcgroup-0.42.2-3.oe2203sp1.x86_64                             1/2
Running scriptlet: libcgroup-0.42.2-3.oe2203sp1.x86_64                             1/2
Installing       : docker-engine-2:18.09.0-316.oe2203sp1.x86_64                     2/2
Running scriptlet: docker-engine-2:18.09.0-316.oe2203sp1.x86_64                     2/2
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /usr/lib/systemd/system/docker.service.
Verifying        : docker-engine-2:18.09.0-316.oe2203sp1.x86_64                     1/2
Verifying        : libcgroup-0.42.2-3.oe2203sp1.x86_64                             2/2
Installed:
docker-engine-2:18.09.0-316.oe2203sp1.x86_64                     libcgroup-0.42.2-3.oe2203sp1.x86_64
Complete!

设置docker开机启动并启动

systemctl enable --now docker

查看docker版本

docker version
Client:
Version:           18.09.0
EulerVersion:      18.09.0.316
API version:       1.39
Go version:        go1.17.3
Git commit:        9b9af2f
Built:             Tue Dec 27 14:25:30 2022
OS/Arch:           linux/amd64
Experimental:      false
Server:
Engine:
Version:          18.09.0
EulerVersion:     18.09.0.316
API version:      1.39 (minimum version 1.12)
Go version:       go1.17.3
Git commit:       9b9af2f
Built:            Tue Dec 27 14:24:56 2022
OS/Arch:          linux/amd64
Experimental:     false

四、K8S软件安装

安装k8s依赖，连接跟踪

dnf install conntrack

k8s master节点安装

dnf install -y kubernetes-kubeadm kubernetes-kubelet kubernetes-master

k8s worker节点安装

dnf install -y kubernetes-kubeadm kubernetes-kubelet kubernetes-node

systemctl enable kubelet

五、K8S集群初始化master

kubeadm init --apiserver-advertise-address=192.168.10.1 --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.20.2 --service-cidr=10.1.0.0/16 --pod-network-cidr=10.244.0.0/16

输出：

[init] Using Kubernetes version: v1.20.2
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING FileExisting-socat]: socat not found in system path
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.1.0.1 192.168.10.1]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master01 localhost] and IPs [192.168.10.1 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master01 localhost] and IPs [192.168.10.1 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 6.502722 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.20" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-master01 as control-plane by adding the labels "node-role.kubernetes.io/master=''" and "node-role.kubernetes.io/control-plane='' (deprecated)"
[mark-control-plane] Marking the node k8s-master01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: jvx2bb.pfd31288qyqcfsn7
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.10.1:6443 --token jvx2bb.pfd31288qyqcfsn7 \
--discovery-token-ca-cert-hash sha256:740fa71f6c5acf156195ce6989cb49b7a64fd061b8bf56e4b1b684cbedafbd40

[root@k8s-master01 ~]# mkdir -p $HOME/.kube
[root@k8s-master01 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@k8s-master01 ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config

六、K8S集群工作节点加入

[root@k8s-node01 ~]kubeadm join 192.168.10.1:6443 --token jvx2bb.pfd31288qyqcfsn7 \
--discovery-token-ca-cert-hash sha256:740fa71f6c5acf156195ce6989cb49b7a64fd061b8bf56e4b1b684cbedafbd40
#
[root@k8s-node02 ~]kubeadm join 192.168.10.1:6443 --token jvx2bb.pfd31288qyqcfsn7 \
--discovery-token-ca-cert-hash sha256:740fa71f6c5acf156195ce6989cb49b7a64fd061b8bf56e4b1b684cbedafbd40
[root@k8s-master01 ~]# kubectl get nodes
NAME           STATUS     ROLES                  AGE     VERSION
k8s-master01   NotReady   control-plane,master   3m59s   v1.20.2
k8s-node01   NotReady   <none>                 18s     v1.20.2
k8s-node02   NotReady   <none>                 10s     v1.20.2

七、K8S集群网络插件使用

[root@k8s-master01 ~]# wget https://docs.projectcalico.org/v3.19/manifests/calico.yaml

由于网络问题，我这边使用离线方式安装插件calico

1. 去github上面下载自己所需的calico离线包，项目地址：

https://github.com/projectcalico/calico

2. 假设要安装最新版本v3.28.0，首先可以下载这个版本的calico.yaml，具体命令是

curl -O -L https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/calico.yaml

3. 下载完成之后可以通过calico.yaml查看需要安装哪些离线包，命令是：

cat calico.yaml | grep image
image: docker.io/calico/cni:v3.28.0
imagePullPolicy: IfNotPresent
image: docker.io/calico/cni:v3.28.0
imagePullPolicy: IfNotPresent
image: docker.io/calico/node:v3.28.0
imagePullPolicy: IfNotPresent
image: docker.io/calico/node:v3.28.0
imagePullPolicy: IfNotPresent
image: docker.io/calico/kube-controllers:v3.28.0
imagePullPolicy: IfNotPresent

4. 通过上述命令，查看到需要安装calico-cni.tar， calico-kube-controllers.tar 和 calico-node.tar三个包，然后需要将这三个包导入到k8s的命名空间中

使用导入命令将这三个包导入到k8s的命名空间中：

docker load -i calico-cni.tar
docker load -i calico-node.tar
docker load -i calico-kube-controllers.tar

5. 导入之后就可以apply calico.yaml 文件了

6. 导入之后查看calico的pod，发现calico和coredns已经起来了

kubectl get pods -n kube-system
NAME                                      READY   STATUS    RESTARTS      AGE
calico-kube-controllers-8d76c5f9b-brv86   1/1     Running   0             22h
calico-node-hxks2                         1/1     Running   0             22h
coredns-66f779496c-9hqsx                  1/1     Running   0             23h
coredns-66f779496c-rcc74                  1/1     Running   0             23h
etcd-kevin-pc                             1/1     Running   4 (28m ago)   23h
kube-apiserver-kevin-pc                   1/1     Running   4 (28m ago)   23h
kube-controller-manager-kevin-pc          1/1     Running   4 (28m ago)   23h
kube-proxy-gglh4                          1/1     Running   1 (28m ago)   23h
kube-scheduler-kevin-pc                   1/1     Running   4 (28m ago)   23h