openeuler22.03 Kubeadm安装k8s 1.29.4版本 ,CRI->containerd, CNI->flannel

  • 操作系统:openEuler 22.03 (LTS-SP3)
  • 容器运行时:containerd://1.7.16
# 此内容为部署完的k8s集群信息

[root@openeuler-1 ~]# kubectl get node -owide
NAME          STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                    KERNEL-VERSION                       CONTAINER-RUNTIME
openeuler-1   Ready    control-plane   11m   v1.29.4   172.8.8.11    <none>        openEuler 22.03 (LTS-SP3)   5.10.0-182.0.0.95.oe2203sp3.x86_64   containerd://1.7.16

安装容器运行时

安装containerd,并配置开机自启动

https://github.com/containerd/containerd/blob/main/docs/getting-started.md

wget https://github.com/containerd/containerd/releases/download/v1.7.16/containerd-1.7.16-linux-amd64.tar.gz
tar Cxzvf /usr/local containerd-1.7.16-linux-amd64.tar.gz
[root@cloud ~]# cat /usr/lib/systemd/system/containerd.service
# Copyright The containerd Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target

[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd

Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5

# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity

# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target

[root@cloud ~]# systemctl daemon-reload
[root@cloud ~]# systemctl enable --now containerd

安装runc

wget https://github.com/opencontainers/runc/releases/download/v1.1.12/runc.amd64
install -m 755 runc.amd64 /usr/local/sbin/runc

安装cni插件

wget https://github.com/containernetworking/plugins/releases/download/v1.4.1/cni-plugins-linux-amd64-v1.4.1.tgz
mkdir -p /opt/cni/bin
tar Cxzvf /opt/cni/bin cni-plugins-linux-amd64-v1.4.1.tgz

# 创建 /etc/crictl.yaml 文件
[root@openeuler-1 ~]# cat /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 2
debug: false
pull-image-on-create: false

生成containerd配置文件

[root@cloud ~]# mkdir /etc/containerd
[root@cloud ~]# containerd config default > /etc/containerd/config.toml

配置 systemd cgroup 驱动

结合 runc 使用 systemd cgroup 驱动,在 /etc/containerd/config.toml 中设置:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  ...
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true

修改sandbox_image镜像地址

  sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.9"

重启containerd
sudo systemctl restart containerd

安装前参数配置

编辑hosts文件,添加本机host地址

[root@cloud ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.0.2 cloud

手动启用 IPv4 数据包转发

modprobe br_netfilter

# 设置所需的 sysctl 参数,参数在重新启动后保持不变
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

vim /etc/sysctl.conf # 设置以下值为1,不然会覆盖成0
net.ipv4.ip_forward=1

# 应用 sysctl 参数而不重新启动
sudo sysctl --system

# 验证
sysctl net.ipv4.ip_forward
sysctl net.bridge.bridge-nf-call-iptables

将 SELinux 设置为 permissive 模式

# 将 SELinux 设置为 permissive 模式(相当于将其禁用)
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

关闭swap

如果 kubelet 未被正确配置使用交换分区,则你必须禁用交换分区。 例如,sudo swapoff -a 将暂时禁用交换分区。要使此更改在重启后保持不变,请确保在如 /etc/fstab、systemd.swap 等配置文件中禁用交换分区,具体取决于你的系统如何配置

安装 k8s

配置yum源,下载 kubelet kubeadm kubectl

如果下载慢可以更换阿里云镜像源。

# 这会覆盖 /etc/yum.repos.d/kubernetes.repo 中现存的所有配置

cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.29/rpm/
enabled=1
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.29/rpm/repodata/repomd.xml.key
EOF

# 下载安装最新的 kubelet kubeadm kubectl
sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

# 立即启动、开机自启
sudo systemctl enable --now kubelet

# 查看各个组件版本
kubectl version --client
kubelet --version
kubeadm version

kubeadm初始化

kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address 172.8.8.11 --image-repository registry.aliyuncs.com/google_containers
  • pod-network-cidr: 规划的pod cidr 段
  • apiserver-advertise-address: 服务器地址
  • image-repository:用阿里云镜像快速下载

结果如下:

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.8.8.11:6443 --token wzz40f.q6mvvg00u4th4wsm \
        --discovery-token-ca-cert-hash sha256:73d5a3305e67ec46d67ff8e4f4e47ba14c5a948f94dca05fcb024c8361c88b61 

安装cni插件 flannel

此时coredns pod状态异常,需要安装cni插件。

官网:https://github.com/flannel-io/flannel#deploying-flannel-manually

wget https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

修改pod cidr段为上面init时的cidr段(默认不用修改):

  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }

执行:

kubectl apply -f kube-flannel.yml 

结果:

[root@openeuler-1 ~]# kubectl get po -A
NAMESPACE      NAME                                  READY   STATUS    RESTARTS   AGE
kube-flannel   kube-flannel-ds-vpdts                 1/1     Running   0          6m3s
kube-system    coredns-857d9ff4c9-45dj5              1/1     Running   0          7m22s
kube-system    coredns-857d9ff4c9-dph2s              1/1     Running   0          7m22s
kube-system    etcd-openeuler-1                      1/1     Running   0          7m38s
kube-system    kube-apiserver-openeuler-1            1/1     Running   0          7m38s
kube-system    kube-controller-manager-openeuler-1   1/1     Running   0          7m38s
kube-system    kube-proxy-f6qxk                      1/1     Running   0          7m23s
kube-system    kube-scheduler-openeuler-1            1/1     Running   0          7m38s

pod service 测试

去除节点上的污点,允许pod调度到该节点

kubectl taint nodes openeuler-1 node-role.kubernetes.io/control-plane:NoSchedule-

pod yaml:

[root@cloud yaml]# cat simple-pod.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    app.kubernetes.io/name: MyApp
spec:
  containers:
  - name: nginx
    image: nginx:1.14.2
    ports:
    - containerPort: 80

service yaml:

[root@cloud yaml]# cat my-service.yaml 
apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  selector:
    app.kubernetes.io/name: MyApp
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 80

执行,并查看结果:

kubectl apply -f simple-pod.yaml 
kubectl apply -f simple-pod.yaml 

[root@cloud yaml]# kubectl get po -owide
NAME    READY   STATUS    RESTARTS   AGE   IP          NODE    NOMINATED NODE   READINESS GATES
nginx   1/1     Running   0          43m   10.10.0.4   cloud   <none>           <none>

[root@cloud yaml]# kubectl get svc -owide
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE   SELECTOR
kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP   82m   <none>
my-service   ClusterIP   10.108.55.244   <none>        80/TCP    35m   app.kubernetes.io/name=MyApp

# curl pod ip 查看结果:
[root@cloud yaml]# curl 10.10.0.4
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...
</body>
</html>

# curl service ip 查看结果:
[root@cloud yaml]# curl 10.108.55.244
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...
</body>
</html>

问题排错

'systemctl status kubelet'
'journalctl -xeu kubelet'

查看service cidr

kubectl cluster-info dump | grep service-cluster-ip-range