# Kubernetes 最佳实践

> 来源：原 mongodb 集合 `t_blog`  ·  整理日期：2026-08-25
> 分类 ID：`2022050319356100000000000001`  ·  排序：4  ·  文章数：8

## 目录

1. [Mongo Replica 集群搭建](#mongo-replica-集群搭建)
2. [kubernetes nfs](#kubernetes-nfs)
3. [Mongo Replica 集群搭建-1](#mongo-replica-集群搭建-1)
4. [kubernetes](#kubernetes)
5. [kubeadm](#kubeadm)
6. [kubernetes containerd](#kubernetes-containerd)
7. [新增加work节点](#新增加work节点)
8. [kubelet](#kubelet)

---

## 1. Mongo Replica 集群搭建

<sub>bid: `2023033021115000000000000029`</sub>

## Replica Set 集群搭建
本篇文章记录了基于mongodb-kubernetes-operator【Replica Set】模式集群搭建。
[1]()
[2](https://github.com/mongodb/mongodb-kubernetes-operator)
[3](https://github.com/mongodb/mongodb-kubernetes-operator/blob/master/docs/install-upgrade.md#install-the-operator-using-kubectl)

### 1. 事前准备

> kubernetes 版本： 1.23.9  
> mongodb 版本：6.0.2  
> nfs： 提前搭建好了kubernetes的nfs系统（具体搭建可以参照）。

### 2. github项目mongodb-kubernetes-operator

[项目地址](https://github.com/mongodb/mongodb-kubernetes-operator)

我们使用mongodb 官网推荐的 mongodb-kubernetes-operator 来进行集群安装。  
[document](https://github.com/mongodb/mongodb-kubernetes-operator/blob/master/docs/install-upgrade.md#install-the-operator-using-kubectl)

我们事先把 mongodb-kubernetes-operator 项目中我们要使用的yaml文件复制到我们的工作目录 mongo, 需要的文件夹如下：

    + mongo
      + config/crd/bases/
      + config/rbac/
      + config/manager/
      + config/samples/mongodb.com_v1_mongodbcommunity_cr.yaml
      + deploy/clusterwide/

    $ pwd
    /usr/local/kubernetes/mongo
    $ ls
    config deploy
    
### 3. 按照document依次进行安装
#### 3.1 创建 mongodb 使用的namespace

    kubectl create ns kube-mongo

#### 3.2 创建集群rbac 权限
  
    # 修改 deploy/clusterwide/cluster_role_binding.yaml 中namespace 为 3.1 中我我们创建的 [kube-mongo]
    kubectl apply -f deploy/clusterwide
    kubectl apply -k config/rbac --namespace kube-mongo

#### 3.3 创建 crd

    kubectl apply -f config/crd/bases/mongodbcommunity.mongodb.com_mongodbcommunity.yaml
    
#### 3.4 安装operator

    kubectl create -f config/manager/manager.yaml --namespace kube-mongo

#### 3.5 安装mongodb
修改 config/samples/mongodb.com_v1_mongodbcommunity_cr.yaml文件  
- 修改 MongoDBCommunity 资源
我们修改 metadata.name=violin-mongodb
我们修改 spec.version= "6.0.2"
我们修改 spec.users.name[0].passwordSecretRef.name= kube-mongo-pw
- 修改 Secret 资源
我们修改 metadata.name= kube-mongo-pw
我们修改 stringData.password= "任意随意写了吧"

然后进行安装

    kubectl apply -f config/samples/mongodb.com_v1_mongodbcommunity_cr.yaml --namespace kube-mongo

#### 3.6 重新创建pvc
由于我使用的是自定义的storageclass，pvc 处于 pending状态，我们删除掉pvc

    kubectl delete pvc data-volume-violin-mongodb-0 -n kube-mongo
    kubectl delete pvc logs-volume-violin-mongodb-0 -n kube-mongo

我们创建新的pvc定义文件：

    # pvc.yaml
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: data-volume-violin-mongodb-0
      namespace: kube-mongo
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
      storageClassName: managed-nfs-storage
    ---
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: data-volume-violin-mongodb-1
      namespace: kube-mongo
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
      storageClassName: managed-nfs-storage
    ---
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: data-volume-violin-mongodb-2
      namespace: kube-mongo
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
      storageClassName: managed-nfs-storage
    ---
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: logs-volume-violin-mongodb-0
      namespace: kube-mongo
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi
      storageClassName: managed-nfs-storage
    ---
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: logs-volume-violin-mongodb-1
      namespace: kube-mongo
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi
      storageClassName: managed-nfs-storage
    ---
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: logs-volume-violin-mongodb-2
      namespace: kube-mongo
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi
      storageClassName: managed-nfs-storage

 我们 创建 pvc 

    kubectl apply -f pvc.yaml 
    # 查看pvc 是否已
    kubectl get pvc -n kube-mongo
    kube-mongo     data-volume-violin-mongodb-0   Bound    pvc-1ac1a5fd-389c-496d-a47c-7df11d5f5a34   10Gi       RWO            managed-nfs-storage   41m
    kube-mongo     data-volume-violin-mongodb-1   Bound    pvc-39008253-8702-4cc5-876a-0561850929bb   10Gi       RWO            managed-nfs-storage   61m
    kube-mongo     data-volume-violin-mongodb-2   Bound    pvc-a7e867d3-b499-486a-9b39-a42ac5a00b43   10Gi       RWO            managed-nfs-storage   61m
    kube-mongo     logs-volume-violin-mongodb-0   Bound    pvc-209ed46c-d558-4a09-8a2f-edfddbed610c   1Gi        RWO            managed-nfs-storage   41m
    kube-mongo     logs-volume-violin-mongodb-1   Bound    pvc-664b307f-ab95-4af3-8851-17a3ecd2b57c   1Gi        RWO            managed-nfs-storage   61m
    kube-mongo     logs-volume-violin-mongodb-2   Bound    pvc-a8f3a8d2-337d-4470-9ef4-5dbe18a4100a   1Gi        RWO            managed-nfs-storage   61m

### 4. 关于删除资源的问题。
我们创建operator 或者 statefuset 出错的时候，可以重新删除他们。

#### 4.1 删除 operator
    
    kubectl delete deployments.apps mongodb-kubernetes-operator -n kube-mongo

#### 4.1 删除 statefulset
    
    kubectl delete statefulsets.apps violin-mongodb -n kube-mongo

### 5. 关于 mongosh的使用

    kubectl get secret <connection-string-secret-name> -n <my-namespace> -o json | jq -r '.data | with_entries(.value |= @base64d)'

    kubectl get secret violin-mongodb-admin-my-user -n kube-mongo -o json | jq -r '.data | with_entries(.value |= @base64d)'

---

## 2. kubernetes nfs

<sub>bid: `2023040608509800000000000030`</sub>

## 1 概述
---
&emsp;&emsp;本篇文章记录了violin-home网站NFS系统搭建过程。  
采用腾讯云的3台公网云轻量服务器（ubuntu系统）  
> NFS服务需要开启 mountd,nfs,nlockmgr,portmapper,rquotad这5个服务.  

&emsp;&emsp;其中 

    # 查看服务使用端口
    # nfs : 2049
    # portmapper: 111
    # mountd, nlockmgr, rquotad 是随机的

    # 所以事先要做好端口规划, 在轻量云服务器上开放对应的端口.
```bash
rpcinfo -p 
```



## 2 nfs 服务器搭建
---

    # install server 
    apt-get install nfs-kernel-server
    
    # enable and start server
    systemctl enable nfs-server
    systemctl start nfs-server

    # confirm nfs-server status 

    # create storage 
    mkdir -p /data/nfs-volume && chmod -R 777 /data/nfs-volume

    # edit config file
    cat > /etc/exports << EOF
    /data/nfs-volume 49.233.4.79(rw,sync,no_root_squash) 43.138.55.43(rw,sync,no_root_squash) 43.138.73.106(rw,sync,no_root_squash)
    EOF

    # view nfs port usage
    rpcinfo -p

    # set port infomation for mountd、rquotad ser
    # edit nfs-common status
    vi /etc/default/nfs-common
    STATDOPTS="--port 31000"
    
    # edit nfs-kernel-server status
    vi /etc/default/nfs-kernel-server
    RPCMOUNTDOPTS="--manage-gids -p 31001"
   
    # edit nlockmgr ,就算进行了这一步操作，也可能不生效。
    vi /etc/modprobe.d/options.conf
    options lockd nlm_udpport=31002 nlm_tcpport=31002
    
    # to add locked
    vi /etc/modules

    # /etc/modules: kernel modules to load at boot time.
    #
    # This file contains the names of kernel modules that should be loaded
    # at boot time, one per line. Lines beginning with "#" are ignored.
    lockd

    vi /etc/sysctl.conf 
    fs.nfs.nlm_udpport=31002
    fs.nfs.nlm_tcpport=31002
    
    # enable sysctl.conf
    /sbin/sysctl -p

    # reload server and service
    systemctl reload nfs-server
    systemctl restart rpcbind.service
    systemctl restart nfs.service

    # view nfs port usage again
    rpcinfo -p

    # to confirm client ip
    showmount -e 49.233.4.79
    # Export list for 49.233.4.79:
    # /data/nfs-volume 43.138.73.106,43.138.55.43,49.233.4.79

## 3 nfs 客户端搭建
---
&emsp;&emsp;下面操作需要在每台node上操作，包括master节点。

    # install client
    apt-get install nfs-common

    # create mount path for client(we use different path to server)
    mkdir /opt/nfs-volume
    
    # mount 
    mount -t nfs 49.233.4.79:/data/nfs-volume /opt/nfs-volume
    df -h | tail -1

    # config auto mount for client
    cat >> /etc/fstab << EOF
    49.233.4.79:/data/nfs-volume /opt/nfs-volume defaults,_netdev 0 0
    EOF

## 4

---

## 3. Mongo Replica 集群搭建-1

<sub>bid: `2023040623407900000000000031`</sub>

## Replica Set 集群搭建
本篇文章记录了基于 helm bitnami/mongodb chat的【Replica Set】模式集群搭建。

[参照文档](https://github.com/bitnami/charts/tree/main/bitnami/mongodb)

### 1. 事前准备

> kubernetes 版本： 1.23.9  
> mongodb 版本：6.0.2  
> nfs： 提前搭建好了kubernetes的nfs系统（具体搭建可以参照）。
> storageclass： 提前准备好了基于nfs的storageclass。  
> helm： 已经安装了helm

### 2. 添加bitnami repo

    helm repo add bitnami https://charts.bitnami.com/bitnami

### 3. 查询 mongodb 相关资源

    helm repo update
    helm search repo mongodb

    NAME                    CHART VERSION   APP VERSION     DESCRIPTION
    bitnami/mongodb         13.9.4          6.0.5           MongoDB(R) is a relational open source NoSQL da...
    bitnami/mongodb-sharded 6.3.3           6.0.5           MongoDB(R) is an open source NoSQL database tha...

### 4. 创建helm 配置文件

    ## values.yaml
    global:
      namespaceOverride: kube-mongo
      storageClass: "managed-nfs-storage"
    nameOverride: violin-mongo
    image:
      pullPolicy: IfNotPresent
    architecture: "replicaset"
    replicaCount: 3
    #auth:
    #  rootUser: "root"
    #  rootPassword: "123654"
    #  usernames:
    #    - "guan"
    #  passwords: 
    #    - "83201048"
    #  databases:
    #    - "violin"
    resources:
      requests:
        cpu: "0.2"
        memory: 200M
      limits: 
        cpu: "0.3"
        memory: 300M
    containerPorts:
      mongodb: 27017
    externalAccess:
      enabled: true
      service:
        externalTrafficPolicy: "Cluster"
        type: "ClusterIP"
    persistence:
      enabled: true
      size: 5Gi 

### 5. 安装 mongodb
  
    helm install mongodb bitnami/mongodb -f values.yaml 

    NAME: mongodb
    LAST DEPLOYED: Fri Apr  7 11:49:07 2023
    NAMESPACE: default
    STATUS: deployed
    REVISION: 1
    TEST SUITE: None
    NOTES:
    CHART NAME: mongodb
    CHART VERSION: 13.9.4
    APP VERSION: 6.0.5
    
    ** Please be patient while the chart is being deployed **
    
    MongoDB&reg; can be accessed on the following DNS name(s) and ports from within your cluster:
    
        mongodb-violin-mongo-0.mongodb-violin-mongo-headless.kube-mongo.svc.cluster.local:27017
        mongodb-violin-mongo-1.mongodb-violin-mongo-headless.kube-mongo.svc.cluster.local:27017
        mongodb-violin-mongo-2.mongodb-violin-mongo-headless.kube-mongo.svc.cluster.local:27017
    
    To get the root password run:
    
        export MONGODB_ROOT_PASSWORD=$(kubectl get secret --namespace kube-mongo mongodb-violin-mongo -o jsonpath="{.data.mongodb-root-password}" | base64 -d)
    
    To get the password for "guan" run:
    
        export MONGODB_PASSWORD=$(kubectl get secret --namespace kube-mongo mongodb-violin-mongo -o jsonpath="{.data.mongodb-passwords}" | base64 -d | awk -F',' '{print $1}')
    
    To connect to your database, create a MongoDB&reg; client container:
    
        kubectl run --namespace kube-mongo mongodb-violin-mongo-client --rm --tty -i --restart='Never' --env="MONGODB_ROOT_PASSWORD=$MONGODB_ROOT_PASSWORD" --image docker.io/bitnami/mongodb:6.0.5-debian-11-r4 --command -- bash
    
    Then, run the following command:
        mongosh admin --host "mongodb-violin-mongo-0.mongodb-violin-mongo-headless.kube-mongo.svc.cluster.local:27017,mongodb-violin-mongo-1.mongodb-violin-mongo-headless.kube-mongo.svc.cluster.local:27017,mongodb-violin-mongo-2.mongodb-violin-mongo-headless.kube-mongo.svc.cluster.local:27017" --authenticationDatabase admin -u root -p $MONGODB_ROOT_PASSWORD
    
    To connect to your database nodes from outside, you need to add both primary and secondary nodes hostnames/IPs to your Mongo client. To obtain them, follow the instructions below:


### 6. connect

    db.createUser({user: "root", pwd: "123654", roles: ["root"]})

---

## 4. kubernetes

<sub>bid: `2023040717276500000000000032`</sub>

## kubernetes 1.25.2 base Containerd

### 1. deploy schedule
NAME | IP | ROLES | VERSION | OS
k8s-master | 192.168.8.30 | master | 1.25.2 | ubuntu 22.04
k8s-node1 | 192.168.8.31 | worker | 1.25.2 | ubuntu 22.04
k8s-node2 | 192.168.8.32 | worker | 1.25.2 | ubuntu 22.04

### 2. preparation

#### 2.1 安装系统工具
```bash
apt-get update && apt-get install -y apt-transport-https
```
#### 2.2 安装 GPG 证书
```bash
curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -
```
#### 2.3 写入软件源
```bash
cat << EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF
```
#### 2.4 vi hostname
```bash
vi /etc/hostname
vi /etc/hosts
reboot
```
#### 2.5 selinux
```bash
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
```
#### 2.6 禁用交换分区
```bash
swapoff -a  
sed -ri 's/.*swap.*/#&/' /etc/fstab
```
#### 2.7 iptables
```bash
echo "1" >/proc/sys/net/bridge/bridge-nf-call-iptables
echo 1 > /proc/sys/net/ipv4/ip_forward
vi /etc/sysctl.conf
net.ipv4.ip_forward = 1

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

sudo sysctl --system
```
#### 2.8 

### 3 install kubectl kubelet kebeadm
#### 3.1 install
```bash
apt-get install -y kubelet=1.25.2-00 kubeadm=1.25.2-00 kubectl=1.25.2-00
```
### 4 Containerd
#### 4.1 install Containerd
```bash
apt-get install containerd.io
```
#### 4.2 generate config file
```bash
containerd config default | sudo tee /etc/containerd/config.toml
```
#### 4.3 edit config.toml
```bash
sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.6"
SystemdCgroup = true
```
#### 4.4 restart
```bash
systemctl restart containerd && systemctl enable containerd
crictl version
```
#### 4.5 edit crictl.yaml
```bash
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
```
#### 4.6 confirm crictl 
```bash
crictl images
```
### 5 kubeadm

#### 5.1 kubeadm init
```bash
kubeadm init --kubernetes-version=1.25.2 \
--apiserver-advertise-address=192.168.8.30 \
--apiserver-bind-port=6443 \
--image-repository=registry.aliyuncs.com/google_containers \
--service-cidr=10.96.0.0/12 \
--cri-socket unix:///var/run/containerd/containerd.sock  \
--ignore-preflight-errors=Swap \
--control-plane-endpoint=k8s-master \
--upload-certs
```
```
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join k8s-master:6443 --token 81jkv9.2n555fci6m6i9afx \
	--discovery-token-ca-cert-hash sha256:9ffe6e7ed7cf4ad6eb0e0dbb5b6f3bae1cac6989c3870653c83fe424715b4008 \
	--control-plane --certificate-key c16ea4431de14998ccb28282b5e6215ab03c9ba94287903c6a8482add718034a

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join k8s-master:6443 --token 81jkv9.2n555fci6m6i9afx \
	--discovery-token-ca-cert-hash sha256:9ffe6e7ed7cf4ad6eb0e0dbb5b6f3bae1cac6989c3870653c83fe424715b4008 
```
#### 5.2 kubeconfig
```bash
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
export KUBECONFIG=/etc/kubernetes/admin.conf
```
#### 5.3 kubectl
```bash
kubectl get noe
```
```
NAME         STATUS     ROLES           AGE    VERSION
k8s-master   NotReady   control-plane   135m   v1.25.2
```

---

## 5. kubeadm

<sub>bid: `2023041008541300000000000036`</sub>

## 1. xxx

22.04.1-Ubuntu

```bash
cat << EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/kubernetes-xenial main
EOF
```
```bash
apt-get update
apt-get install -y kubelet=1.25.2-00 kubeadm=1.25.2-00 kubectl=1.25.2-00
```

```bash
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
```

临时关闭swap分区，当前会话生效，重启失效
永久关闭swap分区
```bash
swapoff -a  
sed -ri 's/.*swap.*/#&/' /etc/fstab
```
```bash
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

sudo sysctl --system
```

```bash
echo "1" >/proc/sys/net/bridge/bridge-nf-call-iptables
echo 1 > /proc/sys/net/ipv4/ip_forward
vi /etc/sysctl.conf
net.ipv4.ip_forward = 1
```

```bash
cat > /etc/sysconfig/network-scripts/ifcfg-eth0 <<EOF
BOOTPROTO=static
DEVICE=eth0
IPADDR=172.16.0.11
PREFIX=32
TYPE=Ethernet
USERCTL=no
ONBOOT=yes
EOF
```
vi /etc/hosts
```bash
echo "######## edit /etc/hosts ########"
echo "" >> /etc/hosts
echo 192.168.8.30 k8s-master >> /etc/hosts
echo 192.168.8.31 k8s-node1 >> /etc/hosts
echo 192.168.8.32 k8s-node2 >> /etc/hosts
```

## about docker and cri-dockerd

```bash
docker pull golang:1.20.2
```
download cri-dockerd
```bash
echo "######### https://github.com/Mirantis/cri-dockerd/releases/tag/v0.3.2 ########'
tar -zxvf cri-dockerd-0.3.2.tar.gz
```


```bash
echo "######## chuangjian guazai dian ########"
mkdir /data
cd /data
mkdir go
cd go
echo "######## download cri-dockerd ########"
echo "######## https://github.com/Mirantis/cri-dockerd/releases/tag/v0.3.2 ########'
echo "######## jieya ########"
tar -zxvf cri-dockerd-0.3.2.tar.gz

echo "######## yunxing golang container ########"
docker run -it -v /data/go:/data/go  golang:1.20.2 /bin/sh
```
enter container 
```bash
cd /data/go/cri-dockerd-0.3.2
mkdir bin
go build -o bin/cri-dockerd
```
out container

```bash
cd /data/go/cri-dockerd-0.3.2
install -o root -g root -m 0755 bin/cri-dockerd /usr/local/bin/cri-dockerd
cp -a packaging/systemd/* /etc/systemd/system
sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service
systemctl daemon-reload
systemctl enable cri-docker.service
systemctl enable --now cri-docker.socket
```

---

## 6. kubernetes containerd

<sub>bid: `2023041415557100000000000037`</sub>

## containerd




## ctr

## crictl

## nerdctl

##

---

## 7. 新增加work节点

<sub>bid: `2023061019139700000000000038`</sub>

##

---

## 8. kubelet

<sub>bid: `2024042923524000000000000040`</sub>

该写点什么么...

---