simple321vip/knowledge-base

Fork 0

Files

T

simple321vip 909c458b07 add kownledge ahout mechine learning

2026-08-25 22:28:36 +08:00

20 KiB

Raw Blame History

Kubernetes 最佳实践

来源：原 mongodb 集合 t_blog · 整理日期：2026-08-25 分类 ID：2022050319356100000000000001 · 排序：4 · 文章数：8

1. Mongo Replica 集群搭建

_{bid: 2023033021115000000000000029}

Replica Set 集群搭建

本篇文章记录了基于mongodb-kubernetes-operator【Replica Set】模式集群搭建。 1 2 3

1. 事前准备

kubernetes 版本： 1.23.9
mongodb 版本：6.0.2
nfs：提前搭建好了kubernetes的nfs系统（具体搭建可以参照）。

2. github项目mongodb-kubernetes-operator

项目地址

我们使用mongodb 官网推荐的 mongodb-kubernetes-operator 来进行集群安装。
document

我们事先把 mongodb-kubernetes-operator 项目中我们要使用的yaml文件复制到我们的工作目录 mongo, 需要的文件夹如下：

+ mongo
  + config/crd/bases/
  + config/rbac/
  + config/manager/
  + config/samples/mongodb.com_v1_mongodbcommunity_cr.yaml
  + deploy/clusterwide/

$ pwd
/usr/local/kubernetes/mongo
$ ls
config deploy

3. 按照document依次进行安装

3.1 创建 mongodb 使用的namespace

kubectl create ns kube-mongo

3.2 创建集群rbac 权限

# 修改 deploy/clusterwide/cluster_role_binding.yaml 中namespace 为 3.1 中我我们创建的 [kube-mongo]
kubectl apply -f deploy/clusterwide
kubectl apply -k config/rbac --namespace kube-mongo

3.3 创建 crd

kubectl apply -f config/crd/bases/mongodbcommunity.mongodb.com_mongodbcommunity.yaml

3.4 安装operator

kubectl create -f config/manager/manager.yaml --namespace kube-mongo

3.5 安装mongodb

修改 config/samples/mongodb.com_v1_mongodbcommunity_cr.yaml文件

修改 MongoDBCommunity 资源我们修改 metadata.name=violin-mongodb 我们修改 spec.version= "6.0.2" 我们修改 spec.users.name[0].passwordSecretRef.name= kube-mongo-pw
修改 Secret 资源我们修改 metadata.name= kube-mongo-pw 我们修改 stringData.password= "任意随意写了吧"

然后进行安装

kubectl apply -f config/samples/mongodb.com_v1_mongodbcommunity_cr.yaml --namespace kube-mongo

3.6 重新创建pvc

由于我使用的是自定义的storageclass，pvc 处于 pending状态，我们删除掉pvc

kubectl delete pvc data-volume-violin-mongodb-0 -n kube-mongo
kubectl delete pvc logs-volume-violin-mongodb-0 -n kube-mongo

我们创建新的pvc定义文件：

# pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-volume-violin-mongodb-0
  namespace: kube-mongo
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: managed-nfs-storage
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-volume-violin-mongodb-1
  namespace: kube-mongo
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: managed-nfs-storage
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-volume-violin-mongodb-2
  namespace: kube-mongo
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: managed-nfs-storage
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: logs-volume-violin-mongodb-0
  namespace: kube-mongo
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: managed-nfs-storage
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: logs-volume-violin-mongodb-1
  namespace: kube-mongo
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: managed-nfs-storage
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: logs-volume-violin-mongodb-2
  namespace: kube-mongo
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: managed-nfs-storage

我们创建 pvc

kubectl apply -f pvc.yaml 
# 查看pvc 是否已
kubectl get pvc -n kube-mongo
kube-mongo     data-volume-violin-mongodb-0   Bound    pvc-1ac1a5fd-389c-496d-a47c-7df11d5f5a34   10Gi       RWO            managed-nfs-storage   41m
kube-mongo     data-volume-violin-mongodb-1   Bound    pvc-39008253-8702-4cc5-876a-0561850929bb   10Gi       RWO            managed-nfs-storage   61m
kube-mongo     data-volume-violin-mongodb-2   Bound    pvc-a7e867d3-b499-486a-9b39-a42ac5a00b43   10Gi       RWO            managed-nfs-storage   61m
kube-mongo     logs-volume-violin-mongodb-0   Bound    pvc-209ed46c-d558-4a09-8a2f-edfddbed610c   1Gi        RWO            managed-nfs-storage   41m
kube-mongo     logs-volume-violin-mongodb-1   Bound    pvc-664b307f-ab95-4af3-8851-17a3ecd2b57c   1Gi        RWO            managed-nfs-storage   61m
kube-mongo     logs-volume-violin-mongodb-2   Bound    pvc-a8f3a8d2-337d-4470-9ef4-5dbe18a4100a   1Gi        RWO            managed-nfs-storage   61m

4. 关于删除资源的问题。

我们创建operator 或者 statefuset 出错的时候，可以重新删除他们。

4.1 删除 operator

kubectl delete deployments.apps mongodb-kubernetes-operator -n kube-mongo

4.1 删除 statefulset

kubectl delete statefulsets.apps violin-mongodb -n kube-mongo

5. 关于 mongosh的使用

kubectl get secret <connection-string-secret-name> -n <my-namespace> -o json | jq -r '.data | with_entries(.value |= @base64d)'

kubectl get secret violin-mongodb-admin-my-user -n kube-mongo -o json | jq -r '.data | with_entries(.value |= @base64d)'

2. kubernetes nfs

_{bid: 2023040608509800000000000030}

1 概述

本篇文章记录了violin-home网站NFS系统搭建过程。
采用腾讯云的3台公网云轻量服务器（ubuntu系统）

NFS服务需要开启 mountd,nfs,nlockmgr,portmapper,rquotad这5个服务.

其中

# 查看服务使用端口
# nfs : 2049
# portmapper: 111
# mountd, nlockmgr, rquotad 是随机的

# 所以事先要做好端口规划, 在轻量云服务器上开放对应的端口.

rpcinfo -p

2 nfs 服务器搭建

# install server 
apt-get install nfs-kernel-server

# enable and start server
systemctl enable nfs-server
systemctl start nfs-server

# confirm nfs-server status 

# create storage 
mkdir -p /data/nfs-volume && chmod -R 777 /data/nfs-volume

# edit config file
cat > /etc/exports << EOF
/data/nfs-volume 49.233.4.79(rw,sync,no_root_squash) 43.138.55.43(rw,sync,no_root_squash) 43.138.73.106(rw,sync,no_root_squash)
EOF

# view nfs port usage
rpcinfo -p

# set port infomation for mountd、rquotad ser
# edit nfs-common status
vi /etc/default/nfs-common
STATDOPTS="--port 31000"

# edit nfs-kernel-server status
vi /etc/default/nfs-kernel-server
RPCMOUNTDOPTS="--manage-gids -p 31001"

# edit nlockmgr ,就算进行了这一步操作，也可能不生效。
vi /etc/modprobe.d/options.conf
options lockd nlm_udpport=31002 nlm_tcpport=31002

# to add locked
vi /etc/modules

# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
lockd

vi /etc/sysctl.conf 
fs.nfs.nlm_udpport=31002
fs.nfs.nlm_tcpport=31002

# enable sysctl.conf
/sbin/sysctl -p

# reload server and service
systemctl reload nfs-server
systemctl restart rpcbind.service
systemctl restart nfs.service

# view nfs port usage again
rpcinfo -p

# to confirm client ip
showmount -e 49.233.4.79
# Export list for 49.233.4.79:
# /data/nfs-volume 43.138.73.106,43.138.55.43,49.233.4.79

3 nfs 客户端搭建

下面操作需要在每台node上操作，包括master节点。

# install client
apt-get install nfs-common

# create mount path for client(we use different path to server)
mkdir /opt/nfs-volume

# mount 
mount -t nfs 49.233.4.79:/data/nfs-volume /opt/nfs-volume
df -h | tail -1

# config auto mount for client
cat >> /etc/fstab << EOF
49.233.4.79:/data/nfs-volume /opt/nfs-volume defaults,_netdev 0 0
EOF

4 3. Mongo Replica 集群搭建-1

_{bid: 2023040623407900000000000031}

Replica Set 集群搭建

本篇文章记录了基于 helm bitnami/mongodb chat的【Replica Set】模式集群搭建。

参照文档

1. 事前准备

kubernetes 版本： 1.23.9
mongodb 版本：6.0.2
nfs：提前搭建好了kubernetes的nfs系统（具体搭建可以参照）。 storageclass：提前准备好了基于nfs的storageclass。
helm：已经安装了helm

2. 添加bitnami repo

helm repo add bitnami https://charts.bitnami.com/bitnami

3. 查询 mongodb 相关资源

helm repo update
helm search repo mongodb

NAME                    CHART VERSION   APP VERSION     DESCRIPTION
bitnami/mongodb         13.9.4          6.0.5           MongoDB(R) is a relational open source NoSQL da...
bitnami/mongodb-sharded 6.3.3           6.0.5           MongoDB(R) is an open source NoSQL database tha...

4. 创建helm 配置文件

## values.yaml
global:
  namespaceOverride: kube-mongo
  storageClass: "managed-nfs-storage"
nameOverride: violin-mongo
image:
  pullPolicy: IfNotPresent
architecture: "replicaset"
replicaCount: 3
#auth:
#  rootUser: "root"
#  rootPassword: "123654"
#  usernames:
#    - "guan"
#  passwords: 
#    - "83201048"
#  databases:
#    - "violin"
resources:
  requests:
    cpu: "0.2"
    memory: 200M
  limits: 
    cpu: "0.3"
    memory: 300M
containerPorts:
  mongodb: 27017
externalAccess:
  enabled: true
  service:
    externalTrafficPolicy: "Cluster"
    type: "ClusterIP"
persistence:
  enabled: true
  size: 5Gi

5. 安装 mongodb

helm install mongodb bitnami/mongodb -f values.yaml 

NAME: mongodb
LAST DEPLOYED: Fri Apr  7 11:49:07 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: mongodb
CHART VERSION: 13.9.4
APP VERSION: 6.0.5

** Please be patient while the chart is being deployed **

MongoDB&reg; can be accessed on the following DNS name(s) and ports from within your cluster:

    mongodb-violin-mongo-0.mongodb-violin-mongo-headless.kube-mongo.svc.cluster.local:27017
    mongodb-violin-mongo-1.mongodb-violin-mongo-headless.kube-mongo.svc.cluster.local:27017
    mongodb-violin-mongo-2.mongodb-violin-mongo-headless.kube-mongo.svc.cluster.local:27017

To get the root password run:

    export MONGODB_ROOT_PASSWORD=$(kubectl get secret --namespace kube-mongo mongodb-violin-mongo -o jsonpath="{.data.mongodb-root-password}" | base64 -d)

To get the password for "guan" run:

    export MONGODB_PASSWORD=$(kubectl get secret --namespace kube-mongo mongodb-violin-mongo -o jsonpath="{.data.mongodb-passwords}" | base64 -d | awk -F',' '{print $1}')

To connect to your database, create a MongoDB&reg; client container:

    kubectl run --namespace kube-mongo mongodb-violin-mongo-client --rm --tty -i --restart='Never' --env="MONGODB_ROOT_PASSWORD=$MONGODB_ROOT_PASSWORD" --image docker.io/bitnami/mongodb:6.0.5-debian-11-r4 --command -- bash

Then, run the following command:
    mongosh admin --host "mongodb-violin-mongo-0.mongodb-violin-mongo-headless.kube-mongo.svc.cluster.local:27017,mongodb-violin-mongo-1.mongodb-violin-mongo-headless.kube-mongo.svc.cluster.local:27017,mongodb-violin-mongo-2.mongodb-violin-mongo-headless.kube-mongo.svc.cluster.local:27017" --authenticationDatabase admin -u root -p $MONGODB_ROOT_PASSWORD

To connect to your database nodes from outside, you need to add both primary and secondary nodes hostnames/IPs to your Mongo client. To obtain them, follow the instructions below:

6. connect

db.createUser({user: "root", pwd: "123654", roles: ["root"]})

4. kubernetes

_{bid: 2023040717276500000000000032}

kubernetes 1.25.2 base Containerd

1. deploy schedule

NAME | IP | ROLES | VERSION | OS k8s-master | 192.168.8.30 | master | 1.25.2 | ubuntu 22.04 k8s-node1 | 192.168.8.31 | worker | 1.25.2 | ubuntu 22.04 k8s-node2 | 192.168.8.32 | worker | 1.25.2 | ubuntu 22.04

2. preparation

2.1 安装系统工具

apt-get update && apt-get install -y apt-transport-https

2.2 安装 GPG 证书

curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -

2.3 写入软件源

cat << EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF

2.4 vi hostname

vi /etc/hostname
vi /etc/hosts
reboot

2.5 selinux

sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

2.6 禁用交换分区

swapoff -a  
sed -ri 's/.*swap.*/#&/' /etc/fstab

2.7 iptables

echo "1" >/proc/sys/net/bridge/bridge-nf-call-iptables
echo 1 > /proc/sys/net/ipv4/ip_forward
vi /etc/sysctl.conf
net.ipv4.ip_forward = 1

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

sudo sysctl --system

2.8

3 install kubectl kubelet kebeadm

3.1 install

apt-get install -y kubelet=1.25.2-00 kubeadm=1.25.2-00 kubectl=1.25.2-00

4 Containerd

4.1 install Containerd

apt-get install containerd.io

4.2 generate config file

containerd config default | sudo tee /etc/containerd/config.toml

4.3 edit config.toml

sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.6"
SystemdCgroup = true

4.4 restart

systemctl restart containerd && systemctl enable containerd
crictl version

4.5 edit crictl.yaml

runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false

4.6 confirm crictl

crictl images

5 kubeadm

5.1 kubeadm init

kubeadm init --kubernetes-version=1.25.2 \
--apiserver-advertise-address=192.168.8.30 \
--apiserver-bind-port=6443 \
--image-repository=registry.aliyuncs.com/google_containers \
--service-cidr=10.96.0.0/12 \
--cri-socket unix:///var/run/containerd/containerd.sock  \
--ignore-preflight-errors=Swap \
--control-plane-endpoint=k8s-master \
--upload-certs

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join k8s-master:6443 --token 81jkv9.2n555fci6m6i9afx \
	--discovery-token-ca-cert-hash sha256:9ffe6e7ed7cf4ad6eb0e0dbb5b6f3bae1cac6989c3870653c83fe424715b4008 \
	--control-plane --certificate-key c16ea4431de14998ccb28282b5e6215ab03c9ba94287903c6a8482add718034a

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join k8s-master:6443 --token 81jkv9.2n555fci6m6i9afx \
	--discovery-token-ca-cert-hash sha256:9ffe6e7ed7cf4ad6eb0e0dbb5b6f3bae1cac6989c3870653c83fe424715b4008

5.2 kubeconfig

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
export KUBECONFIG=/etc/kubernetes/admin.conf

5.3 kubectl

kubectl get noe

NAME         STATUS     ROLES           AGE    VERSION
k8s-master   NotReady   control-plane   135m   v1.25.2

5. kubeadm

_{bid: 2023041008541300000000000036}

1. xxx

22.04.1-Ubuntu

cat << EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/kubernetes-xenial main
EOF

apt-get update
apt-get install -y kubelet=1.25.2-00 kubeadm=1.25.2-00 kubectl=1.25.2-00

sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

临时关闭swap分区，当前会话生效，重启失效永久关闭swap分区

swapoff -a  
sed -ri 's/.*swap.*/#&/' /etc/fstab

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

sudo sysctl --system

echo "1" >/proc/sys/net/bridge/bridge-nf-call-iptables
echo 1 > /proc/sys/net/ipv4/ip_forward
vi /etc/sysctl.conf
net.ipv4.ip_forward = 1

cat > /etc/sysconfig/network-scripts/ifcfg-eth0 <<EOF
BOOTPROTO=static
DEVICE=eth0
IPADDR=172.16.0.11
PREFIX=32
TYPE=Ethernet
USERCTL=no
ONBOOT=yes
EOF

vi /etc/hosts

echo "######## edit /etc/hosts ########"
echo "" >> /etc/hosts
echo 192.168.8.30 k8s-master >> /etc/hosts
echo 192.168.8.31 k8s-node1 >> /etc/hosts
echo 192.168.8.32 k8s-node2 >> /etc/hosts

about docker and cri-dockerd

docker pull golang:1.20.2

download cri-dockerd

echo "######### https://github.com/Mirantis/cri-dockerd/releases/tag/v0.3.2 ########'
tar -zxvf cri-dockerd-0.3.2.tar.gz

echo "######## chuangjian guazai dian ########"
mkdir /data
cd /data
mkdir go
cd go
echo "######## download cri-dockerd ########"
echo "######## https://github.com/Mirantis/cri-dockerd/releases/tag/v0.3.2 ########'
echo "######## jieya ########"
tar -zxvf cri-dockerd-0.3.2.tar.gz

echo "######## yunxing golang container ########"
docker run -it -v /data/go:/data/go  golang:1.20.2 /bin/sh

enter container

cd /data/go/cri-dockerd-0.3.2
mkdir bin
go build -o bin/cri-dockerd

out container

cd /data/go/cri-dockerd-0.3.2
install -o root -g root -m 0755 bin/cri-dockerd /usr/local/bin/cri-dockerd
cp -a packaging/systemd/* /etc/systemd/system
sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service
systemctl daemon-reload
systemctl enable cri-docker.service
systemctl enable --now cri-docker.socket

6. kubernetes containerd

_{bid: 2023041415557100000000000037}

containerd

ctr

crictl

nerdctl

7. 新增加work节点

_{bid: 2023061019139700000000000038}

8. kubelet

_{bid: 2024042923524000000000000040}

该写点什么么...

20 KiB Raw Blame History Unescape Escape

Kubernetes 最佳实践

目录

1. Mongo Replica 集群搭建

Replica Set 集群搭建

1. 事前准备

2. github项目mongodb-kubernetes-operator

3. 按照document依次进行安装

3.1 创建 mongodb 使用的namespace

3.2 创建集群rbac 权限

3.3 创建 crd

3.4 安装operator

3.5 安装mongodb

3.6 重新创建pvc

4. 关于删除资源的问题。

4.1 删除 operator

4.1 删除 statefulset

5. 关于 mongosh的使用

2. kubernetes nfs

1 概述

2 nfs 服务器搭建

3 nfs 客户端搭建

4

3. Mongo Replica 集群搭建-1

Replica Set 集群搭建

1. 事前准备

2. 添加bitnami repo

3. 查询 mongodb 相关资源

4. 创建helm 配置文件

5. 安装 mongodb

6. connect

4. kubernetes

kubernetes 1.25.2 base Containerd

1. deploy schedule

2. preparation

2.1 安装系统工具

2.2 安装 GPG 证书

2.3 写入软件源

2.4 vi hostname

2.5 selinux

2.6 禁用交换分区

2.7 iptables

2.8

3 install kubectl kubelet kebeadm

3.1 install

4 Containerd

4.1 install Containerd

4.2 generate config file

4.3 edit config.toml

4.4 restart

4.5 edit crictl.yaml

4.6 confirm crictl

5 kubeadm

5.1 kubeadm init

5.2 kubeconfig

5.3 kubectl

5. kubeadm

1. xxx

about docker and cri-dockerd

6. kubernetes containerd

containerd

ctr

crictl

nerdctl

7. 新增加work节点

8. kubelet

20 KiB

Raw Blame History