1. Kubernetes集群搭建实战复盘上周终于把生产环境的K8s集群部署上线了整个过程踩了不少坑也积累了一些实战经验。这次部署的是v1.25版本的Kubernetes采用3个master节点5个worker节点的高可用架构网络插件选的Calico存储方案用的Ceph RBD。下面就把这次部署的核心流程和关键注意事项做个完整记录。重要提示生产环境部署前务必做好备份方案我在第一次部署时就因为ETCD配置错误导致需要全部重装1.1 基础环境准备所有节点统一使用Ubuntu 22.04 LTS系统内核升级到5.15版本。硬件配置方面Master节点8核CPU/32GB内存/200GB SSD需要运行ETCDWorker节点16核CPU/64GB内存/500GB SSD运行业务容器先在所有节点执行基础环境配置# 关闭swap sudo swapoff -a sudo sed -i / swap / s/^/#/ /etc/fstab # 设置内核参数 cat EOF | sudo tee /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables 1 net.bridge.bridge-nf-call-iptables 1 EOF sudo sysctl --system1.2 关键组件安装使用官方推荐的kubeadm工具进行安装版本锁定为1.25.6# 安装dockercontainerd也可 sudo apt-get install -y docker.io sudo systemctl enable docker # 安装kubeadm/kubelet/kubectl sudo apt-get install -y apt-transport-https ca-certificates curl curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.25/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg echo deb [signed-by/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.25/deb/ / | sudo tee /etc/apt/sources.list.d/kubernetes.list sudo apt-get update sudo apt-get install -y kubelet1.25.6-1.1 kubeadm1.25.6-1.1 kubectl1.25.6-1.1 sudo apt-mark hold kubelet kubeadm kubectl2. 集群初始化与高可用配置2.1 首个Master节点初始化创建kubeadm配置文件kubeadm-config.yamlapiVersion: kubeadm.k8s.io/v1beta3 kind: InitConfiguration localAPIEndpoint: advertiseAddress: 192.168.1.101 bindPort: 6443 --- apiVersion: kubeadm.k8s.io/v1beta3 kind: ClusterConfiguration kubernetesVersion: v1.25.6 controlPlaneEndpoint: k8s-api.example.com:6443 networking: podSubnet: 10.244.0.0/16 serviceSubnet: 10.96.0.0/12 apiServer: certSANs: - k8s-api.example.com - 192.168.1.101 - 192.168.1.102 - 192.168.1.103 --- apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration cgroupDriver: systemd执行初始化sudo kubeadm init --configkubeadm-config.yaml --upload-certs初始化完成后会输出加入集群的命令务必保存好这些命令。配置kubectlmkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config2.2 安装网络插件选择Calico作为CNI插件kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/tigera-operator.yaml kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/custom-resources.yaml验证网络状态watch kubectl get pods -n calico-system2.3 加入其他Master节点使用初始化时生成的命令在其他两个Master节点执行sudo kubeadm join k8s-api.example.com:6443 --token xxxx \ --discovery-token-ca-cert-hash sha256:xxxx \ --control-plane --certificate-key xxxx验证高可用kubectl get nodes kubectl -n kube-system get pods3. Worker节点加入与配置3.1 节点加入集群在每个Worker节点执行初始化时生成的join命令sudo kubeadm join k8s-api.example.com:6443 --token xxxx \ --discovery-token-ca-cert-hash sha256:xxxx3.2 节点标签与污点设置根据业务需求设置节点标签kubectl label nodes node1 disktypessd kubectl label nodes node2 gputrue对于特殊节点设置污点kubectl taint nodes node3 dedicatedspecial:NoSchedule4. 存储与网络高级配置4.1 Ceph RBD存储配置先在所有节点安装ceph-commonsudo apt-get install -y ceph-common创建StorageClassapiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: ceph-rbd provisioner: kubernetes.io/rbd parameters: monitors: 10.0.0.1:6789,10.0.0.2:6789,10.0.0.3:6789 adminId: admin adminSecretName: ceph-secret adminSecretNamespace: kube-system pool: kube userId: kube userSecretName: ceph-secret-user fsType: ext4 imageFormat: 2 imageFeatures: layering4.2 网络策略配置示例网络策略限制命名空间间通信apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-all namespace: production spec: podSelector: {} policyTypes: - Ingress - Egress5. 监控与日志方案5.1 Prometheus监控部署使用kube-prometheus-stackhelm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \ --namespace monitoring \ --create-namespace \ --values values.yaml5.2 EFK日志收集部署ElasticsearchFluentdKibanahelm install elasticsearch elastic/elasticsearch --namespace logging helm install fluentd fluent/fluentd --namespace logging helm install kibana elastic/kibana --namespace logging6. 常见问题排查记录6.1 节点NotReady问题可能原因及解决方案网络插件未正常运行 → 检查Calico Pod状态kubelet服务异常 →journalctl -u kubelet查看日志节点资源不足 → 检查内存/磁盘空间6.2 Pod一直Pending排查步骤kubectl describe pod pod-name kubectl get events --sort-by.metadata.creationTimestamp常见原因资源不足不满足节点选择器/亲和性规则PV/PVC问题6.3 网络连通性问题诊断工具# 检查DNS解析 kubectl run -it --rm --imagebusybox:1.28 test -- nslookup kubernetes.default # 检查服务连通性 kubectl run -it --rm --imagenicolaka/netshoot test -- curl http://service-name7. 生产环境优化建议7.1 关键参数调优kubelet配置优化apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration maxPods: 150 kubeReserved: cpu: 1 memory: 2Gi systemReserved: cpu: 1 memory: 2Gi evictionHard: memory.available: 500Mi nodefs.available: 10%7.2 安全加固措施启用Pod安全准入控制定期轮换证书限制Dashboard访问启用审计日志# 检查安全配置 kubectl get --raw/metrics | grep apiserver_audit这次部署最大的教训是一定要先在测试环境充分验证所有配置特别是ETCD和网络插件的参数设置。我们因为Calico的IP池配置错误导致第一次部署后全部重来耽误了整整一天时间。另外建议使用Terraform等IaC工具管理节点配置可以大大减少人为操作失误。
Kubernetes高可用集群部署实战与优化指南
1. Kubernetes集群搭建实战复盘上周终于把生产环境的K8s集群部署上线了整个过程踩了不少坑也积累了一些实战经验。这次部署的是v1.25版本的Kubernetes采用3个master节点5个worker节点的高可用架构网络插件选的Calico存储方案用的Ceph RBD。下面就把这次部署的核心流程和关键注意事项做个完整记录。重要提示生产环境部署前务必做好备份方案我在第一次部署时就因为ETCD配置错误导致需要全部重装1.1 基础环境准备所有节点统一使用Ubuntu 22.04 LTS系统内核升级到5.15版本。硬件配置方面Master节点8核CPU/32GB内存/200GB SSD需要运行ETCDWorker节点16核CPU/64GB内存/500GB SSD运行业务容器先在所有节点执行基础环境配置# 关闭swap sudo swapoff -a sudo sed -i / swap / s/^/#/ /etc/fstab # 设置内核参数 cat EOF | sudo tee /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables 1 net.bridge.bridge-nf-call-iptables 1 EOF sudo sysctl --system1.2 关键组件安装使用官方推荐的kubeadm工具进行安装版本锁定为1.25.6# 安装dockercontainerd也可 sudo apt-get install -y docker.io sudo systemctl enable docker # 安装kubeadm/kubelet/kubectl sudo apt-get install -y apt-transport-https ca-certificates curl curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.25/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg echo deb [signed-by/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.25/deb/ / | sudo tee /etc/apt/sources.list.d/kubernetes.list sudo apt-get update sudo apt-get install -y kubelet1.25.6-1.1 kubeadm1.25.6-1.1 kubectl1.25.6-1.1 sudo apt-mark hold kubelet kubeadm kubectl2. 集群初始化与高可用配置2.1 首个Master节点初始化创建kubeadm配置文件kubeadm-config.yamlapiVersion: kubeadm.k8s.io/v1beta3 kind: InitConfiguration localAPIEndpoint: advertiseAddress: 192.168.1.101 bindPort: 6443 --- apiVersion: kubeadm.k8s.io/v1beta3 kind: ClusterConfiguration kubernetesVersion: v1.25.6 controlPlaneEndpoint: k8s-api.example.com:6443 networking: podSubnet: 10.244.0.0/16 serviceSubnet: 10.96.0.0/12 apiServer: certSANs: - k8s-api.example.com - 192.168.1.101 - 192.168.1.102 - 192.168.1.103 --- apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration cgroupDriver: systemd执行初始化sudo kubeadm init --configkubeadm-config.yaml --upload-certs初始化完成后会输出加入集群的命令务必保存好这些命令。配置kubectlmkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config2.2 安装网络插件选择Calico作为CNI插件kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/tigera-operator.yaml kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/custom-resources.yaml验证网络状态watch kubectl get pods -n calico-system2.3 加入其他Master节点使用初始化时生成的命令在其他两个Master节点执行sudo kubeadm join k8s-api.example.com:6443 --token xxxx \ --discovery-token-ca-cert-hash sha256:xxxx \ --control-plane --certificate-key xxxx验证高可用kubectl get nodes kubectl -n kube-system get pods3. Worker节点加入与配置3.1 节点加入集群在每个Worker节点执行初始化时生成的join命令sudo kubeadm join k8s-api.example.com:6443 --token xxxx \ --discovery-token-ca-cert-hash sha256:xxxx3.2 节点标签与污点设置根据业务需求设置节点标签kubectl label nodes node1 disktypessd kubectl label nodes node2 gputrue对于特殊节点设置污点kubectl taint nodes node3 dedicatedspecial:NoSchedule4. 存储与网络高级配置4.1 Ceph RBD存储配置先在所有节点安装ceph-commonsudo apt-get install -y ceph-common创建StorageClassapiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: ceph-rbd provisioner: kubernetes.io/rbd parameters: monitors: 10.0.0.1:6789,10.0.0.2:6789,10.0.0.3:6789 adminId: admin adminSecretName: ceph-secret adminSecretNamespace: kube-system pool: kube userId: kube userSecretName: ceph-secret-user fsType: ext4 imageFormat: 2 imageFeatures: layering4.2 网络策略配置示例网络策略限制命名空间间通信apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-all namespace: production spec: podSelector: {} policyTypes: - Ingress - Egress5. 监控与日志方案5.1 Prometheus监控部署使用kube-prometheus-stackhelm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \ --namespace monitoring \ --create-namespace \ --values values.yaml5.2 EFK日志收集部署ElasticsearchFluentdKibanahelm install elasticsearch elastic/elasticsearch --namespace logging helm install fluentd fluent/fluentd --namespace logging helm install kibana elastic/kibana --namespace logging6. 常见问题排查记录6.1 节点NotReady问题可能原因及解决方案网络插件未正常运行 → 检查Calico Pod状态kubelet服务异常 →journalctl -u kubelet查看日志节点资源不足 → 检查内存/磁盘空间6.2 Pod一直Pending排查步骤kubectl describe pod pod-name kubectl get events --sort-by.metadata.creationTimestamp常见原因资源不足不满足节点选择器/亲和性规则PV/PVC问题6.3 网络连通性问题诊断工具# 检查DNS解析 kubectl run -it --rm --imagebusybox:1.28 test -- nslookup kubernetes.default # 检查服务连通性 kubectl run -it --rm --imagenicolaka/netshoot test -- curl http://service-name7. 生产环境优化建议7.1 关键参数调优kubelet配置优化apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration maxPods: 150 kubeReserved: cpu: 1 memory: 2Gi systemReserved: cpu: 1 memory: 2Gi evictionHard: memory.available: 500Mi nodefs.available: 10%7.2 安全加固措施启用Pod安全准入控制定期轮换证书限制Dashboard访问启用审计日志# 检查安全配置 kubectl get --raw/metrics | grep apiserver_audit这次部署最大的教训是一定要先在测试环境充分验证所有配置特别是ETCD和网络插件的参数设置。我们因为Calico的IP池配置错误导致第一次部署后全部重来耽误了整整一天时间。另外建议使用Terraform等IaC工具管理节点配置可以大大减少人为操作失误。
