一、环境规划与准备1.1 节点规划与硬件配置本次实验使用 3 台物理服务器虚拟机角色分配如下主机名角色配置IP 地址k8s-masterControl Plane4C 8G192.168.1.10k8s-node1Worker4C 8G192.168.1.11k8s-node2Worker4C 8G192.168.1.12三台机器要在同一网段下注本人项目使用四台笔记本jenkinsharbor一台gitlab一台master主节点一台node12两个节点一台。1.2 操作系统与基础环境操作系统统一使用Anolis OS 8.10内核版本 4.18。所有节点执行以下初始化操作# 关闭防火墙# 学习 / 测试环境直接关闭最简单生产环境不建议完全关闭而是精准开放集群所需端口systemctl disable firewalld --now# 关闭 SELinux# SELinux 是 Linux 强制访问控制系统会限制容器与宿主机文件、网络、设备交互容器挂载宿主机目录、日志、存储卷时SELinux 权限拦截直接导致 Pod 启动失败、文件读写报错阻止网桥、iptables 转发破坏 Pod 跨节点通信kubeadm、containerd 运行时会被 SELinux 拦截集群初始化直接报错。setenforce 0 sed -i s/^SELINUXenforcing$/SELINUXdisabled/ /etc/selinux/config# 关闭 swap kubeadm init 初始化会强制检测 swap没关闭直接初始化失败swapoff -a sed -i / swap / s/^\(.*\)$/#\1/ /etc/fstab# 配置主机名# K8s 集群内节点以主机名作为唯一标识注册到 apiserver主机名混乱会导致节点注册失败、证书异常hostnamectl set-hostname k8s-master # 其他节点分别设为 k8s-node1, k8s-node2# 主机名解析# 集群节点之间需要互相用主机名通信不配置 hosts 会出现主机名无法解析cat /etc/hosts EOF 192.168.1.10 k8s-master 192.168.1.11 k8s-node1 192.168.1.12 k8s-node2 EOF# 内核参数调整容器网络核心cat /etc/sysctl.d/k8s.conf EOF net.bridge.bridge-nf-call-iptables 1 net.bridge.bridge-nf-call-ip6tables 1 net.ipv4.ip_forward 1 EOF sysctl --system# 时间同步# K8s 强制要求集群所有节点时间完全一致yum install -y chronyd systemctl enable chronyd --now1.3 网络架构设计Kubernetes 需要规划两个内部网络Pod 网段192.168.58.0/20分配给每个 Pod 的 IPService 网段10.96.0.0/12分配给 Service 的 Cluster IP选择这两个网段的原因是避免与公司/学校内网已有的网段冲突。如果部署在云上还需要注意 VPC 路由表配置。Kubernetes 的Service是一组 Pod 的稳定访问入口、四层负载均衡、虚拟网关。 Pod 有个致命问题 Pod 会随时重启、销毁、扩容缩容每次重建 IP 都会变你没法固定地址访问容器。 Service 就用来解决这个问题给一组功能相同的 Pod 分配固定不变的虚拟 IP二、Kubernetes 集群搭建2.1 容器运行时安装DockerK8s v1.23 的 kubelet 内置了 dockershim可以直接集成 Docker 作为容器运行时无需额外安装 cri-dockerd。# 安装 Docker yum install -y yum-utils yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo yum install -y docker-ce docker-ce-cli containerd.io systemctl enable docker --now2.2 kubeadm 部署集群# 添加 K8s 源 cat /etc/yum.repos.d/kubernetes.repo EOF [kubernetes] nameKubernetes baseurlhttps://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled1 gpgcheck0 EOF# 安装 kubeadm kubelet kubectl yum install -y kubeadm-1.23.17 kubelet-1.23.17 kubectl-1.23.17 systemctl enable kubelet# master 节点初始化 kubeadm init \ --kubernetes-versionv1.23.17 \ --apiserver-advertise-address192.168.1.10 \ --pod-network-cidr192.168.58.0/20 \ --service-cidr10.96.0.0/12 \初始化成功后会输出kubeadm join命令保存下来node 节点加入集群用。# 配置 kubectl管理员权限 mkdir -p $HOME/.kube cp -i /etc/kubernetes/admin.conf $HOME/.kube/config node 节点加入集群 kubeadm join 192.168.1.10:6443 --token xxx --discovery-token-ca-cert-hash sha256:xxx三、CNI 网络插件——Calico BGP 模式3.1 Calico 安装与配置# 下载 Calico 清单 curl https://raw.githubusercontent.com/projectcalico/calico/v3.26.4/manifests/calico.yaml -O 修改 Pod CIDR如果默认不是 192.168.58.0/20 在 calico.yaml 中找到 CALICO_IPV4POOL_CIDR修改为 - name: CALICO_IPV4POOL_CIDR value: 192.168.58.0/20 #根据你自己的ip网段进行修改 应用 Calico kubectl apply -f calico.yaml 确认所有 calico-node Pod 运行正常 kubectl get pods -n kube-system -o wide3.2 跨节点 Pod 通信原理Calico BGP 模式下每个节点上的 calico-node 会与邻居节点建立 BGP 会话自动交换路由信息。这样 Pod 流量直接通过三层路由转发不需要封包解包延迟更低。VXLAN 隧道封装模式云环境 VPC 限制时使用有封包损耗3.3 遇到的问题与解决四、集群高可用与存储4.1 LVS Keepalived 高可用一主两从不需要 LVS Keepalived。如果你是多个 master 节点做高可用可以配置。4.2 local-path-provisioner 本地存储对于有状态服务如 Nacos、MySQL需要持久化存储。实验环境没有 NAS 或云硬盘用 local-path-provisioner 把宿主机目录映射为 PVC# 部署 local-path-provisioner kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml 设为默认 StorageClass kubectl patch storageclass local-path -p {metadata: {annotations:{storageclass.kubernetes.io/is-default-class:true}}}五、中间件容器化部署5.1 NacosStatefulSet PVCNacos 作为注册中心和配置中心采用 Deployment 方式部署通过预先创建的 PVC 实现数据持久化# nacos-deployment.yaml apiVersion: v1 kind: Service metadata: name: nacos namespace: rent-cloud spec: type: NodePort ports: - name: http port: 8848 targetPort: 8848 nodePort: 30000 ~ 32767不要重复或不写自动生成 - name: rpc port: 9848 targetPort: 9848 selector: app: nacos --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: nacos-data-pvc namespace: rent-cloud spec: storageClassName: local-path accessModes: - ReadWriteOnce resources: requests: storage: 2Gi --- apiVersion: apps/v1 kind: Deployment metadata: name: nacos namespace: rent-cloud spec: replicas: 1 selector: matchLabels: app: nacos template: metadata: labels: app: nacos spec: containers: - name: nacos image: nacos/nacos-server:v2.3.0 env: - name: MODE value: standalone ports: - containerPort: 8848 - containerPort: 9848 volumeMounts: - mountPath: /home/nacos/data name: nacos-data volumes: - name: nacos-data persistentVolumeClaim: claimName: nacos-data-pvc #执行 kubectl apply -f nacos-deployment.yaml5.2 MySQL 容器化MySQL 作为有状态服务使用 StatefulSet volumeClaimTemplates 自动创建 PVCapiVersion: v1 kind: Service metadata: name: mysql namespace: rent-cloud spec: ports: - port: 3306 targetPort: 3306 selector: app: mysql --- apiVersion: apps/v1 kind: StatefulSet metadata: name: mysql namespace: rent-cloud spec: serviceName: mysql replicas: 1 selector: matchLabels: app: mysql template: metadata: labels: app: mysql spec: containers: - name: mysql image: mysql:8.0 env: - name: MYSQL_ROOT_PASSWORD value: root ports: - containerPort: 3306 volumeMounts: - mountPath: /var/lib/mysql name: data volumeClaimTemplates: - metadata: name: data spec: storageClassName: local-path accessModes: - ReadWriteOnce resources: requests: storage: 5Gi #执行 kubectl apply -f mysql-statefulset.yaml5.3 Seata、Sentinel 部署Seata sentinel部署要先建命名空间kubectl create namespace seataapiVersion: apps/v1 kind: Deployment metadata: name: seata-server namespace: seata spec: replicas: 1 selector: matchLabels: app: seata-server template: metadata: labels: app: seata-server spec: containers: - name: seata-server image: seataio/seata-server:1.8.0 ports: - containerPort: 8091 #执行 kubectl apply -f seata-deployment.yamlapiVersion: v1 kind: Service metadata: name: sentinel-dashboard namespace: rent-cloud spec: type: NodePort ports: - name: dashboard port: 8858 targetPort: 8080 nodePort: 30000 ~ 32767不要重复 selector: app: sentinel-dashboard --- apiVersion: apps/v1 kind: Deployment metadata: name: sentinel-dashboard namespace: rent-cloud spec: replicas: 1 selector: matchLabels: app: sentinel-dashboard template: metadata: labels: app: sentinel-dashboard spec: containers: - name: sentinel-dashboard image: bladex/sentinel-dashboard:1.8.8 ports: - containerPort: 8080 #执行 kubectl apply -f sentinel-deployment.yaml六、微服务容器化改造6.1 多阶段 Dockerfile 编写多阶段构建可以大幅减小镜像体积。以一个微服务为例第一阶段编译第二阶段运行如果宿主机上有Maven JDK 17后端可以选择单阶段构建。如下↓本人使用的这种#先确认目录存在如果不存在 mkdir -p /root/app/microservice/gateway-service #编写Dockerfile cat gateway-service/Dockerfile EOF FROM openjdk:17-jdk-slim EXPOSE 8088根据自己的端口号修改 VOLUME /tmp ADD target/*.jar /app.jar RUN bash -c touch /app.jar ENTRYPOINT [java,-jar,-Xms128m,-Xmx512m,/app.jar] EOF Maven 编译打包 cd /root/app/microservice/gateway-service mvn clean package -DskipTests 3. 构建 Docker 镜像 docker build -t gateway-service:latest .如果没有Maven JDK 17可以选择多阶段构建# 阶段1Maven 编译 FROM maven:3.8.6-openjdk-17 AS builder WORKDIR /build COPY pom.xml . COPY src ./src RUN mvn clean package -DskipTests # 阶段2运行 FROM openjdk:17-jdk-slim COPY --frombuilder /build/target/*.jar /app.jar ENTRYPOINT [java,-jar,-Xms128m,-Xmx512m,/app.jar] EXPOSE 80886.2 前端 nginx:alpine 镜像构建cat rentweb-frontend/Dockerfile EOF # 阶段1Node 构建 FROM node:22-alpine AS builder WORKDIR /app COPY package*.json ./ RUN npm install COPY . . RUN npm run build #阶段2Nginx 运行 FROM nginx:alpine COPY --frombuilder /app/dist /usr/share/nginx/html COPY nginx.conf /etc/nginx/conf.d/default.conf EXPOSE 80 EOF # 构建 Docker 镜像Dockerfile 内部会执行 npm install npm run build cd /root/app/microservice/rentweb-frontend docker build -t rentweb-frontend:latest .6.3 K8s 资源清单编写Deployment/Service/Ingress在 master 上执行# 先创建命名空间 kubectl create namespace rent-cloud #先创建文件 mkdir -p /root/k8s-manifests 创建微服务的资源文件这里我只展示了gateway其他同理 cat /root/k8s-manifests/gateway-service.yaml EOF apiVersion: apps/v1 kind: Deployment metadata: name: gateway-service namespace: rent-cloud spec: replicas: 1 selector: matchLabels: app: gateway-service template: metadata: labels: app: gateway-service spec: containers: - name: gateway-service image: gateway-service:latest ports: - containerPort: 8088根据自己需求修改 --- apiVersion: v1 kind: Service metadata: name: gateway-service namespace: rent-cloud spec: ports: - port: 8088 targetPort: 8088 selector: app: gateway-service EOF 部署到 K8s kubectl apply -f /root/k8s-manifests/gateway-service.yaml 查看部署状态 kubectl get pods -n rent-cloud -o wide kubectl get svc -n rent-cloud七、GitLab Jenkins CI/CD 流水线7.1 GitLab 部署与仓库配置GitLab 作为代码仓库使用 Docker 方式部署# 创建 GitLab 容器 docker run -d \ --name gitlab \ --restart always \ -p 22:22 -p 80:80 -p 443:443 \ -v /srv/gitlab/config:/etc/gitlab \ -v /srv/gitlab/logs:/var/log/gitlab \ -v /srv/gitlab/data:/var/opt/gitlab \ gitlab/gitlab-ce:15.11.0-ce.0在 GitLab 中创建项目仓库推送代码记录项目 ID 和 GitLab API Token后续 Jenkins 配置需要用到。本人 代码仓库分为前端和后端7.2 Jenkins Declarative Pipeline 7 阶段设计# Jenkins 使用 Docker 部署 docker run -d \ --name jenkins \ --restart always \ -p 8080:8080 -p 50000:50000 \ -v /srv/jenkins:/var/jenkins_home \ -v /var/run/docker.sock:/var/run/docker.sock \ jenkins/jenkins:lts所需插件Git Plugingitlab-pluginGitLab 自动触发 Jenkins 构建PipelineDocker PipelineKubernetes CLI PluginSonarQube Scanner如果安装不了maven插件可以直接在jenkins容器内部署maven# 后端先编译需要容器内安装 Maven JDK 17 # 进入 Jenkins 容器安装 Maven docker exec -it jenkins bash apt-get update apt-get install -y maven openjdk-17-jdk mvn --version # 前端如果构建时提示需要nodejs插件可以在jenkins容器内安装node直接构建不需要宿主机装 Node # 进入 Jenkins 容器安装 Node.js docker exec -it jenkins bash apt-get update apt-get install -y nodejs npm node --version npm --version exit7.3 Harbor 镜像仓库集成# 下载 Harbor 离线安装包 wget https://github.com/goharbor/harbor/releases/download/v2.8.0/harbor-offline-installer-v2.8.0.tgz tar xzf harbor-offline-installer-v2.8.0.tgz 修改 harbor.yml hostname: 192.168.1.20 # 部署 Harbor 的机器 IP http: port: 80 安装 ./install.sh注意使用 HTTP 需要在所有节点 Docker 配置中添加 insecure-registriescat /etc/docker/daemon.json EOF { insecure-registries: [192.168.1.20:80] } EOF systemctl restart docker7.4 Jenkins Declarative Pipeline 7 阶段设计整个流水线分为 7 个阶段1. Checkout → 从 GitLab 拉取代码 2. Parse Services → 解析本次要构建的服务 3. Maven Build → Maven 编译打包 4. Docker Build → 构建镜像并推送到 Harbor 5. Deploy to K8s → 更新 K8s 资源清单 6. Update YAML → 提交更新后的 YAML 到 GitLab 7. Cleanup → 清理工作空间在项目后端中创建jenkinsfilepipeline { agent any environment { HARBOR_URL harbor.internal:80 PROJECT rent-cloud GITLAB_URL http://gitlab.internal } stages { stage(Check Webhook Loop) { steps { script { COMMIT_MSG sh(returnStdout: true, script: git log -1 --pretty%B).trim() if (COMMIT_MSG.startsWith(auto:)) { echo 检测到自动提交跳过构建 currentBuild.result SUCCESS return } } } } stage(Checkout) { steps { checkout scm script { COMMIT_MSG sh(returnStdout: true, script: git log -1 --pretty%B).trim() echo Commit Message: ${COMMIT_MSG} } } } stage(Parse Services) { steps { script { def matcher (COMMIT_MSG ~ /build\s(\S)/) if (matcher.find()) { BUILD_SERVICES matcher[0][1].split(,) echo 本次构建服务: ${BUILD_SERVICES} } else { BUILD_SERVICES [all] echo 未指定服务全量构建 } } } } stage(Maven Build) { steps { script { def builds [:] BUILD_SERVICES.each { service - if (service all) { builds[all] { sh mvn clean package -DskipTests } } else { builds[service] { sh mvn clean package -DskipTests -pl ${service} -am } } } parallel builds } } } stage(Docker Build Push) { steps { script { def builds [:] BUILD_SERVICES.each { service - builds[service] { def tag ${HARBOR_URL}/${PROJECT}/${service}:${env.BUILD_NUMBER} sh docker build -t ${tag} ./${service} sh docker push ${tag} } } parallel builds } } } stage(Deploy to K8s) { steps { script { BUILD_SERVICES.each { service - def tag ${HARBOR_URL}/${PROJECT}/${service}:${env.BUILD_NUMBER} sh sed -i s|image:.*${service}:.*|image: ${tag}| k8s/${service}.yaml kubectl apply -f k8s/${service}.yaml } } } } stage(Update YAML) { steps { withCredentials([string(credentialsId: gitlab-token, variable: TOKEN)]) { script { sh git add k8s/*.yaml git commit -m auto: update image tag for build ${env.BUILD_NUMBER} git push http://oauth2:${TOKEN}${GITLAB_URL}/rent-cloud/rent-system.git HEAD:main } } } } } post { success { echo 构建部署完成 } failure { echo 构建失败请检查日志 } } }在项目前端中创建jenkinsfilepipeline { agent any environment { HARBOR_URL harbor.internal:80 PROJECT rent-cloud GITLAB_URL http://gitlab.internal } stages { stage(Check Webhook Loop) { steps { script { COMMIT_MSG sh(returnStdout: true, script: git log -1 --pretty%B).trim() if (COMMIT_MSG.startsWith(auto:)) { echo 检测到自动提交跳过构建 currentBuild.result SUCCESS return } } } } stage(Checkout) { steps { checkout scm } } stage(Docker Build Push) { steps { script { def tag ${HARBOR_URL}/${PROJECT}/rentweb-frontend:${env.BUILD_NUMBER} sh docker build -t ${tag} ./rentweb-frontend docker push ${tag} } } } stage(Deploy to K8s) { steps { script { def tag ${HARBOR_URL}/${PROJECT}/rentweb-frontend:${env.BUILD_NUMBER} sh sed -i s|image:.*rentweb-frontend:.*|image: ${tag}| k8s/rentweb-frontend.yaml kubectl apply -f k8s/rentweb-frontend.yaml } } } stage(Update YAML) { steps { withCredentials([string(credentialsId: gitlab-token, variable: TOKEN)]) { sh git add k8s/rentweb-frontend.yaml git commit -m auto: update frontend image tag build ${env.BUILD_NUMBER} git push http://oauth2:${TOKEN}${GITLAB_URL}/rent-cloud/rent-system.git HEAD:main } } } } post { success { echo 前端构建部署完成 } failure { echo 前端构建失败请检查日志 } } }7.5 增量构建机制核心亮点这是 CI/CD 流水线的核心优化点。痛点6 个微服务 前端每次提交代码都全量构建耗时 10 分钟以上。大部分时候只改了 1 个服务没必要全部重新编译。方案通过解析 git commit message 中的关键字只构建变更的服务。具体实现开发者在提交代码时在 commit message 中写明要构建的服务git commit -m fix: 修复登录bug build userPipeline 中的 Parse Services 阶段用正则/build\s(\S)/提取服务名如果写多个服务用逗号分隔build user,order如果没有build关键字执行全量构建效果增量构建从 10min 降到 3min效率提升 60% 以上。7.6 Webhook 循环防护问题Pipeline 的最后一步Update YAML会向 GitLab 推送代码而 GitLab 收到推送又会触发 Jenkins Webhook导致无限循环触发构建。解决在 Update YAML 阶段提交时commit message 以auto:开头。然后在 Jenkins Pipeline 的第一阶段检查 commit message如果以auto:开头直接跳过整个构建stage(Check Webhook Loop) { steps { script { if (COMMIT_MSG.startsWith(auto:)) { echo 检测到自动提交跳过构建 currentBuild.result SUCCESS return } } }八、监控与可观测性8.1 Prometheus Grafana 监控栈# 部署 Prometheus Operator kubectl create namespace monitoring helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm install prometheus prometheus-community/kube-prometheus-stack -n monitoring # 访问 Grafana默认密码 admin/prom-operator kubectl port-forward -n monitoring svc/prometheus-grafana 3000:808.2 JVM/GC/HTTP 指标采集微服务引入 Micrometer Prometheus 依赖Spring Boot 3.x 自带dependency groupIdio.micrometer/groupId artifactIdmicrometer-registry-prometheus/artifactId /dependency配置后在每个微服务的 application.yml 中暴露/actuator/prometheus端点management: endpoints: web: exposure: include: health,prometheus metrics: tags: application: ${spring.application.name}在 Prometheus 中配置 ServiceMonitor 采集这些端点Grafana 导入 JVM Micrometer 仪表盘ID 4701即可看到 JVM 内存、GC 次数、线程数、HTTP 请求吞吐量、连接池使用率等指标。8.3 Zipkin 链路追踪微服务引入 Micrometer Tracing Bravedependency groupIdio.micrometer/groupId artifactIdmicrometer-tracing-bridge-brave/artifactId /dependency dependency groupIdio.zipkin.reporter2/groupId artifactIdzipkin-reporter-brave/artifactId /dependencyZipkin 服务端使用 Docker 部署docker run -d --name zipkin -p 30094:9411 openzipkin/zipkin:2.24微服务配置指向 Zipkin 地址spring: zipkin: base-url: http://zipkin-service.monitoring:941 sleuth: sampler: probability: 1.0这样每个请求经过多个微服务时可以在 Zipkin 中看到完整的调用链路和每个环节的耗时。
从零搭建 Kubernetes 集群与 CI/CD 流水线——基于 Spring Cloud Alibaba 微服务实战
一、环境规划与准备1.1 节点规划与硬件配置本次实验使用 3 台物理服务器虚拟机角色分配如下主机名角色配置IP 地址k8s-masterControl Plane4C 8G192.168.1.10k8s-node1Worker4C 8G192.168.1.11k8s-node2Worker4C 8G192.168.1.12三台机器要在同一网段下注本人项目使用四台笔记本jenkinsharbor一台gitlab一台master主节点一台node12两个节点一台。1.2 操作系统与基础环境操作系统统一使用Anolis OS 8.10内核版本 4.18。所有节点执行以下初始化操作# 关闭防火墙# 学习 / 测试环境直接关闭最简单生产环境不建议完全关闭而是精准开放集群所需端口systemctl disable firewalld --now# 关闭 SELinux# SELinux 是 Linux 强制访问控制系统会限制容器与宿主机文件、网络、设备交互容器挂载宿主机目录、日志、存储卷时SELinux 权限拦截直接导致 Pod 启动失败、文件读写报错阻止网桥、iptables 转发破坏 Pod 跨节点通信kubeadm、containerd 运行时会被 SELinux 拦截集群初始化直接报错。setenforce 0 sed -i s/^SELINUXenforcing$/SELINUXdisabled/ /etc/selinux/config# 关闭 swap kubeadm init 初始化会强制检测 swap没关闭直接初始化失败swapoff -a sed -i / swap / s/^\(.*\)$/#\1/ /etc/fstab# 配置主机名# K8s 集群内节点以主机名作为唯一标识注册到 apiserver主机名混乱会导致节点注册失败、证书异常hostnamectl set-hostname k8s-master # 其他节点分别设为 k8s-node1, k8s-node2# 主机名解析# 集群节点之间需要互相用主机名通信不配置 hosts 会出现主机名无法解析cat /etc/hosts EOF 192.168.1.10 k8s-master 192.168.1.11 k8s-node1 192.168.1.12 k8s-node2 EOF# 内核参数调整容器网络核心cat /etc/sysctl.d/k8s.conf EOF net.bridge.bridge-nf-call-iptables 1 net.bridge.bridge-nf-call-ip6tables 1 net.ipv4.ip_forward 1 EOF sysctl --system# 时间同步# K8s 强制要求集群所有节点时间完全一致yum install -y chronyd systemctl enable chronyd --now1.3 网络架构设计Kubernetes 需要规划两个内部网络Pod 网段192.168.58.0/20分配给每个 Pod 的 IPService 网段10.96.0.0/12分配给 Service 的 Cluster IP选择这两个网段的原因是避免与公司/学校内网已有的网段冲突。如果部署在云上还需要注意 VPC 路由表配置。Kubernetes 的Service是一组 Pod 的稳定访问入口、四层负载均衡、虚拟网关。 Pod 有个致命问题 Pod 会随时重启、销毁、扩容缩容每次重建 IP 都会变你没法固定地址访问容器。 Service 就用来解决这个问题给一组功能相同的 Pod 分配固定不变的虚拟 IP二、Kubernetes 集群搭建2.1 容器运行时安装DockerK8s v1.23 的 kubelet 内置了 dockershim可以直接集成 Docker 作为容器运行时无需额外安装 cri-dockerd。# 安装 Docker yum install -y yum-utils yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo yum install -y docker-ce docker-ce-cli containerd.io systemctl enable docker --now2.2 kubeadm 部署集群# 添加 K8s 源 cat /etc/yum.repos.d/kubernetes.repo EOF [kubernetes] nameKubernetes baseurlhttps://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled1 gpgcheck0 EOF# 安装 kubeadm kubelet kubectl yum install -y kubeadm-1.23.17 kubelet-1.23.17 kubectl-1.23.17 systemctl enable kubelet# master 节点初始化 kubeadm init \ --kubernetes-versionv1.23.17 \ --apiserver-advertise-address192.168.1.10 \ --pod-network-cidr192.168.58.0/20 \ --service-cidr10.96.0.0/12 \初始化成功后会输出kubeadm join命令保存下来node 节点加入集群用。# 配置 kubectl管理员权限 mkdir -p $HOME/.kube cp -i /etc/kubernetes/admin.conf $HOME/.kube/config node 节点加入集群 kubeadm join 192.168.1.10:6443 --token xxx --discovery-token-ca-cert-hash sha256:xxx三、CNI 网络插件——Calico BGP 模式3.1 Calico 安装与配置# 下载 Calico 清单 curl https://raw.githubusercontent.com/projectcalico/calico/v3.26.4/manifests/calico.yaml -O 修改 Pod CIDR如果默认不是 192.168.58.0/20 在 calico.yaml 中找到 CALICO_IPV4POOL_CIDR修改为 - name: CALICO_IPV4POOL_CIDR value: 192.168.58.0/20 #根据你自己的ip网段进行修改 应用 Calico kubectl apply -f calico.yaml 确认所有 calico-node Pod 运行正常 kubectl get pods -n kube-system -o wide3.2 跨节点 Pod 通信原理Calico BGP 模式下每个节点上的 calico-node 会与邻居节点建立 BGP 会话自动交换路由信息。这样 Pod 流量直接通过三层路由转发不需要封包解包延迟更低。VXLAN 隧道封装模式云环境 VPC 限制时使用有封包损耗3.3 遇到的问题与解决四、集群高可用与存储4.1 LVS Keepalived 高可用一主两从不需要 LVS Keepalived。如果你是多个 master 节点做高可用可以配置。4.2 local-path-provisioner 本地存储对于有状态服务如 Nacos、MySQL需要持久化存储。实验环境没有 NAS 或云硬盘用 local-path-provisioner 把宿主机目录映射为 PVC# 部署 local-path-provisioner kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml 设为默认 StorageClass kubectl patch storageclass local-path -p {metadata: {annotations:{storageclass.kubernetes.io/is-default-class:true}}}五、中间件容器化部署5.1 NacosStatefulSet PVCNacos 作为注册中心和配置中心采用 Deployment 方式部署通过预先创建的 PVC 实现数据持久化# nacos-deployment.yaml apiVersion: v1 kind: Service metadata: name: nacos namespace: rent-cloud spec: type: NodePort ports: - name: http port: 8848 targetPort: 8848 nodePort: 30000 ~ 32767不要重复或不写自动生成 - name: rpc port: 9848 targetPort: 9848 selector: app: nacos --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: nacos-data-pvc namespace: rent-cloud spec: storageClassName: local-path accessModes: - ReadWriteOnce resources: requests: storage: 2Gi --- apiVersion: apps/v1 kind: Deployment metadata: name: nacos namespace: rent-cloud spec: replicas: 1 selector: matchLabels: app: nacos template: metadata: labels: app: nacos spec: containers: - name: nacos image: nacos/nacos-server:v2.3.0 env: - name: MODE value: standalone ports: - containerPort: 8848 - containerPort: 9848 volumeMounts: - mountPath: /home/nacos/data name: nacos-data volumes: - name: nacos-data persistentVolumeClaim: claimName: nacos-data-pvc #执行 kubectl apply -f nacos-deployment.yaml5.2 MySQL 容器化MySQL 作为有状态服务使用 StatefulSet volumeClaimTemplates 自动创建 PVCapiVersion: v1 kind: Service metadata: name: mysql namespace: rent-cloud spec: ports: - port: 3306 targetPort: 3306 selector: app: mysql --- apiVersion: apps/v1 kind: StatefulSet metadata: name: mysql namespace: rent-cloud spec: serviceName: mysql replicas: 1 selector: matchLabels: app: mysql template: metadata: labels: app: mysql spec: containers: - name: mysql image: mysql:8.0 env: - name: MYSQL_ROOT_PASSWORD value: root ports: - containerPort: 3306 volumeMounts: - mountPath: /var/lib/mysql name: data volumeClaimTemplates: - metadata: name: data spec: storageClassName: local-path accessModes: - ReadWriteOnce resources: requests: storage: 5Gi #执行 kubectl apply -f mysql-statefulset.yaml5.3 Seata、Sentinel 部署Seata sentinel部署要先建命名空间kubectl create namespace seataapiVersion: apps/v1 kind: Deployment metadata: name: seata-server namespace: seata spec: replicas: 1 selector: matchLabels: app: seata-server template: metadata: labels: app: seata-server spec: containers: - name: seata-server image: seataio/seata-server:1.8.0 ports: - containerPort: 8091 #执行 kubectl apply -f seata-deployment.yamlapiVersion: v1 kind: Service metadata: name: sentinel-dashboard namespace: rent-cloud spec: type: NodePort ports: - name: dashboard port: 8858 targetPort: 8080 nodePort: 30000 ~ 32767不要重复 selector: app: sentinel-dashboard --- apiVersion: apps/v1 kind: Deployment metadata: name: sentinel-dashboard namespace: rent-cloud spec: replicas: 1 selector: matchLabels: app: sentinel-dashboard template: metadata: labels: app: sentinel-dashboard spec: containers: - name: sentinel-dashboard image: bladex/sentinel-dashboard:1.8.8 ports: - containerPort: 8080 #执行 kubectl apply -f sentinel-deployment.yaml六、微服务容器化改造6.1 多阶段 Dockerfile 编写多阶段构建可以大幅减小镜像体积。以一个微服务为例第一阶段编译第二阶段运行如果宿主机上有Maven JDK 17后端可以选择单阶段构建。如下↓本人使用的这种#先确认目录存在如果不存在 mkdir -p /root/app/microservice/gateway-service #编写Dockerfile cat gateway-service/Dockerfile EOF FROM openjdk:17-jdk-slim EXPOSE 8088根据自己的端口号修改 VOLUME /tmp ADD target/*.jar /app.jar RUN bash -c touch /app.jar ENTRYPOINT [java,-jar,-Xms128m,-Xmx512m,/app.jar] EOF Maven 编译打包 cd /root/app/microservice/gateway-service mvn clean package -DskipTests 3. 构建 Docker 镜像 docker build -t gateway-service:latest .如果没有Maven JDK 17可以选择多阶段构建# 阶段1Maven 编译 FROM maven:3.8.6-openjdk-17 AS builder WORKDIR /build COPY pom.xml . COPY src ./src RUN mvn clean package -DskipTests # 阶段2运行 FROM openjdk:17-jdk-slim COPY --frombuilder /build/target/*.jar /app.jar ENTRYPOINT [java,-jar,-Xms128m,-Xmx512m,/app.jar] EXPOSE 80886.2 前端 nginx:alpine 镜像构建cat rentweb-frontend/Dockerfile EOF # 阶段1Node 构建 FROM node:22-alpine AS builder WORKDIR /app COPY package*.json ./ RUN npm install COPY . . RUN npm run build #阶段2Nginx 运行 FROM nginx:alpine COPY --frombuilder /app/dist /usr/share/nginx/html COPY nginx.conf /etc/nginx/conf.d/default.conf EXPOSE 80 EOF # 构建 Docker 镜像Dockerfile 内部会执行 npm install npm run build cd /root/app/microservice/rentweb-frontend docker build -t rentweb-frontend:latest .6.3 K8s 资源清单编写Deployment/Service/Ingress在 master 上执行# 先创建命名空间 kubectl create namespace rent-cloud #先创建文件 mkdir -p /root/k8s-manifests 创建微服务的资源文件这里我只展示了gateway其他同理 cat /root/k8s-manifests/gateway-service.yaml EOF apiVersion: apps/v1 kind: Deployment metadata: name: gateway-service namespace: rent-cloud spec: replicas: 1 selector: matchLabels: app: gateway-service template: metadata: labels: app: gateway-service spec: containers: - name: gateway-service image: gateway-service:latest ports: - containerPort: 8088根据自己需求修改 --- apiVersion: v1 kind: Service metadata: name: gateway-service namespace: rent-cloud spec: ports: - port: 8088 targetPort: 8088 selector: app: gateway-service EOF 部署到 K8s kubectl apply -f /root/k8s-manifests/gateway-service.yaml 查看部署状态 kubectl get pods -n rent-cloud -o wide kubectl get svc -n rent-cloud七、GitLab Jenkins CI/CD 流水线7.1 GitLab 部署与仓库配置GitLab 作为代码仓库使用 Docker 方式部署# 创建 GitLab 容器 docker run -d \ --name gitlab \ --restart always \ -p 22:22 -p 80:80 -p 443:443 \ -v /srv/gitlab/config:/etc/gitlab \ -v /srv/gitlab/logs:/var/log/gitlab \ -v /srv/gitlab/data:/var/opt/gitlab \ gitlab/gitlab-ce:15.11.0-ce.0在 GitLab 中创建项目仓库推送代码记录项目 ID 和 GitLab API Token后续 Jenkins 配置需要用到。本人 代码仓库分为前端和后端7.2 Jenkins Declarative Pipeline 7 阶段设计# Jenkins 使用 Docker 部署 docker run -d \ --name jenkins \ --restart always \ -p 8080:8080 -p 50000:50000 \ -v /srv/jenkins:/var/jenkins_home \ -v /var/run/docker.sock:/var/run/docker.sock \ jenkins/jenkins:lts所需插件Git Plugingitlab-pluginGitLab 自动触发 Jenkins 构建PipelineDocker PipelineKubernetes CLI PluginSonarQube Scanner如果安装不了maven插件可以直接在jenkins容器内部署maven# 后端先编译需要容器内安装 Maven JDK 17 # 进入 Jenkins 容器安装 Maven docker exec -it jenkins bash apt-get update apt-get install -y maven openjdk-17-jdk mvn --version # 前端如果构建时提示需要nodejs插件可以在jenkins容器内安装node直接构建不需要宿主机装 Node # 进入 Jenkins 容器安装 Node.js docker exec -it jenkins bash apt-get update apt-get install -y nodejs npm node --version npm --version exit7.3 Harbor 镜像仓库集成# 下载 Harbor 离线安装包 wget https://github.com/goharbor/harbor/releases/download/v2.8.0/harbor-offline-installer-v2.8.0.tgz tar xzf harbor-offline-installer-v2.8.0.tgz 修改 harbor.yml hostname: 192.168.1.20 # 部署 Harbor 的机器 IP http: port: 80 安装 ./install.sh注意使用 HTTP 需要在所有节点 Docker 配置中添加 insecure-registriescat /etc/docker/daemon.json EOF { insecure-registries: [192.168.1.20:80] } EOF systemctl restart docker7.4 Jenkins Declarative Pipeline 7 阶段设计整个流水线分为 7 个阶段1. Checkout → 从 GitLab 拉取代码 2. Parse Services → 解析本次要构建的服务 3. Maven Build → Maven 编译打包 4. Docker Build → 构建镜像并推送到 Harbor 5. Deploy to K8s → 更新 K8s 资源清单 6. Update YAML → 提交更新后的 YAML 到 GitLab 7. Cleanup → 清理工作空间在项目后端中创建jenkinsfilepipeline { agent any environment { HARBOR_URL harbor.internal:80 PROJECT rent-cloud GITLAB_URL http://gitlab.internal } stages { stage(Check Webhook Loop) { steps { script { COMMIT_MSG sh(returnStdout: true, script: git log -1 --pretty%B).trim() if (COMMIT_MSG.startsWith(auto:)) { echo 检测到自动提交跳过构建 currentBuild.result SUCCESS return } } } } stage(Checkout) { steps { checkout scm script { COMMIT_MSG sh(returnStdout: true, script: git log -1 --pretty%B).trim() echo Commit Message: ${COMMIT_MSG} } } } stage(Parse Services) { steps { script { def matcher (COMMIT_MSG ~ /build\s(\S)/) if (matcher.find()) { BUILD_SERVICES matcher[0][1].split(,) echo 本次构建服务: ${BUILD_SERVICES} } else { BUILD_SERVICES [all] echo 未指定服务全量构建 } } } } stage(Maven Build) { steps { script { def builds [:] BUILD_SERVICES.each { service - if (service all) { builds[all] { sh mvn clean package -DskipTests } } else { builds[service] { sh mvn clean package -DskipTests -pl ${service} -am } } } parallel builds } } } stage(Docker Build Push) { steps { script { def builds [:] BUILD_SERVICES.each { service - builds[service] { def tag ${HARBOR_URL}/${PROJECT}/${service}:${env.BUILD_NUMBER} sh docker build -t ${tag} ./${service} sh docker push ${tag} } } parallel builds } } } stage(Deploy to K8s) { steps { script { BUILD_SERVICES.each { service - def tag ${HARBOR_URL}/${PROJECT}/${service}:${env.BUILD_NUMBER} sh sed -i s|image:.*${service}:.*|image: ${tag}| k8s/${service}.yaml kubectl apply -f k8s/${service}.yaml } } } } stage(Update YAML) { steps { withCredentials([string(credentialsId: gitlab-token, variable: TOKEN)]) { script { sh git add k8s/*.yaml git commit -m auto: update image tag for build ${env.BUILD_NUMBER} git push http://oauth2:${TOKEN}${GITLAB_URL}/rent-cloud/rent-system.git HEAD:main } } } } } post { success { echo 构建部署完成 } failure { echo 构建失败请检查日志 } } }在项目前端中创建jenkinsfilepipeline { agent any environment { HARBOR_URL harbor.internal:80 PROJECT rent-cloud GITLAB_URL http://gitlab.internal } stages { stage(Check Webhook Loop) { steps { script { COMMIT_MSG sh(returnStdout: true, script: git log -1 --pretty%B).trim() if (COMMIT_MSG.startsWith(auto:)) { echo 检测到自动提交跳过构建 currentBuild.result SUCCESS return } } } } stage(Checkout) { steps { checkout scm } } stage(Docker Build Push) { steps { script { def tag ${HARBOR_URL}/${PROJECT}/rentweb-frontend:${env.BUILD_NUMBER} sh docker build -t ${tag} ./rentweb-frontend docker push ${tag} } } } stage(Deploy to K8s) { steps { script { def tag ${HARBOR_URL}/${PROJECT}/rentweb-frontend:${env.BUILD_NUMBER} sh sed -i s|image:.*rentweb-frontend:.*|image: ${tag}| k8s/rentweb-frontend.yaml kubectl apply -f k8s/rentweb-frontend.yaml } } } stage(Update YAML) { steps { withCredentials([string(credentialsId: gitlab-token, variable: TOKEN)]) { sh git add k8s/rentweb-frontend.yaml git commit -m auto: update frontend image tag build ${env.BUILD_NUMBER} git push http://oauth2:${TOKEN}${GITLAB_URL}/rent-cloud/rent-system.git HEAD:main } } } } post { success { echo 前端构建部署完成 } failure { echo 前端构建失败请检查日志 } } }7.5 增量构建机制核心亮点这是 CI/CD 流水线的核心优化点。痛点6 个微服务 前端每次提交代码都全量构建耗时 10 分钟以上。大部分时候只改了 1 个服务没必要全部重新编译。方案通过解析 git commit message 中的关键字只构建变更的服务。具体实现开发者在提交代码时在 commit message 中写明要构建的服务git commit -m fix: 修复登录bug build userPipeline 中的 Parse Services 阶段用正则/build\s(\S)/提取服务名如果写多个服务用逗号分隔build user,order如果没有build关键字执行全量构建效果增量构建从 10min 降到 3min效率提升 60% 以上。7.6 Webhook 循环防护问题Pipeline 的最后一步Update YAML会向 GitLab 推送代码而 GitLab 收到推送又会触发 Jenkins Webhook导致无限循环触发构建。解决在 Update YAML 阶段提交时commit message 以auto:开头。然后在 Jenkins Pipeline 的第一阶段检查 commit message如果以auto:开头直接跳过整个构建stage(Check Webhook Loop) { steps { script { if (COMMIT_MSG.startsWith(auto:)) { echo 检测到自动提交跳过构建 currentBuild.result SUCCESS return } } }八、监控与可观测性8.1 Prometheus Grafana 监控栈# 部署 Prometheus Operator kubectl create namespace monitoring helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm install prometheus prometheus-community/kube-prometheus-stack -n monitoring # 访问 Grafana默认密码 admin/prom-operator kubectl port-forward -n monitoring svc/prometheus-grafana 3000:808.2 JVM/GC/HTTP 指标采集微服务引入 Micrometer Prometheus 依赖Spring Boot 3.x 自带dependency groupIdio.micrometer/groupId artifactIdmicrometer-registry-prometheus/artifactId /dependency配置后在每个微服务的 application.yml 中暴露/actuator/prometheus端点management: endpoints: web: exposure: include: health,prometheus metrics: tags: application: ${spring.application.name}在 Prometheus 中配置 ServiceMonitor 采集这些端点Grafana 导入 JVM Micrometer 仪表盘ID 4701即可看到 JVM 内存、GC 次数、线程数、HTTP 请求吞吐量、连接池使用率等指标。8.3 Zipkin 链路追踪微服务引入 Micrometer Tracing Bravedependency groupIdio.micrometer/groupId artifactIdmicrometer-tracing-bridge-brave/artifactId /dependency dependency groupIdio.zipkin.reporter2/groupId artifactIdzipkin-reporter-brave/artifactId /dependencyZipkin 服务端使用 Docker 部署docker run -d --name zipkin -p 30094:9411 openzipkin/zipkin:2.24微服务配置指向 Zipkin 地址spring: zipkin: base-url: http://zipkin-service.monitoring:941 sleuth: sampler: probability: 1.0这样每个请求经过多个微服务时可以在 Zipkin 中看到完整的调用链路和每个环节的耗时。
