Kubernetes持久化存储实战引言Kubernetes的持久化存储是运行有状态应用的关键基础设施。与无状态服务不同有状态应用需要持久化存储来保存数据如数据库、消息队列、文件系统等。Kubernetes通过PersistentVolumePV和PersistentVolumeClaimPVC抽象提供了存储资源的声明式管理支持多种存储后端包括本地存储、网络存储、分布式存储和云存储等。本文将全面介绍Kubernetes存储机制、存储类的使用、存储卷的配置以及有状态应用的部署实践。一、Kubernetes存储架构1.1 存储概念概述Kubernetes的存储模型包含几个核心概念PersistentVolume持久卷是集群层面的存储资源由管理员或通过StorageClass动态创建PersistentVolumeClaim持久卷声明是用户对存储资源的请求类似于Pod消耗Node资源的方式StorageClass提供了动态存储供应的能力简化了PV的管理Volume是Pod中使用的临时或持久存储。┌─────────────────────────────────────────────────────────────┐ │ 用户 (Pod) │ │ 创建PVC │ └─────────────────────┬───────────────────────────────────────┘ │ ┌─────────────────────▼───────────────────────────────────────┐ │ PersistentVolumeClaim │ │ (请求: 10Gi, ReadWriteOnce) │ └─────────────────────┬───────────────────────────────────────┘ │ 绑定 ┌─────────────────────▼───────────────────────────────────────┐ │ PersistentVolume │ │ (容量: 20Gi, 存储类: fast) │ └─────────────────────┬───────────────────────────────────────┘ │ 供应 ┌─────────────────────▼───────────────────────────────────────┐ │ StorageClass │ │ (供应者: csi-driver, 参数: typessd) │ └─────────────────────┬───────────────────────────────────────┘ │ 供应 ┌─────────────────────▼───────────────────────────────────────┐ │ 存储后端 │ │ (NFS / Ceph / AWS EBS / GCE PD / Azure Disk) │ └─────────────────────────────────────────────────────────────┘1.2 Volume类型Kubernetes支持多种Volume类型emptyDir用于临时存储Pod删除时会被清除hostPath将Node上的文件或目录挂载到Podnfs用于网络文件系统存储persistentVolumeClaim引用预先配置的PVconfigMap和secret用于配置和敏感信息cloud存储如awsElasticBlockStore、gcePersistentDisk等。二、PersistentVolume详解2.1 PV创建示例apiVersion: v1 kind: PersistentVolume metadata: name: pv-mysql-data labels: type: local app: mysql spec: capacity: storage: 50Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: local-storage local: path: /data/mysql nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - node-12.2 访问模式ReadWriteOnceRWO是最常用的模式存储卷只能被单个节点以读写方式挂载ReadOnlyManyROX允许存储卷被多个节点以只读方式挂载ReadWriteManyRWX允许存储卷被多个节点以读写方式挂载ReadWriteOncePodRWOP允许单个Pod独占访问存储卷。2.3 回收策略Retain策略保留数据手动处理回收Delete策略自动删除存储资源Recycle策略会删除数据并重新变成可用状态已弃用。三、StorageClass动态供应3.1 创建StorageClassapiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: fast-ssd provisioner: kubernetes.io/gce-pd parameters: type: pd-ssd replication-type: regional-pd fstype: ext4 reclaimPolicy: Retain allowVolumeExpansion: true mountOptions: - debug - noatime volumeBindingMode: WaitForFirstConsumer3.2 CSI存储驱动Container Storage InterfaceCSI是Kubernetes存储的标准接口。apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: csi-rook-ceph-block provisioner: rook.io/block parameters: poolName: replicapool clusterNamespace: rook-ceph imagePrefix: rook/ imageName: ceph-block imageFormat: 2 imageFeatures: layering reclaimPolicy: Retain allowVolumeExpansion: true volumeBindingMode: Immediate四、PersistentVolumeClaim4.1 PVC基本配置apiVersion: v1 kind: PersistentVolumeClaim metadata: name: mysql-pvc namespace: production spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi storageClassName: fast-ssd selector: matchLabels: type: database4.2 PVC在Pod中使用apiVersion: apps/v1 kind: Deployment metadata: name: mysql namespace: production spec: selector: matchLabels: app: mysql template: spec: containers: - name: mysql image: mysql:8.0 ports: - containerPort: 3306 name: mysql env: - name: MYSQL_ROOT_PASSWORD valueFrom: secretKeyRef: name: mysql-secret key: password volumeMounts: - name: mysql-data mountPath: /var/lib/mysql - name: mysql-config mountPath: /etc/mysql/conf.d livenessProbe: exec: command: [mysqladmin, ping, -h, localhost] initialDelaySeconds: 30 periodSeconds: 10 readinessProbe: exec: command: [mysql, -h, localhost, -u, root, -p$MYSQL_ROOT_PASSWORD, -e, SELECT 1] initialDelaySeconds: 20 periodSeconds: 5 volumes: - name: mysql-data persistentVolumeClaim: claimName: mysql-pvc - name: mysql-config configMap: name: mysql-config五、有状态应用部署5.1 StatefulSet简介StatefulSet是Kubernetes用于管理有状态应用的资源对象提供了稳定的网络标识、稳定的持久存储和有序的部署和扩展。apiVersion: apps/v1 kind: StatefulSet metadata: name: mongodb namespace: production spec: serviceName: mongodb-headless replicas: 3 selector: matchLabels: app: mongodb template: metadata: labels: app: mongodb spec: terminationGracePeriodSeconds: 30 containers: - name: mongodb image: mongo:6.0 ports: - containerPort: 27017 name: client - containerPort: 27018 name: replica command: - mongod - --replSet - rs0 - --keyFile - /etc/mongodb-secret/keyfile - --bind_ip_all volumeMounts: - name: mongodb-data mountPath: /data/db - name: mongodb-config mountPath: /data/configdb resources: requests: cpu: 500m memory: 1Gi limits: cpu: 2 memory: 4Gi readinessProbe: exec: command: - mongo - --eval - db.adminCommand(ping) initialDelaySeconds: 10 periodSeconds: 10 volumes: - name: mongodb-config emptyDir: {} volumeClaimTemplates: - metadata: name: mongodb-data spec: accessModes: [ ReadWriteOnce ] storageClassName: fast-ssd resources: requests: storage: 50Gi5.2 Headless ServiceapiVersion: v1 kind: Service metadata: name: mongodb-headless namespace: production spec: clusterIP: None selector: app: mongodb ports: - port: 27017 targetPort: 27017 name: client六、数据备份和恢复6.1 VolumeSnapshotapiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata: name: mysql-backup-20240115 namespace: production spec: volumeSnapshotClassName: csi-aws-vsc source: persistentVolumeClaimName: mysql-pvc6.2 从快照恢复apiVersion: v1 kind: PersistentVolumeClaim metadata: name: mysql-restored-pvc namespace: production spec: dataSource: name: mysql-backup-20240115 kind: VolumeSnapshot apiGroup: snapshot.storage.k8s.io accessModes: - ReadWriteOnce resources: requests: storage: 50Gi storageClassName: fast-ssd6.3 定时备份JobapiVersion: batch/v1 kind: CronJob metadata: name: mysql-backup namespace: production spec: schedule: 0 2 * * * successfulJobsHistoryLimit: 7 failedJobsHistoryLimit: 3 jobTemplate: spec: template: spec: serviceAccountName: backup-sa containers: - name: backup image: mysql:8.0 command: - /bin/bash - -c - | mysqldump --all-databases -u root -p$MYSQL_ROOT_PASSWORD | gzip /backups/backup-$(date %Y%m%d%H%M%S).sql.gz find /backups -type f -mtime 7 -delete env: - name: MYSQL_ROOT_PASSWORD valueFrom: secretKeyRef: name: mysql-secret key: password volumeMounts: - name: backup-storage mountPath: /backups - name: mysql-secret readOnly: true mountPath: /etc/mysql-secret volumes: - name: backup-storage persistentVolumeClaim: claimName: backup-pvc - name: mysql-secret secret: secretName: mysql-secret restartPolicy: OnFailure七、存储性能优化7.1 存储QoSapiVersion: v1 kind: PersistentVolumeClaim metadata: name: high-performance-pvc namespace: production spec: accessModes: - ReadWriteOnce resources: requests: storage: 100Gi storageClassName: premium-storage selector: matchExpressions: - key: performance operator: In values: - high7.2 Volume扩展apiVersion: v1 kind: PersistentVolumeClaim metadata: name: mysql-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 100Gi # 从50Gi扩展到100Gi storageClassName: fast-ssd八、存储监控8.1 VolumeMetricsapiVersion: v1 kind: PersistentVolumeClaim metadata: name: monitored-pvc annotations: volume.beta.kubernetes.io/storage-provisioner: pd.csi.storage.gke.io8.2 Prometheus存储指标# kube-state-metrics会暴露以下指标 # kube_persistentvolume_capacity_bytes # kube_persistentvolume_status_phase # kube_persistentvolumeclaim_info # kube_persistentvolumeclaim_status_phase # kube_persistentvolumeclaim_resource_requests_storage_bytes九、最佳实践9.1 存储规划建议对于生产环境存储规划应考虑使用本地SSD或高性能网络存储作为数据库存储为不同类型的数据配置不同的存储类启用存储卷的自动扩展功能实施定期备份策略监控存储使用量并设置告警。9.2 安全配置apiVersion: v1 kind: PersistentVolumeClaim metadata: name: secure-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi storageClassName: secure-storage --- # StorageClass启用加密 apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: secure-storage provisioner: pd.csi.storage.gke.io parameters: type: pd-standard encrypted: true keyRotationEnabled: true9.3 故障排查清单当存储出现问题时检查PVC状态是否为Pending或LostPV是否存在且状态正确StorageClass配置是否正确存储驱动的Pod是否正常运行Node是否有足够的存储空间PVC的accessMode是否与存储类型兼容。总结Kubernetes的持久化存储机制为有状态应用提供了强大的支持。通过合理使用PV、PVC和StorageClass可以实现存储资源的灵活管理和动态供应。StatefulSet和有状态应用的结合使用使得在Kubernetes上运行数据库和其他有状态服务成为可能。在实际生产环境中需要关注存储的性能、可靠性、安全性和可扩展性建立完善的监控和备份机制确保数据的安全和可用。
Kubernetes持久化存储实战
Kubernetes持久化存储实战引言Kubernetes的持久化存储是运行有状态应用的关键基础设施。与无状态服务不同有状态应用需要持久化存储来保存数据如数据库、消息队列、文件系统等。Kubernetes通过PersistentVolumePV和PersistentVolumeClaimPVC抽象提供了存储资源的声明式管理支持多种存储后端包括本地存储、网络存储、分布式存储和云存储等。本文将全面介绍Kubernetes存储机制、存储类的使用、存储卷的配置以及有状态应用的部署实践。一、Kubernetes存储架构1.1 存储概念概述Kubernetes的存储模型包含几个核心概念PersistentVolume持久卷是集群层面的存储资源由管理员或通过StorageClass动态创建PersistentVolumeClaim持久卷声明是用户对存储资源的请求类似于Pod消耗Node资源的方式StorageClass提供了动态存储供应的能力简化了PV的管理Volume是Pod中使用的临时或持久存储。┌─────────────────────────────────────────────────────────────┐ │ 用户 (Pod) │ │ 创建PVC │ └─────────────────────┬───────────────────────────────────────┘ │ ┌─────────────────────▼───────────────────────────────────────┐ │ PersistentVolumeClaim │ │ (请求: 10Gi, ReadWriteOnce) │ └─────────────────────┬───────────────────────────────────────┘ │ 绑定 ┌─────────────────────▼───────────────────────────────────────┐ │ PersistentVolume │ │ (容量: 20Gi, 存储类: fast) │ └─────────────────────┬───────────────────────────────────────┘ │ 供应 ┌─────────────────────▼───────────────────────────────────────┐ │ StorageClass │ │ (供应者: csi-driver, 参数: typessd) │ └─────────────────────┬───────────────────────────────────────┘ │ 供应 ┌─────────────────────▼───────────────────────────────────────┐ │ 存储后端 │ │ (NFS / Ceph / AWS EBS / GCE PD / Azure Disk) │ └─────────────────────────────────────────────────────────────┘1.2 Volume类型Kubernetes支持多种Volume类型emptyDir用于临时存储Pod删除时会被清除hostPath将Node上的文件或目录挂载到Podnfs用于网络文件系统存储persistentVolumeClaim引用预先配置的PVconfigMap和secret用于配置和敏感信息cloud存储如awsElasticBlockStore、gcePersistentDisk等。二、PersistentVolume详解2.1 PV创建示例apiVersion: v1 kind: PersistentVolume metadata: name: pv-mysql-data labels: type: local app: mysql spec: capacity: storage: 50Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: local-storage local: path: /data/mysql nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - node-12.2 访问模式ReadWriteOnceRWO是最常用的模式存储卷只能被单个节点以读写方式挂载ReadOnlyManyROX允许存储卷被多个节点以只读方式挂载ReadWriteManyRWX允许存储卷被多个节点以读写方式挂载ReadWriteOncePodRWOP允许单个Pod独占访问存储卷。2.3 回收策略Retain策略保留数据手动处理回收Delete策略自动删除存储资源Recycle策略会删除数据并重新变成可用状态已弃用。三、StorageClass动态供应3.1 创建StorageClassapiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: fast-ssd provisioner: kubernetes.io/gce-pd parameters: type: pd-ssd replication-type: regional-pd fstype: ext4 reclaimPolicy: Retain allowVolumeExpansion: true mountOptions: - debug - noatime volumeBindingMode: WaitForFirstConsumer3.2 CSI存储驱动Container Storage InterfaceCSI是Kubernetes存储的标准接口。apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: csi-rook-ceph-block provisioner: rook.io/block parameters: poolName: replicapool clusterNamespace: rook-ceph imagePrefix: rook/ imageName: ceph-block imageFormat: 2 imageFeatures: layering reclaimPolicy: Retain allowVolumeExpansion: true volumeBindingMode: Immediate四、PersistentVolumeClaim4.1 PVC基本配置apiVersion: v1 kind: PersistentVolumeClaim metadata: name: mysql-pvc namespace: production spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi storageClassName: fast-ssd selector: matchLabels: type: database4.2 PVC在Pod中使用apiVersion: apps/v1 kind: Deployment metadata: name: mysql namespace: production spec: selector: matchLabels: app: mysql template: spec: containers: - name: mysql image: mysql:8.0 ports: - containerPort: 3306 name: mysql env: - name: MYSQL_ROOT_PASSWORD valueFrom: secretKeyRef: name: mysql-secret key: password volumeMounts: - name: mysql-data mountPath: /var/lib/mysql - name: mysql-config mountPath: /etc/mysql/conf.d livenessProbe: exec: command: [mysqladmin, ping, -h, localhost] initialDelaySeconds: 30 periodSeconds: 10 readinessProbe: exec: command: [mysql, -h, localhost, -u, root, -p$MYSQL_ROOT_PASSWORD, -e, SELECT 1] initialDelaySeconds: 20 periodSeconds: 5 volumes: - name: mysql-data persistentVolumeClaim: claimName: mysql-pvc - name: mysql-config configMap: name: mysql-config五、有状态应用部署5.1 StatefulSet简介StatefulSet是Kubernetes用于管理有状态应用的资源对象提供了稳定的网络标识、稳定的持久存储和有序的部署和扩展。apiVersion: apps/v1 kind: StatefulSet metadata: name: mongodb namespace: production spec: serviceName: mongodb-headless replicas: 3 selector: matchLabels: app: mongodb template: metadata: labels: app: mongodb spec: terminationGracePeriodSeconds: 30 containers: - name: mongodb image: mongo:6.0 ports: - containerPort: 27017 name: client - containerPort: 27018 name: replica command: - mongod - --replSet - rs0 - --keyFile - /etc/mongodb-secret/keyfile - --bind_ip_all volumeMounts: - name: mongodb-data mountPath: /data/db - name: mongodb-config mountPath: /data/configdb resources: requests: cpu: 500m memory: 1Gi limits: cpu: 2 memory: 4Gi readinessProbe: exec: command: - mongo - --eval - db.adminCommand(ping) initialDelaySeconds: 10 periodSeconds: 10 volumes: - name: mongodb-config emptyDir: {} volumeClaimTemplates: - metadata: name: mongodb-data spec: accessModes: [ ReadWriteOnce ] storageClassName: fast-ssd resources: requests: storage: 50Gi5.2 Headless ServiceapiVersion: v1 kind: Service metadata: name: mongodb-headless namespace: production spec: clusterIP: None selector: app: mongodb ports: - port: 27017 targetPort: 27017 name: client六、数据备份和恢复6.1 VolumeSnapshotapiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata: name: mysql-backup-20240115 namespace: production spec: volumeSnapshotClassName: csi-aws-vsc source: persistentVolumeClaimName: mysql-pvc6.2 从快照恢复apiVersion: v1 kind: PersistentVolumeClaim metadata: name: mysql-restored-pvc namespace: production spec: dataSource: name: mysql-backup-20240115 kind: VolumeSnapshot apiGroup: snapshot.storage.k8s.io accessModes: - ReadWriteOnce resources: requests: storage: 50Gi storageClassName: fast-ssd6.3 定时备份JobapiVersion: batch/v1 kind: CronJob metadata: name: mysql-backup namespace: production spec: schedule: 0 2 * * * successfulJobsHistoryLimit: 7 failedJobsHistoryLimit: 3 jobTemplate: spec: template: spec: serviceAccountName: backup-sa containers: - name: backup image: mysql:8.0 command: - /bin/bash - -c - | mysqldump --all-databases -u root -p$MYSQL_ROOT_PASSWORD | gzip /backups/backup-$(date %Y%m%d%H%M%S).sql.gz find /backups -type f -mtime 7 -delete env: - name: MYSQL_ROOT_PASSWORD valueFrom: secretKeyRef: name: mysql-secret key: password volumeMounts: - name: backup-storage mountPath: /backups - name: mysql-secret readOnly: true mountPath: /etc/mysql-secret volumes: - name: backup-storage persistentVolumeClaim: claimName: backup-pvc - name: mysql-secret secret: secretName: mysql-secret restartPolicy: OnFailure七、存储性能优化7.1 存储QoSapiVersion: v1 kind: PersistentVolumeClaim metadata: name: high-performance-pvc namespace: production spec: accessModes: - ReadWriteOnce resources: requests: storage: 100Gi storageClassName: premium-storage selector: matchExpressions: - key: performance operator: In values: - high7.2 Volume扩展apiVersion: v1 kind: PersistentVolumeClaim metadata: name: mysql-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 100Gi # 从50Gi扩展到100Gi storageClassName: fast-ssd八、存储监控8.1 VolumeMetricsapiVersion: v1 kind: PersistentVolumeClaim metadata: name: monitored-pvc annotations: volume.beta.kubernetes.io/storage-provisioner: pd.csi.storage.gke.io8.2 Prometheus存储指标# kube-state-metrics会暴露以下指标 # kube_persistentvolume_capacity_bytes # kube_persistentvolume_status_phase # kube_persistentvolumeclaim_info # kube_persistentvolumeclaim_status_phase # kube_persistentvolumeclaim_resource_requests_storage_bytes九、最佳实践9.1 存储规划建议对于生产环境存储规划应考虑使用本地SSD或高性能网络存储作为数据库存储为不同类型的数据配置不同的存储类启用存储卷的自动扩展功能实施定期备份策略监控存储使用量并设置告警。9.2 安全配置apiVersion: v1 kind: PersistentVolumeClaim metadata: name: secure-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi storageClassName: secure-storage --- # StorageClass启用加密 apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: secure-storage provisioner: pd.csi.storage.gke.io parameters: type: pd-standard encrypted: true keyRotationEnabled: true9.3 故障排查清单当存储出现问题时检查PVC状态是否为Pending或LostPV是否存在且状态正确StorageClass配置是否正确存储驱动的Pod是否正常运行Node是否有足够的存储空间PVC的accessMode是否与存储类型兼容。总结Kubernetes的持久化存储机制为有状态应用提供了强大的支持。通过合理使用PV、PVC和StorageClass可以实现存储资源的灵活管理和动态供应。StatefulSet和有状态应用的结合使用使得在Kubernetes上运行数据库和其他有状态服务成为可能。在实际生产环境中需要关注存储的性能、可靠性、安全性和可扩展性建立完善的监控和备份机制确保数据的安全和可用。
