昇腾910B双节点K8s集群实战从零到精通的避坑手册在异构计算领域昇腾910B凭借其强大的AI算力正成为越来越多企业的选择。当这样的硬件遇上Kubernetes——这个云原生时代的操作系统会擦出怎样的火花本文将带你用kubeadm在两台昇腾910B服务器上快速搭建生产级Kubernetes集群过程中遇到的每一个坑我们都提前为你标好了警示牌。1. 环境准备昇腾平台的特别注意事项昇腾910B的ARM架构与通用x86环境存在显著差异。在开始前请确保两台服务器满足以下基础条件操作系统推荐Ubuntu 20.04 LTS或CentOS 8需ARM版本昇腾驱动已安装最新版CANN工具包≥5.0.RC2网络配置双节点需位于同一子网建议配置静态IP关键检查点运行npu-smi info应能正常显示昇腾芯片状态若报错需先解决驱动问题。1.1 系统参数调优ARM架构下需要特别关注的内核参数调整# 编辑sysctl配置 cat EOF | sudo tee /etc/sysctl.d/k8s-arm.conf vm.swappiness 0 vm.max_map_count 262144 kernel.panic_on_oops 1 kernel.panic 10 EOF sudo sysctl --system昇腾环境特有配置表格参数推荐值作用说明net.core.rmem_max4194304提升网络吞吐量fs.inotify.max_user_watches524288避免监控节点文件系统事件丢失kernel.sched_latency_ns6000000ARM架构任务调度优化2. 容器运行时为昇腾优化的Docker配置在ARM架构上容器运行时的选择直接影响后续Kubernetes组件的稳定性。以下是经实测验证的配置方案2.1 Docker安装与验证# 安装指定版本Docker curl -fsSL https://get.docker.com | sudo sh sudo usermod -aG docker $USER修改/etc/docker/daemon.json注意ARM架构的特殊参数{ exec-opts: [native.cgroupdriversystemd], log-driver: json-file, log-opts: { max-size: 100m }, storage-driver: overlay2, default-runtime: nvidia, runtimes: { nvidia: { path: /usr/bin/nvidia-container-runtime, runtimeArgs: [] } } }重启服务后验证sudo systemctl restart docker docker run --rm arm64v8/ubuntu uname -m # 应输出aarch642.2 昇腾设备映射配置创建/etc/docker/daemon.json的补充配置sudo mkdir -p /etc/docker cat EOF | sudo tee /etc/docker/daemon.json.d/ascend.conf { device-requests: [ { driver: ascend, count: -1, capabilities: [[gpu]] } ] } EOF3. kubeadm初始化ARM架构的特别参数3.1 预拉取ARM64镜像避免init时因镜像拉取失败sudo kubeadm config images pull --image-repositoryregistry.aliyuncs.com/google_containers3.2 定制init配置文件创建kubeadm-init.yamlapiVersion: kubeadm.k8s.io/v1beta3 kind: InitConfiguration nodeRegistration: kubeletExtraArgs: cgroup-driver: systemd feature-gates: ExecProbeTimeouttrue --- apiVersion: kubeadm.k8s.io/v1beta3 kind: ClusterConfiguration imageRepository: registry.aliyuncs.com/google_containers kubernetesVersion: v1.24.0 controlPlaneEndpoint: MASTER_IP:6443 networking: podSubnet: 192.168.0.0/16 serviceSubnet: 10.96.0.0/12 apiServer: extraArgs: runtime-config: storage.k8s.io/v1true controllerManager: extraArgs: node-cidr-mask-size: 24 scheduler: extraArgs: bind-address: 0.0.0.0执行初始化关键参数说明sudo kubeadm init --configkubeadm-init.yaml \ --ignore-preflight-errorsSwap \ --upload-certs \ --v5重要提示若遇到[ERROR NumCPU]报错需添加--ignore-preflight-errorsNumCPU参数4. 节点加入与网络插件部署4.1 Worker节点加入在Master节点获取join命令kubeadm token create --print-join-command在Worker节点执行时添加ARM架构参数sudo kubeadm join MASTER_IP:6443 --token TOKEN \ --discovery-token-ca-cert-hash sha256:HASH \ --node-labelsacceleratorascend910b \ --ignore-preflight-errorsSwap,NumCPU4.2 网络插件选择昇腾环境下推荐使用Calico网络插件kubectl apply -f https://docs.projectcalico.org/manifests/calico-arm64.yaml验证网络状态watch kubectl get pods -n kube-system # 等待所有Pod变为Running状态5. 昇腾设备插件集成5.1 部署Ascend设备插件创建ascend-device-plugin.yamlapiVersion: apps/v1 kind: DaemonSet metadata: name: ascend-device-plugin-daemonset namespace: kube-system spec: selector: matchLabels: name: ascend-device-plugin-ds template: metadata: labels: name: ascend-device-plugin-ds spec: hostNetwork: true containers: - image: ascend-k8sdeviceplugin:version name: ascend-device-plugin-ctr securityContext: privileged: true volumeMounts: - name: device-plugin mountPath: /var/lib/kubelet/device-plugins - name: ascend-dirver mountPath: /usr/local/Ascend/driver volumes: - name: device-plugin hostPath: path: /var/lib/kubelet/device-plugins - name: ascend-dirver hostPath: path: /usr/local/Ascend/driver部署后验证kubectl describe node | grep ascend # 应显示昇腾资源容量6. 实战验证运行首个昇腾加速Pod创建测试文件ascend-test.yamlapiVersion: v1 kind: Pod metadata: name: ascend-test-pod spec: containers: - name: test-container image: swr.cn-north-4.myhuaweicloud.com/ascend-share/ascend-toolkit:ubuntu18.04-aarch64-5.0.RC2 command: [/bin/bash, -c, while true; do npu-smi info; sleep 5; done] resources: limits: ascend.ai: 1 nodeSelector: accelerator: ascend910b部署并监控kubectl apply -f ascend-test.yaml kubectl logs -f ascend-test-pod # 应周期性输出昇腾芯片状态信息遇到镜像拉取失败时可配置国内镜像仓库sudo mkdir -p /etc/containerd cat EOF | sudo tee /etc/containerd/config.toml [plugins.io.containerd.grpc.v1.cri.registry.mirrors] [plugins.io.containerd.grpc.v1.cri.registry.mirrors.docker.io] endpoint [https://registry-1.docker.io] [plugins.io.containerd.grpc.v1.cri.registry.mirrors.swr.cn-north-4.myhuaweicloud.com] endpoint [https://swr.cn-north-4.myhuaweicloud.com] EOF
在昇腾910B上，用kubeadm快速拉起一个双节点K8s集群（保姆级避坑指南）
昇腾910B双节点K8s集群实战从零到精通的避坑手册在异构计算领域昇腾910B凭借其强大的AI算力正成为越来越多企业的选择。当这样的硬件遇上Kubernetes——这个云原生时代的操作系统会擦出怎样的火花本文将带你用kubeadm在两台昇腾910B服务器上快速搭建生产级Kubernetes集群过程中遇到的每一个坑我们都提前为你标好了警示牌。1. 环境准备昇腾平台的特别注意事项昇腾910B的ARM架构与通用x86环境存在显著差异。在开始前请确保两台服务器满足以下基础条件操作系统推荐Ubuntu 20.04 LTS或CentOS 8需ARM版本昇腾驱动已安装最新版CANN工具包≥5.0.RC2网络配置双节点需位于同一子网建议配置静态IP关键检查点运行npu-smi info应能正常显示昇腾芯片状态若报错需先解决驱动问题。1.1 系统参数调优ARM架构下需要特别关注的内核参数调整# 编辑sysctl配置 cat EOF | sudo tee /etc/sysctl.d/k8s-arm.conf vm.swappiness 0 vm.max_map_count 262144 kernel.panic_on_oops 1 kernel.panic 10 EOF sudo sysctl --system昇腾环境特有配置表格参数推荐值作用说明net.core.rmem_max4194304提升网络吞吐量fs.inotify.max_user_watches524288避免监控节点文件系统事件丢失kernel.sched_latency_ns6000000ARM架构任务调度优化2. 容器运行时为昇腾优化的Docker配置在ARM架构上容器运行时的选择直接影响后续Kubernetes组件的稳定性。以下是经实测验证的配置方案2.1 Docker安装与验证# 安装指定版本Docker curl -fsSL https://get.docker.com | sudo sh sudo usermod -aG docker $USER修改/etc/docker/daemon.json注意ARM架构的特殊参数{ exec-opts: [native.cgroupdriversystemd], log-driver: json-file, log-opts: { max-size: 100m }, storage-driver: overlay2, default-runtime: nvidia, runtimes: { nvidia: { path: /usr/bin/nvidia-container-runtime, runtimeArgs: [] } } }重启服务后验证sudo systemctl restart docker docker run --rm arm64v8/ubuntu uname -m # 应输出aarch642.2 昇腾设备映射配置创建/etc/docker/daemon.json的补充配置sudo mkdir -p /etc/docker cat EOF | sudo tee /etc/docker/daemon.json.d/ascend.conf { device-requests: [ { driver: ascend, count: -1, capabilities: [[gpu]] } ] } EOF3. kubeadm初始化ARM架构的特别参数3.1 预拉取ARM64镜像避免init时因镜像拉取失败sudo kubeadm config images pull --image-repositoryregistry.aliyuncs.com/google_containers3.2 定制init配置文件创建kubeadm-init.yamlapiVersion: kubeadm.k8s.io/v1beta3 kind: InitConfiguration nodeRegistration: kubeletExtraArgs: cgroup-driver: systemd feature-gates: ExecProbeTimeouttrue --- apiVersion: kubeadm.k8s.io/v1beta3 kind: ClusterConfiguration imageRepository: registry.aliyuncs.com/google_containers kubernetesVersion: v1.24.0 controlPlaneEndpoint: MASTER_IP:6443 networking: podSubnet: 192.168.0.0/16 serviceSubnet: 10.96.0.0/12 apiServer: extraArgs: runtime-config: storage.k8s.io/v1true controllerManager: extraArgs: node-cidr-mask-size: 24 scheduler: extraArgs: bind-address: 0.0.0.0执行初始化关键参数说明sudo kubeadm init --configkubeadm-init.yaml \ --ignore-preflight-errorsSwap \ --upload-certs \ --v5重要提示若遇到[ERROR NumCPU]报错需添加--ignore-preflight-errorsNumCPU参数4. 节点加入与网络插件部署4.1 Worker节点加入在Master节点获取join命令kubeadm token create --print-join-command在Worker节点执行时添加ARM架构参数sudo kubeadm join MASTER_IP:6443 --token TOKEN \ --discovery-token-ca-cert-hash sha256:HASH \ --node-labelsacceleratorascend910b \ --ignore-preflight-errorsSwap,NumCPU4.2 网络插件选择昇腾环境下推荐使用Calico网络插件kubectl apply -f https://docs.projectcalico.org/manifests/calico-arm64.yaml验证网络状态watch kubectl get pods -n kube-system # 等待所有Pod变为Running状态5. 昇腾设备插件集成5.1 部署Ascend设备插件创建ascend-device-plugin.yamlapiVersion: apps/v1 kind: DaemonSet metadata: name: ascend-device-plugin-daemonset namespace: kube-system spec: selector: matchLabels: name: ascend-device-plugin-ds template: metadata: labels: name: ascend-device-plugin-ds spec: hostNetwork: true containers: - image: ascend-k8sdeviceplugin:version name: ascend-device-plugin-ctr securityContext: privileged: true volumeMounts: - name: device-plugin mountPath: /var/lib/kubelet/device-plugins - name: ascend-dirver mountPath: /usr/local/Ascend/driver volumes: - name: device-plugin hostPath: path: /var/lib/kubelet/device-plugins - name: ascend-dirver hostPath: path: /usr/local/Ascend/driver部署后验证kubectl describe node | grep ascend # 应显示昇腾资源容量6. 实战验证运行首个昇腾加速Pod创建测试文件ascend-test.yamlapiVersion: v1 kind: Pod metadata: name: ascend-test-pod spec: containers: - name: test-container image: swr.cn-north-4.myhuaweicloud.com/ascend-share/ascend-toolkit:ubuntu18.04-aarch64-5.0.RC2 command: [/bin/bash, -c, while true; do npu-smi info; sleep 5; done] resources: limits: ascend.ai: 1 nodeSelector: accelerator: ascend910b部署并监控kubectl apply -f ascend-test.yaml kubectl logs -f ascend-test-pod # 应周期性输出昇腾芯片状态信息遇到镜像拉取失败时可配置国内镜像仓库sudo mkdir -p /etc/containerd cat EOF | sudo tee /etc/containerd/config.toml [plugins.io.containerd.grpc.v1.cri.registry.mirrors] [plugins.io.containerd.grpc.v1.cri.registry.mirrors.docker.io] endpoint [https://registry-1.docker.io] [plugins.io.containerd.grpc.v1.cri.registry.mirrors.swr.cn-north-4.myhuaweicloud.com] endpoint [https://swr.cn-north-4.myhuaweicloud.com] EOF
