大家好我是JACK本篇是服务器测试百日学习计划Day5。上一篇我们搞清楚了块设备和文件系统今天深入存储系统——搞清楚NVMe、SATA、SAS、RAID各自是什么以及数据怎么从盘走到CPU。一、先把概念分清楚很多人看到lsblk输出一堆sdX和nvmeXnX不知道哪个是什么盘根本原因是没分清这几个层次NVMe协议名走 PCIe 总线低延迟、高带宽、多队列是企业服务器高性能存储主流。SATA走 AHCI/SATA 协议结构简单速度比 NVMe 慢。lsblk看到sdX可能是 SATA 盘也可能是 RAID 逻辑盘不能直接判断。SAS企业级磁盘接口通常配合 HBA 卡和 RAID 控制器使用。RAID不是盘的类型是控制层。把多块物理盘组合成逻辑盘提供冗余和缓存能力常见模式 RAID0/1/5/10。 记住这个关系NVMe 是协议SATA/SAS 是接口体系RAID 是控制层PCIe 是总线M.2/U.2 是形态。二、实机存储设备识别本机配置两块 U.2 NVMe SSD、一张 Broadcom MegaRAID RAID 卡MR9560-16i、两块 Intel M.2 SATA SSD。第一步lsblk 看所有块设备[rootlocalhost ~]# lsblk -dNAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda8:00447.1G0disk sdb8:160447.1G0disk sdc8:320446.6G0disk sdd8:480446.6G0disk sde8:640446.6G0disk sdf8:800446.6G0disk sdg8:960446.6G0disk sdh8:1120446.6G0disk sdi8:1280446.6G0disk sdj8:1440446.6G0disk nvme1n1259:002.9T0disk nvme0n1259:102.9T0disknvme0n1和nvme1n1是 NVMe 盘sda到sdj共10个sdX设备但光看这里还不知道是什么盘继续查。加上分区挂载信息[rootlocalhost ~]# lsblkNAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS nvme1n1259:002.9T0disk ├─nvme1n1p1259:50600M0part /boot/efi ├─nvme1n1p2259:601G0part /boot └─nvme1n1p3259:702.9T0part ├─openeuler-root253:0070G0lvm / ├─openeuler-swap253:104G0lvm[SWAP]└─openeuler-home253:202.8T0lvm /home nvme0n1259:102.9T0disk └─nvme0n1p3259:402.9T0part ├─klas-swap253:304G0lvm ├─klas-backup253:4050G0lvm └─klas-root253:502.9T0lvmnvme1n1 装的是 openEulernvme0n1 装的是 Klas麒麟两块系统盘各跑一个 OS。第二步nvme list 确认 NVMe 盘型号[rootlocalhost ~]# nvme listNode SN Model Namespace Usage Format FW Rev ------------ ---------------- ------------------ ---------- ------------------- ------------ ------- /dev/nvme0n1 D77446D401J852 DERAP44YGM03T2US13.20TB /3.20TB512B 0B D7Y05M1F /dev/nvme1n1 D77446D4017E52 DERAP44YGM03T2US13.20TB /3.20TB512B 0B D7Y05M1F两块 NVMe 型号一致DERAP44YGM03T2US单块 3.2TBU.2 形态通过 PCIe 直连 CPU。第三步lsscsi 看清所有 sdX 是什么盘[rootlocalhost ~]# lsscsi[0:1:124:0]enclosu BROADCOM VirtualSES 03 -[0:3:104:0]disk BROADCOM MR9560-16i5.29/dev/sdc[0:3:105:0]disk BROADCOM MR9560-16i5.29/dev/sdd[0:3:106:0]disk BROADCOM MR9560-16i5.29/dev/sde[0:3:107:0]disk BROADCOM MR9560-16i5.29/dev/sdf[0:3:108:0]disk BROADCOM MR9560-16i5.29/dev/sdi[0:3:109:0]disk BROADCOM MR9560-16i5.29/dev/sdj[0:3:110:0]disk BROADCOM MR9560-16i5.29/dev/sdh[0:3:111:0]disk BROADCOM MR9560-16i5.29/dev/sdg[1:0:0:0]disk ATA INTEL SSDSCKKB48 0100 /dev/sda[2:0:0:0]disk ATA INTEL SSDSCKKB48 0100 /dev/sdb[N:0:0:1]disk DERAP44YGM03T2US__1 /dev/nvme0n1[N:1:0:1]disk DERAP44YGM03T2US__1 /dev/nvme1n1现在全清楚了设备来源类型sdc ~ sdj8块BROADCOM MR9560-16iRAID 控制器逻辑盘sda / sdb2块ATA INTEL SSDSCKKB48M.2 形态 SATA SSDnvme0n1 / nvme1n1DERAP44YGM03T2USU.2 NVMe 直连 PCIesda/sdb 显示 ATA 的原因这是 M.2 形态、SATA 协议的 Intel SSD。M.2 是形态SATA 是协议不是 NVMe。这两种协议共用 M.2 插槽容易混淆。第四步lspci 确认存储控制器[rootlocalhost ~]# lspci | egrep -i SATA|SAS|NVME|SCSI|raid30:04.0 Serial Attached SCSI controller: Huawei Technologies Co., Ltd. HiSilicon SAS3.0HBA(rev30)32:04.0 Serial Attached SCSI controller: Huawei Technologies Co., Ltd. HiSilicon SAS3.0HBA(rev30)38:05.0 SATA controller: Huawei Technologies Co., Ltd. HiSilicon AHCI HBA(rev30)70:04.0 Serial Attached SCSI controller: Huawei Technologies Co., Ltd. HiSilicon SAS3.0HBA(rev30)72:04.0 Serial Attached SCSI controller: Huawei Technologies Co., Ltd. HiSilicon SAS3.0HBA(rev30)83:00.0 RAID bus controller: Broadcom / LSI MegaRAID 12GSAS/PCIe Secure SAS39xx[rootlocalhost ~]# lspci | egrep -i Non81:00.0 Non-Volatile memory controller: DERA Storage Device151582:00.0 Non-Volatile memory controller: DERA Storage Device1515PCIe 设备作用81:00.0 / 82:00.0NVMe 控制器对应 nvme0n1 / nvme1n138:05.0 HiSilicon AHCI HBASATA 控制器挂 sda / sdb83:00.0 Broadcom MegaRAIDRAID 控制器挂 sdc ~ sdj30/32/70/72 HiSilicon SAS HBA华为板载 SAS 通道三、本机存储路径图CPU └─ PCIe Root Port ├─ NVMe Controller (81:00.0) → nvme0n1 (3.2TB系统盘 Klas) ├─ NVMe Controller (82:00.0) → nvme1n1 (3.2TB系统盘 openEuler) ├─ HiSilicon AHCI HBA (38:05.0) → sda / sdb (Intel M.2 SATA SSD × 2) └─ Broadcom MegaRAID (83:00.0) → sdc ~ sdj (RAID0 逻辑盘 × 8)这张图很重要。以后遇到存储慢不能只说磁盘慢要能往下追是 NVMe 本身慢是 RAID 层引入了额外延迟是 PCIe 链路有竞争是跨 NUMA 访问四、storcli64 管理 RAID 控制器storcli64是管理 Broadcom/LSI RAID 卡的专用工具语法是按/c0控制器→/vall逻辑盘→/eall/sall物理盘的层次来查。查控制器总览[rootlocalhost ~]# storcli64 showCLI Version007.2707.0000.0000 Dec18,2023StatusSuccess System Overview:------------------------------------------------------------------------- Ctl Model Ports PDs DGs DNOpt VDs VNOpt Hlth -------------------------------------------------------------------------0MegaRAID9560-16i8GB1688080Opt -------------------------------------------------------------------------关键字段PDs8块物理盘VDs8块逻辑盘DNOpt/VNOpt0表示无异常HlthOpt表示健康。查所有逻辑盘[rootlocalhost ~]# storcli64 /c0/vall showController0StatusSuccess Virtual Drives:----------------------------------------------------------- DG/VD TYPE State Access Consist Cache sCC Size -----------------------------------------------------------6/232 RAID0 Optl RW Yes RWTD ON446.625GB5/233 RAID0 Optl RW Yes RWTD ON446.625GB4/234 RAID0 Optl RW Yes RWTD ON446.625GB3/235 RAID0 Optl RW Yes RWTD ON446.625GB2/236 RAID0 Optl RW Yes RWTD ON446.625GB7/237 RAID0 Optl RW Yes RWTD ON446.625GB1/238 RAID0 Optl RW Yes RWTD ON446.625GB0/239 RAID0 Optl RW Yes RWTD ON446.625GB -----------------------------------------------------------8块物理盘各自做了 RAID0单盘直通对应 sdc ~ sdj每块 446.6GBState 全部 Optl正常。storcli64 常用命令速查# 看控制器列表storcli64 show# 看 c0 控制器总览storcli64 /c0 show# 看所有逻辑盘storcli64 /c0/vall show# 看所有物理盘storcli64 /c0/eall/sall show# 看完整详情storcli64 /c0 show all五、存储异常排查链盘不见了NVMe 不见了lsblk → nvme list → lspci|grep-invme →dmesg|grep-invme → lsmod|grepnvmeRAID/SAS 盘不见了lsblk → lsscsi → storcli64 show →dmesg→ lspci|grep-iraid盘还在但是慢# 先看 IO 指标iostat-x1# 重点关注# %util磁盘利用率是否打满# awaitIO 平均等待时间是否异常高# 对比 NVMe 和 RAID 逻辑盘的数据定位是哪层慢六、Python 存储检测脚本# storage_check.pyimportsubprocessprint( lsblk )print(subprocess.run([lsblk,-d],capture_outputTrue,textTrue).stdout)print( nvme list )print(subprocess.run([nvme,list],capture_outputTrue,textTrue).stdout)print( lsscsi )print(subprocess.run([lsscsi],capture_outputTrue,textTrue).stdout)python3 storage_check.py拿到一台新服务器先跑这个脚本三条命令的输出就能快速建立对存储架构的基本认知。总结层次概念本机实例形态M.2 / U.2NVMe 是 U.2SATA 是 M.2协议NVMe / SATA / SASNVMe 走 PCIeSATA 走 AHCI控制层RAIDBroadcom MegaRAID8块盘各做 RAID0排查工具lsblk / lsscsi / nvme list / storcli64/hiraidadm(华为raid卡专用)四步确认设备类型和健康状态下一篇 Day6 深入 NVMe 架构聊 NVMe 的内部结构、队列机制和健康状态查看敬请期待欢迎关注JACK的服务器笔记我们下篇见
服务器测试百日学习计划——Day5：存储系统全景，搞清楚NVMe/SATA/SAS/RAID的关系
大家好我是JACK本篇是服务器测试百日学习计划Day5。上一篇我们搞清楚了块设备和文件系统今天深入存储系统——搞清楚NVMe、SATA、SAS、RAID各自是什么以及数据怎么从盘走到CPU。一、先把概念分清楚很多人看到lsblk输出一堆sdX和nvmeXnX不知道哪个是什么盘根本原因是没分清这几个层次NVMe协议名走 PCIe 总线低延迟、高带宽、多队列是企业服务器高性能存储主流。SATA走 AHCI/SATA 协议结构简单速度比 NVMe 慢。lsblk看到sdX可能是 SATA 盘也可能是 RAID 逻辑盘不能直接判断。SAS企业级磁盘接口通常配合 HBA 卡和 RAID 控制器使用。RAID不是盘的类型是控制层。把多块物理盘组合成逻辑盘提供冗余和缓存能力常见模式 RAID0/1/5/10。 记住这个关系NVMe 是协议SATA/SAS 是接口体系RAID 是控制层PCIe 是总线M.2/U.2 是形态。二、实机存储设备识别本机配置两块 U.2 NVMe SSD、一张 Broadcom MegaRAID RAID 卡MR9560-16i、两块 Intel M.2 SATA SSD。第一步lsblk 看所有块设备[rootlocalhost ~]# lsblk -dNAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda8:00447.1G0disk sdb8:160447.1G0disk sdc8:320446.6G0disk sdd8:480446.6G0disk sde8:640446.6G0disk sdf8:800446.6G0disk sdg8:960446.6G0disk sdh8:1120446.6G0disk sdi8:1280446.6G0disk sdj8:1440446.6G0disk nvme1n1259:002.9T0disk nvme0n1259:102.9T0disknvme0n1和nvme1n1是 NVMe 盘sda到sdj共10个sdX设备但光看这里还不知道是什么盘继续查。加上分区挂载信息[rootlocalhost ~]# lsblkNAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS nvme1n1259:002.9T0disk ├─nvme1n1p1259:50600M0part /boot/efi ├─nvme1n1p2259:601G0part /boot └─nvme1n1p3259:702.9T0part ├─openeuler-root253:0070G0lvm / ├─openeuler-swap253:104G0lvm[SWAP]└─openeuler-home253:202.8T0lvm /home nvme0n1259:102.9T0disk └─nvme0n1p3259:402.9T0part ├─klas-swap253:304G0lvm ├─klas-backup253:4050G0lvm └─klas-root253:502.9T0lvmnvme1n1 装的是 openEulernvme0n1 装的是 Klas麒麟两块系统盘各跑一个 OS。第二步nvme list 确认 NVMe 盘型号[rootlocalhost ~]# nvme listNode SN Model Namespace Usage Format FW Rev ------------ ---------------- ------------------ ---------- ------------------- ------------ ------- /dev/nvme0n1 D77446D401J852 DERAP44YGM03T2US13.20TB /3.20TB512B 0B D7Y05M1F /dev/nvme1n1 D77446D4017E52 DERAP44YGM03T2US13.20TB /3.20TB512B 0B D7Y05M1F两块 NVMe 型号一致DERAP44YGM03T2US单块 3.2TBU.2 形态通过 PCIe 直连 CPU。第三步lsscsi 看清所有 sdX 是什么盘[rootlocalhost ~]# lsscsi[0:1:124:0]enclosu BROADCOM VirtualSES 03 -[0:3:104:0]disk BROADCOM MR9560-16i5.29/dev/sdc[0:3:105:0]disk BROADCOM MR9560-16i5.29/dev/sdd[0:3:106:0]disk BROADCOM MR9560-16i5.29/dev/sde[0:3:107:0]disk BROADCOM MR9560-16i5.29/dev/sdf[0:3:108:0]disk BROADCOM MR9560-16i5.29/dev/sdi[0:3:109:0]disk BROADCOM MR9560-16i5.29/dev/sdj[0:3:110:0]disk BROADCOM MR9560-16i5.29/dev/sdh[0:3:111:0]disk BROADCOM MR9560-16i5.29/dev/sdg[1:0:0:0]disk ATA INTEL SSDSCKKB48 0100 /dev/sda[2:0:0:0]disk ATA INTEL SSDSCKKB48 0100 /dev/sdb[N:0:0:1]disk DERAP44YGM03T2US__1 /dev/nvme0n1[N:1:0:1]disk DERAP44YGM03T2US__1 /dev/nvme1n1现在全清楚了设备来源类型sdc ~ sdj8块BROADCOM MR9560-16iRAID 控制器逻辑盘sda / sdb2块ATA INTEL SSDSCKKB48M.2 形态 SATA SSDnvme0n1 / nvme1n1DERAP44YGM03T2USU.2 NVMe 直连 PCIesda/sdb 显示 ATA 的原因这是 M.2 形态、SATA 协议的 Intel SSD。M.2 是形态SATA 是协议不是 NVMe。这两种协议共用 M.2 插槽容易混淆。第四步lspci 确认存储控制器[rootlocalhost ~]# lspci | egrep -i SATA|SAS|NVME|SCSI|raid30:04.0 Serial Attached SCSI controller: Huawei Technologies Co., Ltd. HiSilicon SAS3.0HBA(rev30)32:04.0 Serial Attached SCSI controller: Huawei Technologies Co., Ltd. HiSilicon SAS3.0HBA(rev30)38:05.0 SATA controller: Huawei Technologies Co., Ltd. HiSilicon AHCI HBA(rev30)70:04.0 Serial Attached SCSI controller: Huawei Technologies Co., Ltd. HiSilicon SAS3.0HBA(rev30)72:04.0 Serial Attached SCSI controller: Huawei Technologies Co., Ltd. HiSilicon SAS3.0HBA(rev30)83:00.0 RAID bus controller: Broadcom / LSI MegaRAID 12GSAS/PCIe Secure SAS39xx[rootlocalhost ~]# lspci | egrep -i Non81:00.0 Non-Volatile memory controller: DERA Storage Device151582:00.0 Non-Volatile memory controller: DERA Storage Device1515PCIe 设备作用81:00.0 / 82:00.0NVMe 控制器对应 nvme0n1 / nvme1n138:05.0 HiSilicon AHCI HBASATA 控制器挂 sda / sdb83:00.0 Broadcom MegaRAIDRAID 控制器挂 sdc ~ sdj30/32/70/72 HiSilicon SAS HBA华为板载 SAS 通道三、本机存储路径图CPU └─ PCIe Root Port ├─ NVMe Controller (81:00.0) → nvme0n1 (3.2TB系统盘 Klas) ├─ NVMe Controller (82:00.0) → nvme1n1 (3.2TB系统盘 openEuler) ├─ HiSilicon AHCI HBA (38:05.0) → sda / sdb (Intel M.2 SATA SSD × 2) └─ Broadcom MegaRAID (83:00.0) → sdc ~ sdj (RAID0 逻辑盘 × 8)这张图很重要。以后遇到存储慢不能只说磁盘慢要能往下追是 NVMe 本身慢是 RAID 层引入了额外延迟是 PCIe 链路有竞争是跨 NUMA 访问四、storcli64 管理 RAID 控制器storcli64是管理 Broadcom/LSI RAID 卡的专用工具语法是按/c0控制器→/vall逻辑盘→/eall/sall物理盘的层次来查。查控制器总览[rootlocalhost ~]# storcli64 showCLI Version007.2707.0000.0000 Dec18,2023StatusSuccess System Overview:------------------------------------------------------------------------- Ctl Model Ports PDs DGs DNOpt VDs VNOpt Hlth -------------------------------------------------------------------------0MegaRAID9560-16i8GB1688080Opt -------------------------------------------------------------------------关键字段PDs8块物理盘VDs8块逻辑盘DNOpt/VNOpt0表示无异常HlthOpt表示健康。查所有逻辑盘[rootlocalhost ~]# storcli64 /c0/vall showController0StatusSuccess Virtual Drives:----------------------------------------------------------- DG/VD TYPE State Access Consist Cache sCC Size -----------------------------------------------------------6/232 RAID0 Optl RW Yes RWTD ON446.625GB5/233 RAID0 Optl RW Yes RWTD ON446.625GB4/234 RAID0 Optl RW Yes RWTD ON446.625GB3/235 RAID0 Optl RW Yes RWTD ON446.625GB2/236 RAID0 Optl RW Yes RWTD ON446.625GB7/237 RAID0 Optl RW Yes RWTD ON446.625GB1/238 RAID0 Optl RW Yes RWTD ON446.625GB0/239 RAID0 Optl RW Yes RWTD ON446.625GB -----------------------------------------------------------8块物理盘各自做了 RAID0单盘直通对应 sdc ~ sdj每块 446.6GBState 全部 Optl正常。storcli64 常用命令速查# 看控制器列表storcli64 show# 看 c0 控制器总览storcli64 /c0 show# 看所有逻辑盘storcli64 /c0/vall show# 看所有物理盘storcli64 /c0/eall/sall show# 看完整详情storcli64 /c0 show all五、存储异常排查链盘不见了NVMe 不见了lsblk → nvme list → lspci|grep-invme →dmesg|grep-invme → lsmod|grepnvmeRAID/SAS 盘不见了lsblk → lsscsi → storcli64 show →dmesg→ lspci|grep-iraid盘还在但是慢# 先看 IO 指标iostat-x1# 重点关注# %util磁盘利用率是否打满# awaitIO 平均等待时间是否异常高# 对比 NVMe 和 RAID 逻辑盘的数据定位是哪层慢六、Python 存储检测脚本# storage_check.pyimportsubprocessprint( lsblk )print(subprocess.run([lsblk,-d],capture_outputTrue,textTrue).stdout)print( nvme list )print(subprocess.run([nvme,list],capture_outputTrue,textTrue).stdout)print( lsscsi )print(subprocess.run([lsscsi],capture_outputTrue,textTrue).stdout)python3 storage_check.py拿到一台新服务器先跑这个脚本三条命令的输出就能快速建立对存储架构的基本认知。总结层次概念本机实例形态M.2 / U.2NVMe 是 U.2SATA 是 M.2协议NVMe / SATA / SASNVMe 走 PCIeSATA 走 AHCI控制层RAIDBroadcom MegaRAID8块盘各做 RAID0排查工具lsblk / lsscsi / nvme list / storcli64/hiraidadm(华为raid卡专用)四步确认设备类型和健康状态下一篇 Day6 深入 NVMe 架构聊 NVMe 的内部结构、队列机制和健康状态查看敬请期待欢迎关注JACK的服务器笔记我们下篇见
