方案概述基于你的需求我将提供一个完整的、可分阶段实施的监控方案。你可以先安装部分节点完成功能测试再逐步扩展到全量节点。整体架构 第一阶段基础环境搭建1天1.1 部署Consul集群3节点在三台机器上执行以下操作假设IP192.168.1.10, 192.168.1.11, 192.168.1.12bash# 下载安装Consul wget https://releases.hashicorp.com/consul/1.18.0/consul_1.18.0_linux_amd64.zip unzip consul_1.18.0_linux_amd64.zip sudo mv consul /usr/local/bin/ # 创建目录和用户 sudo mkdir -p /etc/consul.d /var/lib/consul sudo useradd --system --home /etc/consul.d --shell /bin/false consul sudo chown -R consul:consul /etc/consul.d /var/lib/consul # 创建配置文件根据实际IP修改 sudo tee /etc/consul.d/server.hcl EOF datacenter dc1 data_dir /var/lib/consul server true bootstrap_expect 3 client_addr 0.0.0.0 bind_addr 0.0.0.0 advertise_addr 192.168.1.10 # 改为本机IP ui_config { enabled true } retry_join [192.168.1.10, 192.168.1.11, 192.168.1.12] ports { http 8500 grpc 8502 } acl { enabled true default_policy deny enable_token_persistence true } EOF # 创建systemd服务 sudo tee /etc/systemd/system/consul.service EOF [Unit] DescriptionConsul Agent Afternetwork.target [Service] Userconsul Groupconsul ExecStart/usr/local/bin/consul agent -config-dir/etc/consul.d ExecReload/bin/kill -HUP \$MAINPID Restarton-failure [Install] WantedBymulti-user.target EOF # 启动Consul sudo systemctl daemon-reload sudo systemctl enable consul sudo systemctl start consul1.2 初始化ACL并获取Token在任意一台Consul Server上执行bash# 获取bootstrap token consul acl bootstrap # 输出示例保存好SecretID # SecretID: 87654321-4321-4321-4321-cba987654321 export CONSUL_TOKEN87654321-4321-4321-4321-cba9876543211.3 部署Prometheusbash# 下载安装Prometheus wget https://github.com/prometheus/prometheus/releases/download/v2.45.0/prometheus-2.45.0.linux-amd64.tar.gz tar xvf prometheus-2.45.0.linux-amd64.tar.gz sudo mv prometheus-2.45.0.linux-amd64 /opt/prometheus # 创建数据目录和用户 sudo mkdir -p /var/lib/prometheus sudo useradd --system --no-create-home prometheus sudo chown -R prometheus:prometheus /opt/prometheus /var/lib/prometheus # 创建基础配置文件后面会逐步完善 sudo tee /opt/prometheus/prometheus.yml EOF global: scrape_interval: 30s evaluation_interval: 30s scrape_configs: - job_name: prometheus static_configs: - targets: [localhost:9090] EOF # 创建systemd服务 sudo tee /etc/systemd/system/prometheus.service EOF [Unit] DescriptionPrometheus Afternetwork.target [Service] Typesimple Userprometheus Groupprometheus ExecStart/opt/prometheus/prometheus \\ --config.file/opt/prometheus/prometheus.yml \\ --storage.tsdb.path/var/lib/prometheus \\ --web.listen-address:9090 Restartalways [Install] WantedBymulti-user.target EOF # 启动Prometheus sudo systemctl daemon-reload sudo systemctl enable prometheus sudo systemctl start prometheus1.4 部署Grafanabash# 安装Grafana wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add - echo deb https://packages.grafana.com/oss/deb stable main | sudo tee /etc/apt/sources.list.d/grafana.list sudo apt-get update sudo apt-get install -y grafana # 启动 sudo systemctl enable grafana-server sudo systemctl start grafana-server # 访问http://你的PrometheusIP:3000 # 默认用户名/密码admin/admin1.5 部署Alertmanagerbash# 下载安装Alertmanager wget https://github.com/prometheus/alertmanager/releases/download/v0.25.0/alertmanager-0.25.0.linux-amd64.tar.gz tar xvf alertmanager-0.25.0.linux-amd64.tar.gz sudo mv alertmanager-0.25.0.linux-amd64 /opt/alertmanager # 创建配置文件 sudo tee /opt/alertmanager/alertmanager.yml EOF global: resolve_timeout: 5m route: group_by: [alertname, env, role] group_wait: 10s group_interval: 10s repeat_interval: 1h receiver: webhook receivers: - name: webhook webhook_configs: - url: http://localhost:8080/alert # 你的告警接收地址 inhibit_rules: - source_match: severity: critical target_match: severity: warning equal: [alertname, env] EOF # 创建systemd服务 sudo tee /etc/systemd/system/alertmanager.service EOF [Unit] DescriptionAlertmanager Afternetwork.target [Service] Typesimple Userprometheus Groupprometheus ExecStart/opt/alertmanager/alertmanager \\ --config.file/opt/alertmanager/alertmanager.yml \\ --storage.path/var/lib/alertmanager Restartalways [Install] WantedBymulti-user.target EOF # 启动 sudo systemctl daemon-reload sudo systemctl enable alertmanager sudo systemctl start alertmanager1.6 验证基础环境bash# 验证Consul curl http://localhost:8500/v1/status/leader curl -H X-Consul-Token: $CONSUL_TOKEN http://localhost:8500/v1/catalog/services # 验证Prometheus curl http://localhost:9090/api/v1/targets # 验证Grafana curl http://localhost:3000/api/health # 验证Alertmanager curl http://localhost:9093/api/v2/status 第二阶段测试节点部署半天2.1 Node Exporter部署方式Node Exporter有两种启动方式根据你的环境选择 方式Asystemd服务推荐生产环境bash# 下载安装Node Exporter wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz tar xvf node_exporter-1.7.0.linux-amd64.tar.gz sudo mv node_exporter-1.7.0.linux-amd64/node_exporter /usr/local/bin/ # 创建专用用户 sudo useradd -rs /bin/false node_exporter # 创建systemd服务 sudo tee /etc/systemd/system/node_exporter.service EOF [Unit] DescriptionNode Exporter Afternetwork.target [Service] Usernode_exporter Groupnode_exporter Typesimple ExecStart/usr/local/bin/node_exporter \\ --web.listen-address:9100 \\ --path.rootfs/ Restartalways [Install] WantedBymulti-user.target EOF # 启动 sudo systemctl daemon-reload sudo systemctl enable node_exporter sudo systemctl start node_exporter # 验证 curl http://localhost:9100/metrics | head -n 5方式BDocker方式测试环境bash# 使用Docker运行 docker run -d \ --name node_exporter \ --restart always \ --nethost \ --pidhost \ -v /:/host:ro,rslave \ prom/node-exporter:latest \ --path.rootfs/host2.2 关于克隆的说明Node Exporter支持克隆/镜像部署没有client ID限制。你可以制作一个包含Node Exporter的黄金镜像使用clustershell批量分发新克隆的机器会自动注册到Consul注意事项每台机器需要有唯一的主机名用于生成唯一的Service IDConsul注册脚本中的service.id必须唯一我们使用node-exporter-$(hostname)确保唯一性2.3 创建Consul注册脚本bash# /tmp/register-to-consul.sh cat /tmp/register-to-consul.sh EOF #!/bin/bash CONSUL_SERVER192.168.1.10:8500 # 你的Consul Server IP TOKEN87654321-4321-4321-4321-cba987654321 # 获取基础信息 PRIVATE_IP$(hostname -I | awk {print $1}) HOSTNAME$(hostname) # 构造服务定义 cat /tmp/service.json SERVICE_EOF { ID: node-exporter-$HOSTNAME, Name: node_exporter, Tags: [huaweicloud, test], # 测试节点打test标签 Address: $PRIVATE_IP, Port: 9100, Meta: { hostname: $HOSTNAME, env: test, role: test-node, registered_at: $(date %Y%m%d-%H%M%S) }, Check: { http: http://$PRIVATE_IP:9100/metrics, interval: 30s, timeout: 5s, deregister_critical_service_after: 10m } } SERVICE_EOF # 注册到Consul curl -X PUT \ -H X-Consul-Token: $TOKEN \ --data /tmp/service.json \ http://$CONSUL_SERVER/v1/agent/service/register echo $(date): 注册 $HOSTNAME ($PRIVATE_IP) EOF chmod x /tmp/register-to-consul.sh2.4 在3台测试节点上执行使用clustershell批量操作bash# 分发注册脚本 clush -w 192.168.1.101,192.168.1.102,192.168.1.103 -c /tmp/register-to-consul.sh --dest/tmp/ # 执行注册 clush -w 192.168.1.101,192.168.1.102,192.168.1.103 bash /tmp/register-to-consul.sh # 添加到crontab每5分钟重新注册 clush -w 192.168.1.101,192.168.1.102,192.168.1.103 \ echo */5 * * * * root bash /tmp/register-to-consul.sh /etc/crontab2.5 验证测试节点注册bash# 查看所有注册的服务 curl -H X-Consul-Token: $CONSUL_TOKEN http://localhost:8500/v1/catalog/services # 查看node_exporter的测试节点 curl -H X-Consul-Token: $CONSUL_TOKEN \ http://localhost:8500/v1/health/service/node_exporter?tagtestpretty 第三阶段Prometheus配置与资源池分组3.1 完整Prometheus配置带分组更新/opt/prometheus/prometheus.ymlyamlglobal: scrape_interval: 30s evaluation_interval: 30s external_labels: monitor: huawei-cloud-ecs alerting: alertmanagers: - static_configs: - targets: [localhost:9093] rule_files: - alerts/*.yml scrape_configs: # 1. 监控Prometheus自身 - job_name: prometheus static_configs: - targets: [localhost:9090] labels: app: prometheus # 2. 监控Consul集群 - job_name: consul static_configs: - targets: [192.168.1.10:8500, 192.168.1.11:8500, 192.168.1.12:8500] metrics_path: /v1/agent/metrics params: format: [prometheus] relabel_configs: - source_labels: [__address__] target_label: instance # 3. 动态发现所有ECS节点 - job_name: huawei-ecs-all consul_sd_configs: - server: 192.168.1.10:8500 services: [node_exporter] token: 87654321-4321-4321-4321-cba987654321 refresh_interval: 30s relabel_configs: # 只保留健康检查通过的实例 - source_labels: [__meta_consul_health] regex: passing action: keep # 提取IP和端口 - source_labels: [__meta_consul_address, __meta_consul_service_port] separator: : target_label: __address__ # 提取元数据作为标签 - source_labels: [__meta_consul_metadata_hostname] target_label: hostname - source_labels: [__meta_consul_metadata_env] target_label: env - source_labels: [__meta_consul_metadata_role] target_label: role # 从Tags中提取标签 - source_labels: [__meta_consul_tags] regex: .*,prod,.* target_label: env_from_tag replacement: prod - source_labels: [__meta_consul_tags] regex: .*,staging,.* target_label: env_from_tag replacement: staging - source_labels: [__meta_consul_tags] regex: .*,test,.* target_label: env_from_tag replacement: test # 添加静态标签 - target_label: cloud_provider replacement: huaweicloud - target_label: discovery replacement: consul # 4. 按环境分组生产环境 - job_name: huawei-ecs-prod consul_sd_configs: - server: 192.168.1.10:8500 services: [node_exporter] token: 87654321-4321-4321-4321-cba987654321 tags: [prod] # 只发现带prod标签的实例 relabel_configs: - source_labels: [__meta_consul_health] regex: passing action: keep - source_labels: [__meta_consul_address, __meta_consul_service_port] separator: : target_label: __address__ - source_labels: [__meta_consul_metadata_role] target_label: role - source_labels: [__meta_consul_metadata_hostname] target_label: hostname - target_label: __scrape_interval__ replacement: 30s # 生产环境抓取更频繁 # 5. 按环境分组测试环境 - job_name: huawei-ecs-test consul_sd_configs: - server: 192.168.1.10:8500 services: [node_exporter] token: 87654321-4321-4321-4321-cba987654321 tags: [test] relabel_configs: - source_labels: [__meta_consul_health] regex: passing action: keep - source_labels: [__meta_consul_address, __meta_consul_service_port] separator: : target_label: __address__ - target_label: __scrape_interval__ replacement: 60s # 测试环境可以抓取频率低一些 # 6. 按角色分组数据库服务器 - job_name: huawei-ecs-database consul_sd_configs: - server: 192.168.1.10:8500 services: [node_exporter] token: 87654321-4321-4321-4321-cba987654321 relabel_configs: - source_labels: [__meta_consul_health] regex: passing action: keep - source_labels: [__meta_consul_metadata_role] regex: db|database|mysql|redis action: keep # 只保留数据库相关角色 - source_labels: [__meta_consul_address, __meta_consul_service_port] separator: : target_label: __address__ # 7. 按业务线分组电商业务 - job_name: huawei-ecs-ecommerce consul_sd_configs: - server: 192.168.1.10:8500 services: [node_exporter] token: 87654321-4321-4321-4321-cba987654321 relabel_configs: - source_labels: [__meta_consul_health] regex: passing action: keep - source_labels: [__meta_consul_metadata_business] regex: ecommerce action: keep - source_labels: [__meta_consul_address, __meta_consul_service_port] separator: : target_label: __address__3.2 按组查询的PromQL示例完成分组配置后你可以使用以下PromQL实现按组查询 promql# 1. 按环境统计节点数量 count by (env) (up{jobhuawei-ecs-all}) # 2. 按角色统计CPU使用率 avg by (role) ( 100 - (avg by (role, instance) (rate(node_cpu_seconds_total{modeidle}[5m])) * 100) ) # 3. 按业务线查看内存使用 sum by (business) ( node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes ) / 1024 / 1024 / 1024 # 4. 查看生产环境各角色的磁盘使用率 avg by (role, instance) ( (node_filesystem_size_bytes{fstype!tmpfs} - node_filesystem_free_bytes) / node_filesystem_size_bytes{fstype!tmpfs} * 100 ) and on(instance) (up{envprod} 1) # 5. 按团队聚合平均负载 avg by (team) (load1) # 6. 多标签分组按环境和角色 avg by (env, role) ( 100 - (avg by (env, role, instance) (rate(node_cpu_seconds_total{modeidle}[5m])) * 100) ) # 7. 使用group()进行标签分组保留指定标签丢弃其他[citation:10] group(node_cpu_seconds_total) by (env, role) # 8. 复杂分组按环境和实例类型 sum by (env, instance_type) ( count by (env, instance_type, instance) (up) ) # 9. 查看测试节点的具体指标 node_load1{envtest} # 10. 使用正则表达式过滤多个角色 up{role~web|api|frontend}3.3 Grafana自定义查询在Grafana中添加Prometheus数据源后创建Dashboard时可以使用变量实现动态分组Dashboard变量配置json{ templating: { list: [ { name: env, type: query, query: label_values(up, env) }, { name: role, type: query, query: label_values(up{env\$env\}, role) }, { name: instance, type: query, query: label_values(up{env\$env\, role\$role\}, instance) } ] } }常用Grafana面板查询promql# CPU使用率带变量 100 - (avg by (instance) (rate(node_cpu_seconds_total{modeidle, env$env, role~$role}[5m])) * 100) # 内存使用率 (1 - node_memory_MemAvailable_bytes{env$env} / node_memory_MemTotal_bytes{env$env}) * 100 # 磁盘使用率TOP 10 topk(10, (node_filesystem_size_bytes{fstype!tmpfs} - node_filesystem_free_bytes) / node_filesystem_size_bytes{fstype!tmpfs} * 100 ) by (instance, mountpoint)⚠️ 第四阶段告警规则配置4.1 创建告警规则文件创建/opt/prometheus/alerts/ecs-alerts.ymlyamlgroups: - name: huawei_ecs_basic_alerts interval: 30s rules: # 实例存活告警 - alert: ECSInstanceDown expr: up{jobhuawei-ecs-all} 0 for: 5m labels: severity: critical team: {{ $labels.team }} annotations: summary: ECS实例 {{ $labels.hostname }} 已离线 description: ECS实例 {{ $labels.hostname }} (环境: {{ $labels.env }}, 角色: {{ $labels.role }}) 已经离线超过5分钟。 - name: huawei_ecs_resource_alerts interval: 30s rules: # CPU使用率告警 - alert: HighCPUUsage expr: 100 - (avg by (instance, env, role, team) (rate(node_cpu_seconds_total{modeidle}[5m])) * 100) 80 for: 10m labels: severity: warning annotations: summary: {{ $labels.env }}环境 {{ $labels.role }} CPU使用率过高 description: {{ $labels.hostname }} CPU使用率已超过80% (当前值: {{ $value }}%) # 内存使用率告警 - alert: HighMemoryUsage expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 90 for: 5m labels: severity: warning annotations: summary: 内存使用率过高 - {{ $labels.hostname }} description: {{ $labels.hostname }} 内存使用率已超过90% (当前值: {{ $value }}%) # 磁盘使用率告警 - alert: DiskSpaceLow expr: (node_filesystem_size_bytes{fstype!tmpfs} - node_filesystem_free_bytes) / node_filesystem_size_bytes{fstype!tmpfs} * 100 85 for: 5m labels: severity: warning annotations: summary: 磁盘空间不足 - {{ $labels.hostname }} description: {{ $labels.hostname }} 磁盘 {{ $labels.device }} 使用率已超过85% - name: huawei_ecs_group_alerts interval: 30s rules: # 按环境分组告警生产环境CPU过高 - alert: ProdHighCPU expr: avg by (env) (100 - (avg by (env, instance) (rate(node_cpu_seconds_total{modeidle}[5m])) * 100)) 70 for: 10m labels: severity: warning env: prod annotations: summary: 生产环境整体CPU使用率过高 description: 生产环境整体CPU使用率已超过70% (当前值: {{ $value }}%) # 按角色分组告警数据库服务器磁盘不足 - alert: DBDiskLow expr: avg by (role) ((node_filesystem_size_bytes{fstype!tmpfs} - node_filesystem_free_bytes) / node_filesystem_size_bytes{fstype!tmpfs} * 100) 80 for: 5m labels: severity: critical role: db annotations: summary: 数据库服务器磁盘空间即将耗尽 description: 数据库服务器平均磁盘使用率已超过80% # 按业务线分组告警电商业务负载过高 - alert: EcommerceHighLoad expr: avg by (business) (load1) 10 for: 10m labels: severity: warning business: ecommerce annotations: summary: 电商业务线负载过高 description: 电商业务线平均负载已超过10 (当前值: {{ $value }})4.2 配置Alertmanager告警路由更新/opt/alertmanager/alertmanager.ymlyamlglobal: resolve_timeout: 5m smtp_smarthost: smtp.example.com:587 smtp_from: alertmanagerexample.com smtp_auth_username: alertmanagerexample.com smtp_auth_password: password route: group_by: [alertname, env, role, severity] group_wait: 30s group_interval: 5m repeat_interval: 4h # 根据标签路由到不同接收器 routes: # 严重告警发送到企业微信邮件 - match: severity: critical receiver: wechat-critical continue: true # 生产环境告警发送到邮件组 - match: env: prod receiver: email-prod continue: true # 数据库相关告警发送到DBA团队 - match: role: db receiver: email-dba # 默认路由 - receiver: default receivers: - name: default email_configs: - to: opsexample.com send_resolved: true - name: email-prod email_configs: - to: prod-teamexample.com send_resolved: true - name: email-dba email_configs: - to: dba-teamexample.com send_resolved: true - name: wechat-critical webhook_configs: - url: http://wechat-bot.example.com/send send_resolved: true inhibit_rules: - source_match: severity: critical target_match: severity: warning equal: [alertname, env, role]4.3 重载配置bash# 检查Prometheus配置 /opt/prometheus/promtool check config /opt/prometheus/prometheus.yml # 热加载 curl -X POST http://localhost:9090/-/reload # 检查Alertmanager配置 /opt/alertmanager/amtool check-config /opt/alertmanager/alertmanager.yml # 热加载 curl -X POST http://localhost:9093/-/reload 第五阶段扩展到全量节点5.1 创建分组配置文件模板bash# 创建模板目录 mkdir -p /tmp/group-templates # Web生产环境 cat /tmp/group-templates/web-prod.conf EOF ENVIRONMENTprod ROLEweb BUSINESSecommerce PROJECTshop-frontend OWNERfrontend-team EOF # Web预发布环境 cat /tmp/group-templates/web-staging.conf EOF ENVIRONMENTstaging ROLEweb BUSINESSecommerce PROJECTshop-frontend OWNERfrontend-team EOF # 数据库生产环境 cat /tmp/group-templates/db-prod.conf EOF ENVIRONMENTprod ROLEdb BUSINESSecommerce PROJECTshop-database OWNERdba-team EOF # Redis生产环境 cat /tmp/group-templates/redis-prod.conf EOF ENVIRONMENTprod ROLEredis BUSINESSecommerce PROJECTshop-cache OWNERbackend-team EOF # 默认配置 cat /tmp/group-templates/default.conf EOF ENVIRONMENTprod ROLEweb BUSINESSdefault PROJECTdefault OWNERops-team EOF5.2 增强版注册脚本bashcat /tmp/register-to-consul-v2.sh EOF #!/bin/bash CONSUL_SERVER192.168.1.10:8500 TOKEN87654321-4321-4321-4321-cba987654321 # 获取基础信息 PRIVATE_IP$(hostname -I | awk {print $1}) HOSTNAME$(hostname) # 从配置文件读取分组信息 GROUP_CONF/etc/server-group.conf if [ -f $GROUP_CONF ]; then source $GROUP_CONF else ENVIRONMENTprod ROLEweb BUSINESSdefault PROJECTdefault OWNERops-team fi # 构造标签 TAGS(huaweicloud $ENVIRONMENT $ROLE $BUSINESS) # 构造服务定义 cat /tmp/service.json SERVICE_EOF { ID: node-exporter-$HOSTNAME, Name: node_exporter, Tags: [${TAGS[0]}, ${TAGS[1]}, ${TAGS[2]}, ${TAGS[3]}], Address: $PRIVATE_IP, Port: 9100, Meta: { hostname: $HOSTNAME, environment: $ENVIRONMENT, role: $ROLE, business: $BUSINESS, project: $PROJECT, owner: $OWNER, registered_at: $(date %Y%m%d-%H%M%S) }, Check: { http: http://$PRIVATE_IP:9100/metrics, interval: 30s, timeout: 5s, deregister_critical_service_after: 10m } } SERVICE_EOF # 注册到Consul curl -X PUT \ -H X-Consul-Token: $TOKEN \ --data /tmp/service.json \ http://$CONSUL_SERVER/v1/agent/service/register echo $(date): 注册 $HOSTNAME ($PRIVATE_IP) [$ENVIRONMENT/$ROLE/$BUSINESS] EOF chmod x /tmp/register-to-consul-v2.sh5.3 使用clustershell批量部署到全量节点bash# 1. 安装Node Exporter到所有节点 clush -a # 下载安装node_exporter wget -q https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz tar xf node_exporter-1.7.0.linux-amd64.tar.gz sudo mv node_exporter-1.7.0.linux-amd64/node_exporter /usr/local/bin/ # 创建systemd服务 sudo tee /etc/systemd/system/node_exporter.service SERVICE_EOF [Unit] DescriptionNode Exporter Afternetwork.target [Service] Usernobody Groupnogroup Typesimple ExecStart/usr/local/bin/node_exporter --web.listen-address:9100 Restartalways [Install] WantedBymulti-user.target SERVICE_EOF # 启动 sudo systemctl daemon-reload sudo systemctl enable node_exporter sudo systemctl start node_exporter # 2. 根据节点角色分发分组配置文件 # Web生产环境 clush -g web-prod -c /tmp/group-templates/web-prod.conf --dest/etc/server-group.conf # Web预发布环境 clush -g web-staging -c /tmp/group-templates/web-staging.conf --dest/etc/server-group.conf # DB生产环境 clush -g db-prod -c /tmp/group-templates/db-prod.conf --dest/etc/server-group.conf # Redis生产环境 clush -g redis-prod -c /tmp/group-templates/redis-prod.conf --dest/etc/server-group.conf # 其他节点使用默认配置 clush -g others -c /tmp/group-templates/default.conf --dest/etc/server-group.conf # 3. 分发注册脚本 clush -a -c /tmp/register-to-consul-v2.sh --dest/usr/local/bin/ clush -a chmod x /usr/local/bin/register-to-consul-v2.sh # 4. 执行首次注册 clush -a /usr/local/bin/register-to-consul-v2.sh # 5. 添加到crontab clush -a echo */5 * * * * root /usr/local/bin/register-to-consul-v2.sh /etc/crontab 第六阶段验证与测试6.1 验证Consul注册bash# 查看所有服务 curl -H X-Consul-Token: $CONSUL_TOKEN http://localhost:8500/v1/catalog/services # 按环境统计 curl -s -H X-Consul-Token: $CONSUL_TOKEN \ http://localhost:8500/v1/health/service/node_exporter?passing | \ jq -r .[].Service.Meta.environment | sort | uniq -c # 按角色统计 curl -s -H X-Consul-Token: $CONSUL_TOKEN \ http://localhost:8500/v1/health/service/node_exporter?passing | \ jq -r .[].Service.Meta.role | sort | uniq -c # 按业务线统计 curl -s -H X-Consul-Token: $CONSUL_TOKEN \ http://localhost:8500/v1/health/service/node_exporter?passing | \ jq -r .[].Service.Meta.business | sort | uniq -c6.2 验证Prometheus Targetsbash# 查看所有targets curl http://localhost:9090/api/v1/targets | jq .data.activeTargets[] | {job: .labels.job, instance: .labels.instance, health: .health} # 按job统计 curl -s http://localhost:9090/api/v1/targets | \ jq .data.activeTargets | group_by(.labels.job) | map({job: .[0].labels.job, count: length})6.3 测试告警bash# 手动触发告警停止一个Node Exporter clush -w 192.168.1.101 sudo systemctl stop node_exporter # 等待5分钟后查看告警 curl http://localhost:9093/api/v2/alerts | jq .[] | {name: .labels.alertname, status: .status.state} # 恢复 clush -w 192.168.1.101 sudo systemctl start node_exporter6.4 Grafana导入Dashboardbash# 推荐导入的Dashboard ID # 8919 - Node Exporter Server Metrics # 11074 - Node Exporter 1-Panel # 1860 - Node Exporter Full # 或者创建自定义Dashboard使用以下PromQLpromql# 自定义分组面板 # 按环境分组CPU使用率 avg by (env) (100 - (avg by (env, instance) (rate(node_cpu_seconds_total{modeidle}[5m])) * 100)) # 按角色分组内存使用 sum by (role) (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / 1024 / 1024 / 1024 # 按业务线分组节点数量 count by (business) (up) # 按团队分组磁盘总量 sum by (team) (node_filesystem_size_bytes{fstype!tmpfs}) / 1024 / 1024 / 1024 / 1024 常见问题解答Q1: Node Exporter启动方式选择A:systemd方式适合生产环境稳定可靠有完善的日志和重启策略Docker方式适合测试环境或容器化部署但需要注意网络模式设置Q2: 克隆机器是否有限制A: Node Exporter本身没有client ID限制。只要确保每台机器的主机名唯一用于生成Service IDIP地址唯一注册到Consul时使用唯一的Service ID我们使用node-exporter-$(hostname)确保唯一性Q3: 如何实现按组查询A: 通过Consul注册时的标签Tags和元数据MetaPrometheus会自动将这些信息转换为标签然后使用PromQL的by子句进行分组查询 promql# 按环境分组 avg by (env) (rate(node_cpu_seconds_total[5m])) # 多标签分组 sum by (env, role) (node_memory_MemTotal_bytes)Q4: 告警规则如何按分组路由A: 在Alertmanager配置中可以根据告警的标签进行路由 yamlroutes: - match: env: prod severity: critical receiver: prod-criticalQ5: 如何测试告警是否正常工作A: 可以手动停止一个Node Exporter服务查看Prometheus Alerts页面确认告警状态变为PENDING→FIRING查看Alertmanager接收到的告警验证通知渠道是否收到消息 总结已完成的功能阶段功能状态第一阶段Consul集群、Prometheus、Grafana、Alertmanager基础环境✅第二阶段3台测试节点部署验证动态发现✅第三阶段Prometheus分组配置按环境/角色/业务线分组✅第四阶段告警规则配置按分组路由✅第五阶段扩展到全量节点分组模板✅第六阶段验证与测试自定义Grafana查询✅可扩展的方向多集群支持配置多个Consul datacenter自定义ExporterMySQL、Nginx、Redis等应用监控日志集成添加Loki实现日志与指标联动自动化运维通过Consul API实现自动扩缩容时的服务注册/注销
Prometheus + Consul 完整监控解决方案（分阶段实施）
方案概述基于你的需求我将提供一个完整的、可分阶段实施的监控方案。你可以先安装部分节点完成功能测试再逐步扩展到全量节点。整体架构 第一阶段基础环境搭建1天1.1 部署Consul集群3节点在三台机器上执行以下操作假设IP192.168.1.10, 192.168.1.11, 192.168.1.12bash# 下载安装Consul wget https://releases.hashicorp.com/consul/1.18.0/consul_1.18.0_linux_amd64.zip unzip consul_1.18.0_linux_amd64.zip sudo mv consul /usr/local/bin/ # 创建目录和用户 sudo mkdir -p /etc/consul.d /var/lib/consul sudo useradd --system --home /etc/consul.d --shell /bin/false consul sudo chown -R consul:consul /etc/consul.d /var/lib/consul # 创建配置文件根据实际IP修改 sudo tee /etc/consul.d/server.hcl EOF datacenter dc1 data_dir /var/lib/consul server true bootstrap_expect 3 client_addr 0.0.0.0 bind_addr 0.0.0.0 advertise_addr 192.168.1.10 # 改为本机IP ui_config { enabled true } retry_join [192.168.1.10, 192.168.1.11, 192.168.1.12] ports { http 8500 grpc 8502 } acl { enabled true default_policy deny enable_token_persistence true } EOF # 创建systemd服务 sudo tee /etc/systemd/system/consul.service EOF [Unit] DescriptionConsul Agent Afternetwork.target [Service] Userconsul Groupconsul ExecStart/usr/local/bin/consul agent -config-dir/etc/consul.d ExecReload/bin/kill -HUP \$MAINPID Restarton-failure [Install] WantedBymulti-user.target EOF # 启动Consul sudo systemctl daemon-reload sudo systemctl enable consul sudo systemctl start consul1.2 初始化ACL并获取Token在任意一台Consul Server上执行bash# 获取bootstrap token consul acl bootstrap # 输出示例保存好SecretID # SecretID: 87654321-4321-4321-4321-cba987654321 export CONSUL_TOKEN87654321-4321-4321-4321-cba9876543211.3 部署Prometheusbash# 下载安装Prometheus wget https://github.com/prometheus/prometheus/releases/download/v2.45.0/prometheus-2.45.0.linux-amd64.tar.gz tar xvf prometheus-2.45.0.linux-amd64.tar.gz sudo mv prometheus-2.45.0.linux-amd64 /opt/prometheus # 创建数据目录和用户 sudo mkdir -p /var/lib/prometheus sudo useradd --system --no-create-home prometheus sudo chown -R prometheus:prometheus /opt/prometheus /var/lib/prometheus # 创建基础配置文件后面会逐步完善 sudo tee /opt/prometheus/prometheus.yml EOF global: scrape_interval: 30s evaluation_interval: 30s scrape_configs: - job_name: prometheus static_configs: - targets: [localhost:9090] EOF # 创建systemd服务 sudo tee /etc/systemd/system/prometheus.service EOF [Unit] DescriptionPrometheus Afternetwork.target [Service] Typesimple Userprometheus Groupprometheus ExecStart/opt/prometheus/prometheus \\ --config.file/opt/prometheus/prometheus.yml \\ --storage.tsdb.path/var/lib/prometheus \\ --web.listen-address:9090 Restartalways [Install] WantedBymulti-user.target EOF # 启动Prometheus sudo systemctl daemon-reload sudo systemctl enable prometheus sudo systemctl start prometheus1.4 部署Grafanabash# 安装Grafana wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add - echo deb https://packages.grafana.com/oss/deb stable main | sudo tee /etc/apt/sources.list.d/grafana.list sudo apt-get update sudo apt-get install -y grafana # 启动 sudo systemctl enable grafana-server sudo systemctl start grafana-server # 访问http://你的PrometheusIP:3000 # 默认用户名/密码admin/admin1.5 部署Alertmanagerbash# 下载安装Alertmanager wget https://github.com/prometheus/alertmanager/releases/download/v0.25.0/alertmanager-0.25.0.linux-amd64.tar.gz tar xvf alertmanager-0.25.0.linux-amd64.tar.gz sudo mv alertmanager-0.25.0.linux-amd64 /opt/alertmanager # 创建配置文件 sudo tee /opt/alertmanager/alertmanager.yml EOF global: resolve_timeout: 5m route: group_by: [alertname, env, role] group_wait: 10s group_interval: 10s repeat_interval: 1h receiver: webhook receivers: - name: webhook webhook_configs: - url: http://localhost:8080/alert # 你的告警接收地址 inhibit_rules: - source_match: severity: critical target_match: severity: warning equal: [alertname, env] EOF # 创建systemd服务 sudo tee /etc/systemd/system/alertmanager.service EOF [Unit] DescriptionAlertmanager Afternetwork.target [Service] Typesimple Userprometheus Groupprometheus ExecStart/opt/alertmanager/alertmanager \\ --config.file/opt/alertmanager/alertmanager.yml \\ --storage.path/var/lib/alertmanager Restartalways [Install] WantedBymulti-user.target EOF # 启动 sudo systemctl daemon-reload sudo systemctl enable alertmanager sudo systemctl start alertmanager1.6 验证基础环境bash# 验证Consul curl http://localhost:8500/v1/status/leader curl -H X-Consul-Token: $CONSUL_TOKEN http://localhost:8500/v1/catalog/services # 验证Prometheus curl http://localhost:9090/api/v1/targets # 验证Grafana curl http://localhost:3000/api/health # 验证Alertmanager curl http://localhost:9093/api/v2/status 第二阶段测试节点部署半天2.1 Node Exporter部署方式Node Exporter有两种启动方式根据你的环境选择 方式Asystemd服务推荐生产环境bash# 下载安装Node Exporter wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz tar xvf node_exporter-1.7.0.linux-amd64.tar.gz sudo mv node_exporter-1.7.0.linux-amd64/node_exporter /usr/local/bin/ # 创建专用用户 sudo useradd -rs /bin/false node_exporter # 创建systemd服务 sudo tee /etc/systemd/system/node_exporter.service EOF [Unit] DescriptionNode Exporter Afternetwork.target [Service] Usernode_exporter Groupnode_exporter Typesimple ExecStart/usr/local/bin/node_exporter \\ --web.listen-address:9100 \\ --path.rootfs/ Restartalways [Install] WantedBymulti-user.target EOF # 启动 sudo systemctl daemon-reload sudo systemctl enable node_exporter sudo systemctl start node_exporter # 验证 curl http://localhost:9100/metrics | head -n 5方式BDocker方式测试环境bash# 使用Docker运行 docker run -d \ --name node_exporter \ --restart always \ --nethost \ --pidhost \ -v /:/host:ro,rslave \ prom/node-exporter:latest \ --path.rootfs/host2.2 关于克隆的说明Node Exporter支持克隆/镜像部署没有client ID限制。你可以制作一个包含Node Exporter的黄金镜像使用clustershell批量分发新克隆的机器会自动注册到Consul注意事项每台机器需要有唯一的主机名用于生成唯一的Service IDConsul注册脚本中的service.id必须唯一我们使用node-exporter-$(hostname)确保唯一性2.3 创建Consul注册脚本bash# /tmp/register-to-consul.sh cat /tmp/register-to-consul.sh EOF #!/bin/bash CONSUL_SERVER192.168.1.10:8500 # 你的Consul Server IP TOKEN87654321-4321-4321-4321-cba987654321 # 获取基础信息 PRIVATE_IP$(hostname -I | awk {print $1}) HOSTNAME$(hostname) # 构造服务定义 cat /tmp/service.json SERVICE_EOF { ID: node-exporter-$HOSTNAME, Name: node_exporter, Tags: [huaweicloud, test], # 测试节点打test标签 Address: $PRIVATE_IP, Port: 9100, Meta: { hostname: $HOSTNAME, env: test, role: test-node, registered_at: $(date %Y%m%d-%H%M%S) }, Check: { http: http://$PRIVATE_IP:9100/metrics, interval: 30s, timeout: 5s, deregister_critical_service_after: 10m } } SERVICE_EOF # 注册到Consul curl -X PUT \ -H X-Consul-Token: $TOKEN \ --data /tmp/service.json \ http://$CONSUL_SERVER/v1/agent/service/register echo $(date): 注册 $HOSTNAME ($PRIVATE_IP) EOF chmod x /tmp/register-to-consul.sh2.4 在3台测试节点上执行使用clustershell批量操作bash# 分发注册脚本 clush -w 192.168.1.101,192.168.1.102,192.168.1.103 -c /tmp/register-to-consul.sh --dest/tmp/ # 执行注册 clush -w 192.168.1.101,192.168.1.102,192.168.1.103 bash /tmp/register-to-consul.sh # 添加到crontab每5分钟重新注册 clush -w 192.168.1.101,192.168.1.102,192.168.1.103 \ echo */5 * * * * root bash /tmp/register-to-consul.sh /etc/crontab2.5 验证测试节点注册bash# 查看所有注册的服务 curl -H X-Consul-Token: $CONSUL_TOKEN http://localhost:8500/v1/catalog/services # 查看node_exporter的测试节点 curl -H X-Consul-Token: $CONSUL_TOKEN \ http://localhost:8500/v1/health/service/node_exporter?tagtestpretty 第三阶段Prometheus配置与资源池分组3.1 完整Prometheus配置带分组更新/opt/prometheus/prometheus.ymlyamlglobal: scrape_interval: 30s evaluation_interval: 30s external_labels: monitor: huawei-cloud-ecs alerting: alertmanagers: - static_configs: - targets: [localhost:9093] rule_files: - alerts/*.yml scrape_configs: # 1. 监控Prometheus自身 - job_name: prometheus static_configs: - targets: [localhost:9090] labels: app: prometheus # 2. 监控Consul集群 - job_name: consul static_configs: - targets: [192.168.1.10:8500, 192.168.1.11:8500, 192.168.1.12:8500] metrics_path: /v1/agent/metrics params: format: [prometheus] relabel_configs: - source_labels: [__address__] target_label: instance # 3. 动态发现所有ECS节点 - job_name: huawei-ecs-all consul_sd_configs: - server: 192.168.1.10:8500 services: [node_exporter] token: 87654321-4321-4321-4321-cba987654321 refresh_interval: 30s relabel_configs: # 只保留健康检查通过的实例 - source_labels: [__meta_consul_health] regex: passing action: keep # 提取IP和端口 - source_labels: [__meta_consul_address, __meta_consul_service_port] separator: : target_label: __address__ # 提取元数据作为标签 - source_labels: [__meta_consul_metadata_hostname] target_label: hostname - source_labels: [__meta_consul_metadata_env] target_label: env - source_labels: [__meta_consul_metadata_role] target_label: role # 从Tags中提取标签 - source_labels: [__meta_consul_tags] regex: .*,prod,.* target_label: env_from_tag replacement: prod - source_labels: [__meta_consul_tags] regex: .*,staging,.* target_label: env_from_tag replacement: staging - source_labels: [__meta_consul_tags] regex: .*,test,.* target_label: env_from_tag replacement: test # 添加静态标签 - target_label: cloud_provider replacement: huaweicloud - target_label: discovery replacement: consul # 4. 按环境分组生产环境 - job_name: huawei-ecs-prod consul_sd_configs: - server: 192.168.1.10:8500 services: [node_exporter] token: 87654321-4321-4321-4321-cba987654321 tags: [prod] # 只发现带prod标签的实例 relabel_configs: - source_labels: [__meta_consul_health] regex: passing action: keep - source_labels: [__meta_consul_address, __meta_consul_service_port] separator: : target_label: __address__ - source_labels: [__meta_consul_metadata_role] target_label: role - source_labels: [__meta_consul_metadata_hostname] target_label: hostname - target_label: __scrape_interval__ replacement: 30s # 生产环境抓取更频繁 # 5. 按环境分组测试环境 - job_name: huawei-ecs-test consul_sd_configs: - server: 192.168.1.10:8500 services: [node_exporter] token: 87654321-4321-4321-4321-cba987654321 tags: [test] relabel_configs: - source_labels: [__meta_consul_health] regex: passing action: keep - source_labels: [__meta_consul_address, __meta_consul_service_port] separator: : target_label: __address__ - target_label: __scrape_interval__ replacement: 60s # 测试环境可以抓取频率低一些 # 6. 按角色分组数据库服务器 - job_name: huawei-ecs-database consul_sd_configs: - server: 192.168.1.10:8500 services: [node_exporter] token: 87654321-4321-4321-4321-cba987654321 relabel_configs: - source_labels: [__meta_consul_health] regex: passing action: keep - source_labels: [__meta_consul_metadata_role] regex: db|database|mysql|redis action: keep # 只保留数据库相关角色 - source_labels: [__meta_consul_address, __meta_consul_service_port] separator: : target_label: __address__ # 7. 按业务线分组电商业务 - job_name: huawei-ecs-ecommerce consul_sd_configs: - server: 192.168.1.10:8500 services: [node_exporter] token: 87654321-4321-4321-4321-cba987654321 relabel_configs: - source_labels: [__meta_consul_health] regex: passing action: keep - source_labels: [__meta_consul_metadata_business] regex: ecommerce action: keep - source_labels: [__meta_consul_address, __meta_consul_service_port] separator: : target_label: __address__3.2 按组查询的PromQL示例完成分组配置后你可以使用以下PromQL实现按组查询 promql# 1. 按环境统计节点数量 count by (env) (up{jobhuawei-ecs-all}) # 2. 按角色统计CPU使用率 avg by (role) ( 100 - (avg by (role, instance) (rate(node_cpu_seconds_total{modeidle}[5m])) * 100) ) # 3. 按业务线查看内存使用 sum by (business) ( node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes ) / 1024 / 1024 / 1024 # 4. 查看生产环境各角色的磁盘使用率 avg by (role, instance) ( (node_filesystem_size_bytes{fstype!tmpfs} - node_filesystem_free_bytes) / node_filesystem_size_bytes{fstype!tmpfs} * 100 ) and on(instance) (up{envprod} 1) # 5. 按团队聚合平均负载 avg by (team) (load1) # 6. 多标签分组按环境和角色 avg by (env, role) ( 100 - (avg by (env, role, instance) (rate(node_cpu_seconds_total{modeidle}[5m])) * 100) ) # 7. 使用group()进行标签分组保留指定标签丢弃其他[citation:10] group(node_cpu_seconds_total) by (env, role) # 8. 复杂分组按环境和实例类型 sum by (env, instance_type) ( count by (env, instance_type, instance) (up) ) # 9. 查看测试节点的具体指标 node_load1{envtest} # 10. 使用正则表达式过滤多个角色 up{role~web|api|frontend}3.3 Grafana自定义查询在Grafana中添加Prometheus数据源后创建Dashboard时可以使用变量实现动态分组Dashboard变量配置json{ templating: { list: [ { name: env, type: query, query: label_values(up, env) }, { name: role, type: query, query: label_values(up{env\$env\}, role) }, { name: instance, type: query, query: label_values(up{env\$env\, role\$role\}, instance) } ] } }常用Grafana面板查询promql# CPU使用率带变量 100 - (avg by (instance) (rate(node_cpu_seconds_total{modeidle, env$env, role~$role}[5m])) * 100) # 内存使用率 (1 - node_memory_MemAvailable_bytes{env$env} / node_memory_MemTotal_bytes{env$env}) * 100 # 磁盘使用率TOP 10 topk(10, (node_filesystem_size_bytes{fstype!tmpfs} - node_filesystem_free_bytes) / node_filesystem_size_bytes{fstype!tmpfs} * 100 ) by (instance, mountpoint)⚠️ 第四阶段告警规则配置4.1 创建告警规则文件创建/opt/prometheus/alerts/ecs-alerts.ymlyamlgroups: - name: huawei_ecs_basic_alerts interval: 30s rules: # 实例存活告警 - alert: ECSInstanceDown expr: up{jobhuawei-ecs-all} 0 for: 5m labels: severity: critical team: {{ $labels.team }} annotations: summary: ECS实例 {{ $labels.hostname }} 已离线 description: ECS实例 {{ $labels.hostname }} (环境: {{ $labels.env }}, 角色: {{ $labels.role }}) 已经离线超过5分钟。 - name: huawei_ecs_resource_alerts interval: 30s rules: # CPU使用率告警 - alert: HighCPUUsage expr: 100 - (avg by (instance, env, role, team) (rate(node_cpu_seconds_total{modeidle}[5m])) * 100) 80 for: 10m labels: severity: warning annotations: summary: {{ $labels.env }}环境 {{ $labels.role }} CPU使用率过高 description: {{ $labels.hostname }} CPU使用率已超过80% (当前值: {{ $value }}%) # 内存使用率告警 - alert: HighMemoryUsage expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 90 for: 5m labels: severity: warning annotations: summary: 内存使用率过高 - {{ $labels.hostname }} description: {{ $labels.hostname }} 内存使用率已超过90% (当前值: {{ $value }}%) # 磁盘使用率告警 - alert: DiskSpaceLow expr: (node_filesystem_size_bytes{fstype!tmpfs} - node_filesystem_free_bytes) / node_filesystem_size_bytes{fstype!tmpfs} * 100 85 for: 5m labels: severity: warning annotations: summary: 磁盘空间不足 - {{ $labels.hostname }} description: {{ $labels.hostname }} 磁盘 {{ $labels.device }} 使用率已超过85% - name: huawei_ecs_group_alerts interval: 30s rules: # 按环境分组告警生产环境CPU过高 - alert: ProdHighCPU expr: avg by (env) (100 - (avg by (env, instance) (rate(node_cpu_seconds_total{modeidle}[5m])) * 100)) 70 for: 10m labels: severity: warning env: prod annotations: summary: 生产环境整体CPU使用率过高 description: 生产环境整体CPU使用率已超过70% (当前值: {{ $value }}%) # 按角色分组告警数据库服务器磁盘不足 - alert: DBDiskLow expr: avg by (role) ((node_filesystem_size_bytes{fstype!tmpfs} - node_filesystem_free_bytes) / node_filesystem_size_bytes{fstype!tmpfs} * 100) 80 for: 5m labels: severity: critical role: db annotations: summary: 数据库服务器磁盘空间即将耗尽 description: 数据库服务器平均磁盘使用率已超过80% # 按业务线分组告警电商业务负载过高 - alert: EcommerceHighLoad expr: avg by (business) (load1) 10 for: 10m labels: severity: warning business: ecommerce annotations: summary: 电商业务线负载过高 description: 电商业务线平均负载已超过10 (当前值: {{ $value }})4.2 配置Alertmanager告警路由更新/opt/alertmanager/alertmanager.ymlyamlglobal: resolve_timeout: 5m smtp_smarthost: smtp.example.com:587 smtp_from: alertmanagerexample.com smtp_auth_username: alertmanagerexample.com smtp_auth_password: password route: group_by: [alertname, env, role, severity] group_wait: 30s group_interval: 5m repeat_interval: 4h # 根据标签路由到不同接收器 routes: # 严重告警发送到企业微信邮件 - match: severity: critical receiver: wechat-critical continue: true # 生产环境告警发送到邮件组 - match: env: prod receiver: email-prod continue: true # 数据库相关告警发送到DBA团队 - match: role: db receiver: email-dba # 默认路由 - receiver: default receivers: - name: default email_configs: - to: opsexample.com send_resolved: true - name: email-prod email_configs: - to: prod-teamexample.com send_resolved: true - name: email-dba email_configs: - to: dba-teamexample.com send_resolved: true - name: wechat-critical webhook_configs: - url: http://wechat-bot.example.com/send send_resolved: true inhibit_rules: - source_match: severity: critical target_match: severity: warning equal: [alertname, env, role]4.3 重载配置bash# 检查Prometheus配置 /opt/prometheus/promtool check config /opt/prometheus/prometheus.yml # 热加载 curl -X POST http://localhost:9090/-/reload # 检查Alertmanager配置 /opt/alertmanager/amtool check-config /opt/alertmanager/alertmanager.yml # 热加载 curl -X POST http://localhost:9093/-/reload 第五阶段扩展到全量节点5.1 创建分组配置文件模板bash# 创建模板目录 mkdir -p /tmp/group-templates # Web生产环境 cat /tmp/group-templates/web-prod.conf EOF ENVIRONMENTprod ROLEweb BUSINESSecommerce PROJECTshop-frontend OWNERfrontend-team EOF # Web预发布环境 cat /tmp/group-templates/web-staging.conf EOF ENVIRONMENTstaging ROLEweb BUSINESSecommerce PROJECTshop-frontend OWNERfrontend-team EOF # 数据库生产环境 cat /tmp/group-templates/db-prod.conf EOF ENVIRONMENTprod ROLEdb BUSINESSecommerce PROJECTshop-database OWNERdba-team EOF # Redis生产环境 cat /tmp/group-templates/redis-prod.conf EOF ENVIRONMENTprod ROLEredis BUSINESSecommerce PROJECTshop-cache OWNERbackend-team EOF # 默认配置 cat /tmp/group-templates/default.conf EOF ENVIRONMENTprod ROLEweb BUSINESSdefault PROJECTdefault OWNERops-team EOF5.2 增强版注册脚本bashcat /tmp/register-to-consul-v2.sh EOF #!/bin/bash CONSUL_SERVER192.168.1.10:8500 TOKEN87654321-4321-4321-4321-cba987654321 # 获取基础信息 PRIVATE_IP$(hostname -I | awk {print $1}) HOSTNAME$(hostname) # 从配置文件读取分组信息 GROUP_CONF/etc/server-group.conf if [ -f $GROUP_CONF ]; then source $GROUP_CONF else ENVIRONMENTprod ROLEweb BUSINESSdefault PROJECTdefault OWNERops-team fi # 构造标签 TAGS(huaweicloud $ENVIRONMENT $ROLE $BUSINESS) # 构造服务定义 cat /tmp/service.json SERVICE_EOF { ID: node-exporter-$HOSTNAME, Name: node_exporter, Tags: [${TAGS[0]}, ${TAGS[1]}, ${TAGS[2]}, ${TAGS[3]}], Address: $PRIVATE_IP, Port: 9100, Meta: { hostname: $HOSTNAME, environment: $ENVIRONMENT, role: $ROLE, business: $BUSINESS, project: $PROJECT, owner: $OWNER, registered_at: $(date %Y%m%d-%H%M%S) }, Check: { http: http://$PRIVATE_IP:9100/metrics, interval: 30s, timeout: 5s, deregister_critical_service_after: 10m } } SERVICE_EOF # 注册到Consul curl -X PUT \ -H X-Consul-Token: $TOKEN \ --data /tmp/service.json \ http://$CONSUL_SERVER/v1/agent/service/register echo $(date): 注册 $HOSTNAME ($PRIVATE_IP) [$ENVIRONMENT/$ROLE/$BUSINESS] EOF chmod x /tmp/register-to-consul-v2.sh5.3 使用clustershell批量部署到全量节点bash# 1. 安装Node Exporter到所有节点 clush -a # 下载安装node_exporter wget -q https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz tar xf node_exporter-1.7.0.linux-amd64.tar.gz sudo mv node_exporter-1.7.0.linux-amd64/node_exporter /usr/local/bin/ # 创建systemd服务 sudo tee /etc/systemd/system/node_exporter.service SERVICE_EOF [Unit] DescriptionNode Exporter Afternetwork.target [Service] Usernobody Groupnogroup Typesimple ExecStart/usr/local/bin/node_exporter --web.listen-address:9100 Restartalways [Install] WantedBymulti-user.target SERVICE_EOF # 启动 sudo systemctl daemon-reload sudo systemctl enable node_exporter sudo systemctl start node_exporter # 2. 根据节点角色分发分组配置文件 # Web生产环境 clush -g web-prod -c /tmp/group-templates/web-prod.conf --dest/etc/server-group.conf # Web预发布环境 clush -g web-staging -c /tmp/group-templates/web-staging.conf --dest/etc/server-group.conf # DB生产环境 clush -g db-prod -c /tmp/group-templates/db-prod.conf --dest/etc/server-group.conf # Redis生产环境 clush -g redis-prod -c /tmp/group-templates/redis-prod.conf --dest/etc/server-group.conf # 其他节点使用默认配置 clush -g others -c /tmp/group-templates/default.conf --dest/etc/server-group.conf # 3. 分发注册脚本 clush -a -c /tmp/register-to-consul-v2.sh --dest/usr/local/bin/ clush -a chmod x /usr/local/bin/register-to-consul-v2.sh # 4. 执行首次注册 clush -a /usr/local/bin/register-to-consul-v2.sh # 5. 添加到crontab clush -a echo */5 * * * * root /usr/local/bin/register-to-consul-v2.sh /etc/crontab 第六阶段验证与测试6.1 验证Consul注册bash# 查看所有服务 curl -H X-Consul-Token: $CONSUL_TOKEN http://localhost:8500/v1/catalog/services # 按环境统计 curl -s -H X-Consul-Token: $CONSUL_TOKEN \ http://localhost:8500/v1/health/service/node_exporter?passing | \ jq -r .[].Service.Meta.environment | sort | uniq -c # 按角色统计 curl -s -H X-Consul-Token: $CONSUL_TOKEN \ http://localhost:8500/v1/health/service/node_exporter?passing | \ jq -r .[].Service.Meta.role | sort | uniq -c # 按业务线统计 curl -s -H X-Consul-Token: $CONSUL_TOKEN \ http://localhost:8500/v1/health/service/node_exporter?passing | \ jq -r .[].Service.Meta.business | sort | uniq -c6.2 验证Prometheus Targetsbash# 查看所有targets curl http://localhost:9090/api/v1/targets | jq .data.activeTargets[] | {job: .labels.job, instance: .labels.instance, health: .health} # 按job统计 curl -s http://localhost:9090/api/v1/targets | \ jq .data.activeTargets | group_by(.labels.job) | map({job: .[0].labels.job, count: length})6.3 测试告警bash# 手动触发告警停止一个Node Exporter clush -w 192.168.1.101 sudo systemctl stop node_exporter # 等待5分钟后查看告警 curl http://localhost:9093/api/v2/alerts | jq .[] | {name: .labels.alertname, status: .status.state} # 恢复 clush -w 192.168.1.101 sudo systemctl start node_exporter6.4 Grafana导入Dashboardbash# 推荐导入的Dashboard ID # 8919 - Node Exporter Server Metrics # 11074 - Node Exporter 1-Panel # 1860 - Node Exporter Full # 或者创建自定义Dashboard使用以下PromQLpromql# 自定义分组面板 # 按环境分组CPU使用率 avg by (env) (100 - (avg by (env, instance) (rate(node_cpu_seconds_total{modeidle}[5m])) * 100)) # 按角色分组内存使用 sum by (role) (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / 1024 / 1024 / 1024 # 按业务线分组节点数量 count by (business) (up) # 按团队分组磁盘总量 sum by (team) (node_filesystem_size_bytes{fstype!tmpfs}) / 1024 / 1024 / 1024 / 1024 常见问题解答Q1: Node Exporter启动方式选择A:systemd方式适合生产环境稳定可靠有完善的日志和重启策略Docker方式适合测试环境或容器化部署但需要注意网络模式设置Q2: 克隆机器是否有限制A: Node Exporter本身没有client ID限制。只要确保每台机器的主机名唯一用于生成Service IDIP地址唯一注册到Consul时使用唯一的Service ID我们使用node-exporter-$(hostname)确保唯一性Q3: 如何实现按组查询A: 通过Consul注册时的标签Tags和元数据MetaPrometheus会自动将这些信息转换为标签然后使用PromQL的by子句进行分组查询 promql# 按环境分组 avg by (env) (rate(node_cpu_seconds_total[5m])) # 多标签分组 sum by (env, role) (node_memory_MemTotal_bytes)Q4: 告警规则如何按分组路由A: 在Alertmanager配置中可以根据告警的标签进行路由 yamlroutes: - match: env: prod severity: critical receiver: prod-criticalQ5: 如何测试告警是否正常工作A: 可以手动停止一个Node Exporter服务查看Prometheus Alerts页面确认告警状态变为PENDING→FIRING查看Alertmanager接收到的告警验证通知渠道是否收到消息 总结已完成的功能阶段功能状态第一阶段Consul集群、Prometheus、Grafana、Alertmanager基础环境✅第二阶段3台测试节点部署验证动态发现✅第三阶段Prometheus分组配置按环境/角色/业务线分组✅第四阶段告警规则配置按分组路由✅第五阶段扩展到全量节点分组模板✅第六阶段验证与测试自定义Grafana查询✅可扩展的方向多集群支持配置多个Consul datacenter自定义ExporterMySQL、Nginx、Redis等应用监控日志集成添加Loki实现日志与指标联动自动化运维通过Consul API实现自动扩缩容时的服务注册/注销
