1. 环境准备与阿里云ECS基础配置在阿里云ECS上搭建Hadoop伪分布式集群前需要先完成基础环境配置。我推荐选择CentOS 7.9或Alibaba Cloud Linux 3作为操作系统镜像这两种系统对Hadoop的兼容性较好。购买ECS实例时建议配置至少2核4GB内存学习场景可用1核2GB但运行效率较低系统盘选择40GB以上。注意购买ECS后需在安全组中放行以下端口9870HDFS NameNode Web UI8088YARN ResourceManager50070HDFS旧版Web UI22SSH连接连接ECS后首先更新系统sudo yum update -y sudo yum install -y vim wget net-tools2. JDK安装与避坑指南Hadoop 3.x需要JDK 8或11实测华为镜像源下载速度最快wget --no-check-certificate https://repo.huaweicloud.com/java/jdk/8u202-b08/jdk-8u202-linux-x64.tar.gz tar -zxvf jdk-8u202-linux-x64.tar.gz sudo mv jdk1.8.0_202 /usr/local/java配置环境变量时容易犯的错是忘记sourcesudo tee -a /etc/profile EOF export JAVA_HOME/usr/local/java export PATH\$PATH:\$JAVA_HOME/bin EOF source /etc/profile # 立即生效验证安装时若报错bash: java: command not found通常是环境变量未生效可尝试hash -r清除缓存。3. Hadoop安装与核心配置3.1 软件包获取与解压推荐使用华为镜像源下载Hadoop 3.3.6截至2024年最新稳定版wget https://repo.huaweicloud.com/apache/hadoop/common/hadoop-3.3.6/hadoop-3.3.6.tar.gz tar -zxvf hadoop-3.3.6.tar.gz -C /opt/ sudo mv /opt/hadoop-3.3.6 /opt/hadoop3.2 环境变量配置关键点在于同时配置bin和sbin目录sudo tee -a /etc/profile EOF export HADOOP_HOME/opt/hadoop export HADOOP_CONF_DIR\$HADOOP_HOME/etc/hadoop export PATH\$PATH:\$HADOOP_HOME/bin:\$HADOOP_HOME/sbin EOF source /etc/profile3.3 配置文件修改需要修改5个核心文件1. hadoop-env.sh定位到export JAVA_HOME修改为export JAVA_HOME/usr/local/java export HADOOP_LOG_DIR/opt/hadoop/logs2. core-site.xml在configuration节点内添加property namefs.defaultFS/name valuehdfs://localhost:9000/value /property property namehadoop.tmp.dir/name value/opt/hadoop/tmp/value /property3. hdfs-site.xml伪分布式需设置副本数为1property namedfs.replication/name value1/value /property property namedfs.namenode.name.dir/name value/opt/hadoop/data/namenode/value /property property namedfs.datanode.data.dir/name value/opt/hadoop/data/datanode/value /property4. SSH免密登录配置Hadoop需要通过SSH管理节点配置步骤如下ssh-keygen -t rsa -P -f ~/.ssh/id_rsa cat ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys chmod 600 ~/.ssh/authorized_keys测试免密登录ssh localhost若仍需密码检查以下权限chmod 700 ~/.ssh restorecon -Rv ~/.ssh # SELinux环境需执行5. 启动集群与排错实战5.1 格式化HDFS首次启动必须格式化hdfs namenode -format常见报错解决Permission denied在hadoop-env.sh中添加export HADOOP_USER_NAMErootJava堆内存不足修改hadoop-env.sh中的HADOOP_HEAPSIZE_MAX5.2 启动服务分步启动更易排查问题start-dfs.sh start-yarn.sh检查进程是否齐全jps正常应显示6个进程NameNode DataNode SecondaryNameNode ResourceManager NodeManager Jps5.3 常见问题排查问题1DataNode未启动检查日志tail -n 100 /opt/hadoop/logs/hadoop-root-datanode-*.log常见原因是clusterID不一致解决方法# 将namenode的clusterID复制到datanode vim /opt/hadoop/data/namenode/current/VERSION vim /opt/hadoop/data/datanode/current/VERSION问题28088端口无法访问检查YARN日志tail -n 100 /opt/hadoop/logs/yarn-root-resourcemanager-*.log可能是内存配置问题修改yarn-site.xmlproperty nameyarn.nodemanager.resource.memory-mb/name value2048/value /property6. 验证与监控通过Web UI验证服务HDFS:http://ECS公网IP:9870YARN:http://ECS公网IP:8088创建测试目录验证HDFShdfs dfs -mkdir /test hdfs dfs -put /etc/hosts /test hdfs dfs -ls /test运行MapReduce测试任务hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar pi 2 5
阿里云ECS从零搭建Hadoop伪分布式集群：一站式配置与排错指南
1. 环境准备与阿里云ECS基础配置在阿里云ECS上搭建Hadoop伪分布式集群前需要先完成基础环境配置。我推荐选择CentOS 7.9或Alibaba Cloud Linux 3作为操作系统镜像这两种系统对Hadoop的兼容性较好。购买ECS实例时建议配置至少2核4GB内存学习场景可用1核2GB但运行效率较低系统盘选择40GB以上。注意购买ECS后需在安全组中放行以下端口9870HDFS NameNode Web UI8088YARN ResourceManager50070HDFS旧版Web UI22SSH连接连接ECS后首先更新系统sudo yum update -y sudo yum install -y vim wget net-tools2. JDK安装与避坑指南Hadoop 3.x需要JDK 8或11实测华为镜像源下载速度最快wget --no-check-certificate https://repo.huaweicloud.com/java/jdk/8u202-b08/jdk-8u202-linux-x64.tar.gz tar -zxvf jdk-8u202-linux-x64.tar.gz sudo mv jdk1.8.0_202 /usr/local/java配置环境变量时容易犯的错是忘记sourcesudo tee -a /etc/profile EOF export JAVA_HOME/usr/local/java export PATH\$PATH:\$JAVA_HOME/bin EOF source /etc/profile # 立即生效验证安装时若报错bash: java: command not found通常是环境变量未生效可尝试hash -r清除缓存。3. Hadoop安装与核心配置3.1 软件包获取与解压推荐使用华为镜像源下载Hadoop 3.3.6截至2024年最新稳定版wget https://repo.huaweicloud.com/apache/hadoop/common/hadoop-3.3.6/hadoop-3.3.6.tar.gz tar -zxvf hadoop-3.3.6.tar.gz -C /opt/ sudo mv /opt/hadoop-3.3.6 /opt/hadoop3.2 环境变量配置关键点在于同时配置bin和sbin目录sudo tee -a /etc/profile EOF export HADOOP_HOME/opt/hadoop export HADOOP_CONF_DIR\$HADOOP_HOME/etc/hadoop export PATH\$PATH:\$HADOOP_HOME/bin:\$HADOOP_HOME/sbin EOF source /etc/profile3.3 配置文件修改需要修改5个核心文件1. hadoop-env.sh定位到export JAVA_HOME修改为export JAVA_HOME/usr/local/java export HADOOP_LOG_DIR/opt/hadoop/logs2. core-site.xml在configuration节点内添加property namefs.defaultFS/name valuehdfs://localhost:9000/value /property property namehadoop.tmp.dir/name value/opt/hadoop/tmp/value /property3. hdfs-site.xml伪分布式需设置副本数为1property namedfs.replication/name value1/value /property property namedfs.namenode.name.dir/name value/opt/hadoop/data/namenode/value /property property namedfs.datanode.data.dir/name value/opt/hadoop/data/datanode/value /property4. SSH免密登录配置Hadoop需要通过SSH管理节点配置步骤如下ssh-keygen -t rsa -P -f ~/.ssh/id_rsa cat ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys chmod 600 ~/.ssh/authorized_keys测试免密登录ssh localhost若仍需密码检查以下权限chmod 700 ~/.ssh restorecon -Rv ~/.ssh # SELinux环境需执行5. 启动集群与排错实战5.1 格式化HDFS首次启动必须格式化hdfs namenode -format常见报错解决Permission denied在hadoop-env.sh中添加export HADOOP_USER_NAMErootJava堆内存不足修改hadoop-env.sh中的HADOOP_HEAPSIZE_MAX5.2 启动服务分步启动更易排查问题start-dfs.sh start-yarn.sh检查进程是否齐全jps正常应显示6个进程NameNode DataNode SecondaryNameNode ResourceManager NodeManager Jps5.3 常见问题排查问题1DataNode未启动检查日志tail -n 100 /opt/hadoop/logs/hadoop-root-datanode-*.log常见原因是clusterID不一致解决方法# 将namenode的clusterID复制到datanode vim /opt/hadoop/data/namenode/current/VERSION vim /opt/hadoop/data/datanode/current/VERSION问题28088端口无法访问检查YARN日志tail -n 100 /opt/hadoop/logs/yarn-root-resourcemanager-*.log可能是内存配置问题修改yarn-site.xmlproperty nameyarn.nodemanager.resource.memory-mb/name value2048/value /property6. 验证与监控通过Web UI验证服务HDFS:http://ECS公网IP:9870YARN:http://ECS公网IP:8088创建测试目录验证HDFShdfs dfs -mkdir /test hdfs dfs -put /etc/hosts /test hdfs dfs -ls /test运行MapReduce测试任务hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar pi 2 5
