Hadoop集群部署方案目录1.网络拓扑 (1)2.软件安装 (1)2.1.修改主机名 (1)2.2.修改host文件 (1)2.3.创建Hadoop 用户 (2)2.4.禁用防火墙 (2)2.5.设置ssh登录免密码 (2)2.6.安装hadoop (4)3.集群配置 (5)3.1.修改脚本 (5)3.1.1................................................ hadoop-env.sh53.1.2................................................... y arn-env.sh53.2.配置文件 (5)3.2.1................................................ core-site.xml53.2.2................................................ hdfs-site.xml73.2.3.............................................. mapred-site.xml103.2.4................................................ yarn-site.xml113.2.5.配置datanode143.3.创建目录 (14)4.启动zk集群 (14)5.启动hadoop (14)5.1.启动所有节点journalnode (14)5.2.格式化h1 namenode (15)5.3.在h1上格式化ZK (15)5.4.启动h1的namenode，zkfc (16)5.5.启动h2上namenode (16)5.6.同步h1上的格式化数据到h2 (16)5.7.启动 HDFS (17)5.8.启动 YARN (18)5.9.启动h2 ResourceManager (18)5.10........................................ h4上启动 JobHistoryServer 195.11.......................................... 查看ResourceManager状态196.浏览器访问 (19)node管理界面 (19)6.1.1............................... http://192.168.121.167:50070196.1.2............................... http://192.168.121.168:50070206.2.ResourceManager管理界面 (20)6.2.1............................... http://192.168.121.167:8088/216.2.2............................... http://192.168.121.168:8088/216.3.JournalNode HTTP 服务 (21)6.3.1............................... http://192.168.121.167:8480/216.4.Datanode HTTP服务 (22)6.4.1.............................. http://192.168.121.167:50075/226.5.jobhistory 管理界面 (22)6.5.1................... http://192.168.121.170:19888/jobhistory227.测试 (23)7.1.上传文件至hdfs (23)7.2.验证hdfs HA (23)7.3.验证yarn (24)1.网络拓扑2.软件安装2.1.修改主机名修改四台机器的主机名，h1，h2，h3，h4查看是否修改成功uname -a2.2.修改host文件修改三台机器/etc/hosts文件配置hadoop节点配置zk节点2.3.创建Hadoop 用户1)创建特定的Hadoop 用户帐号以区分Hadoop 和本机上的其他服务。

sudo groupadd hadoopuseradd hadooppasswd hadoopusermod -a -G hadoop hadoop2)给hadoop用户添加权限，打开/etc/sudoers文件：在root ALL=(ALL) ALL后面添加hadoop ALL=(ALL) ALL2.4.禁用防火墙chkconfig iptables off2.5.设置ssh登录免密码Hadoop 控制脚本(并非守护进程)依赖SSH 来执行针对整个集群的操作。

例如，Namenode是通过SSH（Secure Shell）来启动和停止各个datanode上的各种守护进程的为了支持无缝式工作， SSH 安装好之后，需要允许hadoop 用户无需键入密码即可登陆集群内的机器。

最简单的方法是每个机器创建一个无密码公钥/私钥对1)生成密钥id_dsa.pub为公钥，id_dsa为私钥2)将公钥文件复制成authorized_keys文件3)修改.ssh目录的权限以及authorized_keys 的权限sudo chmod 644 ~/.ssh/authorized_keyssudo chmod 700 ~/.ssh4)测试是否设置成功ssh h15)对h2、h3、h4这几个节点上的hadoop用户也配置ssh，配置过程和上述在h1上一样6)在h2、h3、h4这几个节点上都执行一次“cat ~/.ssh/id_rsa.pub | sshhadoop@h1 'cat >> ~/.ssh/authorized_keys'”命令，将这几个节点上的共钥id_ras.pub拷贝到h1中的authorized_keys文件中拷贝完成后，h1中的authorized_keys文件内容7)将h1中的authorized_keys文件分发到其他节点(h2、h3、h4)上，在hadoop1上，使用scp -r ~/.ssh/authorized_keys hadoop@h1:~/.ssh/ 命令分发8)测试2.6.安装hadoop计算md5值，看文件是否完整sudo tar -zxvf ./hadoop-2.7.3.tar.gz # 解压到/usr/local中sudo mv ./hadoop-2.7.3/ ./hadoop # 将文件夹名改为hadoopsudo chown -R hadoop ./hadoop为了方便，使用hadoop命令或者start-all.sh等命令，修改Master上/etc/profile 新增export HADOOP_HOME=/usr/local/hadoopexport PATH=$PATH:$HADOOP_HOME/binsource /etc/profile使生效3.集群配置3.1.修改脚本3.1.1.hadoop-env.sh$vi hadoop-env.shexport JAVA_HOME=/usr/java/jdk1.8.0_403.1.2.yarn-env.sh$vi yarn-env.shexport JAVA_HOME=/usr/java/jdk1.8.0_403.1.3.hadoop-daemon.sh修改if [ "$HADOOP_PID_DIR" = "" ]; thenHADOOP_PID_DIR=/home/hadoop/tmpfi3.2.配置文件3.2.1.core-site.xml<configuration><property><name>hadoop.tmp.dir</name><value>file:/usr/local/hadoop/tmp</value></property><property><name>fs.defaultFS</name><value>hdfs://cluster</value></property><property><name>fs.trash.interval</name><value>4320</value></property><property><name>fs.trash.checkpoint.interval</name><value>60</value></property><property><name>dfs.ha.fencing.methods</name><value>sshfence</value></property><property><name>dfs.ha.fencing.ssh.private-key-files</name> <value>/home/hadoop/.ssh/id_rsa</value></property><property><name>dfs.ha.fencing.ssh.connect-timeout</name> <value>30000</value><property><name>ha.zookeeper.quorum</name><value>node1:2181,node2:2181,node3:2181</value> </property><property><name>hadoop.security.authorization</name><value>false</value></property></configuration>3.2.2.hdfs-site.xml<configuration><property><name>services</name><value>cluster</value></property><property><name>nodes.cluster</name><value>nn1,nn2</value></property><property><name>node.rpc-address.cluster.nn1</name> <value>h1:8020</value></property><property><name>node.rpc-address.cluster.nn2</name> <value>h2:8020</value><property><name>node.http-address.cluster.nn1</name><value>h1:50070</value></property><property><name>node.http-address.cluster.nn2</name><value>h2:50070</value></property><property><name>node.servicerpc-address.cluster.nn1</name><value>h1:53310</value></property><property><name>node.servicerpc-address.cluster.nn2</name><value>h2:53310</value></property><property><name>node.shared.edits.dir</name><value>qjournal://h1:8485;h2:8485;h3:8485/cluster</value> </property><property><name>dfs.ha.automatic-failover.enabled.cluster</name><value>true</value></property><property><name>dfs.client.failover.proxy.provider.cluster</name><value>node.ha.ConfiguredFailoverP roxyProvider</value></property><property><name>dfs.webhdfs.enabled</name><value>true</value></property><property><name>dfs.permissions.enabled</name><value>false</value></property><property><name>dfs.permissions</name><value>false</value></property><property><name>dfs.replication</name><value>2</value></property><property><name>.dir</name><value>file:/usr/local/hadoop/dfs/name</value></property><property><name>dfs.datanode.data.dir</name><value>file:/usr/local/hadoop/dfs/data</value><property><name>dfs.journalnode.edits.dir</name><value>/usr/local/hadoop/dfs/journalnode</value></property></configuration>3.2.3.mapred-site.xml<configuration><property><name></name><value>yarn</value></property><property><name>mapreduce.jobhistory.address</name><value>h4:10020</value></property><property><name>mapreduce.jobhistory.webapp.address</name><value>h4:19888</value></property><property><name>mapred.job.reuse.jvm.num.tasks</name><value>-1</value></property><property><name>mapreduce.reduce.shuffle.parallelcopies</name> <value>10</value></configuration>3.2.4.yarn-site.xml<configuration><!-- Node Manager Configs --><property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value></property><property><name>yarn.log-aggregation-enable</name><value>true</value></property><property><name>yarn.nodemanager.remote-app-log-dir</name><value>hdfs://cluster/var/log/hadoop-yarn/apps</value> </property><!-- Resource Manager Configs --><property><name>yarn.resourcemanager.connect.retry-interval.ms</name><value>2000</value></property><property><name>yarn.resourcemanager.ha.enabled</name><value>true</value><property><name>yarn.resourcemanager.ha.automatic-failover.enabled</name> <value>true</value></property><property><name>yarn.resourcemanager.ha.automatic-failover.embedded</name> <value>true</value></property><property><name>yarn.resourcemanager.cluster-id</name><value>cluster</value></property><property><name>yarn.resourcemanager.ha.rm-ids</name><value>rm1,rm2</value></property><property><name>yarn.resourcemanager.hostname.rm1</name><value>h1</value></property><property><name>yarn.resourcemanager.hostname.rm2</name><value>h2</value></property><name>yarn.resourcemanager.scheduler.class</name><value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair. FairScheduler</value></property><property><name>yarn.resourcemanager.recovery.enabled</name><value>true</value></property><property><name>yarn.resourcemanager.zk-address</name><value>node1:2181,node2:2181,node3:2181</value></property><property><name>yarn.resourcemanager.store.class</name><value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMSt ateStore</value></property><property><name>yarn.app.mapreduce.am.scheduler.connection.wait.interval-ms</n ame><value>5000</value></property></configuration>3.2.5.配置datanodevi slaves3.3.创建目录mkdir /usr/local/hadoop/dfs/datamkdir /usr/local/hadoop/dfs/namemkdir /usr/local/hadoop/dfs/journalnodemkdir /usr/local/hadoop/tmpmkdir /home/hadoop/tmp4.启动zk集群在zk节点（node1，node2，node3）执行：zkServer.sh start 三个节点都启动后查看状态，一个 leader 两个follower此时执行jps查看进程，启动了QuorumPeerMain5.启动hadoop5.1.启动所有节点journalnode(h1,h2,h3上执行)./hadoop-daemon.sh start journalnode5.2.格式化h1 namenodeh1上执行hdfs namenode -format成功的话，会看到“successfully formatted”和“Exitting with status 0”的提示5.3.在h1上格式化ZK仅第一次需要做，在ZooKeeper集群上建立HA的相应节点，用于保存集群中NameNode的状态数据在h1执行：hdfs zkfc -formatZK5.4.启动h1的namenode，zkfc执行：hadoop-daemon.sh start namenode，hadoop-daemon.sh start zkfc 此时查看进程，zkfc，namenode都启动了5.5.启动h2上namenode5.6.同步h1上的格式化数据到h2h2执行： hdfs namenode -bootstrapStandby5.7.启动 HDFS在h1执行：start-dfs.sh，一键启动所有hdfs进程，包括启动过的进程其他节点上的进程也都启动了也可以单独启动hadoop-daemon.sh start namenode hadoop-daemon.sh start datanode hadoop-daemon.sh start journalnode hadoop-daemon.sh start zkfc5.8.启动 YARN在h1执行：start-yarn.shh3上进程5.9.启动h2 ResourceManageryarn-daemon.sh start resourcemanager5.10.h4上启动 JobHistoryServer执行： mr-jobhistory-daemon.sh start historyserver5.11.查看ResourceManager状态./yarn rmadmin -getServiceState rm16.浏览器访问在windows中访问，某些连接打不开，需要配置host192.168.121.167 h1192.168.121.168 h2192.168.121.169 h3192.168.121.170 h4node管理界面6.1.1.http://192.168.121.167:500706.1.2.http://192.168.121.168:500706.2.ResourceManager管理界面用户可通过该地址在浏览器中查看集群各类信息6.2.1.http://192.168.121.167:8088/6.2.2.http://192.168.121.168:8088/ h2是standby namenode，会自动定向到h16.3.JournalNode HTTP 服务6.3.1.http://192.168.121.167:8480/http://192.168.121.168:8480/http://192.168.121.169:8480/6.4.Datanode HTTP服务6.4.1.http://192.168.121.167:50075/http://192.168.121.168:50075/http://192.168.121.169:50075/http://192.168.121.170:50075/6.5.jobhistory 管理界面6.5.1.http://192.168.121.170:19888/jobhistory7.测试7.1.上传文件至hdfs7.2.验证hdfs HAkill掉h1的namenode查看hadoop fs -ls /，文件还可以正常访问http://192.168.121.167:50070/已经无法访问了http://192.168.121.168:50070/显示变成了active启动kill掉的namenode，变成了standby7.3.验证yarn●添加测试文件●执行map reducehadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2 .7.3.jar wordcount /test /out查看结果。