Hadoop集群搭建(二)HDFSHDFS只是Hadoop最基本的一个服务，很多其他服务，都是基于HDFS 展开的。
所以部署一个HDFS集群，是很核心的一个动作，也是大数据平台的开始。
安装Hadoop集群，首先需要有Zookeeper才可以完成安装。
如果没有Zookeeper，请先部署一套Zookeeper。
另外，JDK以及物理主机的一些设置等。
都请参考下文：Hadoop集群搭建(一) Zookeeper下面开始HDFS的安装HDFS主机分配1.192.168.67.101 c6701 --Namenode+datanode2.192.168.67.102 c6702 --datanode3.192.168.67.103 c6703 --datanode1. 安装HDFS，解压hadoop-2.6.0-EDH-0u2.tar.gz我同时下载2.6和2.7版本的软件，先安装2.6，然后在执行2.6到2.7的升级步骤eradd hdfs2.echo "hdfs:hdfs"| chpasswd3.su - hdfs4.cd /tmp/software5.tar -zxvf hadoop-2.6.0-EDH-0u2.tar.gz -C /home/hdfs/6.mkdir -p /data/hadoop/temp7.mkdir -p /data/hadoop/journal8.mkdir -p /data/hadoop/hdfs/name9.mkdir -p /data/hadoop/hdfs/data10.chown -R hdfs:hdfs /data/hadoop11.chown -R hdfs:hdfs /data/hadoop/temp12.chown -R hdfs:hdfs /data/hadoop/journal13.chown -R hdfs:hdfs /data/hadoop/hdfs/name14.chown -R hdfs:hdfs /data/hadoop/hdfs/data15.$ pwd16./home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop2. 修改core-site.xml对应的参数1.$ cat core-site.xml2.<configuration>3.<!--指定hdfs的nameservice为ns -->4.<property>5.<name>fs.defaultFS</name>6.<value>hdfs://ns</value>7.</property>8.<!--指定hadoop数据临时存放目录-->9.<property>10.<name>hadoop.tmp.dir</name>11.<value>/data/hadoop/temp</value>12.</property>13.14.<property>15.<name>io.file.buffer.size</name>16.<value>4096</value>17.</property>18.<!--指定zookeeper地址-->19.<property>20.<name>ha.zookeeper.quorum</name>21.<value>c6701:2181,c6702:2181,c6703:2181</value>22.</property>23.</configuration>3. 修改hdfs-site.xml对应的参数1.cat hdfs-site.xml2.<configuration>3.<!--指定hdfs的nameservice为ns，需要和core-site.xml中的保持一致,并且ns如果改，整个文件中，全部的ns要都修改，保持统一-->4.<property>5.<name>services</name>6.<value>ns</value>7.</property>8.<!-- ns下面有两个NameNode，分别是nn1，nn2 -->9.<property>10.<name>nodes.ns</name>11.<value>nn1,nn2</value>12.</property>13.<!-- nn1的RPC通信地址-->14.<property>15.<name>node.rpc-address.ns.nn1</name>16.<value>c6701:9000</value>17.</property>18.<!-- nn1的http通信地址-->19.<property>20.<name>node.http-address.ns.nn1</name>21.<value>c6701:50070</value>22.</property>23.<!-- nn2的RPC通信地址-->24.<property>25.<name>node.rpc-address.ns.nn2</name>26.<value>c6702:9000</value>27.</property>28.<!-- nn2的http通信地址-->29.<property>30.<name>node.http-address.ns.nn2</name>31.<value>c6702:50070</value>32.</property>33.<!--指定NameNode的元数据在JournalNode上的存放位置-->34.<property>35.<name>node.shared.edits.dir</name>36.<value>qjournal://c6701:8485;c6702:8485;c6703:8485/ns</value>37.</property>38.<!--指定JournalNode在本地磁盘存放数据的位置-->39.<property>40.<name>dfs.journalnode.edits.dir</name>41.<value>/data/hadoop/journal</value>42.</property>43.<!--开启NameNode故障时自动切换-->44.<property>45.<name>dfs.ha.automatic-failover.enabled</name>46.<value>true</value>47.</property>48.<!--配置失败自动切换实现方式-->49.<property>50.<name>dfs.client.failover.proxy.provider.ns</name>51.<value>node.ha.ConfiguredFailoverProxyProvider</value>52.</property>53.<!--配置隔离机制-->54.<property>55.<name>dfs.ha.fencing.methods</name>56.<value>sshfence</value>57.</property>58.<!--使用隔离机制时需要ssh免登陆-->59.<property>60.<name>dfs.ha.fencing.ssh.private-key-files</name>61.<value>/home/hdfs/.ssh/id_rsa</value>62.</property>63.64.<property>65.<name>.dir</name>66.<value>/data/hadoop/hdfs/name</value>67.</property>68.69.<property>70.<name>dfs.datanode.data.dir</name>71.<value>/data/hadoop/hdfs/data</value>72.</property>73.74.<property>75.<name>dfs.replication</name>76.<value>2</value>77.</property>78.<!--在NN和DN上开启WebHDFS(REST API)功能,不是必须-->79.<property>80.<name>dfs.webhdfs.enabled</name>81.<value>true</value>82.</property>83.</configuration>4. 添加slaves文件1.$ more slaves2.c67013.c67024.c6703--- 安装C6702的hdfs---5. 创建c6702的用户，并为hdfs用户ssh免密1.ssh c6702 "useradd hdfs"2.ssh c6702 "echo "hdfs:hdfs" | chpasswd"3.ssh-copy-id hdfs@c67026. 拷贝软件1.scp -r /tmp/software/hadoop-2.6.0-EDH-0u2.tar.gz root@c6702:/tmp/software/.2.ssh c6702 "chmod 777 /tmp/software/*"7. 创建目录，解压软件1.ssh hdfs@c6702 "mkdir hdfs"2.ssh hdfs@c6702 "tar -zxvf /tmp/software/hadoop-2.6.0-EDH-0u2.tar.gz -C /home/hdfs"3.ssh hdfs@c6702 "ls -al hdfs"4.ssh hdfs@c6702 "ls -al hdfs/hadoop*"复制配置文件1.ssh hdfs@c6702 "rm -rf /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/core-site.xml"2.ssh hdfs@c6702 "rm -rf /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/hdfs-site.xml"3.scp -r /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/core-site.xml hdfs@c6702:/home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/core-site.xml4.scp -r /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/hdfs-site.xml hdfs@c6702:/home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/hdfs-site.xml5.scp -r /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/slaves hdfs@c6702:/home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/slaves创建hdfs需要的目录1.ssh root@c6702 "mkdir -p /data/hadoop"2.ssh root@c6702 " chown -R hdfs:hdfs /data/hadoop"3.ssh hdfs@c6702 "mkdir -p /data/hadoop/temp"4.ssh hdfs@c6702 "mkdir -p /data/hadoop/journal"5.ssh hdfs@c6702 "mkdir -p /data/hadoop/hdfs/name"6.ssh hdfs@c6702 "mkdir -p /data/hadoop/hdfs/data"--- 安装C6703的hdfs---8. 创建c6703的用户，并为hdfs用户ssh免密1.ssh c6703 "useradd hdfs"2.ssh c6703 "echo "hdfs:hdfs" | chpasswd"3.ssh-copy-id hdfs@c67039. 拷贝软件1.scp -r /tmp/software/hadoop-2.6.0-EDH-0u2.tar.gz root@c6703:/tmp/software/.2.ssh c6703 "chmod 777 /tmp/software/*"3.10.创建目录，解压软件4.ssh hdfs@c6703 "mkdir hdfs"5.ssh hdfs@c6703 "tar -zxvf /tmp/software/hadoop-2.6.0-EDH-0u2.tar.gz -C /home/hdfs"6.ssh hdfs@c6703 "ls -al hdfs"7.ssh hdfs@c6703 "ls -al hdfs/hadoop*"复制配置文件1.ssh hdfs@c6703 "rm -rf /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/core-site.xml"2.ssh hdfs@c6703 "rm -rf /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/hdfs-site.xml"3.scp -r /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/core-site.xml hdfs@c6703:/home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/core-site.xml4.scp -r /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/hdfs-site.xml hdfs@c6703:/home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/hdfs-site.xml5.scp -r /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/slaves hdfs@c6703:/home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/slaves创建hdfs需要的目录1.ssh root@c6703 "mkdir -p /data/hadoop"2.ssh root@c6703 " chown -R hdfs:hdfs /data/hadoop"3.ssh hdfs@c6703 "mkdir -p /data/hadoop/temp"4.ssh hdfs@c6703 "mkdir -p /data/hadoop/journal"5.ssh hdfs@c6703 "mkdir -p /data/hadoop/hdfs/name"6.ssh hdfs@c6703 "mkdir -p /data/hadoop/hdfs/data"11. 启动HDFS，先启动三个节点的journalnode/home/hdfs/hadoop-2.6.0-EDH-0u2/sbin/hadoop-daemon.sh start journalnode检查状态1.$ jps2.3958Jps3.3868JournalNode12. 然后启动namenode,首次启动namenode之前，先在其中一个节点(主节点)format namenode信息，信息会存在于.dir指定的路径中1.<name>.dir</name>2.<value>/data/hadoop/hdfs/name</value>1.$ ./hdfs namenode -format2.17/09/2607:52:17 INFO Node: STARTUP_MSG:3./************************************************************4.STARTUP_MSG: Starting NameNode5.STARTUP_MSG: host = /192.168.67.1016.STARTUP_MSG: args = [-format]7.STARTUP_MSG: version = 2.6.0-EDH-0u28.STARTUP_MSG: classpath = /home/hdfs/hadoop-2.6.0-EDHxxxxxxxxxx9.STARTUP_MSG: build = http://gitlab-xxxxx10.STARTUP_MSG: java = 1.8.0_14411.************************************************************/12.17/09/2607:52:17 INFO Node: registered UNIX signal handlers for[TERM, HUP, INT]13.17/09/2607:52:17 INFO Node: createNameNode [-format]14.17/09/2607:52:18 WARN common.Util:Path/data/hadoop/hdfs/name should be specified as a URI in configuration files.Please update hdfs configuration.15.17/09/2607:52:18 WARN common.Util:Path/data/hadoop/hdfs/name should be specified as a URI in configuration files.Please update hdfs configuration.16.Formatting using clusterid: CID-b2f01411-862f-44b2-a6dc-7d17bd48d52217.17/09/2607:52:18 INFO namenode.FSNamesystem:No KeyProvider found.18.17/09/2607:52:18 INFO namenode.FSNamesystem: fsLock is fair:true19.17/09/2607:52:18 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=100020.17/09/2607:52:18 INFO blockmanagement.DatanodeManager: node.datanode.registration.ip-hostname-check=true21.17/09/2607:52:18 INFO blockmanagement.BlockManager: node.startup.delay.block.deletion.sec is set to 000:00:00:00.000will start around 2017Sep2607:52:1823.17/09/2607:52:18 INFO util.GSet:Computing capacity for map BlocksMap24.17/09/2607:52:18 INFO util.GSet: VM type =64-bit25.17/09/2607:52:18 INFO util.GSet:2.0% max memory 966.7 MB =19.3 MB26.17/09/2607:52:18 INFO util.GSet: capacity =2^21=2097152 entries27.17/09/2607:52:18 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false28.17/09/2607:52:18 INFO blockmanagement.BlockManager: defaultReplication=229.17/09/2607:52:18 INFO blockmanagement.BlockManager: maxReplication=51230.17/09/2607:52:18 INFO blockmanagement.BlockManager: minReplication=131.17/09/2607:52:18 INFO blockmanagement.BlockManager: maxReplicationStreams =232.17/09/2607:52:18 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks =false33.17/09/2607:52:18 INFO blockmanagement.BlockManager: replicationRecheckInterval =300034.17/09/2607:52:18 INFO blockmanagement.BlockManager: encryptDataTransfer =falseLog =100036.17/09/2607:52:18 INFO namenode.FSNamesystem: fsOwner = hdfs (auth:SIMPLE)37.17/09/2607:52:18 INFO namenode.FSNamesystem: supergroup = supergroup38.17/09/2607:52:18 INFO namenode.FSNamesystem: isPermissionEnabled =true39.17/09/2607:52:18 INFO namenode.FSNamesystem:Determined nameserviceID: ns40.17/09/2607:52:18 INFO namenode.FSNamesystem: HA Enabled:true41.17/09/2607:52:18 INFO namenode.FSNamesystem:Append Enabled:true42.17/09/2607:52:18 INFO util.GSet:Computing capacity for map INodeMap43.17/09/2607:52:18 INFO util.GSet: VM type =64-bit44.17/09/2607:52:18 INFO util.GSet:1.0% max memory 966.7 MB =9.7 MB45.17/09/2607:52:18 INFO util.GSet: capacity =2^20=1048576 entries46.17/09/2607:52:18 INFO Node:Caching file names occuringmore than 10 times47.17/09/2607:52:18 INFO util.GSet:Computing capacity for map cachedBlocks48.17/09/2607:52:18 INFO util.GSet: VM type =64-bit49.17/09/2607:52:18 INFO util.GSet:0.25% max memory 966.7 MB =2.4 MB50.17/09/2607:52:18 INFO util.GSet: capacity =2^18=262144 entries51.17/09/2607:52:18 INFO namenode.FSNamesystem: node.safemode.threshold-pct =0.999000012874603352.17/09/2607:52:18 INFO namenode.FSNamesystem: node.safemode.min.datanodes =053.17/09/2607:52:18 INFO namenode.FSNamesystem: node.safemode.extension =3000054.17/09/2607:52:18 INFO namenode.FSNamesystem:Retry cache on namenode is enabled55.17/09/2607:52:18 INFO namenode.FSNamesystem:Retry cache will use0.03of total heap and retry cache entry expiry time is600000 millis56.17/09/2607:52:18 INFO util.GSet:Computing capacity for map NameNodeRetryCache57.17/09/2607:52:18 INFO util.GSet: VM type =64-bit58.17/09/2607:52:18 INFO util.GSet:0.029999999329447746% max memory 966.7 MB =297.0 KB59.17/09/2607:52:18 INFO util.GSet: capacity =2^15=32768 entries60.17/09/2607:52:18 INFO namenode.NNConf:ACLs enabled?false61.17/09/2607:52:18 INFO namenode.NNConf:XAttrs enabled?true62.17/09/2607:52:18 INFO namenode.NNConf:Maximum size of an xattr:1638463.17/09/2607:52:19 INFO namenode.FSImage:Allocated new BlockPoolId: BP-144216011-192.168.67.101-150641233975764.17/09/2607:52:19 INFO common.Storage:Storage directory /data/hadoop/hdfs/name has been successfully formatted.65.17/09/2607:52:20 INFO namenode.NNStorageRetentionManager:Going to retain 1 images with txid >=066.17/09/2607:52:20 INFO util.ExitUtil:Exiting with status 067.17/09/2607:52:20 INFO Node: SHUTDOWN_MSG:68./************************************************************69.SHUTDOWN_MSG: Shutting down NameNode at /192.168.67.10170.************************************************************/13. standby namenode需要先执行bootstrapstandby，输出如下1.[hdfs@c6702 sbin]$ ../bin/hdfs namenode -bootstrapstandby2.17/09/2609:44:58 INFO Node: STARTUP_MSG:3./************************************************************4.STARTUP_MSG: Starting NameNode5.STARTUP_MSG: host = /192.168.67.1026.STARTUP_MSG: args = [-bootstrapstandby]7.STARTUP_MSG: version = 2.6.0-EDH-0u28.STARTUP_MSG: classpath = /home/hdfs/haxxx9.STARTUP_MSG: build = http://gitlab-xxxx10.STARTUP_MSG: java = 1.8.0_14411.************************************************************/12.17/09/2609:44:58 INFO Node: registered UNIX signal handlers for[TERM, HUP, INT]13.17/09/2609:44:58 INFO Node: createNameNode [-bootstrapstandby]14.17/09/2609:44:59 WARN common.Util:Path/data/hadoop/hdfs/name should be specified as a URI in configuration files.Please update hdfs configuration.15.17/09/2609:44:59 WARN common.Util:Path/data/hadoop/hdfs/name should be specified as a URI in configuration files.Please update hdfs configuration.16.=====================================================17.About to bootstrap Standby ID nn2 from:service ID: ns19.Other Namenode ID: nn120.Other NN's HTTP address: http://c6701:5007021. Other NN's IPC address: c6701/192.168.67.101:9000space ID:79366220723.Block pool ID: BP-144216011-192.168.67.101-150641233975724.Cluster ID: CID-b2f01411-862f-44b2-a6dc-7d17bd48d522yout version:-6026.=====================================================27.Re-format filesystem in Storage Directory/data/hadoop/hdfs/name ?(Y or N) y28.17/09/2609:45:16 INFO common.Storage:Storage directory /data/hadoop/hdfs/name has been successfully formatted.29.17/09/2609:45:16 WARN common.Util:Path/data/hadoop/hdfs/name should be specified as a URI in configuration files.Please update hdfs configuration.30.17/09/2609:45:16 WARN common.Util:Path/data/hadoop/hdfs/name should be specified as a URI in configuration files.Please update hdfs configuration.31.17/09/2609:45:17 INFO namenode.TransferFsImage:Opening connection tohttp://c6701:50070/imagetransfer?getimage=1&txid=0&storageInf o=-60:793662207:0:CID-b2f01411-862f-44b2-a6dc-7d17bd48d52232.17/09/2609:45:17 INFO namenode.TransferFsImage:Image Transfer timeoutconfigured to 60000 milliseconds33.17/09/2609:45:17 INFO namenode.TransferFsImage:Transfer took 0.01s at 0.00 KB/s34.17/09/2609:45:17 INFO namenode.TransferFsImage:Downloaded file fsimage.ckpt_0000000000000000000 size 351 bytes.35.17/09/2609:45:17 INFO util.ExitUtil:Exiting with status 036.17/09/2609:45:17 INFO Node: SHUTDOWN_MSG:37./************************************************************38.SHUTDOWN_MSG: Shutting down NameNode at /192.168.67.10239.************************************************************/14. 检查状态，namenode还没有启动1.[hdfs@c6702 sbin]$ jps2.4539Jps3.3868JournalNode15. 启动standby namenode，命令和master启动的方式相同1.[hdfs@c6702 sbin]$ ./hadoop-daemon.sh start namenode2.starting namenode, logging to /home/hdfs/hadoop-2.6.0-EDH-0u2/logs/had.out16. 再次检查，namenode已经启动1.[hdfs@c6702 sbin]$ jps2.4640Jps3.4570NameNode4.3868JournalNode17. 格式化zkfc,让在zookeeper中生成ha节点，在master上执行如下命令，完成格式化1.[hdfs@c6701 bin]$ ./hdfs zkfc -formatZK2.17/09/2609:59:20 INFO tools.DFSZKFailoverController:Failover controller configured for NameNode NameNode at c6701/192.168.67.101:90003.17/09/2609:59:20 INFO zookeeper.ZooKeeper:Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/201409:09 GMT4.17/09/2609:59:20 INFO zookeeper.ZooKeeper:Client environment:=5.17/09/2609:59:20 INFO zookeeper.ZooKeeper:Client environment:java.version=1.8.0_1446.17/09/2609:59:20 INFO zookeeper.ZooKeeper:Client environment:java.vendor=Oracle Corporation7.17/09/2609:59:20 INFO zookeeper.ZooKeeper:Client environment:java.home=/usr/local/jdk1.8.0_144/jre8.17/09/2609:59:20 INFO zookeeper.ZooKeeper:Client environment:java.class.path=/home/hdfs/hadoop-2.6.0-EDH-0u2/exxxx9.17/09/2609:59:20 INFO zookeeper.ZooKeeper:Client environment:java.library.path=/home/hdfs/hadoop-2.6.0-EDH-0u2/lib/native10.17/09/2609:59:20 INFO zookeeper.ZooKeeper:Client environment:java.io.tmpdir=/tmp11.17/09/2609:59:20 INFO zookeeper.ZooKeeper:Client environment:piler=<NA>12.17/09/2609:59:20 INFO zookeeper.ZooKeeper:Client environment:=Linux13.17/09/2609:59:20 INFO zookeeper.ZooKeeper:Client environment:os.arch=amd6414.17/09/2609:59:20 INFO zookeeper.ZooKeeper:Client environment:os.version=2.6.32-573.el6.x86_6415.17/09/2609:59:20 INFO zookeeper.ZooKeeper:Client environment:=hdfs16.17/09/2609:59:20 INFO zookeeper.ZooKeeper:Client environment:user.home=/home/hdfs17.17/09/2609:59:20 INFO zookeeper.ZooKeeper:Client environment:user.dir=/home/hdfs/hadoop-2.6.0-EDH-0u2/bin18.17/09/2609:59:20 INFO zookeeper.ZooKeeper:Initiating client connection, connectString=c6701:2181,c6702:2181,c6703:2181 sessionTimeout=5000 watc her=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@20 deea7f19.17/09/2609:59:20 INFO zookeeper.ClientCnxn:Opening socket connection toserver /192.168.67.103:2181.Will not attempt to authen ticate using SASL (unknown error)20.17/09/2609:59:20 INFO zookeeper.ClientCnxn:Socket connection established to /192.168.67.103:2181, initiating session21.17/09/2609:59:20 INFO zookeeper.ClientCnxn:Session establishment complete on server /192.168.67.103:2181, sessionid =0x35ebc5 163710000, negotiated timeout =500022.17/09/2609:59:20 INFO ha.ActiveStandbyElector:Session connected.23.17/09/2609:59:20 INFO ha.ActiveStandbyElector:Successfully created /hadoop-ha/ns in ZK.24.17/09/2609:59:20 INFO zookeeper.ZooKeeper:Session:0x35ebc5163710000closed25.17/09/2609:59:20 INFO zookeeper.ClientCnxn:EventThread shut down18. 格式化完成的检查格式成功后，查看zookeeper中可以看到 <<<<<<<<<<<命令没确认1.[zk: localhost:2181(CONNECTED)1] ls /hadoop-ha19. 启动zkfc，这个就是为namenode使用的1../hadoop-daemon.sh start zkfc2.starting zkfc, logging to /home/hdfs/hadoop-2.6.0-EDH-0u2/logs/hadoop-hd.out3.$ jps4.4272DataNode5.4402JournalNode6.6339Jps7.6277DFSZKFailoverController8.4952NameNode20. 另一个节点启动zkfc，1.ssh hdfs@c67022./home/hdfs/hadoop-2.6.0-EDH-0u2/sbin/hadoop-daemon.sh start zkfc3.$ jps4.4981Jps5.4935DFSZKFailoverController6.4570NameNode7.3868JournalNode21. 注意：进行初始化的时候，必须保证zk集群已经启动了。