The configuration of Hadoop HDFS in Heron Cluster

前序配置详情

HDFS配置

Master:heron01

1. 下载Hadoop

Hadoop下载地址:https://archive.apache.org/dist/hadoop/common/,这里下载的版本为2.7.4版本

2. 解压配置

解压至/home/yitian/hadoop目录下:

yitian@ubuntu:~/hadoop$ ll
total 260452
drwxrwxr-x  3 yitian yitian      4096 Feb 13 00:35 ./
drwxr-xr-x 31 yitian yitian      4096 Feb 13 00:34 ../
drwxr-xr-x 10 yitian yitian      4096 Jul 31  2017 hadoop-2.7.4/
-rwxrw-rw-  1 yitian yitian 266688029 Feb 13 00:33 hadoop-2.7.4.tar.gz*

3. 修改配置文件(目录:/home/yitian/hadoop/hadoop-2.7.4/etc/hadoop)

  • slaves
  • core-site.xml
  • hdfs-site.xml
  • hadoop-env.sh

(1) slaves:

heron02
heron03

(2) core-site.xml:

<configuration>
<property>
         <name>hadoop.tmp.dir</name>
         <value>file:/home/yitian/hadoop/hadoop-2.7.4/tmp</value>
<description>Abase for other temporary directories.</description>
</property>
<property>
         <name>fs.defaultFS</name>
         <value>hdfs://heron01:9000</value>
</property>
</configuration>

(3) hdfs-site.xml:

<configuration>
<property>
         <name>dfs.namenode.secondary.http-address</name>
         <value>heron01:50090</value>
</property>
         <property>
         <name>dfs.replication</name>
         <value>1</value>
</property>
<property>
         <name>dfs.namenode.name.dir</name>
         <value>file:/home/yitian/hadoop/hadoop-2.7.4/tmp/dfs/name</value>
</property>
<property>
         <name>dfs.datanode.data.dir</name>
         <value>file:/home/yitian/hadoop/hadoop-2.7.4/tmp/dfs/data</value>
</property>
</configuration>

(4) hadoop-env.sh

# The java implementation to use.
# export JAVA_HOME=${JAVA_HOME}
export JAVA_HOME=/usr/java/jdk1.8.0_151

4. 打包并分发

将heron01中的/home/yitian/hadoop目录打包,复制到heron02主机中,解压缩至相同目录(/home/yitian/)。

yitian@ubuntu:~$ tar -zcf ./hadoop.tar.gz ./hadoop

5. 配置HADOOP_HOME环境变量

# Hadoop configuration
export HADOOP_HOME=/home/yitian/hadoop/hadoop-2.7.4
export PATH=${HADOOP_HOME}/bin:$PATH

5. 在所有主机中初始化HDFS

yitian@ubuntu:~/.heron/conf/aurora$ cd /home/yitian/hadoop/hadoop-2.7.4/
yitian@ubuntu:~/hadoop/hadoop-2.7.4$ bin/hdfs namenode -format

6. 在heron01中启动HDFS

在/home/yitian/hadoop/hadoop-2.7.4目录下运行该命令:

yitian@ubuntu:~/hadoop/hadoop-2.7.4$ ./sbin/start-dfs.sh

7. 检查运行情况

(1)heron01:

yitian@ubuntu:~/hadoop/hadoop-2.7.4$ jps
14691 SecondaryNameNode
14822 Jps
14476 NameNode

(2)heron02和heron03:

yitian@heron03:~/hadoop/hadoop-2.7.4/etc/hadoop$ jps
5401 DataNode
5481 Jps
2204 QuorumPeerMain

(3)在heron01主机中访问:heron01:50070

image_thumb51

image_thumb61