[Spark] conf/spark-env.sh文件的配置对Spark集群的影响

初次配置spark集群是,由于spark-env.sh配置文件中可以设置的参数很多,而且对standalone、hadoop yarn等集群管理器模式不是很理解。通过实验的方式,发现spark-env.sh配置文件中的参数设置对spark集群的影响。

并探究,spark和hadoop集群相互间的关联和运行情况。

实验一:可以使用local和standlone模式,但无法使用Hadoop YARN模式

1. spark-env.sh配置文件内容如下:

image

注意:这里没有设置和hadoop有关的内容。

2. 在没有启动hadoop集群的情况下,启动Spark集群,并通过jps命令查看master和slave的运行情况:

master主机:

image

slave主机:

image

3. 通过spark的web管理器,查看spark集群的运行情况:

master:master:8080

image

4. 使用spark-submit命令,使用local(单机模式)提交spark-example任务

命令:

cd /usr/spark/spark-1.6.2-bin-hadoop2.6
./bin/run-example SparkPi 10 --master local[2]

运行结果:成功运行,并得到例子中需要计算的Pi值如下。

image

5. 使用spark-submit命令,使用standlone(spark独立集群管理器)提交spark-exmaple任务

命令:

cd /usr/spark/spark-1.6.2-bin-hadoop2.6

./bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://master:7077 ./lib/spark-examples-1.6.2-hadoop2.6.0.jar 

运行结果:同样可以运行成功,输出Pi的值如local模式相同。

通过web页面查看此时spark集群的运行情况:

image

6. 尝试使用Hadoop YARN集群管理器,提交spark-example任务

命令:

cd /usr/spark/spark-1.6.2-bin-hadoop2.6

./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster ./lib/spark-examples-1.6.2-hadoop2.6.0.jar 

运行结果:

image

出现错误。

实验结论:

1. 在spark-env.sh配置文件中,不配置关于hadoop的相关参数时,再启动spark集群后,hadoop集群不会同时启动。

2. 此时,可以使用local(本地模式)和standlone(spark独立集群管理模式),使用spark-submit命令提交任务并运行。但不能使用Hadoop YARN和Apache Mesos模式(这里没有涉及到,但仍需要单独配置)提交任务。

实验二:配置可以使用Hadoop YARN模式下的Spark集群

在master主机中停止上面运行的spark集群,并重新配置spark-env.sh文件。

image

1. spark-env.sh配置文件内容如下:

image

注意:这里的spark-env.sh配置文件中,加入了hadoop的相关参数,其他的基本不变。

2. 仍然在hadoop集群没有启动的情况下,启动spark集群:

master主机:

image

slave主机:

image

可见,无论在spark-env.sh配置文件里有没有添加hadoop集群的相关参数,在不启动hadoop集群的情况下启动spark集群,不会自动的启动hadoop集群。

此时,继续在hadoop集群未启动的情况下,使用Hadoop YARN模式提交spark-submit任务:

命令:

cd /usr/spark/spark-1.6.2-bin-hadoop2.6

./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster ./lib/spark-examples-1.6.2-hadoop2.6.0.jar 

执行命令后,出现异常:

image

错误信息:无法连接到ResouceManager。通过查找资料,并比对hadoop集群的配置信息,在hadoop集群没有配置异常的条件下,应该是hadoop集群没有启动导致的。

3. 启动hadoop集群

在master主机下,开启hadoop集群:

/usr/hadoop/hadoop-2.6.4/sbin/start-dfs.sh 
/usr/hadoop/hadoop-2.6.4/sbin/start-yarn.sh 

image

注意:在之前配置hadoop集群时,需要使用yarn方式,也就是需要修改hadoop配置文件夹下的yarn-site.xml中的内容。具体的参见[Hadoop] Linux(CentOS6.4)下Hadoop单机/集群的安装和配置,这里的hadoop集群配置就是使用的yarn方式

开启后,重新使用jps命令,查看hadoop和spark集群运行情况:

master主机:

image

slave主机:

image

4. 同时开启hadoop集群和spark集群后,重新使用Hadoop YARN模式提交spark-submit任务:

命令同上。

完整的输出结果如下:

[hadoop@master spark-1.6.2-bin-hadoop2.6]$ ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster ./lib/spark-examples-1.6.2-hadoop2.6.0.jar 
17/08/05 03:34:19 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/08/05 03:34:20 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.15.129:8032
17/08/05 03:34:20 INFO yarn.Client: Requesting a new application from cluster with 1 NodeManagers
17/08/05 03:34:20 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
17/08/05 03:34:20 INFO yarn.Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
17/08/05 03:34:20 INFO yarn.Client: Setting up container launch context for our AM
17/08/05 03:34:20 INFO yarn.Client: Setting up the launch environment for our AM container
17/08/05 03:34:21 INFO yarn.Client: Preparing resources for our AM container
Java HotSpot(TM) Client VM warning: You have loaded library /tmp/libnetty-transport-native-epoll9152778255334002025.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
17/08/05 03:34:22 INFO yarn.Client: Uploading resource file:/usr/spark/spark-1.6.2-bin-hadoop2.6/lib/spark-assembly-1.6.2-hadoop2.6.0.jar -> hdfs://master:9000/user/hadoop/.sparkStaging/application_1501929207954_0001/spark-assembly-1.6.2-hadoop2.6.0.jar
17/08/05 03:35:15 INFO yarn.Client: Uploading resource file:/usr/spark/spark-1.6.2-bin-hadoop2.6/lib/spark-examples-1.6.2-hadoop2.6.0.jar -> hdfs://master:9000/user/hadoop/.sparkStaging/application_1501929207954_0001/spark-examples-1.6.2-hadoop2.6.0.jar
17/08/05 03:35:58 INFO yarn.Client: Uploading resource file:/tmp/spark-f4f2a92c-af9b-4560-beb2-e56aaa525503/__spark_conf__8723657871756824681.zip -> hdfs://master:9000/user/hadoop/.sparkStaging/application_1501929207954_0001/__spark_conf__8723657871756824681.zip
17/08/05 03:35:58 INFO spark.SecurityManager: Changing view acls to: hadoop
17/08/05 03:35:58 INFO spark.SecurityManager: Changing modify acls to: hadoop
17/08/05 03:35:58 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); users with modify permissions: Set(hadoop)
17/08/05 03:35:59 INFO yarn.Client: Submitting application 1 to ResourceManager
17/08/05 03:35:59 INFO impl.YarnClientImpl: Submitted application application_1501929207954_0001
17/08/05 03:36:00 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:00 INFO yarn.Client: 
	 client token: N/A
	 diagnostics: N/A
	 ApplicationMaster host: N/A
	 ApplicationMaster RPC port: -1
	 queue: default
	 start time: 1501929359399
	 final status: UNDEFINED
	 tracking URL: http://master:8088/proxy/application_1501929207954_0001/
	 user: hadoop
17/08/05 03:36:01 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:02 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:03 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:04 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:05 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:06 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:07 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:08 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:09 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:10 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:11 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:12 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:13 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:14 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:15 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:16 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:17 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:18 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:19 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:20 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:21 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:22 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:23 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:24 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:25 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:26 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:27 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:28 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:29 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:30 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:31 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:32 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:33 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:34 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:36 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:37 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:38 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:39 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:40 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:41 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:42 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:43 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:44 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:45 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:46 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:47 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:48 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:49 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:50 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:51 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:52 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:53 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:54 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:55 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:56 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:57 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:58 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:36:59 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:37:00 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:37:01 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:37:02 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:37:03 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:37:04 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:37:05 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:37:06 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:37:07 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:37:08 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:37:09 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:37:10 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:37:11 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:37:12 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:37:13 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:37:14 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:37:15 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:37:16 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:37:17 INFO yarn.Client: Application report for application_1501929207954_0001 (state: ACCEPTED)
17/08/05 03:37:18 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:18 INFO yarn.Client: 
	 client token: N/A
	 diagnostics: N/A
	 ApplicationMaster host: 192.168.15.139
	 ApplicationMaster RPC port: 0
	 queue: default
	 start time: 1501929359399
	 final status: UNDEFINED
	 tracking URL: http://master:8088/proxy/application_1501929207954_0001/
	 user: hadoop
17/08/05 03:37:19 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:20 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:21 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:22 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:23 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:24 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:25 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:26 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:27 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:28 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:29 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:30 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:31 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:32 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:33 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:34 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:35 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:36 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:37 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:38 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:39 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:40 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:41 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:42 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:43 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:44 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:45 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:46 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:47 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:48 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:49 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:50 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:51 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:52 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:53 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:54 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:55 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:56 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:57 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:58 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:37:59 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:00 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:01 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:02 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:03 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:04 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:05 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:06 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:07 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:08 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:09 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:10 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:11 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:12 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:13 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:14 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:15 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:16 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:17 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:18 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:19 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:20 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:21 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:22 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:23 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:24 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:25 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:26 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:27 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:28 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:29 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:30 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:31 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:32 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:33 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:34 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:35 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:36 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:37 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:38 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:39 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:40 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:41 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:42 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:43 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:44 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:45 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:46 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:47 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:48 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:49 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:50 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:51 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:52 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:53 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:54 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:55 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:56 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:57 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:58 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:38:59 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:39:00 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:39:01 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:39:02 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:39:03 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:39:04 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:39:05 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:39:06 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:39:07 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:39:08 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:39:09 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:39:10 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:39:11 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:39:12 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:39:13 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:39:14 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:39:15 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:39:16 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:39:17 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:39:18 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:39:19 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:39:20 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:39:21 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:39:22 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:39:23 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:39:24 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:39:25 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:39:27 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:39:28 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:39:29 INFO yarn.Client: Application report for application_1501929207954_0001 (state: RUNNING)
17/08/05 03:39:30 INFO yarn.Client: Application report for application_1501929207954_0001 (state: FINISHED)
17/08/05 03:39:30 INFO yarn.Client: 
	 client token: N/A
	 diagnostics: N/A
	 ApplicationMaster host: 192.168.15.139
	 ApplicationMaster RPC port: 0
	 queue: default
	 start time: 1501929359399
	 final status: SUCCEEDED
	 tracking URL: http://master:8088/proxy/application_1501929207954_0001/
	 user: hadoop
17/08/05 03:39:30 INFO yarn.Client: Deleting staging directory .sparkStaging/application_1501929207954_0001
17/08/05 03:39:30 INFO util.ShutdownHookManager: Shutdown hook called
17/08/05 03:39:30 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-f4f2a92c-af9b-4560-beb2-e56aaa525503
[hadoop@master spark-1.6.2-bin-hadoop2.6]$ 

命令的执行没有错误,但因为使用的是yarn-cluster模式,结果不会输出到控制台(主要包括yarn-Clientyarn-Cluster两种模式,yarn-client模式将会在控制台中输出计算结果)。

注意到上述的:http://master:8088/cluster/app/application_1501929207954_0001,在master主机上的浏览器中打开,可以看到hadoop集群的运行情况:

image

并可以在日志(logs)中查看到spark-example的运行结果输出:

image

实验结论:

1. 无论在spark-env.sh配置文件中,加不加入hadoop集群的配置参数,hadoop集群都不会在spark集群开启时,自动开启。

2. 在需要时用Hadoop YARN集群管理器模式下,提交spark-submit任务时,需要同时配置并开启hadoop集群和spark集群

参考资料: