欢迎关注大数据技术架构与案例微信公众号:过往记忆大数据
过往记忆博客公众号iteblog_hadoop
欢迎关注微信公众号:
过往记忆大数据

在Yarn上运行Apache Zeppelin & Spark

  默认情况下,Apache Zeppelin启动Spark是以本地模式起的,master的值是local[*],我们可以通过修改conf/zeppelin-env.sh文件里面的MASTER的值如下:

export MASTER= yarn-client
export HADOOP_HOME=/home/q/hadoop/hadoop-2.2.0
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop/

然后启动Zeppelin,但是我们有时会发现日志出现了以下的异常信息:

ERROR [2016-01-20 18:28:35,016] ({pool-2-thread-2} Logging.scala[logError]:96) - Error initializing SparkContext.
org.apache.spark.SparkException: Unable to load YARN support
        at org.apache.spark.deploy.SparkHadoopUtil$.liftedTree1$1(SparkHadoopUtil.scala:389)
        at org.apache.spark.deploy.SparkHadoopUtil$.yarn$lzycompute(SparkHadoopUtil.scala:384)
        at org.apache.spark.deploy.SparkHadoopUtil$.yarn(SparkHadoopUtil.scala:384)
        at org.apache.spark.deploy.SparkHadoopUtil$.get(SparkHadoopUtil.scala:401)
        at org.apache.spark.util.Utils$.getSparkOrYarnConfig(Utils.scala:2049)
        at org.apache.spark.storage.BlockManager.(BlockManager.scala:97)
        at org.apache.spark.storage.BlockManager.(BlockManager.scala:173)
        at org.apache.spark.SparkEnv$.create(SparkEnv.scala:347)
        at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:194)
        at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:277)
        at org.apache.spark.SparkContext.(SparkContext.scala:450)
        at org.apache.zeppelin.spark.SparkInterpreter.createSparkContext(SparkInterpreter.java:343)
        at org.apache.zeppelin.spark.SparkInterpreter.getSparkContext(SparkInterpreter.java:145)
        at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:469)
        at org.apache.zeppelin.interpreter.ClassloaderInterpreter.open(ClassloaderInterpreter.java:74)
        at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:68)
        at org.apache.zeppelin.spark.SparkSqlInterpreter.getSparkInterpreter(SparkSqlInterpreter.java:100)
        at org.apache.zeppelin.spark.SparkSqlInterpreter.interpret(SparkSqlInterpreter.java:115)
        at org.apache.zeppelin.interpreter.ClassloaderInterpreter.interpret(ClassloaderInterpreter.java:57)
        at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:93)
        at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:302)
        at org.apache.zeppelin.scheduler.Job.run(Job.java:171)
        at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.deploy.yarn.YarnSparkHadoopUtil
        at scala.tools.nsc.interpreter.AbstractFileClassLoader.findClass(AbstractFileClassLoader.scala:83)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:270)
        at org.apache.spark.util.Utils$.classForName(Utils.scala:173)
        at org.apache.spark.deploy.SparkHadoopUtil$.liftedTree1$1(SparkHadoopUtil.scala:385)
        ... 29 more

导致这个异常的原因是我们在编译Apache Zeppelin没有加入对YARN的依赖,解决的办法是我们可以在编译Apache Zeppelin的时候加上-Pyarn来解决上面的问题:

[iteblog@www.iteblog.com iteblog]$ mvn package -Pspark-1.5  \
           -DskipTests -Dhadoop.version=2.2.0 -Phadoop-2.2 -Pyarn

然后再启动Apache Zeppelin

[iteblog@www.iteblog.com iteblog]$ bin/zeppelin-daemon.sh start

然后启动的Spark Job就会在YARN上跑。所有的日志可以在logs目录下找,在我的例子里面记录日志的文件是zeppelin-interpreter-spark-iteblog-www.iteblog.com.log

  上述的Hadoop为Apache发行版的,如果你运行的是hdp(Hortonworks Data Platform)类型的Hadoop,你除了设置MASTERHADOOP_CONF_DIR之外,还必须设置hdp发行版Hadoop的版本,如下:

export ZEPPELIN_JAVA_OPTS="-Dhdp.version=2.3.1.0-2574"

  关于hdp版本不知道如何获取可以运行下面命令得到:

hdp-select status hadoop-client | sed 's/hadoop-client - \(.*\)/\1/'
# It returned  2.3.1.0-2574

  另外需要注意的是,Apache Zeppelin & Spark在YARN模式上运行只支持yarn-client模式,如果你将MASTER设置成了yarn-cluster,那么将会出现异常:

INFO [2016-01-21 10:36:59,445] ({pool-1-thread-5} Logging.scala[logInfo]:59) - Running Spark version 1.5.2
ERROR [2016-01-21 10:36:59,447] ({pool-1-thread-5} Logging.scala[logError]:96) - Error initializing SparkContext.
org.apache.spark.SparkException: Detected yarn-cluster mode, 
but isn't running on a cluster. Deployment to YARN is not supported directly by SparkContext. Please use spark-submit.
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:404)
	at org.apache.zeppelin.spark.SparkInterpreter.createSparkContext(SparkInterpreter.java:343)
	at org.apache.zeppelin.spark.SparkInterpreter.getSparkContext(SparkInterpreter.java:145)
	at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:469)
	at org.apache.zeppelin.interpreter.ClassloaderInterpreter.open(ClassloaderInterpreter.java:74)
	at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:68)
	at org.apache.zeppelin.interpreter.LazyOpenInterpreter.getProgress(LazyOpenInterpreter.java:109)
	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.getProgress(RemoteInterpreterServer.java:335)
	at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getProgress.getResult(RemoteInterpreterService.java:1090)
	at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getProgress.getResult(RemoteInterpreterService.java:1075)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
本博客文章除特别声明,全部都是原创!
原创文章版权归过往记忆大数据(过往记忆)所有,未经许可不得转载。
本文链接: 【在Yarn上运行Apache Zeppelin & Spark】(https://www.iteblog.com/archives/1571.html)
喜欢 (12)
分享 (0)
发表我的评论
取消评论

表情
本博客评论系统带有自动识别垃圾评论功能,请写一些有意义的评论,谢谢!
(16)个小伙伴在吐槽
  1. 博主:你好,

    export MASTER=yarn-client 
     export SPARK_HOME=/home/hadoop/app/spark-1.6.2-bin-hadoop2.6/ 
    export HADOOP_HOME=/home/hadoop/app/hadoop-2.6.0-cdh5.8.2
    export HADOOP_CONF_DIR=/home/hadoop/app/hadoop-2.6.0-cdh5.8.2/etc/hadoop/ 
    


    设置好后,启动还是本地提交模式呢,在yarn的web界面没看到有任务提交?

    岁月无声2016-11-08 16:54 回复
  2. 你好,我的配置完之后报错信息如下,我不知道怎么解决,请教一下,希望可以得到回复,谢谢!

    ERROR [2016-08-02 11:54:23,176] ({pool-1-thread-5} TThreadPoolServer.java[run]:296) - Error occurred during processing of message.
    org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode=&quot;/user&quot;:hdfs:supergroup:drwxr-xr-x
    	at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257)
    	at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238)
    	at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:216)
    	at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:145)
    	at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138)
    	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6605)
    	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6587)
    	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:6539)
    	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:4329)
    	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:4299)
    	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:4272)
    	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:866)
    	at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.mkdirs(AuthorizationProviderProxyClientProtocol.java:321)
    	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:601)
    	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
    	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
    	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
    	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
    	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
    	at java.security.AccessController.doPrivileged(Native Method)
    	at javax.security.auth.Subject.doAs(Subject.java:422)
    	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
    	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)
    
    	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    	at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
    	at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
    	at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
    	at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3077)
    	at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:3042)
    	at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:956)
    	at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:952)
    	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    	at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:952)
    	at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:945)
    	at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1856)
    	at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:606)
    	at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:282)
    	at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:648)
    	at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:124)
    	at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:56)
    	at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:144)
    	at org.apache.spark.SparkContext.&lt;init&gt;(SparkContext.scala:523)
    	at org.apache.zeppelin.spark.SparkInterpreter.createSparkContext_1(SparkInterpreter.java:488)
    	at org.apache.zeppelin.spark.SparkInterpreter.createSparkContext(SparkInterpreter.java:358)
    	at org.apache.zeppelin.spark.SparkInterpreter.getSparkContext(SparkInterpreter.java:139)
    	at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:716)
    	at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
    	at org.apache.zeppelin.interpreter.LazyOpenInterpreter.getProgress(LazyOpenInterpreter.java:110)
    	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.getProgress(RemoteInterpreterServer.java:447)
    	at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getProgress.getResult(RemoteInterpreterService.java:1701)
    	at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getProgress.getResult(RemoteInterpreterService.java:1686)
    	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
    	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
    	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    	at java.lang.Thread.run(Thread.java:745)
    Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=root, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x
    	at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257)
    	at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238)
    	at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:216)
    	at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:145)
    	at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138)
    

    Kiss the rain2016-08-02 20:38 回复
    • 权限不对啊,你使用root用户写/user目录,但是你应该不属于hdfs的用户组,所以你修改一些启动zeppelin的用户吧。

      w3970907702016-08-04 10:11 回复
  3. 大神,Zeppelin必须部署在hadoop集群的namenode或datanode上面吗?
    可以单独部署,然后提交任务到yarn吗?

    YUZJANG2016-06-12 16:47 回复
    • 不是,你可以部署在任何机器上,然后配置好相关的HADOOP_HOME环境变量就可以将任务提交到yarn上。

      w3970907702016-06-12 19:12 回复
      • 大神,如果独立部署的话,怎么提交任务到yarn集群呢?

        YUZJANG2016-06-13 14:09 回复
        • 这篇文章不就是介绍这个的吗?

          w3970907702016-06-13 15:24 回复
          • 大神,我已经设置了Zeppelin机器上的hadoop,yarn-site.xml和core-site.xml都指向了yarn集群的nn,但是在notebook提交spark任务的时候,日志里面老是提示这个:Connecting to ResourceManager at 0.0.0.0/0.0.0.0:8032,提交不过去啊。

            另外我在这台机器上执行“hadoop fs -ls /”是可以显示集群的hdfs资源的。

            YUZJANG2016-06-14 15:12
    • 你的Hadoop环境变量是配置在conf/zeppelin-env.s里面吗?你的那个错误应该是Zeppelin没有读取到Hadoop相关的配置。

      w3970907702016-06-14 15:52 回复
      • 在zeppelin-env.sh文件里面设置了hadoop的环境变量;除此之外,还在profile文件里面也设置了全局的hadoop环境变量。当然都是指定的同一个地方。

        YUZJANG2016-06-14 16:03 回复
        • 很奇怪,我刚刚又看了一下,我的zeppelin-env.sh文件里面只配置了下面四个配置:

          export MASTER=yarn-client
          export SPARK_HOME=/user/iteblog/spark-1.5.2-bin-2.2.0/
          export HADOOP_HOME=/user/iteblog/hadoop-2.2.0
          export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop/

          然后我启动了zeppelin,可以在YARN上运行。

          w3970907702016-06-14 16:50 回复
          • 确实需要export SPARK_HOME=...这个配置,配置了这个之后,就可以正常提交到yarn集群了。谢谢大神耐心回复 🙂

            YUZJANG2016-06-14 18:09
        • 你好,这是我的配置文件zeppelin-env.sh
          export JAVA_HOME=/usr/local/jdk1.8
          export MASTER=yarn-client
          export HADOOP_HOME=/usr/local/hadoop-2.6.0
          export HADOOP_CONF_DIR=/usr/local/hadoop-2.6.0/etc/hadoop/
          export SPARK_HOME=/usr/local/spark-1.5.2

          但是启动会执行
          sc.version
          java.lang.ClassNotFoundException: org.apache.spark.repl.SparkCommandLine
          at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
          at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
          at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:393)
          at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:68)
          at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:92)
          at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:344)
          at org.apache.zeppelin.scheduler.Job.run(Job.java:176)
          at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)

          漫步2016-06-15 10:34 回复
          • zeppelin编译通过了吗?会不会编译是时候有报错?

            YUZJANG2016-06-16 10:13
        • 大神可否添加一下qq,互相交流一下 1578326883 先谢过

          漫步2016-06-15 10:37 回复
    • 谢谢你回复,编译的时候确实有报错,不过最后还是success了,但是就是测试不成功,这能用%sh,其他的不行,可否添加一下你的qq,请教一下,我的qq(1578326883)互相学习

      漫步2016-06-16 10:31 回复