欢迎关注Hadoop、Spark、Flink、Hive、Hbase、Flume等大数据资料分享微信公共账号:iteblog_hadoop
  1. 文章总数:961
  2. 浏览总数:11,461,275
  3. 评论:3870
  4. 分类目录:103 个
  5. 注册用户数:5832
  6. 最后更新:2018年10月17日
过往记忆博客公众号iteblog_hadoop
欢迎关注微信公众号:
iteblog_hadoop
大数据技术博客公众号bigdata_ai
大数据猿:
bigdata_ai

Spark编译错误笔记

  最近修改了Spark的一些代码,然后编译Spark出现了以下的异常信息:

error file=/iteblog/spark-1.3.1/streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala
message=File line length exceeds 100 characters line=279
error file=/iteblog/spark-1.3.1/streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala
message=File line length exceeds 100 characters line=612
Saving to outputFile=/iteblog/spark-1.3.1/streaming/scalastyle-output.xml
Processed 79 file(s)
Found 2 errors
Found 0 warnings
Found 0 infos
Finished in 946 ms
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Spark Project Parent POM .......................... SUCCESS [3.381s]
[INFO] Spark Project Networking .......................... SUCCESS [13.187s]
[INFO] Spark Project Shuffle Streaming Service ........... SUCCESS [6.580s]
[INFO] Spark Project Core ................................ SUCCESS [3:24.312s]
[INFO] Spark Project Bagel ............................... SUCCESS [9.088s]
[INFO] Spark Project GraphX .............................. SUCCESS [25.691s]
[INFO] Spark Project Streaming ........................... FAILURE [24.015s]
[INFO] Spark Project Catalyst ............................ SKIPPED
[INFO] Spark Project SQL ................................. SKIPPED
[INFO] Spark Project ML Library .......................... SKIPPED
[INFO] Spark Project Tools ............................... SKIPPED
[INFO] Spark Project Hive ................................ SKIPPED
[INFO] Spark Project REPL ................................ SKIPPED
[INFO] Spark Project YARN ................................ SKIPPED
[INFO] Spark Ganglia Integration ......................... SKIPPED
[INFO] Spark Project Assembly ............................ SKIPPED
[INFO] Spark Project External Twitter .................... SKIPPED
[INFO] Spark Project External Flume Sink ................. SKIPPED
[INFO] Spark Project External Flume ...................... SKIPPED
[INFO] Spark Project External MQTT ....................... SKIPPED
[INFO] Spark Project External ZeroMQ ..................... SKIPPED
[INFO] Spark Project External Kafka ...................... SKIPPED
[INFO] Spark Project Examples ............................ SKIPPED
[INFO] Spark Project YARN Shuffle Service ................ SKIPPED
[INFO] Spark Project External Kafka Assembly ............. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4:46.989s
[INFO] Finished at: Wed May 20 14:56:59 CST 2015
[INFO] Final Memory: 82M/530M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.scalastyle:scalastyle-maven-plugin:0.4.0:check (default) on project spark-streaming_2.10: Failed during scalastyle execution: You have 2 Scalastyle violation(s). -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :spark-streaming_2.10

  从上面出错的第一行和第二行可以看出是第279行和第612行的代码长度超过100个字符的原因,因为Spark在编译的时候会检测Scala的代码风格是否符合要求,我们可以通过下面方法解决:

  修改Spark源码的pom.xml文件,找到下面的行:

<failOnViolation>true</failOnViolation>

可以修改成:

@@ -123,6 +123,7 @@
     <log4j.version>1.2.17</log4j.version>
     <hadoop.version>1.0.4</hadoop.version>
     <protobuf.version>2.4.1</protobuf.version>
+    <scalastyle.failonviolation>true</scalastyle.failonviolation>
     <yarn.version>${hadoop.version}</yarn.version>
     <hbase.version>0.94.6</hbase.version>
     <flume.version>1.4.0</flume.version>
@@ -1071,7 +1072,7 @@
         <version>0.4.0</version>
         <configuration>
           <verbose>false</verbose>
-          <failOnViolation>true</failOnViolation>
+          <failOnViolation>${scalastyle.failonviolation}</failOnViolation>
           <includeTestSourceDirectory>false</includeTestSourceDirectory>
           <failOnWarning>false</failOnWarning>
           <sourceDirectory>${basedir}/src/main/scala</sourceDirectory>

上面的+是添加的,-是删除的。然后可以通过下面的方式编译:

[iteblog@iteblog ~]$ ./make-distribution.sh --tgz -Phadoop-2.6 -Pyarn -DskipTests    \
     -Dhadoop.version=2.6.0 -Phive -Pspark-ganglia-lgpl -Dscalastyle.failOnViolation=false

或者直接将failOnViolation选项修改成false:

<failOnViolation>false</failOnViolation>

然后再去编译Spark代码, 这样就不会检测Scala代码规范了。

本博客文章除特别声明,全部都是原创!
转载本文请加上:转载自过往记忆(https://www.iteblog.com/)
本文链接: 【Spark编译错误笔记】(https://www.iteblog.com/archives/1365.html)
喜欢 (2)
分享 (0)
发表我的评论
取消评论

表情
本博客评论系统带有自动识别垃圾评论功能,请写一些有意义的评论,谢谢!