欢迎关注Hadoop、Spark、Flink、Hive、Hbase、Flume等大数据资料分享微信公共账号:iteblog_hadoop
  1. 文章总数:978
  2. 浏览总数:11,976,087
  3. 评论:3938
  4. 分类目录:106 个
  5. 注册用户数:6129
  6. 最后更新:2018年12月15日
过往记忆博客公众号iteblog_hadoop
欢迎关注微信公众号:
iteblog_hadoop
大数据技术博客公众号bigdata_ai
大数据猿:
bigdata_ai

Spark提交作业的一些异常

在提交作业的时候出现了以下的异常信息:

2015-05-05 11:09:28,071 INFO  [Driver] - Attempting to load checkpoint from file hdfs://iteblogcluster/user/iteblog/checkpoint2/checkpoint-1430794986000
2015-05-05 11:09:28,076 WARN  [Driver] - Error reading checkpoint from file hdfs://iteblogcluster/user/iteblog/checkpoint2/checkpoint-1430794986000
java.io.InvalidClassException: org.apache.spark.streaming.Checkpoint; local class incompatible: stream classdesc serialVersionUID = 4260288410130070970, local class serialVersionUID = -1544620567275648823
	at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617)
	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
	at org.apache.spark.streaming.CheckpointReader$$anonfun$read$2.apply(Checkpoint.scala:258)
	at org.apache.spark.streaming.CheckpointReader$$anonfun$read$2.apply(Checkpoint.scala:246)
	at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
	at org.apache.spark.streaming.CheckpointReader$.read(Checkpoint.scala:246)
	at org.apache.spark.streaming.StreamingContext$.getOrCreate(StreamingContext.scala:624)
	at logstatstreaming.LogSplitStreamingKafka$.main(LogSplitStreamingKafka.scala:46)
	at logstatstreaming.LogSplitStreamingKafka.main(LogSplitStreamingKafka.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:480)
2015-05-05 11:09:28,077 ERROR [Driver] - User class threw exception: Failed to read checkpoint from directory /user/iteblog/checkpoint2
org.apache.spark.sparkException: Failed to read checkpoint from directory /user/iteblog/checkpoint2
	at org.apache.spark.streaming.CheckpointReader$.read(Checkpoint.scala:272)
	at org.apache.spark.streaming.StreamingContext$.getOrCreate(StreamingContext.scala:624)
	at logstatstreaming.LogSplitStreamingKafka$.main(LogSplitStreamingKafka.scala:46)
	at logstatstreaming.LogSplitStreamingKafka.main(LogSplitStreamingKafka.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:480)

出现这个异常的原因是两次提交作业的Spark版本不一致导致的,将Spark版本弄成一致即可从Checkpoint目录中恢复。

本博客文章除特别声明,全部都是原创!
转载本文请加上:转载自过往记忆(https://www.iteblog.com/)
本文链接: 【Spark提交作业的一些异常】(https://www.iteblog.com/archives/1348.html)
喜欢 (6)
分享 (0)
发表我的评论
取消评论

表情
本博客评论系统带有自动识别垃圾评论功能,请写一些有意义的评论,谢谢!