欢迎关注Hadoop、Spark、Flink、Hive、Hbase、Flume等大数据资料分享微信公共账号:iteblog_hadoop
  1. 文章总数:978
  2. 浏览总数:11,953,022
  3. 评论:3936
  4. 分类目录:106 个
  5. 注册用户数:6116
  6. 最后更新:2018年12月15日
过往记忆博客公众号iteblog_hadoop
欢迎关注微信公众号:
iteblog_hadoop
大数据技术博客公众号bigdata_ai
大数据猿:
bigdata_ai

基于Hadoop-2.2.0编译flume-ng 1.4.0及错误解决

  写在前面的话,最近发现有很多网站转载我博客的文章,这个我都不介意的,但是这些网站转载我博客都将文章的出处去掉了,直接变成自己的文章了!!我强烈谴责他们,鄙视那些转载文章去掉出处的人!所以为了防止这些,我以后发表文章的时候,将会在文章里面加入一些回复之后才可见的内容!!请大家不要介意,本博客Hadoop相关的文章都是原创,每篇文章都花了很长时间编写,这些都是无偿的,所以转载本文章请标明出处:
  本篇博客地址:《基于Hadoop-2.2.0编译flume-ng 1.4.0及错误解决》:https://www.iteblog.com/archives/1032
  博客地址:过往记忆:https://www.iteblog.com

  关于如何编译Flume 0.9.4可以参见本博客的《Flume-0.9.4源码编译及一些编译出错解决方法》《Flume-0.9.4源码编译依赖的thrift插件安装》
  最近我公司准备将Flume 0.94升级到Flume-ng 1.4,之前在本博客《Flume-0.9.4和Hbase-0.96整合(1)》《Flume-0.9.4和Hbase-0.96整合(2)》《Flume-0.9.4配置Hbase sink》等序列文章中涉及到的都是Flume 0.9.4,在之前开发的各种插件得重新编写到支持Flume-ng 1.4,所以今天来谈谈如何基于Hadoop 2.2.0、Hbase 0.96.0编译Flume-ng 1.4.0。之前特别忙,所以都没时间总结。编译Flume-ng的步骤如下:

一、去Flume官方网站下载相应的src
$ wget http://mirror.bit.edu.cn/apache/flume/1.4.0/apache-flume-1.4.0-src.tar.gz
$ tar -zxf apache-flume-1.4.0-src.tar.gz
$ cd apache-flume-1.4.0-src
二、修改相应的pom文件

  1、修改Hadoop、Hbase相应的版本
  apache-flume-1.4.0-src是基于Hadoop 1.0.1、Hbase 0.92.1或者Hadoop 2.0.0-alpha、Hbase 0.94.2编写的,而公司用到的Hadoop是2.2.0,Hbase是0.96,所以我们需要修改pom文件如下:

<properties>
        <hadoop.version>2.0.0-alpha</hadoop.version>
        <hbase.version>0.94.2</hbase.version>
        <hadoop.common.artifact.id>hadoop-common</hadoop.common.artifact.id>
        <thrift.version>0.8.0</thrift.version>
</properties>

修改成

<properties>
        <hadoop.version>2.2.0</hadoop.version>
        <hbase.version>0.96.0-hadoop2</hbase.version>
        <hadoop.common.artifact.id>hadoop-common</hadoop.common.artifact.id>
        <thrift.version>0.8.0</thrift.version>
 </properties>

  2、修改protobuf的版本
  Flume-ng的用到的protobuf是2.4.1,而目前的Hbase、Hadoop都是2.5.0,所以我们也得修改:

<dependency>
        <groupId>com.google.protobuf</groupId>
        <artifactId>protobuf-java</artifactId>
        <scope>compile</scope>
        <version>2.4.1</version>
</dependency>

修改为

<dependency>
        <groupId>com.google.protobuf</groupId>
        <artifactId>protobuf-java</artifactId>
        <scope>compile</scope>
        <version>2.5.0</version>
</dependency>

  3、修改flume-ng-sinks中Hbase和Hadoop相应的test
将flume-ng-sinks/flume-hdfs-sink/pom.xml和flume-ng-sinks/flume-ng-hbase-sink/pom.xml文件中的下面依赖注释掉:

<dependency>
          <groupId>org.apache.hadoop</groupId>
          <artifactId>hadoop-test</artifactId>
          <scope>test</scope>
</dependency>

和

<dependency>
      <groupId>org.apache.hbase</groupId>
      <artifactId>hbase</artifactId>
      <classifier>tests</classifier>
      <scope>test</scope>
</dependency>

否则你会在编译的时候遇到下面的错误

[ERROR] Failed to execute goal on project flume-hdfs-sink: Could not 
resolve dependencies for project org.apache.flume.flume-ng-sinks:
flume-hdfs-sink:jar:1.4.0: Could not find artifact 
org.apache.hadoop:hadoop-test:jar:2.0.0-alpha in XXX -> [Help 1]

或者

[ERROR] Failed to execute goal on project flume-ng-hbase-sink: 
Could not resolve dependencies for project org.apache.flume.flume-ng-sinks:
flume-ng-hbase-sink:jar:1.4.0: The following artifacts could 
not be resolved: org.apache.hbase:hbase:jar:0.96.0-hadoop2, 
org.apache.hbase:hbase:jar:tests:0.96.0-hadoop2: 
Could not find artifact org.apache.hbase:hbase:jar:0.96.0-hadoop2 in 
XXXX -> [Help 1]

  4、修改flume-ng-sinks/flume-ng-morphline-solr-sink的版本
  flume-ng-morphline-solr-sink的默认为0.4.0,但是目前已经不能从Maven上下载了,所以我们在这修改为如下:

<cdk.version>0.5.0</cdk.version>

否则你会在编译的时候遇到下面的错误:

[ERROR] Failed to execute goal on project flume-ng-morphline-solr-sink: 
Could not resolve dependencies for project org.apache.flume.flume-ng-sinks:
flume-ng-morphline-solr-sink:jar:1.4.0: The following artifacts could 
not be resolved: com.cloudera.cdk:cdk-morphlines-solr-cell:jar:0.4.0, 
com.cloudera.cdk:cdk-morphlines-avro:jar:0.4.0, 
com.cloudera.cdk:cdk-morphlines-twitter:jar:0.4.0, 
com.cloudera.cdk:cdk-morphlines-tika-core:jar:0.4.0, 
com.cloudera.cdk:cdk-morphlines-tika-decompress:jar:0.4.0,
com.cloudera.cdk:cdk-morphlines-solr-core:jar:tests:0.4.0: 
Failure to find com.cloudera.cdk:cdk-morphlines-solr-cell:jar:0.4.0 in
https://repository.cloudera.com/artifactory/cloudera-repos was cached in 
the local repository, resolution will not be reattempted until the update 
interval of cdh.repo has elapsed or updates are forced -> [Help 1]
三、编译Flume源码
$ mvn install -Phadoop-2 -DskipTests -Dtar

编译完成之后,将会出现下面的编译结果:

[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Flume ...................................... SUCCESS [1.303s]
[INFO] Flume NG SDK ...................................... SUCCESS [2.811s]
[INFO] Flume NG Configuration ............................ SUCCESS [0.558s]
[INFO] Flume NG Core ..................................... SUCCESS [2.343s]
[INFO] Flume NG Sinks .................................... SUCCESS [0.097s]
[INFO] Flume NG HDFS Sink ................................ SUCCESS [1.602s]
[INFO] Flume NG IRC Sink ................................. SUCCESS [0.610s]
[INFO] Flume NG Channels ................................. SUCCESS [0.067s]
[INFO] Flume NG JDBC channel ............................. SUCCESS [0.637s]
[INFO] Flume NG file-based channel ....................... SUCCESS [0.758s]
[INFO] Flume NG Node ..................................... SUCCESS [0.687s]
[INFO] Flume NG Embedded Agent ........................... SUCCESS [1.063s]
[INFO] Flume NG HBase Sink ............................... SUCCESS [2.976s]
[INFO] Flume NG ElasticSearch Sink ....................... SUCCESS [1.014s]
[INFO] Flume NG Morphline Solr Sink ...................... SUCCESS [3.125s]
[INFO] Flume Sources ..................................... SUCCESS [0.043s]
[INFO] Flume Scribe Source ............................... SUCCESS [0.865s]
[INFO] Flume JMS Source .................................. SUCCESS [1.013s]
[INFO] Flume legacy Sources .............................. SUCCESS [0.046s]
[INFO] Flume legacy Avro source .......................... SUCCESS [1.466s]
[INFO] Flume legacy Thrift Source ........................ SUCCESS [1.035s]
[INFO] Flume NG Clients .................................. SUCCESS [0.090s]
[INFO] Flume NG Log4j Appender ........................... SUCCESS [4.080s]
[INFO] Flume NG Tools .................................... SUCCESS [1.965s]
[INFO] Flume NG distribution ............................. SUCCESS [4:25.620s]
[INFO] Flume NG Integration Tests ........................ SUCCESS [2.003s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4:58.453s
[INFO] Finished at: Tue May 13 22:51:20 CST 2014
[INFO] Final Memory: 127M/708M

最后将会在flume-ng-dist/target目录下产生以下的文件:

drwxr-xr-x 3 wyp wyp     4096 May 13 22:48 apache-flume-1.4.0-bin
-rw-r--r-- 1 wyp wyp 59087227 May 13 22:49 apache-flume-1.4.0-bin.tar.gz
drwxr-xr-x 3 wyp wyp     4096 May 13 22:49 apache-flume-1.4.0-src
-rw-r--r-- 1 wyp wyp  2496952 May 13 22:51 apache-flume-1.4.0-src.tar.gz
drwxr-xr-x 2 wyp wyp     4096 May 13 22:47 archive-tmp
drwxr-xr-x 3 wyp wyp     4096 May 13 22:47 maven-shared-archive-resources
-rw-r--r-- 1 wyp wyp     1005 May 13 22:51 rat.txt

apache-flume-1.4.0-bin里面的东西我们可以直接拷贝出来,直接用了,关于如何使用flume-ng,我将会在后面的文章介绍,敬请关注本博客!

本博客文章除特别声明,全部都是原创!
转载本文请加上:转载自过往记忆(https://www.iteblog.com/)
本文链接: 【基于Hadoop-2.2.0编译flume-ng 1.4.0及错误解决】(https://www.iteblog.com/archives/1032.html)
喜欢 (2)
分享 (0)
发表我的评论
取消评论

表情
本博客评论系统带有自动识别垃圾评论功能,请写一些有意义的评论,谢谢!
(30)个小伙伴在吐槽
  1. 编译出来的用着会出错
    admln2015-04-10 10:45 回复
    • 什么错误?贴出来看看。
      w3970907702015-04-10 12:22 回复
  2. 学习了
    王老三2015-03-19 19:48 回复
  3. 这个要怎么恢复
    炼心2014-12-09 10:50 回复
  4. 没错,就是遇到这个问题
    月出2014-08-12 21:48 回复
  5. 挺好的
    月出2014-08-12 21:47 回复
  6. 学习了,哈哈...........
    虐杀联盟2014-08-12 11:42 回复
  7. 好技术 文章!
    杜天敏2014-08-08 10:44 回复
  8. 评论一下,能看到么????
    mahatma_wm2014-08-01 17:03 回复
  9. 我要看
    shaofeng099992014-07-27 00:36 回复
  10. 测试下
    shaofeng099992014-07-27 00:36 回复
  11. 正在找,谢谢
    zbx132014-07-22 16:50 回复
  12. 赞 正好需要呢,谢谢
    MyZay2014-07-21 21:58 回复
  13. Great!!!
    MyZay2014-07-21 21:58 回复
  14. 可能是我需要的,多谢
    wanxiaohun2014-07-07 15:22 回复
  15. 写的很好!
    nice2014-07-03 13:46 回复
  16. 看有没有我需要的,多谢
    冬日的阳光2014-06-27 00:13 回复
  17. 学习一下
    lirenyao2014-06-26 15:32 回复
  18. 点赞
    fysoft20062014-06-20 12:17 回复
  19. 谢谢分享。。
    fy2014-06-20 12:13 回复
  20. 想看看如何编译flume
    我与路口2014-06-05 13:44 回复
  21. good
    gggg2014-05-29 11:59 回复
  22. 修改flume-ng-sinks/flume-ng-morphline-solr-sink的版本 找的就是这个
    noapologize2014-05-22 02:53 回复
  23. noapologize2014-05-22 02:52 回复
  24. abc2014-05-19 09:55 回复
  25. cdk-morphlines-solr-cell:jar这个问题纠结死人了,谢谢
    leon2014-05-19 09:48 回复
  26. 修改flume-ng-sinks/flume-ng-morphline-solr-sink的版本 找的就是这个。
    flumer2014-05-18 08:01 回复
  27. 赞!
    2014-05-15 14:08 回复
  28. www2014-05-15 14:08 回复
  29. 正好需要呢,谢谢
    过客2014-05-15 09:51 回复