欢迎关注Hadoop、Spark、Flink、Hive、Hbase、Flume等大数据资料分享微信公共账号:iteblog_hadoop
  1. 文章总数:1059
  2. 浏览总数:14,234,786
  3. 评论:4166
  4. 分类目录:111 个
  5. 注册用户数:7026
  6. 最后更新:2019年9月18日
过往记忆博客公众号iteblog_hadoop
欢迎关注微信公众号:
iteblog_hadoop
大数据技术博客公众号bigdata_ai
开发爱好者社区:
bigdata_ai

用Maven编译Apache flume-ng 1.5.0源码及问题解决

  在今年的5月22号,Flume-ng 1.5.0版本正式发布,关于Flume-ng 1.5.0版本的新特性可以参见本博客的《Apache Flume-ng 1.5.0正式发布》进行了解。关于Apache flume-ng 1.4.0版本的编译可以参见本博客《基于Hadoop-2.2.0编译flume-ng 1.4.0及错误解决》。本文将讲述如何用Maven编译Apache flume-ng 1.5.0源码。

一、到官方网站下载相应版本的flume-ng源码

# wget http://mirror.bit.edu.cn/apache/flume/1.5.0/apache-flume-1.5.0-src.tar.gz
# tar -zxf apache-flume-1.5.0-src.tar.gz
# cd apache-flume-1.5.0-src

二、用maven在相应的Hadoop上编译Flume-ng 1.5.0源码

Flume-ng 1.5.0默认的Hadoop版本是1.0.1,Hbase是0.92.1。我们可以在编译的时候加上一些参数,指定Hadoop 2.2.0版本进行编译:

-Phadoop-2

三、Hadoop、Hbase对应的test不能下载

  如果在编译的时候出现了类似如下的情况,请将相应pom.xml文件中的Hadoop-test和Hbase-test依赖包去掉:

<!--
 User: 过往记忆
 Date: 14-6-16
 Time: 23:45
 bolg: https://www.iteblog.com
 本文地址:https://www.iteblog.com/archives/1043
 过往记忆博客,专注于hadoop、hive、spark、shark、flume的技术博客,大量的干货
 过往记忆博客微信公共帐号:iteblog_hadoop
-->
<dependency>
        <groupId>org.apache.hbase</groupId>
        <artifactId>hbase</artifactId>
        <version>${hbase.version}</version>
        <classifier>tests</classifier>
        <scope>test</scope>
</dependency>

<dependency>
         <groupId>org.apache.hadoop</groupId>
         <artifactId>hadoop-test</artifactId>
         <version>${hadoop.version}</version>
</dependency>

四、elasticsearch-0.90.1.jar read错误

  如果你在编译的过程中出现一下异常:

[ERROR] Failed to execute goal org.apache.maven.plugins:
maven-compiler-plugin:2.3.2:compile (default-compile) on 
project flume-ng-elasticsearch-sink: Compilation failure
[ERROR] error: error reading 
/org/elasticsearch/elasticsearch/0.90.1/elasticsearch-0.90.1.jar; 
error in opening zip file
[ERROR] -> [Help 1]

  只需要到你maven本地仓库将elasticsearch-0.90.1.jar文件删除,然后重新下载即可:

# rm elasticsearch-0.90.1.jar
wget http://central.maven.org/maven2/org/elasticsearch/elasticsearch/0.90.1/elasticsearch-0.90.1.jar

五、ua-parser-1.3.0下载失败

编译过程中,下载ua-parser-1.3.0.pom可能失败,出现如下错误:

[ERROR] Failed to execute goal on project flume-ng-morphline-solr-sink:
 Could not resolve dependencies for project 
org.apache.flume.flume-ng-sinks:flume-ng-morphline-solr-sink:jar:1.5.0: 
Failed to collect dependencies for [org.apache.flume:flume-ng-core:jar:1.5.0 
(compile), org.slf4j:slf4j-api:jar:1.6.1 (compile), 
org.kitesdk:kite-morphlines-all:pom:0.12.0 (compile?),
 org.slf4j:jcl-over-slf4j:jar:1.6.1 (provided), 
org.apache.solr:solr-test-framework:jar:4.3.0 (test), 
org.kitesdk:kite-morphlines-solr-core:jar:tests:0.12.0 (test), 
junit:junit:jar:4.10 (test)]: Failed to read artifact descriptor 
for ua_parser:ua-parser:jar:1.3.0: Could not transfer artifact 
ua_parser:ua-parser:pom:1.3.0 from/to maven-twttr (http://maven.twttr.com):
 Connection to http://maven.twttr.com refused: Connection timed out -> [Help 1]

只需要加一个新的repository即可:

<repository>
  <id>nexus.axiomalaska.com</id>
  <url>http://nexus.axiomalaska.com/nexus/content/repositories/public</url>
</repository>

六、编译Flume-ng 1.5.0源码

# mvn install -Phadoop-2 -DskipTests -Dtar

编译需要一段时间,这个需要看你编译时的网速。如果一切顺利的话,你将会看到以下输出,说明你的Flume-ng 1.5.0编译成功:

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Flume ...................................... SUCCESS [1.360s]
[INFO] Flume NG SDK ...................................... SUCCESS [2.903s]
[INFO] Flume NG Configuration ............................ SUCCESS [0.483s]
[INFO] Flume NG Core ..................................... SUCCESS [2.531s]
[INFO] Flume NG Sinks .................................... SUCCESS [0.110s]
[INFO] Flume NG HDFS Sink ................................ SUCCESS [1.668s]
[INFO] Flume NG IRC Sink ................................. SUCCESS [0.484s]
[INFO] Flume NG Channels ................................. SUCCESS [0.074s]
[INFO] Flume NG JDBC channel ............................. SUCCESS [0.579s]
[INFO] Flume NG file-based channel ....................... SUCCESS [0.712s]
[INFO] Flume NG Spillable Memory channel ................. SUCCESS [0.552s]
[INFO] Flume NG Node ..................................... SUCCESS [0.619s]
[INFO] Flume NG Embedded Agent ........................... SUCCESS [1.001s]
[INFO] Flume NG HBase Sink ............................... SUCCESS [2.373s]
[INFO] Flume NG ElasticSearch Sink ....................... SUCCESS [0.899s]
[INFO] Flume NG Morphline Solr Sink ...................... SUCCESS [2.483s]
[INFO] Flume NG Kite Dataset Sink ........................ SUCCESS [35.925s]
[INFO] Flume Sources ..................................... SUCCESS [0.054s]
[INFO] Flume Scribe Source ............................... SUCCESS [1.231s]
[INFO] Flume JMS Source .................................. SUCCESS [2.133s]
[INFO] Flume Twitter Source .............................. SUCCESS [1.204s]
[INFO] Flume legacy Sources .............................. SUCCESS [0.043s]
[INFO] Flume legacy Avro source .......................... SUCCESS [1.234s]
[INFO] Flume legacy Thrift Source ........................ SUCCESS [1.320s]
[INFO] Flume NG Clients .................................. SUCCESS [0.038s]
[INFO] Flume NG Log4j Appender ........................... SUCCESS [6.175s]
[INFO] Flume NG Tools .................................... SUCCESS [1.559s]
[INFO] Flume NG distribution ............................. SUCCESS [1:51.993s]
[INFO] Flume NG Integration Tests ........................ SUCCESS [1.517s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3:03.921s
[INFO] Finished at: Mon Jun 16 23:37:51 CST 2014
[INFO] Final Memory: 135M/745M
[INFO] ------------------------------------------------------------------------

最后将会在flume-ng-dist/target目录下产生以下的文件:

total 29208
drwxr-xr-x 3 wyp wyp     4096 Jun 16 23:36 apache-flume-1.5.0-bin
-rw-r--r-- 1 wyp wyp 27307850 Jun 16 23:36 apache-flume-1.5.0-bin.tar.gz
drwxr-xr-x 3 wyp wyp     4096 Jun 16 23:36 apache-flume-1.5.0-src
-rw-r--r-- 1 wyp wyp  2543174 Jun 16 23:37 apache-flume-1.5.0-src.tar.gz
drwxr-xr-x 2 wyp wyp     4096 Jun 16 23:36 archive-tmp
drwxr-xr-x 3 wyp wyp     4096 Jun 16 23:36 maven-shared-archive-resources
-rw-r--r-- 1 wyp wyp      990 Jun 16 23:37 rat.txt

apache-flume-1.5.0-bin这个文件夹里面的东西可以直接拿来用。

本文地址:《用Maven编译Apache flume-ng 1.5.0源码及问题解决》:https://www.iteblog.com/archives/1043,过往记忆博客,专注于hadoop、hive、spark、shark、flume的技术博客,大量的干货.过往记忆博客微信公共帐号:iteblog_hadoop

本博客文章除特别声明,全部都是原创!
转载本文请加上:转载自过往记忆(https://www.iteblog.com/)
本文链接: 【用Maven编译Apache flume-ng 1.5.0源码及问题解决】(https://www.iteblog.com/archives/1043.html)
喜欢 (14)
分享 (0)
发表我的评论
取消评论

表情
本博客评论系统带有自动识别垃圾评论功能,请写一些有意义的评论,谢谢!
(23)个小伙伴在吐槽
  1. repository总觉得不太对

    earthnut2014-10-31 15:41 回复
  2. 那个repository写错了

    fc2014-09-23 10:50 回复
  3. 到hdfssink出问题 groupid cannot be empty

    fc2014-09-23 10:34 回复
    • 配置是不是出错啦?

      w3970907702014-09-23 10:36 回复
  4. 有人评论完后看到完整内容了吗,我是各种评论都看不到,谁看到repository 了,麻烦贴一下,谢谢!!

    张炳祥2014-07-23 16:04 回复
  5. 正好遇到了这个问题,谢谢了

    东软_化云为雨2014-07-23 14:52 回复
  6. elasticsearch

    abc2014-07-19 15:52 回复
  7. fffff

    fff2014-07-19 15:45 回复
  8. fsaf

    ds2014-07-18 18:57 回复
  9. 看看怎么解决

    Maker2014-07-15 18:22 回复
  10. 看看怎么解决

    zf2014-07-10 19:43 回复
  11. 看看怎么解决

    hzm2014-07-10 12:11 回复
  12. 看看怎么解决

    aaa2014-07-10 10:10 回复
  13. 感谢 你的信息对我很有用

    cc2014-07-09 15:09 回复
  14. thx

    yeah2014-07-09 14:53 回复
  15. 谢谢分享!

    spring_pear2014-07-08 14:57 回复
  16. 再次点赞

    aaa2014-06-27 23:21 回复
  17. 点赞。。

    fysoft20062014-06-20 12:53 回复
  18. 我也有此问题,正想看看如何解决

    成梦882014-06-19 16:58 回复
  19. 我也有此问题,正想看看如何解决

    成梦2014-06-19 16:42 回复
  20. 正好遇到了这个问题,谢谢了

    qq2014-06-19 13:37 回复
  21. 靠谱

    cenqiongyu2014-06-17 20:29 回复
    • 需要注释掉好多 tests 的行,然后把 hbase那端注释掉,因为在cdh5.1.0没有hbase的包,需要添加hbase的子包如:hbase-server、hbase-client等、然后做install的时候要过滤掉测试代码,如:
      mvn install -Dmaven.test.skip=true -Phadoop-2 -DskipTests -Dtar

      fly2014-09-28 18:00 回复