欢迎关注Hadoop、Spark、Flink、Hive、Hbase、Flume等大数据资料分享微信公共账号:iteblog_hadoop
  1. 文章总数:978
  2. 浏览总数:11,976,096
  3. 评论:3938
  4. 分类目录:106 个
  5. 注册用户数:6129
  6. 最后更新:2018年12月15日
过往记忆博客公众号iteblog_hadoop
欢迎关注微信公众号:
iteblog_hadoop
大数据技术博客公众号bigdata_ai
大数据猿:
bigdata_ai

Flume-0.9.4源码编译及一些编译出错解决方法

  由于需要在Flume里面加入一些我需要的代码,这时候就需要重新编译Flume代码,因为在编译Flume源码的时候出现了很多问题,所以写出这篇博客,以此分享给那些也需要编译代码的人一些参考,这里以如何编译Flume-0.9.4源码为例进行说明。
  首先下载Flume0.9.4源码(可以到https://repository.cloudera.com/content/repositories/releases/com/cloudera/flume-distribution/0.9.4-cdh4.0.0/里面下载)之后,解压及编译:

[wyp@master Desktop]$ unzip flume-distribution-0.9.4-cdh4.0.0-project.zip
[wyp@master Desktop]$ cd flume-0.9.4-cdh4.0.0/
[wyp@master flume-0.9.4-cdh4.0.0]$  mvn package -Pdist -DskipTests -Dtar

这时候会出现下面的错误:

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Flume ............................................. SUCCESS [0.003s]
[INFO] Flume Core ........................................ FAILURE [5.016s]
[INFO] Flume Master Config Web Application ............... SKIPPED
[INFO] Flume Node Web .................................... SKIPPED
[INFO] Flume Distribution Project ........................ SKIPPED
[INFO] A log4j appender for Flume ........................ SKIPPED
[INFO] Flume Hello World Plugin .......................... SKIPPED
[INFO] Flume HBase Plugin ................................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 5.798s
[INFO] Finished at: Wed Jan 22 16:35:13 CST 2014
[INFO] Final Memory: 15M/136M
[INFO] ------------------------------------------------------------------------
[WARNING] The requested profile "dist" could not be activated because it does 
not exist.
[ERROR] Failed to execute goal org.apache.thrift.tools:maven-thrift-plugin:
0.1.10:compile (default) on project flume-core: thrift did not exit cleanly. 
Review output for more information. -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the 
-e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, 
please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/
                 MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with 
the command
[ERROR]   mvn <goals> -rf :flume-core

这是因为编译Flume的时候需要用到thrift插件,可以参考flume-core/pom.xml文件里面的配置:

<plugin>
    <groupId>org.apache.thrift.tools</groupId>
    <artifactId>maven-thrift-plugin</artifactId>
    <version>0.1.10</version>
    <configuration>
      <thriftExecutable>${thrift.executable}</thriftExecutable>
    </configuration>
    <executions>
      <execution>
        <goals>
          <goal>compile</goal>
          <goal>testCompile</goal>
        </goals>
      </execution>
    </executions>
</plugin>

  所以需要安装thrift,可以参考这里进行安装:《Flume-0.9.4源码编译依赖的thrift插件安装》安装完thrift,再环境变量里面设置一下THRIFT_HOME=/usr/local,因为在flume-core/pom.xml里面会用到thrift,如下:

<properties>
  <!-- NB: The version of the thrift compiler must match that of the dependency 
  on the jar file below. -->
  <thrift.executable>${env.THRIFT_HOME}/bin/thrift</thrift.executable>

  <!-- Set default encoding to UTF-8 to remove maven complaints -->
  <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>

  <!-- defaults for flaky test and focused test exclusions -->
  <test.exclude.pattern>$</test.exclude.pattern> <!-- junk pattern -->
  <test.include.pattern>**/Test*.java</test.include.pattern>
</properties>

接着再重新编译Flume源码,这时候将会出现大量类似以下的错误信息:

[WARNING] The requested profile "dist" could not be activated because it does 
not exist.
[ERROR] Failed to execute goal org.apache.maven.plugins:
maven-compiler-plugin:2.3.2:compile (default-compile) on project flume-core: 
Compilation failure: Compilation failure:
[ERROR] /home/wyp/Desktop/flume-src/flume-core/target/generated-sources/
thrift/com/cloudera/flume/conf/thrift/ThriftFlumeClientServer.java:[9,39] 
错误: 程序包org.apache.commons.lang3.builder不存在
[ERROR] /home/wyp/Desktop/flume-src/flume-core/target/generated-sources/
thrift/com/cloudera/flume/conf/thrift/ThriftFlumeClientServer.java:[10,31] 
错误: 程序包org.apache.thrift.scheme不存在
[ERROR] /home/wyp/Desktop/flume-src/flume-core/target/generated-sources/
thrift/com/cloudera/flume/conf/thrift/ThriftFlumeClientServer.java:[11,31] 
错误: 程序包org.apache.thrift.scheme不存在
#########################################################################
                   下面省略了很多类似的错误信息
#########################################################################

请将pom.xml里面libthrift依赖版本修改为0.9.1,如下:

<dependency>
        <groupid>org.apache.thrift</groupid>
        <artifactid>libthrift</artifactid>
        <version>0.6.0-cdh</version>
</dependency>

改为

<dependency>
        <groupid>org.apache.thrift</groupid>
        <artifactid>libthrift</artifactid>
        <version>0.9.1</version>
</dependency>

修改完后再编译。紧接着的错误是类似下面的错误信息:

[WARNING] The requested profile "dist" could not be activated because it does 
not exist.
[ERROR] Failed to execute goal on project flume-config-web: Could not resolve 
dependencies for project com.cloudera:flume-config-web:war:0.9.4-cdh4.0.0: 
Failure to find org.apache.hadoop:hadoop-core:jar:0.20.2-cdh3u3-SNAPSHOT in
https://repository.cloudera.com/content/groups/public/ was cached in the local 
repository, resolution will not be reattempted until the update interval of 
com.cloudera.repository.releases has elapsed or updates are forced -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors,re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read 
the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :flume-config-web

从错误信息可以得到编译Flume的时候依赖的hadoop-core:jar:0.20.2-cdh3u3-SNAPSHOT不能成功下载,解决办法是到pom.xml里面进行如下修改

<dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-core</artifactId>
        <version>0.20.2-cdh3u3-SNAPSHOT</version>
        <exclusions>
          <exclusion>
            <groupId>com.cloudera.cdh</groupId>
            <artifactId>hadoop-ant</artifactId>
          </exclusion>
        </exclusions>
</dependency>

改为

<dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-core</artifactId>
        <version>0.20.2-cdh3u3</version>
        <exclusions>
          <exclusion>
            <groupId>com.cloudera.cdh</groupId>
            <artifactId>hadoop-ant</artifactId>
          </exclusion>
        </exclusions>
</dependency>

修改完保存再编译源码。这时候应该可以顺利的进行编译,最后编译成功的界面如下:

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Flume ............................................. SUCCESS [0.003s]
[INFO] Flume Core ........................................ SUCCESS [12.344s]
[INFO] Flume Master Config Web Application ............... SUCCESS [3.243s]
[INFO] Flume Node Web .................................... SUCCESS [0.885s]
[INFO] Flume Distribution Project ........................ SUCCESS [25.381s]
[INFO] A log4j appender for Flume ........................ SUCCESS [0.251s]
[INFO] Flume Hello World Plugin .......................... SUCCESS [0.133s]
[INFO] Flume HBase Plugin ................................ SUCCESS [20:42.199s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 21:25.138s
[INFO] Finished at: Thu Jan 23 00:50:28 CST 2014
[INFO] Final Memory: 42M/286M
[INFO] ------------------------------------------------------------------------
[WARNING] The requested profile "dist" could not be activated because it does 
not exist.
[wyp@master flume-0.9.4-cdh4.0.0]$

恭喜你,编译成功了!

本博客文章除特别声明,全部都是原创!
转载本文请加上:转载自过往记忆(https://www.iteblog.com/)
本文链接: 【Flume-0.9.4源码编译及一些编译出错解决方法】(https://www.iteblog.com/archives/909.html)
喜欢 (3)
分享 (0)
发表我的评论
取消评论

表情
本博客评论系统带有自动识别垃圾评论功能,请写一些有意义的评论,谢谢!