欢迎关注Hadoop、Spark、Flink、Hive、Hbase、Flume等大数据资料分享微信公共账号:iteblog_hadoop
  1. 文章总数:977
  2. 浏览总数:11,945,297
  3. 评论:3936
  4. 分类目录:106 个
  5. 注册用户数:6113
  6. 最后更新:2018年12月13日
过往记忆博客公众号iteblog_hadoop
欢迎关注微信公众号:
iteblog_hadoop
大数据技术博客公众号bigdata_ai
大数据猿:
bigdata_ai

使用idea阅读Kafka源码

本文涉及到的环境:

  • 操作系统:Windows 7
  • Idea 版本:IntelliJ IDEA 2016.3.4 Build #IU-163.12024.16, built on January 31, 2017
  • Kafka 版本:Kafka 0.8.2.0
  • Gradle 版本:gradle-4.0.1
  • JDK 版本:jdk1.7.0
  • Scala 版本:2.10.4

首先到http://archive.apache.org/dist/kafka/里面下载你需要的Kafka源码,本文选自的是kafka-0.8.2.0。

因为Kafka代码自0.8.x之后就使用 Gradle 来进行编译和构建了,所以如果直接通过 idea 来打开上面下载的Kafka 源码会出现很多错误。我们需要先安装好 Gradle。

https://gradle.org/releases/ 下载你需要的 Gradle,本文选择的是 gradle-4.0.1。下载并解压 Gradle 到一个目录,然后创建一个环境变量 GRADLE_HOME 指向解压的目录,再将 %GRADLE_HOME%\bin 加到 PATH 环境变量中,保存并打开 cmd 窗口,在里面输入 gradle -v,如果输出类似于下面的信息,说明 Gradle 已经加入到环境变量中了。

$ gradle -v

------------------------------------------------------------
Gradle 4.0.1
------------------------------------------------------------

Build time:   2017-07-07 14:02:41 UTC
Revision:     38e5dc0f772daecca1d2681885d3d85414eb6826

Groovy:       2.4.11
Ant:          Apache Ant(TM) version 1.9.6 compiled on June 29 2015
JVM:          1.7.0 (Oracle Corporation 21.0-b17)
OS:           Windows 7 6.1 amd64

当然,将 Gradle 加入到环境变量是可选的,只是可以直接在命令行里面使用 gradle

安装完 Gradle 之后,我们到Kafka的源码目录下,然后在 cmd 窗口执行下面的命令:

$ gradle idea
Starting a Gradle Daemon, 2 busy and 1 incompatible Daemons could not be reused, use --status for details
Download https://repo1.maven.org/maven2//nl/javadude/gradle/plugins/license-gradle-plugin/0.10.0/license-gradle-plugin-0.10.0.pom
Download https://repo1.maven.org/maven2//com/google/guava/guava/17.0/guava-17.0.pom
Download https://repo1.maven.org/maven2//com/mycila/maven-license-plugin/maven-license-plugin/1.10.b1/maven-license-plugin-1.10.b1.pom
Download https://repo1.maven.org/maven2//com/google/guava/guava-parent/17.0/guava-parent-17.0.pom
Download https://repo1.maven.org/maven2//org/sonatype/oss/oss-parent/7/oss-parent-7.pom
Download https://repo1.maven.org/maven2//com/mycila/parent-pom/5/parent-pom-5.pom
Download https://repo1.maven.org/maven2//com/mycila/xmltool/xmltool/3.3/xmltool-3.3.pom
Download https://repo1.maven.org/maven2//org/codehaus/plexus/plexus-utils/2.0.5/plexus-utils-2.0.5.pom
Download https://repo1.maven.org/maven2//org/codehaus/plexus/plexus/2.0.6/plexus-2.0.6.pom
Download https://repo1.maven.org/maven2//com/mycila/maven-license-plugin/maven-license-plugin/1.10.b1/maven-license-plugin-1.10.b1.jar
Download https://repo1.maven.org/maven2//com/google/guava/guava/17.0/guava-17.0.jar
Download https://repo1.maven.org/maven2//com/mycila/xmltool/xmltool/3.3/xmltool-3.3.jar
Download https://repo1.maven.org/maven2//org/codehaus/plexus/plexus-utils/2.0.5/plexus-utils-2.0.5.jar
Download https://repo1.maven.org/maven2//nl/javadude/gradle/plugins/license-gradle-plugin/0.10.0/license-gradle-plugin-0.10.0.jar
Building project 'core' with Scala version 2.10.4
:ideaModule
:ideaProject

.........................................................................
这里省了一堆log
.........................................................................
:contrib:hadoop-producer:idea

BUILD SUCCESSFUL in 1m 7s
9 actionable tasks: 9 executed

gradle 会自动下载好 Kafka 项目依赖的jar包。

1、你在执行 gradle idea 命令的时候,可能会遇到以下的错误:

$ gradle idea
Starting a Gradle Daemon, 1 incompatible Daemon could not be reused, use --status for details
Building project 'core' with Scala version 2.10.4

FAILURE: Build failed with an exception.

* Where:
Build file 'D:\user\iteblog\kafka-0.8.2\build.gradle' line: 131

* What went wrong:
A problem occurred evaluating root project 'kafka-0.8.2'.
> Failed to apply plugin [class 'org.gradle.api.plugins.scala.ScalaBasePlugin']
   > No such property: useAnt for class: org.gradle.api.tasks.scala.ScalaCompileOptions

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED in 9s

我们只需要在 Kafka 源码的 build.gradle 文件里面最顶端添加以下的内容保存即可:

ScalaCompileOptions.metaClass.daemonServer = true
ScalaCompileOptions.metaClass.fork = true
ScalaCompileOptions.metaClass.useAnt = false
ScalaCompileOptions.metaClass.useCompileDaemon = false

2、gradle 默认使用的是 https://repo1.maven.org/maven2/ 源进行下载相关的 jar 依赖,这可能会很慢。我们可以将其修改为 阿里 的 Maven源,只需要将 build.gradle 文件里面的下面代码:

buildscript {
  repositories {
    mavenCentral()
  }
  apply from: file('gradle/buildscript.gradle'), to: buildscript
}

def slf4jlog4j='org.slf4j:slf4j-log4j12:1.7.6'

allprojects {
  apply plugin: 'idea'
  repositories {
    mavenCentral()
  }
}

修改成

buildscript {
  repositories {
    maven { url 'http://maven.aliyun.com/nexus/content/groups/public/' }
  }
  apply from: file('gradle/buildscript.gradle'), to: buildscript
}

def slf4jlog4j='org.slf4j:slf4j-log4j12:1.7.6'

allprojects {
  apply plugin: 'idea'
  repositories {
    maven { url 'http://maven.aliyun.com/nexus/content/groups/public/' }
  }
}

其他国内比较快的 Maven 仓库源可以参见:《推荐几个国内访问比较快的Maven仓库镜像》

运行完 gradle idea 命令之后,我们就可以通过 idea 打开 Kafka 源码了,已经不再有错误,如下:

Kafka源码阅读环境搭建
如果想及时了解Spark、Hadoop或者Hbase相关的文章,欢迎关注微信公共帐号:iteblog_hadoop

关于 Kafka 的源码编译请参见《Apache Kafka-0.8.1.1源码编译》

本博客文章除特别声明,全部都是原创!
转载本文请加上:转载自过往记忆(https://www.iteblog.com/)
本文链接: 【使用idea阅读Kafka源码】(https://www.iteblog.com/archives/2208.html)
喜欢 (14)
分享 (0)
发表我的评论
取消评论

表情
本博客评论系统带有自动识别垃圾评论功能,请写一些有意义的评论,谢谢!