欢迎关注Hadoop、Spark、Flink、Hive、Hbase、Flume等大数据资料分享微信公共账号:iteblog_hadoop
  1. 文章总数:978
  2. 浏览总数:11,982,149
  3. 评论:3939
  4. 分类目录:106 个
  5. 注册用户数:6130
  6. 最后更新:2018年12月15日
过往记忆博客公众号iteblog_hadoop
欢迎关注微信公众号:
iteblog_hadoop
大数据技术博客公众号bigdata_ai
大数据猿:
bigdata_ai

Hadoop1.x程序升级到Hadoop2.x需要的依赖库

  根据官方文档(Apache Hadoop MapReduce - Migrating from Apache Hadoop 1.x to Apache Hadoop 2.x:http://hadoop.apache.org/docs/r2.2.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html)所述,Hadoop2.x是对Hadoop1.x程序兼容的,由于Hadoop2.x对Hadoop1.x做了重大的结构调整,很多程序依赖库被拆分了,所以以前(Hadoop1.x)的依赖库不再可用(不是说API不可用,而是api存放的包不一样),如果需要在Hadoop2.X上面编译,请用下面的依赖库:

<dependencies>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-mapreduce-client-core</artifactId>
            <version>2.2.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>2.2.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-mapreduce-client-common</artifactId>
            <version>2.2.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
            <version>2.2.0</version>
        </dependency>
</dependencies>
记得加上

<dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-mapreduce-client-common</artifactId>
        <version>2.2.0</version>
</dependency>
<dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
        <version>2.2.0</version>
</dependency>

否则运行程序的时候将会出现一下的异常:

Exception in thread "main" java.io.IOException: Cannot initialize Cluster. 
    Please check your configuration for mapreduce.framework.name and the 
    correspond server addresses.
	at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
	at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
	at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
	at org.apache.hadoop.mapred.JobClient.init(JobClient.java:465)
	at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:444)
	at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:826)
	at com.wyp.hadoop.MaxTemperature.main(MaxTemperature.java:41)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke
                           (NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke
                           (DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)

一个简单的Mapreduce程序请参见《编写简单的Mapreduce程序并部署在Hadoop2.2.0上运行》

  如果你不是用Maven管理依赖的类库,你可以分别在${HADOOP_HOME}/share/hadoop/common和${HADOOP_HOME}/share/hadoop/mapreduce目录里面以下几个依赖库,分别是hadoop-mapreduce-client-core-2.2.0.jar、hadoop-common-2.2.0.jar、hadoop-mapreduce-client-common-2.2.0.jar、hadoop-mapreduce-client-jobclient-2.2.0.jar和上面的四个依赖正好对应。
本博客文章除特别声明,全部都是原创!
转载本文请加上:转载自过往记忆(https://www.iteblog.com/)
本文链接: 【Hadoop1.x程序升级到Hadoop2.x需要的依赖库】(https://www.iteblog.com/archives/840.html)
喜欢 (2)
分享 (0)
发表我的评论
取消评论

表情
本博客评论系统带有自动识别垃圾评论功能,请写一些有意义的评论,谢谢!
(3)个小伙伴在吐槽
  1. 感谢作者,解决了自己的升级问题
    zhaoxxxqqq2013-12-01 21:02 回复