欢迎关注Hadoop、Spark、Flink、Hive、Hbase、Flume等大数据资料分享微信公共账号:iteblog_hadoop
  1. 文章总数:978
  2. 浏览总数:11,981,823
  3. 评论:3939
  4. 分类目录:106 个
  5. 注册用户数:6130
  6. 最后更新:2018年12月15日
过往记忆博客公众号iteblog_hadoop
欢迎关注微信公众号:
iteblog_hadoop
大数据技术博客公众号bigdata_ai
大数据猿:
bigdata_ai

如何移动Kafka部分分区的数据

  在《Kafka集群扩展以及重新分布分区》文章中我们介绍了如何重新分布分区,在那里面我们基本上把所有的分区全部移动了,其实我们完全没必要移动所有的分区,而移动其中部分的分区。比如我们想把Broker 1与Broker 7上面的分区数据互换,如下图所示:

可以看出,只有Broker 1与Broker 7上面的分区做了移动。来看看移动分区之前各个分区以及Leader分布情况:

[iteblog@www.iteblog.com kafka]$ ./bin/kafka-topics.sh --topic iteblog --describe   \
		--zookeeper www.iteblog.com
Topic:iteblog	PartitionCount:7	ReplicationFactor:2	Configs:
	Topic: iteblog	Partition: 0	Leader: 1	Replicas: 1,4	Isr: 1,4
	Topic: iteblog	Partition: 1	Leader: 7	Replicas: 7,3	Isr: 3,7
	Topic: iteblog	Partition: 2	Leader: 5	Replicas: 5,7	Isr: 5,7
	Topic: iteblog	Partition: 3	Leader: 6	Replicas: 6,1	Isr: 1,6
	Topic: iteblog	Partition: 4	Leader: 4	Replicas: 4,2	Isr: 4,2
	Topic: iteblog	Partition: 5	Leader: 2	Replicas: 2,5	Isr: 5,2
	Topic: iteblog	Partition: 6	Leader: 3	Replicas: 3,6	Isr: 3,6

将Broker 1与Broker 7上面的分区互换,我们配置了以下的replica计划:

{
    "version": 1,
    "partitions": [
        {
            "topic": "iteblog",
            "partition": 0,
            "replicas": [
                7,
                4
            ]
        },
        {
            "topic": "iteblog",
            "partition": 1,
            "replicas": [
                1,
                3
            ]
        },
        {
            "topic": "iteblog",
            "partition": 3,
            "replicas": [
                6,
                7
            ]
        },
        {
            "topic": "iteblog",
            "partition": 2,
            "replicas": [
                5,
                1
            ]
        }
    ]
}

可以看到只涉及到Broker 1与Broker 7上面的分区,将上面的Json数据保存到iteblog.json文件里面,然后执行它:

[iteblog@www.iteblog.com kafka]$ bin/kafka-reassign-partitions.sh     \
				--zookeeper www.iteblog.com 		 \
				--reassignment-json-file iteblog.json --execute
Current partition replica assignment

{"version":1,"partitions":[{"topic":"iteblog","partition":3,"replicas":[6,1]},
{"topic":"iteblog","partition":5,"replicas":[2,5]},
{"topic":"iteblog","partition":4,"replicas":[4,2]},
{"topic":"iteblog","partition":0,"replicas":[1,4]},
{"topic":"iteblog","partition":6,"replicas":[3,6]},
{"topic":"iteblog","partition":1,"replicas":[7,3]},
{"topic":"iteblog","partition":2,"replicas":[5,7]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions {"version":1,"partitions":[
{"topic":"iteblog","partition":0,"replicas":[7,4]},
{"topic":"iteblog","partition":1,"replicas":[1,3]},
{"topic":"iteblog","partition":3,"replicas":[6,7]},
{"topic":"iteblog","partition":2,"replicas":[5,1]}]}

在运行的时候可以使用verify参数来看看reassign计划是否执行完成:

[iteblog@www.iteblog.com kafka]$ bin/kafka-reassign-partitions.sh      \
				--zookeeper www.iteblog.com                   \
			        --reassignment-json-file iteblog.json --verify
Status of partition reassignment:
Reassignment of partition [iteblog,0] is still in progress
Reassignment of partition [iteblog,1] is still in progress
Reassignment of partition [iteblog,3] is still in progress
Reassignment of partition [iteblog,2] is still in progress


[iteblog@www.iteblog.com kafka]$ bin/kafka-reassign-partitions.sh      \
			       --zookeeper www.iteblog.com                  \
			       --reassignment-json-file iteblog.json --verify
Status of partition reassignment:
Reassignment of partition [iteblog,0] completed successfully
Reassignment of partition [iteblog,1] completed successfully
Reassignment of partition [iteblog,3] completed successfully
Reassignment of partition [iteblog,2] completed successfully

当上面分区全部成功移动完成,然后可以我们再来看看各个分区的分布情况:

[iteblog@www.iteblog.com kafka]$ bin/kafka-topics.sh --topic iteblog --describe \
		--zookeeper www.iteblog.com
Topic:iteblog	PartitionCount:7	ReplicationFactor:2	Configs:
	Topic: iteblog	Partition: 0	Leader: 7	Replicas: 7,4	Isr: 4,7
	Topic: iteblog	Partition: 1	Leader: 1	Replicas: 1,3	Isr: 3,1
	Topic: iteblog	Partition: 2	Leader: 5	Replicas: 5,1	Isr: 5,1
	Topic: iteblog	Partition: 3	Leader: 6	Replicas: 6,7	Isr: 6,7
	Topic: iteblog	Partition: 4	Leader: 4	Replicas: 4,2	Isr: 4,2
	Topic: iteblog	Partition: 5	Leader: 2	Replicas: 2,5	Isr: 5,2
	Topic: iteblog	Partition: 6	Leader: 3	Replicas: 3,6	Isr: 3,6

可以看出已经将Broker 1与Broker 7上面的分区互换了。

本博客文章除特别声明,全部都是原创!
转载本文请加上:转载自过往记忆(https://www.iteblog.com/)
本文链接: 【如何移动Kafka部分分区的数据】(https://www.iteblog.com/archives/1613.html)
喜欢 (3)
分享 (0)
发表我的评论
取消评论

表情
本博客评论系统带有自动识别垃圾评论功能,请写一些有意义的评论,谢谢!