kafka常见问题

如何解决kafka常见问题

• 报错1

[root@slave1 data_mocker]# kafka-console-consumer.sh --bootstrap-server slave1:9092 --topic topic_db
[2023-12-19 18:31:12,770] WARN [Consumer clientId=consumer-1, groupId=console-consumer-92215] Error while fetching metadata with correlation id 2 : {topic_db=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2023-12-19 18:31:12,954] WARN [Consumer clientId=consumer-1, groupId=console-consumer-92215] Error while fetching metadata with correlation id 6 : {topic_db=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

• 解决方案：主题没有创建，需先创建

• 报错2

[root@slave1 data_mocker]# kafka-topics.sh --create --bootstrap-server master:9092 --topic topic_db --partitions 1 --replication-factor 1
[2023-12-19 18:32:48,828] WARN Couldn't resolve server master:9092 from bootstrap.servers as DNS resolution failed for master (org.apache.kafka.clients.ClientUtils)
Exception in thread "main" org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:386)
	at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:55)
	at kafka.admin.TopicCommand$AdminClientTopicService$.createAdminClient(TopicCommand.scala:150)
	at kafka.admin.TopicCommand$AdminClientTopicService$.apply(TopicCommand.scala:154)
	at kafka.admin.TopicCommand$.main(TopicCommand.scala:55)
	at kafka.admin.TopicCommand.main(TopicCommand.scala)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
	at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:88)
	at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:47)
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:346)
	... 5 more

• 解决方案

# 错误原因：没有连接上主机
# 配置如下
主机名：slave1
域名映射：192.168.128.101 slave1
将master改为slave1

• 报错3

[root@slave1 data_mocker]# kafka-topics.sh --create --bootstrap-server slave1:9092 --topic topic_db --partitions 1 --replication-factor 1
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Error while executing topic command : Topic topic_db already exists
[2023-12-19 18:38:38,299] ERROR java.lang.IllegalArgumentException: Topic topic_db already exists
	at kafka.admin.TopicCommand$AdminClientTopicService.createTopic(TopicCommand.scala:177)
	at kafka.admin.TopicCommand$TopicService.createTopic(TopicCommand.scala:134)
	at kafka.admin.TopicCommand$TopicService.createTopic$(TopicCommand.scala:129)
	at kafka.admin.TopicCommand$AdminClientTopicService.createTopic(TopicCommand.scala:157)
	at kafka.admin.TopicCommand$.main(TopicCommand.scala:60)
	at kafka.admin.TopicCommand.main(TopicCommand.scala)
 (kafka.admin.TopicCommand$)

• 解决方案

# 主题topic_db已经存在，需先删除后再重新创建
# 方式1：
# 查看所有主题
kafka-topics.sh --zookeeper slave1:2181,slave1:2181,slave1:2181/kafka --list
# 查看主题信息
kafka-topics.sh --describe --zookeeper slave1:2181,slave1:2181,slave1:2181/kafka --topic topic_db
# 删除主题
kafka-topics.sh --delete --topic topic_db --zookeeper slave1:2181,slave1:2181,slave1:2181/kafka
# 删除该目录下的指定主题
# 在/usr/local/software/kafka-2.2.1/config/server.properties中配置log.dirs=/opt/software/kafka-2.2.1/data
rm -rf /opt/software/kafka-2.2.1/data/topic_db*
rm -rf /opt/software/kafka-2.2.1/data/__consumer_offsets*
# 彻底删除主题
# 启动zookeeper客户端
cd /usr/local/software/zookeeper-3.6.3/
[root@slave1 zookeeper-3.6.3]# ./bin/zkCli.sh
# 查看
[zk: localhost:2181(CONNECTED) 8] ls /brokers/topics
[__consumer_offsets, topic_db]
# 删除
[zk: localhost:2181(CONNECTED) 9] deleteall /brokers/topics/topic_db
[zk: localhost:2181(CONNECTED) 10] ls /brokers/topics
[__consumer_offsets]
[zk: localhost:2181(CONNECTED) 11] deleteall /brokers/topics/__consumer_offsets
[zk: localhost:2181(CONNECTED) 12] ls /brokers/topics
[]
# 最后重启zookeeper和kafka

# 方式2：
vim /usr/local/software/kafka-2.2.1/config/server.properties
# 添加配置如下，并重启kafka
delete.topic.enable=true
# 使用命令删除
kafka-topics.sh --delete --topic topic_db --zookeeper slave1:2181,slave1:2181,slave1:2181/kafka
# 最后重启zookeeper和kafka