2017-07-30 222 views
0

我们正在测试一个节点发生故障的情况下,3比如卡夫卡集群上,用2节点故障

复制因子删除实例后后卡夫卡消费者错误,消费是不断失败。

消费者使用卡夫卡流读取邮件

感谢

这是消费者的故障日志:

08:09:19.667 [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] ERROR o.a.k.c.c.i.ConsumerCoordinator - User provided listener org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener for group ComponentsActivityEventsStream failed on partition assignment 

org.apache.kafka.streams.errors.StreamsException: Store ComponentsActivityStore's change log (ComponentsActivityEventsStream-ComponentsActivityStore-changelog) does not contain partition 0 

    at org.apache.kafka.streams.processor.internals.StoreChangelogReader.validatePartitionExists(StoreChangelogReader.java:87) 

    at org.apache.kafka.streams.processor.internals.ProcessorStateManager.register(ProcessorStateManager.java:165) 

    at org.apache.kafka.streams.processor.internals.AbstractProcessorContext.register(AbstractProcessorContext.java:100) 

    at org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.init(RocksDBSegmentedBytesStore.java:110) 

... 
08:09:19.673 [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] INFO o.a.k.s.p.i.StreamThread - stream-thread [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] Shutting down 

08:09:19.674 [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] INFO o.a.k.c.p.KafkaProducer - Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. 

08:09:19.681 [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] INFO o.a.k.s.p.i.StreamThread - stream-thread [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] Removing all active tasks [] 

08:09:19.681 [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] INFO o.a.k.s.p.i.StreamThread - stream-thread [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] Removing all standby tasks [] 

08:09:19.681 [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] INFO o.a.k.s.p.i.StreamThread - stream-thread [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] Removing all standby tasks [] 

08:09:19.681 [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] INFO o.a.k.s.p.i.StreamThread - stream-thread [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] Stream thread shutdown complete 

08:09:19.681 [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] WARN o.a.k.s.p.i.StreamThread - stream-thread [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] Unexpected state transition from ASSIGNING_PARTITIONS to DEAD. 

08:09:19.681 [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] WARN i.a.k.BaseEventsStream - uncaught exception in stream thread ComponentsActivityEventsStream 

org.apache.kafka.streams.errors.StreamsException: stream-thread [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] Failed to rebalance. 

    at org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:589) 

    at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553) 

    at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527) 

Caused by: org.apache.kafka.streams.errors.StreamsException: Store ComponentsActivityStore's change log (ComponentsActivityEventsStream-ComponentsActivityStore-changelog) does not contain partition 0 

    at org.apache.kafka.streams.processor.internals.StoreChangelogReader.validatePartitionExists(StoreChangelogReader.java:87) 

    at org.apache.kafka.streams.processor.internals.ProcessorStateManager.register(ProcessorStateManager.java:165) 
+0

这个应用程序是否有其所有三个经纪人在其引导服务器列表中? –

+0

这确实是引导服务器列表,谢谢 –

+0

好吧,我会发布答案而不是评论,并请接受为正确的解决方案 –

回答

1

检查所有三家券商都在引导文件的服务器列表中列出。如果您只列出一个代理,并且它恰好是一个代理,那么消费者无法获取它需要的元数据,无法知道哪个节点是您主题中每个分区的领导者。如果无法获得有效的元数据响应,那么即使它们具有数据的活动副本,它也不会连接到其他两个代理中的任何一个。