kafka客户端-无错误但不工作

6g8kf2rb  于 2021-06-06  发布在  Kafka
关注(0)|答案(2)|浏览(362)

我正在用python运行合流的kafka客户端。目前,我在尝试生成和消费消息时没有收到任何错误,但问题是生产者说它成功了,但是消费者找不到任何消息。
我已经创建了一个主题,这是我正在使用的类:

from confluent_kafka import Producer, Consumer
from config import config
import json

class Kafka:
    """
    Kafka Handler.
    """

    def __init__(self, kafka_brokers_sasl, api_key):
        """
        Arguments:
            kafka_brokers_sasl {str} -- String containing kafka brokers separated by comma (no spaces)
            api_key {str} -- Kafka Api Key
        """

        self.driver_options = {
            'bootstrap.servers': kafka_brokers_sasl,
            'sasl.mechanisms': 'PLAIN',
            'security.protocol': 'SASL_SSL',
            'sasl.username': 'token',
            'sasl.password': api_key,
            'log.connection.close' : False,
            #'debug': 'all'
        }

        self.producer_options = {
            'client.id': 'kafka-python-console-sample-producer'
        }
        self.producer_options.update(self.driver_options)

        self.consumer_options = {
            'client.id': 'kafka-python-console-sample-consumer',
            'group.id': 'kafka-python-console-sample-group'
        }
        self.consumer_options.update(self.driver_options)

        self.running = None

    def stop(self):
        self.running = False

    def delivery_report(self, err, msg):
        """ Called once for each message produced to indicate delivery result.
            Triggered by poll() or flush(). """
        if err is not None:
            print('Message delivery failed: {}'.format(err))
        else:
            print('Message delivered to {} [{}]'.format(msg.topic(), msg.partition()))

    def produce(self, topic, data): # Function for producing/uploading data to a Kafka topic

        p = Producer(self.producer_options)

        print("Running?")

        # Asynchronously produce a message, the delivery report callback will be triggered from poll() above, or flush() below, when the message has been successfully delivered or failed permanently.
        p.produce(topic, data, callback=self.delivery_report)

        # Wait for any outstanding messages to be delivered and delivery report callbacks to be triggered.
        p.flush()
        print("Done?")

    def consume(self, topic, method_class=None): # Function for consuming/reading data from a Kafka topic. Works as a listener and triggers the run() function on a method_class
        print("raaa")

        kafka_consumer = Consumer(self.consumer_options)

        kafka_consumer.subscribe([topic])

        # Now loop on the consumer to read messages
        print("Running?")
        self.running = True
        while self.running:
            msg = kafka_consumer.poll()

            print(msg)

            if msg is not None and msg.error() is None:
                print('Message consumed: topic={0}, partition={1}, offset={2}, key={3}, value={4}'.format(
                    msg.topic(),
                    msg.partition(),
                    msg.offset(),
                    msg.key().decode('utf-8'),
                    msg.value().decode('utf-8')))
            else:
                print('No messages consumed')

        print("Here?")
        kafka_consumer.unsubscribe()
        kafka_consumer.close()
        print("Ending?")

mock = {'yas': 'yas', 'yas2': 'yas2'}
kafka = Kafka(config['kafka']['kafka_brokers_sasl'], config['kafka']['api_key'])
kafka.produce(config['kafka']['topic'], json.dumps(mock))
kafka.consume(config['kafka']['topic'])

运行这个我得到了指纹:

Running?
Message delivered to DANIEL_TEST [0]
Done?
raaa
Running?
<cimpl.Message object at 0x104e4c390>
No messages consumed
4szc88ey

4szc88ey1#

我也有同样的问题。这个 driver_options 必须包含ssl证书路径,因此必须设置 'ssl.ca.location': '/etc/pki/tls/cert.pem' 或此处记录的等效位置:https://github.com/ibm-messaging/event-streams-samples/blob/master/kafka-python-console-sample/app.py#l75
然后成功了!

oalqel3c

oalqel3c2#

我不是python方面的Maven,但看起来您是在生成消息之后才开始使用它的? kafka.produce(config['kafka']['topic'], json.dumps(mock)) kafka.consume(config['kafka']['topic']) 您需要在调用product函数之前调用consume函数,因为当您启动一个新的使用者时,该使用者的默认偏移量将是最新的。因此,例如,如果您在偏移量5处生成了一条消息,然后启动了一个新的消费者,那么默认情况下,消费者偏移量将在偏移量6处,并且它不会使用在偏移量5处生成的消息。
解决方案是在生成任何内容之前开始消费,或者将consumer配置设置为从偏移量的开头消费消息。这可以通过设置 auto.offset.resetearliest 但我认为第一种解决办法更简单。

相关问题