dify 使用Redis集群的Celery,抛出异常：ResponseError("'BRPOP'命令的键必须在同一插槽中")

smdnsysy 于 6个月前发布在 Redis

关注(0)|答案(1)|浏览(73)

Self Checks

This is only for bug report, if you would like to ask a question, please head to Discussions .
I have searched for existing issues search for existing issues , including closed ones.
I confirm that I am using English to submit this report (我已阅读并同意 Language Policy ).
Please do not modify this template :) and fill in all the required fields.

Dify version

0.6.8

Cloud or Self Hosted

Self Hosted (Source)

Steps to reproduce

set CELERY_BROKER_URL=redis://:password@host:port/db
celery -A app.celery worker -P gevent -c 1 -Q dataset,generation,mail --loglevel INFO
[2023-07-31 12:58:08,831: INFO/MainProcess] Connected to redis://:**@localhost:6379/1
[2023-07-31 12:58:09,873: INFO/MainProcess] mingle: searching for neighbors
[2023-07-31 12:58:09,873: INFO/MainProcess] mingle: all alone
[2023-07-31 12:58:09,886: INFO/MainProcess] pidbox: Connected to redis://:**@localhost:6379/1.
[2023-07-31 12:58:09,890: INFO/MainProcess] celery@TAKATOST.lan ready.
[2024-05-17 06:19:34,129: INFO/MainProcess] Connected to redis://:**@host:6379/1
[2024-05-17 06:19:34,417: INFO/MainProcess] mingle: searching for neighbors
[2024-05-17 06:19:35,437: CRITICAL/MainProcess] Unrecoverable error: ResponseError("'BRPOP' command keys must in same slot")
Traceback (most recent call last):
File "/Users/wangvic/miniconda3/lib/python3.11/site-packages/celery/worker/worker.py", line 202, in start
self.blueprint.start(self)
File "/Users/wangvic/miniconda3/lib/python3.11/site-packages/celery/bootsteps.py", line 116, in start
step.start(parent)
File "/Users/wangvic/miniconda3/lib/python3.11/site-packages/celery/worker/consumer/consumer.py", line 340, in start
self.sync(c)
File "/Users/wangvic/miniconda3/lib/python3.11/site-packages/celery/worker/consumer/consumer.py", line 41, in sync
replies = self.send_hello(c)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/wangvic/miniconda3/lib/python3.11/site-packages/celery/app/control.py, line 389, in hello
return self._request('hello', from_node=from_node, revoked=revoked)
File "/Users/wangvic/miniconda3/lib/python3.11/site-packages/celery/app/control.py, line 106, in _request
return self._prepare(self.app.control.broadcast(
File "/Users/wangvic/miniconda3/lib/python3.11/site-packages/celery/app/control.py", line 776, in broadcast
return self.mailbox(conn)._broadcast(
File "/Users/wangvic/miniconda3/lib/python3.11/site-packages/kombu/pidbox.py, line 346, in _broadcast
return self._collect(reply_ticket, limit=limit,
这个错误是由于在使用Redis的BRPOP命令时，键必须位于同一个槽中。要解决这个问题，你需要确保所有的键都使用相同的槽。你可以通过设置redis.conf文件中的cluster-enabled选项为yes,并使用CLUSTER NODES命令来配置槽分配。

dify

来源：https://github.com/langgenius/dify/issues/4472

1条答案

按热度按时间

bvjveswy1#

你遇到的错误是由于Redis Cluster的要求，即参与多键操作的BRPOP等所有键必须位于同一个哈希槽中。要解决这个问题，你可以在队列名称中使用"哈希标签",以确保它们被分配到相同的槽位。
例如，将你的队列重命名为包含通用哈希标签：{myapp}:dataset、{myapp}:generation、{myapp}:mail。这确保了与特定队列相关的所有键都在同一个Redis Cluster槽位中，避免了ResponseError。
以下是如何使用这些名称配置Celery的方法：

app.conf.task_default_queue = '{myapp}:default'
app.conf.task_queues = {
    '{myapp}:default': {
        'exchange': '{myapp}:default',
        'routing_key': '{myapp}:default',
    },
}

确保所有生产者和消费者都使用更新后的队列名称。这种方法需要更改代码库中定义和使用队列名称的地方。有关更多详细信息，请参阅Redis Cluster规范和Celery关于路由和队列的文档。

详细信息

来源

此响应旨在帮助你节省时间并提高效率。它不是精确的解决方案,而是你自己研究的一个起点。
帮助我变得更有用！如果这有帮助，请留下一个👍，如果无关紧要，请留下一个👎。

要继续对话，请提及@dosu。*

赞(0）回复(0）举报 6个月前

我来回答