AWS SQS可以在带有django-celery-results后端的Django项目中使用吗?

ct3nt3jp  于 2023-05-08  发布在  Go
关注(0)|答案(1)|浏览(125)

预警:有很多我不明白

我的需求

我需要能够得到celery 任务的结果。成功完成后,我需要将状态更改为'SUCCESS'
例如:
我需要能够得到的结果x + y后,执行add.delay(1,2)的任务如下。
myapp/tasks.py

from celery import shared_task
from time import sleep

@shared_task
def add(x, y):
    sleep(10)
    return x + y

AWS SQS是否适合我的需求?

我读了Celery的Using Amazon SQS,并在底部了解到它是关于结果的。

结果

Amazon Web Services家族中的多个产品可能是存储或发布结果的理想选择,但目前还没有这样的结果后端。

提问:

这是否意味着django-celery-results不能用于AWS SQS?

更多上下文

我在执行什么?
1.我查看我的AWS队列(显示消息为3)
1.在我的本地终端中,我执行celery -A ebdjango worker --loglevel=INFO(参见下面的celery输出)
1.在连接到Django项目的PyCharm Python控制台中,我执行r = add.delay(1,2)

  1. r是一个AsyncResult对象:
>>> r = add.delay(1,2)
>>> r
<AsyncResult: b69c4287-5c82-4873-aa8c-227547511233>

1.在AWS中,我的“可用消息”从3条变为4条
1.在本地,在我的终端中,什么都没有发生(我期望SQS在本地将消息发送回给我?这样不对吗?)
1.我检查r并看到:

>>> r.id
    'b69c4287-5c82-4873-aa8c-227547511233'
    >>> r.status
    'PENDING'
    >>> r.result
    >>> type(r.result)
    <class 'NoneType'>

ebdjango/settings.py

...
AWS_ACCESS_KEY_ID = "XXXXXXXXXXXXXXXXXXX"
AWS_SECRET_ACCESS_KEY = "YYYYYYYYYYYYYYYYYYYYYYYYYYY"

CELERY_BROKER_URL = "sqs://"
CELERY_BROKER_TRANSPORT_OPTIONS = {
    'region': 'us-west-2',
    'visibility_timeout': 3600,
    'predefined_queues': {
        'eb-celery-queue': {
            'url': 'https://sqs.us-west-2.amazonaws.com/12345678910/eb-celery-queue',
            'access_key_id': AWS_ACCESS_KEY_ID,
            'secret_access_key': AWS_SECRET_ACCESS_KEY,
        }
    }
}

CELERY_SEND_EVENTS = False
CELERY_ENABLE_REMOTE_CONTROL = False
CELERY_TASK_DEFAULT_QUEUE = 'eb-celery-queue'
CELERY_WORKER_CONCURRENCY = 1
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_CONTENT_ENCODING = 'utf-8'
CELERY_RESULT_BACKEND = 'django-db' <-- Note: I have django-celery-results installed and set

开始时的celery 产量:

(eb-virt) C:\Users\Jarad\Documents\PyCharm\DEVOPS\ebdjango>celery -A ebdjango worker --loglevel=INFO
[2021-08-27 14:35:31,914: WARNING/MainProcess] No hostname was supplied. Reverting to default 'None'

 -------------- celery@Inspiron v5.1.2 (sun-harmonics)
--- ***** -----
-- ******* ---- Windows-10-10.0.19041-SP0 2021-08-27 14:35:31
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app:         ebdjango:0x1d64e4d4630
- ** ---------- .> transport:   sqs://localhost//
- ** ---------- .> results:
- *** --- * --- .> concurrency: 1 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
 -------------- [queues]
                .> eb-celery-queue  exchange=eb-celery-queue(direct) key=eb-celery-queue

[tasks]
  . ebdjango.celery.debug_task
  . homepage.tasks.add
  . homepage.tasks.count_widgets
  . homepage.tasks.cu
  . homepage.tasks.mul
  . homepage.tasks.rename_widget
  . homepage.tasks.xsum

[2021-08-27 14:35:31,981: WARNING/MainProcess] No hostname was supplied. Reverting to default 'None'
[2021-08-27 14:35:31,981: INFO/MainProcess] Connected to sqs://localhost//
[2021-08-27 14:35:32,306: WARNING/MainProcess] ...
[2021-08-27 14:35:32,307: INFO/MainProcess] celery@Inspiron ready.

我确实注意到1)results:部分显示为空(就像它没有定义一样),2)task events是OFF的,这可能是因为SQS不支持任务事件,但我不确定。似乎我可以设置CELERY_SEND_EVENTS,它对task events输出没有影响。

vuktfyat

vuktfyat1#

为了能够在TaskResults条目中显示所有任务数据,您需要将其添加到'settings.py'

CELERY_RESULT_EXTENDED = True

相关问题