我正在尝试设置DAG以运行Docker映像,但在运行时遇到问题。
我使用的是official Airflow docker-compose file。
我的DAG文件:
from airflow import DAG
from airflow.providers.docker.operators.docker import DockerOperator
from datetime import datetime
with DAG (
"population",
start_date=datetime(2023, 1, 1),
schedule_interval='@daily',
catchup=False
) as dag:
task_a = DockerOperator (
task_id="task_a",
image='population:1.0',
command='python3 population.py',
docker_url='unix://var/run/docker.sock',
network_mode='host'
)
当我访问Airflow UI并运行DAG时,收到错误消息no such file or directory is found
。
错误:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/docker/operators/docker.py", line 411, in execute
self.cli = self._get_cli()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/docker/operators/docker.py", line 442, in _get_cli
base_url=self.docker_url, version=self.api_version, tls=tls_config, timeout=self.timeout
File "/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 197, in __init__
self._version = self._retrieve_server_version()
File "/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 222, in _retrieve_server_version
f'Error while fetching server API version: {e}'
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
[2023-02-20, 19:20:23 UTC] {taskinstance.py:1323} INFO - Marking task as FAILED. dag_id=population, task_id=task_a, execution_date=20230220T192021, start_date=20230220T192023, end_date=20230220T192023
[2023-02-20, 19:20:23 UTC] {standard_task_runner.py:105} ERROR - Failed to execute job 36 for task task_a (Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory')); 80)
[2023-02-20, 19:20:23 UTC] {local_task_job.py:208} INFO - Task exited with return code 1
[2023-02-20, 19:20:23 UTC] {taskinstance.py:2578} INFO - 0 downstream tasks scheduled from follow-on schedule check
我相信这与没有正确地在docker-compose.yml
文件中挂载卷有关,但不确定我需要在文件中的哪里指定卷。
我看到这个question已经问过了,但是没有指定卷行应该去哪里。
1条答案
按热度按时间uajslkp61#
找到了答案:
我遇到了这个post,它建议向
docker-compose.yml
文件添加一个新服务:我还补充道:
低于
x-airflow-common
然后,在DAG文件中:
而且它运行DAG没有问题。看起来这上面也有一个GitHub问题。