尝试从气流DAG运行Docker映像时出现问题

k5ifujac  于 2023-02-21  发布在  Docker
关注(0)|答案(1)|浏览(179)

我正在尝试设置DAG以运行Docker映像,但在运行时遇到问题。
我使用的是official Airflow docker-compose file
我的DAG文件:

from airflow import DAG
from airflow.providers.docker.operators.docker import DockerOperator

from datetime import datetime

with DAG (
    "population", 
    start_date=datetime(2023, 1, 1), 
    schedule_interval='@daily', 
    catchup=False
) as dag:

    task_a = DockerOperator (
        task_id="task_a",
        image='population:1.0',
        command='python3 population.py',
        docker_url='unix://var/run/docker.sock',
        network_mode='host'
    )

当我访问Airflow UI并运行DAG时,收到错误消息no such file or directory is found

错误:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/docker/operators/docker.py", line 411, in execute
    self.cli = self._get_cli()
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/docker/operators/docker.py", line 442, in _get_cli
    base_url=self.docker_url, version=self.api_version, tls=tls_config, timeout=self.timeout
  File "/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
  File "/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 222, in _retrieve_server_version
    f'Error while fetching server API version: {e}'
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
[2023-02-20, 19:20:23 UTC] {taskinstance.py:1323} INFO - Marking task as FAILED. dag_id=population, task_id=task_a, execution_date=20230220T192021, start_date=20230220T192023, end_date=20230220T192023
[2023-02-20, 19:20:23 UTC] {standard_task_runner.py:105} ERROR - Failed to execute job 36 for task task_a (Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory')); 80)
[2023-02-20, 19:20:23 UTC] {local_task_job.py:208} INFO - Task exited with return code 1
[2023-02-20, 19:20:23 UTC] {taskinstance.py:2578} INFO - 0 downstream tasks scheduled from follow-on schedule check

我相信这与没有正确地在docker-compose.yml文件中挂载卷有关,但不确定我需要在文件中的哪里指定卷。
我看到这个question已经问过了,但是没有指定卷行应该去哪里。

uajslkp6

uajslkp61#

找到了答案:
我遇到了这个post,它建议向docker-compose.yml文件添加一个新服务:

services:
  docker-proxy:
    image: bobrik/socat
    command: "TCP4-LISTEN:2375,fork,reuseaddr UNIX-CONNECT:/var/run/docker.sock"
    ports:
      - "2376:2375"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

我还补充道:

- /var/run/docker.sock:/var/run/docker.sock

低于x-airflow-common
然后,在DAG文件中:

docker_url='tcp://docker-proxy:2375',

而且它运行DAG没有问题。看起来这上面也有一个GitHub问题。

相关问题