pyspark Spark连接类型未显示在Airflow中

dzhpxtsq  于 2022-11-01  发布在  Spark
关注(0)|答案(1)|浏览(171)

我有一个docker-compose文件，其中定义了Airflow，Spark，postgreSQL和Redis服务。当我执行docker-compose并打开Airflow UI时，我尝试添加一个Spark连接类型，这样我就可以在Docker上的Airflow中运行一个spark作业。但是Spark并没有显示为连接类型，正如你所看到的：

这是我的docker-compose文件：

  2. # Feel free to modify this file to suit your needs.
  4. ---
  5. version: '3'
  6. x-airflow-common:
  7.   &airflow-common
  8.   # In order to add custom dependencies or upgrade provider packages you can use your extended image.
  9.   # Comment the image line, place your Dockerfile in the directory where you placed the docker-compose.yaml
  10.   # and uncomment the "build" line below, Then run `docker-compose build` to build the images.
  11.   image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.2.3}
  12.   # build: .
  13.   environment:
  14.     &airflow-common-env
  15.     AIRFLOW__CORE__EXECUTOR: CeleryExecutor
  16.     AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
  17.     AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow
  18.     AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
  19.     AIRFLOW__CORE__FERNET_KEY: ''
  20.     AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
  21.     AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
  22.     AIRFLOW__API__AUTH_BACKEND: 'airflow.api.auth.backend.basic_auth'
  23.     ADDITIONAL_AIRFLOW_EXTRAS: apache.spark
  24.     _PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}
  25.   volumes:
  26.     - ./dags:/opt/airflow/dags
  27.     - ./logs:/opt/airflow/logs
  28.     - ./plugins:/opt/airflow/plugins
  29.   user: "${AIRFLOW_UID:-50000}:0"
  30.   depends_on:
  31.     &airflow-common-depends-on
  32.     redis:
  33.       condition: service_healthy
  34.     postgres:
  35.       condition: service_healthy
  37. services:
  38.   postgres:
  39.     image: postgres:13
  40.     environment:
  41.       POSTGRES_USER: airflow
  42.       POSTGRES_PASSWORD: airflow
  43.       POSTGRES_DB: airflow
  44.     volumes:
  45.       - postgres-db-volume:/var/lib/postgresql/data
  46.     healthcheck:
  47.       test: ["CMD", "pg_isready", "-U", "airflow"]
  48.       interval: 5s
  49.       retries: 5
  50.     restart: always
  52.   redis:
  53.     image: redis:latest
  54.     expose:
  55.       - 6379
  56.     healthcheck:
  57.       test: ["CMD", "redis-cli", "ping"]
  58.       interval: 5s
  59.       timeout: 30s
  60.       retries: 50
  61.     restart: always
  63.   airflow-webserver:
  64.     <<: *airflow-common
  65.     command: webserver
  66.     ports:
  67.       - '8282:8080'
  68.     healthcheck:
  69.       test: ["CMD", "curl", "--fail", "http://localhost:8080/health"]
  70.       interval: 10s
  71.       timeout: 10s
  72.       retries: 5
  73.     restart: always
  74.     depends_on:
  75.       <<: *airflow-common-depends-on
  76.       airflow-init:
  77.         condition: service_completed_successfully
  79.   airflow-scheduler:
  80.     <<: *airflow-common
  81.     command: scheduler
  82.     healthcheck:
  83.       test: ["CMD-SHELL", 'airflow jobs check --job-type SchedulerJob --hostname "$${HOSTNAME}"']
  84.       interval: 10s
  85.       timeout: 10s
  86.       retries: 5
  87.     restart: always
  88.     depends_on:
  89.       <<: *airflow-common-depends-on
  90.       airflow-init:
  91.         condition: service_completed_successfully
  93.   airflow-worker:
  94.     <<: *airflow-common
  95.     command: celery worker
  96.     healthcheck:
  97.       test:
  98.         - "CMD-SHELL"
  99.         - 'celery --app airflow.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}"'
  100.       interval: 10s
  101.       timeout: 10s
  102.       retries: 5
  103.     environment:
  104.       <<: *airflow-common-env
  105.       # Required to handle warm shutdown of the celery workers properly
  106.       # See https://airflow.apache.org/docs/docker-stack/entrypoint.html#signal-propagation
  107.       DUMB_INIT_SETSID: "0"
  108.     restart: always
  109.     depends_on:
  110.       <<: *airflow-common-depends-on
  111.       airflow-init:
  112.         condition: service_completed_successfully
  114.   airflow-triggerer:
  115.     <<: *airflow-common
  116.     command: triggerer
  117.     healthcheck:
  118.       test: ["CMD-SHELL", 'airflow jobs check --job-type TriggererJob --hostname "$${HOSTNAME}"']
  119.       interval: 10s
  120.       timeout: 10s
  121.       retries: 5
  122.     restart: always
  123.     depends_on:
  124.       <<: *airflow-common-depends-on
  125.       airflow-init:
  126.         condition: service_completed_successfully
  128.   airflow-init:
  129.     <<: *airflow-common
  130.     entrypoint: /bin/bash
  131.     # yamllint disable rule:line-length
  132.     command:
  133.       - -c
  134.       - |
  135.         function ver() {
  136.           printf "%04d%04d%04d%04d" $${1//./ }
  137.         }
  138.         airflow_version=$$(gosu airflow airflow version)
  139.         airflow_version_comparable=$$(ver $${airflow_version})
  140.         min_airflow_version=2.2.0
  141.         min_airflow_version_comparable=$$(ver $${min_airflow_version})
  142.         if (( airflow_version_comparable < min_airflow_version_comparable )); then
  143.           echo
  144.           echo -e "\033[1;31mERROR!!!: Too old Airflow version $${airflow_version}!\e[0m"
  145.           echo "The minimum Airflow version supported: $${min_airflow_version}. Only use this or higher!"
  146.           echo
  147.           exit 1
  148.         fi
  149.         if [[ -z "${AIRFLOW_UID}" ]]; then
  150.           echo
  151.           echo -e "\033[1;33mWARNING!!!: AIRFLOW_UID not set!\e[0m"
  152.           echo "If you are on Linux, you SHOULD follow the instructions below to set "
  153.           echo "AIRFLOW_UID environment variable, otherwise files will be owned by root."
  154.           echo "For other operating systems you can get rid of the warning with manually created .env file:"
  155.           echo "    See: https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html#setting-the-right-airflow-user"
  156.           echo
  157.         fi
  158.         one_meg=1048576
  159.         mem_available=$$(($$(getconf _PHYS_PAGES) * $$(getconf PAGE_SIZE) / one_meg))
  160.         cpus_available=$$(grep -cE 'cpu[0-9]+' /proc/stat)
  161.         disk_available=$$(df / | tail -1 | awk '{print $$4}')
  162.         warning_resources="false"
  163.         if (( mem_available < 4000 )) ; then
  164.           echo
  165.           echo -e "\033[1;33mWARNING!!!: Not enough memory available for Docker.\e[0m"
  166.           echo "At least 4GB of memory required. You have $$(numfmt --to iec $$((mem_available * one_meg)))"
  167.           echo
  168.           warning_resources="true"
  169.         fi
  170.         if (( cpus_available < 2 )); then
  171.           echo
  172.           echo -e "\033[1;33mWARNING!!!: Not enough CPUS available for Docker.\e[0m"
  173.           echo "At least 2 CPUs recommended. You have $${cpus_available}"
  174.           echo
  175.           warning_resources="true"
  176.         fi
  177.         if (( disk_available < one_meg * 10 )); then
  178.           echo
  179.           echo -e "\033[1;33mWARNING!!!: Not enough Disk space available for Docker.\e[0m"
  180.           echo "At least 10 GBs recommended. You have $$(numfmt --to iec $$((disk_available * 1024 )))"
  181.           echo
  182.           warning_resources="true"
  183.         fi
  184.         if [[ $${warning_resources} == "true" ]]; then
  185.           echo
  186.           echo -e "\033[1;33mWARNING!!!: You have not enough resources to run Airflow (see above)!\e[0m"
  187.           echo "Please follow the instructions to increase amount of resources available:"
  188.           echo "   https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html#before-you-begin"
  189.           echo
  190.         fi
  191.         mkdir -p /sources/logs /sources/dags /sources/plugins
  192.         chown -R "${AIRFLOW_UID}:0" /sources/{logs,dags,plugins}
  193.         exec /entrypoint airflow version
  194.     # yamllint enable rule:line-length
  195.     environment:
  196.       <<: *airflow-common-env
  197.       _AIRFLOW_DB_UPGRADE: 'true'
  198.       _AIRFLOW_WWW_USER_CREATE: 'true'
  199.       _AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
  200.       _AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}
  201.     user: "0:0"
  202.     volumes:
  203.       - .:/sources
  205.   airflow-cli:
  206.     <<: *airflow-common
  207.     profiles:
  208.       - debug
  209.     environment:
  210.       <<: *airflow-common-env
  211.       CONNECTION_CHECK_MAX_COUNT: "0"
  212.     # Workaround for entrypoint issue. See: https://github.com/apache/airflow/issues/16252
  213.     command:
  214.       - bash
  215.       - -c
  216.       - airflow
  218.   flower:
  219.     <<: *airflow-common
  220.     command: celery flower
  221.     ports:
  222.       - 5555:5555
  223.     healthcheck:
  224.       test: ["CMD", "curl", "--fail", "http://localhost:5555/"]
  225.       interval: 10s
  226.       timeout: 10s
  227.       retries: 5
  228.     restart: always
  229.     depends_on:
  230.       <<: *airflow-common-depends-on
  231.       airflow-init:
  232.         condition: service_completed_successfully
  234.   spark:
  235.     image: docker.io/bitnami/spark:3
  236.     environment:
  237.       - SPARK_MODE=master
  238.       - SPARK_RPC_AUTHENTICATION_ENABLED=no
  239.       - SPARK_RPC_ENCRYPTION_ENABLED=no
  240.       - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
  241.       - SPARK_SSL_ENABLED=no
  242.     ports:
  243.       - '8080:8080'
  244.     volumes:
  245.       - ./spark-apps:/opt/spark-apps
  246.   spark-worker:
  247.     image: docker.io/bitnami/spark:3
  248.     environment:
  249.       - SPARK_MODE=worker
  250.       - SPARK_MASTER_URL=spark://spark:7077
  251.       - SPARK_WORKER_MEMORY=1G
  252.       - SPARK_WORKER_CORES=1
  253.       - SPARK_RPC_AUTHENTICATION_ENABLED=no
  254.       - SPARK_RPC_ENCRYPTION_ENABLED=no
  255.       - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
  256.       - SPARK_SSL_ENABLED=no
  257.     ports:
  258.     - '8081:8081'
  260. volumes:
  261.   postgres-db-volume:
pyspark

1条答案

按热度按时间
rsl1atfo

rsl1atfo1#

我也有过类似的经历，试图设置一个Databricks连接，并通过添加一个Dockerfile来解决这个问题--添加Spark提供程序的方法也是一样的。
我花了相当多的时间在网上挖掘，但是在&airflow_common下的docker-compose文件中的注解中跟随了面包屑，这使我找到了这个文档
我使用Docker桌面版本4.12.0和VS代码版本1.62.3作为参考。
操作步骤：
1.在您保存docker-compose.yaml模板的同一目录中，您需要添加另一个名为Dockerfile（即无文件扩展名） 的文件
1.打开Dockerfile并在保存前添加以下代码行 （根据需要更改气流和Spark版本）：

  1. FROM apache/airflow:2.3.0
  2. RUN pip install --no-cache-dir apache-airflow-providers-apache-spark==2.1.3

1.打开您的docker-compose.yaml文件，注解掉image行，并取消注解下面的build行：

  2. # image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.2.3}
  4. build: .

1.一旦保存了docker-compose文件，您需要首先使用以下bash命令构建映像：

  1. docker-compose build

1.如果执行上述命令时没有出现错误，则可以使用以下命令加速气流示例（包括新添加的供应器）：

  1. docker-compose up -d

Docker完成初始化Airflow示例并启动Web服务器后，您应该会看到Spark列在Providers页面下，并可在UI的Connection Types下拉列表中找到：
Airflow connection types after adding Dockerfile
瞧！希望这为您和其他任何在Docker上运行时遇到气流提供程序问题的人解决了这个问题。

展开查看全部
赞(0）分享  回复(0）举报  2022-11-01
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com