pyspark 在docker上运行气流和Spark，发现错误

4xrmg8kj  于 2024-01-06  发布在  Spark
关注(0)|答案(1)|浏览(146)

我从bitnami/spark和apache/airflow分别构建spark镜像和airflow。我在使用sparksubmit操作符运行dag spark submit时发现了这个错误。
“/usr/lib/jvm/java-11-openjdk-amd 64/bin/java：没有这样的文件或目录”
结尾是，
“airflow.exceptions.AirflowException：无法执行：spark-submit --master spark：//spark：7077 --executor-memory 1g --driver-memory 1g --name arrow-spark /home/***/scripts/local_to_postgres_pyspark. py。错误代码为：1。”
My Dockerfile.spark：

  1. FROM bitnami/spark:latest
  3. # Install dependencies
  4. USER root
  5. RUN apt-get update && \
  6.     apt-get install -y gcc curl && \
  7.     apt-get clean && \
  8.     rm -rf /var/lib/apt/lists/*
  10. # Install JBDC Driver
  11. RUN curl -o /opt/bitnami/spark/jars/postgresql-42.6.0.jar https://jdbc.postgresql.org/download/postgresql-42.6.0.jar
  13. COPY ./requirements_for_docker.txt /
  14. RUN pip install -r /requirements_for_docker.txt

字符串
My Dockerfile.airflow：

  1. FROM apache/airflow:2.7.0
  2. USER root
  4. RUN apt-get update && \
  5.     apt-get install -y procps openjdk-11-jre-headless && \
  6.     apt-get clean && \
  7.     rm -rf /var/lib/apt/lists/*
  8. ENV JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
  10. # Install dependencies
  11. USER airflow
  13. COPY requirements_for_docker.txt /tmp/requirements_for_docker.txt
  14. RUN pip install --user --upgrade pip
  15. RUN pip install --no-cache-dir --user -r /tmp/requirements_for_docker.txt
  16. RUN pip install apache-airflow-providers-apache-spark==2.1.3

型
我的docker-compose.airflow.yaml文件：

  1. ---
  2. version: '3.8'
  3. x-airflow-common:
  4.   &airflow-common
  5.   image: airflow-spark:latest
  6.   # build: .
  7.   environment:
  8.     &airflow-common-env
  9.     AIRFLOW__CORE__EXECUTOR: LocalExecutor
  10.     AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
  11.     AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
  12.     AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
  13.     AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
  14.     AIRFLOW__API__AUTH_BACKENDS: 'airflow.api.auth.backend.basic_auth,airflow.api.auth.backend.session'
  15.     LD_LIBRARY_PATH: /usr/lib
  16.     AIRFLOW__SCHEDULER__ENABLE_HEALTH_CHECK: 'true'
  17.     _PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}
  18.     JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
  19.   volumes:
  20.     - ./dags:/opt/airflow/dags
  21.     - ./logs:/opt/airflow/logs
  22.     - ./config:/opt/airflow/config
  23.     - ./plugins:/opt/airflow/plugins
  24.     - ./data_sample:/home/airflow/data_sample
  25.     - ./scripts:/home/airflow/scripts
  26.   user: "${AIRFLOW_UID:-50000}:0"
  27.   depends_on:
  28.     &airflow-common-depends-on
  29.     postgres:
  30.       condition: service_healthy
  32. services:
  33.   postgres:
  34.     image: postgres:13
  35.     environment:
  36.       POSTGRES_USER: airflow
  37.       POSTGRES_PASSWORD: airflow
  38.       POSTGRES_MULTIPLE_DATABASES: "airflow,ownerOfairflow:postgres,ownerOfpostgres"
  39.       MAX_CONNECTIONS: 200
  40.     volumes:
  41.       - postgres-db-volume:/var/lib/postgresql/data
  42.     healthcheck:
  43.       test: ["CMD", "pg_isready", "-U", "airflow"]
  44.       interval: 10s
  45.       retries: 5
  46.       start_period: 5s
  47.     ports:
  48.       - 5432:5432
  49.     restart: always
  51.   pgadmin:
  52.     image: dpage/pgadmin4
  53.     links:
  54.       - postgres
  55.     depends_on:
  56.       - postgres
  57.     restart: always
  58.     ports:
  59.       - "8081:80"
  60.     environment:
  61.       - [email protected]
  62.       - PGADMIN_DEFAULT_PASSWORD=admin
  63.     volumes:
  64.       - ./pgadmin-data:/var/lib/pgadmin
  66.   airflow-webserver:
  67.     <<: *airflow-common
  68.     command: webserver
  69.     ports:
  70.       - "8080:8080"
  71.     healthcheck:
  72.       test: ["CMD", "curl", "--fail", "http://localhost:8080/health"]
  73.       interval: 30s
  74.       timeout: 10s
  75.       retries: 5
  76.       start_period: 30s
  77.     restart: always
  78.     depends_on:
  79.       <<: *airflow-common-depends-on
  80.       airflow-init:
  81.         condition: service_completed_successfully
  83.   airflow-scheduler:
  84.     <<: *airflow-common
  85.     command: scheduler
  86.     healthcheck:
  87.       test: ["CMD", "curl", "--fail", "http://localhost:8974/health"]
  88.       interval: 30s
  89.       timeout: 10s
  90.       retries: 5
  91.       start_period: 30s
  92.     restart: always
  93.     depends_on:
  94.       <<: *airflow-common-depends-on
  95.       airflow-init:
  96.         condition: service_completed_successfully
  98.   airflow-init:
  99.     <<: *airflow-common
  100.     entrypoint: /bin/bash
  101.     command:
  102.       - -c
  103.       - |
  104.         function ver() {
  105.           printf "%04d%04d%04d%04d" $${1//./ }
  106.         }
  107.         airflow_version=$$(AIRFLOW__LOGGING__LOGGING_LEVEL=INFO && gosu airflow airflow version)
  108.         airflow_version_comparable=$$(ver $${airflow_version})
  109.         min_airflow_version=2.2.0
  110.         min_airflow_version_comparable=$$(ver $${min_airflow_version})
  111.         if (( airflow_version_comparable < min_airflow_version_comparable )); then
  112.           echo
  113.           echo -e "\033[1;31mERROR!!!: Too old Airflow version $${airflow_version}!\e[0m"
  114.           echo "The minimum Airflow version supported: $${min_airflow_version}. Only use this or higher!"
  115.           echo
  116.           exit 1
  117.         fi
  118.         if [[ -z "${AIRFLOW_UID}" ]]; then
  119.           echo
  120.           echo -e "\033[1;33mWARNING!!!: AIRFLOW_UID not set!\e[0m"
  121.           echo "If you are on Linux, you SHOULD follow the instructions below to set "
  122.           echo "AIRFLOW_UID environment variable, otherwise files will be owned by root."
  123.           echo "For other operating systems you can get rid of the warning with manually created .env file:"
  124.           echo "    See: https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html#setting-the-right-airflow-user"
  125.           echo
  126.         fi
  127.         one_meg=1048576
  128.         mem_available=$$(($$(getconf _PHYS_PAGES) * $$(getconf PAGE_SIZE) / one_meg))
  129.         cpus_available=$$(grep -cE 'cpu[0-9]+' /proc/stat)
  130.         disk_available=$$(df / | tail -1 | awk '{print $$4}')
  131.         warning_resources="false"
  132.         if (( mem_available < 4000 )) ; then
  133.           echo
  134.           echo -e "\033[1;33mWARNING!!!: Not enough memory available for Docker.\e[0m"
  135.           echo "At least 4GB of memory required. You have $$(numfmt --to iec $$((mem_available * one_meg)))"
  136.           echo
  137.           warning_resources="true"
  138.         fi
  139.         if (( cpus_available < 2 )); then
  140.           echo
  141.           echo -e "\033[1;33mWARNING!!!: Not enough CPUS available for Docker.\e[0m"
  142.           echo "At least 2 CPUs recommended. You have $${cpus_available}"
  143.           echo
  144.           warning_resources="true"
  145.         fi
  146.         if (( disk_available < one_meg * 10 )); then
  147.           echo
  148.           echo -e "\033[1;33mWARNING!!!: Not enough Disk space available for Docker.\e[0m"
  149.           echo "At least 10 GBs recommended. You have $$(numfmt --to iec $$((disk_available * 1024 )))"
  150.           echo
  151.           warning_resources="true"
  152.         fi
  153.         if [[ $${warning_resources} == "true" ]]; then
  154.           echo
  155.           echo -e "\033[1;33mWARNING!!!: You have not enough resources to run Airflow (see above)!\e[0m"
  156.           echo "Please follow the instructions to increase amount of resources available:"
  157.           echo "   https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html#before-you-begin"
  158.           echo
  159.         fi
  160.         mkdir -p /sources/logs /sources/dags /sources/plugins
  161.         chown -R "${AIRFLOW_UID}:0" /sources/{logs,dags,plugins}
  162.         exec /entrypoint airflow version
  163.     environment:
  164.       <<: *airflow-common-env
  165.       _AIRFLOW_DB_UPGRADE: 'true'
  166.       _AIRFLOW_WWW_USER_CREATE: 'true'
  167.       _AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
  168.       _AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}
  169.       _PIP_ADDITIONAL_REQUIREMENTS: ''
  170.     user: "0:0"
  171.     volumes:
  172.       - ${AIRFLOW_PROJ_DIR:-.}:/sources
  174. volumes:
  175.   postgres-db-volume:

型
我的docker-compose.spark.yaml文件：

  1. version: '2'
  3. services:
  4.   spark:
  5.     image: spark-cluster:latest
  6.     environment:
  7.       - SPARK_MODE=master
  8.       - SPARK_RPC_AUTHENTICATION_ENABLED=no
  9.       - SPARK_RPC_ENCRYPTION_ENABLED=no
  10.       - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
  11.       - SPARK_SSL_ENABLED=no
  12.     volumes:
  13.       - ./scripts:/opt/bitnami/spark/scripts
  14.       - ./dags:/opt/bitnami/spark/dags
  15.       - ./data_sample:/opt/bitnami/spark/data_sample
  16.     ports:
  17.       - "8090:8080"
  18.       - "7077:7077"
  20.   spark-worker-1:
  21.     image: spark-cluster:latest
  22.     environment:
  23.       - SPARK_MODE=worker
  24.       - SPARK_MASTER_URL=spark://spark:7077
  25.       - SPARK_WORKER_MEMORY=1G
  26.       - SPARK_WORKER_CORES=1
  27.       - SPARK_RPC_AUTHENTICATION_ENABLED=no
  28.       - SPARK_RPC_ENCRYPTION_ENABLED=no
  29.       - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
  30.       - SPARK_SSL_ENABLED=no
  31.     volumes:
  32.       - ./scripts:/opt/bitnami/spark/scripts
  33.       - ./dags:/opt/bitnami/spark/dags
  34.       - ./data_sample:/opt/bitnami/spark/data_sample
  35.   
  36.   spark-worker-2:
  37.     image: spark-cluster:latest
  38.     environment:
  39.       - SPARK_MODE=worker
  40.       - SPARK_MASTER_URL=spark://spark:7077
  41.       - SPARK_WORKER_MEMORY=1G
  42.       - SPARK_WORKER_CORES=1
  43.       - SPARK_RPC_AUTHENTICATION_ENABLED=no
  44.       - SPARK_RPC_ENCRYPTION_ENABLED=no
  45.       - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
  46.       - SPARK_SSL_ENABLED=no
  47.     volumes:
  48.       - ./scripts:/opt/bitnami/spark/scripts
  49.       - ./dags:/opt/bitnami/spark/dags
  50.       - ./data_sample:/opt/bitnami/spark/data_sample
  51.     
  52.   spark-worker-3:
  53.     image: spark-cluster:latest
  54.     environment:
  55.       - SPARK_MODE=worker
  56.       - SPARK_MASTER_URL=spark://spark:7077
  57.       - SPARK_WORKER_MEMORY=1G
  58.       - SPARK_WORKER_CORES=1
  59.       - SPARK_RPC_AUTHENTICATION_ENABLED=no
  60.       - SPARK_RPC_ENCRYPTION_ENABLED=no
  61.       - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
  62.       - SPARK_SSL_ENABLED=no
  63.     volumes:
  64.       - ./scripts:/opt/bitnami/spark/scripts
  65.       - ./dags:/opt/bitnami/spark/dags
  66.       - ./data_sample:/opt/bitnami/spark/data_sample
  68.   spark-worker-4:
  69.     image: spark-cluster:latest
  70.     environment:
  71.       - SPARK_MODE=worker
  72.       - SPARK_MASTER_URL=spark://spark:7077
  73.       - SPARK_WORKER_MEMORY=1G
  74.       - SPARK_WORKER_CORES=1
  75.       - SPARK_RPC_AUTHENTICATION_ENABLED=no
  76.       - SPARK_RPC_ENCRYPTION_ENABLED=no
  77.       - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
  78.       - SPARK_SSL_ENABLED=no
  79.     volumes:
  80.       - ./scripts:/opt/bitnami/spark/scripts
  81.       - ./dags:/opt/bitnami/spark/dags
  82.       - ./data_sample:/opt/bitnami/spark/data_sample
  83.     
  84.   spark-worker-5:
  85.     image: spark-cluster:latest
  86.     environment:
  87.       - SPARK_MODE=worker
  88.       - SPARK_MASTER_URL=spark://spark:7077
  89.       - SPARK_WORKER_MEMORY=1G
  90.       - SPARK_WORKER_CORES=1
  91.       - SPARK_RPC_AUTHENTICATION_ENABLED=no
  92.       - SPARK_RPC_ENCRYPTION_ENABLED=no
  93.       - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
  94.       - SPARK_SSL_ENABLED=no
  95.     volumes:
  96.       - ./scripts:/opt/bitnami/spark/scripts
  97.       - ./dags:/opt/bitnami/spark/dags
  98.       - ./data_sample:/opt/bitnami/spark/data_sample
  99.     
  100.   spark-worker-6:
  101.     image: spark-cluster:latest
  102.     environment:
  103.       - SPARK_MODE=worker
  104.       - SPARK_MASTER_URL=spark://spark:7077
  105.       - SPARK_WORKER_MEMORY=1G
  106.       - SPARK_WORKER_CORES=1
  107.       - SPARK_RPC_AUTHENTICATION_ENABLED=no
  108.       - SPARK_RPC_ENCRYPTION_ENABLED=no
  109.       - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
  110.       - SPARK_SSL_ENABLED=no
  111.     volumes:
  112.       - ./scripts:/opt/bitnami/spark/scripts
  113.       - ./dags:/opt/bitnami/spark/dags
  114.       - ./data_sample:/opt/bitnami/spark/data_sample

型
我试着加上

  1. ENV JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64

型
从你在我的Dockerfile上看到的。气流

pyspark

1条答案

按热度按时间
xwbd5t1u

xwbd5t1u1#

试着在你的“导出JAVA_HOME”之后添加“导出JAVA_HOME”。
(this是我使用的图像，它可以工作）
同样，如果仍然存在，看看你在spark中使用的python版本和你在气流图像中使用的python版本，在下面的例子中，我不得不改变到python3.11，因为这一点

  1. FROM apache/airflow:2.7.3-python3.11
  3. USER root
  4. RUN apt-get update \
  5.   && apt-get install -y --no-install-recommends \
  6.          openjdk-11-jre-headless \
  7.   && apt-get autoremove -yqq --purge \
  8.   && apt-get clean \
  9.   && rm -rf /var/lib/apt/lists/*
  10. RUN apt update && apt install -y procps
  11. USER airflow
  12. ENV JAVA_HOME=/usr/lib/jvm/java-11-openjdk-arm64
  13. RUN export JAVA_HOME

字符串

展开查看全部
赞(0）分享  回复(0）举报  2024-01-06
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com