docker 容器中缺少Dagster可执行文件

wwodge7n 于 2024-01-06 发布在 Docker

关注(0)|答案(2)|浏览(204)

我一直在尝试创建一个在Dagster云中运行的容器，该容器采用ECS混合部署模型。我能够将容器推送到Dagster，但我不断收到此错误：

dagster_cloud.workspace.ecs.client.EcsServiceError: ECS service failed because task arn:aws:ecs:ap-northeast-1:*****:task/Dagster-Cloud-my-cluster-Cluster/1d151e6d40b44588a4ed4446a949d44a failed: CannotStartContainerError: ResourceInitializationError: failed to create new container runtime task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "dagster": executable file not found in $PATH: unknown
Task logs:
runc create failed: unable to start container process: exec: "dagster": executable file not found in $PATH
For more information about the failure, check the ECS console for logs for task arn:aws:ecs:ap-northeast-1:*****:task/Dagster-Cloud-my-cluster-Cluster/1d151e6d40b44588a4ed4446a949d44a in cluster Dagster-Cloud-my-cluster-Cluster.
  File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1304, in _reconcile
    self._wait_for_new_server_ready(
  File "/dagster-cloud/dagster_cloud/workspace/ecs/launcher.py", line 458, in _wait_for_new_server_ready
    task_arn = self.client.wait_for_new_service(
  File "/dagster-cloud/dagster_cloud/workspace/ecs/client.py", line 491, in wait_for_new_service
    return self.check_service_has_running_task(
  File "/dagster-cloud/dagster_cloud/workspace/ecs/client.py", line 607, in check_service_has_running_task
    self._raise_failed_task(task, container_name, logger)
  File "/dagster-cloud/dagster_cloud/workspace/ecs/client.py", line 526, in _raise_failed_task
    raise EcsServiceError(

字符串
我不知道为什么会发生这种情况，因为我确定我正在安装dagster和dagster-cloud，根据文档。我的Docker容器看起来像这样：

###############################################################################
# Base container
###############################################################################
FROM python:3.11 AS base
# Define the environment variables necessary to work with poetry
ENV POETRY_VERSION=1.5.1 \
  POETRY_HOME="/opt/poetry" \
  POETRY_VIRTUALENVS_IN_PROJECT=true \
  POETRY_NO_INTERACTION=1
# Add the poetry bin to our path
ENV PATH="$POETRY_HOME/bin:$PATH"
###############################################################################
# Poetry installer container
###############################################################################
FROM base AS installer
# Install poetry
RUN curl -sSL https://install.python-poetry.org | python3 -
###############################################################################
# Container that actually builds the application
###############################################################################
FROM base AS builder
# Copy the poetry files from the poetry installer to this container
COPY --from=installer $POETRY_HOME $POETRY_HOME
# Describe the environment variables necessary to install our dependencies
ENV PYTHONFAULTHANDLER=1 \
  PYTHONUNBUFFERED=1 \
  PYTHONHASHSEED=random \
  PIP_NO_CACHE_DIR=off \
  PIP_DISABLE_PIP_VERSION_CHECK=on \
  PIP_DEFAULT_TIMEOUT=100
# Set the working directory to one we can install to easily
WORKDIR /app
# Copy the poetry.lock and pyproject.toml files first to ensure that dependencies
# are only installed when they're updated
COPY poetry.lock pyproject.toml /app/
# Copy in our SSH key so we can retrieve the shared GitHub repo
RUN mkdir -p -m 0700 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts
# Install our project dependencies
RUN --mount=type=ssh poetry install --only dagster --no-ansi --no-root
# Build the project
COPY . /app/
RUN poetry build
###############################################################################
# Create a runtime environment that's much smaller
###############################################################################
FROM python:3.11-alpine AS runtime
# Set the working directory to where Dagster will look for the application
WORKDIR /opt/dagster/app
# Copy the project wheel file
COPY --from=builder /app/dist/*.whl /
# Install the wheel file using pip and then install dagster and dagster-cloud
RUN pip install --no-cache-dir /*.whl \
  && rm -rf /*.whl

型
对于这个包，我的pyproject.toml文件列出了我的依赖项如下：

[tool.poetry.group.dagster.dependencies]
dagster = "^1.5.9"
dagster-aws = "^0.21.9"
pendulum = "^2.1.2"
pandas = "^2.1.3"
openpyxl = "^3.1.2"
dagster-cloud = "^1.5.12"

型
所以，这两个都应该安装，但显然他们没有。我在这里做错了什么？

docker

来源：https://stackoverflow.com/questions/77660998/dagster-executable-missing-from-container

2条答案

按热度按时间

quhf5bfb1#

您可能忘记激活虚拟环境了？如果它位于/.venv，我建议添加

# Activate virtual env (i.e. setting VIRTUAL_ENV and extending PATH)
ENV VIRTUAL_ENV=/.venv \
    PATH="/.venv/bin:$PATH"

字符串
在Dockerfile的结尾。

赞(0）回复(0）举报 2024-01-06

tp5buhyn2#

这个容器有两个问题：
1.我正在推动整个虚拟环境，但这并不包括与一些软件包一起安装的可执行文件，这导致了错误。
1.安装wheel不会安装包的要求。
为了解决这些问题，我使用Poetry构建了一个requirements.txt文件，然后使用pip安装其中的需求。由于这些需求包括dagster和dagster-cloud，因此该软件包最终包含了两个必要的可执行文件。

###############################################################################
# Base container
###############################################################################
FROM python:3.11 AS base
# Define the environment variables necessary to work with poetry
ENV POETRY_VERSION=1.5.1 \
  POETRY_HOME="/opt/poetry" \
  POETRY_VIRTUALENVS_IN_PROJECT=true \
  POETRY_NO_INTERACTION=1
# Add the poetry bin to our path
ENV PATH="$POETRY_HOME/bin:$PATH"
###############################################################################
# Poetry installer container
###############################################################################
FROM base AS installer
# Install poetry
RUN curl -sSL https://install.python-poetry.org | python3 -
###############################################################################
# Container that actually builds the application
###############################################################################
FROM base AS builder
# Copy the poetry files from the poetry installer to this container
COPY --from=installer $POETRY_HOME $POETRY_HOME
# Describe the environment variables necessary to install our dependencies
ENV PYTHONFAULTHANDLER=1 \
  PYTHONUNBUFFERED=1 \
  PYTHONHASHSEED=random \
  PIP_NO_CACHE_DIR=off \
  PIP_DISABLE_PIP_VERSION_CHECK=on \
  PIP_DEFAULT_TIMEOUT=100
# Set the working directory to one we can install to easily
WORKDIR /app
# Copy the poetry.lock and pyproject.toml files first to ensure that dependencies
# are only installed when they're updated
COPY poetry.lock pyproject.toml /app/
# Create a requirements.txt file we can use
RUN poetry export --with dagster --without-hashes --with-credentials -f requirements.txt --output requirements.txt
###############################################################################
# Create a runtime environment that's much smaller
###############################################################################
FROM python:3.11 AS runtime
# Copy in our SSH key so we can retrieve the shared GitHub repo
RUN mkdir -p -m 0700 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts
# # Set the working directory to where Dagster will look for the application
WORKDIR /opt/dagster/app
# # Copy the project files
COPY --from=builder ./app/requirements.txt /opt/dagster/app
COPY . /opt/dagster/app
# Install all the requirements and then install the package itself
RUN --mount=type=ssh pip install -r requirements.txt
RUN pip install -e .

字符串

展开查看全部

赞(0）回复(0）举报 2024-01-06

我来回答

docker 容器中缺少Dagster可执行文件

2条答案

相关问题

热门标签

最新问答