构建镜像并存储在缓存中,但这些阶段仍在docker load和--cache-from之后构建

wqlqzqxt  于 2023-06-21  发布在  Docker
关注(0)|答案(1)|浏览(138)

重新构建我们的Azure管道,以尝试加快它们的速度。
目前,它正在为我们的unit-testingintegration-testing阶段重建相同的依赖阶段...这是非常缓慢和低效的。

  • 我正在构建--target development的图像,将其存储在Cache@2的缓存中。
  • 在我们的UnitTest阶段,一个任务使用docker load -i成功地从缓存中加载了它,并使用docker images进行了确认。
  • 然后是unit-tests阶段的实际运行,我使用了--cache-from=--target unit-tests
  • 在管道中,我可以看到它确认该高速缓存,但它仍然构建缓存图像应该已经具有的阶段:
#4 importing cache manifest from companyapp-api:pr-api
#4 sha256:7c6bf1eebafe5af983d68e3fb7d72c271b8a80918f9799979ebd2b2bea604d10
#4 DONE 0.0s

#5 [python-base 1/1] FROM docker.io/library/python:3.9-slim
#5 sha256:f876c6f14c8c365d299789228d8a0c38ac92e17ea62116c830f5b7c6bc684e47
#5 DONE 0.0s

至于我正在处理的文件

# ./api/docker/Dockerfile

# creating a python base with shared environment variables
FROM python:3.9-slim as python-base
ENV PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1 \
    PIP_NO_CACHE_DIR=off \
    PIP_DISABLE_PIP_VERSION_CHECK=on \
    PIP_DEFAULT_TIMEOUT=100 \
    POETRY_HOME="/opt/poetry" \
    POETRY_VIRTUALENVS_IN_PROJECT=true \
    POETRY_NO_INTERACTION=1 \
    PYSETUP_PATH="/opt/pysetup" \
    VENV_PATH="/opt/pysetup/.venv"

ENV PATH="$POETRY_HOME/bin:$VENV_PATH/bin:$PATH"

# builder-base is used to build dependencies
FROM python-base as builder-base
RUN apt-get update \
    && apt-get install --no-install-recommends -y \
        curl \
        build-essential 
        
# Install Poetry - respects $POETRY_VERSION & $POETRY_HOME
ENV POETRY_VERSION=1.4.1 GET_POETRY_IGNORE_DEPRECATION=1
RUN curl -sSL https://install.python-poetry.org | python3 -

# We copy our Python requirements here to cache them
# and install on ly runtime deps using poetry
WORKDIR $PYSETUP_PATH
COPY ./poetry.lock ./pyproject.toml ./
RUN poetry install --no-dev     

# 'development' stage installs all dev deps and can be used to develop code.
# For example using docker-compose to mount local volume under /app
FROM python-base as development

# Copying poetry and venv into image
COPY --from=builder-base $POETRY_HOME $POETRY_HOME
COPY --from=builder-base $PYSETUP_PATH $PYSETUP_PATH

# Copying in our entrypoint
# COPY ./docker/docker-entrypoint.sh /docker-entrypoint.sh
RUN chmod +x . /opt/pysetup/.venv/bin/activate

# venv already has runtime deps installed we get a quicker install
WORKDIR $PYSETUP_PATH
RUN poetry install
WORKDIR /app
COPY . .
EXPOSE 5000 5672
CMD [ "python", "src/manage.py", "runserver", "0.0.0.0:5000"]

# 'unit-tests' stage runs our unit tests with unittest and coverage.   
FROM development AS unit-tests
RUN coverage run --omit='src/manage.py,src/config/*,*/.venv/*,*/*__init__.py,*/tests.py,*/admin.py' src/manage.py test src --tag=ut && \
    coverage report
# ./pipelines/pr.yaml
# # This is triggered by the PR and branch policies
trigger: none

# Read in the base variable template
variables:
  imageRepository: companyapp
  dockerfilePath: $(Build.SourcesDirectory)
  vmImageName: ubuntu-latest

# Use the ubuntu-latest image
pool:
  vmIMage: $(vmImageName)

stages:
- stage: Build
  displayName: Build image for tests...
  jobs:
  - job: BuildingAndCache
    displayName: Building and caching image for tests...
    steps: 

    - task: Cache@2
      displayName: Creating cache...
      inputs: 
        key: 'docker | "$(Agent.OS)" | cache'
        path: $(Pipeline.Workspace)/docker
        cacheHitVar: CACHE_RESTORED

    - task: Docker@2
      displayName: Building image for tests...
      inputs:
        command: 'build'
        repository: $(imageRepository)-$(service)
        dockerfile: $(dockerFilePath)/$(service)/docker/Dockerfile
        buildContext: $(dockerFilePath)/$(service)
        arguments: |
          --target development
        tags: |
          pr-$(service)
      env:
        DOCKER_BUILDKIT: 1

    - bash: |
        mkdir -p $(Pipeline.Workspace)/docker
        docker save -o $(Pipeline.Workspace)/docker/cache.tar $(imageRepository)-$(service):pr-$(service)
      displayName: Saving image to cache...
      # condition: and(not(canceled()), not(failed()), ne(variables.CACHE_RESTORED, 'true'))
        

- stage: UnitTest
  displayName: Run unit tests...
  jobs:
  - job: UnitTesting
    displayName: Running unit tests...
    steps: 
    - task: Cache@2
      displayName: Checking cache for existing images...
      inputs: 
        key: 'docker | "$(Agent.OS)" | cache'
        path: $(Pipeline.Workspace)/docker
        cacheHitVar: CACHE_RESTORED

    - script: |
        docker load -i $(Pipeline.Workspace)/docker/cache.tar
        docker images
      displayName: Loading existing image from cache...
      condition: and(not(canceled()), eq(variables.CACHE_RESTORED, 'true'))

    - task: Docker@2
      displayName: Running unit-tests...
      inputs:
        command: 'build'
        repository: $(imageRepository)-$(service)
        dockerfile: $(dockerFilePath)/$(service)/docker/Dockerfile
        buildContext: $(dockerFilePath)/$(service)
        arguments: |
          --cache-from=$(imageRepository)-$(service):pr-$(service)
          --target unit-tests
        # tags: |
        #   pr-$(service)
      env:
        DOCKER_BUILDKIT: 1

有什么建议,我做错了什么,以及如何解决它?

我一直在咨询的资源:

ni65a41a

ni65a41a1#

我自己想出来的只是疏忽Dockerfile查找的图像与pr.yaml中构建的图像名称不匹配。
我需要更新:

...
FROM development AS unit-tests
...

要匹配此处正在构建的映像的名称,请执行以下操作:

...
    - task: Docker@2
      displayName: Building image for tests...
      inputs:
        command: 'build'
        repository: $(imageRepository)-$(service)
        dockerfile: $(dockerFilePath)/$(service)/docker/Dockerfile
        buildContext: $(dockerFilePath)/$(service)
        arguments: |
          --target development
        tags: |
          pr-$(service)
      env:
        DOCKER_BUILDKIT: 1

    - bash: |
        mkdir -p $(Pipeline.Workspace)/docker
        docker save -o $(Pipeline.Workspace)/docker/cache.tar $(imageRepository)-$(service):pr-$(service)
      displayName: Saving image to cache...
...

正在构建的镜像名称为:

companyapp-api:pr-api

所以Dockerfile应该是:

...
FROM companyapp-api:pr-api AS unit-tests
...

一旦我这样做了,unit-tests状态需要10s而不是1m20s。

相关问题