重新构建我们的Azure管道,以尝试加快它们的速度。
目前,它正在为我们的unit-testing
和integration-testing
阶段重建相同的依赖阶段...这是非常缓慢和低效的。
- 我正在构建
--target development
的图像,将其存储在Cache@2
的缓存中。 - 在我们的
UnitTest
阶段,一个任务使用docker load -i
成功地从缓存中加载了它,并使用docker images
进行了确认。 - 然后是
unit-tests
阶段的实际运行,我使用了--cache-from=
和--target unit-tests
。 - 在管道中,我可以看到它确认该高速缓存,但它仍然构建缓存图像应该已经具有的阶段:
#4 importing cache manifest from companyapp-api:pr-api
#4 sha256:7c6bf1eebafe5af983d68e3fb7d72c271b8a80918f9799979ebd2b2bea604d10
#4 DONE 0.0s
#5 [python-base 1/1] FROM docker.io/library/python:3.9-slim
#5 sha256:f876c6f14c8c365d299789228d8a0c38ac92e17ea62116c830f5b7c6bc684e47
#5 DONE 0.0s
至于我正在处理的文件
# ./api/docker/Dockerfile
# creating a python base with shared environment variables
FROM python:3.9-slim as python-base
ENV PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1 \
PIP_NO_CACHE_DIR=off \
PIP_DISABLE_PIP_VERSION_CHECK=on \
PIP_DEFAULT_TIMEOUT=100 \
POETRY_HOME="/opt/poetry" \
POETRY_VIRTUALENVS_IN_PROJECT=true \
POETRY_NO_INTERACTION=1 \
PYSETUP_PATH="/opt/pysetup" \
VENV_PATH="/opt/pysetup/.venv"
ENV PATH="$POETRY_HOME/bin:$VENV_PATH/bin:$PATH"
# builder-base is used to build dependencies
FROM python-base as builder-base
RUN apt-get update \
&& apt-get install --no-install-recommends -y \
curl \
build-essential
# Install Poetry - respects $POETRY_VERSION & $POETRY_HOME
ENV POETRY_VERSION=1.4.1 GET_POETRY_IGNORE_DEPRECATION=1
RUN curl -sSL https://install.python-poetry.org | python3 -
# We copy our Python requirements here to cache them
# and install on ly runtime deps using poetry
WORKDIR $PYSETUP_PATH
COPY ./poetry.lock ./pyproject.toml ./
RUN poetry install --no-dev
# 'development' stage installs all dev deps and can be used to develop code.
# For example using docker-compose to mount local volume under /app
FROM python-base as development
# Copying poetry and venv into image
COPY --from=builder-base $POETRY_HOME $POETRY_HOME
COPY --from=builder-base $PYSETUP_PATH $PYSETUP_PATH
# Copying in our entrypoint
# COPY ./docker/docker-entrypoint.sh /docker-entrypoint.sh
RUN chmod +x . /opt/pysetup/.venv/bin/activate
# venv already has runtime deps installed we get a quicker install
WORKDIR $PYSETUP_PATH
RUN poetry install
WORKDIR /app
COPY . .
EXPOSE 5000 5672
CMD [ "python", "src/manage.py", "runserver", "0.0.0.0:5000"]
# 'unit-tests' stage runs our unit tests with unittest and coverage.
FROM development AS unit-tests
RUN coverage run --omit='src/manage.py,src/config/*,*/.venv/*,*/*__init__.py,*/tests.py,*/admin.py' src/manage.py test src --tag=ut && \
coverage report
# ./pipelines/pr.yaml
# # This is triggered by the PR and branch policies
trigger: none
# Read in the base variable template
variables:
imageRepository: companyapp
dockerfilePath: $(Build.SourcesDirectory)
vmImageName: ubuntu-latest
# Use the ubuntu-latest image
pool:
vmIMage: $(vmImageName)
stages:
- stage: Build
displayName: Build image for tests...
jobs:
- job: BuildingAndCache
displayName: Building and caching image for tests...
steps:
- task: Cache@2
displayName: Creating cache...
inputs:
key: 'docker | "$(Agent.OS)" | cache'
path: $(Pipeline.Workspace)/docker
cacheHitVar: CACHE_RESTORED
- task: Docker@2
displayName: Building image for tests...
inputs:
command: 'build'
repository: $(imageRepository)-$(service)
dockerfile: $(dockerFilePath)/$(service)/docker/Dockerfile
buildContext: $(dockerFilePath)/$(service)
arguments: |
--target development
tags: |
pr-$(service)
env:
DOCKER_BUILDKIT: 1
- bash: |
mkdir -p $(Pipeline.Workspace)/docker
docker save -o $(Pipeline.Workspace)/docker/cache.tar $(imageRepository)-$(service):pr-$(service)
displayName: Saving image to cache...
# condition: and(not(canceled()), not(failed()), ne(variables.CACHE_RESTORED, 'true'))
- stage: UnitTest
displayName: Run unit tests...
jobs:
- job: UnitTesting
displayName: Running unit tests...
steps:
- task: Cache@2
displayName: Checking cache for existing images...
inputs:
key: 'docker | "$(Agent.OS)" | cache'
path: $(Pipeline.Workspace)/docker
cacheHitVar: CACHE_RESTORED
- script: |
docker load -i $(Pipeline.Workspace)/docker/cache.tar
docker images
displayName: Loading existing image from cache...
condition: and(not(canceled()), eq(variables.CACHE_RESTORED, 'true'))
- task: Docker@2
displayName: Running unit-tests...
inputs:
command: 'build'
repository: $(imageRepository)-$(service)
dockerfile: $(dockerFilePath)/$(service)/docker/Dockerfile
buildContext: $(dockerFilePath)/$(service)
arguments: |
--cache-from=$(imageRepository)-$(service):pr-$(service)
--target unit-tests
# tags: |
# pr-$(service)
env:
DOCKER_BUILDKIT: 1
有什么建议,我做错了什么,以及如何解决它?
我一直在咨询的资源:
- https://docs.docker.com/build/building/multi-stage/
- https://github.com/michaeloliverx/python-poetry-docker-example/blob/master/docker/Dockerfile
- https://learn.microsoft.com/en-us/azure/devops/pipelines/release/caching?view=azure-devops#docker-images
- How to Enable Docker layer caching in Azure DevOps
- 我试过ChatGPT,它所做的一切就是提供我目前正在做的事情作为答案。
1条答案
按热度按时间ni65a41a1#
我自己想出来的只是疏忽
Dockerfile
查找的图像与pr.yaml
中构建的图像名称不匹配。我需要更新:
要匹配此处正在构建的映像的名称,请执行以下操作:
正在构建的镜像名称为:
所以
Dockerfile
应该是:一旦我这样做了,
unit-tests
状态需要10s而不是1m20s。