How to use Skaffold with kubernetes volumes?
I can't speak for skaffold specifically but the container image build can be improved. If there is layer caching available then only reinstall the dependencies when your requirements.txt
changes. This is documented in the "ADD or COPY" Best Practices.
FROM ubuntu:18.04...COPY requirements.txt /usr/local/app/RUN pip install -r /usr/local/app/requirements.txtCOPY . /usr/local/app...CMD ...
You may need to trigger updates some times if the module versions are loosely defined and say you want a new patch version. I've found requirements should be specific so versions don't slide underneath your application without your knowledge/testing.
Kaniko in-cluster builds
For kaniko builds to make use of a cache in a cluster where there is no persistent storage by default, kaniko needs either a persistent volume mounted (--cache-dir
) or a container image repo (--cache-repo
) with the layers available.
If your goal is to speed up the dev process: Instead of triggering an entirely new deployment process every time you change a line of code, you can switch to a sync-based dev process to deploy once and then update the files within the running containers when editing code.
Skaffold supports file sync to directly update files inside the deployed containers if you change them on your local machine. However, the docs state "File sync is alpha" (https://skaffold.dev/docs/how-tos/filesync/) and I can completely agree from working with it a while ago: The sync mechanism is only one-directional (no sync from container back to local) and pretty buggy, i.e. it crashes frequently when switching git branches, installing dependencies etc. which can be pretty annoying.
If you want a more stable alternative for sync-based Kubernetes development which is very easy to get started with, take a look at DevSpace: https://github.com/devspace-cloud/devspace
I am one of the maintainers of DevSpace and started the project because Skaffold was much too slow for our team and it did not have a file sync back then.
@Matt's answer is a great best practice (+1) - skaffold
in and of itself won't solve the underlying layer cache invalidation issues which results in having to re-install the requirements during every build.
For additional performance, you can cache all the python
packages in a volume
mounted in your pod for example:
apiVersion: v1kind: Podmetadata:name: testspec:containers:- image: test:test name: test-containervolumeMounts:- mountPath: /usr/local/venv name: test-volume- mountPath: /root/.cache/pip name: pip-cachevolumes:- name: test-volume awsElasticBlockStore: volumeID: <volume-id> fsType: ext4- name: pip-cache awsElasticBlockStore: volumeID: <volume-id> fsType: ext4
That way if the build cache is ever invalidated and you have to re-install your requirements.txt
you'd be saving some time by fetching them from cache.
If you're building with kaniko
you can also cache base images to a persistent disk using the kaniko-warmer
, for example:
...volumeMounts:...- mountPath: /cache name: kaniko-warmervolumes:...- name: kaniko-warmer awsElasticBlockStore: volumeID: <volume-id> fsType: ext4
Running the kaniko-warmer
inside the pod: docker run --rm -it -v /cache:/cache --entrypoint /kaniko/warmer gcr.io/kaniko-project/warmer --cache-dir=/cache --image=python:3.7-slim --image=nginx:1.17.3
. Your skaffold.yaml
might look something like:
apiVersion: skaffold/v1beta13kind: Configbuild: artifacts: - image: test:test kaniko: buildContext: localDir: {} cache: hostPath: /cache cluster: namespace: jx dockerConfig: secretName: jenkins-docker-cfg tagPolicy: envTemplate: template: '{{.DOCKER_REGISTRY}}/{{.IMAGE_NAME}}'deploy: kubectl: {}