docker --cache-from with BUILDKIT_INLINE_CACHE does not work every second time
This issue got fixed in newer docker versions, a simple upgrade resolves the issue.
Otherwise the solution described on GitHub can help to not rely on the systems docker version: https://github.com/moby/buildkit/issues/1981#issuecomment-785534131