Any way to make `--cache-to` output smaller? #4381

kohenkatz · 2022-10-27T04:41:13Z

kohenkatz
Oct 27, 2022

I was very happy to see support for --cache-from and --cache-to appear in buildah 1.27.0. (I used to use Kaniko with its --cache argument which did the same thing, but Kaniko does not have proper support for RUN --mount....)

The problem I have is with the size of the cache (and therefore also the build time).

Here's an example file:

# syntax = docker/dockerfile:1.2

# get modules, if they don't change the cache can be used for faster builds
FROM docker.io/library/golang:1.18 AS base
ENV GO111MODULE=on
ENV CGO_ENABLED=0
ENV GOOS=linux
ENV GOARCH=amd64
WORKDIR /src
COPY go.* .
RUN --mount=type=cache,target=/go/pkg/mod \
	--mount=type=secret,id=netrc,target=/root/.netrc \
    go mod download

# build the application
FROM base AS build
WORKDIR /src
# temp mount all files instead of loading into image with COPY
# temp mount module cache
# temp mount go build cache
RUN --mount=type=bind,target=/src \
    --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    --mount=type=secret,id=netrc,target=/root/.netrc \
    go build -ldflags="-w -s" -o /app/liveserver ./cmd/liveserver/*.go

# Import the binary from build stage
FROM gcr.io/distroless/static:nonroot as liveserver
COPY --from=build /app/liveserver /
# this is the numeric version of user nonroot:nonroot to check runAsNonRoot in kubernetes
USER 65532:65532
ENTRYPOINT ["/liveserver"]

Here's the command I use to run it:

buildah bud --secret id=netrc,src=/root/.netrc --cache-from registry.example.com/group/project/cache --cache-to registry.example.com/group/project/cache --layers -f Dockerfile -t liveserver --target liveserver .

I first noticed the issue because the build was running extremely slowly. It turns out that docker.io/library/golang:1.18 is approximately 350MB, and every one of the ENV, WORKDIR, and COPY lines below that was generating another 350MB cache image. By the time this build was done running, it was using several GB of storage space, the vast majority of it being duplicated copies of the golang container image with minor addons.

Under Kaniko (with suitable modifications to replace RUN --mount... with something that works), all of those cache layers for the ENV lines are only a few KB.

I assume this is probably an artifact of how buildah handles the layers - each layer is cumulative from the layers below it - but it produces an enormous amount of wasted cache space.

Is there any way to customize or simplify the caching so it doesn't take up so much space on the server with duplicate data?

rhatdan · 2022-10-27T17:50:33Z

rhatdan
Oct 27, 2022
Maintainer

@nalind @flouthoc PTAL

1 reply

codethief Mar 15, 2024

What is the current state of this? We are currently using Kaniko to build images in containerized CI jobs (Gitlab k8s runners) but, unfortunately, Kaniko's caching performance isn't great, so we have been looking for alternatives, and Buildah had looked quite promising, given the possibility to run it inside containers (w/o privileges) and its recent addition of --cache-to/from.

codethief · 2024-09-15T21:26:17Z

codethief
Sep 15, 2024

@kohenkatz Did you ever find a solution here?

and every one of the ENV, WORKDIR, and COPY lines below that was generating another 350MB cache image.

Which --storage-driver are you using? I'm asking because you mentioned using Buildah as a Kaniko alternative, and people often recommend using --storage-driver vfs to run buildah in such a containerized environment (at least when the container lacks permissions to use (fuse-)overlayfs mounts).

Now, vfs is known for being slow and having to copy data around for every layer. In theory I would not expect this to affect the size and contents of the image layers that end up in the cache (I would expect vfs to still diff the layers' root file systems against each other) but maybe I'm wrong?

1 reply

kohenkatz Jan 20, 2025
Author

Sorry, for not replying sooner. I don't know why I did not get a notification that there was a new post in this discussion.

I have not yet found a solution to enable caching here, though I haven't really been looking recently.

I am using vfs because my CI (GitLab) is running buildah inside a container itself and overlayfs doesn't really support being nested (I've seen some hacks that supposedly make it work, but I am very reluctant to try that in production).

Detuner · 2025-10-24T12:53:07Z

Detuner
Oct 24, 2025

I have the same problem here. Buildah running in k8s behaves the same way: with each line of dockerfile it pushes to a cache repository the whole filesystem image, not just built layer. Building some heavier images gets way too slow due to cache pushing/pulling overhead, some of my builds got 2-3 times slower compared to kaniko builds.
Tried switching between vfs and overlay (fuse), made no difference.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Any way to make `--cache-to` output smaller? #4381

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 3 comments 2 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Any way to make --cache-to output smaller? #4381

Uh oh!

Uh oh!

kohenkatz Oct 27, 2022

Replies: 3 comments · 2 replies

Uh oh!

rhatdan Oct 27, 2022 Maintainer

Uh oh!

Uh oh!

codethief Mar 15, 2024

Uh oh!

Uh oh!

codethief Sep 15, 2024

Uh oh!

kohenkatz Jan 20, 2025 Author

Uh oh!

Detuner Oct 24, 2025

Any way to make `--cache-to` output smaller? #4381

kohenkatz
Oct 27, 2022

Replies: 3 comments 2 replies

rhatdan
Oct 27, 2022
Maintainer

codethief
Sep 15, 2024

kohenkatz Jan 20, 2025
Author

Detuner
Oct 24, 2025