Building GitLab CI images on a flaky network
A pipeline that builds its own image — like the ansible-lint custom Python image — pulls the base image from Docker Hub and packages from PyPI on every build. On a restricted or flaky network, that fails at random with errors like:
failed to resolve reference "docker.io/library/python:3.13-slim": dial tcp 34.204.126.254:443: i/o timeout
Check which addresses actually answer
Docker Hub sits behind many IPv4 addresses, and DNS hands out a different one each time. If some of them are blocked somewhere upstream, pulls work or fail depending on which address you land on. A quick test from the affected host:
for ip in $(getent ahostsv4 registry-1.docker.io | awk '{print $1}' | sort -u); do
printf '%-16s ' "$ip"; curl -s -o /dev/null -w '%{http_code}\n' --max-time 5 \
--resolve registry-1.docker.io:443:$ip https://registry-1.docker.io/v2/ || echo timeout
done
401 means reachable (that's Docker Hub's normal answer to an anonymous request); timeout means that address is dropped. A mix of both points at a firewall, router or ISP, not at the CI configuration. Pinning the good addresses in /etc/hosts is tempting, but which ones answer can change from minute to minute, so it doesn't hold up.
Mirror the base image
Copy the base image into the project's own Container Registry once, from a machine that can reach Docker Hub:
podman pull docker.io/library/python:3.13-slim
podman tag docker.io/library/python:3.13-slim registry.example.com/group/project/python:3.13-slim
podman push registry.example.com/group/project/python:3.13-slim
(docker works the same way.) Then build from it, keeping a way back to Docker Hub for local builds:
ARG BASE_IMAGE=registry.example.com/group/project/python:3.13-slim
FROM ${BASE_IMAGE}
RUN pip install --no-cache-dir ansible-core ansible-lint
docker build --build-arg BASE_IMAGE=docker.io/library/python:3.13-slim -t ansible-lint:local .
The build job already logs in to $CI_REGISTRY, so it can pull the mirrored image without extra setup. Refresh the mirror when you want a newer base.
Retry the build job
build-image:
retry: 2
Rebuilding and pushing the same tag is harmless, so retrying costs nothing but time. It covers the occasional timeout during pip install, which the mirror doesn't.
The real fix
Both of these are workarounds. If the network can't be fixed, the durable option is a pull-through cache (a registry mirror) on a host that can reach Docker Hub, configured as a mirror for the cluster's container runtime and for the image builder.
These commands were run against a GitLab runner on a private RKE2 (Kubernetes) node, not in the Linux cloud container where this entry was drafted.
Created 2026-10-25T03:02:00+01:00 · Edit