Problem with Gitlab private registry and Swarm Stack using web editor #11079
Description
Before you start please confirm the following.
- Yes, I've searched similar issues on GitHub.
- Yes, I've checked whether this issue is covered in the Portainer documentation or knowledge base.
Problem Description
For a long time now, creating a stack using the web editor and filling a docker-compose in using a private registry which is existing in the Registries menu and that we can browse normally and see the related image, does not work as attended.
Stack fails to deploy with the following error :
Feb 7 09:23:15 docker dockerd[3741060]: time="2024-02-07T09:23:15.056607119+01:00" level=error msg="Not continuing with pull after error: errors:\ndenied: requested access to the resource is denied\nunauthorized: authentication required\n"
Feb 7 09:23:15 docker dockerd[3741060]: time="2024-02-07T09:23:15.057113552+01:00" level=info msg="Ignoring extra error returned from registry" error="unauthorized: authentication required"
Feb 7 09:23:15 docker dockerd[3741060]: time="2024-02-07T09:23:15.060066974+01:00" level=error msg="pulling image failed" error="pull access denied for registry.gitlab.com/toto/toto_project/toto-img, repository does not exist or may require 'docker login': denied: requested access to the resource is denied" module=node/agent/taskmanager node.id=os43f7oac0t1ldwqn6i6gxd3g service.id=hyz3frcbj571blhnbhnh0g58e task.id=xmwjcgjpfxvi6jzep19q0y62z
Feb 7 09:23:15 docker dockerd[3741060]: time="2024-02-07T09:23:15.060366644+01:00" level=error msg="fatal task error" error="No such image: registry.gitlab.com/toto/toto_project/toto-img" module=node/agent/taskmanager node.id=os43f7oac0t1ldwqn6i6gxd3g service.id=hyz3frcbj571blhnbhnh0g58e task.id=xmwjcgjpfxvi6jzep19q0y62z
Even if we try to do a docker login, same error is logged without the "repository does not exist or may require 'docker login'" log.
Sometimes, it works for a couple of days before being stuck again with these errors at the next webhook or manual polling.
The same container is successfully deployed if using Services and choosing the registry in the UI, instead of Stack which does not permit to use docker compose.
It sounds like the stack is not able to successfully use Registries credentials added successfully to portainer.
Expected Behavior
Stack is successfully deployed using the Registries set in Portainer and the associatied token with no error.
Actual Behavior
Stack is not successfully deployed using the Registries set in Portainer and the associatied token with authorization / authentication errror.
Steps to Reproduce
- Add a private gitlab registry using the Registries menu with a token that allow to use images (or with all permission, same result)
- Deploy a stack with the web editor using an image in the previously created gitlab registry as a container image
- Stack fails to deploy the image with the previous authentication/authorization error
Portainer logs or screenshots
Feb 7 09:23:15 docker dockerd[3741060]: time="2024-02-07T09:23:15.056607119+01:00" level=error msg="Not continuing with pull after error: errors:\ndenied: requested access to the resource is denied\nunauthorized: authentication required\n"
Feb 7 09:23:15 docker dockerd[3741060]: time="2024-02-07T09:23:15.057113552+01:00" level=info msg="Ignoring extra error returned from registry" error="unauthorized: authentication required"
Feb 7 09:23:15 docker dockerd[3741060]: time="2024-02-07T09:23:15.060066974+01:00" level=error msg="pulling image failed" error="pull access denied for registry.gitlab.com/toto/toto_project/toto-img, repository does not exist or may require 'docker login': denied: requested access to the resource is denied" module=node/agent/taskmanager node.id=os43f7oac0t1ldwqn6i6gxd3g service.id=hyz3frcbj571blhnbhnh0g58e task.id=xmwjcgjpfxvi6jzep19q0y62z
Feb 7 09:23:15 docker dockerd[3741060]: time="2024-02-07T09:23:15.060366644+01:00" level=error msg="fatal task error" error="No such image: registry.gitlab.com/toto/toto_project/toto-img" module=node/agent/taskmanager node.id=os43f7oac0t1ldwqn6i6gxd3g service.id=hyz3frcbj571blhnbhnh0g58e task.id=xmwjcgjpfxvi6jzep19q0y62z
Portainer version
2.19.4
Portainer Edition
Business Edition (BE/EE) with Starter Plus license
Platform and Version
Docker Swarm 24.0.7
OS and Architecture
Debian 11.8 AMD64
Browser
Chrome 121
What command did you use to deploy Portainer?
version: '3.8'
services:
appv2:
image: registry.gitlab.com/toto/toto_project/toto-img:latest
restart: always
ports:
- "6889:443"
extra_hosts:
- "toto.toto.chc:192.168.50.65"
deploy:
resources:
limits:
cpus: '6'
memory: 2048M
update_config:
order: start-first
restart_policy:
condition: anyAdditional Information
We did a new fresh install of Portainer 2.19.4, the result is exactly the same.