Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

controller config caas-image-repo supports private or non dockerhub repos #13192

Merged
merged 35 commits into from
Aug 25, 2021

Conversation

ycliuhw
Copy link
Member

@ycliuhw ycliuhw commented Jul 26, 2021

This PR introduces non-dockerhub/private OCI registry support for:

  • controller pods;
  • model operators;
  • application operators;
  • charm container's base image in sidecar pods;

This PR currently enabled basic auth and auth token support.
The v2 bear token is still a TODO and will be added in the following PRs.

Note: this feature is currently hidden under featureflag: private-registry, but it's not testable without some hack because this bug(it will be fixed soon).

Checklist

  • Requires a pylibjuju change
  • Added integration tests for the PR
  • Added or updated doc.go related to packages changed
  • Comments answer the question of why design decisions were made

QA steps

private repo on non dockerhub

$ cat tttt.json
{
    "serveraddress": "quay.io",
    "auth": "xxxxxx==",
    // "username": "a",
    // "password": "pwd",
    "repository": "quay.io/ycliuhw"
}

$ JUJU_DEV_FEATURE_FLAGS=private-registry juju bootstrap microk8s k1 --config caas-image-repo="'$(cat tttt.json)'"

$ mkubectl get all,pv,pvc,ing -o wide -ncontroller-k1
NAME                                 READY   STATUS    RESTARTS   AGE     IP            NODE                  NOMINATED NODE   READINESS GATES
pod/modeloperator-687cb75dd8-c5p8c   1/1     Running   0          2m51s   10.1.97.208   kelvinliu-m15-ryzen   <none>           <none>
pod/controller-0                     2/2     Running   2          3m48s   10.1.97.207   kelvinliu-m15-ryzen   <none>           <none>

NAME                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)     AGE     SELECTOR
service/controller-service   ClusterIP   10.152.183.22    <none>        17070/TCP   3m51s   app.kubernetes.io/name=controller
service/modeloperator        ClusterIP   10.152.183.161   <none>        17071/TCP   2m51s   operator.juju.is/name=modeloperator,operator.juju.is/target=model

NAME                            READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS      IMAGES                                    SELECTOR
deployment.apps/modeloperator   1/1     1            1           2m51s   juju-operator   quay.io/ycliuhw/jujud-operator:2.9.10.1   operator.juju.is/name=modeloperator,operator.juju.is/target=model

NAME                                       DESIRED   CURRENT   READY   AGE     CONTAINERS      IMAGES                                    SELECTOR
replicaset.apps/modeloperator-687cb75dd8   1         1         1       2m51s   juju-operator   quay.io/ycliuhw/jujud-operator:2.9.10.1   operator.juju.is/name=modeloperator,operator.juju.is/target=model,pod-template-hash=687cb75dd8

NAME                          READY   AGE     CONTAINERS           IMAGES
statefulset.apps/controller   1/1     3m48s   mongodb,api-server   quay.io/ycliuhw/juju-db:4.0,quay.io/ycliuhw/jujud-operator:2.9.10.1

NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                STORAGECLASS        REASON   AGE     VOLUMEMODE
persistentvolume/pvc-30c9754b-9b51-4182-bb14-0ebca15a49f5   20Gi       RWO            Delete           Bound    controller-k1/storage-controller-0   microk8s-hostpath            3m48s   Filesystem

NAME                                         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS        AGE     VOLUMEMODE
persistentvolumeclaim/storage-controller-0   Bound    pvc-30c9754b-9b51-4182-bb14-0ebca15a49f5   20Gi       RWO            microk8s-hostpath   3m48s   Filesystem

$ juju add-model t1

$ juju deploy snappass-test

$ juju deploy cs:~juju/mariadb-k8s-3

$ mkubectl -ncontroller-k1 get pod/controller-0 -o json | jq .spec.imagePullSecrets
[
  {
    "name": "juju-image-pull-secret"
  }
]

$ mkubectl -ncontroller-k1 get secret/juju-image-pull-secret -o json | jq -r '.data[".dockerconfigjson"]' | base64 --decode | jq
{
  "auths": {
    "quay.io": {
      "auth": "xxxxxx==",
      "serveraddress": "quay.io"
    }
  }
}

$ mkubectl -nt1 get pods snappass-test-0 mariadb-k8s-operator-0 modeloperator-56b469cd6-n8dxp -o json |
 jq '.items[].spec.imagePullSecrets'
[
  {
    "name": "juju-image-pull-secret"
  }
]
[
  {
    "name": "juju-image-pull-secret"
  }
]
[
  {
    "name": "juju-image-pull-secret"
  }
]

$ mkubectl -nt1 get secret/juju-image-pull-secret -o json | jq -r '.data[".dockerconfigjson"]' | base64 --decode | jq
{
  "auths": {
    "quay.io": {
      "auth": "xxxxxx==",
      "serveraddress": "quay.io"
    }
  }
}

$ mkubectl -nt1 get pod/snappass-test-0 -o json | jq '.spec.containers[].image'
"quay.io/ycliuhw/charm-base:ubuntu-20.04"
"registry.hub.docker.com/library/redis@sha256:27bea08330830c6e5efc456075d66eb1fb4b76e5518947e8c87556adf1df3e51"
"registry.hub.docker.com/benhoyt/snappass-test@sha256:32506b5ac7bde75b7a76bee003cf8c77f30fc936f446c8c52d457b8fe6fce96f"

$ mkubectl -nt1 get pod/mariadb-k8s-operator-0 -o json | jq '.spec.containers[].image'
"quay.io/ycliuhw/jujud-operator:2.9.10"

$ mkubectl -nt1 get pod/modeloperator-56b469cd6-n8dxp -o json | jq '.spec.containers[].image'
"quay.io/ycliuhw/jujud-operator:2.9.10"

$ mkubectl -ncontroller-k1 get pod/controller-0  -o json | jq '.spec.containers[].image'
"quay.io/ycliuhw/juju-db:4.0"
"quay.io/ycliuhw/jujud-operator:2.9.10"

$ JUJU_DEV_FEATURE_FLAGS=private-registry juju upgrade-controller --agent-stream=develop

public repo on dockerhub

$ juju bootstrap microk8s k1 --config caas-image-repo=ycliuhw && juju add-model t1

$ mkubectl -ncontroller-k1 get pod/controller-0 -o json | jq .spec.imagePullSecrets
null

$ mkubectl -ncontroller-k1 get secret/juju-image-pull-secret
Error from server (NotFound): secrets "juju-image-pull-secret" not found

$ mkubectl -nt1 get pods snappass-test-0 mariadb-k8s-operator-0 modeloperator-84c9b5c68-9259k -o json |
 jq '.items[].spec.imagePullSecrets'
null
null
null

$ mkubectl -nt1 get secret/juju-image-pull-secret
Error from server (NotFound): secrets "juju-image-pull-secret" not found

$ mkubectl -nt1 get pod/snappass-test-0 -o json | jq '.spec.containers[].image'

"ycliuhw/charm-base:ubuntu-20.04"
"registry.hub.docker.com/library/redis@sha256:27bea08330830c6e5efc456075d66eb1fb4b76e5518947e8c87556adf1df3e51"
"registry.hub.docker.com/benhoyt/snappass-test@sha256:32506b5ac7bde75b7a76bee003cf8c77f30fc936f446c8c52d457b8fe6fce96f"

$ mkubectl -nt1 get pod/mariadb-k8s-operator-0 -o json | jq '.spec.containers[].image'
"ycliuhw/jujud-operator:2.9.10"

$ mkubectl -nt1 get pod/modeloperator-56b469cd6-n8dxp -o json | jq '.spec.containers[].image'
Error from server (NotFound): pods "modeloperator-56b469cd6-n8dxp" not found

$ mkubectl -nt1 get pod/modeloperator-84c9b5c68-9259k -o json | jq '.spec.containers[].image'
"ycliuhw/jujud-operator:2.9.10"

$ mkubectl -ncontroller-k1 get pod/controller-0  -o json | jq '.spec.containers[].image'

"ycliuhw/juju-db:4.0"
"ycliuhw/jujud-operator:2.9.10"

Documentation changes

Yes

Bug reference

https://bugs.launchpad.net/juju/+bug/1935953
https://bugs.launchpad.net/juju/+bug/1935830

@ycliuhw ycliuhw changed the title Make juju-db repo configurable like juju operator image; controller config caas-image-repo supports private or non dockerhub repos Aug 10, 2021
@jnsgruk
Copy link
Member

jnsgruk commented Aug 19, 2021

This is looking promising, let's please make sure a section ends up in the docs for this! :)

@ycliuhw
Copy link
Member Author

ycliuhw commented Aug 23, 2021

This is looking promising, let's please make sure a section ends up in the docs for this! :)

Yes, That's in the plan.

@ycliuhw ycliuhw marked this pull request as ready for review August 23, 2021 09:06
@ycliuhw
Copy link
Member Author

ycliuhw commented Aug 23, 2021

@ycliuhw
Copy link
Member Author

ycliuhw commented Aug 23, 2021

Copy link
Member

@wallyworld wallyworld left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks nice.
One issue that needs fixing before landing - the apiserver/params should not use structs from other packages. We need to define bespoke structs in params and use helper methods to marshall/unmarshall.

api/caasapplicationprovisioner/client.go Outdated Show resolved Hide resolved
apiserver/params/params.go Outdated Show resolved Hide resolved
apiserver/params/caas.go Outdated Show resolved Hide resolved
docker/registry/registry.go Outdated Show resolved Hide resolved
docker/registry/registry.go Show resolved Hide resolved
state/docker_resource.go Outdated Show resolved Hide resolved
api/caasmodelconfigmanager/client.go Outdated Show resolved Hide resolved
docker/auth.go Show resolved Hide resolved
@ycliuhw
Copy link
Member Author

ycliuhw commented Aug 24, 2021

!!build!!

@ycliuhw
Copy link
Member Author

ycliuhw commented Aug 25, 2021

$$merge$$

@jujubot jujubot merged commit 57d78e4 into juju:2.9 Aug 25, 2021
@manadart manadart mentioned this pull request Sep 3, 2021
@hpidcock hpidcock mentioned this pull request Sep 5, 2021
jujubot added a commit that referenced this pull request Sep 8, 2021
#13304

Merge from 2.9 bringing forward:
- #13303 from wallyworld/unit-relation-tracing
- #13279 from juju/dependabot/github_actions/actions/setup-go-2.1.4
- #13293 from wallyworld/rotate-secret-watcher
- #13300 from manadart/2.9-net-info-mod-cfg
- #13292 from achilleasa/2.9-use-cidr-in-cidr-fallback-when-matching-machine-addr-cidrs-to-subnets
- #12569 from juju/devVersions
- #13286 from hpidcock/fix-report-hook-error
- #13285 from ycliuhw/fix/registry-basic-auth
- #13289 from achilleasa/2.9-ignore-fan-subnets-when-querying-equinix-api
- #13290 from wallyworld/remote-application-history
- #13281 from ycliuhw/fix/lp-1941055
- #13288 from SimonRichardson/add-logging-for-missing-relation-key
- #13287 from SimonRichardson/trap-upgrade-series-interrupt
- #13283 from SimonRichardson/remote-connections-not-found
- #13284 from wallyworld/remote-relationunit-notfound
- #13276 from manadart/2.9-lease-manager-tick-schedule
- #13282 from manadart/2.9-net-get-bridge-address
- #13280 from SimonRichardson/upstream-raft-part-deux
- #13192 from ycliuhw/fix/lp-1935953
- #13251 from jameinel/2.9-proxy-issue-4
- #13275 from jujubot/increment-to-2.9.13
- #13273 from wallyworld/update-secrets
- #13270 from manadart/2.9-remote-rels-suite

Conflicts:
- .github/workflows/build.yml
- .github/workflows/client-tests.yml
- .github/workflows/static-analysis.yml
- CONTRIBUTING.md
- caas/broker.go
- caas/kubernetes/provider/bootstrap.go
- caas/kubernetes/provider/bootstrap_test.go
- caas/kubernetes/provider/operator.go
- caas/mocks/broker_mock.go
- cloudconfig/podcfg/image.go
- cloudconfig/podcfg/podcfg_test.go
- cmd/juju/commands/upgrademodel.go
- cmd/juju/commands/upgrademodel_test.go
- controller/config_test.go
- feature/flags.go
- scripts/win-installer/setup.iss
- snap/snapcraft.yaml
- version/version.go
- worker/uniter/relation/statetracker.go
- worker/uniter/uniter_test.go
@ycliuhw
Copy link
Member Author

ycliuhw commented Sep 21, 2021

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants