xref: #40182

@dchen1107 is this a blocker for 1.6?

Hi @dchen1107, I'd like to help validate docker 1.13.x and 17.03.x. Can you help me understand what kind of validation currently runs?

Is this expected to be done for k8s 1.7? With an RC cut I'm guessing probably not.

EOL for docker 1.12 is 2017-11-14 according to their maintenance lifecycle page. Is this issue still active, or is there another one that is tracking the validation of newer versions of docker engine?

Yes, it's active and being tracked in this issue (if you see the attached PRs).

• The node/cluster e2e tests are running fine against the cos image with docker 1.13.
• The test for shared PID namespace has been added.
• The test for live-restore is in review and is expected to be merged today. Add node e2e test for Docker's live-restore #50277.
• The test for overlay2 is under development.
• We added benchmarks for docker 1.13 on http://node-perf-dash.k8s.io, and I will run the docker micro-benchmark tool, and provide the data.

I will have a full summary no latter than next Monday.

Unfortunately, at the time of the 1.8 release, docker 17.03-ce will not be a supported version.

Docker CE releases are only supported for a 4 month period, so 17.03 was EOL'd by August.

That means that Kubernetes still warns on every single version of docker-ce that is supported by upstream.

It seems irresponsible to recommend users use software which will not be receiving security updates.
I have no objection with e.g. GKE's container-vm or RHEL or tectonic choosing a version not updated by docker inc because they'll apply updates themselves.

However, for the open source project to recommend unsupported software seems wrong to me.

I can open a new issue for 17.06/17.09 specifically, but this doesn't seem like a scaleable approach.

Additionally, as of last night, 17.06 is no longer officially supported in the open source editions of Docker (only 17.07 and now 17.09).

Edit: @cpuguy83 was kind enough to correct me privately on this -- 17.06 is supported for one more month (the "stable" releases overlap for one month; see https://blog.docker.com/2017/03/docker-enterprise-edition/ which has a graphic that includes CE to illustrate this) ❤️

@luxas all validated versions have been EOLd. 1.13 CSE ("Commercially Supported Edition") might still be supported by docker inc, but that's a closed source fork of docker, not docker 1.13.

Kubernetes probably needs to just explicitly not be recommending specific versions of the engine, but rather support a version of the dockerd API.. and then test against upstream to ensure things aren't breaking.

As far as I can tell, other components (like the Linux kernel, or iptables, or cgroups) we don't declare compatibility with exact numbers, but rather with an ABI or a minimum version.

@tessiof

You can also move away from docker to cri-o or cri-containerd..

My complaint is about what Kubernetes marks as validated/supported.
If Kubernetes quits recommending docker entirely in favor of containerd/cri-o then I'll happily not complain about them recommending unsupported software.

Publishing the Docker API version sounds reasonable.

On the other hand, most of the value of the Docker validation lies within discovering/fixing/working around Docker engine-version-specific bugs. I am not sure I'd compare docker to cgroups or iptables, and I think it's still valuable to publish this information in some form.

Another thing is that we could provide an option (in dockershim) to use a specific docker API version. That way, users can continue upgrading their docker engine while staying on the same API version. Of course, they will have to deal with bugs themselves.

If Kubernetes quits recommending docker entirely in favor of containerd/cri-o then I'll happily not complain about them recommending unsupported software.

I don't think Kubernetes recommends docker. It just happens to be the only feature complete, production-ready integration we have now.
Other CRI implementations are built on more stable grounds, so hopefully this gets better in the future.

FYI, I filed a new issue for using the docker API version: #53221

Seeing this thread and the discussion just raises the question, Why Docker at all? Should we maybe move towards another container engine that maybe follows the Kubernetes release cycle closer?

@roffe That’s what projects like cri-o, cri-containerd, and rktlet are working towards. They’re just not as mature as Docker at this point. (You’ll find all three projects under the kubernetes-incubator GitHub org.)

How long does kubernetes support its releases?

What speaks against following Docker via shorter patch release cycles?

@jimmycuadra I am already running RKT for my core components. but there has been some problems running it for the containers with GC not working as expected and such.

Regarding CRI-O i would love to test it on CoreOS but it requires me to build the OS myself from scratch to make modifications needed to include deps for CRI-O and that is over my head.

@wkornewald i would say Docker itself, it's not rare with regressions and them making breaking changes between versions.

@roffe

@wkornewald i would say Docker itself, it's not rare with regressions and them making breaking changes between versions.

Speaking as a project maintainer on moby/moby, if you find a breaking change it's a bug. Please report as quickly as possible so we can get it fixed in a patch release.

@cpuguy83

Speaking as a project maintainer on moby/moby, if you find a breaking change it's a bug. Please report as quickly as possible so we can get it fixed in a patch release.

Would it be possible to run the (Docker-specific parts of the) Kubernetes test suite before each Docker release in order to ensure there are no breaking changes or regressions? After all, Kubernetes is an important use-case for Docker. Also, other projects building on top of Docker would surely benefit from the additional compatibility guarantees brought by the extra test suite, too.

@wkornewald docker provides release candidates, which allows other projects (like Kubernetes) to run such tests against releases before they start breaking users.

It's rare for a project to take on the burden of running other project's tests for numerous reasons. In this case, it would be an especially large burden since running the Kubernetes test suite is so involved.

An RC period during which projects that care test and report issues is a more normal and, in my opinion, healthy way to handle that issue. The Linux Kernel is probably the most notable user of that model, and it works fairly well for them.

I would much rather argue that the Kubernetes project, or possibly community, should consume docker-ce RCs and alert them if/when regressions are found.

Now that Docker Enterprise announced Kubernetes as an officially supported orchestration platform besides Swarm, wouldn't it mean it's in the best interest of Docker Inc. to work on test suites & regression tests to ensure smooth integration as well? I presume Docker Inc. and/or their Enterprise customers will probably want to run Kubernetes with latest Docker version instead of being stuck with the past/patched Docker versions indefinitely.

I would much rather argue that the Kubernetes project, or possibly community, should consume docker-ce RCs and alert them if/when regressions are found.

I'd love to see the community pick up the work validating Docker RCs and report issues. I think that's orthogonal to validating Docker (in kubernetes) against a fixed API version (#53221).

Now that Docker Enterprise announced Kubernetes as an officially supported orchestration platform besides Swarm, wouldn't it mean it's in the best interest of Docker Inc. to work on test suites & regression tests to ensure smooth integration as well? I presume Docker Inc. and/or their Enterprise customers will probably want to run Kubernetes with latest Docker version instead of being stuck with the past/patched Docker versions indefinitely.

I have no clue how Docker plans to validate Kubernetes releases. Perhaps @cpuguy83 can provide some insight?

I couldn't comment on immediate plans (beta hasn't started yet), but definitely expect to see kubernetes running on top of containerd directly as part of the (Docker) platform instead of sitting on top of the platform.

containerd, once out of beta, will be a much more stable base (much smaller scope) and have much longer support cycle (https://github.com/containerd/containerd/blob/master/RELEASES.md#support-horizon)
See https://github.com/kubernetes-incubator/cri-containerd/ for this integration work.

containerd is already being tested with kube node e2e tests: https://k8s-testgrid.appspot.com/sig-node-containerd, more work to be done for cluster e2e tests (so I'm told).

Hi, I'm trying to locate documentation from kubernetes that recommends running kubernetes 1.8.x or 1.9.x with which docker version on CentOS 7.4, Ubuntu 16.04.

@bamb00 looking for the same...

Check https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.9.md section: External Dependencies.

But to be honest we are using docker ce 17.06.x and it's working ok for us.

What is the preferred storage driver used with Kubernetes 1.9.x in a production environment for CentOS and RHEL? Docker listed the preferrable storage driver is overlay2 but I would like to hear from the user what is being use and why.

Thanks in Advance.

@bamb00 I think this probably isn't the right issue to discuss that on 😉 (Maybe try the community on Slack?)