Update - 2021.9.13

edited by @Bobgy

KFP 1.7.0 is released.
documentation is live: https://www.kubeflow.org/docs/components/pipelines/installation/choose-executor/
choosing an executor will be an option starting from KFP 1.7.

Update - 2021.8.6

edited by @Bobgy

Discussed with @kramachandran and decided to delay the default change for at least one KFP minor version, because argo emissary executor is still in Alpha state. There can be other corner cases that we haven't found, so we should give people some time to test it out.

Therefore, I'd like to propose the following:

• still default to docker executor in 1.7.0 release
• provide emissary executor as an option and document how to migrate
• collect feedback from people who starts to try emissary executor

TODOs:

• docs(kfp): choose a workflow executor. Part of kubeflow/pipelines#5718 website#2857
• Make emissary available in installations
  • Make emissary an option in AI Platform Pipelines #6294
  • Make env/platform-agnostic-emissary
• Make components emissary compatible #6252
• Upgrade kfp-ci test infra to argo v3.1.1 kubeflow/testing@03c6258

Issues discovered when testing argo v3.1.0 with KFP:

• [frontend] artifact preview & visualization breaks with argo v3.1+ #5930
• [bug] condition sample stuck in running state with argo v3.1.0 #5937 (fixed in v3.1.1)
• [frontend] incorrect DAG with argo v3.1.0 #5944 (fixed in v3.1.1)
• [emissary] pod hangs running as non-root argoproj/argo-workflows#6238

=== the following are the original proposal ===

What feature would you like to see?

Update argo to 3.11 such that we can use the emissary executor as decided by @Bobgy in #4645 (comment)

What is the use case or pain point?

The Docker, Kubelet, PNS and K8sapi executors have severe limitations that will be solved by Argo 3.1 and the emissary executor.

Here is an overview https://argoproj.github.io/argo-workflows/workflow-executors/

In a nutshell docker breaks security completely and is incompatible with kubernetes 1.19
#5714 Kubernetes is also moving to containerd anyway and a lot of users want proper non-docker support for a long time #1654

Kubelet needs some configuration and has the same limitation as k8sapi and rootless PNS: "Output artifacts must be saved on volumes (e.g. emptyDir) and not the base image layer (e.g. /tmp)" so it breaks the leightweight python components if there is no volume mounted or the output directory is inside the base layer #4645

One can use a PNS with root rights (PTRACE and CHROOT) to circumvent this, but this obviously breaks the security again.

So only the emissary executor https://argoproj.github.io/argo-workflows/workflow-executors/#emissary-emissary is feasible which does more ore less what i proposed in #4645 and what was started by Argos @alexec in argoproj/argo-workflows#4766

The Argo update has been done before in #5266 and #4693 by @xinbinhuang @Bobgy

Is there a workaround currently?

# Emmissary executor, needs argo 3.1+
# Sadly visualizations are broken(26.04.2021) for k8sapi and emissary

kubectl -n kubeflow patch configmap/workflow-controller-configmap --type='json' -p='[{"op": "replace", "path": "/data/containerRuntimeExecutor", "value": "emissary"}]'

kubectl edit deployment/workflow-controller -n kubeflow
...
    spec:
      containers:
      - env:
        - name: "LEADER_ELECTION_DISABLE"
       	  value: true
      - args:
	- --configmap
        - workflow-controller-configmap
        - --executor-image
        - docker.io/argoproj/argoexec:latest
        command:
        - workflow-controller
        image: docker.io/argoproj/workflow-controller:latest    
        env:
	- name: "LEADER_ELECTION_DISABLE"
          value: "true"
...
kubectl -n kubeflow patch configmap/workflow-controller-configmap --type='json' -p='[{"op": "replace", "path": "/data/containerRuntimeExecutor", "value": "emissary"}]'
kubectl rollout restart deployment workflow-controller -n kubeflow

or manually patching the sdk and using k8sapi as done in https://github.com/kubeflow/pipelines/pull/4645

Or use kfp-tekton

Love this idea? Give it a 👍. We prioritize fulfilling features with the most 👍.