Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Emissary Executor feedback #6249

Closed
Bobgy opened this issue Aug 6, 2021 · 19 comments
Closed

Emissary Executor feedback #6249

Bobgy opened this issue Aug 6, 2021 · 19 comments

Comments

@Bobgy
Copy link
Contributor

Bobgy commented Aug 6, 2021

Part of #5718
Emissary Executor Documentation: https://www.kubeflow.org/docs/components/pipelines/installation/choose-executor/

As proposed, we want to ask for feedback about emissary executor.

Emissary executor is currently in Alpha state, but the Kubeflow Pipelines team is actively helping to make it mature. The team has been able to run Kubeflow Pipelines test infra stably with emissary executor. We are now releasing emissary executor as an option when deploying KFP and we are considering making it the default in the future.

Therefore, welcome feedbacks & bug reports for emissary executor!

For feedbacks, comment on this issue.
For bug reports, create a separate issue logging the details and comment about it here.

@Bobgy
Copy link
Contributor Author

Bobgy commented Aug 6, 2021

If you have already tried emissary executor:

  • thumbs up on this comment if it works well for you
  • thumbs down on this comment if you hit some problems (and welcome a separate comment about the problem you hit)

@Bobgy Bobgy pinned this issue Aug 16, 2021
@wdma
Copy link

wdma commented Oct 1, 2021

Hello @Bogby, I am trying out the emmisary executor with Argo workflows. It appears that simply applying a configmap specifying the emmisary executor is sufficient to break Argo. After application, the workflow will launch, but pods do not initialize. Has anyone else experienced this issue or can you tell me what am I doing wrong?

my configmap

---
{
   "apiVersion": "v1",
   "data": {
      "containerRuntimeExecutor": "emmisary",
      "instanceID": "hippocampus-controller",
      "workflowDefaults": "{\"metadata\": {},\"spec\": {}}"
   },
   "kind": "ConfigMap",
   "metadata": {
      "name": "workflow-controller-configmap",
   }
}

I am testing two different workflows in the default and argo namespaces

  1. hello-world
argo submit --watch https://raw.githubusercontent.com/argoproj/argo-workflows/master/examples/hello-world.yaml
  1. a test workflow
---
{
   "apiVersion": "argoproj.io/v1alpha1",
   "kind": "Workflow",
   "metadata": {
      "annotations": {
         "argo": "workflows"
      },
      "generateName": "hello-test-",
      "labels": {
         "workflows.argoproj.io/archive-strategy": "false"
      }
   },
   "spec": {
      "entrypoint": "entrypoint",
      "parallelism": 3,
      "podGC": {
         "strategy": "OnWorkflowSuccess"
      },
      "securityContext": {
         "runAsNonRoot": true,
 #        "runAsUser": 1000,
#        "runAsGroup": 3000,
#        "fsGroup": 2000,
      },
      "templates": [
         {
            "dag": {
               "tasks": [
                  {
                     "name": "h",
                     "template": "hello"
                  }
               ]
            },
            "name": "entrypoint"
         },
         {
            "name": "hello",
            "script": {
               "image": "alpine:latest",
               "imagePullPolicy": "IfNotPresent",
               "command": ["bin/sh", "-c"],
               "source": "echo Hello; exit",
               "securityContext": {
                 "runAsNonRoot": true,
               },
            },
         }   
      ],
      "ttlStrategy": {
         "secondsAfterSuccess": 5
      }
   }
}
...

@Bobgy
Copy link
Contributor Author

Bobgy commented Oct 15, 2021

Hi @wdma, thank you for trying out!

Yes, when updating the config argo will behave in a completely different mode. If you hit issues, we can help you investigate.

Can you report a separate issue with more details? E.g. can you copy all logs/events for a Pod that do not start?

@wdma
Copy link

wdma commented Oct 15, 2021

Hi @Bobgy Thank you for your response. I have been working through a range of issues on the Argo Github site. They have been very helpful and I have emissary working for the most part. Can you tell me if this (latest) issue might be related to emissary (argoproj/argo-workflows#6942)?

If so, I would greatly appreciate your insights. If not, you can go ahead and close this as I think I am now past my issues with the emissary executor :)

@juliusvonkohout
Copy link
Member

I can report for a team of 10 data scientists that all pipelines in v1 mode work with runasnonroot. I think you should make it the default.

@wdma
Copy link

wdma commented Oct 18, 2021

@Bogby can you tell me if some setting in emissary might be the cause of my current problem? I can now successfully execute a workflow, but I have no control over the serviceaccount and the permissions for whichever serviceaccount is used are insuffucient. I have described the problem in detail here https://stackoverflow.com/questions/69607918/serviceaccount-name-does-not-stick

In short,I think this may be two problems (permissions issue and the service account issue), but I am not certain where to start teasing these two out. I like the fact that I can runasnonroot and just want it to work!

@juliusvonkohout
Copy link
Member

@Bogby can you tell me if some setting in emissary might be the cause of my current problem? I can now successfully execute a workflow, but I have no control over the serviceaccount and the permissions for whichever serviceaccount is used are insuffucient. I have described the problem in detail here https://stackoverflow.com/questions/69607918/serviceaccount-name-does-not-stick

In short,I think this may be two problems (permissions issue and the service account issue), but I am not certain where to start teasing these two out. I like the fact that I can runasnonroot and just want it to work

Just give default-editor (or whatever service account you are using the necessary permissions) a full Kubeflow install does this by default. Since you are using Argo directly without the kubeflow pipelines SDK you are in the wrong GitHub project.

@wdma
Copy link

wdma commented Oct 19, 2021

@juliusvonkohout thank you for your response. I have solved this problem. Will note the project in the future!

@juliusvonkohout
Copy link
Member

@Bobgy Argo itself will make it the default in argo>=3.3. according to https://argoproj.github.io/argo-workflows/workflow-executors/#emissary-emissary

@Bobgy
Copy link
Contributor Author

Bobgy commented Nov 24, 2021

Thanks for the FYI, it's good to see that we are making the same choice!

@stale
Copy link

stale bot commented Mar 2, 2022

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the lifecycle/stale The issue / pull request is stale, any activities remove this label. label Mar 2, 2022
@IronPan IronPan unpinned this issue Mar 25, 2022
@stale stale bot removed the lifecycle/stale The issue / pull request is stale, any activities remove this label. label Mar 25, 2022
@amosaini
Copy link

amosaini commented Jul 6, 2022

#7986 Trying to convert docx to pdf with libreoffice. it fails there.

@ajaypbrt
Copy link

ajaypbrt commented May 10, 2023

This may be relevant here: #9292. Seems to be a bug related to background processes in emissary that is fixed in the latest Argo.

Copy link

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the lifecycle/stale The issue / pull request is stale, any activities remove this label. label Jun 18, 2024
@juliusvonkohout
Copy link
Member

/close

emissary is the only executor still supported

Copy link

@juliusvonkohout: Closing this issue.

In response to this:

/close

emissary is the only executor still supported

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@stale stale bot removed the lifecycle/stale The issue / pull request is stale, any activities remove this label. label Jun 18, 2024
@juliusvonkohout
Copy link
Member

@rimolive if there are any PNS or so leftovers we have to get rid of them. Newer Argo only supports emissary.

@gregsheremeta
Copy link
Contributor

if there are any PNS or so leftovers we have to get rid of them

did you mean leftovers in the code, or PRs, issues, or all of that?

@juliusvonkohout
Copy link
Member

if there are any PNS or so leftovers we have to get rid of them

did you mean leftovers in the code, or PRs, issues, or all of that?

All three areas can be cleaned up. In the code I only found one file so far that contained PNS.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Closed
Development

No branches or pull requests

6 participants