Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[frontend] Kubeflow pipeline experiment compare run cannot retrieve experiment namespace #11419

Open
LehmBook opened this issue Nov 29, 2024 · 0 comments
Open

[frontend] Kubeflow pipeline experiment compare run cannot retrieve experiment namespace #11419

LehmBook opened this issue Nov 29, 2024 · 0 comments
Labels
area/frontend kind/bug

Comments

@LehmBook
Copy link

LehmBook commented Nov 29, 2024 •
edited
Loading

Environment

  • How did you deploy Kubeflow Pipelines (KFP)?
    local deployment
    Kubeflow manifest 1.9.1
    k8s version:1.30.3
    deployed multi-user version kuebflow with the kustomization file in the example folder.
    all images are replaced with images from local storage
    GKE disabled

  • KFP version:
    kubeflow pipeline 2.3.0

Steps to reproduce

  1. checked the kustomization file to see which version of pipeline i deployed
# Kubeflow Pipelines
- ../apps/pipeline/upstream/env/cert-manager/platform-agnostic-multi-user

  1. click pipeline-experiment in the kubeflow UI, then click an experiment, and click compare,it says failed to get associated experiment
    1732847226836

  2. Then with F12, in pipeline-experiment page, when click any experiment or try compare,the result as follow

1732845184641
1732845708771

it says"List experiments failed: Failed to authorize with API: Invalid input error: An experiment cannot have an empty namespace in multi-user mode"
which is refer to the content in the file/backend/src/apiserver/server/experiment_server.go
https://github.com/kubeflow/pipelines/blob/58b2d8d721378fca4f35079f2aaaed92ad6bd36f/backend/src/apiserver/server/experiment_server.go#L342C1-L343C1

4.check if experiment have namespace
1732862362126

All the kubeflow pods are running fine
Also,I want to know if anyone else encountered the same problem or only me.

kubeflow-user-example-com   ctest-0                                                  2/2     Running            0             14d
kubeflow-user-example-com   jupyter-0                                                2/2     Running            0             42h
kubeflow-user-example-com   ml-pipeline-ui-artifact-d6bb58c54-grmxj                  2/2     Running            2 (16h ago)   20d
kubeflow-user-example-com   ml-pipeline-visualizationserver-745dbcb7fb-rfqpz         2/2     Running            0             20d
kubeflow-user-example-com   mnist-pipeline-tsdhj-system-container-impl-1440812143    1/2     ImagePullBackOff   0             7d23h
kubeflow-user-example-com   pvcviewer-codeserver-test-69b55fb6f7-4khh4               2/2     Running            0             14d
kubeflow-user-example-com   pvcviewer-test00-workspace-74cb74f89f-d4s8t              2/2     Running            0             17d
kubeflow-user-example-com   pvcviewer-workspace-8485f49f4f-wkmm7                     2/2     Running            0             2d23h
kubeflow-user-example-com   tensortst-8dcd5dcdb-h77qw                                2/2     Running            0             17d
kubeflow-user-example-com   test10-0                                                 2/2     Running            2 (3d ago)    15d
kubeflow                    admission-webhook-deployment-5c5959bd95-rnmr9            1/1     Running            0             20d
kubeflow                    cache-server-784dff7cfd-768b7                            2/2     Running            0             20d
kubeflow                    centraldashboard-56ff7d88c5-6c8dn                        2/2     Running            0             9d
kubeflow                    jupyter-web-app-deployment-648886f787-4zxx5              2/2     Running            0             13d
kubeflow                    katib-controller-f8df57f77-cqnxp                         1/1     Running            0             20d
kubeflow                    katib-db-manager-5dfb479d86-c9z22                        1/1     Running            0             20d
kubeflow                    katib-mysql-5c54f57f6b-4m2hj                             1/1     Running            0             20d
kubeflow                    katib-ui-6c587c9466-fnzhj                                2/2     Running            0             20d
kubeflow                    kserve-controller-manager-854b47d55d-wmvg4               2/2     Running            0             20d
kubeflow                    kserve-models-web-app-d7c86d76d-wqnfm                    2/2     Running            0             20d
kubeflow                    kubeflow-pipelines-profile-controller-7f78669c58-4r5pj   1/1     Running            0             20d
kubeflow                    metacontroller-0                                         1/1     Running            0             20d
kubeflow                    metadata-envoy-deployment-d4b98db88-qjk25                1/1     Running            0             20d
kubeflow                    metadata-grpc-deployment-6dd4c5fcb6-gvllw                2/2     Running            2 (20d ago)   20d
kubeflow                    metadata-writer-c98794db9-kh52q                          2/2     Running            0             20d
kubeflow                    minio-586bbdcc4-fcszh                                    2/2     Running            0             8d
kubeflow                    ml-pipeline-647947d6d4-5l7tv                             2/2     Running            0             20d
kubeflow                    ml-pipeline-persistenceagent-f4466586-q8wz4              2/2     Running            0             20d
kubeflow                    ml-pipeline-scheduledworkflow-55965d9f58-l4xxh           2/2     Running            0             20d
kubeflow                    ml-pipeline-ui-558979f6d4-2fj4j                          2/2     Running            0             18d
kubeflow                    ml-pipeline-viewer-crd-7db4586d47-z27fv                  2/2     Running            1 (20d ago)   20d
kubeflow                    ml-pipeline-visualizationserver-b687cd666-m6zdf          2/2     Running            0             20d
kubeflow                    mysql-675c9859d8-jklq4                                   2/2     Running            0             20d
kubeflow                    notebook-controller-deployment-546c7d4cb6-8qp5b          2/2     Running            2 (20d ago)   20d
kubeflow                    profiles-deployment-5cb7c65899-k77f5                     3/3     Running            1 (20d ago)   20d
kubeflow                    pvcviewer-controller-manager-d64c9564f-9vnv4             3/3     Running            0             20d
kubeflow                    tensorboard-controller-deployment-5c9c98bcdf-58gfk       3/3     Running            2 (17d ago)   17d
kubeflow                    tensorboards-web-app-deployment-6bbd97c7d4-22cjr         2/2     Running            0             20d
kubeflow                    training-operator-54f754bfcd-sq7k4                       1/1     Running            0             20d
kubeflow                    volumes-web-app-deployment-64cdf449dd-mfhk8              2/2     Running            0             17d
kubeflow                    workflow-controller-986648d9b-v4mzf                      2/2     Running            1 (13d ago)   13d
lsq-pace                    ml-pipeline-ui-artifact-d6bb58c54-bsggh                  2/2     Running            0             17d
lsq-pace                    ml-pipeline-visualizationserver-745dbcb7fb-6ssxg         2/2     Running            0             17d
oauth2-proxy                oauth2-proxy-57d8b58449-6vfvz                            1/1     Running            0             20d
oauth2-proxy                oauth2-proxy-57d8b58449-wjbk4                            1/1     Running            0             20d
tigera-operator             tigera-operator-76ff79f7fd-4bd7f                         1/1     Running            0             25d

Expected result

  1. the experiment of the runs shows correctly in the compare run page
  2. the experiment information fetched correctly in the compare run page and the pipeline-experiment-'my experiment' page

Materials and Reference

Impacted by this bug? Give it a 👍.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/frontend kind/bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@LehmBook