How does the auto creation of endpoints+endpointslices work #6954

jihuiyang · 2024-09-20T20:38:49Z

jihuiyang
Sep 20, 2024

I'm using the kube-prometheus-stack. Im confused to how the endpoints + endpointslices are created. So it auto creates kubelet endponts to monitor port 10250,10255,4194 on 2000+ nodes, in my case this works correctly

kube-prometheus-stack-kubelet                   10.171.100.26:10250,10.171.48.205:10250,10.171.69.67:10250 + 8670 more...   22m

but the endpointslices it generated there is only one slice with the 1000 limit

kube-prometheus-stack-kubelet-spdpp                   IPv4          10250,10255,4194   10.171.103.141,10.171.10.129,10.171.125.165 + 997 more...   5m21s

but for kube-proxy endpoint, it only has 1000 targets for endpoints

kube-prometheus-stack-kube-proxy                10.171.0.111:10249,10.171.0.122:10249,10.171.0.124:10249 + 997 more...      31m

but 20+ endpointslices with 100 ips each

kube-prometheus-stack-kube-proxy-nfmqx                IPv4          10249              10.171.89.28,10.171.125.214,10.171.45.22 + 97 more...       31m
kube-prometheus-stack-kube-proxy-qskx9                IPv4          10249              10.171.117.194,10.171.43.23,10.171.27.55 + 97 more...       31m
kube-prometheus-stack-kube-proxy-t9tv2                IPv4          10249              10.171.91.47,10.171.95.195,10.171.114.23 + 97 more...       31m

jihuiyang · 2024-09-20T20:40:49Z

jihuiyang
Sep 20, 2024
Author

i really want the kubelet endpointslices to be generated the same way as the kube-proxy. i use otel target allocator and it appears it only discovers by endpointslices

0 replies

jihuiyang · 2024-09-20T20:55:05Z

jihuiyang
Sep 20, 2024
Author

looks like i need to pass this

prometheus-operator/Documentation/operator.md

Lines 62 to 63 in 23cbc11

    
             -kubelet-endpointslice 
        
               	Create EndpointSlice objects for kubelet targets.

0 replies

jihuiyang · 2024-09-20T21:11:07Z

jihuiyang
Sep 20, 2024
Author

ts=2024-09-20T21:04:11.876661411Z level=info caller=/home/runner/work/prometheus-operator/prometheus-operator/cmd/operator/main.go:350 msg="Kubernetes API capabilities" endpointslices=true
ts=2024-09-20T21:04:11.932356082Z level=warn caller=/home/runner/work/prometheus-operator/prometheus-operator/cmd/operator/main.go:556 msg="missing permission on resource 'endpointslices' (group: \"discovery.k8s.io\")" reason="missing \"create\" permission on resource \"endpointslices\" (group: \"discovery.k8s.io\") for namespace \"kube-system\""
ts=2024-09-20T21:04:11.932398216Z level=warn caller=/home/runner/work/prometheus-operator/prometheus-operator/cmd/operator/main.go:556 msg="missing permission on resource 'endpointslices' (group: \"discovery.k8s.io\")" reason="missing \"update\" permission on resource \"endpointslices\" (group: \"discovery.k8s.io\") for namespace \"kube-system\""
ts=2024-09-20T21:04:11.932411586Z level=warn caller=/home/runner/work/prometheus-operator/prometheus-operator/cmd/operator/main.go:556 msg="missing permission on resource 'endpointslices' (group: \"discovery.k8s.io\")" reason="missing \"delete\" permission on resource \"endpointslices\" (group: \"discovery.k8s.io\") for namespace \"kube-system\""

looks like just updating the image version of operator in kube-prometheus-stack is not enough, no permission here. Do you have a timeline on when this change will be available to kube-prometheus-stack?

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How does the auto creation of endpoints+endpointslices work #6954

Uh oh!

{{title}}

Uh oh!

Replies: 3 comments

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

How does the auto creation of endpoints+endpointslices work #6954

Uh oh!

jihuiyang Sep 20, 2024

Replies: 3 comments

Uh oh!

jihuiyang Sep 20, 2024 Author

Uh oh!

jihuiyang Sep 20, 2024 Author

Uh oh!

jihuiyang Sep 20, 2024 Author

jihuiyang
Sep 20, 2024

jihuiyang
Sep 20, 2024
Author

jihuiyang
Sep 20, 2024
Author

jihuiyang
Sep 20, 2024
Author