Skip to content

Operator hardcode the Prometheus probe type to HTTPS when https and web.config enabled #8421

@davidquan-broadcom

Description

@davidquan-broadcom

Is there an existing issue for this?

  • I have searched the existing issues

What happened?

Description

This PR addresses the "Multiple Handler" validation error that occurs when users attempt to override the default httpGet probe with a tcpSocket or exec probe, particularly when Web TLS and Basic Auth are enabled via web-config.

Currently, when web.config is configured for HTTPS, the Operator automatically reconciles and enforces an HTTPS httpGet probe. When a user provides a custom probe (e.g., tcpSocket) in the Prometheus CR to bypass authentication or avoid plaintext credential exposure, the StrategicMergePatch logic preserves both the Operator's default and the user's override. This results in a StatefulSet that violates the Kubernetes API schema: may not specify more than 1 handler type.

Actually prometheus support config the TCP Probes when https enabled. And found related Web TLS and Basic Auth PRs still opened for a long time. So could we support the Probe to TCP instead of hardcode the HTTPS?

Related Issues & PRs which still not merged till now :

  1. Readiness probe fails with mTLS enabled and "RequireAndVerifyClientCert" clientAuthType #5419 Specifically addresses the configuration conflict when Web TLS is enabled.
  2. Support for using HTTPS in the Prometheus web UI is incomplete #4273 Addresses the hardcoded probe scheme limitation.
  3. feat: operator support set basic auth for prometheus server #7004 Provides a way to use tcpSocket probes to avoid the credential exposure issues discussed in without requiring new Secret-referencing logic in probes.

Community Prometheus behavior

Prometheus support using the TCP probe when Web TLS and Basic Auth are enabled.

Expected Result

Operator should support using the TCP probe when Web TLS and Basic Auth are enabled instead of HARDCODE of HTTPS.

Actual Result

Operator hardcode the prometheus probe to HTTPS even if config the TCP Probe in prometheus CR.

Source Code

statefulset.go Lines 203-204

	startupProbe, readinessProbe, livenessProbe := cg.BuildProbes()

promcfg.go Lines 1308-1316

	handler.HTTPGet = &v1.HTTPGetAction{
		Path: probePath,
		Port: intstr.FromString(cpf.PortName),
	}
	if cpf.Web != nil && cpf.Web.TLSConfig != nil && cg.IsCompatible() {
		handler.HTTPGet.Scheme = v1.URISchemeHTTPS
	}
	return handler

Prometheus Operator Version

v0.88

Kubernetes Version

1.34

Kubernetes Cluster Type

kind

How did you deploy Prometheus-Operator?

prometheus-operator/kube-prometheus

Manifests

prometheus-operator log output

Message:               creating statefulset failed: StatefulSet.apps "prometheus-prometheus" is invalid: [spec.template.spec.containers[0].livenessProbe.tcpSocket: Forbidden: may not specify more than 1 handler type, spec.template.spec.containers[0].readinessProbe.tcpSocket: Forbidden: may not specify more than 1 handler type, spec.template.spec.containers[0].startupProbe.tcpSocket: Forbidden: may not specify more than 1 handler type]

Anything else?

No response

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions