@simonpasquier hi, i have changed the type of basicAuthUsers to BasicAuth and basis auth info store in secret (should be created maunally first) currently. and when reconcile prometheus, we will set statefulset's probes with basic auth headers automaticly. But there's still have a promblem that, user can get basic auth info by decoding authorization info which in prometheus liveness probe or readiness probe.

There's also the question of the Thanos sidecar and config reloader which require authentication credentials too.
Another option instead of picking up a random user/password from the list provided by the user would be to create a special username with a random password and inject the credentials in the generated passwd file.

There's also the question of the Thanos sidecar and config reloader which require authentication credentials too. Another option instead of picking up a random user/password from the list provided by the user would be to create a special username with a random password and inject the credentials in the generated passwd file.

ok.I will finish it. bytheway, maybe prometheus and other components can exclued health check api for authicated like the issue prometheus/prometheus#9166 mentioned.

I'm on the fence regarding the API: I wonder if the operator should be in charge of producing the brypt hashes or if it should pass the secret data as-is to Prometheus (and let the user provide the hashes). In particular involving the operator with security concerns is always a risk for the long-term maintenance.

After more thinking, I realize that the simplest API might be for the user to provide a secret containing all the credentials where the username would be the key and the password the value:

apiVersion: v1
kind: Secret
metadata:
  name: web-users
stringData:
  foo: $2b$12$hNf2lSsxfm0.i4a.1kVpSOVyBCfIB51VRjgBUyv6kdnyTlgWj81Ay
  bar: $2b$12$hNf2lSsxfm0.i4a.1kVpSOVyBCfIB51VRjgBUyv6kdnyTlgWj81Ay
  prometheus: $2b$12$hNf2lSsxfm0.i4a.1kVpSOVyBCfIB51VRjgBUyv6kdnyTlgWj81Ay
---
apiVersion: v1
kind: Secret
metadata:
  name: example-webconfig
stringData:
  password: test
---
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: example
spec:
  web:
    basicAuthUsers:
      secretName: web-users
      podCredentials:
        username: prometheus
        password:
           key: password
           name: example-webconfig

The operator can fail the reconciliation if it detects invalid hashes in the secret (e.g. using bcrypt.Cost()).

Again sorry for the back and forth but I didn't put a lot of effort in this part before your PR so thanks for doing it!

After more thinking, I realize that the simplest API might be for the user to provide a secret containing all the credentials where the username would be the key and the password the value:

apiVersion: v1
kind: Secret
metadata:
  name: web-users
stringData:
  foo: $2b$12$hNf2lSsxfm0.i4a.1kVpSOVyBCfIB51VRjgBUyv6kdnyTlgWj81Ay
  bar: $2b$12$hNf2lSsxfm0.i4a.1kVpSOVyBCfIB51VRjgBUyv6kdnyTlgWj81Ay
  prometheus: $2b$12$hNf2lSsxfm0.i4a.1kVpSOVyBCfIB51VRjgBUyv6kdnyTlgWj81Ay
---
apiVersion: v1
kind: Secret
metadata:
  name: example-webconfig
stringData:
  password: test
---
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: example
spec:
  web:
    basicAuthUsers:
      secretName: web-users
      podCredentials:
        username: prometheus
        password:
           key: password
           name: example-webconfig

The operator can fail the reconciliation if it detects invalid hashes in the secret (e.g. using bcrypt.Cost()).

Again sorry for the back and forth but I didn't put a lot of effort in this part before your PR so thanks for doing it!

Ok, I'll try to change the API as you recommended, But the way, when i try to set a correct config of thanos sidecar to make sidecar can access to prometheus server normally, i find out that the sidecar currently set --prometheus.http-client config as a plain text, is that ok? coz if we want to use a config file for --promethes.http-client-config https://thanos.io/tip/components/sidecar.md/#configuration, this a lot of work need to do.

when i try to set a correct config of thanos sidecar to make sidecar can access to prometheus server normally, i find out that the sidecar currently set --prometheus.http-client config as a plain text, is that ok? coz if we want to use a config file for --promethes.http-client-config https://thanos.io/tip/components/sidecar.md/#configuration, this a lot of work need to do.

For sure, we don't want the HTTP client config to be in clear-text if it contains sensitive information like password so yes, we'll need to change this and probably use a dedicated secret mounted in the sidecar. And we need the same for the config reloader sidecar. I'd say that it should be a preliminary PR to this one to facilitate the review process.

For context, both thanos and config-reloader sidecars had to disable TLS verification because they only connect through localhost and we didn't want to bother with configuring the right CA.

when i try to set a correct config of thanos sidecar to make sidecar can access to prometheus server normally, i find out that the sidecar currently set --prometheus.http-client config as a plain text, is that ok? coz if we want to use a config file for --promethes.http-client-config https://thanos.io/tip/components/sidecar.md/#configuration, this a lot of work need to do.

For sure, we don't want the HTTP client config to be in clear-text if it contains sensitive information like password so yes, we'll need to change this and probably use a dedicated secret mounted in the sidecar. And we need the same for the config reloader sidecar. I'd say that it should be a preliminary PR to this one to facilitate the review process.

For context, both thanos and config-reloader sidecars had to disable TLS verification because they only connect through localhost and we didn't want to bother with configuring the right CA.

I find that the config-reloader's ReloaderURL is defined here https://github.com/prometheus-operator/prometheus-operator/blob/main/pkg/prometheus/common.go#L456 and did not support config basic auth headers, should i change the thanos config reloader to support this param?

For the config reloader, the idea should be to add CLI arguments to pass the basic-auth user/password in https://github.com/prometheus-operator/prometheus-operator/blob/main/cmd/prometheus-config-reloader/main.go and use something like https://pkg.go.dev/k8s.io/[email protected]/transport#NewBasicAuthRoundTripper to inject the header in the request.

@simonpasquier hi, i have use object reference to secrets as you mentioned, is there any other things i can to to move on？