For "deny" probes, we have the problem of heavy logging [1] due to expected failures. I think we can solve this in several ways but i was wondering if cloudprober would benefit with some notion of throttled logging at a probe level. Once we have throttled logging parameters, we can add a utility function say

func shouldLogForProbe() bool { ...}  // time-based control or one-every-N logging.

then probes can change their probe path to be

func doOneProbe(...) {

   shouldLog := shouldLogForProbe()

   ...
   throttledLog(p.l, shouldLog, msg)
   // OR we could hide the logic for throttling completely by passing a timestamp or probe ID.
   throttledLog(p.l, probeTSorID, msg)
   ...
}

With this we could capture all logs for a single probe and probe writers can also increase the logging per probe with the confidence that there wont be too much log spam.

Alternatively, as a quick fix, we could also add "throttle_log_interval" param to http probe that could be used to limit logging to every 'interval'.

Or a tertiary fix of removing the log line completely.

[1] https://github.com/cloudprober/cloudprober/blob/main/probes/http/http.go#L334