feat: add support to collect metrics using the OTEL library #927

zorocloud · 2024-02-20T11:43:23Z

Adding support to the interceptor proxy to allow for the collection of metrics via a prometheus compatible endpoint or via a OTEL HTTP collector. Resolves #910

Checklist

Commits are signed with Developer Certificate of Origin (DCO)
Changelog has been updated and is aligned with our changelog requirements
Any necessary documentation is added, such as:

JorTurFer

Looks really nice! Could you add some documentation about how to enable these metrics? I think that we could create a new section like Operate and include this information about how to enable prometheus/otel and which metrics are available.

do you think that adding an e2e test could be useful? In KEDA core we cover the metrics too using e2e tests, so we could just add an e2e test like them

zorocloud · 2024-02-27T17:27:46Z

Looks really nice! Could you add some documentation about how to enable these metrics? I think that we could create a new section like Operate and include this information about how to enable prometheus/otel and which metrics are available.

Thanks! Yeah, sure. Will draft some docs up and get the PR updated.

do you think that adding an e2e test could be useful? In KEDA core we cover the metrics too using e2e tests, so we could just add an e2e test like them

That is a good shout. Will get a test for both cases added in and update the PR.

zorocloud · 2024-03-12T15:19:34Z

I have added a brief overview of how to configure each exporter in a Operate.md doc as per your suggestion @JorTurFer and added in some e2e tests (using the core Keda set as a reference). Let me know your thoughts :)

config/interceptor/deployment.yaml

config/interceptor/scaledobject.yaml

JorTurFer · 2024-03-15T21:47:42Z

docs/operate.md

+# Configuring metrics for the KEDA HTTP Add-on interceptor proxy
+
+### Exportable metrics:
+* **Pending request count** - the number of pending requests for a given host.
+* **Total request count** - the total number of requests for a given host with method, path and response code attributes.
+
+There are currently 2 supported methods for exposing metrics from the interceptor proxy service - via a Prometheus compatible metrics endpoint or by pushing metrics to a OTEL HTTP collector.
+
+### Configuring the Prometheus compatible metrics endpoint
+When configured the interceptor proxy can expose metrics on a Prometheus compatible endpoint. This endpoint can be enabled by setting the `KEDA_HTTP_OTEL_PROM_EXPORTER_ENABLED` environment variable to `true` on the interceptor deployment (`true` by default) and by setting `KEDA_HTTP_OTEL_PROM_EXPORTER_PORT` to an unused port for the endpoint to be made avaialble on (`2223` by default).
+
+### Configuring the OTEL HTTP exporter
+When configured the interceptor proxy can export metrics to a OTEL HTTP collector. The OTEL exporter can be enabled by setting the `KEDA_HTTP_OTEL_HTTP_EXPORTER_ENABLED` environment variable to `true` on the interceptor deployment (`false` by default). When enabled the `KEDA_HTTP_OTEL_HTTP_COLLECTOR_ENDPOINT` environment variable must also be configured so the exporter knows what collector to send the metrics to (e.g. opentelemetry-collector.open-telemetry-system:4318). If the collector is exposed on a unsecured endpoint then you can set the `KEDA_HTTP_OTEL_HTTP_COLLECTOR_INSECURE` environment variable to `true` (`false` by default) which will disable client security on the exporter. If you need to provide any headers such as authentication details in order to utilise your OTEL collector you can add them into the `KEDA_HTTP_OTEL_HTTP_HEADERS` environment variable. The frequency at which the metrics are exported can be configured by setting `KEDA_HTTP_OTEL_METRIC_EXPORT_INTERVAL` to the number of seconds you require between each export interval (`30` by default).


PTAL @tomkerkhove

JorTurFer · 2024-03-15T21:49:07Z

tests/utils/setup_test.go

@@ -122,3 +177,35 @@ func TestDeployKEDAHttpAddOn(t *testing.T) {
 	t.Log(string(out))
 	t.Log("KEDA Http Add-on deployed successfully using 'make deploy' command")
 }
+
+func TestSetupOpentelemetryComponents(t *testing.T) {


Could you add the removal of the collection too please? (during the teardown)

JorTurFer · 2024-03-24T20:01:55Z

tests/checks/interceptor_prometheus_metrics/interceptor_prometheus_metrics_test.go

+	assert.True(t, ok, "interceptor_request_count_total is available")
+
+	requestCount := getMetricsValue(val)
+	assert.Equal(t, float64(1), requestCount)


I'd assert this using greater or equal because the Prometheus metrics are persisted over the time, and 2nd and 3rd tries can fail because the metric returns 2,3, etc

Yeah that is a good shout - when I had tested I was just tearing it down after every run so only ever had a count of 1 but that won't always be true

JorTurFer · 2024-03-24T20:05:36Z

tests/checks/interceptor_otel_metrics/interceptor_otel_metrics_test.go

+      - "exec tail -f /dev/null"`
+)
+
+func TestMetricGeneration(t *testing.T) {


This e2e test has failed due to compiling issues

Oops, left a small typo in there. Removed.

JorTurFer · 2024-03-24T20:06:08Z

tests/checks/interceptor_otel_metrics/interceptor_otel_metrics_test.go

+	assert.True(t, ok, "interceptor_request_count_total is available")
+
+	requestCount := getMetricsValue(val)
+	assert.Equal(t, float64(1), requestCount)


I'd assert this using greater or equal because the Prometheus metrics are persisted over the time, and 2nd and 3rd tries can fail because the metric returns 2,3, etc

zorocloud · 2024-03-27T08:17:11Z

Will take a look why the e2e tests failed when I get a sec

JorTurFer

LGTM!

JorTurFer · 2024-04-04T20:07:53Z

There are some static code errors, could you take a look please? https://github.com/kedacore/http-add-on/actions/runs/8453796744/job/23280002430?pr=927

tomkerkhove

Docs LGTM, nice work!

docs/operate.md

zorocloud · 2024-04-16T17:08:09Z

Thanks for the feedback. Remaining suggestions should be addressed and the static checks fixed. Just noticed I need to fix some conflicts too. Will sort that when I next get a sec 👍

zorocloud · 2024-04-17T16:02:56Z

I have just rebased onto main and the unit/e2e tests look like they are passing fine so we should be good now 👍

Edit: @JorTurFer it looks like I have a Snyk failure but can't view them via the details link. Is there another way to view the results?

JorTurFer · 2024-04-21T11:19:13Z

We can ignore Synk, it has transient failures from time to time xD I have marked it as okey manually :)

JorTurFer · 2024-04-21T11:20:37Z

There is other PR ongoing and once that's merged, I'll update your PR and merge it. Thanks for your AWESOME contribution ❤️

JorTurFer · 2024-04-21T14:41:57Z

tests/utils/setup_test.go

+config:
+  exporters:


Suggested change

config:

exporters:

image:

repository: "otel/opentelemetry-collector-contrib"

config:

exporters:

There is a potential breaking change that requires the change above, please commit it 🙏
https://github.com/open-telemetry/opentelemetry-helm-charts/blob/main/charts/opentelemetry-collector/UPGRADING.md#0880-to-0890

Done @JorTurFer :)

Also, I've pushed a change to move the pending request count from a Gauge to a UpDownCounter (noticed some memory issues with the Gauge). Left it in a separate commit rather than squashing locally so it's easier for you to review.

JorTurFer · 2024-04-23T19:49:28Z

There are merge conflicts to be solved, could you solve them? (or enabling me to update your PR xD)

…ible endpoint or by sending metrics to an OTEL http endpoint Signed-off-by: Joe Wogan <[email protected]>

specify otel image for e2e tests Signed-off-by: Joe Wogan <[email protected]>

zorocloud · 2024-04-24T07:35:35Z

Should be sorted now @JorTurFer :)

Also, allowed maintainers to edit the PR, not sure why I didn't select this previously.

JorTurFer

LGTM! ❤️

JorTurFer · 2024-05-18T10:01:15Z

I didn't noticed during the review that we aren't respected the OTEL spec ENV vars: https://opentelemetry.io/docs/specs/otel/protocol/exporter/

I've created an issue to track it, as I guess that you are using the metrics, I share it with you because we will have to change them for future versions: #1031

zorocloud · 2024-05-18T11:28:08Z

Ah sugar, I completely missed that too 🤦. I'm happy to get a PR raised toward the end of next week to fix this if someone isn't already working on it.

JorTurFer · 2024-05-18T12:38:03Z

I'm happy to get a PR raised toward the end of next week to fix this if someone isn't already working on it.

I'd be nice, I just pinged you to make you aware about a possible breaking change, but if you are willing to update the code, it's awesome :)

zorocloud requested a review from a team as a code owner February 20, 2024 11:43

zorocloud mentioned this pull request Feb 20, 2024

Interceptor Proxy - OTEL Instrumentation #910

Closed

JorTurFer reviewed Feb 23, 2024

View reviewed changes

zorocloud force-pushed the interceptor-metrics branch from cd788ad to 2d3f779 Compare March 12, 2024 15:13

JorTurFer reviewed Mar 15, 2024

View reviewed changes

zorocloud force-pushed the interceptor-metrics branch 2 times, most recently from 4a10aa4 to a8e5a05 Compare March 22, 2024 13:00

JorTurFer reviewed Mar 24, 2024

View reviewed changes

zorocloud force-pushed the interceptor-metrics branch from a8e5a05 to 60720b2 Compare March 26, 2024 15:50

zorocloud force-pushed the interceptor-metrics branch 2 times, most recently from 71f7a82 to 7dd67c3 Compare March 27, 2024 14:51

JorTurFer approved these changes Apr 4, 2024

View reviewed changes

tomkerkhove reviewed Apr 5, 2024

View reviewed changes

docs/operate.md Outdated Show resolved Hide resolved

docs/operate.md Outdated Show resolved Hide resolved

zorocloud force-pushed the interceptor-metrics branch from 7dd67c3 to 65135b0 Compare April 16, 2024 17:02

zorocloud force-pushed the interceptor-metrics branch from 65135b0 to adc1c8a Compare April 17, 2024 16:01

JorTurFer enabled auto-merge (squash) April 21, 2024 11:18

JorTurFer reviewed Apr 21, 2024

View reviewed changes

auto-merge was automatically disabled April 23, 2024 16:28
Head branch was pushed to by a user without write access

tomkerkhove assigned JorTurFer Apr 24, 2024

zorocloud added 2 commits April 24, 2024 08:15

feat: Add support to collect metrics using either a prometheus compat…

f5d5f03

…ible endpoint or by sending metrics to an OTEL http endpoint Signed-off-by: Joe Wogan <[email protected]>

fix: use UpDounCounter rather than gauge for pending request count and

06fb14a

specify otel image for e2e tests Signed-off-by: Joe Wogan <[email protected]>

zorocloud force-pushed the interceptor-metrics branch from ceb81d5 to 06fb14a Compare April 24, 2024 07:34

JorTurFer approved these changes Apr 24, 2024

View reviewed changes

JorTurFer enabled auto-merge (squash) April 24, 2024 07:39

JorTurFer merged commit 0237611 into kedacore:main Apr 24, 2024
19 checks passed

zorocloud deleted the interceptor-metrics branch April 24, 2024 08:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add support to collect metrics using the OTEL library #927

feat: add support to collect metrics using the OTEL library #927

zorocloud commented Feb 20, 2024 •

edited by JorTurFer

Loading

JorTurFer left a comment

zorocloud commented Feb 27, 2024 •

edited

Loading

zorocloud commented Mar 12, 2024

JorTurFer Mar 15, 2024

JorTurFer Mar 15, 2024

JorTurFer Mar 24, 2024

zorocloud Mar 26, 2024

JorTurFer Mar 24, 2024

zorocloud Mar 26, 2024

JorTurFer Mar 24, 2024

zorocloud commented Mar 27, 2024

JorTurFer left a comment

JorTurFer commented Apr 4, 2024

tomkerkhove left a comment

zorocloud commented Apr 16, 2024

zorocloud commented Apr 17, 2024 •

edited

Loading

JorTurFer commented Apr 21, 2024

JorTurFer commented Apr 21, 2024

JorTurFer Apr 21, 2024

JorTurFer Apr 21, 2024

zorocloud Apr 23, 2024

JorTurFer commented Apr 23, 2024

zorocloud commented Apr 24, 2024

JorTurFer left a comment

JorTurFer commented May 18, 2024

zorocloud commented May 18, 2024

JorTurFer commented May 18, 2024

feat: add support to collect metrics using the OTEL library #927

feat: add support to collect metrics using the OTEL library #927

Conversation

zorocloud commented Feb 20, 2024 • edited by JorTurFer Loading

Checklist

JorTurFer left a comment

Choose a reason for hiding this comment

zorocloud commented Feb 27, 2024 • edited Loading

zorocloud commented Mar 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zorocloud commented Mar 27, 2024

JorTurFer left a comment

Choose a reason for hiding this comment

JorTurFer commented Apr 4, 2024

tomkerkhove left a comment

Choose a reason for hiding this comment

zorocloud commented Apr 16, 2024

zorocloud commented Apr 17, 2024 • edited Loading

JorTurFer commented Apr 21, 2024

JorTurFer commented Apr 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JorTurFer commented Apr 23, 2024

zorocloud commented Apr 24, 2024

JorTurFer left a comment

Choose a reason for hiding this comment

JorTurFer commented May 18, 2024

zorocloud commented May 18, 2024

JorTurFer commented May 18, 2024

zorocloud commented Feb 20, 2024 •

edited by JorTurFer

Loading

zorocloud commented Feb 27, 2024 •

edited

Loading

zorocloud commented Apr 17, 2024 •

edited

Loading