An alternative I have used to troubleshoot potential intra-Envoy latency issues is the %COMMON_DURATION(START:END:PRECISION)% command operator in Envoy access logging. This helps narrow down if latency is introduced by a downstream/upstream peer or by Envoy itself. However, this does not provide enough granular information as to what might be causing latency if it is caused by Envoy.

To have closer parity to the Apigee Trace tool, maybe it might be worth considering adding this per-filter data to the Tap filter? It would be ideal if for each filter, we can see both:

1. Overall processing duration (useful for troubleshooting transaction latency)
2. Request/response data (headers, payload, and trailers) before/after the filter processes the transaction (useful for troubleshooting malformed requests/responses impacting business logic)

@wbpcode

Nothing is free, more detailed tracing means more overhead in the key path and more complex code base, I think we cannot accept further decrease of the envoy performance.

And in most cases, most users needn't this because the filter of envoy basically is super fast. And if there is performance problem, perf would be more useful than the filter level tracing.

Of course, I am not mean the feature is senseless, I only mean it may doesn't deserve the overhead and investment. I think the best althernative would be the ExecutionContext.

The ExecutionContext provides very fine grained probes (include filter level) that could be extended to record the overhead of per filter. By default, the ExecutionContext is compiled out but it's easy to enable it at the third-party building if it's required.

Nothing is free, more detailed tracing means more overhead in the key path and more complex code base, I think we cannot accept further decrease of the envoy performance.

And in most cases, most users needn't this because the filter of envoy basically is super fast. And if there is performance problem, perf would be more useful than the filter level tracing.

Of course, I am not mean the feature is senseless, I only mean it may doesn't deserve the overhead and investment. I think the best althernative would be the ExecutionContext.

The ExecutionContext provides very fine grained probes (include filter level) that could be extended to record the overhead of per filter. By default, the ExecutionContext is compiled out but it's easy to enable it at the third-party building if it's required.

Thank you for the reply, I appreciate the feedback and information on this feature suggestion.

If more detailed tracing will add more overhead/latency to the key data plane path, then I agree that it probably is not worth adding (unless it was in some sort of "debug" binary separate from the standard Envoy binary).

I am not familiar with the perf and ExecutionContext features/components, but let me do some research to see if this would satisfy what I am looking for.

For ExecutionContext, I found this GitHub issue which explains it in detail, but I am not sure what you mean by perf. Did you mean to refer to envoy-perf?

And in most cases, most users needn't this because the filter of envoy basically is super fast. And if there is performance problem, perf would be more useful than the filter level tracing.

For the standard Envoy filters this is totally reasonable, but my concern is for custom filters built in-house by Envoy operators or by Envoy-based vendors (Solo.io, Tetrate, etc). It is probably still more correct to use envoy-perf and ExecutionContext to debug this in an isolated test/debug environment. However, for hard-to-reproduce production issues, not having the detailed trace available for impacted transactions live in production is a minus.

re to perf

I mean the linux perf tool 🤣