receive: Missing Exemplar Labels when using Receive Router with Capnproto #8224
Description
Thanos, Prometheus and Golang version used:
v0.38.0
Object Storage Provider:
none
What happened:
When using the receiver with capnproto, any exemplars sent to the router lose their labels. Strings are null \u0000 when querying for the exemplars.
What you expected to happen:
Exemplars should keep their labels.
How to reproduce it (as minimally and precisely as possible):
- Deploy Thanos with Receiver in Router setup and
receive.replication-protocol=capnproto - Push a TimeSeries with an Exemplar to the Receive Router over remote-write
- Send query for exemplar
Details
Output with capnproto
curl http://localhost:9090/api/v1/query_exemplars?query=foo_bar | jq
{
"status": "success",
"data": [
{
"seriesLabels": {
"__name__": "foo_bar",
"receive": "true",
"replica": "thanos-receive-ingestor-default-0",
"tenant_id": "default-tenant"
},
"exemplars": [
{
"labels": {
"\u0000\u0000\u0000\u0000\u0000\u0000\u0000": "\u0000\u0000\u0000\u0000\u0000\u0000",
"\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000": "\u0000\u0000\u0000\u0000\u0000\u0000"
},
"timestamp": 1745528638.4,
"value": "1337"
}
]
}
]
}
Output with protobuf
curl http://localhost:9090/api/v1/query_exemplars?query=foo_bar | jq
{
"status": "success",
"data": [
{
"seriesLabels": {
"__name__": "foo_bar",
"receive": "true",
"replica": "thanos-receive-ingestor-default-0",
"tenant_id": "default-tenant"
},
"exemplars": [
{
"labels": {
"span_id": "def456",
"trace_id": "abc123"
},
"timestamp": 1745528606.088,
"value": "1337"
}
]
}
]
}
Repo with full steps to reproduce: https://github.com/lukasboettcher/thanos-exemplars-bug