Hi team, we recently experience the following errors which results in clickhouse crashing

(version 24.1.2.5 (official build))
2025.01.16 23:16:21.972597 [ 565 ] {} <Error> signoz_metrics.distributed_samples_v4.DistributedInsertQueue.default: Renamed `/var/lib/clickhouse/store/dee/dee080fb-ba29-4155-938e-d7fd59792a4c/shard2_replica1/15754034.bin` to `/var/lib/clickhouse/store/dee/dee080fb-ba29-4155-938e-d7fd59792a4c/shard2_replica1/broken/15754034.bin`
2025.01.16 23:16:21.972732 [ 565 ] {} <Error> signoz_metrics.distributed_samples_v4.DistributedInsertQueue.default: Code: 32. DB::Exception: Attempt to read after eof: While sending /var/lib/clickhouse/store/dee/dee080fb-ba29-4155-938e-d7fd59792a4c/shard2_replica1/15754034.bin. (ATTEMPT_TO_READ_AFTER_EOF), Stack trace (when copying this message, always include the lines below):

Any idea what’s the potential cause of the error?

Hi Team, I want to view all of the metrics received by the Signoz for checking all of the metrics received from a host or a service. How can i view that? I tried the query builder but it is not showing me metrics but i can see them in the dashboards. Kindly help!

Hello Team, I am having trouble in configuring signoz for my SaaS product offering running on AKS, and is built with nodejs. Here is what I have done to test out signoz. 1. I have configured Signoz on AKS as per the instructions in the documentation in "Signoz" namespace 2. I have configured ingress as per the Signoz documentation. 3. To do a test, I have auto instrumented my nodejs microservice (entry point at server.js) on a local environment (on My Ubuntu laptop). 4. I have configured the observability (traces, metrics, logs) with base URL as http://<<Static IP of Ingress>> and in my routes in ingress, i have /v1/traces etc. 5. When i run my application for instrumentation, the endpoints are reachable, but I get 404 always for all endpoints. I have dived deep in going over every service, pod details and their logs and unable to find the issue. I am new to the observability setup (even with Grafana + Prometheus + Elk) as a developer, as compared to end user. I request someone to help in configuring and setting up the signoz env on AKS and help me instrument the code. Please share sample end to end setup with code, or share articles, documentation links to achieve them. It will be great if a fully functional sample of ingress for signoz is shared, and instruction to custom install them on AKS, what are the parameters to look for, that will help Signoz to work seamlessly.

Hello everyone Do we have an estimate on when the SigNoz terraform provider would be available for self-hosted instances? At this time of writing you can only use this provider using an API token provided by signoz cloud.

We ran into an issue where we would need to use this new patched version of the OTEL collector but it has not yet been pulled in by SigNoz. This is blocking a certain fix we wish to do with a Tail Sampling Processor policy that relies on this change: https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/36673. Was wondering when will the team update past v0.116.0+ so that we can use the fix in our PROD pipeline? cc @Pranay

Hi team - thanks for a great product. I have a question regarding trace and logs. Is there a way I can get the trace with an api, as it is happening, and not have to wait for a minute or 2 ? I know the trace id

@Vishal Sharma @Prashant Shahi Any idea about below error. that we are getting in clickhouse pods and pods keep restarting. I understood it is related to the insufficient mem. {"date_time":"1736745070.040729","thread_name":"","thread_id":"580","level":"Error","query_id":"","logger_name":"MergeTreeBackgroundExecutor","message":"Exception while executing background task {7165cb86-e3b3-4832-9778-cf6a91b2273c:20250111 18913 18971 1} Code: 241. DB:Exception Memory limit (total) exceeded: would use 6.83 GiB (attempt to allocate chunk of 4224032 bytes), maximum: 6.79 GiB. OvercommitTracker decision: Memory overcommit isn't used. Waiting time or overcommit denominator are set to zero.: (while reading column scope_string): (while reading from part \/var\/lib\/clickhouse\/store\/716\/7165cb86-e3b3-4832-9778-cf6a91b2273c\/20250111_18931_18931_0\/ in table signoz_logs.logs_v2 (7165cb86-e3b3-4832-9778-cf6a91b2273c) located on disk default of type local, from mark 0 with max_rows_to_read = 5888): While executing MergeTreeSequentialSource. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):\n\n0. DB:Exception:Exception(DB:Exception:MessageMasked&&, int, bool) @ 0x000000000c800f1b in \/usr\/bin\/clickhouse\n1. DB:Exception:Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std:type identity&lt;String&gt;:type, std:type identity&lt;long&amp;&gt;:type, std:type identity&lt;String&gt;:type, std::type_identity<char const*>::type, std:type identity&lt;std:basic_string_view<char, std:char traits&lt;char&gt;&gt;&gt;:type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000000c816d0a in \/usr\/bin\/clickhouse\n2. MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000000c816948 in \/usr\/bin\/clickhouse\n3. MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000000c816389 in \/usr\/bin\/clickhouse\n4. MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000000c816389 in \/usr\/bin\/clickhouse\n5. MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000000c816389 in \/usr\/bin\/clickhouse\n6. Allocator<false, false>::alloc(unsigned long, unsigned long) @ 0x000000000c7d560d in \/usr\/bin\/clickhouse\n7. void DB::PODArrayBase<8ul, 4096ul, Allocator<false, false>, 63ul, 64ul>::resize<>(unsigned long) @ 0x0000000007220183 in \/usr\/bin\/clickhouse\n8. DB:SerializationArray:deserializeBinaryBulkWithMultipleStreams(COW<DB:IColumn&gt;immutable ptr&lt;DB:IColumn>&, unsigned long, DB:ISerialization:DeserializeBinaryBulkSettings&, std:shared ptr&lt;DBISerialization:DeserializeBinaryBulkState>&, std::unordered_map<String, COW<DB:IColumn&gt;immutable ptr&lt;DB:IColumn>, std::hash<String>, std::equal_to<String>, std:allocator&lt;std:pair<String const, COW<DB:IColumn&gt;immutable ptr&lt;DB:IColumn>>>>*) const @ 0x00000000108b9585 in \/usr\/bin\/clickhouse\n9. DB:MergeTreeReaderWide:readRows(unsigned long, unsigned long, bool, unsigned long, std:vector&lt;COW&lt;DBIColumn&gt;immutable ptr&lt;DB:IColumn>, std:allocator&lt;COW&lt;DBIColumn&gt;immutable ptr&lt;DB:IColumn>>>&) @ 0x000000001251ab34 in \/usr\/bin\/clickhouse\n10. DB:MergeTreeSequentialSource:generate() @ 0x000000001251ca4d in \/usr\/bin\/clickhouse\n11. DB:ISource:tryGenerate() @ 0x000000001297acf5 in \/usr\/bin\/clickhouse\n12. DB:ISource:work() @ 0x000000001297a743 in \/usr\/bin\/clickhouse\n13. DB:ExecutionThreadContext:executeTask() @ 0x000000001299371a in \/usr\/bin\/clickhouse\n14. DB:PipelineExecutor:executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001298a170 in \/usr\/bin\/clickhouse\n15. DB:PipelineExecutor:executeStep(std::atomic<bool>*) @ 0x0000000012989928 in \/usr\/bin\/clickhouse\n16. DB:PullingPipelineExecutor:pull(DB::Chunk&) @ 0x0000000012998017 in \/usr\/bin\/clickhouse\n17. DB:PullingPipelineExecutor:pull(DB::Block&) @ 0x00000000129981d3 in \/usr\/bin\/clickhouse\n18. DB:MergeTaskExecuteAndFinalizeHorizontalPart:executeImpl() @ 0x000000001233b6f2 in \/usr\/bin\/clickhouse\n19. DB:MergeTaskExecuteAndFinalizeHorizontalPart:execute() @ 0x000000001233b64b in \/usr\/bin\/clickhouse\n20. DB:MergeTask:execute() @ 0x0000000012340d99 in \/usr\/bin\/clickhouse\n21. DB:MergePlainMergeTreeTask:executeStep() @ 0x0000000012723517 in \/usr\/bin\/clickhouse\n22. DB:MergeTreeBackgroundExecutor&lt;DBDynamicRuntimeQueue&gt;:threadFunction() @ 0x00000000123532c4 in \/usr\/bin\/clickhouse\n23. ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000c8eb0c1 in \/usr\/bin\/clickhouse\n24. void std: function:__policy_invoker<void ()>: call impl&lt;std function default alloc func&lt;ThreadFromGlobalPoolImpl&lt;false&gt;:ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std: function:__policy_storage const*) @ 0x000000000c8ee8fa in \/usr\/bin\/clickhouse\n25. void* std: thread proxy[abiv15000]<std:tuple&lt;stdunique ptr&lt;std:__thread_struct, std:default delete&lt;std:__thread_struct>>, void ThreadPoolImpl<std:thread&gt;:scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000c8ed6fe in \/usr\/bin\/clickhouse\n26. ? @ 0x00007f2bbbadb609\n27. ? @ 0x00007f2bbba00353\n (version 24.1.2.5 (official build))","source_file":"src\/Common\/Exception.cpp; void DB::tryLogCurrentExceptionImpl(Poco::Logger *, const std::string &)","source_line":"222"}

{"date_time":"1736745070.040745","thread_name":"","thread_id":"592","level":"Error","query_id":"","logger_name":"MergeTreeBackgroundExecutor","message":"Exception while executing background task {7165cb86-e3b3-4832-9778-cf6a91b2273c::20250111_18664_18732_1}: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 6.83 GiB (attempt to allocate chunk of 4299599 bytes), maximum: 6.79 GiB. OvercommitTracker decision: Memory overcommit isn't used. Waiting time or overcommit denominator are set to zero.: (while reading column attributes_string): (while reading from part \/var\/lib\/clickhouse\/store\/716\/7165cb86-e3b3-4832-9778-cf6a91b2273c\/20250111_18669_18669_0\/ in table signoz_logs.logs_v2 (7165cb86-e3b3-4832-9778-cf6a91b2273c) located on disk default of type local, from mark 0 with max_rows_to_read = 6526): While executing MergeTreeSequentialSource. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):\n\n0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c800f1b in \/usr\/bin\/clickhouse\n1. DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000000c816d0a in \/usr\/bin\/clickhouse\n2. MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000000c816948 in \/usr\/bin\/clickhouse\n3. MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000000c816389 in \/usr\/bin\/clickhouse\n4. MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000000c816389 in \/usr\/bin\/clickhouse\n5. MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000000c816389 in \/usr\/bin\/clickhouse\n6. Allocator<false, false>::realloc(void*, unsigned long, unsigned long, unsigned long) @ 0x000000000c7d5d87 in \/usr\/bin\/clickhouse\n7. void DB::PODArrayBase<1ul, 4096ul, Allocator<false, false>, 63ul, 64ul>::resize_exact<>(unsigned long) @ 0x0000000007226ba6 in \/usr\/bin\/clickhouse\n8. void DB::deserializeBinarySSE2<1>(DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul>&, DB::PODArray<unsigned long, 4096ul, Allocator<false, false>, 63ul, 64ul>&, DB::ReadBuffer&, unsigned long) @ 0x00000000108f8836 in \/usr\/bin\/clickhouse\n9. DB::ISerialization::deserializeBinaryBulkWithMultipleStreams(COW<DB::IColumn>::immutable_ptr<DB::IColumn>&, unsigned long, DB::ISerialization::DeserializeBinaryBulkSettings&, std::shared_ptr<DB::ISerialization::DeserializeBinaryBulkState>&, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>*) const @ 0x00000000108b01d9 in \/usr\/bin\/clickhouse\n10. DB::SerializationTuple::deserializeBinaryBulkWithMultipleStreams(COW<DB::IColumn>::immutable_ptr<DB::IColumn>&, unsigned long, DB::ISerialization::DeserializeBinaryBulkSettings&, std::shared_ptr<DB::ISerialization::DeserializeBinaryBulkState>&, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>*) const @ 0x0000000010902f5a in \/usr\/bin\/clickhouse\n11. DB::SerializationArray::deserializeBinaryBulkWithMultipleStreams(COW<DB::IColumn>::immutable_ptr<DB::IColumn>&, unsigned long, DB::ISerialization::DeserializeBinaryBulkSettings&, std::shared_ptr<DB::ISerialization::DeserializeBinaryBulkState>&, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>*) const @ 0x00000000108b99c8 in \/usr\/bin\/clickhouse\n12. DB::MergeTreeReaderWide::readRows(unsigned long, unsigned long, bool, unsigned long, std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>&) @ 0x000000001251ab34 in \/usr\/bin\/clickhouse\n13. DB::MergeTreeSequentialSource::generate() @ 0x000000001251ca4d in \/usr\/bin\/clickhouse\n14. DB::ISource::tryGenerate() @ 0x000000001297acf5 in \/usr\/bin\/clickhouse\n15. DB::ISource::work() @ 0x000000001297a743 in \/usr\/bin\/clickhouse\n16. DB::ExecutionThreadContext::executeTask() @ 0x000000001299371a in \/usr\/bin\/clickhouse\n17. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001298a170 in \/usr\/bin\/clickhouse\n18. DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x0000000012989928 in \/usr\/bin\/clickhouse\n19. DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x0000000012998017 in \/usr\/bin\/clickhouse\n20. DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x00000000129981d3 in \/usr\/bin\/clickhouse\n21. DB::MergeTask::ExecuteAndFinalizeHorizontalPart::executeImpl() @ 0x000000001233b6f2 in \/usr\/bin\/clickhouse\n22. DB::MergeTask::ExecuteAndFinalizeHorizontalPart::execute() @ 0x000000001233b64b in \/usr\/bin\/clickhouse\n23. DB::MergeTask::execute() @ 0x0000000012340d99 in \/usr\/bin\/clickhouse\n24. DB::MergePlainMergeTreeTask::executeStep() @ 0x0000000012723517 in \/usr\/bin\/clickhouse\n25. DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000123532c4 in \/usr\/bin\/clickhouse\n26.

Hi everyone ! 👋 Just bumped into this specific statement (see below) in the FAQ and was wondering if we have any news on this kind of feature ? AFAIK, I didn't see it on the public roadmap.

Though Honeycomb does have some features like Bubble up and correlation which we don't have currently, but we have some advanced correlation based features in our roadmap

Also is it possible to visualise trace aggregates as heatmaps ? If no, is it planned on the roadmap ? Have a nice day you all 🙂

Hey Everyone! 👋 Our Community Call is just around the corner and we are looking to share success stories of our users. If you're using SigNoz for monitoring and have a story to share, we’d love to hear from you. Simply reply to this thread or DM me. As a token of appreciation, you'll receive some cool SigNoz Swags 😉 Link to Register for Community Call 👇 https://lu.ma/1z2jr0yy

I want to know why accessing v2 and v3 api results in a 404 error.

Hi, my organization currently uses a self-hosted instance of SigNoz deployed on EC2. We are planning to migrate to SigNoz Cloud, but before proceeding, I need to estimate the potential costs. To do this, I need to calculate the size of log, trace, and metric sample ingestions. What would be a good strategy to get a rough estimation of these metrics?

I have a question How much storage is allocated to me in the Docker standalone model? How much data can I store in ClickHouse?

Hello👋*,* I'm trying to send traces from deployed apps on Minikube to SigNoz, which is self-hosted on Docker. When I test the apps locally, the metrics, traces, and logs are being sent correctly, but the data doesn't seem to be transmitted when the apps are deployed on Minikube😭. The interesting thing is that Minikube is also running locally. Could someone help me identify if I'm missing any configuration or setup to make sure the traces are being sent correctly when deployed on Minikube?🤔 Thanks in advance🙏!

👋 Hi everyone! We are using the self-hosted version of SigNoz and are currently trying to monitor our Laravel applications. We have followed the PHP setup documentation provided, but we are unable to find specific instructions or guidance on setting it up for Laravel applications. Could you kindly assist us with the steps or configuration required to integrate SigNoz with Laravel? Any documentation, examples, or tips you could provide would be greatly appreciated.

No errors in otel collector pod logs

Hello everyone, i just deployed signoz using the default values on EKS, every of the pods is up but logs arent showing up in the logs dashboard , what could be the issue? I have tried the troubleshooting guide and the service otel-collector is receiving and active? localhost:4317

Hey everyone! 👋 We've just tagged several "good first issues" in our Helm charts repository (https://github.com/SigNoz/charts) and we'd love your help tackling them. This is a perfect opportunity to: • Get hands-on experience with Helm chart development • Learn Kubernetes best practices • Build your DevOps portfolio • Collaborate with our friendly community Many of these issues focus on chart improvements, documentation updates, and value file enhancements - perfect for both beginners and those looking to deepen their Kubernetes/Helm expertise. Some areas you'll get to work with: • Helm template functions and best practices • Kubernetes resource definitions • Chart dependency management • Values file structuring Whether you're new to Helm or a seasoned DevOps engineer, contributing is a great way to level up your skills while helping the community. To get started: 1. Check out our issues labeled

good-first-issue

2. Comment on any issue you'd like to work on 3. Fork the repo and submit your PR We strive to provide detailed feedback and support throughout the PR process. Link to good first issues: https://github.com/SigNoz/charts/issues?q=is%3Aissue%20state%3Aopen%20label%3A%22good%20first%20issue%22

how to I add my target host herer

Hello, I am using the Signoz collector in a Docker standalone setup. On the frontend, I can see the trace, metric, and log data, but there are no records in the ClickHouse database. What could be the reason for this, and how can I resolve it? I need to query ClickHouse directly to build dashboards.

This message was deleted.

Hello guys, today our self hosted SigNoz stopped working with no any understandable error, just spinning wheel, any ideas what could be wrong?

Hi I am using sftpgo and want to send the metrics to signoz server, but I am not able to see any dashboard, do you have prometheus dashboard sample for configuring

<!channel> Hi Community Members! 👋 As part of our ongoing efforts to improve collaboration and streamline our development process, we have decided to deprecate the

develop

branch in our GitHub repository. Moving forward, the

main

branch will be the single source of truth for all development work. What does this mean for you? • All feature branches should now branch off from and merge back into

main

. • The

develop

branch will no longer be maintained or updated, and will be removed. • CI/CD pipelines and other integrations will be adjusted to work exclusively with the

main

branch. • Open PRs which are raised against

develop

will be closed automatically. You will need to re-create the PRs with

main

as the base branch.

Hi I am trying to put signoz behind https, for that I using traefik Is there a document for that ?

Hello! need a quick advice! I'm trying to improve my signoz environment for incoming high traffic. I have a single signoz server and im thinking about making multiple clickhouse in the same docker with distributed clickhouse. OR i could make a loadbalancer using Nginx and run multiple signoz otel collector?

Service Map Bug - Service Map does not show all hosts, e.g. one of our services make external calls to ELK, ELK does not send requests to SigNoz yet but we still make requests to these hosts and so should see them on the map

Feature Request - space usage, we need to see how spaces is used 1 - by service 2 - by data type (metrics, logs, traces)

@Pranay ^ 🙏

Can anyone confirm if Signoz supports any kind of deduplication of log messages as a pre-processing pipeline?

Hi all, could someone please tell me how to configure the threshold to work on boolean? I try to create a table panel, the value is boolean, but when I try to use threshold to change the color, it give me this warning: Threshold unit (bool) is not valid in comparison with the column unit (bool) is string/bool not supported for threshold? Thanks