For each tier, we are interested in the disk_used and samples
For this agents restart frequency (at least daily) we get
\n- \n
- Tier 0: 0.37 bytes \n
- Tier 1: 4.06 bytes \n
- Tier 2: 27.66 bytes \n
Note that the numbers for Tier 0 and 1 are \"better\" that what is listed in the documentation (Tier 2 is worse). This is the reason also for mentioning Usually On Disk
For an agent with more tiers, a similar approach can be used for calculation
\nNote: These disk_used numbers do not include agent metadata not related to data collection itself and is stored per metric (metric name, chart, context etc) but that doesn't impact higher tiers anyway
I hope this helps
","upvoteCount":2,"url":"https://github.com/netdata/netdata/discussions/18616#discussioncomment-10760972"}}}
-
|
I use 4 tiers and would like to calculate the storage requirements for each one, as well as create a calculator based on the number of metrics collected. According to the documentation, tier 0 requires around 0.6 bytes of storage per sample, tier 1 requires 6 bytes, while tier 2 requires a surprisingly large 18 bytes. So I have two questions:
Thank you! |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments
-
|
Hi @aldem It has to do with update frequency of the higher tiers. Higher tiers are updated every X iterations of data points collected of the previous tiers. So in the default configuration tier 0 collects metrics every second, tier 1 will store 1 point every 60 metrics of tier 0 etc. As collected metrics fill data pages for each tier, the pages are grouped into "blocks", compressed and stored to disk Agent behavior (for example frequent restarts) will result in data pages with just a few data points. This can affect compression, but it can also hurt the compression ratio (because even for 1 data point there is associated metadata that needs to be stored) As an example for a running agent, running localhost:19999/api/v3/node_instances will provide some numbers that can help a bit with the calculations: For each tier, we are interested in the For this agents restart frequency (at least daily) we get
Note that the numbers for Tier 0 and 1 are "better" that what is listed in the documentation (Tier 2 is worse). This is the reason also for mentioning For an agent with more tiers, a similar approach can be used for calculation Note: These I hope this helps |
Beta Was this translation helpful? Give feedback.
-
|
Thank you, @stelfrag - this really helps! I didn't know that it is possible to get exact usage, including number of samples/metrics, via API. Though now I believe that the best approach would be to "auto-tune" based on actual storage usage - let it run for a while, then use |
Beta Was this translation helpful? Give feedback.
Hi @aldem
It has to do with update frequency of the higher tiers. Higher tiers are updated every X iterations of data points collected of the previous tiers.
So in the default configuration tier 0 collects metrics every second, tier 1 will store 1 point every 60 metrics of tier 0 etc.
As collected metrics fill data pages for each tier, the pages are grouped into "blocks", compressed and stored to disk
Agent behavior (for example frequent restarts) will result in data pages with just a few data points. This can affect compression, but it can also hurt the compression ratio (because even for 1 data point there is associated metadata that needs to be stored)
As an example for a running agent…