-
Notifications
You must be signed in to change notification settings - Fork 159
[APP-450] feat: guard time series queries with minimum and maximum granularity #8493
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
…/determine-max-interval-precision
30 days would usually be more taxing as it has to process a lot more raw data. The number of output buckets is less taxing, just still something worth paying attention to.
I don't have a hard cap in mind, somewhere between 500-1000 seems good to me. But if there's a use case for it, we can probably go quite a bit higher. My main concern actually isn't to enforce a hard cap, it's more to avoid accidental/unintended situations where it gets into a state that queries for a large number of buckets. It's fine to query for many buckets if that appears to be a clear user intent. (Kind of similar to "all time", which we didn't want to be too easy to select, but if someone goes to the trouble of selecting a custom time range spanning many years, we don't disallow it.) |
This PR adds a guard around the min/max precision allowed for time series data. It also limits the options provided in the time series aggregation selector on Explore to only those we deem "valid".
The intention of this PR is to adhere to the following heuristic:
Determine the allowed grains for a given interval based on the duration of the interval and the
smallest_time_grainproperty of the metrics viewAllowed grains should result in no more than 731 (see below) buckets and no fewer than 1 bucket (though this bucket may be partial (as in the case of
24h as of latest/hbyday). However, if all grains are above the max, allowyeargranularity. If all grains produce fewer than 1 complete bucket, allow the smallest_time_grain.If the user has explicitly asked for a certain granularity (i.e., there is a
grainparameter in the url) and it is within the allowed grains for the range, render it. If the requested grain is not in the allowable grains, fall back to the range grain (if allowed) or the smallest of the allowed grains.If there is no explicit granularity requested, we should follow the grain derived from the time string (e.g
dayfor P7D andhourfor 7d as of latest/h+1h`) if allowed. If not, fall back to the smallest of the allowed grains.Examples
Time Range:
12M as of latest/d+1dURL Grain:
daySmallest Time Grain:
dayRendered in days.
Time Range:
36M as of latest/d+1dURL Grain:
daySmallest Time Grain:
dayRendered in weeks
Time Range:
12h as of latest/h+1hURL Grain:
hourSmallest Time Grain:
dayIn this case, while
houris the range granularity anddayis normally not allowed, the time series will be aggregated by day despite displaying only a single partial bucket. Because of limitations on Explore, the URL grain will be updated today.This logic can be verified on Canvas via the derived
grainStoreintime-state.tsOn Explore, a number of changes were necessary.
selectedTimeRangefrom theShallowMergeOneLevelDeepKeysso that URL grain is reflected directly and not merged from other fallback statehandleExploreInitandhandleURLChangeto derive this grain state once the time range has been resolvedonSelectRangefunctionFilters.svelteto also properly derive this before triggering a state changeHowever, much of this could be simplified if the
/time-rangesAPI returned these allowed grains directly.Open questions:
Is total number of buckets the correct barometer? Both 30 days in hours and 12 hours in minutes would render 720 buckets. Are these equivalently "taxing" on the database?
What is the appropriate number of buckets to cap at? The API limit is quite high and the practical rendering limit is around 1000 (though can be increased with a rework of the time series component). 731 was somewhat arbitrary, but this was chosen to allow looking at two years worth of days.
Checklist: