This release contains 230 contributions from 29 contributors. We also have 9 new contributors. Thank you all for the contributions!
Some notable changes release are:
- Experimental native histogram ingestion via -blocks-storage.tsdb.enable-native-histograms
flag
- Support for filtering alerts on ListRules API
- Add query rejection mechanism to protect queries
- Introduce token bucket limiter to protect store gateway
- Implement ingester metadata API limits
- Remove -querier.query-store-for-labels-enabled
flag
- Remove -querier.at-modifier-enabled
flag
- Remove oltp_endpoint
config
- ruler.evaluation-delay-duration
mark as deprecated. Use ruler.query-offset
- -querier.max-outstanding-requests-per-tenant
and -query-scheduler.max-outstanding-requests-per-tenant
mark as deprecated. Use frontend.max-outstanding-requests-per-tenant
What's Changed
- [CHANGE] Ingester: Remove
-querier.query-store-for-labels-enabled
flag. Querying long-term store for labels is always enabled. [#5984] - [CHANGE] Server: Instrument
cortex_request_duration_seconds
metric with native histogram. Ifnative-histograms
feature is enabled in monitoring Prometheus then the metric name needs to be updated in your dashboards. [#6056] - [CHANGE] Distributor/Ingester: Change
cortex_distributor_ingester_appends_total
,cortex_distributor_ingester_append_failures_total
,cortex_distributor_ingester_queries_total
, andcortex_distributor_ingester_query_failures_total
metrics to use the ingester ID instead of its IP as the label value. [#6078] - [CHANGE] OTLP: Set
AddMetricSuffixes
to true to always enable metric name normalization. [#6136] - [CHANGE] Querier: Deprecate and enable by default
querier.ingester-metadata-streaming
flag. [#6147] - [CHANGE] QueryFrontend/QueryScheduler: Deprecate
-querier.max-outstanding-requests-per-tenant
and-query-scheduler.max-outstanding-requests-per-tenant
flags. Use frontend.max-outstanding-requests-per-tenant instead. [#6146] - [CHANGE] Ingesters: Enable 'snappy-block' compression on ingester clients by default. [#6148]
- [CHANGE] Ruler: Scheduling
ruler.evaluation-delay-duration
to be deprecated. Ruler will use the highest value betweenruler.evaluation-delay-duration
andruler.query-offset
[#6149] - [CHANGE] Querier: Remove
-querier.at-modifier-enabled
flag. [#6157] - [CHANGE] Tracing: Remove deprecated
oltp_endpoint
config entirely. [#6158] - [CHANGE] Store Gateway: Enable store gateway zone stable shuffle sharding by default. [#6161]
- [FEATURE] Ingester/Distributor: Experimental: Enable native histogram ingestion via
-blocks-storage.tsdb.enable-native-histograms
flag. [#5986] [#6010] [#6020] - [FEATURE] Querier: Enable querying native histogram chunks. [#5944] [#6031]
- [FEATURE] Query Frontend: Support native histogram in query frontend response. [#5996] [#6043]
- [FEATURE] Ruler: Support sending native histogram samples to Ingester. [#6029]
- [FEATURE] Ruler: Add support for filtering out alerts in ListRules API. [#6011]
- [FEATURE] Query Frontend: Added a query rejection mechanism to block resource-intensive queries. [#6005]
- [FEATURE] OTLP: Support ingesting OTLP exponential metrics as native histograms. [#6071] [#6135]
- [FEATURE] Ingester: Add
ingester.instance-limits.max-inflight-query-requests
to allow limiting ingester concurrent queries. [#6081] - [FEATURE] Distributor: Add
validation.max-native-histogram-buckets
to limit max number of bucket count. Distributor will try to automatically reduce histogram resolution until it is within the bucket limit or resolution cannot be reduced anymore. [#6104] - [FEATURE] Store Gateway: Introduce token bucket limiter to enhance store gateway throttling. [#6016]
- [FEATURE] Ruler: Add support for
query_offset
field on RuleGroup and newruler_query_offset
per-tenant limit. [#6085] - [ENHANCEMENT] Ruler: Add support to persist tokens in rulers. [#5987]
- [ENHANCEMENT] Query Frontend/Querier: Added store gateway postings touched count and touched size in Querier stats and log in Query Frontend. [#5892]
- [ENHANCEMENT] Query Frontend/Querier: Returns
warnings
on prometheus query responses. [#5916] - [ENHANCEMENT] Ingester: Allowing to configure
-blocks-storage.tsdb.head-compaction-interval
flag up to 30 min and add a jitter on the first head compaction. [#5919] [#5928] - [ENHANCEMENT] Distributor: Added
max_inflight_push_requests
config to ingester client to protect distributor from OOMKilled. [#5917] - [ENHANCEMENT] Distributor/Querier: Clean stale per-ingester metrics after ingester restarts. [#5930]
- [ENHANCEMENT] Distributor/Ring: Allow disabling detailed ring metrics by ring member. [#5931]
- [ENHANCEMENT] KV: Etcd Added etcd.ping-without-stream-allowed parameter to disable/enable PermitWithoutStream [#5933]
- [ENHANCEMENT] Ingester: Add a new
limits_per_label_set
limit. This limit functions similarly tomax_series_per_metric
, but allowing users to define the maximum number of series per LabelSet. [#5950] [#5993] - [ENHANCEMENT] Store Gateway: Log gRPC requests together with headers configured in
http_request_headers_to_log
. [#5958] - [ENHANCEMENT] Ingester: Add a new experimental
-ingester.labels-string-interning-enabled
flag to enable string interning for metrics labels. [#6057] - [ENHANCEMENT] Ingester: Add link to renew 10% of the ingesters tokens in the admin page. [#6063]
- [ENHANCEMENT] Ruler: Add support for filtering by
state
andhealth
field on Rules API. [#6040] - [ENHANCEMENT] Ruler: Add support for filtering by
match
field on Rules API. [#6083] - [ENHANCEMENT] Distributor: Reduce memory usage when error volume is high. [#6095]
- [ENHANCEMENT] Compactor: Centralize metrics used by compactor and add user label to compactor metrics. [#6096]
- [ENHANCEMENT] Compactor: Add unique execution ID for each compaction cycle in log for easy debugging. [#6097]
- [ENHANCEMENT] Compactor: Differentiate retry and halt error and retry failed compaction only on retriable error. [#6111]
- [ENHANCEMENT] Ruler: Add support for filtering by
state
andhealth
field on Rules API. [#6040] - [ENHANCEMENT] Compactor: Split cleaner cycle for active and deleted tenants. [#6112]
- [ENHANCEMENT] Compactor: Introduce cleaner visit marker. [#6113]
- [ENHANCEMENT] Query Frontend: Add
cortex_query_samples_total
metric. [#6142] - [ENHANCEMENT] Ingester: Implement metadata API limit. [#6128]
- [BUGFIX] Configsdb: Fix endline issue in db password. [#5920]
- [BUGFIX] Ingester: Fix
user
andtype
labels for thecortex_ingester_tsdb_head_samples_appended_total
TSDB metric. [#5952] - [BUGFIX] Querier: Enforce max query length check for
/api/v1/series
API even thoughignoreMaxQueryLength
is set to true. [#6018] - [BUGFIX] Ingester: Fix issue with the minimize token generator where it was not taking in consideration the current ownership of an instance when generating extra tokens. [#6062]
- [BUGFIX] Scheduler: Fix user queue in scheduler that was not thread-safe. [#6077] [#6160]
- [BUGFIX] Ingester: Include out-of-order head compaction when compacting TSDB head. [#6108]
- [BUGFIX] Ingester: Fix
cortex_ingester_tsdb_mmap_chunks_total
metric. [#6134] - [BUGFIX] Query Frontend: Fix query rejection bug for metadata queries. [#6143]
New Contributors
- @Yaxhveer made their first contribution in https://github.com/cortexproject/cortex/pull/5920
- @KrisBuytaert made their first contribution in https://github.com/cortexproject/cortex/pull/5933
- @wilguo made their first contribution in https://github.com/cortexproject/cortex/pull/5935
- @rapphil made their first contribution in https://github.com/cortexproject/cortex/pull/5987
- @harshitasao made their first contribution in https://github.com/cortexproject/cortex/pull/6061
- @deradiri made their first contribution in https://github.com/cortexproject/cortex/pull/6099
- @shekeriev made their first contribution in https://github.com/cortexproject/cortex/pull/6125
- @klingerf made their first contribution in https://github.com/cortexproject/cortex/pull/6131
- @SungJin1212 made their first contribution in https://github.com/cortexproject/cortex/pull/6142
Full Changelog: https://github.com/cortexproject/cortex/compare/v1.17.1...v1.18.0