OBSDOCS-1801: Loki - Adjust Queries for both OTEL and ViaQ #94828
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| <title>Logging alerts</title> | ||
| <productname>{product-title}</productname> | ||
| <productnumber>{product-version}</productnumber> | ||
| <subtitle>Configuring logging alerts.</subtitle> | ||
| <abstract> | ||
| <para>This document provides information about configuring logging alerts. | ||
| </para> | ||
| </abstract> | ||
| <authorgroup> | ||
| <orgname>Red Hat OpenShift Documentation Team</orgname> | ||
| </authorgroup> | ||
| <xi:include href="Common_Content/Legal_Notice.xml" xmlns:xi="http://www.w3.org/2001/XInclude" /> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,36 +1,30 @@ | ||
| // Module included in the following assemblies: | ||
| // | ||
| // * logging_alerts/default-logging-alerts.adoc | ||
|
|
||
| :_mod-docs-content-type: REFERENCE | ||
| [id="logging-vector-collector-alerts_{context}"] | ||
| = {clo} alerts | ||
|
|
||
| The following alerts are generated by the Vector collector. You can view these alerts in the {ocp-product-title} web console. | ||
|
|
||
| .Vector collector alerts | ||
| [cols="2,2,2,1",options="header"] | ||
| |=== | ||
| |Alert |Message |Description |Severity | ||
|
|
||
| |`CollectorNodeDown` | ||
| |`Prometheus could not scrape vector <instance> for more than 10m.` | ||
| |Vector is reporting that Prometheus could not scrape a specific Vector instance. | ||
| |Critical | ||
|
|
||
| |`DiskBufferUsage` | ||
| |`Collectors potentially consuming too much node disk, <value>` | ||
| |Collectors are consuming too much node disk on the host. | ||
| |Warning | ||
|
|
||
| |`CollectorHigh403ForbiddenResponseRate` | ||
|
Contributor
Author
There was a problem hiding this comment. ok, i revised the the target for this PR to 6.3. I'll create a separate PR for 6.2 and older versions. |
||
| |`High rate of "HTTP 403 Forbidden" responses detected for collector <instance> in namespace <namespace> for output <label>. The rate of 403 responses is <rate> over the last 2 minutes, persisting for more than 5 minutes. This could indicate an authorization issue.` | ||
| |At least 10% of sent requests responded with "HTTP 403 Forbidden" for collector "<intance>" in namespace <namespace> for the output "<output>". | ||
| |Critical | ||
| |=== | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,71 @@ | ||||||
| // Module included in the following assemblies: | ||||||
| // | ||||||
| // * logging_alerts/default-logging-alerts.adoc | ||||||
|
|
||||||
| :_mod-docs-content-type: REFERENCE | ||||||
| [id="loki-alerts_{context}"] | ||||||
| = {loki-op} alerts | ||||||
|
|
||||||
| The following alerts are generated by the {loki-op}. You can view these alerts in the {ocp-product-title} web console. | ||||||
|
|
||||||
| .{loki-op} alerts | ||||||
| [cols="2,2,2,1",options="header"] | ||||||
| |=== | ||||||
| |Alert |Message |Description |Severity | ||||||
|
|
||||||
| |`LokiRequestErrors` | ||||||
| |`{{ $labels.job }} {{ $labels.route }} is experiencing <value>% errors.` | ||||||
| |At least 10% of requests result in `5xx` server errors. | ||||||
| |critical | ||||||
|
|
||||||
| |`LokiStackWriteRequestErrors` | ||||||
| |`<value>% of write requests from {{ $labels.job }} in <namespace> are returned with server errors.` | ||||||
| |At least 10% of write requests to the lokistack-gateway result in `5xx` server errors. | ||||||
| |critical | ||||||
|
|
||||||
| |`LokiStackReadRequestErrors` | ||||||
| |`<value>% of query requests from {{ $labels.job }} in <namespace> are returned with server errors.` | ||||||
| |At least 10% of query requests to the lokistack-gateway result in `5xx` server errors. | ||||||
| |critical | ||||||
|
|
||||||
| |`LokiRequestPanics` | ||||||
| |`{{ $labels.job }} is experiencing an increase of <value> panics.` | ||||||
| |A panic was triggered. | ||||||
| |critical | ||||||
|
|
||||||
| |`LokiRequestLatency` | ||||||
| |`{{ $labels.job }} {{ $labels.route }} is experiencing <value>s 99th percentile latency.` | ||||||
| |The 99th percentile is experiencing latency higher than 1 second. | ||||||
| |critical | ||||||
|
|
||||||
| |`LokiTenantRateLimit` | ||||||
| |`{{ $labels.job }} {{ $labels.route }} is experiencing 429 errors.` | ||||||
| |At least 10% of requests are received the rate limit error code. | ||||||
| |warning | ||||||
|
|
||||||
| |`LokiStorageSlowWrite` | ||||||
| |`The storage path is experiencing slow write response rates.` | ||||||
| |The storage path is experiencing slow read response rates. | ||||||
| |warning | ||||||
|
|
||||||
| |`LokiWritePathHighLoad` | ||||||
| |`The write path is experiencing high load.`` | ||||||
| |The write path is experiencing high load causing backpressure storage flushing. | ||||||
| |warning | ||||||
|
|
||||||
| |`LokiReadPathHighLoad` | ||||||
| |`The read path is experiencing high load.` | ||||||
| |The read path has a high volume of queries, causing longer response times. | ||||||
| |warning | ||||||
|
|
||||||
| |`LokiDiscardedSamplesWarning` | ||||||
| |`Loki in namespace "<namespace>" is discarding samples in the "<tenant>" tenant during ingestion. Samples are discarded because of "<reason>" at a rate of <value> samples per second.` | ||||||
| |Loki is discarding samples during ingestion because they fail validation. | ||||||
| |warning | ||||||
|
|
||||||
| |`LokistackSchemaUpgradesRequired` | ||||||
| |`The LokiStack "{{ $labels.stack_name }}" in namespace "<namespace>" is using a storage schema | ||||||
| configuration that does not contain the latest schema version. It is recommended to update the schema configuration to update the schema version to the latest` | ||||||
|
Contributor
There was a problem hiding this comment. Not sure about this one, but the sentence is confusing to me.
Suggested change
Contributor
Author
There was a problem hiding this comment. Good catch, @mburke5678! This is the output text. I believe this should first be fixed upstream first, cc @xperimental |
||||||
| |One or more of the deployed LokiStacks contains an outdated storage schema configuration. | ||||||
| |warning | ||||||
| |=== | ||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,6 +1,6 @@ | ||
| // Module included in the following assemblies: | ||
| // | ||
| // * logging_alerts/custom-logging-alerts.adoc | ||
|
|
||
| :_mod-docs-content-type: REFERENCE | ||
| [id="loki-rbac-rules-permissions_{context}"] | ||
|
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.