Skip to content

Commit

Permalink
Merge branch 'master' into bump-banyandb-api
Browse files Browse the repository at this point in the history
  • Loading branch information
hanahmily authored May 8, 2024
2 parents fc47061 + b3044ee commit abd80c4
Show file tree
Hide file tree
Showing 14 changed files with 617 additions and 6 deletions.
65 changes: 60 additions & 5 deletions docs/en/api/metrics-query-expression.md
Original file line number Diff line number Diff line change
Expand Up @@ -232,9 +232,8 @@ Expression:
top_n(<metric_name>, <top_number>, <order>)
```

`top_number` is the number of the top results, should be a positive integer.

`order` is the order of the top results. The value of `order` can be `asc` or `des`.
- `top_number` is the number of the top results, should be a positive integer.
- `order` is the order of the top results. The value of `order` can be `asc` or `des`.

For example:
If we want to query the current service's top 10 instances with the highest `service_instance_cpm` metric value, we can use the following expression
Expand Down Expand Up @@ -273,11 +272,11 @@ AggregateLabels Operation takes an expression and performs an aggregate calculat

Expression:
```text
aggregate_labels(Expression, AggregateType<Optional>(<label1_name>,<label2_name>...))
aggregate_labels(Expression, <AggregateType>(<label1_name>,<label2_name>...))
```

- `AggregateType` is the type of the aggregation operation.
- `<label1_name>,<label2_name>...` is the label names that need to be aggregated. If not specified, all labels will be aggregated.
- `<label1_name>,<label2_name>...` is the label names that need to be aggregated. If not specified, all labels will be aggregated. Optional.

| AggregateType | Definition | ExpressionResultType |
|---------------|----------------------------------------------------|----------------------|
Expand Down Expand Up @@ -379,6 +378,62 @@ V(T1)-V(T1-2), V(T2)-V(T1-1), V(T3)-V(T1)
### Result Type
TIME_SERIES_VALUES.

## Sort Operation
### SortValues Operation
SortValues Operation takes an expression and sorts the values of the input expression result.

Expression:
```text
sort_values(Expression, <limit>, <order>)
```
- `limit` is the number of the sort results, should be a positive integer, if not specified, will return all results. Optional.
- `order` is the order of the sort results. The value of `order` can be `asc` or `des`.

For example:
If we want to sort the `service_resp_time` metric values in descending order and get the top 10 values, we can use the following expression:
```text
sort_values(service_resp_time, 10, des)
```

#### Result Type
The result type follows the input expression.

### SortLabelValues Operation
SortLabelValues Operation takes an expression and sorts the label values of the input expression result. This function uses `natural sort order`.

Expression:
```text
sort_label_values(Expression, <order>, <label1_name>, <label2_name> ...)
```
- `order` is the order of the sort results. The value of `order` can be `asc` or `des`.
- `<label1_name>, <label2_name> ...` is the label names that need to be sorted by their values. At least one label name should be specified.
The labels in the head of the list will be sorted first, and if the label not be included in the expression result will be ignored.

For example:
If we want to sort the `service_percentile` metric label values in descending order by the `p` label, we can use the following expression:
```text
sort_label_values(service_percentile{p='50,75,90,95,99'}, des, p)
```

For multiple labels, assume the metric has 2 labels:
```text
metric{label1='a', label2='2a'}
metric{label1='a', label2='2c'}
metric{label1='b', label2='2a'}
metric{label1='b', label2='2c'}
```
If we want to sort the `metric` metric label values in descending order by the `label1` and `label2` labels, we can use the following expression:
```text
sort_label_values(metric, des, label1, label2)
```
And the result will be:
```text
metric{label1='b', label2='2c'}
metric{label1='b', label2='2a'}
metric{label1='a', label2='2c'}
metric{label1='a', label2='2a'}
```

## Expression Query Example
### Labeled Value Metrics
```text
Expand Down
1 change: 1 addition & 0 deletions docs/en/changes/changes.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,7 @@
* Add component definition(ID=152) for `c3p0`(JDBC3 Connection and Statement Pooling).
* Fix MQE `top_n` global query.
* Fix inaccurate Pulsar and Bookkeeper metrics.
* MQE support `sort_values` and `sort_label_values` functions.

#### UI

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,10 @@ DES options { caseInsensitive=true; }: 'des';
// AGGREGATE_LABELS
AGGREGATE_LABELS: 'aggregate_labels';

// Sort
SORT_VALUES: 'sort_values';
SORT_LABEL_VALUES: 'sort_label_values';

// Literals
INTEGER: Digit+;
DECIMAL: Digit+ DOT Digit+;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,8 @@ expression
| topN L_PAREN metric COMMA INTEGER COMMA order R_PAREN #topNOP
| relabels L_PAREN expression COMMA label COMMA replaceLabel R_PAREN #relablesOP
| aggregateLabels L_PAREN expression COMMA aggregateLabelsFunc R_PAREN #aggregateLabelsOp
| sort_values L_PAREN expression (COMMA INTEGER)? COMMA order R_PAREN #sortValuesOP
| sort_label_values L_PAREN expression COMMA order COMMA labelNameList R_PAREN #sortLabelValuesOP
;

expressionList
Expand Down Expand Up @@ -91,3 +93,9 @@ aggregateLabelsFunc: aggregateLabelsFuncName (L_PAREN labelNameList R_PAREN)?;

aggregateLabelsFuncName:
AVG | SUM | MAX | MIN;

sort_values:
SORT_VALUES;

sort_label_values:
SORT_LABEL_VALUES;
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import lombok.extern.slf4j.Slf4j;
import org.apache.skywalking.mqe.rt.grammar.MQEParser;
import org.apache.skywalking.mqe.rt.grammar.MQEParserBaseVisitor;
Expand All @@ -31,6 +32,8 @@
import org.apache.skywalking.mqe.rt.operation.CompareOp;
import org.apache.skywalking.mqe.rt.operation.LogicalFunctionOp;
import org.apache.skywalking.mqe.rt.operation.MathematicalFunctionOp;
import org.apache.skywalking.mqe.rt.operation.SortLabelValuesOp;
import org.apache.skywalking.mqe.rt.operation.SortValuesOp;
import org.apache.skywalking.mqe.rt.operation.TrendOp;
import org.apache.skywalking.mqe.rt.type.ExpressionResult;
import org.apache.skywalking.mqe.rt.exception.IllegalExpressionException;
Expand Down Expand Up @@ -311,6 +314,42 @@ public ExpressionResult visitTrendOP(MQEParser.TrendOPContext ctx) {
}
}

@Override
public ExpressionResult visitSortValuesOP(MQEParser.SortValuesOPContext ctx) {
ExpressionResult result = visit(ctx.expression());
int order = ctx.order().getStart().getType();
Optional<Integer> limit = Optional.empty();
if (ctx.INTEGER() != null) {
limit = Optional.of(Integer.valueOf(ctx.INTEGER().getText()));
}
try {
return SortValuesOp.doSortValuesOp(result, limit, order);
} catch (IllegalExpressionException e) {
ExpressionResult errorResult = new ExpressionResult();
errorResult.setType(ExpressionResultType.UNKNOWN);
errorResult.setError(e.getMessage());
return errorResult;
}
}

@Override
public ExpressionResult visitSortLabelValuesOP(MQEParser.SortLabelValuesOPContext ctx) {
ExpressionResult result = visit(ctx.expression());
int order = ctx.order().getStart().getType();
List<String> labelNames = new ArrayList<>();
for (MQEParser.LabelNameContext labelNameContext : ctx.labelNameList().labelName()) {
labelNames.add(labelNameContext.getText());
}
try {
return SortLabelValuesOp.doSortLabelValuesOp(result, order, labelNames);
} catch (IllegalExpressionException e) {
ExpressionResult errorResult = new ExpressionResult();
errorResult.setType(ExpressionResultType.UNKNOWN);
errorResult.setError(e.getMessage());
return errorResult;
}
}

@Override
public abstract ExpressionResult visitMetric(MQEParser.MetricContext ctx);

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/

package org.apache.skywalking.mqe.rt.operation;

import java.util.Comparator;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import lombok.extern.slf4j.Slf4j;
import org.apache.skywalking.mqe.rt.exception.IllegalExpressionException;
import org.apache.skywalking.mqe.rt.grammar.MQEParser;
import org.apache.skywalking.mqe.rt.type.ExpressionResult;
import org.apache.skywalking.mqe.rt.type.MQEValues;
import org.apache.skywalking.oap.server.core.query.type.KeyValue;
import org.apache.skywalking.oap.server.library.util.CollectionUtils;

import static java.util.stream.Collectors.groupingBy;
import static java.util.stream.Collectors.toList;

@Slf4j
public class SortLabelValuesOp {
public static ExpressionResult doSortLabelValuesOp(ExpressionResult expResult,
int order,
List<String> labelNames) throws IllegalExpressionException {
if (CollectionUtils.isNotEmpty(labelNames)) {
labelNames = labelNames.stream().distinct().collect(toList());
if (MQEParser.ASC == order) {
expResult.setResults(
sort(expResult.getResults(), labelNames, labelNames.get(0), Comparator.naturalOrder()));
} else if (MQEParser.DES == order) {
expResult.setResults(
sort(expResult.getResults(), labelNames, labelNames.get(0), Comparator.reverseOrder()));
} else {
throw new IllegalExpressionException("Unsupported sort order.");
}
}
return expResult;
}

private static List<MQEValues> sort(List<MQEValues> results,
List<String> sortLabels,
String currentSortLabel,
Comparator<String> comparator) {
if (!sortLabels.contains(currentSortLabel)) {
log.error("Current sort label {} not found in the sort labels {} ", currentSortLabel, sortLabels);
return results;
}
if (sortLabels.indexOf(
currentSortLabel) == sortLabels.size() - 1) { //only one label or the latest label no need to group
results.sort(Comparator.comparing(mqeValues -> mqeValues.getMetric()
.getLabels()
.stream()
.filter(kv -> kv.getKey().equals(currentSortLabel))
.findFirst()
.orElse(new KeyValue(currentSortLabel, ""))
.getValue(), comparator));
} else {
LinkedHashMap<KeyValue, List<MQEValues>> groupResult = group(results, currentSortLabel);
LinkedHashMap<KeyValue, List<MQEValues>> sortedGroup = new LinkedHashMap<>(groupResult.size());
for (Map.Entry<KeyValue, List<MQEValues>> entry : groupResult.entrySet()) {
//sort by next label
List<MQEValues> sortedResult = sort(
entry.getValue(), sortLabels, sortLabels.get(sortLabels.indexOf(currentSortLabel) + 1), comparator);
sortedGroup.put(entry.getKey(), sortedResult);
}
//combine the sorted group
results = sortedGroup.keySet()
.stream()
.sorted(Comparator.comparing(KeyValue::getValue, comparator))
.flatMap(keyValue -> sortedGroup.get(keyValue).stream())
.collect(toList());
}
return results;
}

//group by current label for sub sorting
private static LinkedHashMap<KeyValue, List<MQEValues>> group(List<MQEValues> results, String labelName) {
return results
.stream()
.collect(groupingBy(
mqeValues -> mqeValues.getMetric().getLabels()
.stream()
.filter(kv -> kv.getKey().equals(labelName))
.findFirst().orElse(new KeyValue(labelName, "")),
LinkedHashMap::new,
toList()
));
}
}

Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/

package org.apache.skywalking.mqe.rt.operation;

import java.util.Comparator;
import java.util.List;
import java.util.Optional;
import java.util.stream.Collectors;
import org.apache.skywalking.mqe.rt.exception.IllegalExpressionException;
import org.apache.skywalking.mqe.rt.grammar.MQEParser;
import org.apache.skywalking.mqe.rt.type.ExpressionResult;
import org.apache.skywalking.mqe.rt.type.MQEValue;

public class SortValuesOp {
public static ExpressionResult doSortValuesOp(ExpressionResult expResult,
Optional<Integer> limit,
int order) throws IllegalExpressionException {
if (MQEParser.ASC == order || MQEParser.DES == order) {
expResult.getResults().forEach(mqeValues -> {
List<MQEValue> values = mqeValues.getValues()
.stream()
// Filter out empty values
.filter(mqeValue -> !mqeValue.isEmptyValue())
.sorted(MQEParser.ASC == order ? Comparator.comparingDouble(
MQEValue::getDoubleValue) :
Comparator.comparingDouble(MQEValue::getDoubleValue)
.reversed())
.collect(
Collectors.toList());
if (limit.isPresent() && limit.get() < values.size()) {
mqeValues.setValues(values.subList(0, limit.get()));
} else {
mqeValues.setValues(values);
}
});
} else {
throw new IllegalExpressionException("Unsupported sort order.");
}
return expResult;
}
}
Loading

0 comments on commit abd80c4

Please sign in to comment.