Skip to content

Commit 49adca3

Browse files
dwdoughertydwdougherty
authored
DEV: FT.AGGREGATE expression precedence (#1780)
1 parent a40a591 commit 49adca3

File tree

3 files changed

+378
-2
lines changed

3 files changed

+378
-2
lines changed

‎content/develop/ai/search-and-query/advanced-concepts/aggregations-syntax.md

Lines changed: 376 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,376 @@
1+
---
2+
categories:
3+
- docs
4+
- develop
5+
- stack
6+
- oss
7+
- rs
8+
- rc
9+
- oss
10+
- kubernetes
11+
- clients
12+
description: Order of operations for the FT.AGGREGATE command
13+
linkTitle: Aggregation syntax
14+
title: FT.AGGREGATE order of operations
15+
weight: 2
16+
---
17+
18+
## Overview
19+
20+
[`FT.AGGREGATE`]({{< relref "/commands/ft.aggregate" >}}) is a powerful Redis Query Engine (RQE) command for performing advanced data aggregation, filtering, sorting, and transformations on indexed hash or JSON documents. This reference page provides a structured breakdown of syntax, ordering rules, and best practices.
21+
22+
The [main aggregations page]({{< relref "/develop/ai/search-and-query/advanced-concepts/aggregations" >}}) has a simple diagram showing how `FT.AGGREGATE` pipelines are constructed, but it doesn't tell the whole story. For example, you can create more complex aggregation pipelines by applying multiple `REDUCE` functions under a single `GROUPBY` clause, or you can chain groupings and mix in additional mapping steps:
23+
24+
`GROUPBY` ... `REDUCE` ... `APPLY` ... `GROUPBY` ... `REDUCE`
25+
26+
{{< note >}}
27+
The examples on this page are based on a hypothetical "products" data set, which you can [download here](./data/products.txt).
28+
{{< /note >}}
29+
30+
## Syntax and expression ordering
31+
32+
The `FT.AGGREGATE` command processes multiple expressions in a pipeline. Below is the recommended order:
33+
34+
1. `index` – the name of your index, which must be the first argument.
35+
1. `query` – your query, which must be the second argument.
36+
1. `FILTER` – filters raw documents before transformations or aggregation.
37+
1. `LOAD` – loads document fields.
38+
1. `APPLY` – applies transformations on fields.
39+
1. `GROUPBY` – groups results by specific fields.
40+
1. `REDUCE` – performs aggregations. For example, `SUM`, `COUNT`, and `AVG`.
41+
1. `SORTBY` – orders the results based on specified fields.
42+
1. `LIMIT` – restricts the number of results returned.
43+
1. `DIALECT 2` - provides for more comprehensive query syntax, for example using parameters in `FILTER` expressions.
44+
45+
Other keywords will be discussed toward the end of this page.
46+
47+
## When to use `@`
48+
49+
You must add `@` at the start of a field name in the following circumstances:
50+
51+
- When referencing fields loaded from documents. In the following example, `price` is a document field and must be prefixed with `@`.
52+
53+
```sh
54+
FT.AGGREGATE products "*"
55+
LOAD 1 @price
56+
APPLY "@price * 1.1" AS adjusted_price
57+
SORTBY 2 @adjusted_price DESC
58+
LIMIT 0 10
59+
60+
1) (integer) 200
61+
2) 1) "price"
62+
2) "623"
63+
3) "adjusted_price"
64+
4) "685.3"
65+
3) 1) "price"
66+
2) "619.75"
67+
3) "adjusted_price"
68+
4) "681.725"
69+
.
70+
.
71+
.
72+
```
73+
74+
- When referencing fields inside a `FILTER` clause that were loaded from documents.
75+
76+
```sh
77+
FT.AGGREGATE products "*"
78+
LOAD 1 @rating
79+
FILTER "@rating >= 4.5"
80+
LIMIT 0 10
81+
82+
1) (integer) 5
83+
2) 1) "rating"
84+
2) "4.5"
85+
3) 1) "rating"
86+
2) "4.8"
87+
4) 1) "rating"
88+
2) "4.5"
89+
.
90+
.
91+
.
92+
```
93+
94+
- When referencing fields inside `GROUPBY` or `REDUCE` clauses.
95+
96+
```sh
97+
FT.AGGREGATE products "*"
98+
GROUPBY 1 @category
99+
REDUCE SUM 1 @price AS total_price
100+
LIMIT 0 10
101+
102+
1) (integer) 6
103+
2) 1) "category"
104+
2) "Toys"
105+
3) "total_price"
106+
4) "9799.25"
107+
3) 1) "category"
108+
2) "Electronics"
109+
3) "total_price"
110+
4) "10683.5"
111+
4) 1) "category"
112+
2) "Apparel"
113+
3) "total_price"
114+
4) "10273.5"
115+
.
116+
.
117+
.
118+
```
119+
120+
- When referencing fields created by `REDUCE` in `APPLY` or `FILTER` clauses.
121+
122+
```sh
123+
FT.AGGREGATE products "*"
124+
GROUPBY 1 @category
125+
REDUCE SUM 1 @price AS total_price
126+
APPLY "@total_price * 1.2" AS boosted_price
127+
FILTER "@total_price > 1000"
128+
LIMIT 0 10
129+
130+
1) (integer) 6
131+
2) 1) "category"
132+
2) "Toys"
133+
3) "total_price"
134+
4) "9799.25"
135+
5) "boosted_price"
136+
6) "11759.1"
137+
3) 1) "category"
138+
2) "Electronics"
139+
3) "total_price"
140+
4) "10683.5"
141+
5) "boosted_price"
142+
6) "12820.2"
143+
.
144+
.
145+
.
146+
```
147+
148+
- When referencing fields created by `APPLY` in another `APPLY` or `FILTER` clause.
149+
150+
```sh
151+
FT.AGGREGATE products "*"
152+
LOAD 2 @price @discount
153+
APPLY "@price - @discount" AS net_price
154+
APPLY "@net_price * 1.1" AS marked_up
155+
FILTER "@net_price > 200"
156+
LIMIT 0 10
157+
158+
1) (integer) 60
159+
2) 1) "price"
160+
2) "220"
161+
3) "discount"
162+
4) "0"
163+
5) "net_price"
164+
6) "220"
165+
7) "marked_up"
166+
8) "242"
167+
3) 1) "price"
168+
2) "223.25"
169+
3) "discount"
170+
4) "1.5"
171+
5) "net_price"
172+
6) "221.75"
173+
7) "marked_up"
174+
8) "243.925"
175+
.
176+
.
177+
.
178+
```
179+
180+
- When referencing fields created by `APPLY` in a `SORTBY` clause.
181+
182+
```sh
183+
FT.AGGREGATE products "*"
184+
LOAD 2 @price @discount
185+
APPLY "@price - @discount" AS net_price
186+
SORTBY 2 @net_price DESC
187+
LIMIT 0 10
188+
189+
1) (integer) 200
190+
2) 1) "price"
191+
2) "623"
192+
3) "discount"
193+
4) "6"
194+
5) "net_price"
195+
6) "617"
196+
3) 1) "price"
197+
2) "619.75"
198+
3) "discount"
199+
4) "4.5"
200+
5) "net_price"
201+
6) "615.25"
202+
.
203+
.
204+
.
205+
```
206+
207+
## GROUPBY with multiple REDUCE operations
208+
209+
You can use multiple `REDUCE` operations after `GROUPBY` for different aggregations.
210+
211+
```sh
212+
FT.AGGREGATE products "*"
213+
GROUPBY 1 @category
214+
REDUCE COUNT 0 AS product_count
215+
REDUCE SUM 1 @price AS total_price
216+
REDUCE AVG 1 @rating AS avg_rating
217+
SORTBY 2 @total_price DESC
218+
LIMIT 0 10
219+
220+
1) (integer) 6
221+
2) 1) "category"
222+
2) "Groceries"
223+
3) "product_count"
224+
4) "44"
225+
5) "total_price"
226+
6) "13495.25"
227+
7) "avg_rating"
228+
8) "3.94090909091"
229+
3) 1) "category"
230+
2) "Home"
231+
3) "product_count"
232+
4) "40"
233+
5) "total_price"
234+
6) "11026.75"
235+
7) "avg_rating"
236+
8) "3.78"
237+
.
238+
.
239+
.
240+
```
241+
242+
## Multiple APPLY operations followed by GROUPBY and REDUCE
243+
244+
You can use `APPLY` in various ways before and after `GROUPBY` and `REDUCE`.
245+
246+
```sh
247+
FT.AGGREGATE products "*"
248+
LOAD 3 @price @discount @quantity
249+
APPLY "@price - @discount" AS final_price
250+
APPLY "@final_price * @quantity" AS total_revenue
251+
GROUPBY 1 @category
252+
REDUCE SUM 1 @total_revenue AS total_category_revenue
253+
SORTBY 2 @total_category_revenue DESC
254+
LIMIT 0 10
255+
256+
1) (integer) 6
257+
2) 1) "category"
258+
2) "Groceries"
259+
3) "total_category_revenue"
260+
4) "81373"
261+
3) 1) "category"
262+
2) "Home"
263+
3) "total_category_revenue"
264+
4) "55797.5"
265+
.
266+
.
267+
.
268+
```
269+
270+
## FILTER and PARAMS
271+
272+
Use `FILTER` to remove unwanted records, and `PARAMS` to pass values to parameterized queries.
273+
274+
```sh
275+
FT.AGGREGATE products "*"
276+
LOAD 3 @price @rating @quantity
277+
FILTER "@price >= 500"
278+
FILTER "@rating >= 4.0"
279+
APPLY "@price * @quantity" AS total_value
280+
SORTBY 2 @total_value DESC
281+
LIMIT 0 10
282+
DIALECT 2
283+
284+
1) (integer) 200
285+
2) 1) "price"
286+
2) "606.75"
287+
3) "rating"
288+
4) "4.2"
289+
5) "quantity"
290+
6) "10"
291+
7) "total_value"
292+
8) "6067.5"
293+
3) 1) "price"
294+
2) "541.75"
295+
3) "rating"
296+
4) "4.5"
297+
5) "quantity"
298+
6) "10"
299+
7) "total_value"
300+
8) "5417.5"
301+
.
302+
.
303+
.
304+
```
305+
306+
## Placement of FILTER before and after GROUPBY/APPLY
307+
308+
- **Before GROUPBY:** Removes records before aggregation.
309+
- **After GROUPBY:** Filters based on aggregated results.
310+
311+
## LOAD after GROUPBY/REDUCE
312+
313+
This is not allowed and you'll get a syntax error.
314+
315+
## Placement rules for specific parameters
316+
317+
| Parameter | Placement |
318+
|----- |----- |
319+
| `TIMEOUT` | Can be placed anywhere. |
320+
| `LIMIT` | Must be at the end, before `DIALECT`. |
321+
| `WITHCURSOR` | Must be at the end, before `DIALECT`. |
322+
| `SCORER` | Must be placed between the query and pipeline operations. |
323+
| `ADDSCORES` | Must be placed between the query and pipeline operations. |
324+
| `DIALECT` | Must be at the end. |
325+
326+
## LIMIT and WITHCURSOR used together
327+
328+
While you wouldn't ordinarily use `LIMIT` and `WITHCURSOR` together in the same query, you can do so if necessary.
329+
`LIMIT`, as the name suggests, will limit the total number of results returned for the given query. `WITHCURSOR` will paginate the results in chunks of size `COUNT`. You can use the [cursor API]({{< relref "/develop/ai/search-and-query/advanced-concepts/aggregations/#cursor-api" >}}) to retrieve more results, as shown below.
330+
331+
```sh
332+
FT.AGGREGATE products "*"
333+
GROUPBY 1 @category
334+
REDUCE COUNT 0 AS product_count
335+
LIMIT 0 100
336+
WITHCURSOR COUNT 3
337+
338+
1) 1) (integer) 6
339+
2) 1) "category"
340+
2) "Toys"
341+
3) "product_count"
342+
4) "28"
343+
3) 1) "category"
344+
2) "Electronics"
345+
3) "product_count"
346+
4) "31"
347+
4) 1) "category"
348+
2) "Apparel"
349+
3) "product_count"
350+
4) "36"
351+
2) (integer) 89400486
352+
127.0.0.1:6379> FT.CURSOR READ products 89400486 COUNT 3
353+
1) 1) (integer) 0
354+
2) 1) "category"
355+
2) "Home"
356+
3) "product_count"
357+
4) "40"
358+
3) 1) "category"
359+
2) "Groceries"
360+
3) "product_count"
361+
4) "44"
362+
4) 1) "category"
363+
2) "Books"
364+
3) "product_count"
365+
4) "21"
366+
2) (integer) 89400486
367+
.
368+
.
369+
.
370+
```
371+
372+
See the following resources for more information:
373+
374+
- [Aggregations]({{< relref "/develop/ai/search-and-query/advanced-concepts/aggregations" >}}) discussion page.
375+
- [`FT.AGGREGATE` command page](https://redis.io/docs/latest/commands/ft.aggregate/)
376+
- [RQE source code](https://github.com/RediSearch/RediSearch/tree/master/src/aggregate)

‎content/develop/ai/search-and-query/advanced-concepts/autocomplete.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ categories:
1414
description: Learn how to use the autocomplete feature of Redis for efficient prefix-based suggestion retrieval.
1515
linkTitle: Autocomplete
1616
title: Autocomplete with Redis
17-
weight: 1
17+
weight: 3
1818
---
1919

2020
## Overview

‎content/develop/ai/search-and-query/advanced-concepts/stopwords.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ categories:
1414
description: Stop words support
1515
linkTitle: Stop words
1616
title: Stop words
17-
weight: 1
17+
weight: 4
1818
---
1919

2020
Redis Open Source has a default list of [stop words](https://en.wikipedia.org/wiki/Stop_words). These are words that are usually so common that they do not add much information to search, but take up a lot of space and CPU time in the index.

0 commit comments

Comments
 (0)