You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardexpand all lines: docs/en/stack/ml/anomaly-detection/ml-detect-categories.asciidoc
+2-2
Original file line number
Diff line number
Diff line change
@@ -33,8 +33,8 @@ Avoid using human-generated data for categorization analysis.
33
33
[[creating-categorization-jobs]]
34
34
== Creating categorization jobs
35
35
36
-
. In {kib}, navigate to **{ml-app} > Anomaly Detection > Jobs**.
37
-
. Click **Create {anomaly-jobs}**, select the {data-view} you want to analyze.
36
+
. In {kib}, navigate to *Jobs*. To open *Jobs*, find **{ml-app} > Anomaly Detection** in the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field].
37
+
. Click **Create job**, select the {data-view} you want to analyze.
38
38
. Select the **Categorization** wizard from the list.
39
39
. Choose a categorization detector - it's the `count` function in this example - and the field you want to categorize - the `message` field in this example.
Copy file name to clipboardexpand all lines: docs/en/stack/ml/anomaly-detection/ml-jobs-from-visuals.asciidoc
+1-1
Original file line number
Diff line number
Diff line change
@@ -40,7 +40,7 @@ NOTE: You need to have a compatible visualization on **Dashboard** to create an
40
40
which is based on the {kib} sample flight data set. Select the `Flight count`
41
41
visualization from the dashboard.
42
42
43
-
. Go to **Analytics > Dashboard** and select a dashboard with a compatible
43
+
. Go to **Analytics > Dashboard** from the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field]. Select a dashboard with a compatible
44
44
visualization.
45
45
. Open the **Options (...) menu** for the panel, then select **More**.
46
46
. Select **Create {anomaly-job}**. The option is only displayed if the
Population analysis is a method of detecting anomalies by comparing the behavior of entities or events within a specified population.
5
+
In this approach, {ml} analytics create a profile of what is considered "typical" behavior for users, machines, or other entities over a specified time period.
6
+
An entity is considered as anomalous when its behavior deviates from that of the population, indicating abnormal activity compared to the rest of the population.
7
+
8
+
This type of analysis is most effective when the behavior within a group is generally homogeneous, allowing for the identification of unusual patterns.
9
+
However, it is less useful when members of the population show vastly different behaviors.
10
+
In such cases, you can segment your data into groups with similar behaviors and run separate jobs for each.
11
+
This can be done by using a query filter in the datafeed or by applying the `partition_field_name` to split the analysis across different groups.
12
+
13
+
Population analysis is resource-efficient and scales well, enabling the analysis of populations consisting of hundreds of thousands or even millions of entities with a lower resource footprint than analyzing each series individually.
14
+
15
+
16
+
17
+
[discrete]
18
+
[[population-recommendations]]
19
+
== Recommendations
20
+
21
+
* Use population analysis when the behavior within a group is mostly homogeneous, as it helps identify anomalous patterns effectively.
22
+
* Leverage population analysis when dealing with large-scale datasets.
23
+
* Avoid using population analysis when members of the population exhibit vastly different behaviors, as it may not be effective.
24
+
25
+
26
+
[discrete]
27
+
[[creating-population-jobs]]
28
+
== Creating population jobs
29
+
30
+
. In {kib}, navigate to *Jobs*. To open *Jobs*, find **{ml-app} > Anomaly Detection** in the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field].
31
+
. Click **Create job**, select the {data-source} you want to analyze.
32
+
. Select the **Population** wizard from the list.
33
+
. Choose a population field - it's the `clientip` field in this example - and the metric you want to use for the analysis - `Mean(bytes)` in this example.
34
+
+
35
+
--
36
+
[role="screenshot"]
37
+
image::images/ml-population-wizard.png[Creating a population job in Kibana]
38
+
--
39
+
. Click **Next**.
40
+
. Provide a job ID and click **Next**.
41
+
. If the validation is successful, click **Next** to review the summary of the job creation.
42
+
. Click **Create job**.
43
+
44
+
[%collapsible]
45
+
.API example
46
+
====
47
+
To specify the population, use the `over_field_name` property. For example:
48
+
49
+
[source,console]
50
+
----------------------------------
51
+
PUT _ml/anomaly_detectors/population
52
+
{
53
+
"description" : "Population analysis",
54
+
"analysis_config" : {
55
+
"bucket_span":"15m",
56
+
"influencers": [
57
+
"clientip"
58
+
],
59
+
"detectors": [
60
+
{
61
+
"function": "mean",
62
+
"field_name": "bytes",
63
+
"over_field_name": "clientip" <1>
64
+
}
65
+
]
66
+
},
67
+
"data_description" : {
68
+
"time_field":"timestamp",
69
+
"time_format": "epoch_ms"
70
+
}
71
+
}
72
+
----------------------------------
73
+
// TEST[skip:needs-licence]
74
+
75
+
<1> This `over_field_name` property indicates that the metrics for each client (as identified by their IP address) are analyzed relative to other clients in each bucket.
76
+
====
77
+
78
+
[discrete]
79
+
[[population-job-results]]
80
+
=== Viewing the job results
81
+
82
+
Use the **Anomaly Explorer** in {kib} to view the analysis results:
83
+
84
+
[role="screenshot"]
85
+
image::images/ml-population-anomalies.png["Population results in the Anomaly Explorer"]
86
+
87
+
The results are often quite sparse.
88
+
There might be just a few data points for the selected time period.
89
+
Population analysis is particularly useful when you have many entities and the data for specific entitles is sporadic or sparse.
90
+
If you click on a section in the timeline or swim lanes, you can see more details about the anomalies:
91
+
92
+
[role="screenshot"]
93
+
image::images/ml-population-anomaly.png["Anomaly details for a specific user"]
94
+
95
+
In this example, the client IP address `167.145.234.154` received a high volume of bytes on the date and time shown.
96
+
This event is anomalous because the mean is four times higher than the expected behavior of the population.
Copy file name to clipboardexpand all lines: docs/en/stack/ml/anomaly-detection/ml-revert-model-snapshot.asciidoc
+1-1
Original file line number
Diff line number
Diff line change
@@ -7,7 +7,7 @@ resilience. It makes it possible to reset the model to a previous state in case
7
7
of a system failure or if the model changed significantly due to a one-off
8
8
event.
9
9
10
-
. In {kib}, navigate to **{ml-app} > Anomaly Detection > Jobs**.
10
+
. In {kib}, navigate to *Jobs*. To open *Jobs*, find **{ml-app} > Anomaly Detection** in the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field].
11
11
. Locate the {anomaly-job} whose model you want to revert in the job table.
12
12
. Open the job details and navigate to the **Model Snapshots** tab.
Copy file name to clipboardexpand all lines: docs/en/stack/ml/df-analytics/ml-dfa-shared.asciidoc
+1-1
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
tag::dfa-deploy-model[]
2
2
. To deploy {dfanalytics} model in a pipeline, navigate to **Machine Learning** >
3
-
**Model Management** > **Trained models** in {kib}.
3
+
**Model Management** > **Trained models** in the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field] in {kib}.
4
4
5
5
. Find the model you want to deploy in the list and click **Deploy model** in
Copy file name to clipboardexpand all lines: docs/en/stack/ml/get-started/ml-gs-visualizer.asciidoc
+1-1
Original file line number
Diff line number
Diff line change
@@ -17,7 +17,7 @@ exception for your {kib} URL.
17
17
18
18
--
19
19
20
-
. Click *Machine Learning* in the {kib} main menu.
20
+
. Open *Machine Learning* from the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field].
Copy file name to clipboardexpand all lines: docs/en/stack/ml/nlp/ml-nlp-e5.asciidoc
+2-2
Original file line number
Diff line number
Diff line change
@@ -92,7 +92,7 @@ NOTE: For most cases, the preferred version is the **Intel and Linux optimized**
92
92
[[trained-model-e5]]
93
93
==== Using the Trained Models page
94
94
95
-
1. In {kib}, navigate to **{ml-app}** > **Trained Models**. E5 can be found in
95
+
1. In {kib}, navigate to **{ml-app}** > **Trained Models** from the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field]. E5 can be found in
96
96
the list of trained models. There are two versions available: one portable
97
97
version which runs on any hardware and one version which is optimized for Intel®
98
98
silicon. You can see which model is recommended to use based on your hardware
. Repeat step 2 and step 3 on all master-eligible nodes.
251
251
. {ref}/restart-cluster.html#restart-cluster-rolling[Restart] the
252
252
master-eligible nodes one by one.
253
-
. Navigate to the **Trained Models** page in {kib}, E5 can be found in the
253
+
. Navigate to the **Trained Models** page from the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field] in {kib}. E5 can be found in the
254
254
list of trained models.
255
255
. Click the **Add trained model** button, select the E5 model version you
256
256
downloaded in step 1 and want to deploy and click **Download**. The selected
Copy file name to clipboardexpand all lines: docs/en/stack/ml/nlp/ml-nlp-elser.asciidoc
+3-3
Original file line number
Diff line number
Diff line change
@@ -351,7 +351,7 @@ master-eligible nodes can reach the server you specify.
351
351
. Repeat step 5 on all master-eligible nodes.
352
352
. {ref}/restart-cluster.html#restart-cluster-rolling[Restart] the
353
353
master-eligible nodes one by one.
354
-
. Navigate to the **Trained Models** page in {kib}, ELSER can be found in the
354
+
. Navigate to the **Trained Models** page from the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field] in {kib}. ELSER can be found in the
355
355
list of trained models.
356
356
. Click the **Add trained model** button, select the ELSER model version you
357
357
downloaded in step 1 and want to deploy, and click **Download**. The selected
. Repeat step 2 and step 3 on all master-eligible nodes.
392
392
. {ref}/restart-cluster.html#restart-cluster-rolling[Restart] the
393
393
master-eligible nodes one by one.
394
-
. Navigate to the **Trained Models** page in {kib}, ELSER can be found in the
394
+
. Navigate to the **Trained Models** page from the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field] in {kib}. ELSER can be found in the
395
395
list of trained models.
396
396
. Click the **Add trained model** button, select the ELSER model version you
397
397
downloaded in step 1 and want to deploy and click **Download**. The selected
@@ -407,7 +407,7 @@ allocations and threads per allocation values.
407
407
== Testing ELSER
408
408
409
409
You can test the deployed model in {kib}. Navigate to **Model Management** >
410
-
**Trained Models**, locate the deployed ELSER model in the list of trained
410
+
**Trained Models** from the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field] in {kib}. Locate the deployed ELSER model in the list of trained
411
411
models, then select **Test model** from the Actions menu.
412
412
413
413
You can use data from an existing index to test the model. Select the index,
Copy file name to clipboardexpand all lines: docs/en/stack/ml/nlp/ml-nlp-inference.asciidoc
+1-1
Original file line number
Diff line number
Diff line change
@@ -18,7 +18,7 @@ can use it to perform {nlp} tasks in ingest pipelines.
18
18
== Add an {infer} processor to an ingest pipeline
19
19
20
20
In {kib}, you can create and edit pipelines in **{stack-manage-app}** >
21
-
**Ingest Pipelines**.
21
+
**Ingest Pipelines**. To open **Ingest Pipelines**, find **{stack-manage-app}** in the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field].
22
22
23
23
[role="screenshot"]
24
24
image::images/ml-nlp-pipeline-lang.png[Creating a pipeline in the Stack Management app,align="center"]
Copy file name to clipboardexpand all lines: docs/en/stack/ml/nlp/ml-nlp-ner-example.asciidoc
+1-1
Original file line number
Diff line number
Diff line change
@@ -294,7 +294,7 @@ You can create a tag cloud to visualize your data processed by the {infer}
294
294
pipeline. A tag cloud is a visualization that scales words by the frequency at
295
295
which they occur. It is a handy tool for viewing the entities found in the data.
296
296
297
-
In {kib}, open **Stack management** > **{data-sources-cap}**, and create a new
297
+
In {kib}, open **Stack management** > **{data-sources-cap}** from the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field], and create a new
298
298
{data-source} from the `les-miserables-infer` index pattern.
299
299
300
300
Open **Dashboard** and create a new dashboard. Select the
Copy file name to clipboardexpand all lines: docs/en/stack/ml/setup.asciidoc
+39-40
Original file line number
Diff line number
Diff line change
@@ -11,17 +11,16 @@
11
11
12
12
To use the {stack} {ml-features}, you must have:
13
13
14
-
[%interactive]
15
-
- [ ] the {subscriptions}[appropriate subscription] level or the free trial
14
+
- the {subscriptions}[appropriate subscription] level or the free trial
16
15
period activated
17
-
- [ ] `xpack.ml.enabled` set to its default value of `true` on every node in the
16
+
- `xpack.ml.enabled` set to its default value of `true` on every node in the
18
17
cluster (refer to {ref}/ml-settings.html[{ml-cap} settings in {es}])
19
-
- [ ] `ml` value defined in the list of `node.roles` on the
18
+
- `ml` value defined in the list of `node.roles` on the
20
19
{ref}/modules-node.html#ml-node[{ml} nodes]
21
-
- [ ] {ml} features visible in the {kib} space
22
-
- [ ] security privileges assigned to the user that:
23
-
* grant use of {ml-features}, and
24
-
* grant access to source and destination indices.
20
+
- {ml} features visible in the {kib} space
21
+
- security privileges assigned to the user that:
22
+
* grant use of {ml-features}, and
23
+
* grant access to source and destination indices.
25
24
26
25
TIP: The fastest way to get started with {ml-features} is to
27
26
{ess-trial}[start a free 14-day trial of {ess}] in the cloud.
@@ -39,12 +38,15 @@ the two main categories:
39
38
* *<<kib-security-privileges>>*: uses the {ml-features} in {kib} and does not
40
39
use Dev Tools. It requires either {kib} feature privileges or {es} security
41
40
privileges and is granted the most permissive combination of both. {kib} feature
42
-
privileges are recommended if you control job level visibility via _Spaces_.
41
+
privileges are recommended if you control job level visibility via **Spaces**.
43
42
{ml-cap} features must be visible in the relevant space. Refer to
44
43
<<kib-visibility-spaces>> for configuration information.
45
44
46
-
You can configure these privileges under **{stack-manage-app}** > _Security_ in
47
-
{kib} or via the respective {es} security APIs.
45
+
You can configure these privileges
46
+
47
+
- under **Security**. To open Security, find **{stack-manage-app}** in the main menu or
48
+
use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field].
49
+
- via the respective {es} security APIs.
48
50
49
51
50
52
[discrete]
@@ -55,19 +57,17 @@ If you use {ml} APIs, you must have the following cluster and index privileges:
55
57
56
58
For full access:
57
59
58
-
[%interactive]
59
-
* [ ] `machine_learning_admin` built-in role or the equivalent cluster
60
+
* `machine_learning_admin` built-in role or the equivalent cluster
60
61
privileges
61
-
* [ ] `read` and `view_index_metadata` on source indices
62
-
* [ ] `read`, `manage`, and `index` on destination indices (for
62
+
* `read` and `view_index_metadata` on source indices
63
+
* `read`, `manage`, and `index` on destination indices (for
63
64
{dfanalytics-jobs} only)
64
65
65
66
For read-only access:
66
67
67
-
[%interactive]
68
-
* [ ] `machine_learning_user` built-in role or the equivalent cluster privileges
69
-
* [ ] `read` index privileges on source indices
70
-
* [ ] `read` index privileges on destination indices (for {dfanalytics-jobs}
68
+
* `machine_learning_user` built-in role or the equivalent cluster privileges
69
+
* `read` index privileges on source indices
70
+
* `read` index privileges on destination indices (for {dfanalytics-jobs}
71
71
only)
72
72
73
73
IMPORTANT: The `machine_learning_admin` and `machine_learning_user` built-in
@@ -92,19 +92,21 @@ visualizations as well as {ml} job, trained model and module saved objects.
92
92
93
93
In {kib}, the {ml-features} must be visible in your
94
94
{kibana-ref}/xpack-spaces.html#spaces-control-feature-visibility[space]. To
95
-
control which features are visible in your space, use **{stack-manage-app}** >
96
-
_{kib}_ > _Spaces_.
95
+
manage which features are visible in your space, go to **{stack-manage-app}** >
96
+
**{kib}** > **Spaces** or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field]
97
+
to locate **Spaces** directly.
97
98
98
99
[role="screenshot"]
99
100
image::spaces.jpg["Manage spaces in {kib}"]
100
101
101
102
In addition to index privileges, source {data-sources} must also exist in the
102
-
same space as your {ml} jobs. These can be configured in **{stack-manage-app}**
103
-
> _{kib}_ > _{data-sources-caps}_.
103
+
same space as your {ml} jobs. You can configure these under **{data-sources-caps}**. To open **{data-sources-caps}**,
104
+
find **{stack-manage-app}** > **{kib}** in the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field].
104
105
105
106
106
107
Each {ml} job and trained model can be assigned to all, one, or multiple spaces.
107
-
This can be configured in **{stack-manage-app} > Alerts and Insights > Machine Learning**.
108
+
This can be configured in **Machine Learning**. To open **Machine Learning**, find **{stack-manage-app} > Alerts and Insights** in the main menu,
109
+
or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field].
108
110
You can edit the spaces that a job or model is assigned to by clicking the
109
111
icons in the **Spaces** column.
110
112
@@ -118,22 +120,20 @@ image::assign-job-spaces.jpg["Assign machine learning jobs to spaces"]
118
120
119
121
Within a {kib} space, for full access to the {ml-features}, you must have:
![mergify[bot]](https://avatars.githubusercontent.com/in/10562?v=4&size=40){kind=avatar}
0 commit comments