Skip to content

Commit 32389c6

Browse files
cwillumsbcd90
authored and
vagimeli
committed
Add documentation for OCSF field mapping and correlation engine API (#4549)
* fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * add missing param descriptions (#4555) Signed-off-by: Subhobrata Dey <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * Revert "fix#4500 ocsf fields and api" This reverts commit c6db296. merge conflict with same edits on another branch# * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * Revert "fix#4500 ocsf fields and api" This reverts commit 7a70dc3. merge conflict with second branch :wq * Revert "fix#4500 ocsf fields and api" This reverts commit ef13a74. * fix#4500 edits post merge conflict Signed-off-by: cwillum <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> --------- Signed-off-by: cwillum <[email protected]> Signed-off-by: Subhobrata Dey <[email protected]> Co-authored-by: Subhobrata Dey <[email protected]>
1 parent a63b578 commit 32389c6

File tree

7 files changed

+248
-2
lines changed

7 files changed

+248
-2
lines changed

‎_security-analytics/api-tools/correlation-eng.md

Lines changed: 224 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,224 @@
1+
---
2+
layout: default
3+
title: Correlation engine APIs
4+
parent: API tools
5+
nav_order: 55
6+
---
7+
8+
9+
# Correlation engine APIs
10+
11+
Correlation engine APIs allow you to create new correlation rules, view findings and correlations within a certain time window, and perform other tasks.
12+
13+
14+
## Create correlation rules between log types
15+
16+
This API is used to create correlation rules:
17+
18+
```json
19+
POST /_plugins/_security_analytics/correlation/rules
20+
```
21+
22+
### Request fields
23+
24+
| Field | Type | Description |
25+
| :--- | :--- |:--- |
26+
| `index` | String | The name of the index used as the log source. |
27+
| `query` | String | The query used to filter security logs for correlation. |
28+
| `category` | String | The log type associated with the log source. |
29+
30+
#### Example request
31+
32+
```json
33+
POST /_plugins/_security_analytics/correlation/rules
34+
{
35+
"correlate": [
36+
{
37+
"index": "vpc_flow",
38+
"query": "dstaddr:4.5.6.7 or dstaddr:4.5.6.6",
39+
"category": "network"
40+
},
41+
{
42+
"index": "windows",
43+
"query": "winlog.event_data.SubjectDomainName:NTAUTHORI*",
44+
"category": "windows"
45+
},
46+
{
47+
"index": "ad_logs",
48+
"query": "ResultType:50126",
49+
"category": "ad_ldap"
50+
},
51+
{
52+
"index": "app_logs",
53+
"query": "endpoint:/customer_records.txt",
54+
"category": "others_application"
55+
}
56+
]
57+
}
58+
```
59+
{% include copy-curl.html %}
60+
61+
#### Example response
62+
63+
```json
64+
{
65+
"_id": "DxKEUIkBpIjg64IK4nXg",
66+
"_version": 1,
67+
"rule": {
68+
"name": null,
69+
"correlate": [
70+
{
71+
"index": "vpc_flow",
72+
"query": "dstaddr:4.5.6.7 or dstaddr:4.5.6.6",
73+
"category": "network"
74+
},
75+
{
76+
"index": "windows",
77+
"query": "winlog.event_data.SubjectDomainName:NTAUTHORI*",
78+
"category": "windows"
79+
},
80+
{
81+
"index": "ad_logs",
82+
"query": "ResultType:50126",
83+
"category": "ad_ldap"
84+
},
85+
{
86+
"index": "app_logs",
87+
"query": "endpoint:/customer_records.txt",
88+
"category": "others_application"
89+
}
90+
]
91+
}
92+
}
93+
```
94+
95+
### Response fields
96+
97+
| Field | Type | Description |
98+
| :--- | :--- |:--- |
99+
| `_id` | String | The Id for the new rule. |
100+
101+
102+
## List all findings and their correlations within a time window
103+
104+
This API provides a list of all findings and their correlations within a specified time window:
105+
106+
```json
107+
GET /_plugins/_security_analytics/correlations?start_timestamp=<start time in milliseconds>&end_timestamp=<end time in milliseconds>
108+
```
109+
110+
### Query parameters
111+
112+
| Parameter | Type | Description |
113+
| :--- | :--- |:--- |
114+
| `start_timestamp` | Number | Start time for the time window, in milliseconds. |
115+
| `end_timestamp` | Number | End time for the time window, in milliseconds. |
116+
117+
#### Example request
118+
119+
```json
120+
GET /_plugins/_security_analytics/correlations?start_timestamp=1689289210000&end_timestamp=1689300010000
121+
```
122+
{% include copy-curl.html %}
123+
124+
#### Example response
125+
126+
```json
127+
{
128+
"findings": [
129+
{
130+
"finding1": "931de5f0-a276-45d5-9cdb-83e1045a3630",
131+
"logType1": "network",
132+
"finding2": "1e6f6a12-83f1-4a38-9bb8-648f196859cc",
133+
"logType2": "test_windows",
134+
"rules": [
135+
"nqI2TokBgL5wWFPZ6Gfu"
136+
]
137+
}
138+
]
139+
}
140+
```
141+
142+
### Response fields
143+
144+
| Field | Type | Description |
145+
| :--- | :--- |:--- |
146+
| `finding1` | String | The Id for a first finding in the correlation. |
147+
| `logType1` | String | The log type associated with the first finding. |
148+
| `finding2` | String | The Id for a second finding in the correlation. |
149+
| `logType2` | String | The log type associated with the second finding. |
150+
| `rules` | Array | A list of correlation rule IDs associated with the correlated findings. |
151+
152+
153+
## List correlations for a finding belonging to a log type
154+
155+
This API is used to list correlations for specific findings and the log types associated with them:
156+
157+
```json
158+
GET /_plugins/_security_analytics/findings/correlate?finding=425dce0b-f5ee-4889-b0c0-7d15669f0871&detector_type=ad_ldap&nearby_findings=20&time_window=10m
159+
```
160+
161+
### Query parameters
162+
163+
| Parameter | Type | Description |
164+
| :--- | :--- |:--- |
165+
| `finding` | String | The finding ID. |
166+
| `detector_type` | String | The log type for the detector. |
167+
| `nearby_findings` | Number | The number of nearby findings with respect to the given finding Id. |
168+
| `time_window` | String | Sets a time window in which all of the correlations must have occurred together. |
169+
170+
171+
#### Example request
172+
173+
```json
174+
GET /_plugins/_security_analytics/findings/correlate?finding=425dce0b-f5ee-4889-b0c0-7d15669f0871&detector_type=ad_ldap&nearby_findings=20&time_window=10m
175+
```
176+
{% include copy-curl.html %}
177+
178+
#### Example response
179+
180+
```json
181+
{
182+
"findings": [
183+
{
184+
"finding": "5c661104-aaa9-484b-a91f-9cad4ae6d5f5",
185+
"detector_type": "others_application",
186+
"score": 0.000015182109564193524
187+
},
188+
{
189+
"finding": "2485b623-6573-42f4-a055-9b927e38a65f",
190+
"detector_type": "ad_ldap",
191+
"score": 0.000001615897872397909
192+
},
193+
{
194+
"finding": "051e00ad-5996-4c41-be20-f992451d1331",
195+
"detector_type": "windows",
196+
"score": 0.000016230604160227813
197+
},
198+
{
199+
"finding": "f11ca8a3-50d7-4074-a951-51439aa9e67b",
200+
"detector_type": "s3",
201+
"score": 0.000001759401811796124
202+
},
203+
{
204+
"finding": "9b86980e-5fb7-4c5a-bd1b-879a1e3baf12",
205+
"detector_type": "network",
206+
"score": 0.0000016306962606904563
207+
},
208+
{
209+
"finding": "e7dea5a1-164f-48f9-880e-4ba33e508713",
210+
"detector_type": "network",
211+
"score": 0.00001632626481296029
212+
}
213+
]
214+
}
215+
```
216+
217+
### Response fields
218+
219+
| Field | Type | Description |
220+
| :--- | :--- |:--- |
221+
| `finding` | String | The finding ID. |
222+
| `detector_type` | String | The log type associated with the finding. |
223+
| `score` | Number | The correlation score for the correlated finding. The score is based on the proximity of relevant findings in the threat scenario defined by the correlation rule. |
224+

‎_security-analytics/api-tools/detector-api.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ Field | Type | Description
2626
`enabled` | Boolean | Sets the detector as either active (true) or inactive (false). Default is `true` when a new detector is created. Required.
2727
`name` | String | Name of the detector. Name should only consist of upper and lowercase letters, numbers 0-9, hyphens, spaces, and underscores. Use between 5 and 50 characters. Required.
2828
`detector_type` | String | The log type that defines the detector. Options are `linux`, `network` ,`windows`, `ad_ldap`, `apache_access`, `cloudtrail`, `dns`, and `s3`. Required.
29-
`schedule` | Object | The schedule that determines how often the detector runs. For information on specifying fixed intervals in the API, see [Cron expression reference]({{site.url}}{{site.baseurl}}/monitoring-plugins/alerting/cron/).
29+
`schedule` | Object | The schedule that determines how often the detector runs. For information about specifying fixed intervals in the API, see the [Cron expression reference]({{site.url}}{{site.baseurl}}/monitoring-plugins/alerting/cron/).
3030
`schedule.period` | Object | Details for the frequency of the schedule.
3131
`schedule.period.interval` | Integer | The interval at which the detector runs.
3232
`schedule.period.unit` | String | The interval's unit of time.

‎_security-analytics/api-tools/index.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,4 +18,5 @@ The APIs for Security Analytics are separated into the following categories:
1818
* [Rules APIs]({{site.url}}{{site.baseurl}}/security-analytics/api-tools/rule-api/)
1919
* [Mappings APIs]({{site.url}}{{site.baseurl}}/security-analytics/api-tools/mappings-api/)
2020
* [Alerts and findings APIs]({{site.url}}{{site.baseurl}}/security-analytics/api-tools/alert-finding-api/)
21+
* [Correlation engine APIs]({{site.url}}{{site.baseurl}}/security-analytics/api-tools/correlation-eng/)
2122

‎_security-analytics/api-tools/mappings-api.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,18 @@ The following APIs can be used for a number of tasks related to mappings, from c
1111

1212
## Get Mappings View
1313

14+
15+
this API returns a view of the fields contained in an index used as a log source.
16+
17+
### Request fields
18+
19+
The following fields are used to get field mappings.
20+
21+
Field | Type | Description
22+
:--- | :--- |:--- |
23+
| `index_name` | String | The name of the index used for log ingestion. |
24+
| `rule_topic` | String | The log type of the index. |
25+
1426
### Example request
1527

1628
```json

‎_security-analytics/api-tools/rule-api.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ The following APIs can be used for a number of tasks related to rules, from sear
1111

1212
## Create Custom Rule
1313

14-
The Create custom rule API uses Sigma security rule formatting to create a custom rule. For information on how to write a rule in Sigma format, see information provided at [Sigma's GitHub repository](https://github.com/SigmaHQ/sigma).
14+
The Create Custom Rule API uses Sigma security rule formatting to create a custom rule. For information about how to write a rule in Sigma format, see information provided at [Sigma's GitHub repository](https://github.com/SigmaHQ/sigma).
1515

1616
```json
1717
POST /_plugins/_security_analytics/rules?category=windows

‎_security-analytics/sec-analytics-config/detectors-config.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,15 @@ Security Analytics takes advantage of prepackaged Sigma rules for security event
6060

6161
Although the ECS rule field names are largely self-explanatory, you can find predefined mappings of the Sigma rule field names to ECS rule field names, for all supported log types, in the GitHub Security Analytics repository. Navigate to the [OSMappings](https://github.com/opensearch-project/security-analytics/tree/main/src/main/resources/OSMapping) folder, choose the folder named for the log type, and open the `fieldmappings.yml` file. For example, to see the Sigma rule fields that correspond to ECS rule fields for the Windows log type, open the [fieldmappings.yml file](https://github.com/opensearch-project/security-analytics/blob/main/src/main/resources/OSMapping/windows/fieldmappings.yml) in the **windows** folder.
6262

63+
#### Amazon Security Lake logs
64+
65+
[Amazon Security Lake](https://docs.aws.amazon.com/security-lake/latest/userguide/what-is-security-lake.html) converts security log and event data to the [Open Cybersecurity Schema Framework](https://docs.aws.amazon.com/security-lake/latest/userguide/open-cybersecurity-schema-framework.html) (OCSF) to normalize combined data and facilitate its management. OpenSearch supports ingestion of log data from Security Lake in the OCSF format, and Security Analytics can automatically map fields from OCSF to ECS (the default field-mapping schema).
66+
67+
The Security Lake log types that can be used as log sources for detector creation include CloudTrail, Route 53, and VPC Flow. Given that Route 53 is a log that captures DNS activity, its log type should be specified as **DNS logs** when [defining a detector](#step-1-define-a-detector). Furthermore, because logs such as CloudTrail can conceivably be captured in both raw format and OCSF, it's good practice to name indexes in a way that keeps these logs separate and easily identifiable. This becomes helpful when specifying an index name in any of the APIs associated with Security Analytics.
68+
69+
To reveal fields for a log index in either raw format or OCSF, use the [Get Mappings View]({{site.url}}{{site.baseurl}}/security-analytics/api-tools/mappings-api/#get-mappings-view) API and specify the index in the `index_name` field of the request.
70+
{: .tip }
71+
6372
### Automatically mapped fields
6473

6574
Once you select a data source and log type, the system attempts to automatically map fields between the log and rule fields. Expand **Automatically mapped fields** to show the list of these mappings. When the field names are similar to one another, the system can successfully match the two, as shown in the following image.

‎images/Security/detector_rules.png

10.5 KB
Loading

0 commit comments

Comments
 (0)