[axonius][user] Add Axonius User datastream

packages/axonius/_dev/build/docs/README.md

@@ -17,11 +17,15 @@ This integration collects log messages of the following type:
 
 - `Adapter`: Collect details of all adapters (endpoint: `/api/v2/adapters`).
 
+- `User`: Collect details of all users (endpoint: `/api/v2/users`).
+
 ### Supported use cases
 
-Integrating the Axonius Adapters with Elastic SIEM provides clear visibility into adapter health and data-collection performance across the environment. The dashboard highlights overall adapter status, offering a quick understanding of which integrations are functioning normally and which require attention. Essential adapter details are surfaced to help analysts validate configurations, identify failing plugins, and understand the distribution of adapters across nodes.
+Integrating the Axonius Adapter and User Data streams with Elastic SIEM provides centralized visibility into both data-collection health and user identity context across the environment. Together, these data streams help analysts understand how data is being ingested through adapters and how that data maps to user identities and access posture.
+
+The dashboards highlight overall adapter status and connection behavior, making it easy to identify healthy integrations, failing plugins, and error-prone connections that may impact asset visibility. At the same time, user-focused views surface role distribution and essential identity attributes, helping analysts quickly assess access patterns and identify high-privileged or unusual user activity. Consolidated user details and source information provide clarity on where identity data originates and whether coverage gaps exist.
 
-It also provides insight into connection behavior for each adapter, revealing patterns of active, inactive, and error-prone connections. Error-specific views make it easy to spot problematic integrations and prioritize troubleshooting efforts. These insights enable teams to maintain reliable data ingestion, reduce blind spots, and ensure complete and accurate asset collection across all connected systems.
+By combining adapter health insights with user identity visibility, security teams can ensure reliable data ingestion, detect identity-related anomalies, reduce blind spots, and streamline investigations that depend on accurate, end-to-end context from both integrations and users.
 
 ## What do I need to use this integration?
 
@@ -39,7 +43,13 @@ To collect data through the Axonius APIs, you need to provide the **URL**, **API
 2. Your instance URL is your Base **URL**.
 3. Navigate to **User Settings > API Key**.
 4. Generate an **API Key**.
-5. Copy both values including **API Key and Secret Key** and store them securely for use in the Integration configuration.
+5. If you do not see the API Key tab in your user settings, follow these steps:
+    1.  Go to **System Settings** > **User and Role Management** > **Service Accounts**.
+    2. Create a Service Account, and then generate an **API Key**.
+6. Copy both values including **API Key and Secret Key** and store them securely for use in the Integration configuration.
+
+**Note:**
+To generate or reset an API key, your role must be **Admin**, and you must have **API Access** permissions, which include **API Access Enabled** and **Reset API Key**.
 
 ## How do I deploy this integration?
 
@@ -106,6 +116,16 @@ The `adapter` data stream provides adapter logs from axonius.
 
 {{ event "adapter" }}
 
+### User
+
+The `user` data stream provides user events from axonius.
+
+#### user fields
+
+{{ fields "user" }}
+
+{{ event "user" }}
+
 ### Inputs used
 {{/* All inputs used by this package will be automatically listed here. */}}
 {{ inputDocs }}
@@ -115,7 +135,8 @@ The `adapter` data stream provides adapter logs from axonius.
 These APIs are used with this integration:
 
 * Adapter (endpoint: `/api/v2/adapters`)
+* User (endpoint: `/api/v2/users`)
 
-#### ILM Policy
+### ILM Policy
 
-To facilitate adapter data, source data stream-backed indices `.ds-logs-axonius.adapter-*` are allowed to contain duplicates from each polling interval. ILM policy `logs-axonius.adapter-default_policy` is added to these source indices, so it doesn't lead to unbounded growth. This means that in these source indices data will be deleted after `30 days` from ingested date.
+To facilitate adapter and user data, source data stream-backed indices `.ds-logs-axonius.adapter-*` and `.ds-logs-axonius.user-*` respectively are allowed to contain duplicates from each polling interval. ILM policies `logs-axonius.adapter-default_policy` and `logs-axonius.user-default_policy` are added to these source indices, so it doesn't lead to unbounded growth. This means that in these source indices data will be deleted after `30 days` from ingested date.

packages/axonius/_dev/deploy/docker/files/config.yml

@@ -97,3 +97,61 @@ rules:
             ]
           }
           `}}
+  - path: /api/v2/users
+    methods: ['GET']
+    request_headers:
+      Accept: application/json
+      api-key: xxxx
+      api-secret: xxxx
+    responses:
+      - status_code: 200
+        headers:
+          Content-Type:
+            - 'application/json'
+        body: |-
+          {{ minify_json `
+          {
+            "users": [
+              {
+                "role_id": "63622vfed93d274d9489dbbgresdcv6a1cf",
+                "data_scope_id": "fgreg63622d93d274d9489db6a1cf",
+                "allowed_scopes_impersonation": [
+                  "63622d93d27cvdsfa4d9489db6a1cf",
+                  "63622d93d2dvfwe74d9489db6a1cc"
+                ],
+                "first_name": "alias",
+                "last_name": "doe",
+                "email": "[email protected]",
+                "title": "Security Analyst",
+                "department": "test",
+                "user_name": "alias.doe",
+                "last_updated": "Sun, 11 Mar 2025 18:53:09 GMT",
+                "source": "test source",
+                "uuid": "63622d93d274ihvbngvbhd9489db6a1cf",
+                "role_name": "test role",
+                "data_scope_name": "test data scope",
+                "last_login": "Sun, 09 Mar 2025 18:53:09 GMT"
+              },
+              {
+                "role_id": "63622d93d27xcvdgre4d9489dbbgresdcv6a1cf",
+                "data_scope_id": "fgreghtgrtefdgrt63622d93d274d9489db6a1cf",
+                "allowed_scopes_impersonation": [
+                  "63622d93d27vdfrew4d9489db6a1cf",
+                  "63622d93d27fvdwe4d9489db6a1cc"
+                ],
+                "first_name": "bob",
+                "last_name": "doe",
+                "email": "[email protected]",
+                "title": "Cloud Operations Engineer",
+                "department": "sample",
+                "user_name": "bob.doe",
+                "last_updated": "Mon, 12 Mar 2025 18:53:09 GMT",
+                "source": "sample source",
+                "uuid": "78ij63622d93d274d9489db6a1cf",
+                "role_name": " role",
+                "data_scope_name": "sample data scope",
+                "last_login": "Mon, 08 Mar 2025 18:53:09 GMT"
+              }
+            ]
+          }
+          `}}

packages/axonius/changelog.yml

@@ -3,4 +3,4 @@
   changes:
     - description: Initial release.
       type: enhancement
-      link: https://github.com/elastic/integrations/pull/16171
+      link: https://github.com/elastic/integrations/pull/16142

packages/axonius/data_stream/user/_dev/test/pipeline/test-common-config.yml

@@ -0,0 +1,3 @@
+fields:
+  tags:
+    - preserve_duplicate_custom_fields

packages/axonius/data_stream/user/_dev/test/pipeline/test-user.log

@@ -0,0 +1,3 @@
+{"role_id":"63622d93d274d9489dbbgresdcv6a1cf","data_scope_id":"fgreg63622d93d274d9489db6a1cf","allowed_scopes_impersonation":["63622d93d274d9489db6a1cf","63622d93d274d9489db6a1cc"],"first_name":"john","last_name":"doe","email":"[email protected]","title":"Software Engineer","department":"example","user_name":"john.doe","last_updated":"Sat, 10 Mar 2025 18:53:09 GMT","source":"example source","uuid":"63622d93d274d9489db6a1cf","role_name":"example role","data_scope_name":"example data scope","last_login":"Sat, 08 Mar 2025 18:53:09 GMT"}
+{"role_id":"63622vfed93d274d9489dbbgresdcv6a1cf","data_scope_id":"fgreg63622d93d274d9489db6a1cf","allowed_scopes_impersonation":["63622d93d27cvdsfa4d9489db6a1cf","63622d93d2dvfwe74d9489db6a1cc"],"first_name":"alias","last_name":"doe","email":"[email protected]","title":"Security Analyst","department":"test","user_name":"alias.doe","last_updated":"Sun, 11 Mar 2025 18:53:09 GMT","source":"test source","uuid":"63622d93d274ihvbngvbhd9489db6a1cf","role_name":"test role","data_scope_name":"test data scope","last_login":"Sun, 09 Mar 2025 18:53:09 GMT"}
+{"role_id":"63622d93d27xcvdgre4d9489dbbgresdcv6a1cf","data_scope_id":"fgreghtgrtefdgrt63622d93d274d9489db6a1cf","allowed_scopes_impersonation":["63622d93d27vdfrew4d9489db6a1cf","63622d93d27fvdwe4d9489db6a1cc"],"first_name":"bob","last_name":"doe","email":"[email protected]","title":"Cloud Operations Engineer","department":"sample","user_name":"bob.doe","last_updated":"Mon, 12 Mar 2025 18:53:09 GMT","source":"sample source","uuid":"78ij63622d93d274d9489db6a1cf","role_name":" role","data_scope_name":"sample data scope","last_login":"Mon, 08 Mar 2025 18:53:09 GMT"}

packages/axonius/data_stream/user/_dev/test/pipeline/test-user.log-expected.json

@@ -0,0 +1,160 @@
+{
+    "expected": [
+        {
+            "axonius": {
+                "user": {
+                    "allowed_scopes_impersonation": [
+                        "63622d93d274d9489db6a1cf",
+                        "63622d93d274d9489db6a1cc"
+                    ],
+                    "data_scope_id": "fgreg63622d93d274d9489db6a1cf",
+                    "data_scope_name": "example data scope",
+                    "department": "example",
+                    "email": "[email protected]",
+                    "first_name": "john",
+                    "last_login": "2025-03-08T18:53:09.000Z",
+                    "last_name": "doe",
+                    "last_updated": "2025-03-10T18:53:09.000Z",
+                    "role_id": "63622d93d274d9489dbbgresdcv6a1cf",
+                    "role_name": "example role",
+                    "source": "example source",
+                    "title": "Software Engineer",
+                    "user_name": "john.doe",
+                    "uuid": "63622d93d274d9489db6a1cf"
+                }
+            },
+            "ecs": {
+                "version": "9.2.0"
+            },
+            "event": {
+                "kind": "event",
+                "original": "{\"role_id\":\"63622d93d274d9489dbbgresdcv6a1cf\",\"data_scope_id\":\"fgreg63622d93d274d9489db6a1cf\",\"allowed_scopes_impersonation\":[\"63622d93d274d9489db6a1cf\",\"63622d93d274d9489db6a1cc\"],\"first_name\":\"john\",\"last_name\":\"doe\",\"email\":\"[email protected]\",\"title\":\"Software Engineer\",\"department\":\"example\",\"user_name\":\"john.doe\",\"last_updated\":\"Sat, 10 Mar 2025 18:53:09 GMT\",\"source\":\"example source\",\"uuid\":\"63622d93d274d9489db6a1cf\",\"role_name\":\"example role\",\"data_scope_name\":\"example data scope\",\"last_login\":\"Sat, 08 Mar 2025 18:53:09 GMT\"}"
+            },
+            "related": {
+                "user": [
+                    "john doe",
+                    "[email protected]",
+                    "john.doe",
+                    "63622d93d274d9489db6a1cf"
+                ]
+            },
+            "tags": [
+                "preserve_duplicate_custom_fields"
+            ],
+            "user": {
+                "domain": "example.com",
+                "email": "[email protected]",
+                "full_name": "john doe",
+                "id": "63622d93d274d9489db6a1cf",
+                "name": "john.doe",
+                "roles": [
+                    "example role"
+                ]
+            }
+        },
+        {
+            "axonius": {
+                "user": {
+                    "allowed_scopes_impersonation": [
+                        "63622d93d27cvdsfa4d9489db6a1cf",
+                        "63622d93d2dvfwe74d9489db6a1cc"
+                    ],
+                    "data_scope_id": "fgreg63622d93d274d9489db6a1cf",
+                    "data_scope_name": "test data scope",
+                    "department": "test",
+                    "email": "[email protected]",
+                    "first_name": "alias",
+                    "last_login": "2025-03-09T18:53:09.000Z",
+                    "last_name": "doe",
+                    "last_updated": "2025-03-11T18:53:09.000Z",
+                    "role_id": "63622vfed93d274d9489dbbgresdcv6a1cf",
+                    "role_name": "test role",
+                    "source": "test source",
+                    "title": "Security Analyst",
+                    "user_name": "alias.doe",
+                    "uuid": "63622d93d274ihvbngvbhd9489db6a1cf"
+                }
+            },
+            "ecs": {
+                "version": "9.2.0"
+            },
+            "event": {
+                "kind": "event",
+                "original": "{\"role_id\":\"63622vfed93d274d9489dbbgresdcv6a1cf\",\"data_scope_id\":\"fgreg63622d93d274d9489db6a1cf\",\"allowed_scopes_impersonation\":[\"63622d93d27cvdsfa4d9489db6a1cf\",\"63622d93d2dvfwe74d9489db6a1cc\"],\"first_name\":\"alias\",\"last_name\":\"doe\",\"email\":\"[email protected]\",\"title\":\"Security Analyst\",\"department\":\"test\",\"user_name\":\"alias.doe\",\"last_updated\":\"Sun, 11 Mar 2025 18:53:09 GMT\",\"source\":\"test source\",\"uuid\":\"63622d93d274ihvbngvbhd9489db6a1cf\",\"role_name\":\"test role\",\"data_scope_name\":\"test data scope\",\"last_login\":\"Sun, 09 Mar 2025 18:53:09 GMT\"}"
+            },
+            "related": {
+                "user": [
+                    "alias doe",
+                    "[email protected]",
+                    "alias.doe",
+                    "63622d93d274ihvbngvbhd9489db6a1cf"
+                ]
+            },
+            "tags": [
+                "preserve_duplicate_custom_fields"
+            ],
+            "user": {
+                "domain": "example.com",
+                "email": "[email protected]",
+                "full_name": "alias doe",
+                "id": "63622d93d274ihvbngvbhd9489db6a1cf",
+                "name": "alias.doe",
+                "roles": [
+                    "test role"
+                ]
+            }
+        },
+        {
+            "axonius": {
+                "user": {
+                    "allowed_scopes_impersonation": [
+                        "63622d93d27vdfrew4d9489db6a1cf",
+                        "63622d93d27fvdwe4d9489db6a1cc"
+                    ],
+                    "data_scope_id": "fgreghtgrtefdgrt63622d93d274d9489db6a1cf",
+                    "data_scope_name": "sample data scope",
+                    "department": "sample",
+                    "email": "[email protected]",
+                    "first_name": "bob",
+                    "last_login": "2025-03-08T18:53:09.000Z",
+                    "last_name": "doe",
+                    "last_updated": "2025-03-12T18:53:09.000Z",
+                    "role_id": "63622d93d27xcvdgre4d9489dbbgresdcv6a1cf",
+                    "role_name": " role",
+                    "source": "sample source",
+                    "title": "Cloud Operations Engineer",
+                    "user_name": "bob.doe",
+                    "uuid": "78ij63622d93d274d9489db6a1cf"
+                }
+            },
+            "ecs": {
+                "version": "9.2.0"
+            },
+            "event": {
+                "kind": "event",
+                "original": "{\"role_id\":\"63622d93d27xcvdgre4d9489dbbgresdcv6a1cf\",\"data_scope_id\":\"fgreghtgrtefdgrt63622d93d274d9489db6a1cf\",\"allowed_scopes_impersonation\":[\"63622d93d27vdfrew4d9489db6a1cf\",\"63622d93d27fvdwe4d9489db6a1cc\"],\"first_name\":\"bob\",\"last_name\":\"doe\",\"email\":\"[email protected]\",\"title\":\"Cloud Operations Engineer\",\"department\":\"sample\",\"user_name\":\"bob.doe\",\"last_updated\":\"Mon, 12 Mar 2025 18:53:09 GMT\",\"source\":\"sample source\",\"uuid\":\"78ij63622d93d274d9489db6a1cf\",\"role_name\":\" role\",\"data_scope_name\":\"sample data scope\",\"last_login\":\"Mon, 08 Mar 2025 18:53:09 GMT\"}"
+            },
+            "related": {
+                "user": [
+                    "bob doe",
+                    "[email protected]",
+                    "bob.doe",
+                    "78ij63622d93d274d9489db6a1cf"
+                ]
+            },
+            "tags": [
+                "preserve_duplicate_custom_fields"
+            ],
+            "user": {
+                "domain": "example.com",
+                "email": "[email protected]",
+                "full_name": "bob doe",
+                "id": "78ij63622d93d274d9489db6a1cf",
+                "name": "bob.doe",
+                "roles": [
+                    " role"
+                ]
+            }
+        }
+    ]
+}

packages/axonius/data_stream/user/_dev/test/system/test-default-config.yml

@@ -0,0 +1,12 @@
+input: cel
+service: axonius
+vars:
+  url: http://{{Hostname}}:{{Port}}
+  api_key: xxxx
+  secret_key: xxxx
+data_stream:
+  vars:
+    preserve_original_event: true
+    preserve_duplicate_custom_fields: true
+assert:
+  hit_count: 2

packages/axonius/data_stream/user/agent/stream/cel.yml.hbs

@@ -0,0 +1,79 @@
+config_version: 2
+interval: {{interval}}
+resource.tracer:
+  enabled: {{enable_request_tracer}}
+  filename: "../../logs/cel/http-request-trace-*.ndjson"
+  maxbackups: 5
+{{#if proxy_url}}
+resource.proxy_url: {{proxy_url}}
+{{/if}}
+{{#if ssl}}
+resource.ssl: {{ssl}}
+{{/if}}
+{{#if http_client_timeout}}
+resource.timeout: {{http_client_timeout}}
+{{/if}}
+resource.url: {{url}}
+
+state:
+  api_key: {{api_key}}
+  secret_key: {{secret_key}}
+redact:
+  fields:
+    - api_key
+    - secret_key
+program: |
+  request(
+    "GET",
+    state.url.trim_right("/") + "/api/v2/users"
+  ).with({
+    "Header":{
+        "Accept": ["application/json"],
+        "api-key": [state.api_key],
+        "api-secret": [state.secret_key],
+    }
+  }).do_request().as(resp, resp.StatusCode == 200 ?
+    resp.Body.decode_json().as(body,
+      {
+        "events": body.?users.orValue([]).map(e, {
+          "message": e.encode_json(),
+        }),
+        "api_key": state.api_key,
+        "secret_key": state.secret_key,
+      }
+    )
+  :
+    {
+      "events": {
+        "error": {
+          "code": string(resp.StatusCode),
+          "id": string(resp.Status),
+          "message": "GET" + state.url.trim_right("/") + "/api/v2/users/: " + (
+            size(resp.Body) != 0 ?
+              string(resp.Body)
+            :
+              string(resp.Status) + ' (' + string(resp.StatusCode) + ')'
+          ),
+        },
+      },
+      "api_key": state.api_key,
+      "secret_key": state.secret_key,
+    }
+  )
+tags:
+{{#if preserve_original_event}}
+  - preserve_original_event
+{{/if}}
+{{#if preserve_duplicate_custom_fields}}
+  - preserve_duplicate_custom_fields
+{{/if}}
+{{#each tags as |tag|}}
+  - {{tag}}
+{{/each}}
+{{#contains "forwarded" tags}}
+publisher_pipeline.disable_host: true
+{{/contains}}
+{{#if processors}}
+processors:
+{{processors}}
+{{/if}}