google_scc: Add support for Security Command Center API v2 and update finding data stream to support Cloud Detection and Response (CDR) workflow

Some fields inside "google_scc.finding.resource" are moved into "google_scc.finding.resource.gcp_metadata"
as per v2 API schema. This change allows to choose between v1 and v2 (recommended) APIs to fetch
findings. Location-based findings can be fetched to support Security Command Center data
residency feature. Field "google_scc.finding.vulnerability.cve.cvssv3.base_score" data
type is updated to "double" to better suit data. In dashboards, ECS fields are used
in preference to custom fields.

Added "do_not_log_failure: true" to the asset and source data streams to prevent logging of expected
pagination end failures, which have been causing system test failures from version 8.19 onwards.
This issue arose due to the addition of Fleet health status updates in the httpjson input.

Add ECS mappings and latest transform to finding data stream to help with the
Cloud Native Vulnerability Management (CNVM)[1] and Cloud Security Posture Management (CSPM)[2] workflow.

[1] https://www.elastic.co/guide/en/security/current/vuln-management-overview.html
[2] https://www.elastic.co/docs/solutions/security/cloud/cloud-security-posture-management

Author: brijesh-elastic

packages/google_scc/_dev/build/docs/README.md

@@ -89,6 +89,27 @@ If installing in GCP-Cloud Environment, No need to provide any credentials and m
       - topic
       - subscription name 
 
+## Troubleshooting
+
+### Breaking Changes
+
+#### Support for Elastic Vulnerability & Misconfiguration Findings page.
+
+Version `2.0.0` of the Google Security Command Center integration adds support for [Elastic Cloud Security workflow](https://www.elastic.co/docs/solutions/security/cloud/ingest-third-party-cloud-security-data#_ingest_third_party_security_posture_and_vulnerability_data). The enhancement enables the users of Google Security Command Center integration to ingest vulnerabilities and misconfiguration findings from Google Security Command Center platform into Elastic and get insights directly from [Vulnerability Findings page](https://www.elastic.co/docs/solutions/security/cloud/findings-page-3) and [Misconfiguration Findings page](https://www.elastic.co/docs/solutions/security/cloud/findings-page).
+Version `2.0.0` adds [Elastic Latest Transform](https://www.elastic.co/docs/explore-analyze/transforms/transform-overview#latest-transform-overview) which copies the latest findings from source indices matching the pattern `logs-google_scc.finding-*` into new destination indices matching the pattern `security_solution-google_scc.vulnerability_latest-*` and `security_solution-google_scc.misconfiguration_latest-`. The Elastic Findings pages will display findings based on the destination indices.
+
+For existing users of Google Security Command Center integration, before upgrading to version `2.0.0` please ensure following requirements are met:
+
+1. Users need [Elastic Security solution](https://www.elastic.co/docs/solutions/security) which has requirements documented [here](https://www.elastic.co/docs/solutions/security/get-started/elastic-security-requirements).
+2. To use transforms, users must have:
+   - at least one [transform node](https://www.elastic.co/docs/deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles#transform-node-role),
+   - management features visible in the Kibana space, and
+   - security privileges that:
+     - grant use of transforms, and
+     - grant access to source and destination indices
+   For more details on Transform Setup, refer to the link [here](https://www.elastic.co/docs/explore-analyze/transforms/transform-setup)
+3. Because the latest copy of findings is now indexed in two places, that is, in both source and destination indices, users must anticipate storage requirements accordingly.
+
 ## Logs reference
 
 ### Asset

packages/google_scc/_dev/deploy/docker/files/config.yml

@@ -1 +1,3 @@
 {"finding":{"name":"organizations/515665165161/sources/98481484454154454545/findings/414rfrhjebhrbhjbr444454hv54545","parent":"organizations/515665165161/sources/98481484454154454545","resourceName":"//cloudresourcemanager.googleapis.com/projects/45455445554","state":"ACTIVE","category":"application","externalUri":"http://www.adwait.com","sourceProperties":{},"securityMarks":{"name":"organizations/515665165161/sources/98481484454154454545/findings/414rfrhjebhrbhjbr444454hv54545/securityMarks"},"eventTime":"2023-06-02T05:17:41.936Z","createTime":"2020-02-19T13:37:43.858Z","severity":"CRITICAL","canonicalName":"organizations/515665165161/sources/98481484454154454545/findings/414rfrhjebhrbhjbr444454hv54545","mute":"UNMUTED","muteUpdateTime":"2022-03-23T05:50:21.804Z","externalSystems":{"test":{"name":"organizations/515665165161/sources/98481484454154454545/findings/414rfrhjebhrbhjbr444454hv54545/externalSystems/test","assignees":["primary"],"externalUid":"test_scc_finding_2","status":"updated1","externalSystemUpdateTime":"2022-01-05T05:00:35.674Z"}},"muteInitiator":"Unmuted by john@gmail.com"},"resource":{"name":"//cloudresourcemanager.googleapis.com/projects/45455445554"}}
+{"finding":{"canonicalName":"projects/101234567893/sources/01234567899876543210/locations/global/findings/452f80114abcdef0123456789abcdefd","category":"SOFTWARE_VULNERABILITY","contacts":{"security":{"contacts":[{"email":"security@example.com"}]}},"createTime":"2025-06-28T16:56:07.323Z","description":"Calling any of the Parse functions on Go source code which contains //line directives with very large line numbers can cause an infinite loop due to integer overflow.","eventTime":"2025-06-30T06:22:44.440Z","findingClass":"VULNERABILITY","mute":"UNDEFINED","muteUpdateTime":"1970-01-01T00:00:00Z","name":"organizations/901234567899/sources/01234567899876543210/locations/global/findings/452f80114abcdef0123456789abcdefd","parent":"organizations/901234567899/sources/01234567899876543210/locations/global","parentDisplayName":"Vulnerability Assessment","resourceName":"//compute.googleapis.com/projects/elastic-siem/zones/us-central1-c/instances/xyz-abcd-1234-abc-12-123","securityMarks":{"name":"organizations/901234567899/sources/01234567899876543210/locations/global/findings/452f80114abcdef0123456789abcdefd/securityMarks"},"severity":"HIGH","state":"ACTIVE","vulnerability":{"cve":{"cvssv3":{"attackComplexity":"ATTACK_COMPLEXITY_LOW","attackVector":"ATTACK_VECTOR_NETWORK","availabilityImpact":"IMPACT_HIGH","baseScore":7.5,"confidentialityImpact":"IMPACT_NONE","integrityImpact":"IMPACT_NONE","privilegesRequired":"PRIVILEGES_REQUIRED_NONE","scope":"SCOPE_UNCHANGED","userInteraction":"USER_INTERACTION_NONE"},"exploitReleaseDate":"1970-01-01T00:00:00Z","exploitationActivity":"NO_KNOWN","firstExploitationDate":"1970-01-01T00:00:00Z","id":"CVE-2023-24537","impact":"LOW","references":[{"source":"More Info","uri":"https://security-tracker.debian.org/tracker/CVE-2023-24537"},{"source":"More Info","uri":"https://access.redhat.com/security/cve/CVE-2023-24537"},{"source":"More Info","uri":"http://people.ubuntu.com/~ubuntu-security/cve/CVE-2023-24537"}],"upstreamFixAvailable":true},"fixedPackage":{"cpeUri":"cpe:/o:nvd:nvd:1.1","packageName":"go","packageType":"GO_STDLIB","packageVersion":"1.19.8"},"offendingPackage":{"cpeUri":"cpe:/o:nvd:nvd:1.1","packageName":"go","packageType":"GO_STDLIB","packageVersion":"1.13.8"},"securityBulletin":{"submissionTime":"1970-01-01T00:00:00Z"}}},"resource":{"cloudProvider":"GOOGLE_CLOUD_PLATFORM","displayName":"abc-wxyz-abcd-pqr-12-123","gcpMetadata":{"folders":[{"resourceFolder":"//cloudresourcemanager.googleapis.com/folders/801234567893","resourceFolderDisplayName":"Security"},{"resourceFolder":"//cloudresourcemanager.googleapis.com/folders/901234567891","resourceFolderDisplayName":"Engineering"},{"resourceFolder":"//cloudresourcemanager.googleapis.com/folders/901234567891","resourceFolderDisplayName":"Research and Development"}],"organization":"organizations/901234567899","parent":"//cloudresourcemanager.googleapis.com/projects/101234567893","parentDisplayName":"elastic-siem","project":"//cloudresourcemanager.googleapis.com/projects/101234567893","projectDisplayName":"elastic-siem"},"location":"us-central1-c","name":"//compute.googleapis.com/projects/elastic-siem/zones/us-central1-c/instances/abc-wxyz-abcd-pqr-12-123","resourcePath":{"nodes":[{"displayName":"elastic-siem","id":"projects/101234567893","nodeType":"GCP_PROJECT"},{"displayName":"Security","id":"folders/801234567893","nodeType":"GCP_FOLDER"},{"displayName":"Engineering","id":"folders/901234567891","nodeType":"GCP_FOLDER"},{"displayName":"Research and Development","id":"folders/901234567891","nodeType":"GCP_FOLDER"},{"id":"organizations/901234567899","nodeType":"GCP_ORGANIZATION"}]},"resourcePathString":"organizations/901234567899/folders/901234567891/folders/901234567891/folders/801234567893/projects/101234567893","service":"compute.googleapis.com","type":"google.compute.Instance"}}
+{"finding":{"canonicalName":"projects/101235645003/sources/14012345677654321025/locations/global/findings/9eaabcdef01234567890123456789147","category":"COMPUTE_SECURE_BOOT_DISABLED","contacts":{"security":{"contacts":[{"email":"security@example.com"}]}},"createTime":"2025-06-04T07:56:02.904Z","description":"Using secure boot helps protect your virtual machines against rootkits, boot- and kernel-level malware. Learn more at: https://cloud.google.com/security/shielded-cloud/shielded-vm","eventTime":"2025-06-04T07:44:08.656Z","externalUri":"https://console.cloud.google.com/compute/instancesDetail/zones/us-central1-b/instances/instance-20250528-112341?project=test-siem","findingClass":"MISCONFIGURATION","mute":"UNDEFINED","muteInfo":{"staticMute":{"applyTime":"1970-01-01T00:00:00Z","state":"UNDEFINED"}},"muteUpdateTime":"1970-01-01T00:00:00Z","name":"organizations/012345601234/sources/14012345677654321025/locations/global/findings/9eaabcdef01234567890123456789147","parent":"organizations/012345601234/sources/14012345677654321025/locations/global","parentDisplayName":"Security Health Analytics","resourceName":"//compute.googleapis.com/projects/test-siem/zones/us-central1-b/instances/instance-20250528-112341","securityMarks":{"name":"organizations/012345601234/sources/14012345677654321025/locations/global/findings/9eaabcdef01234567890123456789147/securityMarks"},"severity":"MEDIUM","sourceProperties":{"ExceptionInstructions":"Add the security mark \"allow_compute_secure_boot_disabled\" to the asset with a value of \"true\" to prevent this finding from being activated again.","Explanation":"Using secure boot helps protect your virtual machines against rootkits, boot- and kernel-level malware. Learn more at: https://cloud.google.com/security/shielded-cloud/shielded-vm","ReactivationCount":0,"Recommendation":"Go to https://console.cloud.google.com/compute/instancesDetail/zones/us-central1-b/instances/instance-20250528-112341?project=test-siem and click \"Stop\". Once the instance has stopped, scroll down to the \"Shielded VM\" section and check the \"Turn on Secure Boot\" checkbox. Then you can start the instance back up by clicking the \"Start\" button.","ResourcePath":["projects/test-siem/","folders/801234567893/","folders/901234567891/","folders/901234567890/","organizations/012345601234/"],"ScannerName":"COMPUTE_INSTANCE_SCANNER"},"state":"ACTIVE"},"resource":{"cloudProvider":"GOOGLE_CLOUD_PLATFORM","displayName":"instance-20250528-112341","gcpMetadata":{"folders":[{"resourceFolder":"//cloudresourcemanager.googleapis.com/folders/801234567893","resourceFolderDisplayName":"Security"},{"resourceFolder":"//cloudresourcemanager.googleapis.com/folders/901234567891","resourceFolderDisplayName":"Engineering"},{"resourceFolder":"//cloudresourcemanager.googleapis.com/folders/901234567890","resourceFolderDisplayName":"Research and Development"}],"organization":"organizations/012345601234","parent":"//cloudresourcemanager.googleapis.com/projects/101235645003","parentDisplayName":"test-siem","project":"//cloudresourcemanager.googleapis.com/projects/101235645003","projectDisplayName":"test-siem"},"location":"us-central1-b","name":"//compute.googleapis.com/projects/test-siem/zones/us-central1-b/instances/instance-20250528-112341","resourcePath":{"nodes":[{"displayName":"test-siem","id":"projects/101235645003","nodeType":"GCP_PROJECT"},{"displayName":"Security","id":"folders/801234567893","nodeType":"GCP_FOLDER"},{"displayName":"Engineering","id":"folders/901234567891","nodeType":"GCP_FOLDER"},{"displayName":"Research and Development","id":"folders/901234567890","nodeType":"GCP_FOLDER"},{"id":"organizations/012345601234","nodeType":"GCP_ORGANIZATION"}]},"resourcePathString":"organizations/012345601234/folders/901234567890/folders/901234567891/folders/801234567893/projects/101235645003","service":"compute.googleapis.com","type":"google.compute.Instance"}}

packages/google_scc/_dev/deploy/docker/files/finding.log

@@ -1,4 +1,15 @@
 # newer versions go on top
+- version: "2.0.0"
+  changes:
+    - description: | 
+        Add support for Findings v2 API. Parse and convert camelCase fields into snake_case fields.
+        Some fields inside "google_scc.finding.resource" are moved into "google_scc.finding.resource.gcp_metadata" as per v2 API schema.
+        This change allows to choose between v1 and v2 (recommended) APIs to fetch findings.
+        Location-based findings can be fetched to support Security Command Center data residency feature.
+        Field "google_scc.finding.vulnerability.cve.cvssv3.base_score" data type is updated to "float" to better suit data.
+        Add mapping changes in `finding` datastream for Cloud Detection and Response (CDR) vulnerability and misconfiguration workflow.
+      type: breaking-change
+      link: https://github.com/elastic/integrations/pull/14629
 - version: "1.10.1"
   changes:
     - description: Add temporary processor to remove the fields added by the Agentless policy.

packages/google_scc/changelog.yml

@@ -681,4 +681,4 @@
         },
         null
     ]
-}
+}

packages/google_scc/data_stream/asset/_dev/test/pipeline/test-asset.log-expected.json

@@ -1,4 +1,7 @@
 input: httpjson
+skip:
+  reason: "The fleet health status changes to degraded when the HTTPJSON template's value evaluation comes up empty, which leads to system test failures but does not interrupt the data flow."
+  link: https://github.com/elastic/beats/issues/45664
 service: google_scc
 vars:
   credentials_type: credentials_json

packages/google_scc/data_stream/asset/_dev/test/system/test-default-config.yml

@@ -33,12 +33,14 @@ request.transforms:
 response.pagination:
   - set:
       target: url.params.readTime
-      value: '[[formatDate (parseDate .last_response.body.readTime) "RFC3339"]]'
+      value: '[[if index .last_response.body "readTime"]][[formatDate (parseDate .last_response.body.readTime) "RFC3339"]][[end]]'
       fail_on_template_error: true
+      do_not_log_failure: true
   - set:
       target: url.params.pageToken
       value: '[[if index .last_response.body "nextPageToken"]][[.last_response.body.nextPageToken]][[end]]'
       fail_on_template_error: true
+      do_not_log_failure: true
 response.split:
   target: body.assets
   ignore_empty_value: true

packages/google_scc/data_stream/asset/agent/stream/httpjson.yml.hbs

@@ -1,108 +1,77 @@
 {
-    "@timestamp": "2023-07-03T06:24:10.638Z",
+    "@timestamp": "2025-07-30T10:45:24.425Z",
     "agent": {
-        "ephemeral_id": "7ab58b6a-e33a-470d-b529-80d7f867ce64",
-        "id": "4c00a899-0103-47cf-a91d-fa52a48711c8",
-        "name": "docker-fleet-agent",
+        "ephemeral_id": "f9c21eae-c360-427b-afdf-3662d2d99c30",
+        "id": "89379132-64b3-4465-ae5e-8d179096ac1b",
+        "name": "elastic-agent-69527",
         "type": "filebeat",
-        "version": "8.8.0"
+        "version": "8.19.0"
     },
     "data_stream": {
         "dataset": "google_scc.asset",
-        "namespace": "ep",
+        "namespace": "56937",
         "type": "logs"
     },
     "ecs": {
         "version": "8.11.0"
     },
     "elastic_agent": {
-        "id": "4c00a899-0103-47cf-a91d-fa52a48711c8",
-        "snapshot": false,
-        "version": "8.8.0"
+        "id": "89379132-64b3-4465-ae5e-8d179096ac1b",
+        "snapshot": true,
+        "version": "8.19.0"
     },
     "event": {
         "agent_id_status": "verified",
         "category": [
             "host"
         ],
-        "created": "2023-07-03T06:24:26.934Z",
+        "created": "2025-07-30T10:45:24.425Z",
         "dataset": "google_scc.asset",
-        "id": "f14c38ac40-2",
-        "ingested": "2023-07-03T06:24:30Z",
+        "ingested": "2025-07-30T10:45:27Z",
         "kind": "event",
+        "original": "{\"ancestors\":[\"organizations/523456987520\"],\"assetType\":\"cloudbilling.googleapis.com/BillingAccount\",\"name\":\"//cloudbilling.googleapis.com/billingAccounts/012345-A08098-1Ab2CD\",\"resource\":{\"data\":{\"displayName\":\"New\",\"name\":\"billingAccounts/012345-A08098-1Ab2CD\"},\"discoveryDocumentUri\":\"https://cloudbilling.googleapis.com/$discovery/rest\",\"discoveryName\":\"BillingAccount\",\"location\":\"global\",\"version\":\"v1\"},\"updateTime\":\"2022-11-17T12:20:17.601902Z\"}",
         "type": [
             "info"
         ]
     },
     "google_scc": {
         "asset": {
             "ancestors": [
-                "projects/123456987522",
-                "folders/123456987520",
                 "organizations/523456987520"
             ],
-            "prior": {
-                "ancestors": [
-                    "projects/123456987522",
-                    "folders/123456987520",
-                    "organizations/523456987520"
-                ],
-                "name": "//logging.googleapis.com/projects/123456987522/locations/global/buckets/_Default",
-                "resource": {
-                    "data": {
-                        "analyticsEnabled": true,
-                        "description": "Default bucket",
-                        "lifecycleState": "ACTIVE",
-                        "name": "projects/123456987522/locations/global/buckets/_Default",
-                        "retentionDays": 30
-                    },
-                    "discovery": {
-                        "document_uri": "https://logging.googleapis.com/$discovery/rest",
-                        "name": "LogBucket"
-                    },
-                    "location": "global",
-                    "parent": "//cloudresourcemanager.googleapis.com/projects/123456987522",
-                    "version": "v2"
-                },
-                "type": "logging.googleapis.com/LogBucket",
-                "update_time": "2023-05-27T18:53:48.843Z"
-            },
-            "prior_asset_state": "PRESENT",
+            "name": "//cloudbilling.googleapis.com/billingAccounts/012345-A08098-1Ab2CD",
             "resource": {
                 "data": {
-                    "description": "Default bucket",
-                    "lifecycleState": "ACTIVE",
-                    "name": "projects/123456987522/locations/global/buckets/_Default",
-                    "retentionDays": 30
+                    "displayName": "New",
+                    "name": "billingAccounts/012345-A08098-1Ab2CD"
                 },
                 "discovery": {
-                    "document_uri": "https://logging.googleapis.com/$discovery/rest",
-                    "name": "LogBucket"
+                    "document_uri": "https://cloudbilling.googleapis.com/$discovery/rest",
+                    "name": "BillingAccount"
                 },
                 "location": "global",
-                "parent": "//cloudresourcemanager.googleapis.com/projects/123456987522",
-                "version": "v2"
+                "version": "v1"
             },
-            "update_time": "2023-05-28T06:59:48.052Z",
-            "window": {
-                "start_time": "2023-05-28T06:59:48.052Z"
-            }
+            "type": "cloudbilling.googleapis.com/BillingAccount",
+            "update_time": "2022-11-17T12:20:17.601Z"
         }
     },
     "host": {
-        "name": "//logging.googleapis.com/projects/123456987522/locations/global/buckets/_Default",
-        "type": "logging.googleapis.com/LogBucket"
+        "name": "//cloudbilling.googleapis.com/billingAccounts/012345-A08098-1Ab2CD",
+        "type": "cloudbilling.googleapis.com/BillingAccount"
     },
     "input": {
-        "type": "gcp-pubsub"
+        "type": "httpjson"
     },
     "related": {
         "hosts": [
-            "//logging.googleapis.com/projects/123456987522/locations/global/buckets/_Default"
+            "//cloudbilling.googleapis.com/billingAccounts/012345-A08098-1Ab2CD"
         ]
     },
     "tags": [
+        "preserve_original_event",
+        "preserve_duplicate_custom_fields",
         "forwarded",
         "google_scc-asset"
     ]
-}
+}

packages/google_scc/data_stream/asset/sample_event.json

@@ -76,4 +76,4 @@
             ]
         }
     ]
-}
+}