Notifications: Add support for webhooks (#7311)

* Add go-httpbin

* First round of changes

* move webhooks to separated model,fix err handliing

* flake8

* Uset contant instead of strings

* Add basic API endpoints

* Add owner of endpoint

* Update go-httpbin

* Basic GUI

* per line

* upgrade go-httpbin, move db_mig

* Disable view and changes if not enabled in setting

* Fix full text of status

* Update go-httpbin

* Move migration

* Rename model + flake8

* Rebase db mig

* Rearange setting buttons, add connectivity validator

* Handle more generic errors from 'requests'

* flake8

* Rewrite YAML template to JSON request body

* update go-httpbin

* Update go-httpbin

* Inc db_mig

* Upgrade

* Ruff

* Update httpbin, move db_mig, use as_view

* Fix nones, more verbose "missing template"

* Prepare templates

* Usable by admins only

* API tests

* Add main unittests

* Update 4xx test

* Docs: add Transition graph

* ruff

* Rewrite

* Start "webhook.endpoint" in unit-tests

* Extend webhook_status_cleanup, add note to related places

* More tests

* Small adjustments

* Set max_length

* Better handle nones

* Add basic doc + fix findings_list

* Update docs

* Clean ruff

* Fix db_mig

* Fix long notes

* Clean ruff

* Move "webhook.endpoint" from debug docker to dev

* Make fields "editable=False"

* Try to fix accesslint

* Use class-based choices

* Shorter default timeout

* Update dojo/notifications/views.py

Co-authored-by: Cody Maffucci <[email protected]>

* Finish preprocess_request

* Update dojo/notifications/helper.py

Co-authored-by: Cody Maffucci <[email protected]>

* Show error-times as hint

* Try to fix accesslint

* Rename `url` to `url_ui` and add `url_api`

* inc db_mig

* Accept any 2xx as successful

* Add permission checker for item in menu

* Fix editing for superadmin

---------

Co-authored-by: Cody Maffucci <[email protected]>

.github/workflows/rest-framework-tests.yml

@@ -34,8 +34,8 @@ jobs:
         run: docker/setEnv.sh unit_tests_cicd
 
       # phased startup so we can use the exit code from unit test container
-      - name: Start Postgres
-        run: docker compose up -d postgres
+      - name: Start Postgres and webhook.endpoint
+        run: docker compose up -d postgres webhook.endpoint
 
       # no celery or initializer needed for unit tests
       - name: Unit tests

docker-compose.override.dev.yml

@@ -53,3 +53,5 @@ services:
         published: 8025
         protocol: tcp
         mode: host
+  "webhook.endpoint":
+    image: mccutchen/go-httpbin:v2.14.0@sha256:e0f398a0a29e7cf00a2467326344d70b4d89d0786d8f9a3287c2a0371c804823

docker-compose.override.unit_tests.yml

@@ -51,6 +51,8 @@ services:
   redis:
     image: busybox:1.36.1-musl
     entrypoint: ['echo', 'skipping', 'redis']
+  "webhook.endpoint":
+    image: mccutchen/go-httpbin:v2.14.0@sha256:e0f398a0a29e7cf00a2467326344d70b4d89d0786d8f9a3287c2a0371c804823
 volumes:
   defectdojo_postgres_unit_tests: {}
   defectdojo_media_unit_tests: {}

docker-compose.override.unit_tests_cicd.yml

@@ -50,6 +50,8 @@ services:
   redis:
     image: busybox:1.36.1-musl
     entrypoint: ['echo', 'skipping', 'redis']
+  "webhook.endpoint":
+    image: mccutchen/go-httpbin:v2.14.0@sha256:e0f398a0a29e7cf00a2467326344d70b4d89d0786d8f9a3287c2a0371c804823
 volumes:
   defectdojo_postgres_unit_tests: {}
   defectdojo_media_unit_tests: {}

docs/content/en/integrations/burp-plugin.md

@@ -2,7 +2,7 @@
 title: "Defect Dojo Burp plugin"
 description: "Export findings directly from Burp to DefectDojo."
 draft: false
-weight: 8
+weight: 9
 ---
 
 **Please note: The DefectDojo Burp Plugin has been sunset and is no longer a supported feature.**

docs/content/en/integrations/exporting.md

@@ -2,7 +2,7 @@
 title: "Exporting"
 description: "DefectDojo has the ability to export findings."
 draft: false
-weight: 11
+weight: 12
 ---
 
 

docs/content/en/integrations/google-sheets-sync.md

@@ -2,7 +2,7 @@
 title: "Google Sheets synchronisation"
 description: "Export finding details to Google Sheets and upload changes from Google Sheets."
 draft: false
-weight: 7
+weight: 8
 ---
 
 **Please note - the Google Sheets feature has been deprecated as of DefectDojo version 2.21.0 - these documents are for reference only.**

docs/content/en/integrations/languages.md

@@ -2,7 +2,7 @@
 title: "Languages and lines of code"
 description: "You can import an analysis of languages used in a project, including lines of code."
 draft: false
-weight: 9
+weight: 10
 ---
 
 ## Import of languages for a project

docs/content/en/integrations/notification_webhooks/_index.md

@@ -0,0 +1,79 @@
+---
+title: "Notification Webhooks (experimental)"
+description: "How to setup and use webhooks"
+weight: 7
+chapter: true
+---
+
+Webhooks are HTTP requests coming from the DefectDojo instance towards user-defined webserver which expects this kind of incoming traffic.
+
+## Transition graph:
+
+It is not unusual that in some cases webhook can not be performed. It is usually connected to network issues, server misconfiguration, or running upgrades on the server. DefectDojo needs to react to these outages. It might temporarily or permanently disable related endpoints. The following graph shows how it might change the status of the webhook definition based on HTTP responses (or manual user interaction).
+
+```mermaid
+flowchart TD
+
+    START{{Endpoint created}}
+    ALL{All states}
+    STATUS_ACTIVE([STATUS_ACTIVE])
+    STATUS_INACTIVE_TMP
+    STATUS_INACTIVE_PERMANENT
+    STATUS_ACTIVE_TMP([STATUS_ACTIVE_TMP])
+    END{{Endpoint removed}}
+
+    START ==> STATUS_ACTIVE
+    STATUS_ACTIVE --HTTP 200 or 201 --> STATUS_ACTIVE
+    STATUS_ACTIVE --HTTP 5xx <br>or HTTP 429 <br>or Timeout--> STATUS_INACTIVE_TMP
+    STATUS_ACTIVE --Any HTTP 4xx response<br>or any other HTTP response<br>or non-HTTP error--> STATUS_INACTIVE_PERMANENT
+    STATUS_INACTIVE_TMP -.After 60s.-> STATUS_ACTIVE_TMP
+    STATUS_ACTIVE_TMP --HTTP 5xx <br>or HTTP 429 <br>or Timeout <br>within 24h<br>from the first error-->STATUS_INACTIVE_TMP
+    STATUS_ACTIVE_TMP -.After 24h.-> STATUS_ACTIVE
+    STATUS_ACTIVE_TMP --HTTP 200 or 201 --> STATUS_ACTIVE_TMP
+    STATUS_ACTIVE_TMP --HTTP 5xx <br>or HTTP 429 <br>or Timeout <br>within 24h from the first error<br>or any other HTTP response or error--> STATUS_INACTIVE_PERMANENT
+    ALL ==Activation by user==> STATUS_ACTIVE
+    ALL ==Deactivation by user==> STATUS_INACTIVE_PERMANENT
+    ALL ==Removal of endpoint by user==> END
+```
+
+Notes: 
+
+1. Transitions:
+    - bold: manual changes by user
+    - dotted: automated by celery
+    - others: based on responses on webhooks
+1. Nodes:
+    - Stadium-shaped: Active - following webhook can be sent
+    - Rectangles: Inactive - performing of webhook will fail (and not retried)
+    - Hexagonal: Initial and final states
+    - Rhombus: All states (meta node to make the graph more readable)
+
+## Body and Headers
+
+The body of each request is JSON which contains data about related events like names and IDs of affected elements.
+Examples of bodies are on pages related to each event (see below).
+
+Each request contains the following headers. They might be useful for better handling of events by server this process events.
+
+```yaml
+User-Agent: DefectDojo-<version of DD>
+X-DefectDojo-Event: <name of the event>
+X-DefectDojo-Instance: <Base URL for DD instance>
+```
+## Disclaimer
+
+This functionality is new and in experimental mode. This means Functionality might generate breaking changes in following DefectDojo releases and might not be considered final.
+
+However, the community is open to feedback to make this functionality better and transform it stable as soon as possible.
+
+## Roadmap
+
+There are a couple of known issues that are expected to be implemented as soon as core functionality is considered ready.
+
+- Support events - Not only adding products, product types, engagements, tests, or upload of new scans but also events around SLA
+- User webhook - right now only admins can define webhooks; in the future also users will be able to define their own
+- Improvement in UI - add filtering and pagination of webhook endpoints
+
+## Events
+
+<!-- Hugo automatically renders list of subpages here -->

docs/content/en/integrations/notification_webhooks/engagement_added.md

@@ -0,0 +1,38 @@
+---
+title: "Event: engagement_added"
+weight: 3
+chapter: true
+---
+
+## Event HTTP header
+```yaml
+X-DefectDojo-Event: engagement_added
+```
+
+## Event HTTP body
+```json
+{
+    "description": null,
+    "engagement": {
+        "id": 7,
+        "name": "notif eng",
+        "url_api": "http://localhost:8080/api/v2/engagements/7/",
+        "url_ui": "http://localhost:8080/engagement/7"
+    },
+    "product": {
+        "id": 4,
+        "name": "notif prod",
+        "url_api": "http://localhost:8080/api/v2/products/4/",
+        "url_ui": "http://localhost:8080/product/4"
+    },
+    "product_type": {
+        "id": 4,
+        "name": "notif prod type",
+        "url_api": "http://localhost:8080/api/v2/product_types/4/",
+        "url_ui": "http://localhost:8080/product/type/4"
+    },
+    "url_api": "http://localhost:8080/api/v2/engagements/7/",
+    "url_ui": "http://localhost:8080/engagement/7",
+    "user": null
+}
+```

docs/content/en/integrations/notification_webhooks/product_added.md

@@ -0,0 +1,32 @@
+---
+title: "Event: product_added"
+weight: 2
+chapter: true
+---
+
+## Event HTTP header
+```yaml
+X-DefectDojo-Event: product_added
+```
+
+## Event HTTP body
+```json
+{
+    "description": null,
+    "product": {
+        "id": 4,
+        "name": "notif prod",
+        "url_api": "http://localhost:8080/api/v2/products/4/",
+        "url_ui": "http://localhost:8080/product/4"
+    },
+    "product_type": {
+        "id": 4,
+        "name": "notif prod type",
+        "url_api": "http://localhost:8080/api/v2/product_types/4/",
+        "url_ui": "http://localhost:8080/product/type/4"
+    },
+    "url_api": "http://localhost:8080/api/v2/products/4/",
+    "url_ui": "http://localhost:8080/product/4",
+    "user": null
+}
+```

docs/content/en/integrations/notification_webhooks/product_type_added.md

@@ -0,0 +1,26 @@
+---
+title: "Event: product_type_added"
+weight: 1
+chapter: true
+---
+
+## Event HTTP header
+```yaml
+X-DefectDojo-Event: product_type_added
+```
+
+## Event HTTP body
+```json
+{
+    "description": null,
+    "product_type": {
+        "id": 4,
+        "name": "notif prod type",
+        "url_api": "http://localhost:8080/api/v2/product_types/4/",
+        "url_ui": "http://localhost:8080/product/type/4"
+    },
+    "url_api": "http://localhost:8080/api/v2/product_types/4/",
+    "url_ui": "http://localhost:8080/product/type/4",
+    "user": null
+}
+```

docs/content/en/integrations/notification_webhooks/scan_added.md

@@ -0,0 +1,90 @@
+---
+title: "Event: scan_added and scan_added_empty"
+weight: 5
+chapter: true
+---
+
+Event `scan_added_empty` describes a situation when reimport did not affect the existing test (no finding has been created or closed). 
+
+## Event HTTP header for scan_added
+```yaml
+X-DefectDojo-Event: scan_added
+```
+
+## Event HTTP header for scan_added_empty
+```yaml
+X-DefectDojo-Event: scan_added_empty
+```
+
+## Event HTTP body
+```json
+{
+    "description": null,
+    "engagement": {
+        "id": 7,
+        "name": "notif eng",
+        "url_api": "http://localhost:8080/api/v2/engagements/7/",
+        "url_ui": "http://localhost:8080/engagement/7"
+    },
+    "finding_count": 4,
+    "findings": {
+        "mitigated": [
+            {
+                "id": 233,
+                "severity": "Medium",
+                "title": "Mitigated Finding",
+                "url_api": "http://localhost:8080/api/v2/findings/233/",
+                "url_ui": "http://localhost:8080/finding/233"
+            }
+        ],
+        "new": [
+            {
+                "id": 232,
+                "severity": "Critical",
+                "title": "New Finding",
+                "url_api": "http://localhost:8080/api/v2/findings/232/",
+                "url_ui": "http://localhost:8080/finding/232"
+            }
+        ],
+        "reactivated": [
+            {
+                "id": 234,
+                "severity": "Low",
+                "title": "Reactivated Finding",
+                "url_api": "http://localhost:8080/api/v2/findings/234/",
+                "url_ui": "http://localhost:8080/finding/234"
+            }
+        ],
+        "untouched": [
+            {
+                "id": 235,
+                "severity": "Info",
+                "title": "Untouched Finding",
+                "url_api": "http://localhost:8080/api/v2/findings/235/",
+                "url_ui": "http://localhost:8080/finding/235"
+            }
+        ]
+    },
+    "product": {
+        "id": 4,
+        "name": "notif prod",
+        "url_api": "http://localhost:8080/api/v2/products/4/",
+        "url_ui": "http://localhost:8080/product/4"
+    },
+    "product_type": {
+        "id": 4,
+        "name": "notif prod type",
+        "url_api": "http://localhost:8080/api/v2/product_types/4/",
+        "url_ui": "http://localhost:8080/product/type/4"
+    },
+    "test": {
+        "id": 90,
+        "title": "notif test",
+        "url_api": "http://localhost:8080/api/v2/tests/90/",
+        "url_ui": "http://localhost:8080/test/90"
+    },
+    "url_api": "http://localhost:8080/api/v2/tests/90/",
+    "url_ui": "http://localhost:8080/test/90",
+    "user": null
+}
+```

docs/content/en/integrations/notification_webhooks/test_added.md

@@ -0,0 +1,44 @@
+---
+title: "Event: test_added"
+weight: 4
+chapter: true
+---
+
+## Event HTTP header
+```yaml
+X-DefectDojo-Event: test_added
+```
+
+## Event HTTP body
+```json
+{
+    "description": null,
+    "engagement": {
+        "id": 7,
+        "name": "notif eng",
+        "url_api": "http://localhost:8080/api/v2/engagements/7/",
+        "url_ui": "http://localhost:8080/engagement/7"
+    },
+    "product": {
+        "id": 4,
+        "name": "notif prod",
+        "url_api": "http://localhost:8080/api/v2/products/4/",
+        "url_ui": "http://localhost:8080/product/4"
+    },
+    "product_type": {
+        "id": 4,
+        "name": "notif prod type",
+        "url_api": "http://localhost:8080/api/v2/product_types/4/",
+        "url_ui": "http://localhost:8080/product/type/4"
+    },
+    "test": {
+        "id": 90,
+        "title": "notif test",
+        "url_api": "http://localhost:8080/api/v2/tests/90/",
+        "url_ui": "http://localhost:8080/test/90"
+    },
+    "url_api": "http://localhost:8080/api/v2/tests/90/",
+    "url_ui": "http://localhost:8080/test/90",
+    "user": null
+}
+```