Event Integrations Structurizr POC

.github/CODEOWNERS

@@ -39,12 +39,15 @@ bitwarden_license/src/Sso @bitwarden/team-auth-dev
 src/Identity @bitwarden/team-auth-dev
 src/Core/Identity @bitwarden/team-auth-dev
 src/Core/IdentityServer @bitwarden/team-auth-dev
+docs/auth/* @bitwarden/team-auth-dev
 
 # Key Management team
 **/KeyManagement @bitwarden/team-key-management-dev
+docs/key_management/* @bitwarden/team-key-management-dev
 
 # Tools team
 **/Tools @bitwarden/team-tools-dev
+docs/tools/* @bitwarden/team-tools-dev
 
 # Dirt (Data Insights & Reporting) team
 src/Api/Controllers/Dirt @bitwarden/team-data-insights-and-reporting-dev
@@ -56,6 +59,7 @@ test/Core.Test/Dirt @bitwarden/team-data-insights-and-reporting-dev
 # Vault team
 **/Vault @bitwarden/team-vault-dev
 **/Vault/AuthorizationHandlers @bitwarden/team-vault-dev @bitwarden/team-admin-console-dev  # joint ownership over authorization handlers that affect organization users
+docs/vault/* @bitwarden/team-vault-dev
 
 # Admin Console team
 **/AdminConsole @bitwarden/team-admin-console-dev
@@ -64,6 +68,7 @@ bitwarden_license/src/test/Scim.IntegrationTest @bitwarden/team-admin-console-de
 bitwarden_license/src/test/Scim.ScimTest @bitwarden/team-admin-console-dev
 src/Events @bitwarden/team-admin-console-dev
 src/EventsProcessor @bitwarden/team-admin-console-dev
+docs/admin_console/* @bitwarden/team-admin-console-dev
 
 # Billing team
 **/*billing* @bitwarden/team-billing-dev
@@ -80,6 +85,7 @@ src/EventsProcessor @bitwarden/team-admin-console-dev
 **/Billing @bitwarden/team-billing-dev
 src/Admin/Controllers/ToolsController.cs @bitwarden/team-billing-dev
 src/Admin/Views/Tools @bitwarden/team-billing-dev
+docs/billing/* @bitwarden/team-billing-dev
 
 # Platform team
 .github/workflows/build.yml @bitwarden/team-platform-dev
@@ -93,7 +99,10 @@ src/Admin/Views/Tools @bitwarden/team-billing-dev
 **/.dockerignore @bitwarden/team-platform-dev
 **/Dockerfile @bitwarden/team-platform-dev
 **/entrypoint.sh @bitwarden/team-platform-dev
+docs/platform/* @bitwarden/team-platform-dev
 
 # Multiple owners - DO NOT REMOVE (BRE)
 **/packages.lock.json
 Directory.Build.props
+
+docs/ @bitwarden/dept-architecture

docs/.gitignore

@@ -0,0 +1,2 @@
+.structurizr
+export

docs/admin_console/event_integrations/building_a_new_integration.md

@@ -0,0 +1,145 @@
+# Building a new integration
+
+These are all the pieces required in the process of building out a new integration. For
+clarity in naming, these assume a new integration called "Example".
+
+## IntegrationType
+
+Add a new type to `IntegrationType` for the new integration.
+
+## Configuration Models
+
+The configuration models are the classes that will determine what is stored in the database for
+`OrganizationIntegration` and `OrganizationIntegrationConfiguration`. The `Configuration` columns are the
+serialized version of the corresponding objects and represent the coonfiguration details for this integration
+and event type.
+
+1. `ExampleIntegration`
+    - Configuration details for the whole integration (e.g. a token in Slack).
+    - Applies to every event type configuration defined for this integration.
+    - Maps to the JSON structure stored in `Configuration` in ``OrganizationIntegration`.
+2. `ExampleIntegrationConfiguration`
+    - Configuration details that could change from event to event (e.g. channelId in Slack).
+    - Maps to the JSON structure stored in `Configuration` in `OrganizationIntegrationConfiguration`.
+3. `ExampleIntegrationConfigurationDetails`
+    - Combined configuration of both Integration _and_ IntegrationConfiguration.
+    - This will be the deserialized version of the `MergedConfiguration` in
+      `OrganizationIntegrationConfigurationDetails`.
+
+## Request Models
+
+1. Add a new case to the switch method in `OrganizationIntegrationRequestModel.Validate`.
+2. Add a new case to the switch method in `OrganizationIntegrationConfigurationRequestModel.IsValidForType`.
+
+## Integration Handler
+
+e.g. `ExampleIntegrationHandler`
+- This is where the actual code will go to perform the integration (i.e. send an HTTP request, etc.).
+- Handlers receive an `IntegrationMessage<T>` where `<T>` is the `ExampleIntegrationConfigurationDetails`
+  defined above. This has the Configuration as well as the rendered template message to be sent.
+- Handlers return an `IntegrationHandlerResult` with details about if the request - success / failure,
+  if it can be retried, when it should be delayed until, etc.
+- The scope of the handler is simply to do the integration and report the result.
+  Everything else (such as how many times to retry, when to retry, what to do with failures)
+  is done in the Listener.
+
+## GlobalSettings
+
+### RabbitMQ
+Add the queue names for the integration. These are typically set with a default value so
+that they will be created when first accessed in code by RabbitMQ.
+
+1. `ExampleEventQueueName`
+2. `ExampleIntegrationQueueName`
+3. `ExampleIntegrationRetryQueueName`
+
+### Azure Service Bus
+Add the subscription names to use for ASB for this integration. Similar to RabbitMQ a
+default value is provided so that we don't require configuring it in secrets but allow
+it to be overridden. **However**, unlike RabbitMQ these subscriptions must exist prior
+to the code accessing them. They will not be created on the fly. See [Deploying a new
+integration](#deploying-a-new-integration) below
+
+1. `ExmpleEventSubscriptionName`
+2. `ExmpleIntegrationSubscriptionName`
+
+#### Service Bus Emulator, local config
+In order to create ASB resources locally, we need to also update the `servicebusemulator_config.json` file
+to include any new subscriptions.
+- Under the existing event topic (`event-logging`) add a subscription for the event level for this
+  new integration (`events-example-subscription`).
+- Under the existing integration topic (`event-integrations`) add a new subscription for the integration
+  level messages (`integration-example-subscription`).
+    - Copy the correlation filter from the other integration level subscriptions. It should filter based on
+      the `IntegrationType.ToRoutingKey`, or in this example `example`.
+
+These names added here are what must match the values provided in the secrets or the defaults provided
+in Global Settings. This must be in place (and the local ASB emulator restarted) before you can use any
+code locally that accesses ASB resources.
+
+## ServiceCollectionExtensions
+In our `ServiceCollectionExtensions`, we pull all the above pieces together to start listeners on each message
+tier with handlers to process the integration. There are a number of helper methods in here to make this simple
+to add a new integration - one call per platform.
+
+Also note that if an integration needs a custom singleton / service defined, the add listeners method is a
+good place to set that up. For instance, `SlackIntegrationHandler` needs a `SlackService`, so the singleton
+declaration is right above the add integration method for slack. Same thing for webhooks when it comes to
+defining a custom HttpClient by name.
+
+1. In `AddRabbitMqListeners` add the integration:
+``` csharp
+        services.AddRabbitMqIntegration<ExampleIntegrationConfigurationDetails, ExampleIntegrationHandler>(
+            globalSettings.EventLogging.RabbitMq.ExampleEventsQueueName,
+            globalSettings.EventLogging.RabbitMq.ExampleIntegrationQueueName,
+            globalSettings.EventLogging.RabbitMq.ExampleIntegrationRetryQueueName,
+            globalSettings.EventLogging.RabbitMq.MaxRetries,
+            IntegrationType.Example);
+```
+
+2. In `AddAzureServiceBusListeners` add the integration:
+``` csharp
+services.AddAzureServiceBusIntegration<ExampleIntegrationConfigurationDetails, ExampleIntegrationHandler>(
+            eventSubscriptionName: globalSettings.EventLogging.AzureServiceBus.ExampleEventSubscriptionName,
+            integrationSubscriptionName: globalSettings.EventLogging.AzureServiceBus.ExampleIntegrationSubscriptionName,
+            integrationType: IntegrationType.Example,
+            globalSettings: globalSettings);
+```
+
+# Deploying a new integration
+
+## RabbitMQ
+
+RabbitMQ dynamically creates queues and exchanges when they are first accessed in code.
+Therefore, there is no need to manually create queues when deploying a new integration.
+They can be created and configured ahead of time, but it's not required. Note that once
+they are created, if any configurations need to be changed, the queue or exchange must be
+deleted and recreated.
+
+## Azure Service Bus
+
+Unlike RabbitMQ, ASB resources **must** be allocated before the code accesses them and
+will not be created on the fly. This means that any subscriptions needed for a new
+integration must be created in ASB before that code is deployed.
+
+The two subscriptions created above in Global Settings and `servicebusemulator_config.json`
+need to be created in the Azure portal or CLI for the environment before deploying the
+code.
+
+1. `ExmpleEventSubscriptionName`
+    - This subscription is a fan-out subscription from the main event topic.
+    - As such, it will start receiving all the events as soon as it is declared.
+    - This can create a backlog before the integration-specific handler is declared and deployed.
+    - One strategy to avoid this is to create the subscription with a false filter (e.g. `1 = 0`).
+        - This will create the subscription, but the filter will ensure that no messages
+          actually land in the subscription.
+        - Code can be deployed that references the subscription, because the subscription
+          legitimately exists (it is simply empty).
+        - When the code is in place, and we're ready to start receiving messages on the new
+          integration, we simply remove the filter to return the subscription to receiving
+          all messages via fan-out.
+2. `ExmpleIntegrationSubscriptionName`
+    - This subscription must be created before the new integration code can be deployed.
+    - However, it is not fan-out, but rather a filter based on the `IntegrationType.ToRoutingKey`.
+    - Therefore, it won't start receiving messages until organizations have active configurations.
+      This means there's no risk of building up a backlog by declaring it ahead of time.

docs/admin_console/event_integrations/models.dsl

@@ -0,0 +1,195 @@
+!element server {
+    azure_service_bus = container "Azure Service Bus" {
+        description "AMQP service used for pub/sub architecture for Events and Integrations"
+        tags "Events", "Azure", "ASB"
+
+        event_topic = component "Event Topic" {
+            description "The main entry point for all events in the system. When an event occurs, it is published to this topic."
+            tags "Events", "ASB", "Event Tier"
+        }
+
+        integration_topic = component "Integration Topic" {
+            description "Events that have integrations configured are processed and put on the integration topic with a routing key for their specific integration handler to process."
+            tags "Events", "ASB", "Integrations", "Integration Tier"
+        }
+
+        eventsWriteSub = component "events-write-subscription" {
+            description "Subscription for EventRepositoryHandler to write all events into azure table storage."
+            tags "ASB", "Subscription", "Event Tier"
+        }
+
+        eventsSlackSub = component "events-slack-subscription" {
+            description "Subscription for slack-specific EventIntegrationHandler which publishes processed events to the integration tier if there is a Slack integration configured."
+            tags "ASB", "Subscription", "Event Tier", "Slack"
+        }
+
+        eventsWebhookSub = component "events-webhook-subscription" {
+            description "Subscription for webhook-specific EventIntegrationHandler which publishes processed events to the integration tier if there is a webhook integration configured."
+            tags "ASB", "Subscription", "Event Tier", "Webhook"
+        }
+
+        eventsHecSub = component "events-hec-subscription" {
+            description "Subscription for HEC-specific EventIntegrationHandler which publishes processed events to the integration tier if there is a HEC integration configured."
+            tags "ASB", "Subscription", "Event Tier", "HEC"
+        }
+
+        integrationSlackSub = component "integration-slack-subscription" {
+            description "Integration-level subscription for Slack IntegrationMessages. Correlation filter: Label = 'slack'."
+            tags "ASB", "Subscription", "Integration Tier", "Slack"
+        }
+
+        integrationWebhookSub = component "integration-webhook-subscription" {
+            description "Integration-level subscription for Webhook IntegrationMessages. Correlation filter: Label = 'webhook'."
+            tags "ASB", "Subscription", "Integration Tier", "Webhook"
+        }
+
+        integrationHecSub = component "integration-hec-subscription" {
+            description "Integration-level subscription for HEC IntegrationMessages. Correlation filter: Label = 'hec'."
+            tags "ASB", "Subscription", "Integration Tier", "HEC"
+        }
+    }
+
+    rabbit_mq = container "RabbitMQ" {
+        tags "Events"
+        tags "RabbitMQ"
+
+        event_exchange = component "Event Exchange" {
+            tags "Events", "Event Tier"
+        }
+
+        integration_exchange = component "Integration Exchange" {
+            tags "Events", "Integrations", "Integration Tier"
+        }
+
+        eventsWriteQueue = component "events-write-queue" {
+            description "Queue for EventRepositoryHandler to write all events into the database."
+            tags "RabbitMQ", "Queue", "Event Tier"
+        }
+
+        eventsSlackQueue = component "events-slack-queue" {
+            description "Queue for slack-specific EventIntegrationHandler which publishes processed events to the integration tier if there is a Slack integration configured."
+            tags "RabbitMQ", "Queue", "Event Tier", "Slack"
+        }
+
+        eventsWebhookQueue = component "events-webhook-queue" {
+            description "Queue for webhook-specific EventIntegrationHandler which publishes processed events to the integration tier if there is a webhook integration configured."
+            tags "RabbitMQ", "Queue", "Event Tier", "Webhook"
+        }
+
+        eventsHecQueue = component "events-hec-queue" {
+            description "Queue for HEC-specific EventIntegrationHandler which publishes processed events to the integration tier if there is a HEC integration configured."
+            tags "RabbitMQ", "Queue", "Event Tier", "HEC"
+        }
+
+        integrationSlackQueue = component "integration-slack-queue" {
+            description "Integration-level queue for Slack IntegrationMessages. Routing key = 'slack'."
+            tags "RabbitMQ", "Queue", "Integration Tier", "Slack"
+        }
+
+        integrationWebhookQueue = component "integration-webhook-queue" {
+            description "Integration-level queue for Webhook IntegrationMessages. Routing key = 'webhook'."
+            tags "RabbitMQ", "Queue", "Integration Tier", "Webhook"
+        }
+
+        integrationHecQueue = component "integration-hec-queue" {
+            description "Integration-level queue for HEC IntegrationMessages. Routing key = 'hec'."
+            tags "RabbitMQ", "Queue", "Integration Tier", "HEC"
+        }
+
+        integrationSlackRetryQueue = component "integration-slack-retry-queue" {
+            description "Integration-level retry queue for Slack IntegrationMessages. Routing key Label = 'slack-retry'."
+            tags "RabbitMQ", "Queue", "Integration Tier", "Slack"
+        }
+
+        integrationWebhookRetryQueue = component "integration-webhook-retry-queue" {
+            description "Integration-level retry queue for Webhook IntegrationMessages. Routing key = 'webhook-retry'."
+            tags "RabbitMQ", "Queue", "Integration Tier", "Webhook"
+        }
+
+        integrationHecRetryQueue = component "integration-hec-retry-queue" {
+            description "Integration-level retry queue for HEC IntegrationMessages. Routing key = 'hec-retry'."
+            tags "RabbitMQ", "Queue", "Integration Tier", "HEC"
+        }
+    }
+}
+
+!element server.events_processor {
+    !docs "architecture.md"
+
+    event_repository_handler = component "EventRepositoryHandler" {
+        description "Handles all events, passing them off to the IEventWriteService with the `persistent` key for long term storage."
+    }
+    event_integration_handler = component "EventIntegrationHandler" {
+        description "Fetches the relevent configurations when an event comes in and hands the event to its paired integration handler for processing."
+    }
+    slack_integration_handler = component "SlackIntegrationHandler" {
+        description "Processes Slack IntegrationMessages, posting them to the configured channels."
+    }
+    webhook_integration_handler = component "WebhookIntegrationHandler" {
+        description "Processes Webhook and HEC IntegrationMessages, posting them to the configured URI."
+    }
+    integration_configuration_details_cache_service = component "IntegrationConfigurationDetailsCacheService" {
+        description "Caches all configurations for integrations in memory so that events can be handled without adding database load."
+    }
+    slack_service = component "SlackService" {
+        description "Handles all API interaction with Slack."
+    }
+    http_client = component "HttpClient" {
+        description "Performs any Http functions for Webhooks / HEC."
+    }
+    integration_filter_service = component "IntegrationFilterService" {
+        description "Processes filters from configurations to determine if an event should be processed out to the integration."
+    }
+}
+
+!element server.events {
+    event_repository_handler = component "EventRepositoryHandler" {
+        description "Handles all events, passing them off to the IEventWriteService with the `persistent` key for long term storage."
+    }
+    event_integration_handler = component "EventIntegrationHandler" {
+        description "Fetches the relevent configurations when an event comes in and hands the event to its paired integration handler for processing."
+    }
+    slack_integration_handler = component "SlackIntegrationHandler" {
+        description "Processes Slack IntegrationMessages, posting them to the configured channels."
+    }
+    webhook_integration_handler = component "WebhookIntegrationHandler" {
+        description "Processes Webhook and HEC IntegrationMessages, posting them to the configured URI."
+    }
+    integration_configuration_details_cache_service = component "IntegrationConfigurationDetailsCacheService" {
+        description "Caches all configurations for integrations in memory so that events can be handled without adding database load."
+    }
+    slack_service = component "SlackService" {
+        description "Handles all API interaction with Slack."
+    }
+    http_client = component "HttpClient" {
+        description "Performs any Http functions for Webhooks / HEC."
+    }
+    integration_filter_service = component "IntegrationFilterService" {
+        description "Processes filters from configurations to determine if an event should be processed out to the integration."
+    }
+}
+
+external_services = softwareSystem "External Services" {
+    tags "External", "Events", "Integrations"
+    description "External services (e.g. SIEM, Slack, et al) that consume events via integrations"
+
+    slack = container "Slack" {
+        tags "External", "Events", "Integrations", "Slack"
+        description "Slack messaging service. Receives messages via configured event integrations."
+    }
+
+    splunk = container "Splunk" {
+        tags "External", "Events", "Integrations", "Splunk"
+        description "Splunk SIEM service. Receives events via configured event integrations."
+    }
+
+    datadog = container "Datadog" {
+        tags "External", "Events", "Integrations", "Datadog"
+        description "Datadog SIEM service. Receives events via configured event integrations."
+    }
+
+    crowdstrike = container "Crowdstrike Falcon" {
+        tags "External", "Events", "Integrations", "Crowdstrike Falcon", "Crowdstrike"
+        description "Crowdstrike Falcon SIEM service. Receives events via configured event integrations."
+    }
+}