data [required]
\n[object]
The schema data.
attributes [required]
\nobject
Defines the pipeline’s name and its components (sources, processors, and destinations).
config [required]
\nobject
Specifies the pipeline's configuration, including its sources, processors, and destinations.
destinations [required]
\n[ <oneOf>]
A list of destination components where processed logs are sent.
Option 1
\nobject
The datadog_logs destination forwards logs to Datadog Log Management.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be datadog_logs. \nAllowed enum values: datadog_logs
default: datadog_logs
Option 2
\nobject
The amazon_s3 destination sends your logs in Datadog-rehydratable format to an Amazon S3 bucket for archiving.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
bucket [required]
\nstring
S3 bucket name.
id [required]
\nstring
Unique identifier for the destination component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
key_prefix
\nstring
Optional prefix for object keys.
region [required]
\nstring
AWS region of the S3 bucket.
storage_class [required]
\nenum
S3 storage class. \nAllowed enum values: STANDARD,REDUCED_REDUNDANCY,INTELLIGENT_TIERING,STANDARD_IA,EXPRESS_ONEZONE,ONEZONE_IA,GLACIER,GLACIER_IR,DEEP_ARCHIVE
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. Always amazon_s3. \nAllowed enum values: amazon_s3
default: amazon_s3
Option 3
\nobject
The google_cloud_storage destination stores logs in a Google Cloud Storage (GCS) bucket.\nIt requires a bucket name, GCP authentication, and metadata fields.
acl [required]
\nenum
Access control list setting for objects written to the bucket. \nAllowed enum values: private,project-private,public-read,authenticated-read,bucket-owner-read,bucket-owner-full-control
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
bucket [required]
\nstring
Name of the GCS bucket.
id [required]
\nstring
Unique identifier for the destination component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
key_prefix
\nstring
Optional prefix for object keys within the GCS bucket.
metadata
\n[object]
Custom metadata to attach to each object uploaded to the GCS bucket.
name [required]
\nstring
The metadata key.
value [required]
\nstring
The metadata value.
storage_class [required]
\nenum
Storage class used for objects stored in GCS. \nAllowed enum values: STANDARD,NEARLINE,COLDLINE,ARCHIVE
type [required]
\nenum
The destination type. Always google_cloud_storage. \nAllowed enum values: google_cloud_storage
default: google_cloud_storage
Option 4
\nobject
The splunk_hec destination forwards logs to Splunk using the HTTP Event Collector (HEC).
auto_extract_timestamp
\nboolean
If true, Splunk tries to extract timestamps from incoming log events.\nIf false, Splunk assigns the time the event was received.
encoding
\nenum
Encoding format for log events. \nAllowed enum values: json,raw_message
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
index
\nstring
Optional name of the Splunk index where logs are written.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
sourcetype
\nstring
The Splunk sourcetype to assign to log events.
type [required]
\nenum
The destination type. Always splunk_hec. \nAllowed enum values: splunk_hec
default: splunk_hec
Option 5
\nobject
The sumo_logic destination forwards logs to Sumo Logic.
encoding
\nenum
The output encoding format. \nAllowed enum values: json,raw_message,logfmt
header_custom_fields
\n[object]
A list of custom headers to include in the request to Sumo Logic.
name [required]
\nstring
The header field name.
value [required]
\nstring
The header field value.
header_host_name
\nstring
Optional override for the host name header.
header_source_category
\nstring
Optional override for the source category header.
header_source_name
\nstring
Optional override for the source name header.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be sumo_logic. \nAllowed enum values: sumo_logic
default: sumo_logic
Option 6
\nobject
The elasticsearch destination writes logs to an Elasticsearch cluster.
api_version
\nenum
The Elasticsearch API version to use. Set to auto to auto-detect. \nAllowed enum values: auto,v6,v7,v8
bulk_index
\nstring
The index to write logs to in Elasticsearch.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be elasticsearch. \nAllowed enum values: elasticsearch
default: elasticsearch
Option 7
\nobject
The rsyslog destination forwards logs to an external rsyslog server over TCP or UDP using the syslog protocol.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
keepalive
\nint64
Optional socket keepalive duration in milliseconds.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. The value should always be rsyslog. \nAllowed enum values: rsyslog
default: rsyslog
Option 8
\nobject
The syslog_ng destination forwards logs to an external syslog-ng server over TCP or UDP using the syslog protocol.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
keepalive
\nint64
Optional socket keepalive duration in milliseconds.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. The value should always be syslog_ng. \nAllowed enum values: syslog_ng
default: syslog_ng
Option 9
\nobject
The azure_storage destination forwards logs to an Azure Blob Storage container.
blob_prefix
\nstring
Optional prefix for blobs written to the container.
container_name [required]
\nstring
The name of the Azure Blob Storage container to store logs in.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be azure_storage. \nAllowed enum values: azure_storage
default: azure_storage
Option 10
\nobject
The microsoft_sentinel destination forwards logs to Microsoft Sentinel.
client_id [required]
\nstring
Azure AD client ID used for authentication.
dcr_immutable_id [required]
\nstring
The immutable ID of the Data Collection Rule (DCR).
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
table [required]
\nstring
The name of the Log Analytics table where logs are sent.
tenant_id [required]
\nstring
Azure AD tenant ID.
type [required]
\nenum
The destination type. The value should always be microsoft_sentinel. \nAllowed enum values: microsoft_sentinel
default: microsoft_sentinel
Option 11
\nobject
The google_chronicle destination sends logs to Google Chronicle.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
customer_id [required]
\nstring
The Google Chronicle customer ID.
encoding
\nenum
The encoding format for the logs sent to Chronicle. \nAllowed enum values: json,raw_message
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
log_type
\nstring
The log type metadata associated with the Chronicle destination.
type [required]
\nenum
The destination type. The value should always be google_chronicle. \nAllowed enum values: google_chronicle
default: google_chronicle
Option 12
\nobject
The new_relic destination sends logs to the New Relic platform.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The New Relic region. \nAllowed enum values: us,eu
type [required]
\nenum
The destination type. The value should always be new_relic. \nAllowed enum values: new_relic
default: new_relic
Option 13
\nobject
The sentinel_one destination sends logs to SentinelOne.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The SentinelOne region to send logs to. \nAllowed enum values: us,eu,ca,data_set_us
type [required]
\nenum
The destination type. The value should always be sentinel_one. \nAllowed enum values: sentinel_one
default: sentinel_one
Option 14
\nobject
The opensearch destination writes logs to an OpenSearch cluster.
bulk_index
\nstring
The index to write logs to.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be opensearch. \nAllowed enum values: opensearch
default: opensearch
Option 15
\nobject
The amazon_opensearch destination writes logs to Amazon OpenSearch.
auth [required]
\nobject
Authentication settings for the Amazon OpenSearch destination.\nThe strategy field determines whether basic or AWS-based authentication is used.
assume_role
\nstring
The ARN of the role to assume (used with aws strategy).
aws_region
\nstring
AWS region
external_id
\nstring
External ID for the assumed role (used with aws strategy).
session_name
\nstring
Session name for the assumed role (used with aws strategy).
strategy [required]
\nenum
The authentication strategy to use. \nAllowed enum values: basic,aws
bulk_index
\nstring
The index to write logs to.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be amazon_opensearch. \nAllowed enum values: amazon_opensearch
default: amazon_opensearch
processors
\n[ <oneOf>]
A list of processors that transform or enrich log data.
Option 1
\nobject
The filter processor allows conditional processing of logs based on a Datadog search query. Logs that match the include query are passed through; others are discarded.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs should pass through the filter. Logs that match this query continue to downstream components; others are dropped.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be filter. \nAllowed enum values: filter
default: filter
Option 2
\nobject
The parse_json processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string.
field [required]
\nstring
The name of the log field that contains a JSON string.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be parse_json. \nAllowed enum values: parse_json
default: parse_json
Option 3
\nobject
The Quota Processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert.
drop_events [required]
\nboolean
If set to true, logs that matched the quota filter and sent after the quota has been met are dropped; only logs that did not match the filter query continue through the pipeline.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
ignore_when_missing_partitions
\nboolean
If true, the processor skips quota checks when partition fields are missing from the logs.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
name [required]
\nstring
Name of the quota.
overflow_action
\nenum
The action to take when the quota is exceeded. Options:
\ndrop: Drop the event.
no_action: Let the event pass through.
overflow_routing: Route to an overflow destination.
Allowed enum values: drop,no_action,overflow_routing
overrides
\n[object]
A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit.
fields [required]
\n[object]
A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
partition_fields
\n[string]
A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values.
type [required]
\nenum
The processor type. The value should always be quota. \nAllowed enum values: quota
default: quota
Option 4
\nobject
The add_fields processor adds static key-value fields to logs.
fields [required]
\n[object]
A list of static fields (key-value pairs) that is added to each log event processed by this component.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be add_fields. \nAllowed enum values: add_fields
default: add_fields
Option 5
\nobject
The remove_fields processor deletes specified fields from logs.
fields [required]
\n[string]
A list of field names to be removed from each log event.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
The PipelineRemoveFieldsProcessor inputs.
type [required]
\nenum
The processor type. The value should always be remove_fields. \nAllowed enum values: remove_fields
default: remove_fields
Option 6
\nobject
The rename_fields processor changes field names.
fields [required]
\n[object]
A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields.
destination [required]
\nstring
The field name to assign the renamed value to.
preserve_source [required]
\nboolean
Indicates whether the original field, that is received from the source, should be kept (true) or removed (false) after renaming.
source [required]
\nstring
The original field name in the log event that should be renamed.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be rename_fields. \nAllowed enum values: rename_fields
default: rename_fields
Option 7
\nobject
The generate_datadog_metrics processor creates custom metrics from logs and sends them to Datadog.\nMetrics can be counters, gauges, or distributions and optionally grouped by log fields.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
metrics [required]
\n[object]
Configuration for generating individual metrics.
group_by
\n[string]
Optional fields used to group the metric series.
include [required]
\nstring
Datadog filter query to match logs for metric generation.
metric_type [required]
\nenum
Type of metric to create. \nAllowed enum values: count,gauge,distribution
name [required]
\nstring
Name of the custom metric to be created.
value [required]
\n<oneOf>
Specifies how the value of the generated metric is computed.
Option 1
\nobject
Strategy that increments a generated metric by one for each matching event.
strategy [required]
\nenum
Increments the metric by 1 for each matching event. \nAllowed enum values: increment_by_one
Option 2
\nobject
Strategy that increments a generated metric based on the value of a log field.
field [required]
\nstring
Name of the log field containing the numeric value to increment the metric by.
strategy [required]
\nenum
Uses a numeric field in the log event as the metric increment. \nAllowed enum values: increment_by_field
type [required]
\nenum
The processor type. Always generate_datadog_metrics. \nAllowed enum values: generate_datadog_metrics
default: generate_datadog_metrics
Option 8
\nobject
The sample processor allows probabilistic sampling of logs at a fixed rate.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
percentage
\ndouble
The percentage of logs to sample.
rate
\nint64
Number of events to sample (1 in N).
type [required]
\nenum
The processor type. The value should always be sample. \nAllowed enum values: sample
default: sample
Option 9
\nobject
The parse_grok processor extracts structured fields from unstructured log messages using Grok patterns.
disable_library_rules
\nboolean
If set to true, disables the default Grok rules provided by Datadog.
id [required]
\nstring
A unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
rules [required]
\n[object]
The list of Grok parsing rules. If multiple matching rules are provided, they are evaluated in order. The first successful match is applied.
match_rules [required]
\n[object]
A list of Grok parsing rules that define how to extract fields from the source field.\nEach rule must contain a name and a valid Grok pattern.
name [required]
\nstring
The name of the rule.
rule [required]
\nstring
The definition of the Grok rule.
source [required]
\nstring
The name of the field in the log event to apply the Grok rules to.
support_rules [required]
\n[object]
A list of Grok helper rules that can be referenced by the parsing rules.
name [required]
\nstring
The name of the Grok helper rule.
rule [required]
\nstring
The definition of the Grok helper rule.
type [required]
\nenum
The processor type. The value should always be parse_grok. \nAllowed enum values: parse_grok
default: parse_grok
Option 10
\nobject
The sensitive_data_scanner processor detects and optionally redacts sensitive data in log events.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
rules [required]
\n[object]
A list of rules for identifying and acting on sensitive data patterns.
keyword_options
\nobject
Configuration for keywords used to reinforce sensitive data pattern detection.
keywords [required]
\n[string]
A list of keywords to match near the sensitive pattern.
proximity [required]
\nint64
Maximum number of tokens between a keyword and a sensitive value match.
name [required]
\nstring
A name identifying the rule.
on_match [required]
\n<oneOf>
Defines what action to take when sensitive data is matched.
Option 1
\nobject
Configuration for completely redacting matched sensitive data.
action [required]
\nenum
Action type that completely replaces the matched sensitive data with a fixed replacement string to remove all visibility. \nAllowed enum values: redact
options [required]
\nobject
Configuration for fully redacting sensitive data.
replace [required]
\nstring
The ObservabilityPipelineSensitiveDataScannerProcessorActionRedactOptions replace.
Option 2
\nobject
Configuration for hashing matched sensitive values.
action [required]
\nenum
Action type that replaces the matched sensitive data with a hashed representation, preserving structure while securing content. \nAllowed enum values: hash
options
\nobject
The ObservabilityPipelineSensitiveDataScannerProcessorActionHash options.
Option 3
\nobject
Configuration for partially redacting matched sensitive data.
action [required]
\nenum
Action type that redacts part of the sensitive data while preserving a configurable number of characters, typically used for masking purposes (e.g., show last 4 digits of a credit card). \nAllowed enum values: partial_redact
options [required]
\nobject
Controls how partial redaction is applied, including character count and direction.
characters [required]
\nint64
The ObservabilityPipelineSensitiveDataScannerProcessorActionPartialRedactOptions characters.
direction [required]
\nenum
Indicates whether to redact characters from the first or last part of the matched value. \nAllowed enum values: first,last
pattern [required]
\n<oneOf>
Pattern detection configuration for identifying sensitive data using either a custom regex or a library reference.
Option 1
\nobject
Defines a custom regex-based pattern for identifying sensitive data in logs.
options [required]
\nobject
Options for defining a custom regex pattern.
rule [required]
\nstring
A regular expression used to detect sensitive values. Must be a valid regex.
type [required]
\nenum
Indicates a custom regular expression is used for matching. \nAllowed enum values: custom
Option 2
\nobject
Specifies a pattern from Datadog’s sensitive data detection library to match known sensitive data types.
options [required]
\nobject
Options for selecting a predefined library pattern and enabling keyword support.
id [required]
\nstring
Identifier for a predefined pattern from the sensitive data scanner pattern library.
use_recommended_keywords
\nboolean
Whether to augment the pattern with recommended keywords (optional).
type [required]
\nenum
Indicates that a predefined library pattern is used. \nAllowed enum values: library
scope [required]
\n<oneOf>
Determines which parts of the log the pattern-matching rule should be applied to.
Option 1
\nobject
Includes only specific fields for sensitive data scanning.
options [required]
\nobject
Fields to which the scope rule applies.
fields [required]
\n[string]
The ObservabilityPipelineSensitiveDataScannerProcessorScopeOptions fields.
target [required]
\nenum
Applies the rule only to included fields. \nAllowed enum values: include
Option 2
\nobject
Excludes specific fields from sensitive data scanning.
options [required]
\nobject
Fields to which the scope rule applies.
fields [required]
\n[string]
The ObservabilityPipelineSensitiveDataScannerProcessorScopeOptions fields.
target [required]
\nenum
Excludes specific fields from processing. \nAllowed enum values: exclude
Option 3
\nobject
Applies scanning across all available fields.
target [required]
\nenum
Applies the rule to all fields. \nAllowed enum values: all
tags [required]
\n[string]
Tags assigned to this rule for filtering and classification.
type [required]
\nenum
The processor type. The value should always be sensitive_data_scanner. \nAllowed enum values: sensitive_data_scanner
default: sensitive_data_scanner
Option 11
\nobject
The ocsf_mapper processor transforms logs into the OCSF schema using a predefined mapping configuration.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mappings [required]
\n[object]
A list of mapping rules to convert events to the OCSF format.
include [required]
\nstring
A Datadog search query used to select the logs that this mapping should apply to.
mapping [required]
\n<oneOf>
Defines a single mapping rule for transforming logs into the OCSF schema.
Option 1
\nenum
Predefined library mappings for common log formats. \nAllowed enum values: CloudTrail Account Change,GCP Cloud Audit CreateBucket,GCP Cloud Audit CreateSink,GCP Cloud Audit SetIamPolicy,GCP Cloud Audit UpdateSink,Github Audit Log API Activity,Google Workspace Admin Audit addPrivilege,Microsoft 365 Defender Incident,Microsoft 365 Defender UserLoggedIn,Okta System Log Authentication,Palo Alto Networks Firewall Traffic
type [required]
\nenum
The processor type. The value should always be ocsf_mapper. \nAllowed enum values: ocsf_mapper
default: ocsf_mapper
Option 12
\nobject
The add_env_vars processor adds environment variable values to log events.
id [required]
\nstring
The unique identifier for this component. Used to reference this processor in the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
type [required]
\nenum
The processor type. The value should always be add_env_vars. \nAllowed enum values: add_env_vars
default: add_env_vars
variables [required]
\n[object]
A list of environment variable mappings to apply to log fields.
field [required]
\nstring
The target field in the log event.
name [required]
\nstring
The name of the environment variable to read.
Option 13
\nobject
The dedupe processor removes duplicate fields in log events.
fields [required]
\n[string]
A list of log field paths to check for duplicates.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mode [required]
\nenum
The deduplication mode to apply to the fields. \nAllowed enum values: match,ignore
type [required]
\nenum
The processor type. The value should always be dedupe. \nAllowed enum values: dedupe
default: dedupe
Option 14
\nobject
The enrichment_table processor enriches logs using a static CSV file or GeoIP database.
file
\nobject
Defines a static enrichment table loaded from a CSV file.
encoding [required]
\nobject
File encoding format.
delimiter [required]
\nstring
The encoding delimiter.
includes_headers [required]
\nboolean
The encoding includes_headers.
type [required]
\nenum
Specifies the encoding format (e.g., CSV) used for enrichment tables. \nAllowed enum values: csv
key [required]
\n[object]
Key fields used to look up enrichment values.
column [required]
\nstring
The items column.
comparison [required]
\nenum
Defines how to compare key fields for enrichment table lookups. \nAllowed enum values: equals
field [required]
\nstring
The items field.
path [required]
\nstring
Path to the CSV file.
schema [required]
\n[object]
Schema defining column names and their types.
column [required]
\nstring
The items column.
type [required]
\nenum
Declares allowed data types for enrichment table columns. \nAllowed enum values: string,boolean,integer,float,date,timestamp
geoip
\nobject
Uses a GeoIP database to enrich logs based on an IP field.
key_field [required]
\nstring
Path to the IP field in the log.
locale [required]
\nstring
Locale used to resolve geographical names.
path [required]
\nstring
Path to the GeoIP database file.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
target [required]
\nstring
Path where enrichment results should be stored in the log.
type [required]
\nenum
The processor type. The value should always be enrichment_table. \nAllowed enum values: enrichment_table
default: enrichment_table
Option 15
\nobject
The reduce processor aggregates and merges logs based on matching keys and merge strategies.
group_by [required]
\n[string]
A list of fields used to group log events for merging.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
merge_strategies [required]
\n[object]
List of merge strategies defining how values from grouped events should be combined.
path [required]
\nstring
The field path in the log event.
strategy [required]
\nenum
The merge strategy to apply. \nAllowed enum values: discard,retain,sum,max,min,array,concat,concat_newline,concat_raw,shortest_array,longest_array,flat_unique
type [required]
\nenum
The processor type. The value should always be reduce. \nAllowed enum values: reduce
default: reduce
Option 16
\nobject
The throttle processor limits the number of events that pass through over a given time window.
group_by
\n[string]
Optional list of fields used to group events before the threshold has been reached.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
threshold [required]
\nint64
the number of events allowed in a given time window. Events sent after the threshold has been reached, are dropped.
type [required]
\nenum
The processor type. The value should always be throttle. \nAllowed enum values: throttle
default: throttle
window [required]
\ndouble
The time window in seconds over which the threshold applies.
sources [required]
\n[ <oneOf>]
A list of configured data sources for the pipeline.
Option 1
\nobject
The kafka source ingests data from Apache Kafka topics.
group_id [required]
\nstring
Consumer group ID used by the Kafka client.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
librdkafka_options
\n[object]
Optional list of advanced Kafka client configuration options, defined as key-value pairs.
name [required]
\nstring
The name of the librdkafka configuration option to set.
value [required]
\nstring
The value assigned to the specified librdkafka configuration option.
sasl
\nobject
Specifies the SASL mechanism for authenticating with a Kafka cluster.
mechanism
\nenum
SASL mechanism used for Kafka authentication. \nAllowed enum values: PLAIN,SCRAM-SHA-256,SCRAM-SHA-512
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
topics [required]
\n[string]
A list of Kafka topic names to subscribe to. The source ingests messages from each topic specified.
type [required]
\nenum
The source type. The value should always be kafka. \nAllowed enum values: kafka
default: kafka
Option 2
\nobject
The datadog_agent source collects logs from the Datadog Agent.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be datadog_agent. \nAllowed enum values: datadog_agent
default: datadog_agent
Option 3
\nobject
The splunk_tcp source receives logs from a Splunk Universal Forwarder over TCP.\nTLS is supported for secure transmission.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always splunk_tcp. \nAllowed enum values: splunk_tcp
default: splunk_tcp
Option 4
\nobject
The splunk_hec source implements the Splunk HTTP Event Collector (HEC) API.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always splunk_hec. \nAllowed enum values: splunk_hec
default: splunk_hec
Option 5
\nobject
The amazon_s3 source ingests logs from an Amazon S3 bucket.\nIt supports AWS authentication and TLS encryption.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
region [required]
\nstring
AWS region where the S3 bucket resides.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always amazon_s3. \nAllowed enum values: amazon_s3
default: amazon_s3
Option 6
\nobject
The fluentd source ingests logs from a Fluentd-compatible service.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be `fluentd. \nAllowed enum values: fluentd
default: fluentd
Option 7
\nobject
The fluent_bit source ingests logs from Fluent Bit.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be fluent_bit. \nAllowed enum values: fluent_bit
default: fluent_bit
Option 8
\nobject
The http_server source collects logs over HTTP POST from external services.
auth_strategy [required]
\nenum
HTTP authentication method. \nAllowed enum values: none,plain
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
Unique ID for the HTTP server source.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be http_server. \nAllowed enum values: http_server
default: http_server
Option 9
\nobject
The sumo_logic source receives logs from Sumo Logic collectors.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
type [required]
\nenum
The source type. The value should always be sumo_logic. \nAllowed enum values: sumo_logic
default: sumo_logic
Option 10
\nobject
The rsyslog source listens for logs over TCP or UDP from an rsyslog server using the syslog protocol.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
mode [required]
\nenum
Protocol used by the syslog source to receive messages. \nAllowed enum values: tcp,udp
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be rsyslog. \nAllowed enum values: rsyslog
default: rsyslog
Option 11
\nobject
The syslog_ng source listens for logs over TCP or UDP from a syslog-ng server using the syslog protocol.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
mode [required]
\nenum
Protocol used by the syslog source to receive messages. \nAllowed enum values: tcp,udp
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be syslog_ng. \nAllowed enum values: syslog_ng
default: syslog_ng
Option 12
\nobject
The amazon_data_firehose source ingests logs from AWS Data Firehose.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be amazon_data_firehose. \nAllowed enum values: amazon_data_firehose
default: amazon_data_firehose
Option 13
\nobject
The google_pubsub source ingests logs from a Google Cloud Pub/Sub subscription.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
project [required]
\nstring
The GCP project ID that owns the Pub/Sub subscription.
subscription [required]
\nstring
The Pub/Sub subscription name from which messages are consumed.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be google_pubsub. \nAllowed enum values: google_pubsub
default: google_pubsub
Option 14
\nobject
The http_client source scrapes logs from HTTP endpoints at regular intervals.
auth_strategy
\nenum
Optional authentication strategy for HTTP requests. \nAllowed enum values: basic,bearer
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
scrape_interval_secs
\nint64
The interval (in seconds) between HTTP scrape requests.
scrape_timeout_secs
\nint64
The timeout (in seconds) for each scrape request.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be http_client. \nAllowed enum values: http_client
default: http_client
Option 15
\nobject
The logstash source ingests logs from a Logstash forwarder.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be logstash. \nAllowed enum values: logstash
default: logstash
name [required]
\nstring
Name of the pipeline.
id [required]
\nstring
Unique identifier for the pipeline.
type [required]
\nstring
The resource type identifier. For pipeline resources, this should always be set to pipelines.
default: pipelines
meta
\nobject
Metadata about the response.
totalCount
\nint64
The total number of pipelines.
data [required]
\n[object]
The schema data.
attributes [required]
\nobject
Defines the pipeline’s name and its components (sources, processors, and destinations).
config [required]
\nobject
Specifies the pipeline's configuration, including its sources, processors, and destinations.
destinations [required]
\n[ <oneOf>]
A list of destination components where processed logs are sent.
Option 1
\nobject
The datadog_logs destination forwards logs to Datadog Log Management.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be datadog_logs. \nAllowed enum values: datadog_logs
default: datadog_logs
Option 2
\nobject
The amazon_s3 destination sends your logs in Datadog-rehydratable format to an Amazon S3 bucket for archiving.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
bucket [required]
\nstring
S3 bucket name.
id [required]
\nstring
Unique identifier for the destination component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
key_prefix
\nstring
Optional prefix for object keys.
region [required]
\nstring
AWS region of the S3 bucket.
storage_class [required]
\nenum
S3 storage class. \nAllowed enum values: STANDARD,REDUCED_REDUNDANCY,INTELLIGENT_TIERING,STANDARD_IA,EXPRESS_ONEZONE,ONEZONE_IA,GLACIER,GLACIER_IR,DEEP_ARCHIVE
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. Always amazon_s3. \nAllowed enum values: amazon_s3
default: amazon_s3
Option 3
\nobject
The google_cloud_storage destination stores logs in a Google Cloud Storage (GCS) bucket.\nIt requires a bucket name, GCP authentication, and metadata fields.
acl [required]
\nenum
Access control list setting for objects written to the bucket. \nAllowed enum values: private,project-private,public-read,authenticated-read,bucket-owner-read,bucket-owner-full-control
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
bucket [required]
\nstring
Name of the GCS bucket.
id [required]
\nstring
Unique identifier for the destination component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
key_prefix
\nstring
Optional prefix for object keys within the GCS bucket.
metadata
\n[object]
Custom metadata to attach to each object uploaded to the GCS bucket.
name [required]
\nstring
The metadata key.
value [required]
\nstring
The metadata value.
storage_class [required]
\nenum
Storage class used for objects stored in GCS. \nAllowed enum values: STANDARD,NEARLINE,COLDLINE,ARCHIVE
type [required]
\nenum
The destination type. Always google_cloud_storage. \nAllowed enum values: google_cloud_storage
default: google_cloud_storage
Option 4
\nobject
The splunk_hec destination forwards logs to Splunk using the HTTP Event Collector (HEC).
auto_extract_timestamp
\nboolean
If true, Splunk tries to extract timestamps from incoming log events.\nIf false, Splunk assigns the time the event was received.
encoding
\nenum
Encoding format for log events. \nAllowed enum values: json,raw_message
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
index
\nstring
Optional name of the Splunk index where logs are written.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
sourcetype
\nstring
The Splunk sourcetype to assign to log events.
type [required]
\nenum
The destination type. Always splunk_hec. \nAllowed enum values: splunk_hec
default: splunk_hec
Option 5
\nobject
The sumo_logic destination forwards logs to Sumo Logic.
encoding
\nenum
The output encoding format. \nAllowed enum values: json,raw_message,logfmt
header_custom_fields
\n[object]
A list of custom headers to include in the request to Sumo Logic.
name [required]
\nstring
The header field name.
value [required]
\nstring
The header field value.
header_host_name
\nstring
Optional override for the host name header.
header_source_category
\nstring
Optional override for the source category header.
header_source_name
\nstring
Optional override for the source name header.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be sumo_logic. \nAllowed enum values: sumo_logic
default: sumo_logic
Option 6
\nobject
The elasticsearch destination writes logs to an Elasticsearch cluster.
api_version
\nenum
The Elasticsearch API version to use. Set to auto to auto-detect. \nAllowed enum values: auto,v6,v7,v8
bulk_index
\nstring
The index to write logs to in Elasticsearch.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be elasticsearch. \nAllowed enum values: elasticsearch
default: elasticsearch
Option 7
\nobject
The rsyslog destination forwards logs to an external rsyslog server over TCP or UDP using the syslog protocol.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
keepalive
\nint64
Optional socket keepalive duration in milliseconds.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. The value should always be rsyslog. \nAllowed enum values: rsyslog
default: rsyslog
Option 8
\nobject
The syslog_ng destination forwards logs to an external syslog-ng server over TCP or UDP using the syslog protocol.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
keepalive
\nint64
Optional socket keepalive duration in milliseconds.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. The value should always be syslog_ng. \nAllowed enum values: syslog_ng
default: syslog_ng
Option 9
\nobject
The azure_storage destination forwards logs to an Azure Blob Storage container.
blob_prefix
\nstring
Optional prefix for blobs written to the container.
container_name [required]
\nstring
The name of the Azure Blob Storage container to store logs in.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be azure_storage. \nAllowed enum values: azure_storage
default: azure_storage
Option 10
\nobject
The microsoft_sentinel destination forwards logs to Microsoft Sentinel.
client_id [required]
\nstring
Azure AD client ID used for authentication.
dcr_immutable_id [required]
\nstring
The immutable ID of the Data Collection Rule (DCR).
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
table [required]
\nstring
The name of the Log Analytics table where logs are sent.
tenant_id [required]
\nstring
Azure AD tenant ID.
type [required]
\nenum
The destination type. The value should always be microsoft_sentinel. \nAllowed enum values: microsoft_sentinel
default: microsoft_sentinel
Option 11
\nobject
The google_chronicle destination sends logs to Google Chronicle.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
customer_id [required]
\nstring
The Google Chronicle customer ID.
encoding
\nenum
The encoding format for the logs sent to Chronicle. \nAllowed enum values: json,raw_message
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
log_type
\nstring
The log type metadata associated with the Chronicle destination.
type [required]
\nenum
The destination type. The value should always be google_chronicle. \nAllowed enum values: google_chronicle
default: google_chronicle
Option 12
\nobject
The new_relic destination sends logs to the New Relic platform.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The New Relic region. \nAllowed enum values: us,eu
type [required]
\nenum
The destination type. The value should always be new_relic. \nAllowed enum values: new_relic
default: new_relic
Option 13
\nobject
The sentinel_one destination sends logs to SentinelOne.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The SentinelOne region to send logs to. \nAllowed enum values: us,eu,ca,data_set_us
type [required]
\nenum
The destination type. The value should always be sentinel_one. \nAllowed enum values: sentinel_one
default: sentinel_one
Option 14
\nobject
The opensearch destination writes logs to an OpenSearch cluster.
bulk_index
\nstring
The index to write logs to.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be opensearch. \nAllowed enum values: opensearch
default: opensearch
Option 15
\nobject
The amazon_opensearch destination writes logs to Amazon OpenSearch.
auth [required]
\nobject
Authentication settings for the Amazon OpenSearch destination.\nThe strategy field determines whether basic or AWS-based authentication is used.
assume_role
\nstring
The ARN of the role to assume (used with aws strategy).
aws_region
\nstring
AWS region
external_id
\nstring
External ID for the assumed role (used with aws strategy).
session_name
\nstring
Session name for the assumed role (used with aws strategy).
strategy [required]
\nenum
The authentication strategy to use. \nAllowed enum values: basic,aws
bulk_index
\nstring
The index to write logs to.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be amazon_opensearch. \nAllowed enum values: amazon_opensearch
default: amazon_opensearch
processors
\n[ <oneOf>]
A list of processors that transform or enrich log data.
Option 1
\nobject
The filter processor allows conditional processing of logs based on a Datadog search query. Logs that match the include query are passed through; others are discarded.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs should pass through the filter. Logs that match this query continue to downstream components; others are dropped.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be filter. \nAllowed enum values: filter
default: filter
Option 2
\nobject
The parse_json processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string.
field [required]
\nstring
The name of the log field that contains a JSON string.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be parse_json. \nAllowed enum values: parse_json
default: parse_json
Option 3
\nobject
The Quota Processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert.
drop_events [required]
\nboolean
If set to true, logs that matched the quota filter and sent after the quota has been met are dropped; only logs that did not match the filter query continue through the pipeline.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
ignore_when_missing_partitions
\nboolean
If true, the processor skips quota checks when partition fields are missing from the logs.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
name [required]
\nstring
Name of the quota.
overflow_action
\nenum
The action to take when the quota is exceeded. Options:
\ndrop: Drop the event.
no_action: Let the event pass through.
overflow_routing: Route to an overflow destination.
Allowed enum values: drop,no_action,overflow_routing
overrides
\n[object]
A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit.
fields [required]
\n[object]
A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
partition_fields
\n[string]
A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values.
type [required]
\nenum
The processor type. The value should always be quota. \nAllowed enum values: quota
default: quota
Option 4
\nobject
The add_fields processor adds static key-value fields to logs.
fields [required]
\n[object]
A list of static fields (key-value pairs) that is added to each log event processed by this component.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be add_fields. \nAllowed enum values: add_fields
default: add_fields
Option 5
\nobject
The remove_fields processor deletes specified fields from logs.
fields [required]
\n[string]
A list of field names to be removed from each log event.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
The PipelineRemoveFieldsProcessor inputs.
type [required]
\nenum
The processor type. The value should always be remove_fields. \nAllowed enum values: remove_fields
default: remove_fields
Option 6
\nobject
The rename_fields processor changes field names.
fields [required]
\n[object]
A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields.
destination [required]
\nstring
The field name to assign the renamed value to.
preserve_source [required]
\nboolean
Indicates whether the original field, that is received from the source, should be kept (true) or removed (false) after renaming.
source [required]
\nstring
The original field name in the log event that should be renamed.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be rename_fields. \nAllowed enum values: rename_fields
default: rename_fields
Option 7
\nobject
The generate_datadog_metrics processor creates custom metrics from logs and sends them to Datadog.\nMetrics can be counters, gauges, or distributions and optionally grouped by log fields.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
metrics [required]
\n[object]
Configuration for generating individual metrics.
group_by
\n[string]
Optional fields used to group the metric series.
include [required]
\nstring
Datadog filter query to match logs for metric generation.
metric_type [required]
\nenum
Type of metric to create. \nAllowed enum values: count,gauge,distribution
name [required]
\nstring
Name of the custom metric to be created.
value [required]
\n<oneOf>
Specifies how the value of the generated metric is computed.
Option 1
\nobject
Strategy that increments a generated metric by one for each matching event.
strategy [required]
\nenum
Increments the metric by 1 for each matching event. \nAllowed enum values: increment_by_one
Option 2
\nobject
Strategy that increments a generated metric based on the value of a log field.
field [required]
\nstring
Name of the log field containing the numeric value to increment the metric by.
strategy [required]
\nenum
Uses a numeric field in the log event as the metric increment. \nAllowed enum values: increment_by_field
type [required]
\nenum
The processor type. Always generate_datadog_metrics. \nAllowed enum values: generate_datadog_metrics
default: generate_datadog_metrics
Option 8
\nobject
The sample processor allows probabilistic sampling of logs at a fixed rate.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
percentage
\ndouble
The percentage of logs to sample.
rate
\nint64
Number of events to sample (1 in N).
type [required]
\nenum
The processor type. The value should always be sample. \nAllowed enum values: sample
default: sample
Option 9
\nobject
The parse_grok processor extracts structured fields from unstructured log messages using Grok patterns.
disable_library_rules
\nboolean
If set to true, disables the default Grok rules provided by Datadog.
id [required]
\nstring
A unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
rules [required]
\n[object]
The list of Grok parsing rules. If multiple matching rules are provided, they are evaluated in order. The first successful match is applied.
match_rules [required]
\n[object]
A list of Grok parsing rules that define how to extract fields from the source field.\nEach rule must contain a name and a valid Grok pattern.
name [required]
\nstring
The name of the rule.
rule [required]
\nstring
The definition of the Grok rule.
source [required]
\nstring
The name of the field in the log event to apply the Grok rules to.
support_rules [required]
\n[object]
A list of Grok helper rules that can be referenced by the parsing rules.
name [required]
\nstring
The name of the Grok helper rule.
rule [required]
\nstring
The definition of the Grok helper rule.
type [required]
\nenum
The processor type. The value should always be parse_grok. \nAllowed enum values: parse_grok
default: parse_grok
Option 10
\nobject
The sensitive_data_scanner processor detects and optionally redacts sensitive data in log events.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
rules [required]
\n[object]
A list of rules for identifying and acting on sensitive data patterns.
keyword_options
\nobject
Configuration for keywords used to reinforce sensitive data pattern detection.
keywords [required]
\n[string]
A list of keywords to match near the sensitive pattern.
proximity [required]
\nint64
Maximum number of tokens between a keyword and a sensitive value match.
name [required]
\nstring
A name identifying the rule.
on_match [required]
\n<oneOf>
Defines what action to take when sensitive data is matched.
Option 1
\nobject
Configuration for completely redacting matched sensitive data.
action [required]
\nenum
Action type that completely replaces the matched sensitive data with a fixed replacement string to remove all visibility. \nAllowed enum values: redact
options [required]
\nobject
Configuration for fully redacting sensitive data.
replace [required]
\nstring
The ObservabilityPipelineSensitiveDataScannerProcessorActionRedactOptions replace.
Option 2
\nobject
Configuration for hashing matched sensitive values.
action [required]
\nenum
Action type that replaces the matched sensitive data with a hashed representation, preserving structure while securing content. \nAllowed enum values: hash
options
\nobject
The ObservabilityPipelineSensitiveDataScannerProcessorActionHash options.
Option 3
\nobject
Configuration for partially redacting matched sensitive data.
action [required]
\nenum
Action type that redacts part of the sensitive data while preserving a configurable number of characters, typically used for masking purposes (e.g., show last 4 digits of a credit card). \nAllowed enum values: partial_redact
options [required]
\nobject
Controls how partial redaction is applied, including character count and direction.
characters [required]
\nint64
The ObservabilityPipelineSensitiveDataScannerProcessorActionPartialRedactOptions characters.
direction [required]
\nenum
Indicates whether to redact characters from the first or last part of the matched value. \nAllowed enum values: first,last
pattern [required]
\n<oneOf>
Pattern detection configuration for identifying sensitive data using either a custom regex or a library reference.
Option 1
\nobject
Defines a custom regex-based pattern for identifying sensitive data in logs.
options [required]
\nobject
Options for defining a custom regex pattern.
rule [required]
\nstring
A regular expression used to detect sensitive values. Must be a valid regex.
type [required]
\nenum
Indicates a custom regular expression is used for matching. \nAllowed enum values: custom
Option 2
\nobject
Specifies a pattern from Datadog’s sensitive data detection library to match known sensitive data types.
options [required]
\nobject
Options for selecting a predefined library pattern and enabling keyword support.
id [required]
\nstring
Identifier for a predefined pattern from the sensitive data scanner pattern library.
use_recommended_keywords
\nboolean
Whether to augment the pattern with recommended keywords (optional).
type [required]
\nenum
Indicates that a predefined library pattern is used. \nAllowed enum values: library
scope [required]
\n<oneOf>
Determines which parts of the log the pattern-matching rule should be applied to.
Option 1
\nobject
Includes only specific fields for sensitive data scanning.
options [required]
\nobject
Fields to which the scope rule applies.
fields [required]
\n[string]
The ObservabilityPipelineSensitiveDataScannerProcessorScopeOptions fields.
target [required]
\nenum
Applies the rule only to included fields. \nAllowed enum values: include
Option 2
\nobject
Excludes specific fields from sensitive data scanning.
options [required]
\nobject
Fields to which the scope rule applies.
fields [required]
\n[string]
The ObservabilityPipelineSensitiveDataScannerProcessorScopeOptions fields.
target [required]
\nenum
Excludes specific fields from processing. \nAllowed enum values: exclude
Option 3
\nobject
Applies scanning across all available fields.
target [required]
\nenum
Applies the rule to all fields. \nAllowed enum values: all
tags [required]
\n[string]
Tags assigned to this rule for filtering and classification.
type [required]
\nenum
The processor type. The value should always be sensitive_data_scanner. \nAllowed enum values: sensitive_data_scanner
default: sensitive_data_scanner
Option 11
\nobject
The ocsf_mapper processor transforms logs into the OCSF schema using a predefined mapping configuration.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mappings [required]
\n[object]
A list of mapping rules to convert events to the OCSF format.
include [required]
\nstring
A Datadog search query used to select the logs that this mapping should apply to.
mapping [required]
\n<oneOf>
Defines a single mapping rule for transforming logs into the OCSF schema.
Option 1
\nenum
Predefined library mappings for common log formats. \nAllowed enum values: CloudTrail Account Change,GCP Cloud Audit CreateBucket,GCP Cloud Audit CreateSink,GCP Cloud Audit SetIamPolicy,GCP Cloud Audit UpdateSink,Github Audit Log API Activity,Google Workspace Admin Audit addPrivilege,Microsoft 365 Defender Incident,Microsoft 365 Defender UserLoggedIn,Okta System Log Authentication,Palo Alto Networks Firewall Traffic
type [required]
\nenum
The processor type. The value should always be ocsf_mapper. \nAllowed enum values: ocsf_mapper
default: ocsf_mapper
Option 12
\nobject
The add_env_vars processor adds environment variable values to log events.
id [required]
\nstring
The unique identifier for this component. Used to reference this processor in the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
type [required]
\nenum
The processor type. The value should always be add_env_vars. \nAllowed enum values: add_env_vars
default: add_env_vars
variables [required]
\n[object]
A list of environment variable mappings to apply to log fields.
field [required]
\nstring
The target field in the log event.
name [required]
\nstring
The name of the environment variable to read.
Option 13
\nobject
The dedupe processor removes duplicate fields in log events.
fields [required]
\n[string]
A list of log field paths to check for duplicates.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mode [required]
\nenum
The deduplication mode to apply to the fields. \nAllowed enum values: match,ignore
type [required]
\nenum
The processor type. The value should always be dedupe. \nAllowed enum values: dedupe
default: dedupe
Option 14
\nobject
The enrichment_table processor enriches logs using a static CSV file or GeoIP database.
file
\nobject
Defines a static enrichment table loaded from a CSV file.
encoding [required]
\nobject
File encoding format.
delimiter [required]
\nstring
The encoding delimiter.
includes_headers [required]
\nboolean
The encoding includes_headers.
type [required]
\nenum
Specifies the encoding format (e.g., CSV) used for enrichment tables. \nAllowed enum values: csv
key [required]
\n[object]
Key fields used to look up enrichment values.
column [required]
\nstring
The items column.
comparison [required]
\nenum
Defines how to compare key fields for enrichment table lookups. \nAllowed enum values: equals
field [required]
\nstring
The items field.
path [required]
\nstring
Path to the CSV file.
schema [required]
\n[object]
Schema defining column names and their types.
column [required]
\nstring
The items column.
type [required]
\nenum
Declares allowed data types for enrichment table columns. \nAllowed enum values: string,boolean,integer,float,date,timestamp
geoip
\nobject
Uses a GeoIP database to enrich logs based on an IP field.
key_field [required]
\nstring
Path to the IP field in the log.
locale [required]
\nstring
Locale used to resolve geographical names.
path [required]
\nstring
Path to the GeoIP database file.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
target [required]
\nstring
Path where enrichment results should be stored in the log.
type [required]
\nenum
The processor type. The value should always be enrichment_table. \nAllowed enum values: enrichment_table
default: enrichment_table
Option 15
\nobject
The reduce processor aggregates and merges logs based on matching keys and merge strategies.
group_by [required]
\n[string]
A list of fields used to group log events for merging.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
merge_strategies [required]
\n[object]
List of merge strategies defining how values from grouped events should be combined.
path [required]
\nstring
The field path in the log event.
strategy [required]
\nenum
The merge strategy to apply. \nAllowed enum values: discard,retain,sum,max,min,array,concat,concat_newline,concat_raw,shortest_array,longest_array,flat_unique
type [required]
\nenum
The processor type. The value should always be reduce. \nAllowed enum values: reduce
default: reduce
Option 16
\nobject
The throttle processor limits the number of events that pass through over a given time window.
group_by
\n[string]
Optional list of fields used to group events before the threshold has been reached.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
threshold [required]
\nint64
the number of events allowed in a given time window. Events sent after the threshold has been reached, are dropped.
type [required]
\nenum
The processor type. The value should always be throttle. \nAllowed enum values: throttle
default: throttle
window [required]
\ndouble
The time window in seconds over which the threshold applies.
Option 17
\nobject
The datadog_tags processor includes or excludes specific Datadog tags in your logs.
action [required]
\nenum
The action to take on tags with matching keys. \nAllowed enum values: include,exclude
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
keys [required]
\n[string]
A list of tag keys.
mode [required]
\nenum
The processing mode. \nAllowed enum values: filter
type [required]
\nenum
The processor type. The value should always be datadog_tags. \nAllowed enum values: datadog_tags
default: datadog_tags
sources [required]
\n[ <oneOf>]
A list of configured data sources for the pipeline.
Option 1
\nobject
The kafka source ingests data from Apache Kafka topics.
group_id [required]
\nstring
Consumer group ID used by the Kafka client.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
librdkafka_options
\n[object]
Optional list of advanced Kafka client configuration options, defined as key-value pairs.
name [required]
\nstring
The name of the librdkafka configuration option to set.
value [required]
\nstring
The value assigned to the specified librdkafka configuration option.
sasl
\nobject
Specifies the SASL mechanism for authenticating with a Kafka cluster.
mechanism
\nenum
SASL mechanism used for Kafka authentication. \nAllowed enum values: PLAIN,SCRAM-SHA-256,SCRAM-SHA-512
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
topics [required]
\n[string]
A list of Kafka topic names to subscribe to. The source ingests messages from each topic specified.
type [required]
\nenum
The source type. The value should always be kafka. \nAllowed enum values: kafka
default: kafka
Option 2
\nobject
The datadog_agent source collects logs from the Datadog Agent.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be datadog_agent. \nAllowed enum values: datadog_agent
default: datadog_agent
Option 3
\nobject
The splunk_tcp source receives logs from a Splunk Universal Forwarder over TCP.\nTLS is supported for secure transmission.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always splunk_tcp. \nAllowed enum values: splunk_tcp
default: splunk_tcp
Option 4
\nobject
The splunk_hec source implements the Splunk HTTP Event Collector (HEC) API.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always splunk_hec. \nAllowed enum values: splunk_hec
default: splunk_hec
Option 5
\nobject
The amazon_s3 source ingests logs from an Amazon S3 bucket.\nIt supports AWS authentication and TLS encryption.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
region [required]
\nstring
AWS region where the S3 bucket resides.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always amazon_s3. \nAllowed enum values: amazon_s3
default: amazon_s3
Option 6
\nobject
The fluentd source ingests logs from a Fluentd-compatible service.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be `fluentd. \nAllowed enum values: fluentd
default: fluentd
Option 7
\nobject
The fluent_bit source ingests logs from Fluent Bit.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be fluent_bit. \nAllowed enum values: fluent_bit
default: fluent_bit
Option 8
\nobject
The http_server source collects logs over HTTP POST from external services.
auth_strategy [required]
\nenum
HTTP authentication method. \nAllowed enum values: none,plain
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
Unique ID for the HTTP server source.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be http_server. \nAllowed enum values: http_server
default: http_server
Option 9
\nobject
The sumo_logic source receives logs from Sumo Logic collectors.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
type [required]
\nenum
The source type. The value should always be sumo_logic. \nAllowed enum values: sumo_logic
default: sumo_logic
Option 10
\nobject
The rsyslog source listens for logs over TCP or UDP from an rsyslog server using the syslog protocol.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
mode [required]
\nenum
Protocol used by the syslog source to receive messages. \nAllowed enum values: tcp,udp
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be rsyslog. \nAllowed enum values: rsyslog
default: rsyslog
Option 11
\nobject
The syslog_ng source listens for logs over TCP or UDP from a syslog-ng server using the syslog protocol.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
mode [required]
\nenum
Protocol used by the syslog source to receive messages. \nAllowed enum values: tcp,udp
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be syslog_ng. \nAllowed enum values: syslog_ng
default: syslog_ng
Option 12
\nobject
The amazon_data_firehose source ingests logs from AWS Data Firehose.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be amazon_data_firehose. \nAllowed enum values: amazon_data_firehose
default: amazon_data_firehose
Option 13
\nobject
The google_pubsub source ingests logs from a Google Cloud Pub/Sub subscription.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
project [required]
\nstring
The GCP project ID that owns the Pub/Sub subscription.
subscription [required]
\nstring
The Pub/Sub subscription name from which messages are consumed.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be google_pubsub. \nAllowed enum values: google_pubsub
default: google_pubsub
Option 14
\nobject
The http_client source scrapes logs from HTTP endpoints at regular intervals.
auth_strategy
\nenum
Optional authentication strategy for HTTP requests. \nAllowed enum values: basic,bearer
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
scrape_interval_secs
\nint64
The interval (in seconds) between HTTP scrape requests.
scrape_timeout_secs
\nint64
The timeout (in seconds) for each scrape request.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be http_client. \nAllowed enum values: http_client
default: http_client
Option 15
\nobject
The logstash source ingests logs from a Logstash forwarder.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be logstash. \nAllowed enum values: logstash
default: logstash
name [required]
\nstring
Name of the pipeline.
id [required]
\nstring
Unique identifier for the pipeline.
type [required]
\nstring
The resource type identifier. For pipeline resources, this should always be set to pipelines.
default: pipelines
meta
\nobject
Metadata about the response.
totalCount
\nint64
The total number of pipelines.
data [required]
\nobject
Contains the pipeline’s ID, type, and configuration attributes.
attributes [required]
\nobject
Defines the pipeline’s name and its components (sources, processors, and destinations).
config [required]
\nobject
Specifies the pipeline's configuration, including its sources, processors, and destinations.
destinations [required]
\n[ <oneOf>]
A list of destination components where processed logs are sent.
Option 1
\nobject
The datadog_logs destination forwards logs to Datadog Log Management.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be datadog_logs. \nAllowed enum values: datadog_logs
default: datadog_logs
Option 2
\nobject
The amazon_s3 destination sends your logs in Datadog-rehydratable format to an Amazon S3 bucket for archiving.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
bucket [required]
\nstring
S3 bucket name.
id [required]
\nstring
Unique identifier for the destination component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
key_prefix
\nstring
Optional prefix for object keys.
region [required]
\nstring
AWS region of the S3 bucket.
storage_class [required]
\nenum
S3 storage class. \nAllowed enum values: STANDARD,REDUCED_REDUNDANCY,INTELLIGENT_TIERING,STANDARD_IA,EXPRESS_ONEZONE,ONEZONE_IA,GLACIER,GLACIER_IR,DEEP_ARCHIVE
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. Always amazon_s3. \nAllowed enum values: amazon_s3
default: amazon_s3
Option 3
\nobject
The google_cloud_storage destination stores logs in a Google Cloud Storage (GCS) bucket.\nIt requires a bucket name, GCP authentication, and metadata fields.
acl [required]
\nenum
Access control list setting for objects written to the bucket. \nAllowed enum values: private,project-private,public-read,authenticated-read,bucket-owner-read,bucket-owner-full-control
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
bucket [required]
\nstring
Name of the GCS bucket.
id [required]
\nstring
Unique identifier for the destination component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
key_prefix
\nstring
Optional prefix for object keys within the GCS bucket.
metadata
\n[object]
Custom metadata to attach to each object uploaded to the GCS bucket.
name [required]
\nstring
The metadata key.
value [required]
\nstring
The metadata value.
storage_class [required]
\nenum
Storage class used for objects stored in GCS. \nAllowed enum values: STANDARD,NEARLINE,COLDLINE,ARCHIVE
type [required]
\nenum
The destination type. Always google_cloud_storage. \nAllowed enum values: google_cloud_storage
default: google_cloud_storage
Option 4
\nobject
The splunk_hec destination forwards logs to Splunk using the HTTP Event Collector (HEC).
auto_extract_timestamp
\nboolean
If true, Splunk tries to extract timestamps from incoming log events.\nIf false, Splunk assigns the time the event was received.
encoding
\nenum
Encoding format for log events. \nAllowed enum values: json,raw_message
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
index
\nstring
Optional name of the Splunk index where logs are written.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
sourcetype
\nstring
The Splunk sourcetype to assign to log events.
type [required]
\nenum
The destination type. Always splunk_hec. \nAllowed enum values: splunk_hec
default: splunk_hec
Option 5
\nobject
The sumo_logic destination forwards logs to Sumo Logic.
encoding
\nenum
The output encoding format. \nAllowed enum values: json,raw_message,logfmt
header_custom_fields
\n[object]
A list of custom headers to include in the request to Sumo Logic.
name [required]
\nstring
The header field name.
value [required]
\nstring
The header field value.
header_host_name
\nstring
Optional override for the host name header.
header_source_category
\nstring
Optional override for the source category header.
header_source_name
\nstring
Optional override for the source name header.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be sumo_logic. \nAllowed enum values: sumo_logic
default: sumo_logic
Option 6
\nobject
The elasticsearch destination writes logs to an Elasticsearch cluster.
api_version
\nenum
The Elasticsearch API version to use. Set to auto to auto-detect. \nAllowed enum values: auto,v6,v7,v8
bulk_index
\nstring
The index to write logs to in Elasticsearch.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be elasticsearch. \nAllowed enum values: elasticsearch
default: elasticsearch
Option 7
\nobject
The rsyslog destination forwards logs to an external rsyslog server over TCP or UDP using the syslog protocol.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
keepalive
\nint64
Optional socket keepalive duration in milliseconds.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. The value should always be rsyslog. \nAllowed enum values: rsyslog
default: rsyslog
Option 8
\nobject
The syslog_ng destination forwards logs to an external syslog-ng server over TCP or UDP using the syslog protocol.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
keepalive
\nint64
Optional socket keepalive duration in milliseconds.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. The value should always be syslog_ng. \nAllowed enum values: syslog_ng
default: syslog_ng
Option 9
\nobject
The azure_storage destination forwards logs to an Azure Blob Storage container.
blob_prefix
\nstring
Optional prefix for blobs written to the container.
container_name [required]
\nstring
The name of the Azure Blob Storage container to store logs in.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be azure_storage. \nAllowed enum values: azure_storage
default: azure_storage
Option 10
\nobject
The microsoft_sentinel destination forwards logs to Microsoft Sentinel.
client_id [required]
\nstring
Azure AD client ID used for authentication.
dcr_immutable_id [required]
\nstring
The immutable ID of the Data Collection Rule (DCR).
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
table [required]
\nstring
The name of the Log Analytics table where logs are sent.
tenant_id [required]
\nstring
Azure AD tenant ID.
type [required]
\nenum
The destination type. The value should always be microsoft_sentinel. \nAllowed enum values: microsoft_sentinel
default: microsoft_sentinel
Option 11
\nobject
The google_chronicle destination sends logs to Google Chronicle.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
customer_id [required]
\nstring
The Google Chronicle customer ID.
encoding
\nenum
The encoding format for the logs sent to Chronicle. \nAllowed enum values: json,raw_message
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
log_type
\nstring
The log type metadata associated with the Chronicle destination.
type [required]
\nenum
The destination type. The value should always be google_chronicle. \nAllowed enum values: google_chronicle
default: google_chronicle
Option 12
\nobject
The new_relic destination sends logs to the New Relic platform.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The New Relic region. \nAllowed enum values: us,eu
type [required]
\nenum
The destination type. The value should always be new_relic. \nAllowed enum values: new_relic
default: new_relic
Option 13
\nobject
The sentinel_one destination sends logs to SentinelOne.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The SentinelOne region to send logs to. \nAllowed enum values: us,eu,ca,data_set_us
type [required]
\nenum
The destination type. The value should always be sentinel_one. \nAllowed enum values: sentinel_one
default: sentinel_one
Option 14
\nobject
The opensearch destination writes logs to an OpenSearch cluster.
bulk_index
\nstring
The index to write logs to.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be opensearch. \nAllowed enum values: opensearch
default: opensearch
Option 15
\nobject
The amazon_opensearch destination writes logs to Amazon OpenSearch.
auth [required]
\nobject
Authentication settings for the Amazon OpenSearch destination.\nThe strategy field determines whether basic or AWS-based authentication is used.
assume_role
\nstring
The ARN of the role to assume (used with aws strategy).
aws_region
\nstring
AWS region
external_id
\nstring
External ID for the assumed role (used with aws strategy).
session_name
\nstring
Session name for the assumed role (used with aws strategy).
strategy [required]
\nenum
The authentication strategy to use. \nAllowed enum values: basic,aws
bulk_index
\nstring
The index to write logs to.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be amazon_opensearch. \nAllowed enum values: amazon_opensearch
default: amazon_opensearch
processors
\n[ <oneOf>]
A list of processors that transform or enrich log data.
Option 1
\nobject
The filter processor allows conditional processing of logs based on a Datadog search query. Logs that match the include query are passed through; others are discarded.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs should pass through the filter. Logs that match this query continue to downstream components; others are dropped.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be filter. \nAllowed enum values: filter
default: filter
Option 2
\nobject
The parse_json processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string.
field [required]
\nstring
The name of the log field that contains a JSON string.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be parse_json. \nAllowed enum values: parse_json
default: parse_json
Option 3
\nobject
The Quota Processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert.
drop_events [required]
\nboolean
If set to true, logs that matched the quota filter and sent after the quota has been met are dropped; only logs that did not match the filter query continue through the pipeline.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
ignore_when_missing_partitions
\nboolean
If true, the processor skips quota checks when partition fields are missing from the logs.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
name [required]
\nstring
Name of the quota.
overflow_action
\nenum
The action to take when the quota is exceeded. Options:
\ndrop: Drop the event.
no_action: Let the event pass through.
overflow_routing: Route to an overflow destination.
Allowed enum values: drop,no_action,overflow_routing
overrides
\n[object]
A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit.
fields [required]
\n[object]
A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
partition_fields
\n[string]
A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values.
type [required]
\nenum
The processor type. The value should always be quota. \nAllowed enum values: quota
default: quota
Option 4
\nobject
The add_fields processor adds static key-value fields to logs.
fields [required]
\n[object]
A list of static fields (key-value pairs) that is added to each log event processed by this component.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be add_fields. \nAllowed enum values: add_fields
default: add_fields
Option 5
\nobject
The remove_fields processor deletes specified fields from logs.
fields [required]
\n[string]
A list of field names to be removed from each log event.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
The PipelineRemoveFieldsProcessor inputs.
type [required]
\nenum
The processor type. The value should always be remove_fields. \nAllowed enum values: remove_fields
default: remove_fields
Option 6
\nobject
The rename_fields processor changes field names.
fields [required]
\n[object]
A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields.
destination [required]
\nstring
The field name to assign the renamed value to.
preserve_source [required]
\nboolean
Indicates whether the original field, that is received from the source, should be kept (true) or removed (false) after renaming.
source [required]
\nstring
The original field name in the log event that should be renamed.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be rename_fields. \nAllowed enum values: rename_fields
default: rename_fields
Option 7
\nobject
The generate_datadog_metrics processor creates custom metrics from logs and sends them to Datadog.\nMetrics can be counters, gauges, or distributions and optionally grouped by log fields.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
metrics [required]
\n[object]
Configuration for generating individual metrics.
group_by
\n[string]
Optional fields used to group the metric series.
include [required]
\nstring
Datadog filter query to match logs for metric generation.
metric_type [required]
\nenum
Type of metric to create. \nAllowed enum values: count,gauge,distribution
name [required]
\nstring
Name of the custom metric to be created.
value [required]
\n<oneOf>
Specifies how the value of the generated metric is computed.
Option 1
\nobject
Strategy that increments a generated metric by one for each matching event.
strategy [required]
\nenum
Increments the metric by 1 for each matching event. \nAllowed enum values: increment_by_one
Option 2
\nobject
Strategy that increments a generated metric based on the value of a log field.
field [required]
\nstring
Name of the log field containing the numeric value to increment the metric by.
strategy [required]
\nenum
Uses a numeric field in the log event as the metric increment. \nAllowed enum values: increment_by_field
type [required]
\nenum
The processor type. Always generate_datadog_metrics. \nAllowed enum values: generate_datadog_metrics
default: generate_datadog_metrics
Option 8
\nobject
The sample processor allows probabilistic sampling of logs at a fixed rate.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
percentage
\ndouble
The percentage of logs to sample.
rate
\nint64
Number of events to sample (1 in N).
type [required]
\nenum
The processor type. The value should always be sample. \nAllowed enum values: sample
default: sample
Option 9
\nobject
The parse_grok processor extracts structured fields from unstructured log messages using Grok patterns.
disable_library_rules
\nboolean
If set to true, disables the default Grok rules provided by Datadog.
id [required]
\nstring
A unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
rules [required]
\n[object]
The list of Grok parsing rules. If multiple matching rules are provided, they are evaluated in order. The first successful match is applied.
match_rules [required]
\n[object]
A list of Grok parsing rules that define how to extract fields from the source field.\nEach rule must contain a name and a valid Grok pattern.
name [required]
\nstring
The name of the rule.
rule [required]
\nstring
The definition of the Grok rule.
source [required]
\nstring
The name of the field in the log event to apply the Grok rules to.
support_rules [required]
\n[object]
A list of Grok helper rules that can be referenced by the parsing rules.
name [required]
\nstring
The name of the Grok helper rule.
rule [required]
\nstring
The definition of the Grok helper rule.
type [required]
\nenum
The processor type. The value should always be parse_grok. \nAllowed enum values: parse_grok
default: parse_grok
Option 10
\nobject
The sensitive_data_scanner processor detects and optionally redacts sensitive data in log events.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
rules [required]
\n[object]
A list of rules for identifying and acting on sensitive data patterns.
keyword_options
\nobject
Configuration for keywords used to reinforce sensitive data pattern detection.
keywords [required]
\n[string]
A list of keywords to match near the sensitive pattern.
proximity [required]
\nint64
Maximum number of tokens between a keyword and a sensitive value match.
name [required]
\nstring
A name identifying the rule.
on_match [required]
\n<oneOf>
Defines what action to take when sensitive data is matched.
Option 1
\nobject
Configuration for completely redacting matched sensitive data.
action [required]
\nenum
Action type that completely replaces the matched sensitive data with a fixed replacement string to remove all visibility. \nAllowed enum values: redact
options [required]
\nobject
Configuration for fully redacting sensitive data.
replace [required]
\nstring
The ObservabilityPipelineSensitiveDataScannerProcessorActionRedactOptions replace.
Option 2
\nobject
Configuration for hashing matched sensitive values.
action [required]
\nenum
Action type that replaces the matched sensitive data with a hashed representation, preserving structure while securing content. \nAllowed enum values: hash
options
\nobject
The ObservabilityPipelineSensitiveDataScannerProcessorActionHash options.
Option 3
\nobject
Configuration for partially redacting matched sensitive data.
action [required]
\nenum
Action type that redacts part of the sensitive data while preserving a configurable number of characters, typically used for masking purposes (e.g., show last 4 digits of a credit card). \nAllowed enum values: partial_redact
options [required]
\nobject
Controls how partial redaction is applied, including character count and direction.
characters [required]
\nint64
The ObservabilityPipelineSensitiveDataScannerProcessorActionPartialRedactOptions characters.
direction [required]
\nenum
Indicates whether to redact characters from the first or last part of the matched value. \nAllowed enum values: first,last
pattern [required]
\n<oneOf>
Pattern detection configuration for identifying sensitive data using either a custom regex or a library reference.
Option 1
\nobject
Defines a custom regex-based pattern for identifying sensitive data in logs.
options [required]
\nobject
Options for defining a custom regex pattern.
rule [required]
\nstring
A regular expression used to detect sensitive values. Must be a valid regex.
type [required]
\nenum
Indicates a custom regular expression is used for matching. \nAllowed enum values: custom
Option 2
\nobject
Specifies a pattern from Datadog’s sensitive data detection library to match known sensitive data types.
options [required]
\nobject
Options for selecting a predefined library pattern and enabling keyword support.
id [required]
\nstring
Identifier for a predefined pattern from the sensitive data scanner pattern library.
use_recommended_keywords
\nboolean
Whether to augment the pattern with recommended keywords (optional).
type [required]
\nenum
Indicates that a predefined library pattern is used. \nAllowed enum values: library
scope [required]
\n<oneOf>
Determines which parts of the log the pattern-matching rule should be applied to.
Option 1
\nobject
Includes only specific fields for sensitive data scanning.
options [required]
\nobject
Fields to which the scope rule applies.
fields [required]
\n[string]
The ObservabilityPipelineSensitiveDataScannerProcessorScopeOptions fields.
target [required]
\nenum
Applies the rule only to included fields. \nAllowed enum values: include
Option 2
\nobject
Excludes specific fields from sensitive data scanning.
options [required]
\nobject
Fields to which the scope rule applies.
fields [required]
\n[string]
The ObservabilityPipelineSensitiveDataScannerProcessorScopeOptions fields.
target [required]
\nenum
Excludes specific fields from processing. \nAllowed enum values: exclude
Option 3
\nobject
Applies scanning across all available fields.
target [required]
\nenum
Applies the rule to all fields. \nAllowed enum values: all
tags [required]
\n[string]
Tags assigned to this rule for filtering and classification.
type [required]
\nenum
The processor type. The value should always be sensitive_data_scanner. \nAllowed enum values: sensitive_data_scanner
default: sensitive_data_scanner
Option 11
\nobject
The ocsf_mapper processor transforms logs into the OCSF schema using a predefined mapping configuration.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mappings [required]
\n[object]
A list of mapping rules to convert events to the OCSF format.
include [required]
\nstring
A Datadog search query used to select the logs that this mapping should apply to.
mapping [required]
\n<oneOf>
Defines a single mapping rule for transforming logs into the OCSF schema.
Option 1
\nenum
Predefined library mappings for common log formats. \nAllowed enum values: CloudTrail Account Change,GCP Cloud Audit CreateBucket,GCP Cloud Audit CreateSink,GCP Cloud Audit SetIamPolicy,GCP Cloud Audit UpdateSink,Github Audit Log API Activity,Google Workspace Admin Audit addPrivilege,Microsoft 365 Defender Incident,Microsoft 365 Defender UserLoggedIn,Okta System Log Authentication,Palo Alto Networks Firewall Traffic
type [required]
\nenum
The processor type. The value should always be ocsf_mapper. \nAllowed enum values: ocsf_mapper
default: ocsf_mapper
Option 12
\nobject
The add_env_vars processor adds environment variable values to log events.
id [required]
\nstring
The unique identifier for this component. Used to reference this processor in the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
type [required]
\nenum
The processor type. The value should always be add_env_vars. \nAllowed enum values: add_env_vars
default: add_env_vars
variables [required]
\n[object]
A list of environment variable mappings to apply to log fields.
field [required]
\nstring
The target field in the log event.
name [required]
\nstring
The name of the environment variable to read.
Option 13
\nobject
The dedupe processor removes duplicate fields in log events.
fields [required]
\n[string]
A list of log field paths to check for duplicates.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mode [required]
\nenum
The deduplication mode to apply to the fields. \nAllowed enum values: match,ignore
type [required]
\nenum
The processor type. The value should always be dedupe. \nAllowed enum values: dedupe
default: dedupe
Option 14
\nobject
The enrichment_table processor enriches logs using a static CSV file or GeoIP database.
file
\nobject
Defines a static enrichment table loaded from a CSV file.
encoding [required]
\nobject
File encoding format.
delimiter [required]
\nstring
The encoding delimiter.
includes_headers [required]
\nboolean
The encoding includes_headers.
type [required]
\nenum
Specifies the encoding format (e.g., CSV) used for enrichment tables. \nAllowed enum values: csv
key [required]
\n[object]
Key fields used to look up enrichment values.
column [required]
\nstring
The items column.
comparison [required]
\nenum
Defines how to compare key fields for enrichment table lookups. \nAllowed enum values: equals
field [required]
\nstring
The items field.
path [required]
\nstring
Path to the CSV file.
schema [required]
\n[object]
Schema defining column names and their types.
column [required]
\nstring
The items column.
type [required]
\nenum
Declares allowed data types for enrichment table columns. \nAllowed enum values: string,boolean,integer,float,date,timestamp
geoip
\nobject
Uses a GeoIP database to enrich logs based on an IP field.
key_field [required]
\nstring
Path to the IP field in the log.
locale [required]
\nstring
Locale used to resolve geographical names.
path [required]
\nstring
Path to the GeoIP database file.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
target [required]
\nstring
Path where enrichment results should be stored in the log.
type [required]
\nenum
The processor type. The value should always be enrichment_table. \nAllowed enum values: enrichment_table
default: enrichment_table
Option 15
\nobject
The reduce processor aggregates and merges logs based on matching keys and merge strategies.
group_by [required]
\n[string]
A list of fields used to group log events for merging.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
merge_strategies [required]
\n[object]
List of merge strategies defining how values from grouped events should be combined.
path [required]
\nstring
The field path in the log event.
strategy [required]
\nenum
The merge strategy to apply. \nAllowed enum values: discard,retain,sum,max,min,array,concat,concat_newline,concat_raw,shortest_array,longest_array,flat_unique
type [required]
\nenum
The processor type. The value should always be reduce. \nAllowed enum values: reduce
default: reduce
Option 16
\nobject
The throttle processor limits the number of events that pass through over a given time window.
group_by
\n[string]
Optional list of fields used to group events before the threshold has been reached.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
threshold [required]
\nint64
the number of events allowed in a given time window. Events sent after the threshold has been reached, are dropped.
type [required]
\nenum
The processor type. The value should always be throttle. \nAllowed enum values: throttle
default: throttle
window [required]
\ndouble
The time window in seconds over which the threshold applies.
sources [required]
\n[ <oneOf>]
A list of configured data sources for the pipeline.
Option 1
\nobject
The kafka source ingests data from Apache Kafka topics.
group_id [required]
\nstring
Consumer group ID used by the Kafka client.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
librdkafka_options
\n[object]
Optional list of advanced Kafka client configuration options, defined as key-value pairs.
name [required]
\nstring
The name of the librdkafka configuration option to set.
value [required]
\nstring
The value assigned to the specified librdkafka configuration option.
sasl
\nobject
Specifies the SASL mechanism for authenticating with a Kafka cluster.
mechanism
\nenum
SASL mechanism used for Kafka authentication. \nAllowed enum values: PLAIN,SCRAM-SHA-256,SCRAM-SHA-512
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
topics [required]
\n[string]
A list of Kafka topic names to subscribe to. The source ingests messages from each topic specified.
type [required]
\nenum
The source type. The value should always be kafka. \nAllowed enum values: kafka
default: kafka
Option 2
\nobject
The datadog_agent source collects logs from the Datadog Agent.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be datadog_agent. \nAllowed enum values: datadog_agent
default: datadog_agent
Option 3
\nobject
The splunk_tcp source receives logs from a Splunk Universal Forwarder over TCP.\nTLS is supported for secure transmission.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always splunk_tcp. \nAllowed enum values: splunk_tcp
default: splunk_tcp
Option 4
\nobject
The splunk_hec source implements the Splunk HTTP Event Collector (HEC) API.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always splunk_hec. \nAllowed enum values: splunk_hec
default: splunk_hec
Option 5
\nobject
The amazon_s3 source ingests logs from an Amazon S3 bucket.\nIt supports AWS authentication and TLS encryption.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
region [required]
\nstring
AWS region where the S3 bucket resides.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always amazon_s3. \nAllowed enum values: amazon_s3
default: amazon_s3
Option 6
\nobject
The fluentd source ingests logs from a Fluentd-compatible service.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be `fluentd. \nAllowed enum values: fluentd
default: fluentd
Option 7
\nobject
The fluent_bit source ingests logs from Fluent Bit.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be fluent_bit. \nAllowed enum values: fluent_bit
default: fluent_bit
Option 8
\nobject
The http_server source collects logs over HTTP POST from external services.
auth_strategy [required]
\nenum
HTTP authentication method. \nAllowed enum values: none,plain
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
Unique ID for the HTTP server source.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be http_server. \nAllowed enum values: http_server
default: http_server
Option 9
\nobject
The sumo_logic source receives logs from Sumo Logic collectors.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
type [required]
\nenum
The source type. The value should always be sumo_logic. \nAllowed enum values: sumo_logic
default: sumo_logic
Option 10
\nobject
The rsyslog source listens for logs over TCP or UDP from an rsyslog server using the syslog protocol.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
mode [required]
\nenum
Protocol used by the syslog source to receive messages. \nAllowed enum values: tcp,udp
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be rsyslog. \nAllowed enum values: rsyslog
default: rsyslog
Option 11
\nobject
The syslog_ng source listens for logs over TCP or UDP from a syslog-ng server using the syslog protocol.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
mode [required]
\nenum
Protocol used by the syslog source to receive messages. \nAllowed enum values: tcp,udp
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be syslog_ng. \nAllowed enum values: syslog_ng
default: syslog_ng
Option 12
\nobject
The amazon_data_firehose source ingests logs from AWS Data Firehose.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be amazon_data_firehose. \nAllowed enum values: amazon_data_firehose
default: amazon_data_firehose
Option 13
\nobject
The google_pubsub source ingests logs from a Google Cloud Pub/Sub subscription.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
project [required]
\nstring
The GCP project ID that owns the Pub/Sub subscription.
subscription [required]
\nstring
The Pub/Sub subscription name from which messages are consumed.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be google_pubsub. \nAllowed enum values: google_pubsub
default: google_pubsub
Option 14
\nobject
The http_client source scrapes logs from HTTP endpoints at regular intervals.
auth_strategy
\nenum
Optional authentication strategy for HTTP requests. \nAllowed enum values: basic,bearer
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
scrape_interval_secs
\nint64
The interval (in seconds) between HTTP scrape requests.
scrape_timeout_secs
\nint64
The timeout (in seconds) for each scrape request.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be http_client. \nAllowed enum values: http_client
default: http_client
Option 15
\nobject
The logstash source ingests logs from a Logstash forwarder.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be logstash. \nAllowed enum values: logstash
default: logstash
name [required]
\nstring
Name of the pipeline.
id [required]
\nstring
Unique identifier for the pipeline.
type [required]
\nstring
The resource type identifier. For pipeline resources, this should always be set to pipelines.
default: pipelines
data [required]
\nobject
Contains the pipeline’s ID, type, and configuration attributes.
attributes [required]
\nobject
Defines the pipeline’s name and its components (sources, processors, and destinations).
config [required]
\nobject
Specifies the pipeline's configuration, including its sources, processors, and destinations.
destinations [required]
\n[ <oneOf>]
A list of destination components where processed logs are sent.
Option 1
\nobject
The datadog_logs destination forwards logs to Datadog Log Management.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be datadog_logs. \nAllowed enum values: datadog_logs
default: datadog_logs
Option 2
\nobject
The amazon_s3 destination sends your logs in Datadog-rehydratable format to an Amazon S3 bucket for archiving.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
bucket [required]
\nstring
S3 bucket name.
id [required]
\nstring
Unique identifier for the destination component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
key_prefix
\nstring
Optional prefix for object keys.
region [required]
\nstring
AWS region of the S3 bucket.
storage_class [required]
\nenum
S3 storage class. \nAllowed enum values: STANDARD,REDUCED_REDUNDANCY,INTELLIGENT_TIERING,STANDARD_IA,EXPRESS_ONEZONE,ONEZONE_IA,GLACIER,GLACIER_IR,DEEP_ARCHIVE
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. Always amazon_s3. \nAllowed enum values: amazon_s3
default: amazon_s3
Option 3
\nobject
The google_cloud_storage destination stores logs in a Google Cloud Storage (GCS) bucket.\nIt requires a bucket name, GCP authentication, and metadata fields.
acl [required]
\nenum
Access control list setting for objects written to the bucket. \nAllowed enum values: private,project-private,public-read,authenticated-read,bucket-owner-read,bucket-owner-full-control
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
bucket [required]
\nstring
Name of the GCS bucket.
id [required]
\nstring
Unique identifier for the destination component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
key_prefix
\nstring
Optional prefix for object keys within the GCS bucket.
metadata
\n[object]
Custom metadata to attach to each object uploaded to the GCS bucket.
name [required]
\nstring
The metadata key.
value [required]
\nstring
The metadata value.
storage_class [required]
\nenum
Storage class used for objects stored in GCS. \nAllowed enum values: STANDARD,NEARLINE,COLDLINE,ARCHIVE
type [required]
\nenum
The destination type. Always google_cloud_storage. \nAllowed enum values: google_cloud_storage
default: google_cloud_storage
Option 4
\nobject
The splunk_hec destination forwards logs to Splunk using the HTTP Event Collector (HEC).
auto_extract_timestamp
\nboolean
If true, Splunk tries to extract timestamps from incoming log events.\nIf false, Splunk assigns the time the event was received.
encoding
\nenum
Encoding format for log events. \nAllowed enum values: json,raw_message
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
index
\nstring
Optional name of the Splunk index where logs are written.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
sourcetype
\nstring
The Splunk sourcetype to assign to log events.
type [required]
\nenum
The destination type. Always splunk_hec. \nAllowed enum values: splunk_hec
default: splunk_hec
Option 5
\nobject
The sumo_logic destination forwards logs to Sumo Logic.
encoding
\nenum
The output encoding format. \nAllowed enum values: json,raw_message,logfmt
header_custom_fields
\n[object]
A list of custom headers to include in the request to Sumo Logic.
name [required]
\nstring
The header field name.
value [required]
\nstring
The header field value.
header_host_name
\nstring
Optional override for the host name header.
header_source_category
\nstring
Optional override for the source category header.
header_source_name
\nstring
Optional override for the source name header.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be sumo_logic. \nAllowed enum values: sumo_logic
default: sumo_logic
Option 6
\nobject
The elasticsearch destination writes logs to an Elasticsearch cluster.
api_version
\nenum
The Elasticsearch API version to use. Set to auto to auto-detect. \nAllowed enum values: auto,v6,v7,v8
bulk_index
\nstring
The index to write logs to in Elasticsearch.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be elasticsearch. \nAllowed enum values: elasticsearch
default: elasticsearch
Option 7
\nobject
The rsyslog destination forwards logs to an external rsyslog server over TCP or UDP using the syslog protocol.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
keepalive
\nint64
Optional socket keepalive duration in milliseconds.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. The value should always be rsyslog. \nAllowed enum values: rsyslog
default: rsyslog
Option 8
\nobject
The syslog_ng destination forwards logs to an external syslog-ng server over TCP or UDP using the syslog protocol.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
keepalive
\nint64
Optional socket keepalive duration in milliseconds.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. The value should always be syslog_ng. \nAllowed enum values: syslog_ng
default: syslog_ng
Option 9
\nobject
The azure_storage destination forwards logs to an Azure Blob Storage container.
blob_prefix
\nstring
Optional prefix for blobs written to the container.
container_name [required]
\nstring
The name of the Azure Blob Storage container to store logs in.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be azure_storage. \nAllowed enum values: azure_storage
default: azure_storage
Option 10
\nobject
The microsoft_sentinel destination forwards logs to Microsoft Sentinel.
client_id [required]
\nstring
Azure AD client ID used for authentication.
dcr_immutable_id [required]
\nstring
The immutable ID of the Data Collection Rule (DCR).
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
table [required]
\nstring
The name of the Log Analytics table where logs are sent.
tenant_id [required]
\nstring
Azure AD tenant ID.
type [required]
\nenum
The destination type. The value should always be microsoft_sentinel. \nAllowed enum values: microsoft_sentinel
default: microsoft_sentinel
Option 11
\nobject
The google_chronicle destination sends logs to Google Chronicle.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
customer_id [required]
\nstring
The Google Chronicle customer ID.
encoding
\nenum
The encoding format for the logs sent to Chronicle. \nAllowed enum values: json,raw_message
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
log_type
\nstring
The log type metadata associated with the Chronicle destination.
type [required]
\nenum
The destination type. The value should always be google_chronicle. \nAllowed enum values: google_chronicle
default: google_chronicle
Option 12
\nobject
The new_relic destination sends logs to the New Relic platform.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The New Relic region. \nAllowed enum values: us,eu
type [required]
\nenum
The destination type. The value should always be new_relic. \nAllowed enum values: new_relic
default: new_relic
Option 13
\nobject
The sentinel_one destination sends logs to SentinelOne.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The SentinelOne region to send logs to. \nAllowed enum values: us,eu,ca,data_set_us
type [required]
\nenum
The destination type. The value should always be sentinel_one. \nAllowed enum values: sentinel_one
default: sentinel_one
Option 14
\nobject
The opensearch destination writes logs to an OpenSearch cluster.
bulk_index
\nstring
The index to write logs to.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be opensearch. \nAllowed enum values: opensearch
default: opensearch
Option 15
\nobject
The amazon_opensearch destination writes logs to Amazon OpenSearch.
auth [required]
\nobject
Authentication settings for the Amazon OpenSearch destination.\nThe strategy field determines whether basic or AWS-based authentication is used.
assume_role
\nstring
The ARN of the role to assume (used with aws strategy).
aws_region
\nstring
AWS region
external_id
\nstring
External ID for the assumed role (used with aws strategy).
session_name
\nstring
Session name for the assumed role (used with aws strategy).
strategy [required]
\nenum
The authentication strategy to use. \nAllowed enum values: basic,aws
bulk_index
\nstring
The index to write logs to.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be amazon_opensearch. \nAllowed enum values: amazon_opensearch
default: amazon_opensearch
processors
\n[ <oneOf>]
A list of processors that transform or enrich log data.
Option 1
\nobject
The filter processor allows conditional processing of logs based on a Datadog search query. Logs that match the include query are passed through; others are discarded.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs should pass through the filter. Logs that match this query continue to downstream components; others are dropped.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be filter. \nAllowed enum values: filter
default: filter
Option 2
\nobject
The parse_json processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string.
field [required]
\nstring
The name of the log field that contains a JSON string.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be parse_json. \nAllowed enum values: parse_json
default: parse_json
Option 3
\nobject
The Quota Processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert.
drop_events [required]
\nboolean
If set to true, logs that matched the quota filter and sent after the quota has been met are dropped; only logs that did not match the filter query continue through the pipeline.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
ignore_when_missing_partitions
\nboolean
If true, the processor skips quota checks when partition fields are missing from the logs.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
name [required]
\nstring
Name of the quota.
overflow_action
\nenum
The action to take when the quota is exceeded. Options:
\ndrop: Drop the event.
no_action: Let the event pass through.
overflow_routing: Route to an overflow destination.
Allowed enum values: drop,no_action,overflow_routing
overrides
\n[object]
A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit.
fields [required]
\n[object]
A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
partition_fields
\n[string]
A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values.
type [required]
\nenum
The processor type. The value should always be quota. \nAllowed enum values: quota
default: quota
Option 4
\nobject
The add_fields processor adds static key-value fields to logs.
fields [required]
\n[object]
A list of static fields (key-value pairs) that is added to each log event processed by this component.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be add_fields. \nAllowed enum values: add_fields
default: add_fields
Option 5
\nobject
The remove_fields processor deletes specified fields from logs.
fields [required]
\n[string]
A list of field names to be removed from each log event.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
The PipelineRemoveFieldsProcessor inputs.
type [required]
\nenum
The processor type. The value should always be remove_fields. \nAllowed enum values: remove_fields
default: remove_fields
Option 6
\nobject
The rename_fields processor changes field names.
fields [required]
\n[object]
A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields.
destination [required]
\nstring
The field name to assign the renamed value to.
preserve_source [required]
\nboolean
Indicates whether the original field, that is received from the source, should be kept (true) or removed (false) after renaming.
source [required]
\nstring
The original field name in the log event that should be renamed.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be rename_fields. \nAllowed enum values: rename_fields
default: rename_fields
Option 7
\nobject
The generate_datadog_metrics processor creates custom metrics from logs and sends them to Datadog.\nMetrics can be counters, gauges, or distributions and optionally grouped by log fields.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
metrics [required]
\n[object]
Configuration for generating individual metrics.
group_by
\n[string]
Optional fields used to group the metric series.
include [required]
\nstring
Datadog filter query to match logs for metric generation.
metric_type [required]
\nenum
Type of metric to create. \nAllowed enum values: count,gauge,distribution
name [required]
\nstring
Name of the custom metric to be created.
value [required]
\n<oneOf>
Specifies how the value of the generated metric is computed.
Option 1
\nobject
Strategy that increments a generated metric by one for each matching event.
strategy [required]
\nenum
Increments the metric by 1 for each matching event. \nAllowed enum values: increment_by_one
Option 2
\nobject
Strategy that increments a generated metric based on the value of a log field.
field [required]
\nstring
Name of the log field containing the numeric value to increment the metric by.
strategy [required]
\nenum
Uses a numeric field in the log event as the metric increment. \nAllowed enum values: increment_by_field
type [required]
\nenum
The processor type. Always generate_datadog_metrics. \nAllowed enum values: generate_datadog_metrics
default: generate_datadog_metrics
Option 8
\nobject
The sample processor allows probabilistic sampling of logs at a fixed rate.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
percentage
\ndouble
The percentage of logs to sample.
rate
\nint64
Number of events to sample (1 in N).
type [required]
\nenum
The processor type. The value should always be sample. \nAllowed enum values: sample
default: sample
Option 9
\nobject
The parse_grok processor extracts structured fields from unstructured log messages using Grok patterns.
disable_library_rules
\nboolean
If set to true, disables the default Grok rules provided by Datadog.
id [required]
\nstring
A unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
rules [required]
\n[object]
The list of Grok parsing rules. If multiple matching rules are provided, they are evaluated in order. The first successful match is applied.
match_rules [required]
\n[object]
A list of Grok parsing rules that define how to extract fields from the source field.\nEach rule must contain a name and a valid Grok pattern.
name [required]
\nstring
The name of the rule.
rule [required]
\nstring
The definition of the Grok rule.
source [required]
\nstring
The name of the field in the log event to apply the Grok rules to.
support_rules [required]
\n[object]
A list of Grok helper rules that can be referenced by the parsing rules.
name [required]
\nstring
The name of the Grok helper rule.
rule [required]
\nstring
The definition of the Grok helper rule.
type [required]
\nenum
The processor type. The value should always be parse_grok. \nAllowed enum values: parse_grok
default: parse_grok
Option 10
\nobject
The sensitive_data_scanner processor detects and optionally redacts sensitive data in log events.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
rules [required]
\n[object]
A list of rules for identifying and acting on sensitive data patterns.
keyword_options
\nobject
Configuration for keywords used to reinforce sensitive data pattern detection.
keywords [required]
\n[string]
A list of keywords to match near the sensitive pattern.
proximity [required]
\nint64
Maximum number of tokens between a keyword and a sensitive value match.
name [required]
\nstring
A name identifying the rule.
on_match [required]
\n<oneOf>
Defines what action to take when sensitive data is matched.
Option 1
\nobject
Configuration for completely redacting matched sensitive data.
action [required]
\nenum
Action type that completely replaces the matched sensitive data with a fixed replacement string to remove all visibility. \nAllowed enum values: redact
options [required]
\nobject
Configuration for fully redacting sensitive data.
replace [required]
\nstring
The ObservabilityPipelineSensitiveDataScannerProcessorActionRedactOptions replace.
Option 2
\nobject
Configuration for hashing matched sensitive values.
action [required]
\nenum
Action type that replaces the matched sensitive data with a hashed representation, preserving structure while securing content. \nAllowed enum values: hash
options
\nobject
The ObservabilityPipelineSensitiveDataScannerProcessorActionHash options.
Option 3
\nobject
Configuration for partially redacting matched sensitive data.
action [required]
\nenum
Action type that redacts part of the sensitive data while preserving a configurable number of characters, typically used for masking purposes (e.g., show last 4 digits of a credit card). \nAllowed enum values: partial_redact
options [required]
\nobject
Controls how partial redaction is applied, including character count and direction.
characters [required]
\nint64
The ObservabilityPipelineSensitiveDataScannerProcessorActionPartialRedactOptions characters.
direction [required]
\nenum
Indicates whether to redact characters from the first or last part of the matched value. \nAllowed enum values: first,last
pattern [required]
\n<oneOf>
Pattern detection configuration for identifying sensitive data using either a custom regex or a library reference.
Option 1
\nobject
Defines a custom regex-based pattern for identifying sensitive data in logs.
options [required]
\nobject
Options for defining a custom regex pattern.
rule [required]
\nstring
A regular expression used to detect sensitive values. Must be a valid regex.
type [required]
\nenum
Indicates a custom regular expression is used for matching. \nAllowed enum values: custom
Option 2
\nobject
Specifies a pattern from Datadog’s sensitive data detection library to match known sensitive data types.
options [required]
\nobject
Options for selecting a predefined library pattern and enabling keyword support.
id [required]
\nstring
Identifier for a predefined pattern from the sensitive data scanner pattern library.
use_recommended_keywords
\nboolean
Whether to augment the pattern with recommended keywords (optional).
type [required]
\nenum
Indicates that a predefined library pattern is used. \nAllowed enum values: library
scope [required]
\n<oneOf>
Determines which parts of the log the pattern-matching rule should be applied to.
Option 1
\nobject
Includes only specific fields for sensitive data scanning.
options [required]
\nobject
Fields to which the scope rule applies.
fields [required]
\n[string]
The ObservabilityPipelineSensitiveDataScannerProcessorScopeOptions fields.
target [required]
\nenum
Applies the rule only to included fields. \nAllowed enum values: include
Option 2
\nobject
Excludes specific fields from sensitive data scanning.
options [required]
\nobject
Fields to which the scope rule applies.
fields [required]
\n[string]
The ObservabilityPipelineSensitiveDataScannerProcessorScopeOptions fields.
target [required]
\nenum
Excludes specific fields from processing. \nAllowed enum values: exclude
Option 3
\nobject
Applies scanning across all available fields.
target [required]
\nenum
Applies the rule to all fields. \nAllowed enum values: all
tags [required]
\n[string]
Tags assigned to this rule for filtering and classification.
type [required]
\nenum
The processor type. The value should always be sensitive_data_scanner. \nAllowed enum values: sensitive_data_scanner
default: sensitive_data_scanner
Option 11
\nobject
The ocsf_mapper processor transforms logs into the OCSF schema using a predefined mapping configuration.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mappings [required]
\n[object]
A list of mapping rules to convert events to the OCSF format.
include [required]
\nstring
A Datadog search query used to select the logs that this mapping should apply to.
mapping [required]
\n<oneOf>
Defines a single mapping rule for transforming logs into the OCSF schema.
Option 1
\nenum
Predefined library mappings for common log formats. \nAllowed enum values: CloudTrail Account Change,GCP Cloud Audit CreateBucket,GCP Cloud Audit CreateSink,GCP Cloud Audit SetIamPolicy,GCP Cloud Audit UpdateSink,Github Audit Log API Activity,Google Workspace Admin Audit addPrivilege,Microsoft 365 Defender Incident,Microsoft 365 Defender UserLoggedIn,Okta System Log Authentication,Palo Alto Networks Firewall Traffic
type [required]
\nenum
The processor type. The value should always be ocsf_mapper. \nAllowed enum values: ocsf_mapper
default: ocsf_mapper
Option 12
\nobject
The add_env_vars processor adds environment variable values to log events.
id [required]
\nstring
The unique identifier for this component. Used to reference this processor in the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
type [required]
\nenum
The processor type. The value should always be add_env_vars. \nAllowed enum values: add_env_vars
default: add_env_vars
variables [required]
\n[object]
A list of environment variable mappings to apply to log fields.
field [required]
\nstring
The target field in the log event.
name [required]
\nstring
The name of the environment variable to read.
Option 13
\nobject
The dedupe processor removes duplicate fields in log events.
fields [required]
\n[string]
A list of log field paths to check for duplicates.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mode [required]
\nenum
The deduplication mode to apply to the fields. \nAllowed enum values: match,ignore
type [required]
\nenum
The processor type. The value should always be dedupe. \nAllowed enum values: dedupe
default: dedupe
Option 14
\nobject
The enrichment_table processor enriches logs using a static CSV file or GeoIP database.
file
\nobject
Defines a static enrichment table loaded from a CSV file.
encoding [required]
\nobject
File encoding format.
delimiter [required]
\nstring
The encoding delimiter.
includes_headers [required]
\nboolean
The encoding includes_headers.
type [required]
\nenum
Specifies the encoding format (e.g., CSV) used for enrichment tables. \nAllowed enum values: csv
key [required]
\n[object]
Key fields used to look up enrichment values.
column [required]
\nstring
The items column.
comparison [required]
\nenum
Defines how to compare key fields for enrichment table lookups. \nAllowed enum values: equals
field [required]
\nstring
The items field.
path [required]
\nstring
Path to the CSV file.
schema [required]
\n[object]
Schema defining column names and their types.
column [required]
\nstring
The items column.
type [required]
\nenum
Declares allowed data types for enrichment table columns. \nAllowed enum values: string,boolean,integer,float,date,timestamp
geoip
\nobject
Uses a GeoIP database to enrich logs based on an IP field.
key_field [required]
\nstring
Path to the IP field in the log.
locale [required]
\nstring
Locale used to resolve geographical names.
path [required]
\nstring
Path to the GeoIP database file.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
target [required]
\nstring
Path where enrichment results should be stored in the log.
type [required]
\nenum
The processor type. The value should always be enrichment_table. \nAllowed enum values: enrichment_table
default: enrichment_table
Option 15
\nobject
The reduce processor aggregates and merges logs based on matching keys and merge strategies.
group_by [required]
\n[string]
A list of fields used to group log events for merging.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
merge_strategies [required]
\n[object]
List of merge strategies defining how values from grouped events should be combined.
path [required]
\nstring
The field path in the log event.
strategy [required]
\nenum
The merge strategy to apply. \nAllowed enum values: discard,retain,sum,max,min,array,concat,concat_newline,concat_raw,shortest_array,longest_array,flat_unique
type [required]
\nenum
The processor type. The value should always be reduce. \nAllowed enum values: reduce
default: reduce
Option 16
\nobject
The throttle processor limits the number of events that pass through over a given time window.
group_by
\n[string]
Optional list of fields used to group events before the threshold has been reached.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
threshold [required]
\nint64
the number of events allowed in a given time window. Events sent after the threshold has been reached, are dropped.
type [required]
\nenum
The processor type. The value should always be throttle. \nAllowed enum values: throttle
default: throttle
window [required]
\ndouble
The time window in seconds over which the threshold applies.
Option 17
\nobject
The datadog_tags processor includes or excludes specific Datadog tags in your logs.
action [required]
\nenum
The action to take on tags with matching keys. \nAllowed enum values: include,exclude
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
keys [required]
\n[string]
A list of tag keys.
mode [required]
\nenum
The processing mode. \nAllowed enum values: filter
type [required]
\nenum
The processor type. The value should always be datadog_tags. \nAllowed enum values: datadog_tags
default: datadog_tags
sources [required]
\n[ <oneOf>]
A list of configured data sources for the pipeline.
Option 1
\nobject
The kafka source ingests data from Apache Kafka topics.
group_id [required]
\nstring
Consumer group ID used by the Kafka client.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
librdkafka_options
\n[object]
Optional list of advanced Kafka client configuration options, defined as key-value pairs.
name [required]
\nstring
The name of the librdkafka configuration option to set.
value [required]
\nstring
The value assigned to the specified librdkafka configuration option.
sasl
\nobject
Specifies the SASL mechanism for authenticating with a Kafka cluster.
mechanism
\nenum
SASL mechanism used for Kafka authentication. \nAllowed enum values: PLAIN,SCRAM-SHA-256,SCRAM-SHA-512
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
topics [required]
\n[string]
A list of Kafka topic names to subscribe to. The source ingests messages from each topic specified.
type [required]
\nenum
The source type. The value should always be kafka. \nAllowed enum values: kafka
default: kafka
Option 2
\nobject
The datadog_agent source collects logs from the Datadog Agent.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be datadog_agent. \nAllowed enum values: datadog_agent
default: datadog_agent
Option 3
\nobject
The splunk_tcp source receives logs from a Splunk Universal Forwarder over TCP.\nTLS is supported for secure transmission.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always splunk_tcp. \nAllowed enum values: splunk_tcp
default: splunk_tcp
Option 4
\nobject
The splunk_hec source implements the Splunk HTTP Event Collector (HEC) API.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always splunk_hec. \nAllowed enum values: splunk_hec
default: splunk_hec
Option 5
\nobject
The amazon_s3 source ingests logs from an Amazon S3 bucket.\nIt supports AWS authentication and TLS encryption.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
region [required]
\nstring
AWS region where the S3 bucket resides.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always amazon_s3. \nAllowed enum values: amazon_s3
default: amazon_s3
Option 6
\nobject
The fluentd source ingests logs from a Fluentd-compatible service.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be `fluentd. \nAllowed enum values: fluentd
default: fluentd
Option 7
\nobject
The fluent_bit source ingests logs from Fluent Bit.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be fluent_bit. \nAllowed enum values: fluent_bit
default: fluent_bit
Option 8
\nobject
The http_server source collects logs over HTTP POST from external services.
auth_strategy [required]
\nenum
HTTP authentication method. \nAllowed enum values: none,plain
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
Unique ID for the HTTP server source.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be http_server. \nAllowed enum values: http_server
default: http_server
Option 9
\nobject
The sumo_logic source receives logs from Sumo Logic collectors.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
type [required]
\nenum
The source type. The value should always be sumo_logic. \nAllowed enum values: sumo_logic
default: sumo_logic
Option 10
\nobject
The rsyslog source listens for logs over TCP or UDP from an rsyslog server using the syslog protocol.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
mode [required]
\nenum
Protocol used by the syslog source to receive messages. \nAllowed enum values: tcp,udp
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be rsyslog. \nAllowed enum values: rsyslog
default: rsyslog
Option 11
\nobject
The syslog_ng source listens for logs over TCP or UDP from a syslog-ng server using the syslog protocol.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
mode [required]
\nenum
Protocol used by the syslog source to receive messages. \nAllowed enum values: tcp,udp
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be syslog_ng. \nAllowed enum values: syslog_ng
default: syslog_ng
Option 12
\nobject
The amazon_data_firehose source ingests logs from AWS Data Firehose.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be amazon_data_firehose. \nAllowed enum values: amazon_data_firehose
default: amazon_data_firehose
Option 13
\nobject
The google_pubsub source ingests logs from a Google Cloud Pub/Sub subscription.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
project [required]
\nstring
The GCP project ID that owns the Pub/Sub subscription.
subscription [required]
\nstring
The Pub/Sub subscription name from which messages are consumed.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be google_pubsub. \nAllowed enum values: google_pubsub
default: google_pubsub
Option 14
\nobject
The http_client source scrapes logs from HTTP endpoints at regular intervals.
auth_strategy
\nenum
Optional authentication strategy for HTTP requests. \nAllowed enum values: basic,bearer
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
scrape_interval_secs
\nint64
The interval (in seconds) between HTTP scrape requests.
scrape_timeout_secs
\nint64
The timeout (in seconds) for each scrape request.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be http_client. \nAllowed enum values: http_client
default: http_client
Option 15
\nobject
The logstash source ingests logs from a Logstash forwarder.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be logstash. \nAllowed enum values: logstash
default: logstash
name [required]
\nstring
Name of the pipeline.
id [required]
\nstring
Unique identifier for the pipeline.
type [required]
\nstring
The resource type identifier. For pipeline resources, this should always be set to pipelines.
default: pipelines
data [required]
\nobject
Contains the the pipeline configuration.
attributes [required]
\nobject
Defines the pipeline’s name and its components (sources, processors, and destinations).
config [required]
\nobject
Specifies the pipeline's configuration, including its sources, processors, and destinations.
destinations [required]
\n[ <oneOf>]
A list of destination components where processed logs are sent.
Option 1
\nobject
The datadog_logs destination forwards logs to Datadog Log Management.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be datadog_logs. \nAllowed enum values: datadog_logs
default: datadog_logs
Option 2
\nobject
The amazon_s3 destination sends your logs in Datadog-rehydratable format to an Amazon S3 bucket for archiving.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
bucket [required]
\nstring
S3 bucket name.
id [required]
\nstring
Unique identifier for the destination component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
key_prefix
\nstring
Optional prefix for object keys.
region [required]
\nstring
AWS region of the S3 bucket.
storage_class [required]
\nenum
S3 storage class. \nAllowed enum values: STANDARD,REDUCED_REDUNDANCY,INTELLIGENT_TIERING,STANDARD_IA,EXPRESS_ONEZONE,ONEZONE_IA,GLACIER,GLACIER_IR,DEEP_ARCHIVE
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. Always amazon_s3. \nAllowed enum values: amazon_s3
default: amazon_s3
Option 3
\nobject
The google_cloud_storage destination stores logs in a Google Cloud Storage (GCS) bucket.\nIt requires a bucket name, GCP authentication, and metadata fields.
acl [required]
\nenum
Access control list setting for objects written to the bucket. \nAllowed enum values: private,project-private,public-read,authenticated-read,bucket-owner-read,bucket-owner-full-control
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
bucket [required]
\nstring
Name of the GCS bucket.
id [required]
\nstring
Unique identifier for the destination component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
key_prefix
\nstring
Optional prefix for object keys within the GCS bucket.
metadata
\n[object]
Custom metadata to attach to each object uploaded to the GCS bucket.
name [required]
\nstring
The metadata key.
value [required]
\nstring
The metadata value.
storage_class [required]
\nenum
Storage class used for objects stored in GCS. \nAllowed enum values: STANDARD,NEARLINE,COLDLINE,ARCHIVE
type [required]
\nenum
The destination type. Always google_cloud_storage. \nAllowed enum values: google_cloud_storage
default: google_cloud_storage
Option 4
\nobject
The splunk_hec destination forwards logs to Splunk using the HTTP Event Collector (HEC).
auto_extract_timestamp
\nboolean
If true, Splunk tries to extract timestamps from incoming log events.\nIf false, Splunk assigns the time the event was received.
encoding
\nenum
Encoding format for log events. \nAllowed enum values: json,raw_message
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
index
\nstring
Optional name of the Splunk index where logs are written.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
sourcetype
\nstring
The Splunk sourcetype to assign to log events.
type [required]
\nenum
The destination type. Always splunk_hec. \nAllowed enum values: splunk_hec
default: splunk_hec
Option 5
\nobject
The sumo_logic destination forwards logs to Sumo Logic.
encoding
\nenum
The output encoding format. \nAllowed enum values: json,raw_message,logfmt
header_custom_fields
\n[object]
A list of custom headers to include in the request to Sumo Logic.
name [required]
\nstring
The header field name.
value [required]
\nstring
The header field value.
header_host_name
\nstring
Optional override for the host name header.
header_source_category
\nstring
Optional override for the source category header.
header_source_name
\nstring
Optional override for the source name header.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be sumo_logic. \nAllowed enum values: sumo_logic
default: sumo_logic
Option 6
\nobject
The elasticsearch destination writes logs to an Elasticsearch cluster.
api_version
\nenum
The Elasticsearch API version to use. Set to auto to auto-detect. \nAllowed enum values: auto,v6,v7,v8
bulk_index
\nstring
The index to write logs to in Elasticsearch.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be elasticsearch. \nAllowed enum values: elasticsearch
default: elasticsearch
Option 7
\nobject
The rsyslog destination forwards logs to an external rsyslog server over TCP or UDP using the syslog protocol.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
keepalive
\nint64
Optional socket keepalive duration in milliseconds.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. The value should always be rsyslog. \nAllowed enum values: rsyslog
default: rsyslog
Option 8
\nobject
The syslog_ng destination forwards logs to an external syslog-ng server over TCP or UDP using the syslog protocol.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
keepalive
\nint64
Optional socket keepalive duration in milliseconds.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. The value should always be syslog_ng. \nAllowed enum values: syslog_ng
default: syslog_ng
Option 9
\nobject
The azure_storage destination forwards logs to an Azure Blob Storage container.
blob_prefix
\nstring
Optional prefix for blobs written to the container.
container_name [required]
\nstring
The name of the Azure Blob Storage container to store logs in.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be azure_storage. \nAllowed enum values: azure_storage
default: azure_storage
Option 10
\nobject
The microsoft_sentinel destination forwards logs to Microsoft Sentinel.
client_id [required]
\nstring
Azure AD client ID used for authentication.
dcr_immutable_id [required]
\nstring
The immutable ID of the Data Collection Rule (DCR).
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
table [required]
\nstring
The name of the Log Analytics table where logs are sent.
tenant_id [required]
\nstring
Azure AD tenant ID.
type [required]
\nenum
The destination type. The value should always be microsoft_sentinel. \nAllowed enum values: microsoft_sentinel
default: microsoft_sentinel
Option 11
\nobject
The google_chronicle destination sends logs to Google Chronicle.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
customer_id [required]
\nstring
The Google Chronicle customer ID.
encoding
\nenum
The encoding format for the logs sent to Chronicle. \nAllowed enum values: json,raw_message
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
log_type
\nstring
The log type metadata associated with the Chronicle destination.
type [required]
\nenum
The destination type. The value should always be google_chronicle. \nAllowed enum values: google_chronicle
default: google_chronicle
Option 12
\nobject
The new_relic destination sends logs to the New Relic platform.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The New Relic region. \nAllowed enum values: us,eu
type [required]
\nenum
The destination type. The value should always be new_relic. \nAllowed enum values: new_relic
default: new_relic
Option 13
\nobject
The sentinel_one destination sends logs to SentinelOne.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The SentinelOne region to send logs to. \nAllowed enum values: us,eu,ca,data_set_us
type [required]
\nenum
The destination type. The value should always be sentinel_one. \nAllowed enum values: sentinel_one
default: sentinel_one
Option 14
\nobject
The opensearch destination writes logs to an OpenSearch cluster.
bulk_index
\nstring
The index to write logs to.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be opensearch. \nAllowed enum values: opensearch
default: opensearch
Option 15
\nobject
The amazon_opensearch destination writes logs to Amazon OpenSearch.
auth [required]
\nobject
Authentication settings for the Amazon OpenSearch destination.\nThe strategy field determines whether basic or AWS-based authentication is used.
assume_role
\nstring
The ARN of the role to assume (used with aws strategy).
aws_region
\nstring
AWS region
external_id
\nstring
External ID for the assumed role (used with aws strategy).
session_name
\nstring
Session name for the assumed role (used with aws strategy).
strategy [required]
\nenum
The authentication strategy to use. \nAllowed enum values: basic,aws
bulk_index
\nstring
The index to write logs to.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be amazon_opensearch. \nAllowed enum values: amazon_opensearch
default: amazon_opensearch
processors
\n[ <oneOf>]
A list of processors that transform or enrich log data.
Option 1
\nobject
The filter processor allows conditional processing of logs based on a Datadog search query. Logs that match the include query are passed through; others are discarded.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs should pass through the filter. Logs that match this query continue to downstream components; others are dropped.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be filter. \nAllowed enum values: filter
default: filter
Option 2
\nobject
The parse_json processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string.
field [required]
\nstring
The name of the log field that contains a JSON string.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be parse_json. \nAllowed enum values: parse_json
default: parse_json
Option 3
\nobject
The Quota Processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert.
drop_events [required]
\nboolean
If set to true, logs that matched the quota filter and sent after the quota has been met are dropped; only logs that did not match the filter query continue through the pipeline.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
ignore_when_missing_partitions
\nboolean
If true, the processor skips quota checks when partition fields are missing from the logs.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
name [required]
\nstring
Name of the quota.
overflow_action
\nenum
The action to take when the quota is exceeded. Options:
\ndrop: Drop the event.
no_action: Let the event pass through.
overflow_routing: Route to an overflow destination.
Allowed enum values: drop,no_action,overflow_routing
overrides
\n[object]
A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit.
fields [required]
\n[object]
A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
partition_fields
\n[string]
A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values.
type [required]
\nenum
The processor type. The value should always be quota. \nAllowed enum values: quota
default: quota
Option 4
\nobject
The add_fields processor adds static key-value fields to logs.
fields [required]
\n[object]
A list of static fields (key-value pairs) that is added to each log event processed by this component.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be add_fields. \nAllowed enum values: add_fields
default: add_fields
Option 5
\nobject
The remove_fields processor deletes specified fields from logs.
fields [required]
\n[string]
A list of field names to be removed from each log event.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
The PipelineRemoveFieldsProcessor inputs.
type [required]
\nenum
The processor type. The value should always be remove_fields. \nAllowed enum values: remove_fields
default: remove_fields
Option 6
\nobject
The rename_fields processor changes field names.
fields [required]
\n[object]
A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields.
destination [required]
\nstring
The field name to assign the renamed value to.
preserve_source [required]
\nboolean
Indicates whether the original field, that is received from the source, should be kept (true) or removed (false) after renaming.
source [required]
\nstring
The original field name in the log event that should be renamed.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be rename_fields. \nAllowed enum values: rename_fields
default: rename_fields
Option 7
\nobject
The generate_datadog_metrics processor creates custom metrics from logs and sends them to Datadog.\nMetrics can be counters, gauges, or distributions and optionally grouped by log fields.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
metrics [required]
\n[object]
Configuration for generating individual metrics.
group_by
\n[string]
Optional fields used to group the metric series.
include [required]
\nstring
Datadog filter query to match logs for metric generation.
metric_type [required]
\nenum
Type of metric to create. \nAllowed enum values: count,gauge,distribution
name [required]
\nstring
Name of the custom metric to be created.
value [required]
\n<oneOf>
Specifies how the value of the generated metric is computed.
Option 1
\nobject
Strategy that increments a generated metric by one for each matching event.
strategy [required]
\nenum
Increments the metric by 1 for each matching event. \nAllowed enum values: increment_by_one
Option 2
\nobject
Strategy that increments a generated metric based on the value of a log field.
field [required]
\nstring
Name of the log field containing the numeric value to increment the metric by.
strategy [required]
\nenum
Uses a numeric field in the log event as the metric increment. \nAllowed enum values: increment_by_field
type [required]
\nenum
The processor type. Always generate_datadog_metrics. \nAllowed enum values: generate_datadog_metrics
default: generate_datadog_metrics
Option 8
\nobject
The sample processor allows probabilistic sampling of logs at a fixed rate.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
percentage
\ndouble
The percentage of logs to sample.
rate
\nint64
Number of events to sample (1 in N).
type [required]
\nenum
The processor type. The value should always be sample. \nAllowed enum values: sample
default: sample
Option 9
\nobject
The parse_grok processor extracts structured fields from unstructured log messages using Grok patterns.
disable_library_rules
\nboolean
If set to true, disables the default Grok rules provided by Datadog.
id [required]
\nstring
A unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
rules [required]
\n[object]
The list of Grok parsing rules. If multiple matching rules are provided, they are evaluated in order. The first successful match is applied.
match_rules [required]
\n[object]
A list of Grok parsing rules that define how to extract fields from the source field.\nEach rule must contain a name and a valid Grok pattern.
name [required]
\nstring
The name of the rule.
rule [required]
\nstring
The definition of the Grok rule.
source [required]
\nstring
The name of the field in the log event to apply the Grok rules to.
support_rules [required]
\n[object]
A list of Grok helper rules that can be referenced by the parsing rules.
name [required]
\nstring
The name of the Grok helper rule.
rule [required]
\nstring
The definition of the Grok helper rule.
type [required]
\nenum
The processor type. The value should always be parse_grok. \nAllowed enum values: parse_grok
default: parse_grok
Option 10
\nobject
The sensitive_data_scanner processor detects and optionally redacts sensitive data in log events.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
rules [required]
\n[object]
A list of rules for identifying and acting on sensitive data patterns.
keyword_options
\nobject
Configuration for keywords used to reinforce sensitive data pattern detection.
keywords [required]
\n[string]
A list of keywords to match near the sensitive pattern.
proximity [required]
\nint64
Maximum number of tokens between a keyword and a sensitive value match.
name [required]
\nstring
A name identifying the rule.
on_match [required]
\n<oneOf>
Defines what action to take when sensitive data is matched.
Option 1
\nobject
Configuration for completely redacting matched sensitive data.
action [required]
\nenum
Action type that completely replaces the matched sensitive data with a fixed replacement string to remove all visibility. \nAllowed enum values: redact
options [required]
\nobject
Configuration for fully redacting sensitive data.
replace [required]
\nstring
The ObservabilityPipelineSensitiveDataScannerProcessorActionRedactOptions replace.
Option 2
\nobject
Configuration for hashing matched sensitive values.
action [required]
\nenum
Action type that replaces the matched sensitive data with a hashed representation, preserving structure while securing content. \nAllowed enum values: hash
options
\nobject
The ObservabilityPipelineSensitiveDataScannerProcessorActionHash options.
Option 3
\nobject
Configuration for partially redacting matched sensitive data.
action [required]
\nenum
Action type that redacts part of the sensitive data while preserving a configurable number of characters, typically used for masking purposes (e.g., show last 4 digits of a credit card). \nAllowed enum values: partial_redact
options [required]
\nobject
Controls how partial redaction is applied, including character count and direction.
characters [required]
\nint64
The ObservabilityPipelineSensitiveDataScannerProcessorActionPartialRedactOptions characters.
direction [required]
\nenum
Indicates whether to redact characters from the first or last part of the matched value. \nAllowed enum values: first,last
pattern [required]
\n<oneOf>
Pattern detection configuration for identifying sensitive data using either a custom regex or a library reference.
Option 1
\nobject
Defines a custom regex-based pattern for identifying sensitive data in logs.
options [required]
\nobject
Options for defining a custom regex pattern.
rule [required]
\nstring
A regular expression used to detect sensitive values. Must be a valid regex.
type [required]
\nenum
Indicates a custom regular expression is used for matching. \nAllowed enum values: custom
Option 2
\nobject
Specifies a pattern from Datadog’s sensitive data detection library to match known sensitive data types.
options [required]
\nobject
Options for selecting a predefined library pattern and enabling keyword support.
id [required]
\nstring
Identifier for a predefined pattern from the sensitive data scanner pattern library.
use_recommended_keywords
\nboolean
Whether to augment the pattern with recommended keywords (optional).
type [required]
\nenum
Indicates that a predefined library pattern is used. \nAllowed enum values: library
scope [required]
\n<oneOf>
Determines which parts of the log the pattern-matching rule should be applied to.
Option 1
\nobject
Includes only specific fields for sensitive data scanning.
options [required]
\nobject
Fields to which the scope rule applies.
fields [required]
\n[string]
The ObservabilityPipelineSensitiveDataScannerProcessorScopeOptions fields.
target [required]
\nenum
Applies the rule only to included fields. \nAllowed enum values: include
Option 2
\nobject
Excludes specific fields from sensitive data scanning.
options [required]
\nobject
Fields to which the scope rule applies.
fields [required]
\n[string]
The ObservabilityPipelineSensitiveDataScannerProcessorScopeOptions fields.
target [required]
\nenum
Excludes specific fields from processing. \nAllowed enum values: exclude
Option 3
\nobject
Applies scanning across all available fields.
target [required]
\nenum
Applies the rule to all fields. \nAllowed enum values: all
tags [required]
\n[string]
Tags assigned to this rule for filtering and classification.
type [required]
\nenum
The processor type. The value should always be sensitive_data_scanner. \nAllowed enum values: sensitive_data_scanner
default: sensitive_data_scanner
Option 11
\nobject
The ocsf_mapper processor transforms logs into the OCSF schema using a predefined mapping configuration.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mappings [required]
\n[object]
A list of mapping rules to convert events to the OCSF format.
include [required]
\nstring
A Datadog search query used to select the logs that this mapping should apply to.
mapping [required]
\n<oneOf>
Defines a single mapping rule for transforming logs into the OCSF schema.
Option 1
\nenum
Predefined library mappings for common log formats. \nAllowed enum values: CloudTrail Account Change,GCP Cloud Audit CreateBucket,GCP Cloud Audit CreateSink,GCP Cloud Audit SetIamPolicy,GCP Cloud Audit UpdateSink,Github Audit Log API Activity,Google Workspace Admin Audit addPrivilege,Microsoft 365 Defender Incident,Microsoft 365 Defender UserLoggedIn,Okta System Log Authentication,Palo Alto Networks Firewall Traffic
type [required]
\nenum
The processor type. The value should always be ocsf_mapper. \nAllowed enum values: ocsf_mapper
default: ocsf_mapper
Option 12
\nobject
The add_env_vars processor adds environment variable values to log events.
id [required]
\nstring
The unique identifier for this component. Used to reference this processor in the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
type [required]
\nenum
The processor type. The value should always be add_env_vars. \nAllowed enum values: add_env_vars
default: add_env_vars
variables [required]
\n[object]
A list of environment variable mappings to apply to log fields.
field [required]
\nstring
The target field in the log event.
name [required]
\nstring
The name of the environment variable to read.
Option 13
\nobject
The dedupe processor removes duplicate fields in log events.
fields [required]
\n[string]
A list of log field paths to check for duplicates.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mode [required]
\nenum
The deduplication mode to apply to the fields. \nAllowed enum values: match,ignore
type [required]
\nenum
The processor type. The value should always be dedupe. \nAllowed enum values: dedupe
default: dedupe
Option 14
\nobject
The enrichment_table processor enriches logs using a static CSV file or GeoIP database.
file
\nobject
Defines a static enrichment table loaded from a CSV file.
encoding [required]
\nobject
File encoding format.
delimiter [required]
\nstring
The encoding delimiter.
includes_headers [required]
\nboolean
The encoding includes_headers.
type [required]
\nenum
Specifies the encoding format (e.g., CSV) used for enrichment tables. \nAllowed enum values: csv
key [required]
\n[object]
Key fields used to look up enrichment values.
column [required]
\nstring
The items column.
comparison [required]
\nenum
Defines how to compare key fields for enrichment table lookups. \nAllowed enum values: equals
field [required]
\nstring
The items field.
path [required]
\nstring
Path to the CSV file.
schema [required]
\n[object]
Schema defining column names and their types.
column [required]
\nstring
The items column.
type [required]
\nenum
Declares allowed data types for enrichment table columns. \nAllowed enum values: string,boolean,integer,float,date,timestamp
geoip
\nobject
Uses a GeoIP database to enrich logs based on an IP field.
key_field [required]
\nstring
Path to the IP field in the log.
locale [required]
\nstring
Locale used to resolve geographical names.
path [required]
\nstring
Path to the GeoIP database file.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
target [required]
\nstring
Path where enrichment results should be stored in the log.
type [required]
\nenum
The processor type. The value should always be enrichment_table. \nAllowed enum values: enrichment_table
default: enrichment_table
Option 15
\nobject
The reduce processor aggregates and merges logs based on matching keys and merge strategies.
group_by [required]
\n[string]
A list of fields used to group log events for merging.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
merge_strategies [required]
\n[object]
List of merge strategies defining how values from grouped events should be combined.
path [required]
\nstring
The field path in the log event.
strategy [required]
\nenum
The merge strategy to apply. \nAllowed enum values: discard,retain,sum,max,min,array,concat,concat_newline,concat_raw,shortest_array,longest_array,flat_unique
type [required]
\nenum
The processor type. The value should always be reduce. \nAllowed enum values: reduce
default: reduce
Option 16
\nobject
The throttle processor limits the number of events that pass through over a given time window.
group_by
\n[string]
Optional list of fields used to group events before the threshold has been reached.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
threshold [required]
\nint64
the number of events allowed in a given time window. Events sent after the threshold has been reached, are dropped.
type [required]
\nenum
The processor type. The value should always be throttle. \nAllowed enum values: throttle
default: throttle
window [required]
\ndouble
The time window in seconds over which the threshold applies.
sources [required]
\n[ <oneOf>]
A list of configured data sources for the pipeline.
Option 1
\nobject
The kafka source ingests data from Apache Kafka topics.
group_id [required]
\nstring
Consumer group ID used by the Kafka client.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
librdkafka_options
\n[object]
Optional list of advanced Kafka client configuration options, defined as key-value pairs.
name [required]
\nstring
The name of the librdkafka configuration option to set.
value [required]
\nstring
The value assigned to the specified librdkafka configuration option.
sasl
\nobject
Specifies the SASL mechanism for authenticating with a Kafka cluster.
mechanism
\nenum
SASL mechanism used for Kafka authentication. \nAllowed enum values: PLAIN,SCRAM-SHA-256,SCRAM-SHA-512
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
topics [required]
\n[string]
A list of Kafka topic names to subscribe to. The source ingests messages from each topic specified.
type [required]
\nenum
The source type. The value should always be kafka. \nAllowed enum values: kafka
default: kafka
Option 2
\nobject
The datadog_agent source collects logs from the Datadog Agent.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be datadog_agent. \nAllowed enum values: datadog_agent
default: datadog_agent
Option 3
\nobject
The splunk_tcp source receives logs from a Splunk Universal Forwarder over TCP.\nTLS is supported for secure transmission.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always splunk_tcp. \nAllowed enum values: splunk_tcp
default: splunk_tcp
Option 4
\nobject
The splunk_hec source implements the Splunk HTTP Event Collector (HEC) API.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always splunk_hec. \nAllowed enum values: splunk_hec
default: splunk_hec
Option 5
\nobject
The amazon_s3 source ingests logs from an Amazon S3 bucket.\nIt supports AWS authentication and TLS encryption.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
region [required]
\nstring
AWS region where the S3 bucket resides.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always amazon_s3. \nAllowed enum values: amazon_s3
default: amazon_s3
Option 6
\nobject
The fluentd source ingests logs from a Fluentd-compatible service.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be `fluentd. \nAllowed enum values: fluentd
default: fluentd
Option 7
\nobject
The fluent_bit source ingests logs from Fluent Bit.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be fluent_bit. \nAllowed enum values: fluent_bit
default: fluent_bit
Option 8
\nobject
The http_server source collects logs over HTTP POST from external services.
auth_strategy [required]
\nenum
HTTP authentication method. \nAllowed enum values: none,plain
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
Unique ID for the HTTP server source.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be http_server. \nAllowed enum values: http_server
default: http_server
Option 9
\nobject
The sumo_logic source receives logs from Sumo Logic collectors.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
type [required]
\nenum
The source type. The value should always be sumo_logic. \nAllowed enum values: sumo_logic
default: sumo_logic
Option 10
\nobject
The rsyslog source listens for logs over TCP or UDP from an rsyslog server using the syslog protocol.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
mode [required]
\nenum
Protocol used by the syslog source to receive messages. \nAllowed enum values: tcp,udp
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be rsyslog. \nAllowed enum values: rsyslog
default: rsyslog
Option 11
\nobject
The syslog_ng source listens for logs over TCP or UDP from a syslog-ng server using the syslog protocol.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
mode [required]
\nenum
Protocol used by the syslog source to receive messages. \nAllowed enum values: tcp,udp
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be syslog_ng. \nAllowed enum values: syslog_ng
default: syslog_ng
Option 12
\nobject
The amazon_data_firehose source ingests logs from AWS Data Firehose.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be amazon_data_firehose. \nAllowed enum values: amazon_data_firehose
default: amazon_data_firehose
Option 13
\nobject
The google_pubsub source ingests logs from a Google Cloud Pub/Sub subscription.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
project [required]
\nstring
The GCP project ID that owns the Pub/Sub subscription.
subscription [required]
\nstring
The Pub/Sub subscription name from which messages are consumed.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be google_pubsub. \nAllowed enum values: google_pubsub
default: google_pubsub
Option 14
\nobject
The http_client source scrapes logs from HTTP endpoints at regular intervals.
auth_strategy
\nenum
Optional authentication strategy for HTTP requests. \nAllowed enum values: basic,bearer
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
scrape_interval_secs
\nint64
The interval (in seconds) between HTTP scrape requests.
scrape_timeout_secs
\nint64
The timeout (in seconds) for each scrape request.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be http_client. \nAllowed enum values: http_client
default: http_client
Option 15
\nobject
The logstash source ingests logs from a Logstash forwarder.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be logstash. \nAllowed enum values: logstash
default: logstash
name [required]
\nstring
Name of the pipeline.
type [required]
\nstring
The resource type identifier. For pipeline resources, this should always be set to pipelines.
default: pipelines
data [required]
\nobject
Contains the the pipeline configuration.
attributes [required]
\nobject
Defines the pipeline’s name and its components (sources, processors, and destinations).
config [required]
\nobject
Specifies the pipeline's configuration, including its sources, processors, and destinations.
destinations [required]
\n[ <oneOf>]
A list of destination components where processed logs are sent.
Option 1
\nobject
The datadog_logs destination forwards logs to Datadog Log Management.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be datadog_logs. \nAllowed enum values: datadog_logs
default: datadog_logs
Option 2
\nobject
The amazon_s3 destination sends your logs in Datadog-rehydratable format to an Amazon S3 bucket for archiving.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
bucket [required]
\nstring
S3 bucket name.
id [required]
\nstring
Unique identifier for the destination component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
key_prefix
\nstring
Optional prefix for object keys.
region [required]
\nstring
AWS region of the S3 bucket.
storage_class [required]
\nenum
S3 storage class. \nAllowed enum values: STANDARD,REDUCED_REDUNDANCY,INTELLIGENT_TIERING,STANDARD_IA,EXPRESS_ONEZONE,ONEZONE_IA,GLACIER,GLACIER_IR,DEEP_ARCHIVE
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. Always amazon_s3. \nAllowed enum values: amazon_s3
default: amazon_s3
Option 3
\nobject
The google_cloud_storage destination stores logs in a Google Cloud Storage (GCS) bucket.\nIt requires a bucket name, GCP authentication, and metadata fields.
acl [required]
\nenum
Access control list setting for objects written to the bucket. \nAllowed enum values: private,project-private,public-read,authenticated-read,bucket-owner-read,bucket-owner-full-control
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
bucket [required]
\nstring
Name of the GCS bucket.
id [required]
\nstring
Unique identifier for the destination component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
key_prefix
\nstring
Optional prefix for object keys within the GCS bucket.
metadata
\n[object]
Custom metadata to attach to each object uploaded to the GCS bucket.
name [required]
\nstring
The metadata key.
value [required]
\nstring
The metadata value.
storage_class [required]
\nenum
Storage class used for objects stored in GCS. \nAllowed enum values: STANDARD,NEARLINE,COLDLINE,ARCHIVE
type [required]
\nenum
The destination type. Always google_cloud_storage. \nAllowed enum values: google_cloud_storage
default: google_cloud_storage
Option 4
\nobject
The splunk_hec destination forwards logs to Splunk using the HTTP Event Collector (HEC).
auto_extract_timestamp
\nboolean
If true, Splunk tries to extract timestamps from incoming log events.\nIf false, Splunk assigns the time the event was received.
encoding
\nenum
Encoding format for log events. \nAllowed enum values: json,raw_message
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
index
\nstring
Optional name of the Splunk index where logs are written.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
sourcetype
\nstring
The Splunk sourcetype to assign to log events.
type [required]
\nenum
The destination type. Always splunk_hec. \nAllowed enum values: splunk_hec
default: splunk_hec
Option 5
\nobject
The sumo_logic destination forwards logs to Sumo Logic.
encoding
\nenum
The output encoding format. \nAllowed enum values: json,raw_message,logfmt
header_custom_fields
\n[object]
A list of custom headers to include in the request to Sumo Logic.
name [required]
\nstring
The header field name.
value [required]
\nstring
The header field value.
header_host_name
\nstring
Optional override for the host name header.
header_source_category
\nstring
Optional override for the source category header.
header_source_name
\nstring
Optional override for the source name header.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be sumo_logic. \nAllowed enum values: sumo_logic
default: sumo_logic
Option 6
\nobject
The elasticsearch destination writes logs to an Elasticsearch cluster.
api_version
\nenum
The Elasticsearch API version to use. Set to auto to auto-detect. \nAllowed enum values: auto,v6,v7,v8
bulk_index
\nstring
The index to write logs to in Elasticsearch.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be elasticsearch. \nAllowed enum values: elasticsearch
default: elasticsearch
Option 7
\nobject
The rsyslog destination forwards logs to an external rsyslog server over TCP or UDP using the syslog protocol.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
keepalive
\nint64
Optional socket keepalive duration in milliseconds.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. The value should always be rsyslog. \nAllowed enum values: rsyslog
default: rsyslog
Option 8
\nobject
The syslog_ng destination forwards logs to an external syslog-ng server over TCP or UDP using the syslog protocol.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
keepalive
\nint64
Optional socket keepalive duration in milliseconds.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. The value should always be syslog_ng. \nAllowed enum values: syslog_ng
default: syslog_ng
Option 9
\nobject
The azure_storage destination forwards logs to an Azure Blob Storage container.
blob_prefix
\nstring
Optional prefix for blobs written to the container.
container_name [required]
\nstring
The name of the Azure Blob Storage container to store logs in.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be azure_storage. \nAllowed enum values: azure_storage
default: azure_storage
Option 10
\nobject
The microsoft_sentinel destination forwards logs to Microsoft Sentinel.
client_id [required]
\nstring
Azure AD client ID used for authentication.
dcr_immutable_id [required]
\nstring
The immutable ID of the Data Collection Rule (DCR).
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
table [required]
\nstring
The name of the Log Analytics table where logs are sent.
tenant_id [required]
\nstring
Azure AD tenant ID.
type [required]
\nenum
The destination type. The value should always be microsoft_sentinel. \nAllowed enum values: microsoft_sentinel
default: microsoft_sentinel
Option 11
\nobject
The google_chronicle destination sends logs to Google Chronicle.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
customer_id [required]
\nstring
The Google Chronicle customer ID.
encoding
\nenum
The encoding format for the logs sent to Chronicle. \nAllowed enum values: json,raw_message
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
log_type
\nstring
The log type metadata associated with the Chronicle destination.
type [required]
\nenum
The destination type. The value should always be google_chronicle. \nAllowed enum values: google_chronicle
default: google_chronicle
Option 12
\nobject
The new_relic destination sends logs to the New Relic platform.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The New Relic region. \nAllowed enum values: us,eu
type [required]
\nenum
The destination type. The value should always be new_relic. \nAllowed enum values: new_relic
default: new_relic
Option 13
\nobject
The sentinel_one destination sends logs to SentinelOne.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The SentinelOne region to send logs to. \nAllowed enum values: us,eu,ca,data_set_us
type [required]
\nenum
The destination type. The value should always be sentinel_one. \nAllowed enum values: sentinel_one
default: sentinel_one
Option 14
\nobject
The opensearch destination writes logs to an OpenSearch cluster.
bulk_index
\nstring
The index to write logs to.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be opensearch. \nAllowed enum values: opensearch
default: opensearch
Option 15
\nobject
The amazon_opensearch destination writes logs to Amazon OpenSearch.
auth [required]
\nobject
Authentication settings for the Amazon OpenSearch destination.\nThe strategy field determines whether basic or AWS-based authentication is used.
assume_role
\nstring
The ARN of the role to assume (used with aws strategy).
aws_region
\nstring
AWS region
external_id
\nstring
External ID for the assumed role (used with aws strategy).
session_name
\nstring
Session name for the assumed role (used with aws strategy).
strategy [required]
\nenum
The authentication strategy to use. \nAllowed enum values: basic,aws
bulk_index
\nstring
The index to write logs to.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be amazon_opensearch. \nAllowed enum values: amazon_opensearch
default: amazon_opensearch
processors
\n[ <oneOf>]
A list of processors that transform or enrich log data.
Option 1
\nobject
The filter processor allows conditional processing of logs based on a Datadog search query. Logs that match the include query are passed through; others are discarded.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs should pass through the filter. Logs that match this query continue to downstream components; others are dropped.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be filter. \nAllowed enum values: filter
default: filter
Option 2
\nobject
The parse_json processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string.
field [required]
\nstring
The name of the log field that contains a JSON string.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be parse_json. \nAllowed enum values: parse_json
default: parse_json
Option 3
\nobject
The Quota Processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert.
drop_events [required]
\nboolean
If set to true, logs that matched the quota filter and sent after the quota has been met are dropped; only logs that did not match the filter query continue through the pipeline.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
ignore_when_missing_partitions
\nboolean
If true, the processor skips quota checks when partition fields are missing from the logs.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
name [required]
\nstring
Name of the quota.
overflow_action
\nenum
The action to take when the quota is exceeded. Options:
\ndrop: Drop the event.
no_action: Let the event pass through.
overflow_routing: Route to an overflow destination.
Allowed enum values: drop,no_action,overflow_routing
overrides
\n[object]
A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit.
fields [required]
\n[object]
A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
partition_fields
\n[string]
A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values.
type [required]
\nenum
The processor type. The value should always be quota. \nAllowed enum values: quota
default: quota
Option 4
\nobject
The add_fields processor adds static key-value fields to logs.
fields [required]
\n[object]
A list of static fields (key-value pairs) that is added to each log event processed by this component.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be add_fields. \nAllowed enum values: add_fields
default: add_fields
Option 5
\nobject
The remove_fields processor deletes specified fields from logs.
fields [required]
\n[string]
A list of field names to be removed from each log event.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
The PipelineRemoveFieldsProcessor inputs.
type [required]
\nenum
The processor type. The value should always be remove_fields. \nAllowed enum values: remove_fields
default: remove_fields
Option 6
\nobject
The rename_fields processor changes field names.
fields [required]
\n[object]
A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields.
destination [required]
\nstring
The field name to assign the renamed value to.
preserve_source [required]
\nboolean
Indicates whether the original field, that is received from the source, should be kept (true) or removed (false) after renaming.
source [required]
\nstring
The original field name in the log event that should be renamed.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be rename_fields. \nAllowed enum values: rename_fields
default: rename_fields
Option 7
\nobject
The generate_datadog_metrics processor creates custom metrics from logs and sends them to Datadog.\nMetrics can be counters, gauges, or distributions and optionally grouped by log fields.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
metrics [required]
\n[object]
Configuration for generating individual metrics.
group_by
\n[string]
Optional fields used to group the metric series.
include [required]
\nstring
Datadog filter query to match logs for metric generation.
metric_type [required]
\nenum
Type of metric to create. \nAllowed enum values: count,gauge,distribution
name [required]
\nstring
Name of the custom metric to be created.
value [required]
\n<oneOf>
Specifies how the value of the generated metric is computed.
Option 1
\nobject
Strategy that increments a generated metric by one for each matching event.
strategy [required]
\nenum
Increments the metric by 1 for each matching event. \nAllowed enum values: increment_by_one
Option 2
\nobject
Strategy that increments a generated metric based on the value of a log field.
field [required]
\nstring
Name of the log field containing the numeric value to increment the metric by.
strategy [required]
\nenum
Uses a numeric field in the log event as the metric increment. \nAllowed enum values: increment_by_field
type [required]
\nenum
The processor type. Always generate_datadog_metrics. \nAllowed enum values: generate_datadog_metrics
default: generate_datadog_metrics
Option 8
\nobject
The sample processor allows probabilistic sampling of logs at a fixed rate.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
percentage
\ndouble
The percentage of logs to sample.
rate
\nint64
Number of events to sample (1 in N).
type [required]
\nenum
The processor type. The value should always be sample. \nAllowed enum values: sample
default: sample
Option 9
\nobject
The parse_grok processor extracts structured fields from unstructured log messages using Grok patterns.
disable_library_rules
\nboolean
If set to true, disables the default Grok rules provided by Datadog.
id [required]
\nstring
A unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
rules [required]
\n[object]
The list of Grok parsing rules. If multiple matching rules are provided, they are evaluated in order. The first successful match is applied.
match_rules [required]
\n[object]
A list of Grok parsing rules that define how to extract fields from the source field.\nEach rule must contain a name and a valid Grok pattern.
name [required]
\nstring
The name of the rule.
rule [required]
\nstring
The definition of the Grok rule.
source [required]
\nstring
The name of the field in the log event to apply the Grok rules to.
support_rules [required]
\n[object]
A list of Grok helper rules that can be referenced by the parsing rules.
name [required]
\nstring
The name of the Grok helper rule.
rule [required]
\nstring
The definition of the Grok helper rule.
type [required]
\nenum
The processor type. The value should always be parse_grok. \nAllowed enum values: parse_grok
default: parse_grok
Option 10
\nobject
The sensitive_data_scanner processor detects and optionally redacts sensitive data in log events.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
rules [required]
\n[object]
A list of rules for identifying and acting on sensitive data patterns.
keyword_options
\nobject
Configuration for keywords used to reinforce sensitive data pattern detection.
keywords [required]
\n[string]
A list of keywords to match near the sensitive pattern.
proximity [required]
\nint64
Maximum number of tokens between a keyword and a sensitive value match.
name [required]
\nstring
A name identifying the rule.
on_match [required]
\n<oneOf>
Defines what action to take when sensitive data is matched.
Option 1
\nobject
Configuration for completely redacting matched sensitive data.
action [required]
\nenum
Action type that completely replaces the matched sensitive data with a fixed replacement string to remove all visibility. \nAllowed enum values: redact
options [required]
\nobject
Configuration for fully redacting sensitive data.
replace [required]
\nstring
The ObservabilityPipelineSensitiveDataScannerProcessorActionRedactOptions replace.
Option 2
\nobject
Configuration for hashing matched sensitive values.
action [required]
\nenum
Action type that replaces the matched sensitive data with a hashed representation, preserving structure while securing content. \nAllowed enum values: hash
options
\nobject
The ObservabilityPipelineSensitiveDataScannerProcessorActionHash options.
Option 3
\nobject
Configuration for partially redacting matched sensitive data.
action [required]
\nenum
Action type that redacts part of the sensitive data while preserving a configurable number of characters, typically used for masking purposes (e.g., show last 4 digits of a credit card). \nAllowed enum values: partial_redact
options [required]
\nobject
Controls how partial redaction is applied, including character count and direction.
characters [required]
\nint64
The ObservabilityPipelineSensitiveDataScannerProcessorActionPartialRedactOptions characters.
direction [required]
\nenum
Indicates whether to redact characters from the first or last part of the matched value. \nAllowed enum values: first,last
pattern [required]
\n<oneOf>
Pattern detection configuration for identifying sensitive data using either a custom regex or a library reference.
Option 1
\nobject
Defines a custom regex-based pattern for identifying sensitive data in logs.
options [required]
\nobject
Options for defining a custom regex pattern.
rule [required]
\nstring
A regular expression used to detect sensitive values. Must be a valid regex.
type [required]
\nenum
Indicates a custom regular expression is used for matching. \nAllowed enum values: custom
Option 2
\nobject
Specifies a pattern from Datadog’s sensitive data detection library to match known sensitive data types.
options [required]
\nobject
Options for selecting a predefined library pattern and enabling keyword support.
id [required]
\nstring
Identifier for a predefined pattern from the sensitive data scanner pattern library.
use_recommended_keywords
\nboolean
Whether to augment the pattern with recommended keywords (optional).
type [required]
\nenum
Indicates that a predefined library pattern is used. \nAllowed enum values: library
scope [required]
\n<oneOf>
Determines which parts of the log the pattern-matching rule should be applied to.
Option 1
\nobject
Includes only specific fields for sensitive data scanning.
options [required]
\nobject
Fields to which the scope rule applies.
fields [required]
\n[string]
The ObservabilityPipelineSensitiveDataScannerProcessorScopeOptions fields.
target [required]
\nenum
Applies the rule only to included fields. \nAllowed enum values: include
Option 2
\nobject
Excludes specific fields from sensitive data scanning.
options [required]
\nobject
Fields to which the scope rule applies.
fields [required]
\n[string]
The ObservabilityPipelineSensitiveDataScannerProcessorScopeOptions fields.
target [required]
\nenum
Excludes specific fields from processing. \nAllowed enum values: exclude
Option 3
\nobject
Applies scanning across all available fields.
target [required]
\nenum
Applies the rule to all fields. \nAllowed enum values: all
tags [required]
\n[string]
Tags assigned to this rule for filtering and classification.
type [required]
\nenum
The processor type. The value should always be sensitive_data_scanner. \nAllowed enum values: sensitive_data_scanner
default: sensitive_data_scanner
Option 11
\nobject
The ocsf_mapper processor transforms logs into the OCSF schema using a predefined mapping configuration.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mappings [required]
\n[object]
A list of mapping rules to convert events to the OCSF format.
include [required]
\nstring
A Datadog search query used to select the logs that this mapping should apply to.
mapping [required]
\n<oneOf>
Defines a single mapping rule for transforming logs into the OCSF schema.
Option 1
\nenum
Predefined library mappings for common log formats. \nAllowed enum values: CloudTrail Account Change,GCP Cloud Audit CreateBucket,GCP Cloud Audit CreateSink,GCP Cloud Audit SetIamPolicy,GCP Cloud Audit UpdateSink,Github Audit Log API Activity,Google Workspace Admin Audit addPrivilege,Microsoft 365 Defender Incident,Microsoft 365 Defender UserLoggedIn,Okta System Log Authentication,Palo Alto Networks Firewall Traffic
type [required]
\nenum
The processor type. The value should always be ocsf_mapper. \nAllowed enum values: ocsf_mapper
default: ocsf_mapper
Option 12
\nobject
The add_env_vars processor adds environment variable values to log events.
id [required]
\nstring
The unique identifier for this component. Used to reference this processor in the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
type [required]
\nenum
The processor type. The value should always be add_env_vars. \nAllowed enum values: add_env_vars
default: add_env_vars
variables [required]
\n[object]
A list of environment variable mappings to apply to log fields.
field [required]
\nstring
The target field in the log event.
name [required]
\nstring
The name of the environment variable to read.
Option 13
\nobject
The dedupe processor removes duplicate fields in log events.
fields [required]
\n[string]
A list of log field paths to check for duplicates.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mode [required]
\nenum
The deduplication mode to apply to the fields. \nAllowed enum values: match,ignore
type [required]
\nenum
The processor type. The value should always be dedupe. \nAllowed enum values: dedupe
default: dedupe
Option 14
\nobject
The enrichment_table processor enriches logs using a static CSV file or GeoIP database.
file
\nobject
Defines a static enrichment table loaded from a CSV file.
encoding [required]
\nobject
File encoding format.
delimiter [required]
\nstring
The encoding delimiter.
includes_headers [required]
\nboolean
The encoding includes_headers.
type [required]
\nenum
Specifies the encoding format (e.g., CSV) used for enrichment tables. \nAllowed enum values: csv
key [required]
\n[object]
Key fields used to look up enrichment values.
column [required]
\nstring
The items column.
comparison [required]
\nenum
Defines how to compare key fields for enrichment table lookups. \nAllowed enum values: equals
field [required]
\nstring
The items field.
path [required]
\nstring
Path to the CSV file.
schema [required]
\n[object]
Schema defining column names and their types.
column [required]
\nstring
The items column.
type [required]
\nenum
Declares allowed data types for enrichment table columns. \nAllowed enum values: string,boolean,integer,float,date,timestamp
geoip
\nobject
Uses a GeoIP database to enrich logs based on an IP field.
key_field [required]
\nstring
Path to the IP field in the log.
locale [required]
\nstring
Locale used to resolve geographical names.
path [required]
\nstring
Path to the GeoIP database file.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
target [required]
\nstring
Path where enrichment results should be stored in the log.
type [required]
\nenum
The processor type. The value should always be enrichment_table. \nAllowed enum values: enrichment_table
default: enrichment_table
Option 15
\nobject
The reduce processor aggregates and merges logs based on matching keys and merge strategies.
group_by [required]
\n[string]
A list of fields used to group log events for merging.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
merge_strategies [required]
\n[object]
List of merge strategies defining how values from grouped events should be combined.
path [required]
\nstring
The field path in the log event.
strategy [required]
\nenum
The merge strategy to apply. \nAllowed enum values: discard,retain,sum,max,min,array,concat,concat_newline,concat_raw,shortest_array,longest_array,flat_unique
type [required]
\nenum
The processor type. The value should always be reduce. \nAllowed enum values: reduce
default: reduce
Option 16
\nobject
The throttle processor limits the number of events that pass through over a given time window.
group_by
\n[string]
Optional list of fields used to group events before the threshold has been reached.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
threshold [required]
\nint64
the number of events allowed in a given time window. Events sent after the threshold has been reached, are dropped.
type [required]
\nenum
The processor type. The value should always be throttle. \nAllowed enum values: throttle
default: throttle
window [required]
\ndouble
The time window in seconds over which the threshold applies.
Option 17
\nobject
The datadog_tags processor includes or excludes specific Datadog tags in your logs.
action [required]
\nenum
The action to take on tags with matching keys. \nAllowed enum values: include,exclude
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
keys [required]
\n[string]
A list of tag keys.
mode [required]
\nenum
The processing mode. \nAllowed enum values: filter
type [required]
\nenum
The processor type. The value should always be datadog_tags. \nAllowed enum values: datadog_tags
default: datadog_tags
sources [required]
\n[ <oneOf>]
A list of configured data sources for the pipeline.
Option 1
\nobject
The kafka source ingests data from Apache Kafka topics.
group_id [required]
\nstring
Consumer group ID used by the Kafka client.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
librdkafka_options
\n[object]
Optional list of advanced Kafka client configuration options, defined as key-value pairs.
name [required]
\nstring
The name of the librdkafka configuration option to set.
value [required]
\nstring
The value assigned to the specified librdkafka configuration option.
sasl
\nobject
Specifies the SASL mechanism for authenticating with a Kafka cluster.
mechanism
\nenum
SASL mechanism used for Kafka authentication. \nAllowed enum values: PLAIN,SCRAM-SHA-256,SCRAM-SHA-512
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
topics [required]
\n[string]
A list of Kafka topic names to subscribe to. The source ingests messages from each topic specified.
type [required]
\nenum
The source type. The value should always be kafka. \nAllowed enum values: kafka
default: kafka
Option 2
\nobject
The datadog_agent source collects logs from the Datadog Agent.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be datadog_agent. \nAllowed enum values: datadog_agent
default: datadog_agent
Option 3
\nobject
The splunk_tcp source receives logs from a Splunk Universal Forwarder over TCP.\nTLS is supported for secure transmission.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always splunk_tcp. \nAllowed enum values: splunk_tcp
default: splunk_tcp
Option 4
\nobject
The splunk_hec source implements the Splunk HTTP Event Collector (HEC) API.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always splunk_hec. \nAllowed enum values: splunk_hec
default: splunk_hec
Option 5
\nobject
The amazon_s3 source ingests logs from an Amazon S3 bucket.\nIt supports AWS authentication and TLS encryption.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
region [required]
\nstring
AWS region where the S3 bucket resides.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always amazon_s3. \nAllowed enum values: amazon_s3
default: amazon_s3
Option 6
\nobject
The fluentd source ingests logs from a Fluentd-compatible service.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be `fluentd. \nAllowed enum values: fluentd
default: fluentd
Option 7
\nobject
The fluent_bit source ingests logs from Fluent Bit.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be fluent_bit. \nAllowed enum values: fluent_bit
default: fluent_bit
Option 8
\nobject
The http_server source collects logs over HTTP POST from external services.
auth_strategy [required]
\nenum
HTTP authentication method. \nAllowed enum values: none,plain
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
Unique ID for the HTTP server source.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be http_server. \nAllowed enum values: http_server
default: http_server
Option 9
\nobject
The sumo_logic source receives logs from Sumo Logic collectors.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
type [required]
\nenum
The source type. The value should always be sumo_logic. \nAllowed enum values: sumo_logic
default: sumo_logic
Option 10
\nobject
The rsyslog source listens for logs over TCP or UDP from an rsyslog server using the syslog protocol.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
mode [required]
\nenum
Protocol used by the syslog source to receive messages. \nAllowed enum values: tcp,udp
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be rsyslog. \nAllowed enum values: rsyslog
default: rsyslog
Option 11
\nobject
The syslog_ng source listens for logs over TCP or UDP from a syslog-ng server using the syslog protocol.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
mode [required]
\nenum
Protocol used by the syslog source to receive messages. \nAllowed enum values: tcp,udp
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be syslog_ng. \nAllowed enum values: syslog_ng
default: syslog_ng
Option 12
\nobject
The amazon_data_firehose source ingests logs from AWS Data Firehose.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be amazon_data_firehose. \nAllowed enum values: amazon_data_firehose
default: amazon_data_firehose
Option 13
\nobject
The google_pubsub source ingests logs from a Google Cloud Pub/Sub subscription.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
project [required]
\nstring
The GCP project ID that owns the Pub/Sub subscription.
subscription [required]
\nstring
The Pub/Sub subscription name from which messages are consumed.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be google_pubsub. \nAllowed enum values: google_pubsub
default: google_pubsub
Option 14
\nobject
The http_client source scrapes logs from HTTP endpoints at regular intervals.
auth_strategy
\nenum
Optional authentication strategy for HTTP requests. \nAllowed enum values: basic,bearer
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
scrape_interval_secs
\nint64
The interval (in seconds) between HTTP scrape requests.
scrape_timeout_secs
\nint64
The timeout (in seconds) for each scrape request.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be http_client. \nAllowed enum values: http_client
default: http_client
Option 15
\nobject
The logstash source ingests logs from a Logstash forwarder.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be logstash. \nAllowed enum values: logstash
default: logstash
name [required]
\nstring
Name of the pipeline.
type [required]
\nstring
The resource type identifier. For pipeline resources, this should always be set to pipelines.
default: pipelines
data [required]
\nobject
Contains the the pipeline configuration.
attributes [required]
\nobject
Defines the pipeline’s name and its components (sources, processors, and destinations).
config [required]
\nobject
Specifies the pipeline's configuration, including its sources, processors, and destinations.
destinations [required]
\n[ <oneOf>]
A list of destination components where processed logs are sent.
Option 1
\nobject
The datadog_logs destination forwards logs to Datadog Log Management.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be datadog_logs. \nAllowed enum values: datadog_logs
default: datadog_logs
Option 2
\nobject
The amazon_s3 destination sends your logs in Datadog-rehydratable format to an Amazon S3 bucket for archiving.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
bucket [required]
\nstring
S3 bucket name.
id [required]
\nstring
Unique identifier for the destination component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
key_prefix
\nstring
Optional prefix for object keys.
region [required]
\nstring
AWS region of the S3 bucket.
storage_class [required]
\nenum
S3 storage class. \nAllowed enum values: STANDARD,REDUCED_REDUNDANCY,INTELLIGENT_TIERING,STANDARD_IA,EXPRESS_ONEZONE,ONEZONE_IA,GLACIER,GLACIER_IR,DEEP_ARCHIVE
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. Always amazon_s3. \nAllowed enum values: amazon_s3
default: amazon_s3
Option 3
\nobject
The google_cloud_storage destination stores logs in a Google Cloud Storage (GCS) bucket.\nIt requires a bucket name, GCP authentication, and metadata fields.
acl [required]
\nenum
Access control list setting for objects written to the bucket. \nAllowed enum values: private,project-private,public-read,authenticated-read,bucket-owner-read,bucket-owner-full-control
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
bucket [required]
\nstring
Name of the GCS bucket.
id [required]
\nstring
Unique identifier for the destination component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
key_prefix
\nstring
Optional prefix for object keys within the GCS bucket.
metadata
\n[object]
Custom metadata to attach to each object uploaded to the GCS bucket.
name [required]
\nstring
The metadata key.
value [required]
\nstring
The metadata value.
storage_class [required]
\nenum
Storage class used for objects stored in GCS. \nAllowed enum values: STANDARD,NEARLINE,COLDLINE,ARCHIVE
type [required]
\nenum
The destination type. Always google_cloud_storage. \nAllowed enum values: google_cloud_storage
default: google_cloud_storage
Option 4
\nobject
The splunk_hec destination forwards logs to Splunk using the HTTP Event Collector (HEC).
auto_extract_timestamp
\nboolean
If true, Splunk tries to extract timestamps from incoming log events.\nIf false, Splunk assigns the time the event was received.
encoding
\nenum
Encoding format for log events. \nAllowed enum values: json,raw_message
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
index
\nstring
Optional name of the Splunk index where logs are written.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
sourcetype
\nstring
The Splunk sourcetype to assign to log events.
type [required]
\nenum
The destination type. Always splunk_hec. \nAllowed enum values: splunk_hec
default: splunk_hec
Option 5
\nobject
The sumo_logic destination forwards logs to Sumo Logic.
encoding
\nenum
The output encoding format. \nAllowed enum values: json,raw_message,logfmt
header_custom_fields
\n[object]
A list of custom headers to include in the request to Sumo Logic.
name [required]
\nstring
The header field name.
value [required]
\nstring
The header field value.
header_host_name
\nstring
Optional override for the host name header.
header_source_category
\nstring
Optional override for the source category header.
header_source_name
\nstring
Optional override for the source name header.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be sumo_logic. \nAllowed enum values: sumo_logic
default: sumo_logic
Option 6
\nobject
The elasticsearch destination writes logs to an Elasticsearch cluster.
api_version
\nenum
The Elasticsearch API version to use. Set to auto to auto-detect. \nAllowed enum values: auto,v6,v7,v8
bulk_index
\nstring
The index to write logs to in Elasticsearch.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be elasticsearch. \nAllowed enum values: elasticsearch
default: elasticsearch
Option 7
\nobject
The rsyslog destination forwards logs to an external rsyslog server over TCP or UDP using the syslog protocol.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
keepalive
\nint64
Optional socket keepalive duration in milliseconds.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. The value should always be rsyslog. \nAllowed enum values: rsyslog
default: rsyslog
Option 8
\nobject
The syslog_ng destination forwards logs to an external syslog-ng server over TCP or UDP using the syslog protocol.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
keepalive
\nint64
Optional socket keepalive duration in milliseconds.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. The value should always be syslog_ng. \nAllowed enum values: syslog_ng
default: syslog_ng
Option 9
\nobject
The azure_storage destination forwards logs to an Azure Blob Storage container.
blob_prefix
\nstring
Optional prefix for blobs written to the container.
container_name [required]
\nstring
The name of the Azure Blob Storage container to store logs in.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be azure_storage. \nAllowed enum values: azure_storage
default: azure_storage
Option 10
\nobject
The microsoft_sentinel destination forwards logs to Microsoft Sentinel.
client_id [required]
\nstring
Azure AD client ID used for authentication.
dcr_immutable_id [required]
\nstring
The immutable ID of the Data Collection Rule (DCR).
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
table [required]
\nstring
The name of the Log Analytics table where logs are sent.
tenant_id [required]
\nstring
Azure AD tenant ID.
type [required]
\nenum
The destination type. The value should always be microsoft_sentinel. \nAllowed enum values: microsoft_sentinel
default: microsoft_sentinel
Option 11
\nobject
The google_chronicle destination sends logs to Google Chronicle.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
customer_id [required]
\nstring
The Google Chronicle customer ID.
encoding
\nenum
The encoding format for the logs sent to Chronicle. \nAllowed enum values: json,raw_message
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
log_type
\nstring
The log type metadata associated with the Chronicle destination.
type [required]
\nenum
The destination type. The value should always be google_chronicle. \nAllowed enum values: google_chronicle
default: google_chronicle
Option 12
\nobject
The new_relic destination sends logs to the New Relic platform.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The New Relic region. \nAllowed enum values: us,eu
type [required]
\nenum
The destination type. The value should always be new_relic. \nAllowed enum values: new_relic
default: new_relic
Option 13
\nobject
The sentinel_one destination sends logs to SentinelOne.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The SentinelOne region to send logs to. \nAllowed enum values: us,eu,ca,data_set_us
type [required]
\nenum
The destination type. The value should always be sentinel_one. \nAllowed enum values: sentinel_one
default: sentinel_one
Option 14
\nobject
The opensearch destination writes logs to an OpenSearch cluster.
bulk_index
\nstring
The index to write logs to.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be opensearch. \nAllowed enum values: opensearch
default: opensearch
Option 15
\nobject
The amazon_opensearch destination writes logs to Amazon OpenSearch.
auth [required]
\nobject
Authentication settings for the Amazon OpenSearch destination.\nThe strategy field determines whether basic or AWS-based authentication is used.
assume_role
\nstring
The ARN of the role to assume (used with aws strategy).
aws_region
\nstring
AWS region
external_id
\nstring
External ID for the assumed role (used with aws strategy).
session_name
\nstring
Session name for the assumed role (used with aws strategy).
strategy [required]
\nenum
The authentication strategy to use. \nAllowed enum values: basic,aws
bulk_index
\nstring
The index to write logs to.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be amazon_opensearch. \nAllowed enum values: amazon_opensearch
default: amazon_opensearch
processors
\n[ <oneOf>]
A list of processors that transform or enrich log data.
Option 1
\nobject
The filter processor allows conditional processing of logs based on a Datadog search query. Logs that match the include query are passed through; others are discarded.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs should pass through the filter. Logs that match this query continue to downstream components; others are dropped.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be filter. \nAllowed enum values: filter
default: filter
Option 2
\nobject
The parse_json processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string.
field [required]
\nstring
The name of the log field that contains a JSON string.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be parse_json. \nAllowed enum values: parse_json
default: parse_json
Option 3
\nobject
The Quota Processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert.
drop_events [required]
\nboolean
If set to true, logs that matched the quota filter and sent after the quota has been met are dropped; only logs that did not match the filter query continue through the pipeline.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
ignore_when_missing_partitions
\nboolean
If true, the processor skips quota checks when partition fields are missing from the logs.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
name [required]
\nstring
Name of the quota.
overflow_action
\nenum
The action to take when the quota is exceeded. Options:
\ndrop: Drop the event.
no_action: Let the event pass through.
overflow_routing: Route to an overflow destination.
Allowed enum values: drop,no_action,overflow_routing
overrides
\n[object]
A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit.
fields [required]
\n[object]
A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
partition_fields
\n[string]
A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values.
type [required]
\nenum
The processor type. The value should always be quota. \nAllowed enum values: quota
default: quota
Option 4
\nobject
The add_fields processor adds static key-value fields to logs.
fields [required]
\n[object]
A list of static fields (key-value pairs) that is added to each log event processed by this component.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be add_fields. \nAllowed enum values: add_fields
default: add_fields
Option 5
\nobject
The remove_fields processor deletes specified fields from logs.
fields [required]
\n[string]
A list of field names to be removed from each log event.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
The PipelineRemoveFieldsProcessor inputs.
type [required]
\nenum
The processor type. The value should always be remove_fields. \nAllowed enum values: remove_fields
default: remove_fields
Option 6
\nobject
The rename_fields processor changes field names.
fields [required]
\n[object]
A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields.
destination [required]
\nstring
The field name to assign the renamed value to.
preserve_source [required]
\nboolean
Indicates whether the original field, that is received from the source, should be kept (true) or removed (false) after renaming.
source [required]
\nstring
The original field name in the log event that should be renamed.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be rename_fields. \nAllowed enum values: rename_fields
default: rename_fields
Option 7
\nobject
The generate_datadog_metrics processor creates custom metrics from logs and sends them to Datadog.\nMetrics can be counters, gauges, or distributions and optionally grouped by log fields.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
metrics [required]
\n[object]
Configuration for generating individual metrics.
group_by
\n[string]
Optional fields used to group the metric series.
include [required]
\nstring
Datadog filter query to match logs for metric generation.
metric_type [required]
\nenum
Type of metric to create. \nAllowed enum values: count,gauge,distribution
name [required]
\nstring
Name of the custom metric to be created.
value [required]
\n<oneOf>
Specifies how the value of the generated metric is computed.
Option 1
\nobject
Strategy that increments a generated metric by one for each matching event.
strategy [required]
\nenum
Increments the metric by 1 for each matching event. \nAllowed enum values: increment_by_one
Option 2
\nobject
Strategy that increments a generated metric based on the value of a log field.
field [required]
\nstring
Name of the log field containing the numeric value to increment the metric by.
strategy [required]
\nenum
Uses a numeric field in the log event as the metric increment. \nAllowed enum values: increment_by_field
type [required]
\nenum
The processor type. Always generate_datadog_metrics. \nAllowed enum values: generate_datadog_metrics
default: generate_datadog_metrics
Option 8
\nobject
The sample processor allows probabilistic sampling of logs at a fixed rate.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
percentage
\ndouble
The percentage of logs to sample.
rate
\nint64
Number of events to sample (1 in N).
type [required]
\nenum
The processor type. The value should always be sample. \nAllowed enum values: sample
default: sample
Option 9
\nobject
The parse_grok processor extracts structured fields from unstructured log messages using Grok patterns.
disable_library_rules
\nboolean
If set to true, disables the default Grok rules provided by Datadog.
id [required]
\nstring
A unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
rules [required]
\n[object]
The list of Grok parsing rules. If multiple matching rules are provided, they are evaluated in order. The first successful match is applied.
match_rules [required]
\n[object]
A list of Grok parsing rules that define how to extract fields from the source field.\nEach rule must contain a name and a valid Grok pattern.
name [required]
\nstring
The name of the rule.
rule [required]
\nstring
The definition of the Grok rule.
source [required]
\nstring
The name of the field in the log event to apply the Grok rules to.
support_rules [required]
\n[object]
A list of Grok helper rules that can be referenced by the parsing rules.
name [required]
\nstring
The name of the Grok helper rule.
rule [required]
\nstring
The definition of the Grok helper rule.
type [required]
\nenum
The processor type. The value should always be parse_grok. \nAllowed enum values: parse_grok
default: parse_grok
Option 10
\nobject
The sensitive_data_scanner processor detects and optionally redacts sensitive data in log events.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
rules [required]
\n[object]
A list of rules for identifying and acting on sensitive data patterns.
keyword_options
\nobject
Configuration for keywords used to reinforce sensitive data pattern detection.
keywords [required]
\n[string]
A list of keywords to match near the sensitive pattern.
proximity [required]
\nint64
Maximum number of tokens between a keyword and a sensitive value match.
name [required]
\nstring
A name identifying the rule.
on_match [required]
\n<oneOf>
Defines what action to take when sensitive data is matched.
Option 1
\nobject
Configuration for completely redacting matched sensitive data.
action [required]
\nenum
Action type that completely replaces the matched sensitive data with a fixed replacement string to remove all visibility. \nAllowed enum values: redact
options [required]
\nobject
Configuration for fully redacting sensitive data.
replace [required]
\nstring
The ObservabilityPipelineSensitiveDataScannerProcessorActionRedactOptions replace.
Option 2
\nobject
Configuration for hashing matched sensitive values.
action [required]
\nenum
Action type that replaces the matched sensitive data with a hashed representation, preserving structure while securing content. \nAllowed enum values: hash
options
\nobject
The ObservabilityPipelineSensitiveDataScannerProcessorActionHash options.
Option 3
\nobject
Configuration for partially redacting matched sensitive data.
action [required]
\nenum
Action type that redacts part of the sensitive data while preserving a configurable number of characters, typically used for masking purposes (e.g., show last 4 digits of a credit card). \nAllowed enum values: partial_redact
options [required]
\nobject
Controls how partial redaction is applied, including character count and direction.
characters [required]
\nint64
The ObservabilityPipelineSensitiveDataScannerProcessorActionPartialRedactOptions characters.
direction [required]
\nenum
Indicates whether to redact characters from the first or last part of the matched value. \nAllowed enum values: first,last
pattern [required]
\n<oneOf>
Pattern detection configuration for identifying sensitive data using either a custom regex or a library reference.
Option 1
\nobject
Defines a custom regex-based pattern for identifying sensitive data in logs.
options [required]
\nobject
Options for defining a custom regex pattern.
rule [required]
\nstring
A regular expression used to detect sensitive values. Must be a valid regex.
type [required]
\nenum
Indicates a custom regular expression is used for matching. \nAllowed enum values: custom
Option 2
\nobject
Specifies a pattern from Datadog’s sensitive data detection library to match known sensitive data types.
options [required]
\nobject
Options for selecting a predefined library pattern and enabling keyword support.
id [required]
\nstring
Identifier for a predefined pattern from the sensitive data scanner pattern library.
use_recommended_keywords
\nboolean
Whether to augment the pattern with recommended keywords (optional).
type [required]
\nenum
Indicates that a predefined library pattern is used. \nAllowed enum values: library
scope [required]
\n<oneOf>
Determines which parts of the log the pattern-matching rule should be applied to.
Option 1
\nobject
Includes only specific fields for sensitive data scanning.
options [required]
\nobject
Fields to which the scope rule applies.
fields [required]
\n[string]
The ObservabilityPipelineSensitiveDataScannerProcessorScopeOptions fields.
target [required]
\nenum
Applies the rule only to included fields. \nAllowed enum values: include
Option 2
\nobject
Excludes specific fields from sensitive data scanning.
options [required]
\nobject
Fields to which the scope rule applies.
fields [required]
\n[string]
The ObservabilityPipelineSensitiveDataScannerProcessorScopeOptions fields.
target [required]
\nenum
Excludes specific fields from processing. \nAllowed enum values: exclude
Option 3
\nobject
Applies scanning across all available fields.
target [required]
\nenum
Applies the rule to all fields. \nAllowed enum values: all
tags [required]
\n[string]
Tags assigned to this rule for filtering and classification.
type [required]
\nenum
The processor type. The value should always be sensitive_data_scanner. \nAllowed enum values: sensitive_data_scanner
default: sensitive_data_scanner
Option 11
\nobject
The ocsf_mapper processor transforms logs into the OCSF schema using a predefined mapping configuration.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mappings [required]
\n[object]
A list of mapping rules to convert events to the OCSF format.
include [required]
\nstring
A Datadog search query used to select the logs that this mapping should apply to.
mapping [required]
\n<oneOf>
Defines a single mapping rule for transforming logs into the OCSF schema.
Option 1
\nenum
Predefined library mappings for common log formats. \nAllowed enum values: CloudTrail Account Change,GCP Cloud Audit CreateBucket,GCP Cloud Audit CreateSink,GCP Cloud Audit SetIamPolicy,GCP Cloud Audit UpdateSink,Github Audit Log API Activity,Google Workspace Admin Audit addPrivilege,Microsoft 365 Defender Incident,Microsoft 365 Defender UserLoggedIn,Okta System Log Authentication,Palo Alto Networks Firewall Traffic
type [required]
\nenum
The processor type. The value should always be ocsf_mapper. \nAllowed enum values: ocsf_mapper
default: ocsf_mapper
Option 12
\nobject
The add_env_vars processor adds environment variable values to log events.
id [required]
\nstring
The unique identifier for this component. Used to reference this processor in the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
type [required]
\nenum
The processor type. The value should always be add_env_vars. \nAllowed enum values: add_env_vars
default: add_env_vars
variables [required]
\n[object]
A list of environment variable mappings to apply to log fields.
field [required]
\nstring
The target field in the log event.
name [required]
\nstring
The name of the environment variable to read.
Option 13
\nobject
The dedupe processor removes duplicate fields in log events.
fields [required]
\n[string]
A list of log field paths to check for duplicates.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mode [required]
\nenum
The deduplication mode to apply to the fields. \nAllowed enum values: match,ignore
type [required]
\nenum
The processor type. The value should always be dedupe. \nAllowed enum values: dedupe
default: dedupe
Option 14
\nobject
The enrichment_table processor enriches logs using a static CSV file or GeoIP database.
file
\nobject
Defines a static enrichment table loaded from a CSV file.
encoding [required]
\nobject
File encoding format.
delimiter [required]
\nstring
The encoding delimiter.
includes_headers [required]
\nboolean
The encoding includes_headers.
type [required]
\nenum
Specifies the encoding format (e.g., CSV) used for enrichment tables. \nAllowed enum values: csv
key [required]
\n[object]
Key fields used to look up enrichment values.
column [required]
\nstring
The items column.
comparison [required]
\nenum
Defines how to compare key fields for enrichment table lookups. \nAllowed enum values: equals
field [required]
\nstring
The items field.
path [required]
\nstring
Path to the CSV file.
schema [required]
\n[object]
Schema defining column names and their types.
column [required]
\nstring
The items column.
type [required]
\nenum
Declares allowed data types for enrichment table columns. \nAllowed enum values: string,boolean,integer,float,date,timestamp
geoip
\nobject
Uses a GeoIP database to enrich logs based on an IP field.
key_field [required]
\nstring
Path to the IP field in the log.
locale [required]
\nstring
Locale used to resolve geographical names.
path [required]
\nstring
Path to the GeoIP database file.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
target [required]
\nstring
Path where enrichment results should be stored in the log.
type [required]
\nenum
The processor type. The value should always be enrichment_table. \nAllowed enum values: enrichment_table
default: enrichment_table
Option 15
\nobject
The reduce processor aggregates and merges logs based on matching keys and merge strategies.
group_by [required]
\n[string]
A list of fields used to group log events for merging.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
merge_strategies [required]
\n[object]
List of merge strategies defining how values from grouped events should be combined.
path [required]
\nstring
The field path in the log event.
strategy [required]
\nenum
The merge strategy to apply. \nAllowed enum values: discard,retain,sum,max,min,array,concat,concat_newline,concat_raw,shortest_array,longest_array,flat_unique
type [required]
\nenum
The processor type. The value should always be reduce. \nAllowed enum values: reduce
default: reduce
Option 16
\nobject
The throttle processor limits the number of events that pass through over a given time window.
group_by
\n[string]
Optional list of fields used to group events before the threshold has been reached.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
threshold [required]
\nint64
the number of events allowed in a given time window. Events sent after the threshold has been reached, are dropped.
type [required]
\nenum
The processor type. The value should always be throttle. \nAllowed enum values: throttle
default: throttle
window [required]
\ndouble
The time window in seconds over which the threshold applies.
sources [required]
\n[ <oneOf>]
A list of configured data sources for the pipeline.
Option 1
\nobject
The kafka source ingests data from Apache Kafka topics.
group_id [required]
\nstring
Consumer group ID used by the Kafka client.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
librdkafka_options
\n[object]
Optional list of advanced Kafka client configuration options, defined as key-value pairs.
name [required]
\nstring
The name of the librdkafka configuration option to set.
value [required]
\nstring
The value assigned to the specified librdkafka configuration option.
sasl
\nobject
Specifies the SASL mechanism for authenticating with a Kafka cluster.
mechanism
\nenum
SASL mechanism used for Kafka authentication. \nAllowed enum values: PLAIN,SCRAM-SHA-256,SCRAM-SHA-512
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
topics [required]
\n[string]
A list of Kafka topic names to subscribe to. The source ingests messages from each topic specified.
type [required]
\nenum
The source type. The value should always be kafka. \nAllowed enum values: kafka
default: kafka
Option 2
\nobject
The datadog_agent source collects logs from the Datadog Agent.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be datadog_agent. \nAllowed enum values: datadog_agent
default: datadog_agent
Option 3
\nobject
The splunk_tcp source receives logs from a Splunk Universal Forwarder over TCP.\nTLS is supported for secure transmission.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always splunk_tcp. \nAllowed enum values: splunk_tcp
default: splunk_tcp
Option 4
\nobject
The splunk_hec source implements the Splunk HTTP Event Collector (HEC) API.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always splunk_hec. \nAllowed enum values: splunk_hec
default: splunk_hec
Option 5
\nobject
The amazon_s3 source ingests logs from an Amazon S3 bucket.\nIt supports AWS authentication and TLS encryption.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
region [required]
\nstring
AWS region where the S3 bucket resides.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always amazon_s3. \nAllowed enum values: amazon_s3
default: amazon_s3
Option 6
\nobject
The fluentd source ingests logs from a Fluentd-compatible service.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be `fluentd. \nAllowed enum values: fluentd
default: fluentd
Option 7
\nobject
The fluent_bit source ingests logs from Fluent Bit.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be fluent_bit. \nAllowed enum values: fluent_bit
default: fluent_bit
Option 8
\nobject
The http_server source collects logs over HTTP POST from external services.
auth_strategy [required]
\nenum
HTTP authentication method. \nAllowed enum values: none,plain
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
Unique ID for the HTTP server source.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be http_server. \nAllowed enum values: http_server
default: http_server
Option 9
\nobject
The sumo_logic source receives logs from Sumo Logic collectors.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
type [required]
\nenum
The source type. The value should always be sumo_logic. \nAllowed enum values: sumo_logic
default: sumo_logic
Option 10
\nobject
The rsyslog source listens for logs over TCP or UDP from an rsyslog server using the syslog protocol.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
mode [required]
\nenum
Protocol used by the syslog source to receive messages. \nAllowed enum values: tcp,udp
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be rsyslog. \nAllowed enum values: rsyslog
default: rsyslog
Option 11
\nobject
The syslog_ng source listens for logs over TCP or UDP from a syslog-ng server using the syslog protocol.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
mode [required]
\nenum
Protocol used by the syslog source to receive messages. \nAllowed enum values: tcp,udp
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be syslog_ng. \nAllowed enum values: syslog_ng
default: syslog_ng
Option 12
\nobject
The amazon_data_firehose source ingests logs from AWS Data Firehose.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be amazon_data_firehose. \nAllowed enum values: amazon_data_firehose
default: amazon_data_firehose
Option 13
\nobject
The google_pubsub source ingests logs from a Google Cloud Pub/Sub subscription.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
project [required]
\nstring
The GCP project ID that owns the Pub/Sub subscription.
subscription [required]
\nstring
The Pub/Sub subscription name from which messages are consumed.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be google_pubsub. \nAllowed enum values: google_pubsub
default: google_pubsub
Option 14
\nobject
The http_client source scrapes logs from HTTP endpoints at regular intervals.
auth_strategy
\nenum
Optional authentication strategy for HTTP requests. \nAllowed enum values: basic,bearer
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
scrape_interval_secs
\nint64
The interval (in seconds) between HTTP scrape requests.
scrape_timeout_secs
\nint64
The timeout (in seconds) for each scrape request.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be http_client. \nAllowed enum values: http_client
default: http_client
Option 15
\nobject
The logstash source ingests logs from a Logstash forwarder.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be logstash. \nAllowed enum values: logstash
default: logstash
name [required]
\nstring
Name of the pipeline.
type [required]
\nstring
The resource type identifier. For pipeline resources, this should always be set to pipelines.
default: pipelines
data [required]
\nobject
Contains the the pipeline configuration.
attributes [required]
\nobject
Defines the pipeline’s name and its components (sources, processors, and destinations).
config [required]
\nobject
Specifies the pipeline's configuration, including its sources, processors, and destinations.
destinations [required]
\n[ <oneOf>]
A list of destination components where processed logs are sent.
Option 1
\nobject
The datadog_logs destination forwards logs to Datadog Log Management.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be datadog_logs. \nAllowed enum values: datadog_logs
default: datadog_logs
Option 2
\nobject
The amazon_s3 destination sends your logs in Datadog-rehydratable format to an Amazon S3 bucket for archiving.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
bucket [required]
\nstring
S3 bucket name.
id [required]
\nstring
Unique identifier for the destination component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
key_prefix
\nstring
Optional prefix for object keys.
region [required]
\nstring
AWS region of the S3 bucket.
storage_class [required]
\nenum
S3 storage class. \nAllowed enum values: STANDARD,REDUCED_REDUNDANCY,INTELLIGENT_TIERING,STANDARD_IA,EXPRESS_ONEZONE,ONEZONE_IA,GLACIER,GLACIER_IR,DEEP_ARCHIVE
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. Always amazon_s3. \nAllowed enum values: amazon_s3
default: amazon_s3
Option 3
\nobject
The google_cloud_storage destination stores logs in a Google Cloud Storage (GCS) bucket.\nIt requires a bucket name, GCP authentication, and metadata fields.
acl [required]
\nenum
Access control list setting for objects written to the bucket. \nAllowed enum values: private,project-private,public-read,authenticated-read,bucket-owner-read,bucket-owner-full-control
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
bucket [required]
\nstring
Name of the GCS bucket.
id [required]
\nstring
Unique identifier for the destination component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
key_prefix
\nstring
Optional prefix for object keys within the GCS bucket.
metadata
\n[object]
Custom metadata to attach to each object uploaded to the GCS bucket.
name [required]
\nstring
The metadata key.
value [required]
\nstring
The metadata value.
storage_class [required]
\nenum
Storage class used for objects stored in GCS. \nAllowed enum values: STANDARD,NEARLINE,COLDLINE,ARCHIVE
type [required]
\nenum
The destination type. Always google_cloud_storage. \nAllowed enum values: google_cloud_storage
default: google_cloud_storage
Option 4
\nobject
The splunk_hec destination forwards logs to Splunk using the HTTP Event Collector (HEC).
auto_extract_timestamp
\nboolean
If true, Splunk tries to extract timestamps from incoming log events.\nIf false, Splunk assigns the time the event was received.
encoding
\nenum
Encoding format for log events. \nAllowed enum values: json,raw_message
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
index
\nstring
Optional name of the Splunk index where logs are written.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
sourcetype
\nstring
The Splunk sourcetype to assign to log events.
type [required]
\nenum
The destination type. Always splunk_hec. \nAllowed enum values: splunk_hec
default: splunk_hec
Option 5
\nobject
The sumo_logic destination forwards logs to Sumo Logic.
encoding
\nenum
The output encoding format. \nAllowed enum values: json,raw_message,logfmt
header_custom_fields
\n[object]
A list of custom headers to include in the request to Sumo Logic.
name [required]
\nstring
The header field name.
value [required]
\nstring
The header field value.
header_host_name
\nstring
Optional override for the host name header.
header_source_category
\nstring
Optional override for the source category header.
header_source_name
\nstring
Optional override for the source name header.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be sumo_logic. \nAllowed enum values: sumo_logic
default: sumo_logic
Option 6
\nobject
The elasticsearch destination writes logs to an Elasticsearch cluster.
api_version
\nenum
The Elasticsearch API version to use. Set to auto to auto-detect. \nAllowed enum values: auto,v6,v7,v8
bulk_index
\nstring
The index to write logs to in Elasticsearch.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be elasticsearch. \nAllowed enum values: elasticsearch
default: elasticsearch
Option 7
\nobject
The rsyslog destination forwards logs to an external rsyslog server over TCP or UDP using the syslog protocol.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
keepalive
\nint64
Optional socket keepalive duration in milliseconds.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. The value should always be rsyslog. \nAllowed enum values: rsyslog
default: rsyslog
Option 8
\nobject
The syslog_ng destination forwards logs to an external syslog-ng server over TCP or UDP using the syslog protocol.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
keepalive
\nint64
Optional socket keepalive duration in milliseconds.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. The value should always be syslog_ng. \nAllowed enum values: syslog_ng
default: syslog_ng
Option 9
\nobject
The azure_storage destination forwards logs to an Azure Blob Storage container.
blob_prefix
\nstring
Optional prefix for blobs written to the container.
container_name [required]
\nstring
The name of the Azure Blob Storage container to store logs in.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be azure_storage. \nAllowed enum values: azure_storage
default: azure_storage
Option 10
\nobject
The microsoft_sentinel destination forwards logs to Microsoft Sentinel.
client_id [required]
\nstring
Azure AD client ID used for authentication.
dcr_immutable_id [required]
\nstring
The immutable ID of the Data Collection Rule (DCR).
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
table [required]
\nstring
The name of the Log Analytics table where logs are sent.
tenant_id [required]
\nstring
Azure AD tenant ID.
type [required]
\nenum
The destination type. The value should always be microsoft_sentinel. \nAllowed enum values: microsoft_sentinel
default: microsoft_sentinel
Option 11
\nobject
The google_chronicle destination sends logs to Google Chronicle.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
customer_id [required]
\nstring
The Google Chronicle customer ID.
encoding
\nenum
The encoding format for the logs sent to Chronicle. \nAllowed enum values: json,raw_message
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
log_type
\nstring
The log type metadata associated with the Chronicle destination.
type [required]
\nenum
The destination type. The value should always be google_chronicle. \nAllowed enum values: google_chronicle
default: google_chronicle
Option 12
\nobject
The new_relic destination sends logs to the New Relic platform.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The New Relic region. \nAllowed enum values: us,eu
type [required]
\nenum
The destination type. The value should always be new_relic. \nAllowed enum values: new_relic
default: new_relic
Option 13
\nobject
The sentinel_one destination sends logs to SentinelOne.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The SentinelOne region to send logs to. \nAllowed enum values: us,eu,ca,data_set_us
type [required]
\nenum
The destination type. The value should always be sentinel_one. \nAllowed enum values: sentinel_one
default: sentinel_one
Option 14
\nobject
The opensearch destination writes logs to an OpenSearch cluster.
bulk_index
\nstring
The index to write logs to.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be opensearch. \nAllowed enum values: opensearch
default: opensearch
Option 15
\nobject
The amazon_opensearch destination writes logs to Amazon OpenSearch.
auth [required]
\nobject
Authentication settings for the Amazon OpenSearch destination.\nThe strategy field determines whether basic or AWS-based authentication is used.
assume_role
\nstring
The ARN of the role to assume (used with aws strategy).
aws_region
\nstring
AWS region
external_id
\nstring
External ID for the assumed role (used with aws strategy).
session_name
\nstring
Session name for the assumed role (used with aws strategy).
strategy [required]
\nenum
The authentication strategy to use. \nAllowed enum values: basic,aws
bulk_index
\nstring
The index to write logs to.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be amazon_opensearch. \nAllowed enum values: amazon_opensearch
default: amazon_opensearch
processors
\n[ <oneOf>]
A list of processors that transform or enrich log data.
Option 1
\nobject
The filter processor allows conditional processing of logs based on a Datadog search query. Logs that match the include query are passed through; others are discarded.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs should pass through the filter. Logs that match this query continue to downstream components; others are dropped.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be filter. \nAllowed enum values: filter
default: filter
Option 2
\nobject
The parse_json processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string.
field [required]
\nstring
The name of the log field that contains a JSON string.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be parse_json. \nAllowed enum values: parse_json
default: parse_json
Option 3
\nobject
The Quota Processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert.
drop_events [required]
\nboolean
If set to true, logs that matched the quota filter and sent after the quota has been met are dropped; only logs that did not match the filter query continue through the pipeline.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
ignore_when_missing_partitions
\nboolean
If true, the processor skips quota checks when partition fields are missing from the logs.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
name [required]
\nstring
Name of the quota.
overflow_action
\nenum
The action to take when the quota is exceeded. Options:
\ndrop: Drop the event.
no_action: Let the event pass through.
overflow_routing: Route to an overflow destination.
Allowed enum values: drop,no_action,overflow_routing
overrides
\n[object]
A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit.
fields [required]
\n[object]
A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
partition_fields
\n[string]
A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values.
type [required]
\nenum
The processor type. The value should always be quota. \nAllowed enum values: quota
default: quota
Option 4
\nobject
The add_fields processor adds static key-value fields to logs.
fields [required]
\n[object]
A list of static fields (key-value pairs) that is added to each log event processed by this component.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be add_fields. \nAllowed enum values: add_fields
default: add_fields
Option 5
\nobject
The remove_fields processor deletes specified fields from logs.
fields [required]
\n[string]
A list of field names to be removed from each log event.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
The PipelineRemoveFieldsProcessor inputs.
type [required]
\nenum
The processor type. The value should always be remove_fields. \nAllowed enum values: remove_fields
default: remove_fields
Option 6
\nobject
The rename_fields processor changes field names.
fields [required]
\n[object]
A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields.
destination [required]
\nstring
The field name to assign the renamed value to.
preserve_source [required]
\nboolean
Indicates whether the original field, that is received from the source, should be kept (true) or removed (false) after renaming.
source [required]
\nstring
The original field name in the log event that should be renamed.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be rename_fields. \nAllowed enum values: rename_fields
default: rename_fields
Option 7
\nobject
The generate_datadog_metrics processor creates custom metrics from logs and sends them to Datadog.\nMetrics can be counters, gauges, or distributions and optionally grouped by log fields.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
metrics [required]
\n[object]
Configuration for generating individual metrics.
group_by
\n[string]
Optional fields used to group the metric series.
include [required]
\nstring
Datadog filter query to match logs for metric generation.
metric_type [required]
\nenum
Type of metric to create. \nAllowed enum values: count,gauge,distribution
name [required]
\nstring
Name of the custom metric to be created.
value [required]
\n<oneOf>
Specifies how the value of the generated metric is computed.
Option 1
\nobject
Strategy that increments a generated metric by one for each matching event.
strategy [required]
\nenum
Increments the metric by 1 for each matching event. \nAllowed enum values: increment_by_one
Option 2
\nobject
Strategy that increments a generated metric based on the value of a log field.
field [required]
\nstring
Name of the log field containing the numeric value to increment the metric by.
strategy [required]
\nenum
Uses a numeric field in the log event as the metric increment. \nAllowed enum values: increment_by_field
type [required]
\nenum
The processor type. Always generate_datadog_metrics. \nAllowed enum values: generate_datadog_metrics
default: generate_datadog_metrics
Option 8
\nobject
The sample processor allows probabilistic sampling of logs at a fixed rate.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
percentage
\ndouble
The percentage of logs to sample.
rate
\nint64
Number of events to sample (1 in N).
type [required]
\nenum
The processor type. The value should always be sample. \nAllowed enum values: sample
default: sample
Option 9
\nobject
The parse_grok processor extracts structured fields from unstructured log messages using Grok patterns.
disable_library_rules
\nboolean
If set to true, disables the default Grok rules provided by Datadog.
id [required]
\nstring
A unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
rules [required]
\n[object]
The list of Grok parsing rules. If multiple matching rules are provided, they are evaluated in order. The first successful match is applied.
match_rules [required]
\n[object]
A list of Grok parsing rules that define how to extract fields from the source field.\nEach rule must contain a name and a valid Grok pattern.
name [required]
\nstring
The name of the rule.
rule [required]
\nstring
The definition of the Grok rule.
source [required]
\nstring
The name of the field in the log event to apply the Grok rules to.
support_rules [required]
\n[object]
A list of Grok helper rules that can be referenced by the parsing rules.
name [required]
\nstring
The name of the Grok helper rule.
rule [required]
\nstring
The definition of the Grok helper rule.
type [required]
\nenum
The processor type. The value should always be parse_grok. \nAllowed enum values: parse_grok
default: parse_grok
Option 10
\nobject
The sensitive_data_scanner processor detects and optionally redacts sensitive data in log events.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
rules [required]
\n[object]
A list of rules for identifying and acting on sensitive data patterns.
keyword_options
\nobject
Configuration for keywords used to reinforce sensitive data pattern detection.
keywords [required]
\n[string]
A list of keywords to match near the sensitive pattern.
proximity [required]
\nint64
Maximum number of tokens between a keyword and a sensitive value match.
name [required]
\nstring
A name identifying the rule.
on_match [required]
\n<oneOf>
Defines what action to take when sensitive data is matched.
Option 1
\nobject
Configuration for completely redacting matched sensitive data.
action [required]
\nenum
Action type that completely replaces the matched sensitive data with a fixed replacement string to remove all visibility. \nAllowed enum values: redact
options [required]
\nobject
Configuration for fully redacting sensitive data.
replace [required]
\nstring
The ObservabilityPipelineSensitiveDataScannerProcessorActionRedactOptions replace.
Option 2
\nobject
Configuration for hashing matched sensitive values.
action [required]
\nenum
Action type that replaces the matched sensitive data with a hashed representation, preserving structure while securing content. \nAllowed enum values: hash
options
\nobject
The ObservabilityPipelineSensitiveDataScannerProcessorActionHash options.
Option 3
\nobject
Configuration for partially redacting matched sensitive data.
action [required]
\nenum
Action type that redacts part of the sensitive data while preserving a configurable number of characters, typically used for masking purposes (e.g., show last 4 digits of a credit card). \nAllowed enum values: partial_redact
options [required]
\nobject
Controls how partial redaction is applied, including character count and direction.
characters [required]
\nint64
The ObservabilityPipelineSensitiveDataScannerProcessorActionPartialRedactOptions characters.
direction [required]
\nenum
Indicates whether to redact characters from the first or last part of the matched value. \nAllowed enum values: first,last
pattern [required]
\n<oneOf>
Pattern detection configuration for identifying sensitive data using either a custom regex or a library reference.
Option 1
\nobject
Defines a custom regex-based pattern for identifying sensitive data in logs.
options [required]
\nobject
Options for defining a custom regex pattern.
rule [required]
\nstring
A regular expression used to detect sensitive values. Must be a valid regex.
type [required]
\nenum
Indicates a custom regular expression is used for matching. \nAllowed enum values: custom
Option 2
\nobject
Specifies a pattern from Datadog’s sensitive data detection library to match known sensitive data types.
options [required]
\nobject
Options for selecting a predefined library pattern and enabling keyword support.
id [required]
\nstring
Identifier for a predefined pattern from the sensitive data scanner pattern library.
use_recommended_keywords
\nboolean
Whether to augment the pattern with recommended keywords (optional).
type [required]
\nenum
Indicates that a predefined library pattern is used. \nAllowed enum values: library
scope [required]
\n<oneOf>
Determines which parts of the log the pattern-matching rule should be applied to.
Option 1
\nobject
Includes only specific fields for sensitive data scanning.
options [required]
\nobject
Fields to which the scope rule applies.
fields [required]
\n[string]
The ObservabilityPipelineSensitiveDataScannerProcessorScopeOptions fields.
target [required]
\nenum
Applies the rule only to included fields. \nAllowed enum values: include
Option 2
\nobject
Excludes specific fields from sensitive data scanning.
options [required]
\nobject
Fields to which the scope rule applies.
fields [required]
\n[string]
The ObservabilityPipelineSensitiveDataScannerProcessorScopeOptions fields.
target [required]
\nenum
Excludes specific fields from processing. \nAllowed enum values: exclude
Option 3
\nobject
Applies scanning across all available fields.
target [required]
\nenum
Applies the rule to all fields. \nAllowed enum values: all
tags [required]
\n[string]
Tags assigned to this rule for filtering and classification.
type [required]
\nenum
The processor type. The value should always be sensitive_data_scanner. \nAllowed enum values: sensitive_data_scanner
default: sensitive_data_scanner
Option 11
\nobject
The ocsf_mapper processor transforms logs into the OCSF schema using a predefined mapping configuration.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mappings [required]
\n[object]
A list of mapping rules to convert events to the OCSF format.
include [required]
\nstring
A Datadog search query used to select the logs that this mapping should apply to.
mapping [required]
\n<oneOf>
Defines a single mapping rule for transforming logs into the OCSF schema.
Option 1
\nenum
Predefined library mappings for common log formats. \nAllowed enum values: CloudTrail Account Change,GCP Cloud Audit CreateBucket,GCP Cloud Audit CreateSink,GCP Cloud Audit SetIamPolicy,GCP Cloud Audit UpdateSink,Github Audit Log API Activity,Google Workspace Admin Audit addPrivilege,Microsoft 365 Defender Incident,Microsoft 365 Defender UserLoggedIn,Okta System Log Authentication,Palo Alto Networks Firewall Traffic
type [required]
\nenum
The processor type. The value should always be ocsf_mapper. \nAllowed enum values: ocsf_mapper
default: ocsf_mapper
Option 12
\nobject
The add_env_vars processor adds environment variable values to log events.
id [required]
\nstring
The unique identifier for this component. Used to reference this processor in the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
type [required]
\nenum
The processor type. The value should always be add_env_vars. \nAllowed enum values: add_env_vars
default: add_env_vars
variables [required]
\n[object]
A list of environment variable mappings to apply to log fields.
field [required]
\nstring
The target field in the log event.
name [required]
\nstring
The name of the environment variable to read.
Option 13
\nobject
The dedupe processor removes duplicate fields in log events.
fields [required]
\n[string]
A list of log field paths to check for duplicates.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mode [required]
\nenum
The deduplication mode to apply to the fields. \nAllowed enum values: match,ignore
type [required]
\nenum
The processor type. The value should always be dedupe. \nAllowed enum values: dedupe
default: dedupe
Option 14
\nobject
The enrichment_table processor enriches logs using a static CSV file or GeoIP database.
file
\nobject
Defines a static enrichment table loaded from a CSV file.
encoding [required]
\nobject
File encoding format.
delimiter [required]
\nstring
The encoding delimiter.
includes_headers [required]
\nboolean
The encoding includes_headers.
type [required]
\nenum
Specifies the encoding format (e.g., CSV) used for enrichment tables. \nAllowed enum values: csv
key [required]
\n[object]
Key fields used to look up enrichment values.
column [required]
\nstring
The items column.
comparison [required]
\nenum
Defines how to compare key fields for enrichment table lookups. \nAllowed enum values: equals
field [required]
\nstring
The items field.
path [required]
\nstring
Path to the CSV file.
schema [required]
\n[object]
Schema defining column names and their types.
column [required]
\nstring
The items column.
type [required]
\nenum
Declares allowed data types for enrichment table columns. \nAllowed enum values: string,boolean,integer,float,date,timestamp
geoip
\nobject
Uses a GeoIP database to enrich logs based on an IP field.
key_field [required]
\nstring
Path to the IP field in the log.
locale [required]
\nstring
Locale used to resolve geographical names.
path [required]
\nstring
Path to the GeoIP database file.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
target [required]
\nstring
Path where enrichment results should be stored in the log.
type [required]
\nenum
The processor type. The value should always be enrichment_table. \nAllowed enum values: enrichment_table
default: enrichment_table
Option 15
\nobject
The reduce processor aggregates and merges logs based on matching keys and merge strategies.
group_by [required]
\n[string]
A list of fields used to group log events for merging.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
merge_strategies [required]
\n[object]
List of merge strategies defining how values from grouped events should be combined.
path [required]
\nstring
The field path in the log event.
strategy [required]
\nenum
The merge strategy to apply. \nAllowed enum values: discard,retain,sum,max,min,array,concat,concat_newline,concat_raw,shortest_array,longest_array,flat_unique
type [required]
\nenum
The processor type. The value should always be reduce. \nAllowed enum values: reduce
default: reduce
Option 16
\nobject
The throttle processor limits the number of events that pass through over a given time window.
group_by
\n[string]
Optional list of fields used to group events before the threshold has been reached.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
threshold [required]
\nint64
the number of events allowed in a given time window. Events sent after the threshold has been reached, are dropped.
type [required]
\nenum
The processor type. The value should always be throttle. \nAllowed enum values: throttle
default: throttle
window [required]
\ndouble
The time window in seconds over which the threshold applies.
Option 17
\nobject
The datadog_tags processor includes or excludes specific Datadog tags in your logs.
action [required]
\nenum
The action to take on tags with matching keys. \nAllowed enum values: include,exclude
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
keys [required]
\n[string]
A list of tag keys.
mode [required]
\nenum
The processing mode. \nAllowed enum values: filter
type [required]
\nenum
The processor type. The value should always be datadog_tags. \nAllowed enum values: datadog_tags
default: datadog_tags
sources [required]
\n[ <oneOf>]
A list of configured data sources for the pipeline.
Option 1
\nobject
The kafka source ingests data from Apache Kafka topics.
group_id [required]
\nstring
Consumer group ID used by the Kafka client.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
librdkafka_options
\n[object]
Optional list of advanced Kafka client configuration options, defined as key-value pairs.
name [required]
\nstring
The name of the librdkafka configuration option to set.
value [required]
\nstring
The value assigned to the specified librdkafka configuration option.
sasl
\nobject
Specifies the SASL mechanism for authenticating with a Kafka cluster.
mechanism
\nenum
SASL mechanism used for Kafka authentication. \nAllowed enum values: PLAIN,SCRAM-SHA-256,SCRAM-SHA-512
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
topics [required]
\n[string]
A list of Kafka topic names to subscribe to. The source ingests messages from each topic specified.
type [required]
\nenum
The source type. The value should always be kafka. \nAllowed enum values: kafka
default: kafka
Option 2
\nobject
The datadog_agent source collects logs from the Datadog Agent.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be datadog_agent. \nAllowed enum values: datadog_agent
default: datadog_agent
Option 3
\nobject
The splunk_tcp source receives logs from a Splunk Universal Forwarder over TCP.\nTLS is supported for secure transmission.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always splunk_tcp. \nAllowed enum values: splunk_tcp
default: splunk_tcp
Option 4
\nobject
The splunk_hec source implements the Splunk HTTP Event Collector (HEC) API.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always splunk_hec. \nAllowed enum values: splunk_hec
default: splunk_hec
Option 5
\nobject
The amazon_s3 source ingests logs from an Amazon S3 bucket.\nIt supports AWS authentication and TLS encryption.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
region [required]
\nstring
AWS region where the S3 bucket resides.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always amazon_s3. \nAllowed enum values: amazon_s3
default: amazon_s3
Option 6
\nobject
The fluentd source ingests logs from a Fluentd-compatible service.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be `fluentd. \nAllowed enum values: fluentd
default: fluentd
Option 7
\nobject
The fluent_bit source ingests logs from Fluent Bit.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be fluent_bit. \nAllowed enum values: fluent_bit
default: fluent_bit
Option 8
\nobject
The http_server source collects logs over HTTP POST from external services.
auth_strategy [required]
\nenum
HTTP authentication method. \nAllowed enum values: none,plain
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
Unique ID for the HTTP server source.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be http_server. \nAllowed enum values: http_server
default: http_server
Option 9
\nobject
The sumo_logic source receives logs from Sumo Logic collectors.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
type [required]
\nenum
The source type. The value should always be sumo_logic. \nAllowed enum values: sumo_logic
default: sumo_logic
Option 10
\nobject
The rsyslog source listens for logs over TCP or UDP from an rsyslog server using the syslog protocol.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
mode [required]
\nenum
Protocol used by the syslog source to receive messages. \nAllowed enum values: tcp,udp
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be rsyslog. \nAllowed enum values: rsyslog
default: rsyslog
Option 11
\nobject
The syslog_ng source listens for logs over TCP or UDP from a syslog-ng server using the syslog protocol.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
mode [required]
\nenum
Protocol used by the syslog source to receive messages. \nAllowed enum values: tcp,udp
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be syslog_ng. \nAllowed enum values: syslog_ng
default: syslog_ng
Option 12
\nobject
The amazon_data_firehose source ingests logs from AWS Data Firehose.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be amazon_data_firehose. \nAllowed enum values: amazon_data_firehose
default: amazon_data_firehose
Option 13
\nobject
The google_pubsub source ingests logs from a Google Cloud Pub/Sub subscription.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
project [required]
\nstring
The GCP project ID that owns the Pub/Sub subscription.
subscription [required]
\nstring
The Pub/Sub subscription name from which messages are consumed.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be google_pubsub. \nAllowed enum values: google_pubsub
default: google_pubsub
Option 14
\nobject
The http_client source scrapes logs from HTTP endpoints at regular intervals.
auth_strategy
\nenum
Optional authentication strategy for HTTP requests. \nAllowed enum values: basic,bearer
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
scrape_interval_secs
\nint64
The interval (in seconds) between HTTP scrape requests.
scrape_timeout_secs
\nint64
The timeout (in seconds) for each scrape request.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be http_client. \nAllowed enum values: http_client
default: http_client
Option 15
\nobject
The logstash source ingests logs from a Logstash forwarder.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be logstash. \nAllowed enum values: logstash
default: logstash
name [required]
\nstring
Name of the pipeline.
type [required]
\nstring
The resource type identifier. For pipeline resources, this should always be set to pipelines.
default: pipelines
data [required]
\nobject
Contains the pipeline’s ID, type, and configuration attributes.
attributes [required]
\nobject
Defines the pipeline’s name and its components (sources, processors, and destinations).
config [required]
\nobject
Specifies the pipeline's configuration, including its sources, processors, and destinations.
destinations [required]
\n[ <oneOf>]
A list of destination components where processed logs are sent.
Option 1
\nobject
The datadog_logs destination forwards logs to Datadog Log Management.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be datadog_logs. \nAllowed enum values: datadog_logs
default: datadog_logs
Option 2
\nobject
The amazon_s3 destination sends your logs in Datadog-rehydratable format to an Amazon S3 bucket for archiving.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
bucket [required]
\nstring
S3 bucket name.
id [required]
\nstring
Unique identifier for the destination component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
key_prefix
\nstring
Optional prefix for object keys.
region [required]
\nstring
AWS region of the S3 bucket.
storage_class [required]
\nenum
S3 storage class. \nAllowed enum values: STANDARD,REDUCED_REDUNDANCY,INTELLIGENT_TIERING,STANDARD_IA,EXPRESS_ONEZONE,ONEZONE_IA,GLACIER,GLACIER_IR,DEEP_ARCHIVE
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. Always amazon_s3. \nAllowed enum values: amazon_s3
default: amazon_s3
Option 3
\nobject
The google_cloud_storage destination stores logs in a Google Cloud Storage (GCS) bucket.\nIt requires a bucket name, GCP authentication, and metadata fields.
acl [required]
\nenum
Access control list setting for objects written to the bucket. \nAllowed enum values: private,project-private,public-read,authenticated-read,bucket-owner-read,bucket-owner-full-control
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
bucket [required]
\nstring
Name of the GCS bucket.
id [required]
\nstring
Unique identifier for the destination component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
key_prefix
\nstring
Optional prefix for object keys within the GCS bucket.
metadata
\n[object]
Custom metadata to attach to each object uploaded to the GCS bucket.
name [required]
\nstring
The metadata key.
value [required]
\nstring
The metadata value.
storage_class [required]
\nenum
Storage class used for objects stored in GCS. \nAllowed enum values: STANDARD,NEARLINE,COLDLINE,ARCHIVE
type [required]
\nenum
The destination type. Always google_cloud_storage. \nAllowed enum values: google_cloud_storage
default: google_cloud_storage
Option 4
\nobject
The splunk_hec destination forwards logs to Splunk using the HTTP Event Collector (HEC).
auto_extract_timestamp
\nboolean
If true, Splunk tries to extract timestamps from incoming log events.\nIf false, Splunk assigns the time the event was received.
encoding
\nenum
Encoding format for log events. \nAllowed enum values: json,raw_message
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
index
\nstring
Optional name of the Splunk index where logs are written.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
sourcetype
\nstring
The Splunk sourcetype to assign to log events.
type [required]
\nenum
The destination type. Always splunk_hec. \nAllowed enum values: splunk_hec
default: splunk_hec
Option 5
\nobject
The sumo_logic destination forwards logs to Sumo Logic.
encoding
\nenum
The output encoding format. \nAllowed enum values: json,raw_message,logfmt
header_custom_fields
\n[object]
A list of custom headers to include in the request to Sumo Logic.
name [required]
\nstring
The header field name.
value [required]
\nstring
The header field value.
header_host_name
\nstring
Optional override for the host name header.
header_source_category
\nstring
Optional override for the source category header.
header_source_name
\nstring
Optional override for the source name header.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be sumo_logic. \nAllowed enum values: sumo_logic
default: sumo_logic
Option 6
\nobject
The elasticsearch destination writes logs to an Elasticsearch cluster.
api_version
\nenum
The Elasticsearch API version to use. Set to auto to auto-detect. \nAllowed enum values: auto,v6,v7,v8
bulk_index
\nstring
The index to write logs to in Elasticsearch.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be elasticsearch. \nAllowed enum values: elasticsearch
default: elasticsearch
Option 7
\nobject
The rsyslog destination forwards logs to an external rsyslog server over TCP or UDP using the syslog protocol.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
keepalive
\nint64
Optional socket keepalive duration in milliseconds.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. The value should always be rsyslog. \nAllowed enum values: rsyslog
default: rsyslog
Option 8
\nobject
The syslog_ng destination forwards logs to an external syslog-ng server over TCP or UDP using the syslog protocol.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
keepalive
\nint64
Optional socket keepalive duration in milliseconds.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. The value should always be syslog_ng. \nAllowed enum values: syslog_ng
default: syslog_ng
Option 9
\nobject
The azure_storage destination forwards logs to an Azure Blob Storage container.
blob_prefix
\nstring
Optional prefix for blobs written to the container.
container_name [required]
\nstring
The name of the Azure Blob Storage container to store logs in.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be azure_storage. \nAllowed enum values: azure_storage
default: azure_storage
Option 10
\nobject
The microsoft_sentinel destination forwards logs to Microsoft Sentinel.
client_id [required]
\nstring
Azure AD client ID used for authentication.
dcr_immutable_id [required]
\nstring
The immutable ID of the Data Collection Rule (DCR).
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
table [required]
\nstring
The name of the Log Analytics table where logs are sent.
tenant_id [required]
\nstring
Azure AD tenant ID.
type [required]
\nenum
The destination type. The value should always be microsoft_sentinel. \nAllowed enum values: microsoft_sentinel
default: microsoft_sentinel
Option 11
\nobject
The google_chronicle destination sends logs to Google Chronicle.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
customer_id [required]
\nstring
The Google Chronicle customer ID.
encoding
\nenum
The encoding format for the logs sent to Chronicle. \nAllowed enum values: json,raw_message
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
log_type
\nstring
The log type metadata associated with the Chronicle destination.
type [required]
\nenum
The destination type. The value should always be google_chronicle. \nAllowed enum values: google_chronicle
default: google_chronicle
Option 12
\nobject
The new_relic destination sends logs to the New Relic platform.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The New Relic region. \nAllowed enum values: us,eu
type [required]
\nenum
The destination type. The value should always be new_relic. \nAllowed enum values: new_relic
default: new_relic
Option 13
\nobject
The sentinel_one destination sends logs to SentinelOne.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The SentinelOne region to send logs to. \nAllowed enum values: us,eu,ca,data_set_us
type [required]
\nenum
The destination type. The value should always be sentinel_one. \nAllowed enum values: sentinel_one
default: sentinel_one
Option 14
\nobject
The opensearch destination writes logs to an OpenSearch cluster.
bulk_index
\nstring
The index to write logs to.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be opensearch. \nAllowed enum values: opensearch
default: opensearch
Option 15
\nobject
The amazon_opensearch destination writes logs to Amazon OpenSearch.
auth [required]
\nobject
Authentication settings for the Amazon OpenSearch destination.\nThe strategy field determines whether basic or AWS-based authentication is used.
assume_role
\nstring
The ARN of the role to assume (used with aws strategy).
aws_region
\nstring
AWS region
external_id
\nstring
External ID for the assumed role (used with aws strategy).
session_name
\nstring
Session name for the assumed role (used with aws strategy).
strategy [required]
\nenum
The authentication strategy to use. \nAllowed enum values: basic,aws
bulk_index
\nstring
The index to write logs to.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be amazon_opensearch. \nAllowed enum values: amazon_opensearch
default: amazon_opensearch
processors
\n[ <oneOf>]
A list of processors that transform or enrich log data.
Option 1
\nobject
The filter processor allows conditional processing of logs based on a Datadog search query. Logs that match the include query are passed through; others are discarded.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs should pass through the filter. Logs that match this query continue to downstream components; others are dropped.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be filter. \nAllowed enum values: filter
default: filter
Option 2
\nobject
The parse_json processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string.
field [required]
\nstring
The name of the log field that contains a JSON string.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be parse_json. \nAllowed enum values: parse_json
default: parse_json
Option 3
\nobject
The Quota Processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert.
drop_events [required]
\nboolean
If set to true, logs that matched the quota filter and sent after the quota has been met are dropped; only logs that did not match the filter query continue through the pipeline.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
ignore_when_missing_partitions
\nboolean
If true, the processor skips quota checks when partition fields are missing from the logs.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
name [required]
\nstring
Name of the quota.
overflow_action
\nenum
The action to take when the quota is exceeded. Options:
\ndrop: Drop the event.
no_action: Let the event pass through.
overflow_routing: Route to an overflow destination.
Allowed enum values: drop,no_action,overflow_routing
overrides
\n[object]
A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit.
fields [required]
\n[object]
A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
partition_fields
\n[string]
A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values.
type [required]
\nenum
The processor type. The value should always be quota. \nAllowed enum values: quota
default: quota
Option 4
\nobject
The add_fields processor adds static key-value fields to logs.
fields [required]
\n[object]
A list of static fields (key-value pairs) that is added to each log event processed by this component.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be add_fields. \nAllowed enum values: add_fields
default: add_fields
Option 5
\nobject
The remove_fields processor deletes specified fields from logs.
fields [required]
\n[string]
A list of field names to be removed from each log event.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
The PipelineRemoveFieldsProcessor inputs.
type [required]
\nenum
The processor type. The value should always be remove_fields. \nAllowed enum values: remove_fields
default: remove_fields
Option 6
\nobject
The rename_fields processor changes field names.
fields [required]
\n[object]
A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields.
destination [required]
\nstring
The field name to assign the renamed value to.
preserve_source [required]
\nboolean
Indicates whether the original field, that is received from the source, should be kept (true) or removed (false) after renaming.
source [required]
\nstring
The original field name in the log event that should be renamed.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be rename_fields. \nAllowed enum values: rename_fields
default: rename_fields
Option 7
\nobject
The generate_datadog_metrics processor creates custom metrics from logs and sends them to Datadog.\nMetrics can be counters, gauges, or distributions and optionally grouped by log fields.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
metrics [required]
\n[object]
Configuration for generating individual metrics.
group_by
\n[string]
Optional fields used to group the metric series.
include [required]
\nstring
Datadog filter query to match logs for metric generation.
metric_type [required]
\nenum
Type of metric to create. \nAllowed enum values: count,gauge,distribution
name [required]
\nstring
Name of the custom metric to be created.
value [required]
\n<oneOf>
Specifies how the value of the generated metric is computed.
Option 1
\nobject
Strategy that increments a generated metric by one for each matching event.
strategy [required]
\nenum
Increments the metric by 1 for each matching event. \nAllowed enum values: increment_by_one
Option 2
\nobject
Strategy that increments a generated metric based on the value of a log field.
field [required]
\nstring
Name of the log field containing the numeric value to increment the metric by.
strategy [required]
\nenum
Uses a numeric field in the log event as the metric increment. \nAllowed enum values: increment_by_field
type [required]
\nenum
The processor type. Always generate_datadog_metrics. \nAllowed enum values: generate_datadog_metrics
default: generate_datadog_metrics
Option 8
\nobject
The sample processor allows probabilistic sampling of logs at a fixed rate.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
percentage
\ndouble
The percentage of logs to sample.
rate
\nint64
Number of events to sample (1 in N).
type [required]
\nenum
The processor type. The value should always be sample. \nAllowed enum values: sample
default: sample
Option 9
\nobject
The parse_grok processor extracts structured fields from unstructured log messages using Grok patterns.
disable_library_rules
\nboolean
If set to true, disables the default Grok rules provided by Datadog.
id [required]
\nstring
A unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
rules [required]
\n[object]
The list of Grok parsing rules. If multiple matching rules are provided, they are evaluated in order. The first successful match is applied.
match_rules [required]
\n[object]
A list of Grok parsing rules that define how to extract fields from the source field.\nEach rule must contain a name and a valid Grok pattern.
name [required]
\nstring
The name of the rule.
rule [required]
\nstring
The definition of the Grok rule.
source [required]
\nstring
The name of the field in the log event to apply the Grok rules to.
support_rules [required]
\n[object]
A list of Grok helper rules that can be referenced by the parsing rules.
name [required]
\nstring
The name of the Grok helper rule.
rule [required]
\nstring
The definition of the Grok helper rule.
type [required]
\nenum
The processor type. The value should always be parse_grok. \nAllowed enum values: parse_grok
default: parse_grok
Option 10
\nobject
The sensitive_data_scanner processor detects and optionally redacts sensitive data in log events.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
rules [required]
\n[object]
A list of rules for identifying and acting on sensitive data patterns.
keyword_options
\nobject
Configuration for keywords used to reinforce sensitive data pattern detection.
keywords [required]
\n[string]
A list of keywords to match near the sensitive pattern.
proximity [required]
\nint64
Maximum number of tokens between a keyword and a sensitive value match.
name [required]
\nstring
A name identifying the rule.
on_match [required]
\n<oneOf>
Defines what action to take when sensitive data is matched.
Option 1
\nobject
Configuration for completely redacting matched sensitive data.
action [required]
\nenum
Action type that completely replaces the matched sensitive data with a fixed replacement string to remove all visibility. \nAllowed enum values: redact
options [required]
\nobject
Configuration for fully redacting sensitive data.
replace [required]
\nstring
The ObservabilityPipelineSensitiveDataScannerProcessorActionRedactOptions replace.
Option 2
\nobject
Configuration for hashing matched sensitive values.
action [required]
\nenum
Action type that replaces the matched sensitive data with a hashed representation, preserving structure while securing content. \nAllowed enum values: hash
options
\nobject
The ObservabilityPipelineSensitiveDataScannerProcessorActionHash options.
Option 3
\nobject
Configuration for partially redacting matched sensitive data.
action [required]
\nenum
Action type that redacts part of the sensitive data while preserving a configurable number of characters, typically used for masking purposes (e.g., show last 4 digits of a credit card). \nAllowed enum values: partial_redact
options [required]
\nobject
Controls how partial redaction is applied, including character count and direction.
characters [required]
\nint64
The ObservabilityPipelineSensitiveDataScannerProcessorActionPartialRedactOptions characters.
direction [required]
\nenum
Indicates whether to redact characters from the first or last part of the matched value. \nAllowed enum values: first,last
pattern [required]
\n<oneOf>
Pattern detection configuration for identifying sensitive data using either a custom regex or a library reference.
Option 1
\nobject
Defines a custom regex-based pattern for identifying sensitive data in logs.
options [required]
\nobject
Options for defining a custom regex pattern.
rule [required]
\nstring
A regular expression used to detect sensitive values. Must be a valid regex.
type [required]
\nenum
Indicates a custom regular expression is used for matching. \nAllowed enum values: custom
Option 2
\nobject
Specifies a pattern from Datadog’s sensitive data detection library to match known sensitive data types.
options [required]
\nobject
Options for selecting a predefined library pattern and enabling keyword support.
id [required]
\nstring
Identifier for a predefined pattern from the sensitive data scanner pattern library.
use_recommended_keywords
\nboolean
Whether to augment the pattern with recommended keywords (optional).
type [required]
\nenum
Indicates that a predefined library pattern is used. \nAllowed enum values: library
scope [required]
\n<oneOf>
Determines which parts of the log the pattern-matching rule should be applied to.
Option 1
\nobject
Includes only specific fields for sensitive data scanning.
options [required]
\nobject
Fields to which the scope rule applies.
fields [required]
\n[string]
The ObservabilityPipelineSensitiveDataScannerProcessorScopeOptions fields.
target [required]
\nenum
Applies the rule only to included fields. \nAllowed enum values: include
Option 2
\nobject
Excludes specific fields from sensitive data scanning.
options [required]
\nobject
Fields to which the scope rule applies.
fields [required]
\n[string]
The ObservabilityPipelineSensitiveDataScannerProcessorScopeOptions fields.
target [required]
\nenum
Excludes specific fields from processing. \nAllowed enum values: exclude
Option 3
\nobject
Applies scanning across all available fields.
target [required]
\nenum
Applies the rule to all fields. \nAllowed enum values: all
tags [required]
\n[string]
Tags assigned to this rule for filtering and classification.
type [required]
\nenum
The processor type. The value should always be sensitive_data_scanner. \nAllowed enum values: sensitive_data_scanner
default: sensitive_data_scanner
Option 11
\nobject
The ocsf_mapper processor transforms logs into the OCSF schema using a predefined mapping configuration.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mappings [required]
\n[object]
A list of mapping rules to convert events to the OCSF format.
include [required]
\nstring
A Datadog search query used to select the logs that this mapping should apply to.
mapping [required]
\n<oneOf>
Defines a single mapping rule for transforming logs into the OCSF schema.
Option 1
\nenum
Predefined library mappings for common log formats. \nAllowed enum values: CloudTrail Account Change,GCP Cloud Audit CreateBucket,GCP Cloud Audit CreateSink,GCP Cloud Audit SetIamPolicy,GCP Cloud Audit UpdateSink,Github Audit Log API Activity,Google Workspace Admin Audit addPrivilege,Microsoft 365 Defender Incident,Microsoft 365 Defender UserLoggedIn,Okta System Log Authentication,Palo Alto Networks Firewall Traffic
type [required]
\nenum
The processor type. The value should always be ocsf_mapper. \nAllowed enum values: ocsf_mapper
default: ocsf_mapper
Option 12
\nobject
The add_env_vars processor adds environment variable values to log events.
id [required]
\nstring
The unique identifier for this component. Used to reference this processor in the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
type [required]
\nenum
The processor type. The value should always be add_env_vars. \nAllowed enum values: add_env_vars
default: add_env_vars
variables [required]
\n[object]
A list of environment variable mappings to apply to log fields.
field [required]
\nstring
The target field in the log event.
name [required]
\nstring
The name of the environment variable to read.
Option 13
\nobject
The dedupe processor removes duplicate fields in log events.
fields [required]
\n[string]
A list of log field paths to check for duplicates.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mode [required]
\nenum
The deduplication mode to apply to the fields. \nAllowed enum values: match,ignore
type [required]
\nenum
The processor type. The value should always be dedupe. \nAllowed enum values: dedupe
default: dedupe
Option 14
\nobject
The enrichment_table processor enriches logs using a static CSV file or GeoIP database.
file
\nobject
Defines a static enrichment table loaded from a CSV file.
encoding [required]
\nobject
File encoding format.
delimiter [required]
\nstring
The encoding delimiter.
includes_headers [required]
\nboolean
The encoding includes_headers.
type [required]
\nenum
Specifies the encoding format (e.g., CSV) used for enrichment tables. \nAllowed enum values: csv
key [required]
\n[object]
Key fields used to look up enrichment values.
column [required]
\nstring
The items column.
comparison [required]
\nenum
Defines how to compare key fields for enrichment table lookups. \nAllowed enum values: equals
field [required]
\nstring
The items field.
path [required]
\nstring
Path to the CSV file.
schema [required]
\n[object]
Schema defining column names and their types.
column [required]
\nstring
The items column.
type [required]
\nenum
Declares allowed data types for enrichment table columns. \nAllowed enum values: string,boolean,integer,float,date,timestamp
geoip
\nobject
Uses a GeoIP database to enrich logs based on an IP field.
key_field [required]
\nstring
Path to the IP field in the log.
locale [required]
\nstring
Locale used to resolve geographical names.
path [required]
\nstring
Path to the GeoIP database file.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
target [required]
\nstring
Path where enrichment results should be stored in the log.
type [required]
\nenum
The processor type. The value should always be enrichment_table. \nAllowed enum values: enrichment_table
default: enrichment_table
Option 15
\nobject
The reduce processor aggregates and merges logs based on matching keys and merge strategies.
group_by [required]
\n[string]
A list of fields used to group log events for merging.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
merge_strategies [required]
\n[object]
List of merge strategies defining how values from grouped events should be combined.
path [required]
\nstring
The field path in the log event.
strategy [required]
\nenum
The merge strategy to apply. \nAllowed enum values: discard,retain,sum,max,min,array,concat,concat_newline,concat_raw,shortest_array,longest_array,flat_unique
type [required]
\nenum
The processor type. The value should always be reduce. \nAllowed enum values: reduce
default: reduce
Option 16
\nobject
The throttle processor limits the number of events that pass through over a given time window.
group_by
\n[string]
Optional list of fields used to group events before the threshold has been reached.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
threshold [required]
\nint64
the number of events allowed in a given time window. Events sent after the threshold has been reached, are dropped.
type [required]
\nenum
The processor type. The value should always be throttle. \nAllowed enum values: throttle
default: throttle
window [required]
\ndouble
The time window in seconds over which the threshold applies.
sources [required]
\n[ <oneOf>]
A list of configured data sources for the pipeline.
Option 1
\nobject
The kafka source ingests data from Apache Kafka topics.
group_id [required]
\nstring
Consumer group ID used by the Kafka client.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
librdkafka_options
\n[object]
Optional list of advanced Kafka client configuration options, defined as key-value pairs.
name [required]
\nstring
The name of the librdkafka configuration option to set.
value [required]
\nstring
The value assigned to the specified librdkafka configuration option.
sasl
\nobject
Specifies the SASL mechanism for authenticating with a Kafka cluster.
mechanism
\nenum
SASL mechanism used for Kafka authentication. \nAllowed enum values: PLAIN,SCRAM-SHA-256,SCRAM-SHA-512
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
topics [required]
\n[string]
A list of Kafka topic names to subscribe to. The source ingests messages from each topic specified.
type [required]
\nenum
The source type. The value should always be kafka. \nAllowed enum values: kafka
default: kafka
Option 2
\nobject
The datadog_agent source collects logs from the Datadog Agent.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be datadog_agent. \nAllowed enum values: datadog_agent
default: datadog_agent
Option 3
\nobject
The splunk_tcp source receives logs from a Splunk Universal Forwarder over TCP.\nTLS is supported for secure transmission.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always splunk_tcp. \nAllowed enum values: splunk_tcp
default: splunk_tcp
Option 4
\nobject
The splunk_hec source implements the Splunk HTTP Event Collector (HEC) API.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always splunk_hec. \nAllowed enum values: splunk_hec
default: splunk_hec
Option 5
\nobject
The amazon_s3 source ingests logs from an Amazon S3 bucket.\nIt supports AWS authentication and TLS encryption.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
region [required]
\nstring
AWS region where the S3 bucket resides.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always amazon_s3. \nAllowed enum values: amazon_s3
default: amazon_s3
Option 6
\nobject
The fluentd source ingests logs from a Fluentd-compatible service.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be `fluentd. \nAllowed enum values: fluentd
default: fluentd
Option 7
\nobject
The fluent_bit source ingests logs from Fluent Bit.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be fluent_bit. \nAllowed enum values: fluent_bit
default: fluent_bit
Option 8
\nobject
The http_server source collects logs over HTTP POST from external services.
auth_strategy [required]
\nenum
HTTP authentication method. \nAllowed enum values: none,plain
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
Unique ID for the HTTP server source.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be http_server. \nAllowed enum values: http_server
default: http_server
Option 9
\nobject
The sumo_logic source receives logs from Sumo Logic collectors.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
type [required]
\nenum
The source type. The value should always be sumo_logic. \nAllowed enum values: sumo_logic
default: sumo_logic
Option 10
\nobject
The rsyslog source listens for logs over TCP or UDP from an rsyslog server using the syslog protocol.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
mode [required]
\nenum
Protocol used by the syslog source to receive messages. \nAllowed enum values: tcp,udp
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be rsyslog. \nAllowed enum values: rsyslog
default: rsyslog
Option 11
\nobject
The syslog_ng source listens for logs over TCP or UDP from a syslog-ng server using the syslog protocol.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
mode [required]
\nenum
Protocol used by the syslog source to receive messages. \nAllowed enum values: tcp,udp
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be syslog_ng. \nAllowed enum values: syslog_ng
default: syslog_ng
Option 12
\nobject
The amazon_data_firehose source ingests logs from AWS Data Firehose.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be amazon_data_firehose. \nAllowed enum values: amazon_data_firehose
default: amazon_data_firehose
Option 13
\nobject
The google_pubsub source ingests logs from a Google Cloud Pub/Sub subscription.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
project [required]
\nstring
The GCP project ID that owns the Pub/Sub subscription.
subscription [required]
\nstring
The Pub/Sub subscription name from which messages are consumed.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be google_pubsub. \nAllowed enum values: google_pubsub
default: google_pubsub
Option 14
\nobject
The http_client source scrapes logs from HTTP endpoints at regular intervals.
auth_strategy
\nenum
Optional authentication strategy for HTTP requests. \nAllowed enum values: basic,bearer
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
scrape_interval_secs
\nint64
The interval (in seconds) between HTTP scrape requests.
scrape_timeout_secs
\nint64
The timeout (in seconds) for each scrape request.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be http_client. \nAllowed enum values: http_client
default: http_client
Option 15
\nobject
The logstash source ingests logs from a Logstash forwarder.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be logstash. \nAllowed enum values: logstash
default: logstash
name [required]
\nstring
Name of the pipeline.
id [required]
\nstring
Unique identifier for the pipeline.
type [required]
\nstring
The resource type identifier. For pipeline resources, this should always be set to pipelines.
default: pipelines
data [required]
\nobject
Contains the pipeline’s ID, type, and configuration attributes.
attributes [required]
\nobject
Defines the pipeline’s name and its components (sources, processors, and destinations).
config [required]
\nobject
Specifies the pipeline's configuration, including its sources, processors, and destinations.
destinations [required]
\n[ <oneOf>]
A list of destination components where processed logs are sent.
Option 1
\nobject
The datadog_logs destination forwards logs to Datadog Log Management.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be datadog_logs. \nAllowed enum values: datadog_logs
default: datadog_logs
Option 2
\nobject
The amazon_s3 destination sends your logs in Datadog-rehydratable format to an Amazon S3 bucket for archiving.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
bucket [required]
\nstring
S3 bucket name.
id [required]
\nstring
Unique identifier for the destination component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
key_prefix
\nstring
Optional prefix for object keys.
region [required]
\nstring
AWS region of the S3 bucket.
storage_class [required]
\nenum
S3 storage class. \nAllowed enum values: STANDARD,REDUCED_REDUNDANCY,INTELLIGENT_TIERING,STANDARD_IA,EXPRESS_ONEZONE,ONEZONE_IA,GLACIER,GLACIER_IR,DEEP_ARCHIVE
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. Always amazon_s3. \nAllowed enum values: amazon_s3
default: amazon_s3
Option 3
\nobject
The google_cloud_storage destination stores logs in a Google Cloud Storage (GCS) bucket.\nIt requires a bucket name, GCP authentication, and metadata fields.
acl [required]
\nenum
Access control list setting for objects written to the bucket. \nAllowed enum values: private,project-private,public-read,authenticated-read,bucket-owner-read,bucket-owner-full-control
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
bucket [required]
\nstring
Name of the GCS bucket.
id [required]
\nstring
Unique identifier for the destination component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
key_prefix
\nstring
Optional prefix for object keys within the GCS bucket.
metadata
\n[object]
Custom metadata to attach to each object uploaded to the GCS bucket.
name [required]
\nstring
The metadata key.
value [required]
\nstring
The metadata value.
storage_class [required]
\nenum
Storage class used for objects stored in GCS. \nAllowed enum values: STANDARD,NEARLINE,COLDLINE,ARCHIVE
type [required]
\nenum
The destination type. Always google_cloud_storage. \nAllowed enum values: google_cloud_storage
default: google_cloud_storage
Option 4
\nobject
The splunk_hec destination forwards logs to Splunk using the HTTP Event Collector (HEC).
auto_extract_timestamp
\nboolean
If true, Splunk tries to extract timestamps from incoming log events.\nIf false, Splunk assigns the time the event was received.
encoding
\nenum
Encoding format for log events. \nAllowed enum values: json,raw_message
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
index
\nstring
Optional name of the Splunk index where logs are written.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
sourcetype
\nstring
The Splunk sourcetype to assign to log events.
type [required]
\nenum
The destination type. Always splunk_hec. \nAllowed enum values: splunk_hec
default: splunk_hec
Option 5
\nobject
The sumo_logic destination forwards logs to Sumo Logic.
encoding
\nenum
The output encoding format. \nAllowed enum values: json,raw_message,logfmt
header_custom_fields
\n[object]
A list of custom headers to include in the request to Sumo Logic.
name [required]
\nstring
The header field name.
value [required]
\nstring
The header field value.
header_host_name
\nstring
Optional override for the host name header.
header_source_category
\nstring
Optional override for the source category header.
header_source_name
\nstring
Optional override for the source name header.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be sumo_logic. \nAllowed enum values: sumo_logic
default: sumo_logic
Option 6
\nobject
The elasticsearch destination writes logs to an Elasticsearch cluster.
api_version
\nenum
The Elasticsearch API version to use. Set to auto to auto-detect. \nAllowed enum values: auto,v6,v7,v8
bulk_index
\nstring
The index to write logs to in Elasticsearch.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be elasticsearch. \nAllowed enum values: elasticsearch
default: elasticsearch
Option 7
\nobject
The rsyslog destination forwards logs to an external rsyslog server over TCP or UDP using the syslog protocol.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
keepalive
\nint64
Optional socket keepalive duration in milliseconds.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. The value should always be rsyslog. \nAllowed enum values: rsyslog
default: rsyslog
Option 8
\nobject
The syslog_ng destination forwards logs to an external syslog-ng server over TCP or UDP using the syslog protocol.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
keepalive
\nint64
Optional socket keepalive duration in milliseconds.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. The value should always be syslog_ng. \nAllowed enum values: syslog_ng
default: syslog_ng
Option 9
\nobject
The azure_storage destination forwards logs to an Azure Blob Storage container.
blob_prefix
\nstring
Optional prefix for blobs written to the container.
container_name [required]
\nstring
The name of the Azure Blob Storage container to store logs in.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be azure_storage. \nAllowed enum values: azure_storage
default: azure_storage
Option 10
\nobject
The microsoft_sentinel destination forwards logs to Microsoft Sentinel.
client_id [required]
\nstring
Azure AD client ID used for authentication.
dcr_immutable_id [required]
\nstring
The immutable ID of the Data Collection Rule (DCR).
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
table [required]
\nstring
The name of the Log Analytics table where logs are sent.
tenant_id [required]
\nstring
Azure AD tenant ID.
type [required]
\nenum
The destination type. The value should always be microsoft_sentinel. \nAllowed enum values: microsoft_sentinel
default: microsoft_sentinel
Option 11
\nobject
The google_chronicle destination sends logs to Google Chronicle.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
customer_id [required]
\nstring
The Google Chronicle customer ID.
encoding
\nenum
The encoding format for the logs sent to Chronicle. \nAllowed enum values: json,raw_message
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
log_type
\nstring
The log type metadata associated with the Chronicle destination.
type [required]
\nenum
The destination type. The value should always be google_chronicle. \nAllowed enum values: google_chronicle
default: google_chronicle
Option 12
\nobject
The new_relic destination sends logs to the New Relic platform.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The New Relic region. \nAllowed enum values: us,eu
type [required]
\nenum
The destination type. The value should always be new_relic. \nAllowed enum values: new_relic
default: new_relic
Option 13
\nobject
The sentinel_one destination sends logs to SentinelOne.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The SentinelOne region to send logs to. \nAllowed enum values: us,eu,ca,data_set_us
type [required]
\nenum
The destination type. The value should always be sentinel_one. \nAllowed enum values: sentinel_one
default: sentinel_one
Option 14
\nobject
The opensearch destination writes logs to an OpenSearch cluster.
bulk_index
\nstring
The index to write logs to.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be opensearch. \nAllowed enum values: opensearch
default: opensearch
Option 15
\nobject
The amazon_opensearch destination writes logs to Amazon OpenSearch.
auth [required]
\nobject
Authentication settings for the Amazon OpenSearch destination.\nThe strategy field determines whether basic or AWS-based authentication is used.
assume_role
\nstring
The ARN of the role to assume (used with aws strategy).
aws_region
\nstring
AWS region
external_id
\nstring
External ID for the assumed role (used with aws strategy).
session_name
\nstring
Session name for the assumed role (used with aws strategy).
strategy [required]
\nenum
The authentication strategy to use. \nAllowed enum values: basic,aws
bulk_index
\nstring
The index to write logs to.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be amazon_opensearch. \nAllowed enum values: amazon_opensearch
default: amazon_opensearch
processors
\n[ <oneOf>]
A list of processors that transform or enrich log data.
Option 1
\nobject
The filter processor allows conditional processing of logs based on a Datadog search query. Logs that match the include query are passed through; others are discarded.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs should pass through the filter. Logs that match this query continue to downstream components; others are dropped.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be filter. \nAllowed enum values: filter
default: filter
Option 2
\nobject
The parse_json processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string.
field [required]
\nstring
The name of the log field that contains a JSON string.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be parse_json. \nAllowed enum values: parse_json
default: parse_json
Option 3
\nobject
The Quota Processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert.
drop_events [required]
\nboolean
If set to true, logs that matched the quota filter and sent after the quota has been met are dropped; only logs that did not match the filter query continue through the pipeline.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
ignore_when_missing_partitions
\nboolean
If true, the processor skips quota checks when partition fields are missing from the logs.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
name [required]
\nstring
Name of the quota.
overflow_action
\nenum
The action to take when the quota is exceeded. Options:
\ndrop: Drop the event.
no_action: Let the event pass through.
overflow_routing: Route to an overflow destination.
Allowed enum values: drop,no_action,overflow_routing
overrides
\n[object]
A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit.
fields [required]
\n[object]
A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
partition_fields
\n[string]
A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values.
type [required]
\nenum
The processor type. The value should always be quota. \nAllowed enum values: quota
default: quota
Option 4
\nobject
The add_fields processor adds static key-value fields to logs.
fields [required]
\n[object]
A list of static fields (key-value pairs) that is added to each log event processed by this component.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be add_fields. \nAllowed enum values: add_fields
default: add_fields
Option 5
\nobject
The remove_fields processor deletes specified fields from logs.
fields [required]
\n[string]
A list of field names to be removed from each log event.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
The PipelineRemoveFieldsProcessor inputs.
type [required]
\nenum
The processor type. The value should always be remove_fields. \nAllowed enum values: remove_fields
default: remove_fields
Option 6
\nobject
The rename_fields processor changes field names.
fields [required]
\n[object]
A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields.
destination [required]
\nstring
The field name to assign the renamed value to.
preserve_source [required]
\nboolean
Indicates whether the original field, that is received from the source, should be kept (true) or removed (false) after renaming.
source [required]
\nstring
The original field name in the log event that should be renamed.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be rename_fields. \nAllowed enum values: rename_fields
default: rename_fields
Option 7
\nobject
The generate_datadog_metrics processor creates custom metrics from logs and sends them to Datadog.\nMetrics can be counters, gauges, or distributions and optionally grouped by log fields.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
metrics [required]
\n[object]
Configuration for generating individual metrics.
group_by
\n[string]
Optional fields used to group the metric series.
include [required]
\nstring
Datadog filter query to match logs for metric generation.
metric_type [required]
\nenum
Type of metric to create. \nAllowed enum values: count,gauge,distribution
name [required]
\nstring
Name of the custom metric to be created.
value [required]
\n<oneOf>
Specifies how the value of the generated metric is computed.
Option 1
\nobject
Strategy that increments a generated metric by one for each matching event.
strategy [required]
\nenum
Increments the metric by 1 for each matching event. \nAllowed enum values: increment_by_one
Option 2
\nobject
Strategy that increments a generated metric based on the value of a log field.
field [required]
\nstring
Name of the log field containing the numeric value to increment the metric by.
strategy [required]
\nenum
Uses a numeric field in the log event as the metric increment. \nAllowed enum values: increment_by_field
type [required]
\nenum
The processor type. Always generate_datadog_metrics. \nAllowed enum values: generate_datadog_metrics
default: generate_datadog_metrics
Option 8
\nobject
The sample processor allows probabilistic sampling of logs at a fixed rate.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
percentage
\ndouble
The percentage of logs to sample.
rate
\nint64
Number of events to sample (1 in N).
type [required]
\nenum
The processor type. The value should always be sample. \nAllowed enum values: sample
default: sample
Option 9
\nobject
The parse_grok processor extracts structured fields from unstructured log messages using Grok patterns.
disable_library_rules
\nboolean
If set to true, disables the default Grok rules provided by Datadog.
id [required]
\nstring
A unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
rules [required]
\n[object]
The list of Grok parsing rules. If multiple matching rules are provided, they are evaluated in order. The first successful match is applied.
match_rules [required]
\n[object]
A list of Grok parsing rules that define how to extract fields from the source field.\nEach rule must contain a name and a valid Grok pattern.
name [required]
\nstring
The name of the rule.
rule [required]
\nstring
The definition of the Grok rule.
source [required]
\nstring
The name of the field in the log event to apply the Grok rules to.
support_rules [required]
\n[object]
A list of Grok helper rules that can be referenced by the parsing rules.
name [required]
\nstring
The name of the Grok helper rule.
rule [required]
\nstring
The definition of the Grok helper rule.
type [required]
\nenum
The processor type. The value should always be parse_grok. \nAllowed enum values: parse_grok
default: parse_grok
Option 10
\nobject
The sensitive_data_scanner processor detects and optionally redacts sensitive data in log events.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
rules [required]
\n[object]
A list of rules for identifying and acting on sensitive data patterns.
keyword_options
\nobject
Configuration for keywords used to reinforce sensitive data pattern detection.
keywords [required]
\n[string]
A list of keywords to match near the sensitive pattern.
proximity [required]
\nint64
Maximum number of tokens between a keyword and a sensitive value match.
name [required]
\nstring
A name identifying the rule.
on_match [required]
\n<oneOf>
Defines what action to take when sensitive data is matched.
Option 1
\nobject
Configuration for completely redacting matched sensitive data.
action [required]
\nenum
Action type that completely replaces the matched sensitive data with a fixed replacement string to remove all visibility. \nAllowed enum values: redact
options [required]
\nobject
Configuration for fully redacting sensitive data.
replace [required]
\nstring
The ObservabilityPipelineSensitiveDataScannerProcessorActionRedactOptions replace.
Option 2
\nobject
Configuration for hashing matched sensitive values.
action [required]
\nenum
Action type that replaces the matched sensitive data with a hashed representation, preserving structure while securing content. \nAllowed enum values: hash
options
\nobject
The ObservabilityPipelineSensitiveDataScannerProcessorActionHash options.
Option 3
\nobject
Configuration for partially redacting matched sensitive data.
action [required]
\nenum
Action type that redacts part of the sensitive data while preserving a configurable number of characters, typically used for masking purposes (e.g., show last 4 digits of a credit card). \nAllowed enum values: partial_redact
options [required]
\nobject
Controls how partial redaction is applied, including character count and direction.
characters [required]
\nint64
The ObservabilityPipelineSensitiveDataScannerProcessorActionPartialRedactOptions characters.
direction [required]
\nenum
Indicates whether to redact characters from the first or last part of the matched value. \nAllowed enum values: first,last
pattern [required]
\n<oneOf>
Pattern detection configuration for identifying sensitive data using either a custom regex or a library reference.
Option 1
\nobject
Defines a custom regex-based pattern for identifying sensitive data in logs.
options [required]
\nobject
Options for defining a custom regex pattern.
rule [required]
\nstring
A regular expression used to detect sensitive values. Must be a valid regex.
type [required]
\nenum
Indicates a custom regular expression is used for matching. \nAllowed enum values: custom
Option 2
\nobject
Specifies a pattern from Datadog’s sensitive data detection library to match known sensitive data types.
options [required]
\nobject
Options for selecting a predefined library pattern and enabling keyword support.
id [required]
\nstring
Identifier for a predefined pattern from the sensitive data scanner pattern library.
use_recommended_keywords
\nboolean
Whether to augment the pattern with recommended keywords (optional).
type [required]
\nenum
Indicates that a predefined library pattern is used. \nAllowed enum values: library
scope [required]
\n<oneOf>
Determines which parts of the log the pattern-matching rule should be applied to.
Option 1
\nobject
Includes only specific fields for sensitive data scanning.
options [required]
\nobject
Fields to which the scope rule applies.
fields [required]
\n[string]
The ObservabilityPipelineSensitiveDataScannerProcessorScopeOptions fields.
target [required]
\nenum
Applies the rule only to included fields. \nAllowed enum values: include
Option 2
\nobject
Excludes specific fields from sensitive data scanning.
options [required]
\nobject
Fields to which the scope rule applies.
fields [required]
\n[string]
The ObservabilityPipelineSensitiveDataScannerProcessorScopeOptions fields.
target [required]
\nenum
Excludes specific fields from processing. \nAllowed enum values: exclude
Option 3
\nobject
Applies scanning across all available fields.
target [required]
\nenum
Applies the rule to all fields. \nAllowed enum values: all
tags [required]
\n[string]
Tags assigned to this rule for filtering and classification.
type [required]
\nenum
The processor type. The value should always be sensitive_data_scanner. \nAllowed enum values: sensitive_data_scanner
default: sensitive_data_scanner
Option 11
\nobject
The ocsf_mapper processor transforms logs into the OCSF schema using a predefined mapping configuration.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mappings [required]
\n[object]
A list of mapping rules to convert events to the OCSF format.
include [required]
\nstring
A Datadog search query used to select the logs that this mapping should apply to.
mapping [required]
\n<oneOf>
Defines a single mapping rule for transforming logs into the OCSF schema.
Option 1
\nenum
Predefined library mappings for common log formats. \nAllowed enum values: CloudTrail Account Change,GCP Cloud Audit CreateBucket,GCP Cloud Audit CreateSink,GCP Cloud Audit SetIamPolicy,GCP Cloud Audit UpdateSink,Github Audit Log API Activity,Google Workspace Admin Audit addPrivilege,Microsoft 365 Defender Incident,Microsoft 365 Defender UserLoggedIn,Okta System Log Authentication,Palo Alto Networks Firewall Traffic
type [required]
\nenum
The processor type. The value should always be ocsf_mapper. \nAllowed enum values: ocsf_mapper
default: ocsf_mapper
Option 12
\nobject
The add_env_vars processor adds environment variable values to log events.
id [required]
\nstring
The unique identifier for this component. Used to reference this processor in the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
type [required]
\nenum
The processor type. The value should always be add_env_vars. \nAllowed enum values: add_env_vars
default: add_env_vars
variables [required]
\n[object]
A list of environment variable mappings to apply to log fields.
field [required]
\nstring
The target field in the log event.
name [required]
\nstring
The name of the environment variable to read.
Option 13
\nobject
The dedupe processor removes duplicate fields in log events.
fields [required]
\n[string]
A list of log field paths to check for duplicates.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mode [required]
\nenum
The deduplication mode to apply to the fields. \nAllowed enum values: match,ignore
type [required]
\nenum
The processor type. The value should always be dedupe. \nAllowed enum values: dedupe
default: dedupe
Option 14
\nobject
The enrichment_table processor enriches logs using a static CSV file or GeoIP database.
file
\nobject
Defines a static enrichment table loaded from a CSV file.
encoding [required]
\nobject
File encoding format.
delimiter [required]
\nstring
The encoding delimiter.
includes_headers [required]
\nboolean
The encoding includes_headers.
type [required]
\nenum
Specifies the encoding format (e.g., CSV) used for enrichment tables. \nAllowed enum values: csv
key [required]
\n[object]
Key fields used to look up enrichment values.
column [required]
\nstring
The items column.
comparison [required]
\nenum
Defines how to compare key fields for enrichment table lookups. \nAllowed enum values: equals
field [required]
\nstring
The items field.
path [required]
\nstring
Path to the CSV file.
schema [required]
\n[object]
Schema defining column names and their types.
column [required]
\nstring
The items column.
type [required]
\nenum
Declares allowed data types for enrichment table columns. \nAllowed enum values: string,boolean,integer,float,date,timestamp
geoip
\nobject
Uses a GeoIP database to enrich logs based on an IP field.
key_field [required]
\nstring
Path to the IP field in the log.
locale [required]
\nstring
Locale used to resolve geographical names.
path [required]
\nstring
Path to the GeoIP database file.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
target [required]
\nstring
Path where enrichment results should be stored in the log.
type [required]
\nenum
The processor type. The value should always be enrichment_table. \nAllowed enum values: enrichment_table
default: enrichment_table
Option 15
\nobject
The reduce processor aggregates and merges logs based on matching keys and merge strategies.
group_by [required]
\n[string]
A list of fields used to group log events for merging.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
merge_strategies [required]
\n[object]
List of merge strategies defining how values from grouped events should be combined.
path [required]
\nstring
The field path in the log event.
strategy [required]
\nenum
The merge strategy to apply. \nAllowed enum values: discard,retain,sum,max,min,array,concat,concat_newline,concat_raw,shortest_array,longest_array,flat_unique
type [required]
\nenum
The processor type. The value should always be reduce. \nAllowed enum values: reduce
default: reduce
Option 16
\nobject
The throttle processor limits the number of events that pass through over a given time window.
group_by
\n[string]
Optional list of fields used to group events before the threshold has been reached.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
threshold [required]
\nint64
the number of events allowed in a given time window. Events sent after the threshold has been reached, are dropped.
type [required]
\nenum
The processor type. The value should always be throttle. \nAllowed enum values: throttle
default: throttle
window [required]
\ndouble
The time window in seconds over which the threshold applies.
Option 17
\nobject
The datadog_tags processor includes or excludes specific Datadog tags in your logs.
action [required]
\nenum
The action to take on tags with matching keys. \nAllowed enum values: include,exclude
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
keys [required]
\n[string]
A list of tag keys.
mode [required]
\nenum
The processing mode. \nAllowed enum values: filter
type [required]
\nenum
The processor type. The value should always be datadog_tags. \nAllowed enum values: datadog_tags
default: datadog_tags
sources [required]
\n[ <oneOf>]
A list of configured data sources for the pipeline.
Option 1
\nobject
The kafka source ingests data from Apache Kafka topics.
group_id [required]
\nstring
Consumer group ID used by the Kafka client.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
librdkafka_options
\n[object]
Optional list of advanced Kafka client configuration options, defined as key-value pairs.
name [required]
\nstring
The name of the librdkafka configuration option to set.
value [required]
\nstring
The value assigned to the specified librdkafka configuration option.
sasl
\nobject
Specifies the SASL mechanism for authenticating with a Kafka cluster.
mechanism
\nenum
SASL mechanism used for Kafka authentication. \nAllowed enum values: PLAIN,SCRAM-SHA-256,SCRAM-SHA-512
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
topics [required]
\n[string]
A list of Kafka topic names to subscribe to. The source ingests messages from each topic specified.
type [required]
\nenum
The source type. The value should always be kafka. \nAllowed enum values: kafka
default: kafka
Option 2
\nobject
The datadog_agent source collects logs from the Datadog Agent.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be datadog_agent. \nAllowed enum values: datadog_agent
default: datadog_agent
Option 3
\nobject
The splunk_tcp source receives logs from a Splunk Universal Forwarder over TCP.\nTLS is supported for secure transmission.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always splunk_tcp. \nAllowed enum values: splunk_tcp
default: splunk_tcp
Option 4
\nobject
The splunk_hec source implements the Splunk HTTP Event Collector (HEC) API.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always splunk_hec. \nAllowed enum values: splunk_hec
default: splunk_hec
Option 5
\nobject
The amazon_s3 source ingests logs from an Amazon S3 bucket.\nIt supports AWS authentication and TLS encryption.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
region [required]
\nstring
AWS region where the S3 bucket resides.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always amazon_s3. \nAllowed enum values: amazon_s3
default: amazon_s3
Option 6
\nobject
The fluentd source ingests logs from a Fluentd-compatible service.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be `fluentd. \nAllowed enum values: fluentd
default: fluentd
Option 7
\nobject
The fluent_bit source ingests logs from Fluent Bit.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be fluent_bit. \nAllowed enum values: fluent_bit
default: fluent_bit
Option 8
\nobject
The http_server source collects logs over HTTP POST from external services.
auth_strategy [required]
\nenum
HTTP authentication method. \nAllowed enum values: none,plain
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
Unique ID for the HTTP server source.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be http_server. \nAllowed enum values: http_server
default: http_server
Option 9
\nobject
The sumo_logic source receives logs from Sumo Logic collectors.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
type [required]
\nenum
The source type. The value should always be sumo_logic. \nAllowed enum values: sumo_logic
default: sumo_logic
Option 10
\nobject
The rsyslog source listens for logs over TCP or UDP from an rsyslog server using the syslog protocol.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
mode [required]
\nenum
Protocol used by the syslog source to receive messages. \nAllowed enum values: tcp,udp
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be rsyslog. \nAllowed enum values: rsyslog
default: rsyslog
Option 11
\nobject
The syslog_ng source listens for logs over TCP or UDP from a syslog-ng server using the syslog protocol.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
mode [required]
\nenum
Protocol used by the syslog source to receive messages. \nAllowed enum values: tcp,udp
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be syslog_ng. \nAllowed enum values: syslog_ng
default: syslog_ng
Option 12
\nobject
The amazon_data_firehose source ingests logs from AWS Data Firehose.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be amazon_data_firehose. \nAllowed enum values: amazon_data_firehose
default: amazon_data_firehose
Option 13
\nobject
The google_pubsub source ingests logs from a Google Cloud Pub/Sub subscription.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
project [required]
\nstring
The GCP project ID that owns the Pub/Sub subscription.
subscription [required]
\nstring
The Pub/Sub subscription name from which messages are consumed.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be google_pubsub. \nAllowed enum values: google_pubsub
default: google_pubsub
Option 14
\nobject
The http_client source scrapes logs from HTTP endpoints at regular intervals.
auth_strategy
\nenum
Optional authentication strategy for HTTP requests. \nAllowed enum values: basic,bearer
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
scrape_interval_secs
\nint64
The interval (in seconds) between HTTP scrape requests.
scrape_timeout_secs
\nint64
The timeout (in seconds) for each scrape request.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be http_client. \nAllowed enum values: http_client
default: http_client
Option 15
\nobject
The logstash source ingests logs from a Logstash forwarder.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be logstash. \nAllowed enum values: logstash
default: logstash
name [required]
\nstring
Name of the pipeline.
id [required]
\nstring
Unique identifier for the pipeline.
type [required]
\nstring
The resource type identifier. For pipeline resources, this should always be set to pipelines.
default: pipelines
data [required]
\nobject
Contains the pipeline’s ID, type, and configuration attributes.
attributes [required]
\nobject
Defines the pipeline’s name and its components (sources, processors, and destinations).
config [required]
\nobject
Specifies the pipeline's configuration, including its sources, processors, and destinations.
destinations [required]
\n[ <oneOf>]
A list of destination components where processed logs are sent.
Option 1
\nobject
The datadog_logs destination forwards logs to Datadog Log Management.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be datadog_logs. \nAllowed enum values: datadog_logs
default: datadog_logs
Option 2
\nobject
The amazon_s3 destination sends your logs in Datadog-rehydratable format to an Amazon S3 bucket for archiving.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
bucket [required]
\nstring
S3 bucket name.
id [required]
\nstring
Unique identifier for the destination component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
key_prefix
\nstring
Optional prefix for object keys.
region [required]
\nstring
AWS region of the S3 bucket.
storage_class [required]
\nenum
S3 storage class. \nAllowed enum values: STANDARD,REDUCED_REDUNDANCY,INTELLIGENT_TIERING,STANDARD_IA,EXPRESS_ONEZONE,ONEZONE_IA,GLACIER,GLACIER_IR,DEEP_ARCHIVE
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. Always amazon_s3. \nAllowed enum values: amazon_s3
default: amazon_s3
Option 3
\nobject
The google_cloud_storage destination stores logs in a Google Cloud Storage (GCS) bucket.\nIt requires a bucket name, GCP authentication, and metadata fields.
acl [required]
\nenum
Access control list setting for objects written to the bucket. \nAllowed enum values: private,project-private,public-read,authenticated-read,bucket-owner-read,bucket-owner-full-control
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
bucket [required]
\nstring
Name of the GCS bucket.
id [required]
\nstring
Unique identifier for the destination component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
key_prefix
\nstring
Optional prefix for object keys within the GCS bucket.
metadata
\n[object]
Custom metadata to attach to each object uploaded to the GCS bucket.
name [required]
\nstring
The metadata key.
value [required]
\nstring
The metadata value.
storage_class [required]
\nenum
Storage class used for objects stored in GCS. \nAllowed enum values: STANDARD,NEARLINE,COLDLINE,ARCHIVE
type [required]
\nenum
The destination type. Always google_cloud_storage. \nAllowed enum values: google_cloud_storage
default: google_cloud_storage
Option 4
\nobject
The splunk_hec destination forwards logs to Splunk using the HTTP Event Collector (HEC).
auto_extract_timestamp
\nboolean
If true, Splunk tries to extract timestamps from incoming log events.\nIf false, Splunk assigns the time the event was received.
encoding
\nenum
Encoding format for log events. \nAllowed enum values: json,raw_message
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
index
\nstring
Optional name of the Splunk index where logs are written.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
sourcetype
\nstring
The Splunk sourcetype to assign to log events.
type [required]
\nenum
The destination type. Always splunk_hec. \nAllowed enum values: splunk_hec
default: splunk_hec
Option 5
\nobject
The sumo_logic destination forwards logs to Sumo Logic.
encoding
\nenum
The output encoding format. \nAllowed enum values: json,raw_message,logfmt
header_custom_fields
\n[object]
A list of custom headers to include in the request to Sumo Logic.
name [required]
\nstring
The header field name.
value [required]
\nstring
The header field value.
header_host_name
\nstring
Optional override for the host name header.
header_source_category
\nstring
Optional override for the source category header.
header_source_name
\nstring
Optional override for the source name header.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be sumo_logic. \nAllowed enum values: sumo_logic
default: sumo_logic
Option 6
\nobject
The elasticsearch destination writes logs to an Elasticsearch cluster.
api_version
\nenum
The Elasticsearch API version to use. Set to auto to auto-detect. \nAllowed enum values: auto,v6,v7,v8
bulk_index
\nstring
The index to write logs to in Elasticsearch.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be elasticsearch. \nAllowed enum values: elasticsearch
default: elasticsearch
Option 7
\nobject
The rsyslog destination forwards logs to an external rsyslog server over TCP or UDP using the syslog protocol.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
keepalive
\nint64
Optional socket keepalive duration in milliseconds.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. The value should always be rsyslog. \nAllowed enum values: rsyslog
default: rsyslog
Option 8
\nobject
The syslog_ng destination forwards logs to an external syslog-ng server over TCP or UDP using the syslog protocol.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
keepalive
\nint64
Optional socket keepalive duration in milliseconds.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. The value should always be syslog_ng. \nAllowed enum values: syslog_ng
default: syslog_ng
Option 9
\nobject
The azure_storage destination forwards logs to an Azure Blob Storage container.
blob_prefix
\nstring
Optional prefix for blobs written to the container.
container_name [required]
\nstring
The name of the Azure Blob Storage container to store logs in.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be azure_storage. \nAllowed enum values: azure_storage
default: azure_storage
Option 10
\nobject
The microsoft_sentinel destination forwards logs to Microsoft Sentinel.
client_id [required]
\nstring
Azure AD client ID used for authentication.
dcr_immutable_id [required]
\nstring
The immutable ID of the Data Collection Rule (DCR).
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
table [required]
\nstring
The name of the Log Analytics table where logs are sent.
tenant_id [required]
\nstring
Azure AD tenant ID.
type [required]
\nenum
The destination type. The value should always be microsoft_sentinel. \nAllowed enum values: microsoft_sentinel
default: microsoft_sentinel
Option 11
\nobject
The google_chronicle destination sends logs to Google Chronicle.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
customer_id [required]
\nstring
The Google Chronicle customer ID.
encoding
\nenum
The encoding format for the logs sent to Chronicle. \nAllowed enum values: json,raw_message
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
log_type
\nstring
The log type metadata associated with the Chronicle destination.
type [required]
\nenum
The destination type. The value should always be google_chronicle. \nAllowed enum values: google_chronicle
default: google_chronicle
Option 12
\nobject
The new_relic destination sends logs to the New Relic platform.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The New Relic region. \nAllowed enum values: us,eu
type [required]
\nenum
The destination type. The value should always be new_relic. \nAllowed enum values: new_relic
default: new_relic
Option 13
\nobject
The sentinel_one destination sends logs to SentinelOne.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The SentinelOne region to send logs to. \nAllowed enum values: us,eu,ca,data_set_us
type [required]
\nenum
The destination type. The value should always be sentinel_one. \nAllowed enum values: sentinel_one
default: sentinel_one
Option 14
\nobject
The opensearch destination writes logs to an OpenSearch cluster.
bulk_index
\nstring
The index to write logs to.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be opensearch. \nAllowed enum values: opensearch
default: opensearch
Option 15
\nobject
The amazon_opensearch destination writes logs to Amazon OpenSearch.
auth [required]
\nobject
Authentication settings for the Amazon OpenSearch destination.\nThe strategy field determines whether basic or AWS-based authentication is used.
assume_role
\nstring
The ARN of the role to assume (used with aws strategy).
aws_region
\nstring
AWS region
external_id
\nstring
External ID for the assumed role (used with aws strategy).
session_name
\nstring
Session name for the assumed role (used with aws strategy).
strategy [required]
\nenum
The authentication strategy to use. \nAllowed enum values: basic,aws
bulk_index
\nstring
The index to write logs to.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be amazon_opensearch. \nAllowed enum values: amazon_opensearch
default: amazon_opensearch
processors
\n[ <oneOf>]
A list of processors that transform or enrich log data.
Option 1
\nobject
The filter processor allows conditional processing of logs based on a Datadog search query. Logs that match the include query are passed through; others are discarded.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs should pass through the filter. Logs that match this query continue to downstream components; others are dropped.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be filter. \nAllowed enum values: filter
default: filter
Option 2
\nobject
The parse_json processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string.
field [required]
\nstring
The name of the log field that contains a JSON string.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be parse_json. \nAllowed enum values: parse_json
default: parse_json
Option 3
\nobject
The Quota Processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert.
drop_events [required]
\nboolean
If set to true, logs that matched the quota filter and sent after the quota has been met are dropped; only logs that did not match the filter query continue through the pipeline.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
ignore_when_missing_partitions
\nboolean
If true, the processor skips quota checks when partition fields are missing from the logs.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
name [required]
\nstring
Name of the quota.
overflow_action
\nenum
The action to take when the quota is exceeded. Options:
\ndrop: Drop the event.
no_action: Let the event pass through.
overflow_routing: Route to an overflow destination.
Allowed enum values: drop,no_action,overflow_routing
overrides
\n[object]
A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit.
fields [required]
\n[object]
A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
partition_fields
\n[string]
A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values.
type [required]
\nenum
The processor type. The value should always be quota. \nAllowed enum values: quota
default: quota
Option 4
\nobject
The add_fields processor adds static key-value fields to logs.
fields [required]
\n[object]
A list of static fields (key-value pairs) that is added to each log event processed by this component.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be add_fields. \nAllowed enum values: add_fields
default: add_fields
Option 5
\nobject
The remove_fields processor deletes specified fields from logs.
fields [required]
\n[string]
A list of field names to be removed from each log event.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
The PipelineRemoveFieldsProcessor inputs.
type [required]
\nenum
The processor type. The value should always be remove_fields. \nAllowed enum values: remove_fields
default: remove_fields
Option 6
\nobject
The rename_fields processor changes field names.
fields [required]
\n[object]
A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields.
destination [required]
\nstring
The field name to assign the renamed value to.
preserve_source [required]
\nboolean
Indicates whether the original field, that is received from the source, should be kept (true) or removed (false) after renaming.
source [required]
\nstring
The original field name in the log event that should be renamed.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be rename_fields. \nAllowed enum values: rename_fields
default: rename_fields
Option 7
\nobject
The generate_datadog_metrics processor creates custom metrics from logs and sends them to Datadog.\nMetrics can be counters, gauges, or distributions and optionally grouped by log fields.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
metrics [required]
\n[object]
Configuration for generating individual metrics.
group_by
\n[string]
Optional fields used to group the metric series.
include [required]
\nstring
Datadog filter query to match logs for metric generation.
metric_type [required]
\nenum
Type of metric to create. \nAllowed enum values: count,gauge,distribution
name [required]
\nstring
Name of the custom metric to be created.
value [required]
\n<oneOf>
Specifies how the value of the generated metric is computed.
Option 1
\nobject
Strategy that increments a generated metric by one for each matching event.
strategy [required]
\nenum
Increments the metric by 1 for each matching event. \nAllowed enum values: increment_by_one
Option 2
\nobject
Strategy that increments a generated metric based on the value of a log field.
field [required]
\nstring
Name of the log field containing the numeric value to increment the metric by.
strategy [required]
\nenum
Uses a numeric field in the log event as the metric increment. \nAllowed enum values: increment_by_field
type [required]
\nenum
The processor type. Always generate_datadog_metrics. \nAllowed enum values: generate_datadog_metrics
default: generate_datadog_metrics
Option 8
\nobject
The sample processor allows probabilistic sampling of logs at a fixed rate.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
percentage
\ndouble
The percentage of logs to sample.
rate
\nint64
Number of events to sample (1 in N).
type [required]
\nenum
The processor type. The value should always be sample. \nAllowed enum values: sample
default: sample
Option 9
\nobject
The parse_grok processor extracts structured fields from unstructured log messages using Grok patterns.
disable_library_rules
\nboolean
If set to true, disables the default Grok rules provided by Datadog.
id [required]
\nstring
A unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
rules [required]
\n[object]
The list of Grok parsing rules. If multiple matching rules are provided, they are evaluated in order. The first successful match is applied.
match_rules [required]
\n[object]
A list of Grok parsing rules that define how to extract fields from the source field.\nEach rule must contain a name and a valid Grok pattern.
name [required]
\nstring
The name of the rule.
rule [required]
\nstring
The definition of the Grok rule.
source [required]
\nstring
The name of the field in the log event to apply the Grok rules to.
support_rules [required]
\n[object]
A list of Grok helper rules that can be referenced by the parsing rules.
name [required]
\nstring
The name of the Grok helper rule.
rule [required]
\nstring
The definition of the Grok helper rule.
type [required]
\nenum
The processor type. The value should always be parse_grok. \nAllowed enum values: parse_grok
default: parse_grok
Option 10
\nobject
The sensitive_data_scanner processor detects and optionally redacts sensitive data in log events.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
rules [required]
\n[object]
A list of rules for identifying and acting on sensitive data patterns.
keyword_options
\nobject
Configuration for keywords used to reinforce sensitive data pattern detection.
keywords [required]
\n[string]
A list of keywords to match near the sensitive pattern.
proximity [required]
\nint64
Maximum number of tokens between a keyword and a sensitive value match.
name [required]
\nstring
A name identifying the rule.
on_match [required]
\n<oneOf>
Defines what action to take when sensitive data is matched.
Option 1
\nobject
Configuration for completely redacting matched sensitive data.
action [required]
\nenum
Action type that completely replaces the matched sensitive data with a fixed replacement string to remove all visibility. \nAllowed enum values: redact
options [required]
\nobject
Configuration for fully redacting sensitive data.
replace [required]
\nstring
The ObservabilityPipelineSensitiveDataScannerProcessorActionRedactOptions replace.
Option 2
\nobject
Configuration for hashing matched sensitive values.
action [required]
\nenum
Action type that replaces the matched sensitive data with a hashed representation, preserving structure while securing content. \nAllowed enum values: hash
options
\nobject
The ObservabilityPipelineSensitiveDataScannerProcessorActionHash options.
Option 3
\nobject
Configuration for partially redacting matched sensitive data.
action [required]
\nenum
Action type that redacts part of the sensitive data while preserving a configurable number of characters, typically used for masking purposes (e.g., show last 4 digits of a credit card). \nAllowed enum values: partial_redact
options [required]
\nobject
Controls how partial redaction is applied, including character count and direction.
characters [required]
\nint64
The ObservabilityPipelineSensitiveDataScannerProcessorActionPartialRedactOptions characters.
direction [required]
\nenum
Indicates whether to redact characters from the first or last part of the matched value. \nAllowed enum values: first,last
pattern [required]
\n<oneOf>
Pattern detection configuration for identifying sensitive data using either a custom regex or a library reference.
Option 1
\nobject
Defines a custom regex-based pattern for identifying sensitive data in logs.
options [required]
\nobject
Options for defining a custom regex pattern.
rule [required]
\nstring
A regular expression used to detect sensitive values. Must be a valid regex.
type [required]
\nenum
Indicates a custom regular expression is used for matching. \nAllowed enum values: custom
Option 2
\nobject
Specifies a pattern from Datadog’s sensitive data detection library to match known sensitive data types.
options [required]
\nobject
Options for selecting a predefined library pattern and enabling keyword support.
id [required]
\nstring
Identifier for a predefined pattern from the sensitive data scanner pattern library.
use_recommended_keywords
\nboolean
Whether to augment the pattern with recommended keywords (optional).
type [required]
\nenum
Indicates that a predefined library pattern is used. \nAllowed enum values: library
scope [required]
\n<oneOf>
Determines which parts of the log the pattern-matching rule should be applied to.
Option 1
\nobject
Includes only specific fields for sensitive data scanning.
options [required]
\nobject
Fields to which the scope rule applies.
fields [required]
\n[string]
The ObservabilityPipelineSensitiveDataScannerProcessorScopeOptions fields.
target [required]
\nenum
Applies the rule only to included fields. \nAllowed enum values: include
Option 2
\nobject
Excludes specific fields from sensitive data scanning.
options [required]
\nobject
Fields to which the scope rule applies.
fields [required]
\n[string]
The ObservabilityPipelineSensitiveDataScannerProcessorScopeOptions fields.
target [required]
\nenum
Excludes specific fields from processing. \nAllowed enum values: exclude
Option 3
\nobject
Applies scanning across all available fields.
target [required]
\nenum
Applies the rule to all fields. \nAllowed enum values: all
tags [required]
\n[string]
Tags assigned to this rule for filtering and classification.
type [required]
\nenum
The processor type. The value should always be sensitive_data_scanner. \nAllowed enum values: sensitive_data_scanner
default: sensitive_data_scanner
Option 11
\nobject
The ocsf_mapper processor transforms logs into the OCSF schema using a predefined mapping configuration.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mappings [required]
\n[object]
A list of mapping rules to convert events to the OCSF format.
include [required]
\nstring
A Datadog search query used to select the logs that this mapping should apply to.
mapping [required]
\n<oneOf>
Defines a single mapping rule for transforming logs into the OCSF schema.
Option 1
\nenum
Predefined library mappings for common log formats. \nAllowed enum values: CloudTrail Account Change,GCP Cloud Audit CreateBucket,GCP Cloud Audit CreateSink,GCP Cloud Audit SetIamPolicy,GCP Cloud Audit UpdateSink,Github Audit Log API Activity,Google Workspace Admin Audit addPrivilege,Microsoft 365 Defender Incident,Microsoft 365 Defender UserLoggedIn,Okta System Log Authentication,Palo Alto Networks Firewall Traffic
type [required]
\nenum
The processor type. The value should always be ocsf_mapper. \nAllowed enum values: ocsf_mapper
default: ocsf_mapper
Option 12
\nobject
The add_env_vars processor adds environment variable values to log events.
id [required]
\nstring
The unique identifier for this component. Used to reference this processor in the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
type [required]
\nenum
The processor type. The value should always be add_env_vars. \nAllowed enum values: add_env_vars
default: add_env_vars
variables [required]
\n[object]
A list of environment variable mappings to apply to log fields.
field [required]
\nstring
The target field in the log event.
name [required]
\nstring
The name of the environment variable to read.
Option 13
\nobject
The dedupe processor removes duplicate fields in log events.
fields [required]
\n[string]
A list of log field paths to check for duplicates.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mode [required]
\nenum
The deduplication mode to apply to the fields. \nAllowed enum values: match,ignore
type [required]
\nenum
The processor type. The value should always be dedupe. \nAllowed enum values: dedupe
default: dedupe
Option 14
\nobject
The enrichment_table processor enriches logs using a static CSV file or GeoIP database.
file
\nobject
Defines a static enrichment table loaded from a CSV file.
encoding [required]
\nobject
File encoding format.
delimiter [required]
\nstring
The encoding delimiter.
includes_headers [required]
\nboolean
The encoding includes_headers.
type [required]
\nenum
Specifies the encoding format (e.g., CSV) used for enrichment tables. \nAllowed enum values: csv
key [required]
\n[object]
Key fields used to look up enrichment values.
column [required]
\nstring
The items column.
comparison [required]
\nenum
Defines how to compare key fields for enrichment table lookups. \nAllowed enum values: equals
field [required]
\nstring
The items field.
path [required]
\nstring
Path to the CSV file.
schema [required]
\n[object]
Schema defining column names and their types.
column [required]
\nstring
The items column.
type [required]
\nenum
Declares allowed data types for enrichment table columns. \nAllowed enum values: string,boolean,integer,float,date,timestamp
geoip
\nobject
Uses a GeoIP database to enrich logs based on an IP field.
key_field [required]
\nstring
Path to the IP field in the log.
locale [required]
\nstring
Locale used to resolve geographical names.
path [required]
\nstring
Path to the GeoIP database file.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
target [required]
\nstring
Path where enrichment results should be stored in the log.
type [required]
\nenum
The processor type. The value should always be enrichment_table. \nAllowed enum values: enrichment_table
default: enrichment_table
Option 15
\nobject
The reduce processor aggregates and merges logs based on matching keys and merge strategies.
group_by [required]
\n[string]
A list of fields used to group log events for merging.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
merge_strategies [required]
\n[object]
List of merge strategies defining how values from grouped events should be combined.
path [required]
\nstring
The field path in the log event.
strategy [required]
\nenum
The merge strategy to apply. \nAllowed enum values: discard,retain,sum,max,min,array,concat,concat_newline,concat_raw,shortest_array,longest_array,flat_unique
type [required]
\nenum
The processor type. The value should always be reduce. \nAllowed enum values: reduce
default: reduce
Option 16
\nobject
The throttle processor limits the number of events that pass through over a given time window.
group_by
\n[string]
Optional list of fields used to group events before the threshold has been reached.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
threshold [required]
\nint64
the number of events allowed in a given time window. Events sent after the threshold has been reached, are dropped.
type [required]
\nenum
The processor type. The value should always be throttle. \nAllowed enum values: throttle
default: throttle
window [required]
\ndouble
The time window in seconds over which the threshold applies.
sources [required]
\n[ <oneOf>]
A list of configured data sources for the pipeline.
Option 1
\nobject
The kafka source ingests data from Apache Kafka topics.
group_id [required]
\nstring
Consumer group ID used by the Kafka client.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
librdkafka_options
\n[object]
Optional list of advanced Kafka client configuration options, defined as key-value pairs.
name [required]
\nstring
The name of the librdkafka configuration option to set.
value [required]
\nstring
The value assigned to the specified librdkafka configuration option.
sasl
\nobject
Specifies the SASL mechanism for authenticating with a Kafka cluster.
mechanism
\nenum
SASL mechanism used for Kafka authentication. \nAllowed enum values: PLAIN,SCRAM-SHA-256,SCRAM-SHA-512
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
topics [required]
\n[string]
A list of Kafka topic names to subscribe to. The source ingests messages from each topic specified.
type [required]
\nenum
The source type. The value should always be kafka. \nAllowed enum values: kafka
default: kafka
Option 2
\nobject
The datadog_agent source collects logs from the Datadog Agent.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be datadog_agent. \nAllowed enum values: datadog_agent
default: datadog_agent
Option 3
\nobject
The splunk_tcp source receives logs from a Splunk Universal Forwarder over TCP.\nTLS is supported for secure transmission.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always splunk_tcp. \nAllowed enum values: splunk_tcp
default: splunk_tcp
Option 4
\nobject
The splunk_hec source implements the Splunk HTTP Event Collector (HEC) API.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always splunk_hec. \nAllowed enum values: splunk_hec
default: splunk_hec
Option 5
\nobject
The amazon_s3 source ingests logs from an Amazon S3 bucket.\nIt supports AWS authentication and TLS encryption.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
region [required]
\nstring
AWS region where the S3 bucket resides.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always amazon_s3. \nAllowed enum values: amazon_s3
default: amazon_s3
Option 6
\nobject
The fluentd source ingests logs from a Fluentd-compatible service.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be `fluentd. \nAllowed enum values: fluentd
default: fluentd
Option 7
\nobject
The fluent_bit source ingests logs from Fluent Bit.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be fluent_bit. \nAllowed enum values: fluent_bit
default: fluent_bit
Option 8
\nobject
The http_server source collects logs over HTTP POST from external services.
auth_strategy [required]
\nenum
HTTP authentication method. \nAllowed enum values: none,plain
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
Unique ID for the HTTP server source.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be http_server. \nAllowed enum values: http_server
default: http_server
Option 9
\nobject
The sumo_logic source receives logs from Sumo Logic collectors.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
type [required]
\nenum
The source type. The value should always be sumo_logic. \nAllowed enum values: sumo_logic
default: sumo_logic
Option 10
\nobject
The rsyslog source listens for logs over TCP or UDP from an rsyslog server using the syslog protocol.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
mode [required]
\nenum
Protocol used by the syslog source to receive messages. \nAllowed enum values: tcp,udp
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be rsyslog. \nAllowed enum values: rsyslog
default: rsyslog
Option 11
\nobject
The syslog_ng source listens for logs over TCP or UDP from a syslog-ng server using the syslog protocol.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
mode [required]
\nenum
Protocol used by the syslog source to receive messages. \nAllowed enum values: tcp,udp
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be syslog_ng. \nAllowed enum values: syslog_ng
default: syslog_ng
Option 12
\nobject
The amazon_data_firehose source ingests logs from AWS Data Firehose.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be amazon_data_firehose. \nAllowed enum values: amazon_data_firehose
default: amazon_data_firehose
Option 13
\nobject
The google_pubsub source ingests logs from a Google Cloud Pub/Sub subscription.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
project [required]
\nstring
The GCP project ID that owns the Pub/Sub subscription.
subscription [required]
\nstring
The Pub/Sub subscription name from which messages are consumed.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be google_pubsub. \nAllowed enum values: google_pubsub
default: google_pubsub
Option 14
\nobject
The http_client source scrapes logs from HTTP endpoints at regular intervals.
auth_strategy
\nenum
Optional authentication strategy for HTTP requests. \nAllowed enum values: basic,bearer
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
scrape_interval_secs
\nint64
The interval (in seconds) between HTTP scrape requests.
scrape_timeout_secs
\nint64
The timeout (in seconds) for each scrape request.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be http_client. \nAllowed enum values: http_client
default: http_client
Option 15
\nobject
The logstash source ingests logs from a Logstash forwarder.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be logstash. \nAllowed enum values: logstash
default: logstash
name [required]
\nstring
Name of the pipeline.
id [required]
\nstring
Unique identifier for the pipeline.
type [required]
\nstring
The resource type identifier. For pipeline resources, this should always be set to pipelines.
default: pipelines
data [required]
\nobject
Contains the pipeline’s ID, type, and configuration attributes.
attributes [required]
\nobject
Defines the pipeline’s name and its components (sources, processors, and destinations).
config [required]
\nobject
Specifies the pipeline's configuration, including its sources, processors, and destinations.
destinations [required]
\n[ <oneOf>]
A list of destination components where processed logs are sent.
Option 1
\nobject
The datadog_logs destination forwards logs to Datadog Log Management.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be datadog_logs. \nAllowed enum values: datadog_logs
default: datadog_logs
Option 2
\nobject
The amazon_s3 destination sends your logs in Datadog-rehydratable format to an Amazon S3 bucket for archiving.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
bucket [required]
\nstring
S3 bucket name.
id [required]
\nstring
Unique identifier for the destination component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
key_prefix
\nstring
Optional prefix for object keys.
region [required]
\nstring
AWS region of the S3 bucket.
storage_class [required]
\nenum
S3 storage class. \nAllowed enum values: STANDARD,REDUCED_REDUNDANCY,INTELLIGENT_TIERING,STANDARD_IA,EXPRESS_ONEZONE,ONEZONE_IA,GLACIER,GLACIER_IR,DEEP_ARCHIVE
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. Always amazon_s3. \nAllowed enum values: amazon_s3
default: amazon_s3
Option 3
\nobject
The google_cloud_storage destination stores logs in a Google Cloud Storage (GCS) bucket.\nIt requires a bucket name, GCP authentication, and metadata fields.
acl [required]
\nenum
Access control list setting for objects written to the bucket. \nAllowed enum values: private,project-private,public-read,authenticated-read,bucket-owner-read,bucket-owner-full-control
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
bucket [required]
\nstring
Name of the GCS bucket.
id [required]
\nstring
Unique identifier for the destination component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
key_prefix
\nstring
Optional prefix for object keys within the GCS bucket.
metadata
\n[object]
Custom metadata to attach to each object uploaded to the GCS bucket.
name [required]
\nstring
The metadata key.
value [required]
\nstring
The metadata value.
storage_class [required]
\nenum
Storage class used for objects stored in GCS. \nAllowed enum values: STANDARD,NEARLINE,COLDLINE,ARCHIVE
type [required]
\nenum
The destination type. Always google_cloud_storage. \nAllowed enum values: google_cloud_storage
default: google_cloud_storage
Option 4
\nobject
The splunk_hec destination forwards logs to Splunk using the HTTP Event Collector (HEC).
auto_extract_timestamp
\nboolean
If true, Splunk tries to extract timestamps from incoming log events.\nIf false, Splunk assigns the time the event was received.
encoding
\nenum
Encoding format for log events. \nAllowed enum values: json,raw_message
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
index
\nstring
Optional name of the Splunk index where logs are written.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
sourcetype
\nstring
The Splunk sourcetype to assign to log events.
type [required]
\nenum
The destination type. Always splunk_hec. \nAllowed enum values: splunk_hec
default: splunk_hec
Option 5
\nobject
The sumo_logic destination forwards logs to Sumo Logic.
encoding
\nenum
The output encoding format. \nAllowed enum values: json,raw_message,logfmt
header_custom_fields
\n[object]
A list of custom headers to include in the request to Sumo Logic.
name [required]
\nstring
The header field name.
value [required]
\nstring
The header field value.
header_host_name
\nstring
Optional override for the host name header.
header_source_category
\nstring
Optional override for the source category header.
header_source_name
\nstring
Optional override for the source name header.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be sumo_logic. \nAllowed enum values: sumo_logic
default: sumo_logic
Option 6
\nobject
The elasticsearch destination writes logs to an Elasticsearch cluster.
api_version
\nenum
The Elasticsearch API version to use. Set to auto to auto-detect. \nAllowed enum values: auto,v6,v7,v8
bulk_index
\nstring
The index to write logs to in Elasticsearch.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be elasticsearch. \nAllowed enum values: elasticsearch
default: elasticsearch
Option 7
\nobject
The rsyslog destination forwards logs to an external rsyslog server over TCP or UDP using the syslog protocol.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
keepalive
\nint64
Optional socket keepalive duration in milliseconds.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. The value should always be rsyslog. \nAllowed enum values: rsyslog
default: rsyslog
Option 8
\nobject
The syslog_ng destination forwards logs to an external syslog-ng server over TCP or UDP using the syslog protocol.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
keepalive
\nint64
Optional socket keepalive duration in milliseconds.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. The value should always be syslog_ng. \nAllowed enum values: syslog_ng
default: syslog_ng
Option 9
\nobject
The azure_storage destination forwards logs to an Azure Blob Storage container.
blob_prefix
\nstring
Optional prefix for blobs written to the container.
container_name [required]
\nstring
The name of the Azure Blob Storage container to store logs in.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be azure_storage. \nAllowed enum values: azure_storage
default: azure_storage
Option 10
\nobject
The microsoft_sentinel destination forwards logs to Microsoft Sentinel.
client_id [required]
\nstring
Azure AD client ID used for authentication.
dcr_immutable_id [required]
\nstring
The immutable ID of the Data Collection Rule (DCR).
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
table [required]
\nstring
The name of the Log Analytics table where logs are sent.
tenant_id [required]
\nstring
Azure AD tenant ID.
type [required]
\nenum
The destination type. The value should always be microsoft_sentinel. \nAllowed enum values: microsoft_sentinel
default: microsoft_sentinel
Option 11
\nobject
The google_chronicle destination sends logs to Google Chronicle.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
customer_id [required]
\nstring
The Google Chronicle customer ID.
encoding
\nenum
The encoding format for the logs sent to Chronicle. \nAllowed enum values: json,raw_message
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
log_type
\nstring
The log type metadata associated with the Chronicle destination.
type [required]
\nenum
The destination type. The value should always be google_chronicle. \nAllowed enum values: google_chronicle
default: google_chronicle
Option 12
\nobject
The new_relic destination sends logs to the New Relic platform.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The New Relic region. \nAllowed enum values: us,eu
type [required]
\nenum
The destination type. The value should always be new_relic. \nAllowed enum values: new_relic
default: new_relic
Option 13
\nobject
The sentinel_one destination sends logs to SentinelOne.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The SentinelOne region to send logs to. \nAllowed enum values: us,eu,ca,data_set_us
type [required]
\nenum
The destination type. The value should always be sentinel_one. \nAllowed enum values: sentinel_one
default: sentinel_one
Option 14
\nobject
The opensearch destination writes logs to an OpenSearch cluster.
bulk_index
\nstring
The index to write logs to.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be opensearch. \nAllowed enum values: opensearch
default: opensearch
Option 15
\nobject
The amazon_opensearch destination writes logs to Amazon OpenSearch.
auth [required]
\nobject
Authentication settings for the Amazon OpenSearch destination.\nThe strategy field determines whether basic or AWS-based authentication is used.
assume_role
\nstring
The ARN of the role to assume (used with aws strategy).
aws_region
\nstring
AWS region
external_id
\nstring
External ID for the assumed role (used with aws strategy).
session_name
\nstring
Session name for the assumed role (used with aws strategy).
strategy [required]
\nenum
The authentication strategy to use. \nAllowed enum values: basic,aws
bulk_index
\nstring
The index to write logs to.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be amazon_opensearch. \nAllowed enum values: amazon_opensearch
default: amazon_opensearch
processors
\n[ <oneOf>]
A list of processors that transform or enrich log data.
Option 1
\nobject
The filter processor allows conditional processing of logs based on a Datadog search query. Logs that match the include query are passed through; others are discarded.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs should pass through the filter. Logs that match this query continue to downstream components; others are dropped.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be filter. \nAllowed enum values: filter
default: filter
Option 2
\nobject
The parse_json processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string.
field [required]
\nstring
The name of the log field that contains a JSON string.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be parse_json. \nAllowed enum values: parse_json
default: parse_json
Option 3
\nobject
The Quota Processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert.
drop_events [required]
\nboolean
If set to true, logs that matched the quota filter and sent after the quota has been met are dropped; only logs that did not match the filter query continue through the pipeline.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
ignore_when_missing_partitions
\nboolean
If true, the processor skips quota checks when partition fields are missing from the logs.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
name [required]
\nstring
Name of the quota.
overflow_action
\nenum
The action to take when the quota is exceeded. Options:
\ndrop: Drop the event.
no_action: Let the event pass through.
overflow_routing: Route to an overflow destination.
Allowed enum values: drop,no_action,overflow_routing
overrides
\n[object]
A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit.
fields [required]
\n[object]
A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
partition_fields
\n[string]
A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values.
type [required]
\nenum
The processor type. The value should always be quota. \nAllowed enum values: quota
default: quota
Option 4
\nobject
The add_fields processor adds static key-value fields to logs.
fields [required]
\n[object]
A list of static fields (key-value pairs) that is added to each log event processed by this component.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be add_fields. \nAllowed enum values: add_fields
default: add_fields
Option 5
\nobject
The remove_fields processor deletes specified fields from logs.
fields [required]
\n[string]
A list of field names to be removed from each log event.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
The PipelineRemoveFieldsProcessor inputs.
type [required]
\nenum
The processor type. The value should always be remove_fields. \nAllowed enum values: remove_fields
default: remove_fields
Option 6
\nobject
The rename_fields processor changes field names.
fields [required]
\n[object]
A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields.
destination [required]
\nstring
The field name to assign the renamed value to.
preserve_source [required]
\nboolean
Indicates whether the original field, that is received from the source, should be kept (true) or removed (false) after renaming.
source [required]
\nstring
The original field name in the log event that should be renamed.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be rename_fields. \nAllowed enum values: rename_fields
default: rename_fields
Option 7
\nobject
The generate_datadog_metrics processor creates custom metrics from logs and sends them to Datadog.\nMetrics can be counters, gauges, or distributions and optionally grouped by log fields.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
metrics [required]
\n[object]
Configuration for generating individual metrics.
group_by
\n[string]
Optional fields used to group the metric series.
include [required]
\nstring
Datadog filter query to match logs for metric generation.
metric_type [required]
\nenum
Type of metric to create. \nAllowed enum values: count,gauge,distribution
name [required]
\nstring
Name of the custom metric to be created.
value [required]
\n<oneOf>
Specifies how the value of the generated metric is computed.
Option 1
\nobject
Strategy that increments a generated metric by one for each matching event.
strategy [required]
\nenum
Increments the metric by 1 for each matching event. \nAllowed enum values: increment_by_one
Option 2
\nobject
Strategy that increments a generated metric based on the value of a log field.
field [required]
\nstring
Name of the log field containing the numeric value to increment the metric by.
strategy [required]
\nenum
Uses a numeric field in the log event as the metric increment. \nAllowed enum values: increment_by_field
type [required]
\nenum
The processor type. Always generate_datadog_metrics. \nAllowed enum values: generate_datadog_metrics
default: generate_datadog_metrics
Option 8
\nobject
The sample processor allows probabilistic sampling of logs at a fixed rate.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
percentage
\ndouble
The percentage of logs to sample.
rate
\nint64
Number of events to sample (1 in N).
type [required]
\nenum
The processor type. The value should always be sample. \nAllowed enum values: sample
default: sample
Option 9
\nobject
The parse_grok processor extracts structured fields from unstructured log messages using Grok patterns.
disable_library_rules
\nboolean
If set to true, disables the default Grok rules provided by Datadog.
id [required]
\nstring
A unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
rules [required]
\n[object]
The list of Grok parsing rules. If multiple matching rules are provided, they are evaluated in order. The first successful match is applied.
match_rules [required]
\n[object]
A list of Grok parsing rules that define how to extract fields from the source field.\nEach rule must contain a name and a valid Grok pattern.
name [required]
\nstring
The name of the rule.
rule [required]
\nstring
The definition of the Grok rule.
source [required]
\nstring
The name of the field in the log event to apply the Grok rules to.
support_rules [required]
\n[object]
A list of Grok helper rules that can be referenced by the parsing rules.
name [required]
\nstring
The name of the Grok helper rule.
rule [required]
\nstring
The definition of the Grok helper rule.
type [required]
\nenum
The processor type. The value should always be parse_grok. \nAllowed enum values: parse_grok
default: parse_grok
Option 10
\nobject
The sensitive_data_scanner processor detects and optionally redacts sensitive data in log events.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
rules [required]
\n[object]
A list of rules for identifying and acting on sensitive data patterns.
keyword_options
\nobject
Configuration for keywords used to reinforce sensitive data pattern detection.
keywords [required]
\n[string]
A list of keywords to match near the sensitive pattern.
proximity [required]
\nint64
Maximum number of tokens between a keyword and a sensitive value match.
name [required]
\nstring
A name identifying the rule.
on_match [required]
\n<oneOf>
Defines what action to take when sensitive data is matched.
Option 1
\nobject
Configuration for completely redacting matched sensitive data.
action [required]
\nenum
Action type that completely replaces the matched sensitive data with a fixed replacement string to remove all visibility. \nAllowed enum values: redact
options [required]
\nobject
Configuration for fully redacting sensitive data.
replace [required]
\nstring
The ObservabilityPipelineSensitiveDataScannerProcessorActionRedactOptions replace.
Option 2
\nobject
Configuration for hashing matched sensitive values.
action [required]
\nenum
Action type that replaces the matched sensitive data with a hashed representation, preserving structure while securing content. \nAllowed enum values: hash
options
\nobject
The ObservabilityPipelineSensitiveDataScannerProcessorActionHash options.
Option 3
\nobject
Configuration for partially redacting matched sensitive data.
action [required]
\nenum
Action type that redacts part of the sensitive data while preserving a configurable number of characters, typically used for masking purposes (e.g., show last 4 digits of a credit card). \nAllowed enum values: partial_redact
options [required]
\nobject
Controls how partial redaction is applied, including character count and direction.
characters [required]
\nint64
The ObservabilityPipelineSensitiveDataScannerProcessorActionPartialRedactOptions characters.
direction [required]
\nenum
Indicates whether to redact characters from the first or last part of the matched value. \nAllowed enum values: first,last
pattern [required]
\n<oneOf>
Pattern detection configuration for identifying sensitive data using either a custom regex or a library reference.
Option 1
\nobject
Defines a custom regex-based pattern for identifying sensitive data in logs.
options [required]
\nobject
Options for defining a custom regex pattern.
rule [required]
\nstring
A regular expression used to detect sensitive values. Must be a valid regex.
type [required]
\nenum
Indicates a custom regular expression is used for matching. \nAllowed enum values: custom
Option 2
\nobject
Specifies a pattern from Datadog’s sensitive data detection library to match known sensitive data types.
options [required]
\nobject
Options for selecting a predefined library pattern and enabling keyword support.
id [required]
\nstring
Identifier for a predefined pattern from the sensitive data scanner pattern library.
use_recommended_keywords
\nboolean
Whether to augment the pattern with recommended keywords (optional).
type [required]
\nenum
Indicates that a predefined library pattern is used. \nAllowed enum values: library
scope [required]
\n<oneOf>
Determines which parts of the log the pattern-matching rule should be applied to.
Option 1
\nobject
Includes only specific fields for sensitive data scanning.
options [required]
\nobject
Fields to which the scope rule applies.
fields [required]
\n[string]
The ObservabilityPipelineSensitiveDataScannerProcessorScopeOptions fields.
target [required]
\nenum
Applies the rule only to included fields. \nAllowed enum values: include
Option 2
\nobject
Excludes specific fields from sensitive data scanning.
options [required]
\nobject
Fields to which the scope rule applies.
fields [required]
\n[string]
The ObservabilityPipelineSensitiveDataScannerProcessorScopeOptions fields.
target [required]
\nenum
Excludes specific fields from processing. \nAllowed enum values: exclude
Option 3
\nobject
Applies scanning across all available fields.
target [required]
\nenum
Applies the rule to all fields. \nAllowed enum values: all
tags [required]
\n[string]
Tags assigned to this rule for filtering and classification.
type [required]
\nenum
The processor type. The value should always be sensitive_data_scanner. \nAllowed enum values: sensitive_data_scanner
default: sensitive_data_scanner
Option 11
\nobject
The ocsf_mapper processor transforms logs into the OCSF schema using a predefined mapping configuration.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mappings [required]
\n[object]
A list of mapping rules to convert events to the OCSF format.
include [required]
\nstring
A Datadog search query used to select the logs that this mapping should apply to.
mapping [required]
\n<oneOf>
Defines a single mapping rule for transforming logs into the OCSF schema.
Option 1
\nenum
Predefined library mappings for common log formats. \nAllowed enum values: CloudTrail Account Change,GCP Cloud Audit CreateBucket,GCP Cloud Audit CreateSink,GCP Cloud Audit SetIamPolicy,GCP Cloud Audit UpdateSink,Github Audit Log API Activity,Google Workspace Admin Audit addPrivilege,Microsoft 365 Defender Incident,Microsoft 365 Defender UserLoggedIn,Okta System Log Authentication,Palo Alto Networks Firewall Traffic
type [required]
\nenum
The processor type. The value should always be ocsf_mapper. \nAllowed enum values: ocsf_mapper
default: ocsf_mapper
Option 12
\nobject
The add_env_vars processor adds environment variable values to log events.
id [required]
\nstring
The unique identifier for this component. Used to reference this processor in the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
type [required]
\nenum
The processor type. The value should always be add_env_vars. \nAllowed enum values: add_env_vars
default: add_env_vars
variables [required]
\n[object]
A list of environment variable mappings to apply to log fields.
field [required]
\nstring
The target field in the log event.
name [required]
\nstring
The name of the environment variable to read.
Option 13
\nobject
The dedupe processor removes duplicate fields in log events.
fields [required]
\n[string]
A list of log field paths to check for duplicates.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mode [required]
\nenum
The deduplication mode to apply to the fields. \nAllowed enum values: match,ignore
type [required]
\nenum
The processor type. The value should always be dedupe. \nAllowed enum values: dedupe
default: dedupe
Option 14
\nobject
The enrichment_table processor enriches logs using a static CSV file or GeoIP database.
file
\nobject
Defines a static enrichment table loaded from a CSV file.
encoding [required]
\nobject
File encoding format.
delimiter [required]
\nstring
The encoding delimiter.
includes_headers [required]
\nboolean
The encoding includes_headers.
type [required]
\nenum
Specifies the encoding format (e.g., CSV) used for enrichment tables. \nAllowed enum values: csv
key [required]
\n[object]
Key fields used to look up enrichment values.
column [required]
\nstring
The items column.
comparison [required]
\nenum
Defines how to compare key fields for enrichment table lookups. \nAllowed enum values: equals
field [required]
\nstring
The items field.
path [required]
\nstring
Path to the CSV file.
schema [required]
\n[object]
Schema defining column names and their types.
column [required]
\nstring
The items column.
type [required]
\nenum
Declares allowed data types for enrichment table columns. \nAllowed enum values: string,boolean,integer,float,date,timestamp
geoip
\nobject
Uses a GeoIP database to enrich logs based on an IP field.
key_field [required]
\nstring
Path to the IP field in the log.
locale [required]
\nstring
Locale used to resolve geographical names.
path [required]
\nstring
Path to the GeoIP database file.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
target [required]
\nstring
Path where enrichment results should be stored in the log.
type [required]
\nenum
The processor type. The value should always be enrichment_table. \nAllowed enum values: enrichment_table
default: enrichment_table
Option 15
\nobject
The reduce processor aggregates and merges logs based on matching keys and merge strategies.
group_by [required]
\n[string]
A list of fields used to group log events for merging.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
merge_strategies [required]
\n[object]
List of merge strategies defining how values from grouped events should be combined.
path [required]
\nstring
The field path in the log event.
strategy [required]
\nenum
The merge strategy to apply. \nAllowed enum values: discard,retain,sum,max,min,array,concat,concat_newline,concat_raw,shortest_array,longest_array,flat_unique
type [required]
\nenum
The processor type. The value should always be reduce. \nAllowed enum values: reduce
default: reduce
Option 16
\nobject
The throttle processor limits the number of events that pass through over a given time window.
group_by
\n[string]
Optional list of fields used to group events before the threshold has been reached.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
threshold [required]
\nint64
the number of events allowed in a given time window. Events sent after the threshold has been reached, are dropped.
type [required]
\nenum
The processor type. The value should always be throttle. \nAllowed enum values: throttle
default: throttle
window [required]
\ndouble
The time window in seconds over which the threshold applies.
Option 17
\nobject
The datadog_tags processor includes or excludes specific Datadog tags in your logs.
action [required]
\nenum
The action to take on tags with matching keys. \nAllowed enum values: include,exclude
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
keys [required]
\n[string]
A list of tag keys.
mode [required]
\nenum
The processing mode. \nAllowed enum values: filter
type [required]
\nenum
The processor type. The value should always be datadog_tags. \nAllowed enum values: datadog_tags
default: datadog_tags
sources [required]
\n[ <oneOf>]
A list of configured data sources for the pipeline.
Option 1
\nobject
The kafka source ingests data from Apache Kafka topics.
group_id [required]
\nstring
Consumer group ID used by the Kafka client.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
librdkafka_options
\n[object]
Optional list of advanced Kafka client configuration options, defined as key-value pairs.
name [required]
\nstring
The name of the librdkafka configuration option to set.
value [required]
\nstring
The value assigned to the specified librdkafka configuration option.
sasl
\nobject
Specifies the SASL mechanism for authenticating with a Kafka cluster.
mechanism
\nenum
SASL mechanism used for Kafka authentication. \nAllowed enum values: PLAIN,SCRAM-SHA-256,SCRAM-SHA-512
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
topics [required]
\n[string]
A list of Kafka topic names to subscribe to. The source ingests messages from each topic specified.
type [required]
\nenum
The source type. The value should always be kafka. \nAllowed enum values: kafka
default: kafka
Option 2
\nobject
The datadog_agent source collects logs from the Datadog Agent.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be datadog_agent. \nAllowed enum values: datadog_agent
default: datadog_agent
Option 3
\nobject
The splunk_tcp source receives logs from a Splunk Universal Forwarder over TCP.\nTLS is supported for secure transmission.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always splunk_tcp. \nAllowed enum values: splunk_tcp
default: splunk_tcp
Option 4
\nobject
The splunk_hec source implements the Splunk HTTP Event Collector (HEC) API.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always splunk_hec. \nAllowed enum values: splunk_hec
default: splunk_hec
Option 5
\nobject
The amazon_s3 source ingests logs from an Amazon S3 bucket.\nIt supports AWS authentication and TLS encryption.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
region [required]
\nstring
AWS region where the S3 bucket resides.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always amazon_s3. \nAllowed enum values: amazon_s3
default: amazon_s3
Option 6
\nobject
The fluentd source ingests logs from a Fluentd-compatible service.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be `fluentd. \nAllowed enum values: fluentd
default: fluentd
Option 7
\nobject
The fluent_bit source ingests logs from Fluent Bit.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be fluent_bit. \nAllowed enum values: fluent_bit
default: fluent_bit
Option 8
\nobject
The http_server source collects logs over HTTP POST from external services.
auth_strategy [required]
\nenum
HTTP authentication method. \nAllowed enum values: none,plain
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
Unique ID for the HTTP server source.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be http_server. \nAllowed enum values: http_server
default: http_server
Option 9
\nobject
The sumo_logic source receives logs from Sumo Logic collectors.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
type [required]
\nenum
The source type. The value should always be sumo_logic. \nAllowed enum values: sumo_logic
default: sumo_logic
Option 10
\nobject
The rsyslog source listens for logs over TCP or UDP from an rsyslog server using the syslog protocol.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
mode [required]
\nenum
Protocol used by the syslog source to receive messages. \nAllowed enum values: tcp,udp
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be rsyslog. \nAllowed enum values: rsyslog
default: rsyslog
Option 11
\nobject
The syslog_ng source listens for logs over TCP or UDP from a syslog-ng server using the syslog protocol.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
mode [required]
\nenum
Protocol used by the syslog source to receive messages. \nAllowed enum values: tcp,udp
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be syslog_ng. \nAllowed enum values: syslog_ng
default: syslog_ng
Option 12
\nobject
The amazon_data_firehose source ingests logs from AWS Data Firehose.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be amazon_data_firehose. \nAllowed enum values: amazon_data_firehose
default: amazon_data_firehose
Option 13
\nobject
The google_pubsub source ingests logs from a Google Cloud Pub/Sub subscription.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
project [required]
\nstring
The GCP project ID that owns the Pub/Sub subscription.
subscription [required]
\nstring
The Pub/Sub subscription name from which messages are consumed.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be google_pubsub. \nAllowed enum values: google_pubsub
default: google_pubsub
Option 14
\nobject
The http_client source scrapes logs from HTTP endpoints at regular intervals.
auth_strategy
\nenum
Optional authentication strategy for HTTP requests. \nAllowed enum values: basic,bearer
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
scrape_interval_secs
\nint64
The interval (in seconds) between HTTP scrape requests.
scrape_timeout_secs
\nint64
The timeout (in seconds) for each scrape request.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be http_client. \nAllowed enum values: http_client
default: http_client
Option 15
\nobject
The logstash source ingests logs from a Logstash forwarder.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be logstash. \nAllowed enum values: logstash
default: logstash
name [required]
\nstring
Name of the pipeline.
id [required]
\nstring
Unique identifier for the pipeline.
type [required]
\nstring
The resource type identifier. For pipeline resources, this should always be set to pipelines.
default: pipelines
data [required]
\nobject
Contains the pipeline’s ID, type, and configuration attributes.
attributes [required]
\nobject
Defines the pipeline’s name and its components (sources, processors, and destinations).
config [required]
\nobject
Specifies the pipeline's configuration, including its sources, processors, and destinations.
destinations [required]
\n[ <oneOf>]
A list of destination components where processed logs are sent.
Option 1
\nobject
The datadog_logs destination forwards logs to Datadog Log Management.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be datadog_logs. \nAllowed enum values: datadog_logs
default: datadog_logs
Option 2
\nobject
The amazon_s3 destination sends your logs in Datadog-rehydratable format to an Amazon S3 bucket for archiving.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
bucket [required]
\nstring
S3 bucket name.
id [required]
\nstring
Unique identifier for the destination component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
key_prefix
\nstring
Optional prefix for object keys.
region [required]
\nstring
AWS region of the S3 bucket.
storage_class [required]
\nenum
S3 storage class. \nAllowed enum values: STANDARD,REDUCED_REDUNDANCY,INTELLIGENT_TIERING,STANDARD_IA,EXPRESS_ONEZONE,ONEZONE_IA,GLACIER,GLACIER_IR,DEEP_ARCHIVE
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. Always amazon_s3. \nAllowed enum values: amazon_s3
default: amazon_s3
Option 3
\nobject
The google_cloud_storage destination stores logs in a Google Cloud Storage (GCS) bucket.\nIt requires a bucket name, GCP authentication, and metadata fields.
acl [required]
\nenum
Access control list setting for objects written to the bucket. \nAllowed enum values: private,project-private,public-read,authenticated-read,bucket-owner-read,bucket-owner-full-control
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
bucket [required]
\nstring
Name of the GCS bucket.
id [required]
\nstring
Unique identifier for the destination component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
key_prefix
\nstring
Optional prefix for object keys within the GCS bucket.
metadata
\n[object]
Custom metadata to attach to each object uploaded to the GCS bucket.
name [required]
\nstring
The metadata key.
value [required]
\nstring
The metadata value.
storage_class [required]
\nenum
Storage class used for objects stored in GCS. \nAllowed enum values: STANDARD,NEARLINE,COLDLINE,ARCHIVE
type [required]
\nenum
The destination type. Always google_cloud_storage. \nAllowed enum values: google_cloud_storage
default: google_cloud_storage
Option 4
\nobject
The splunk_hec destination forwards logs to Splunk using the HTTP Event Collector (HEC).
auto_extract_timestamp
\nboolean
If true, Splunk tries to extract timestamps from incoming log events.\nIf false, Splunk assigns the time the event was received.
encoding
\nenum
Encoding format for log events. \nAllowed enum values: json,raw_message
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
index
\nstring
Optional name of the Splunk index where logs are written.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
sourcetype
\nstring
The Splunk sourcetype to assign to log events.
type [required]
\nenum
The destination type. Always splunk_hec. \nAllowed enum values: splunk_hec
default: splunk_hec
Option 5
\nobject
The sumo_logic destination forwards logs to Sumo Logic.
encoding
\nenum
The output encoding format. \nAllowed enum values: json,raw_message,logfmt
header_custom_fields
\n[object]
A list of custom headers to include in the request to Sumo Logic.
name [required]
\nstring
The header field name.
value [required]
\nstring
The header field value.
header_host_name
\nstring
Optional override for the host name header.
header_source_category
\nstring
Optional override for the source category header.
header_source_name
\nstring
Optional override for the source name header.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be sumo_logic. \nAllowed enum values: sumo_logic
default: sumo_logic
Option 6
\nobject
The elasticsearch destination writes logs to an Elasticsearch cluster.
api_version
\nenum
The Elasticsearch API version to use. Set to auto to auto-detect. \nAllowed enum values: auto,v6,v7,v8
bulk_index
\nstring
The index to write logs to in Elasticsearch.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be elasticsearch. \nAllowed enum values: elasticsearch
default: elasticsearch
Option 7
\nobject
The rsyslog destination forwards logs to an external rsyslog server over TCP or UDP using the syslog protocol.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
keepalive
\nint64
Optional socket keepalive duration in milliseconds.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. The value should always be rsyslog. \nAllowed enum values: rsyslog
default: rsyslog
Option 8
\nobject
The syslog_ng destination forwards logs to an external syslog-ng server over TCP or UDP using the syslog protocol.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
keepalive
\nint64
Optional socket keepalive duration in milliseconds.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. The value should always be syslog_ng. \nAllowed enum values: syslog_ng
default: syslog_ng
Option 9
\nobject
The azure_storage destination forwards logs to an Azure Blob Storage container.
blob_prefix
\nstring
Optional prefix for blobs written to the container.
container_name [required]
\nstring
The name of the Azure Blob Storage container to store logs in.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be azure_storage. \nAllowed enum values: azure_storage
default: azure_storage
Option 10
\nobject
The microsoft_sentinel destination forwards logs to Microsoft Sentinel.
client_id [required]
\nstring
Azure AD client ID used for authentication.
dcr_immutable_id [required]
\nstring
The immutable ID of the Data Collection Rule (DCR).
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
table [required]
\nstring
The name of the Log Analytics table where logs are sent.
tenant_id [required]
\nstring
Azure AD tenant ID.
type [required]
\nenum
The destination type. The value should always be microsoft_sentinel. \nAllowed enum values: microsoft_sentinel
default: microsoft_sentinel
Option 11
\nobject
The google_chronicle destination sends logs to Google Chronicle.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
customer_id [required]
\nstring
The Google Chronicle customer ID.
encoding
\nenum
The encoding format for the logs sent to Chronicle. \nAllowed enum values: json,raw_message
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
log_type
\nstring
The log type metadata associated with the Chronicle destination.
type [required]
\nenum
The destination type. The value should always be google_chronicle. \nAllowed enum values: google_chronicle
default: google_chronicle
Option 12
\nobject
The new_relic destination sends logs to the New Relic platform.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The New Relic region. \nAllowed enum values: us,eu
type [required]
\nenum
The destination type. The value should always be new_relic. \nAllowed enum values: new_relic
default: new_relic
Option 13
\nobject
The sentinel_one destination sends logs to SentinelOne.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The SentinelOne region to send logs to. \nAllowed enum values: us,eu,ca,data_set_us
type [required]
\nenum
The destination type. The value should always be sentinel_one. \nAllowed enum values: sentinel_one
default: sentinel_one
Option 14
\nobject
The opensearch destination writes logs to an OpenSearch cluster.
bulk_index
\nstring
The index to write logs to.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be opensearch. \nAllowed enum values: opensearch
default: opensearch
Option 15
\nobject
The amazon_opensearch destination writes logs to Amazon OpenSearch.
auth [required]
\nobject
Authentication settings for the Amazon OpenSearch destination.\nThe strategy field determines whether basic or AWS-based authentication is used.
assume_role
\nstring
The ARN of the role to assume (used with aws strategy).
aws_region
\nstring
AWS region
external_id
\nstring
External ID for the assumed role (used with aws strategy).
session_name
\nstring
Session name for the assumed role (used with aws strategy).
strategy [required]
\nenum
The authentication strategy to use. \nAllowed enum values: basic,aws
bulk_index
\nstring
The index to write logs to.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be amazon_opensearch. \nAllowed enum values: amazon_opensearch
default: amazon_opensearch
processors
\n[ <oneOf>]
A list of processors that transform or enrich log data.
Option 1
\nobject
The filter processor allows conditional processing of logs based on a Datadog search query. Logs that match the include query are passed through; others are discarded.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs should pass through the filter. Logs that match this query continue to downstream components; others are dropped.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be filter. \nAllowed enum values: filter
default: filter
Option 2
\nobject
The parse_json processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string.
field [required]
\nstring
The name of the log field that contains a JSON string.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be parse_json. \nAllowed enum values: parse_json
default: parse_json
Option 3
\nobject
The Quota Processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert.
drop_events [required]
\nboolean
If set to true, logs that matched the quota filter and sent after the quota has been met are dropped; only logs that did not match the filter query continue through the pipeline.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
ignore_when_missing_partitions
\nboolean
If true, the processor skips quota checks when partition fields are missing from the logs.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
name [required]
\nstring
Name of the quota.
overflow_action
\nenum
The action to take when the quota is exceeded. Options:
\ndrop: Drop the event.
no_action: Let the event pass through.
overflow_routing: Route to an overflow destination.
Allowed enum values: drop,no_action,overflow_routing
overrides
\n[object]
A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit.
fields [required]
\n[object]
A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
partition_fields
\n[string]
A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values.
type [required]
\nenum
The processor type. The value should always be quota. \nAllowed enum values: quota
default: quota
Option 4
\nobject
The add_fields processor adds static key-value fields to logs.
fields [required]
\n[object]
A list of static fields (key-value pairs) that is added to each log event processed by this component.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be add_fields. \nAllowed enum values: add_fields
default: add_fields
Option 5
\nobject
The remove_fields processor deletes specified fields from logs.
fields [required]
\n[string]
A list of field names to be removed from each log event.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
The PipelineRemoveFieldsProcessor inputs.
type [required]
\nenum
The processor type. The value should always be remove_fields. \nAllowed enum values: remove_fields
default: remove_fields
Option 6
\nobject
The rename_fields processor changes field names.
fields [required]
\n[object]
A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields.
destination [required]
\nstring
The field name to assign the renamed value to.
preserve_source [required]
\nboolean
Indicates whether the original field, that is received from the source, should be kept (true) or removed (false) after renaming.
source [required]
\nstring
The original field name in the log event that should be renamed.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be rename_fields. \nAllowed enum values: rename_fields
default: rename_fields
Option 7
\nobject
The generate_datadog_metrics processor creates custom metrics from logs and sends them to Datadog.\nMetrics can be counters, gauges, or distributions and optionally grouped by log fields.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
metrics [required]
\n[object]
Configuration for generating individual metrics.
group_by
\n[string]
Optional fields used to group the metric series.
include [required]
\nstring
Datadog filter query to match logs for metric generation.
metric_type [required]
\nenum
Type of metric to create. \nAllowed enum values: count,gauge,distribution
name [required]
\nstring
Name of the custom metric to be created.
value [required]
\n<oneOf>
Specifies how the value of the generated metric is computed.
Option 1
\nobject
Strategy that increments a generated metric by one for each matching event.
strategy [required]
\nenum
Increments the metric by 1 for each matching event. \nAllowed enum values: increment_by_one
Option 2
\nobject
Strategy that increments a generated metric based on the value of a log field.
field [required]
\nstring
Name of the log field containing the numeric value to increment the metric by.
strategy [required]
\nenum
Uses a numeric field in the log event as the metric increment. \nAllowed enum values: increment_by_field
type [required]
\nenum
The processor type. Always generate_datadog_metrics. \nAllowed enum values: generate_datadog_metrics
default: generate_datadog_metrics
Option 8
\nobject
The sample processor allows probabilistic sampling of logs at a fixed rate.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
percentage
\ndouble
The percentage of logs to sample.
rate
\nint64
Number of events to sample (1 in N).
type [required]
\nenum
The processor type. The value should always be sample. \nAllowed enum values: sample
default: sample
Option 9
\nobject
The parse_grok processor extracts structured fields from unstructured log messages using Grok patterns.
disable_library_rules
\nboolean
If set to true, disables the default Grok rules provided by Datadog.
id [required]
\nstring
A unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
rules [required]
\n[object]
The list of Grok parsing rules. If multiple matching rules are provided, they are evaluated in order. The first successful match is applied.
match_rules [required]
\n[object]
A list of Grok parsing rules that define how to extract fields from the source field.\nEach rule must contain a name and a valid Grok pattern.
name [required]
\nstring
The name of the rule.
rule [required]
\nstring
The definition of the Grok rule.
source [required]
\nstring
The name of the field in the log event to apply the Grok rules to.
support_rules [required]
\n[object]
A list of Grok helper rules that can be referenced by the parsing rules.
name [required]
\nstring
The name of the Grok helper rule.
rule [required]
\nstring
The definition of the Grok helper rule.
type [required]
\nenum
The processor type. The value should always be parse_grok. \nAllowed enum values: parse_grok
default: parse_grok
Option 10
\nobject
The sensitive_data_scanner processor detects and optionally redacts sensitive data in log events.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
rules [required]
\n[object]
A list of rules for identifying and acting on sensitive data patterns.
keyword_options
\nobject
Configuration for keywords used to reinforce sensitive data pattern detection.
keywords [required]
\n[string]
A list of keywords to match near the sensitive pattern.
proximity [required]
\nint64
Maximum number of tokens between a keyword and a sensitive value match.
name [required]
\nstring
A name identifying the rule.
on_match [required]
\n<oneOf>
Defines what action to take when sensitive data is matched.
Option 1
\nobject
Configuration for completely redacting matched sensitive data.
action [required]
\nenum
Action type that completely replaces the matched sensitive data with a fixed replacement string to remove all visibility. \nAllowed enum values: redact
options [required]
\nobject
Configuration for fully redacting sensitive data.
replace [required]
\nstring
The ObservabilityPipelineSensitiveDataScannerProcessorActionRedactOptions replace.
Option 2
\nobject
Configuration for hashing matched sensitive values.
action [required]
\nenum
Action type that replaces the matched sensitive data with a hashed representation, preserving structure while securing content. \nAllowed enum values: hash
options
\nobject
The ObservabilityPipelineSensitiveDataScannerProcessorActionHash options.
Option 3
\nobject
Configuration for partially redacting matched sensitive data.
action [required]
\nenum
Action type that redacts part of the sensitive data while preserving a configurable number of characters, typically used for masking purposes (e.g., show last 4 digits of a credit card). \nAllowed enum values: partial_redact
options [required]
\nobject
Controls how partial redaction is applied, including character count and direction.
characters [required]
\nint64
The ObservabilityPipelineSensitiveDataScannerProcessorActionPartialRedactOptions characters.
direction [required]
\nenum
Indicates whether to redact characters from the first or last part of the matched value. \nAllowed enum values: first,last
pattern [required]
\n<oneOf>
Pattern detection configuration for identifying sensitive data using either a custom regex or a library reference.
Option 1
\nobject
Defines a custom regex-based pattern for identifying sensitive data in logs.
options [required]
\nobject
Options for defining a custom regex pattern.
rule [required]
\nstring
A regular expression used to detect sensitive values. Must be a valid regex.
type [required]
\nenum
Indicates a custom regular expression is used for matching. \nAllowed enum values: custom
Option 2
\nobject
Specifies a pattern from Datadog’s sensitive data detection library to match known sensitive data types.
options [required]
\nobject
Options for selecting a predefined library pattern and enabling keyword support.
id [required]
\nstring
Identifier for a predefined pattern from the sensitive data scanner pattern library.
use_recommended_keywords
\nboolean
Whether to augment the pattern with recommended keywords (optional).
type [required]
\nenum
Indicates that a predefined library pattern is used. \nAllowed enum values: library
scope [required]
\n<oneOf>
Determines which parts of the log the pattern-matching rule should be applied to.
Option 1
\nobject
Includes only specific fields for sensitive data scanning.
options [required]
\nobject
Fields to which the scope rule applies.
fields [required]
\n[string]
The ObservabilityPipelineSensitiveDataScannerProcessorScopeOptions fields.
target [required]
\nenum
Applies the rule only to included fields. \nAllowed enum values: include
Option 2
\nobject
Excludes specific fields from sensitive data scanning.
options [required]
\nobject
Fields to which the scope rule applies.
fields [required]
\n[string]
The ObservabilityPipelineSensitiveDataScannerProcessorScopeOptions fields.
target [required]
\nenum
Excludes specific fields from processing. \nAllowed enum values: exclude
Option 3
\nobject
Applies scanning across all available fields.
target [required]
\nenum
Applies the rule to all fields. \nAllowed enum values: all
tags [required]
\n[string]
Tags assigned to this rule for filtering and classification.
type [required]
\nenum
The processor type. The value should always be sensitive_data_scanner. \nAllowed enum values: sensitive_data_scanner
default: sensitive_data_scanner
Option 11
\nobject
The ocsf_mapper processor transforms logs into the OCSF schema using a predefined mapping configuration.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mappings [required]
\n[object]
A list of mapping rules to convert events to the OCSF format.
include [required]
\nstring
A Datadog search query used to select the logs that this mapping should apply to.
mapping [required]
\n<oneOf>
Defines a single mapping rule for transforming logs into the OCSF schema.
Option 1
\nenum
Predefined library mappings for common log formats. \nAllowed enum values: CloudTrail Account Change,GCP Cloud Audit CreateBucket,GCP Cloud Audit CreateSink,GCP Cloud Audit SetIamPolicy,GCP Cloud Audit UpdateSink,Github Audit Log API Activity,Google Workspace Admin Audit addPrivilege,Microsoft 365 Defender Incident,Microsoft 365 Defender UserLoggedIn,Okta System Log Authentication,Palo Alto Networks Firewall Traffic
type [required]
\nenum
The processor type. The value should always be ocsf_mapper. \nAllowed enum values: ocsf_mapper
default: ocsf_mapper
Option 12
\nobject
The add_env_vars processor adds environment variable values to log events.
id [required]
\nstring
The unique identifier for this component. Used to reference this processor in the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
type [required]
\nenum
The processor type. The value should always be add_env_vars. \nAllowed enum values: add_env_vars
default: add_env_vars
variables [required]
\n[object]
A list of environment variable mappings to apply to log fields.
field [required]
\nstring
The target field in the log event.
name [required]
\nstring
The name of the environment variable to read.
Option 13
\nobject
The dedupe processor removes duplicate fields in log events.
fields [required]
\n[string]
A list of log field paths to check for duplicates.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mode [required]
\nenum
The deduplication mode to apply to the fields. \nAllowed enum values: match,ignore
type [required]
\nenum
The processor type. The value should always be dedupe. \nAllowed enum values: dedupe
default: dedupe
Option 14
\nobject
The enrichment_table processor enriches logs using a static CSV file or GeoIP database.
file
\nobject
Defines a static enrichment table loaded from a CSV file.
encoding [required]
\nobject
File encoding format.
delimiter [required]
\nstring
The encoding delimiter.
includes_headers [required]
\nboolean
The encoding includes_headers.
type [required]
\nenum
Specifies the encoding format (e.g., CSV) used for enrichment tables. \nAllowed enum values: csv
key [required]
\n[object]
Key fields used to look up enrichment values.
column [required]
\nstring
The items column.
comparison [required]
\nenum
Defines how to compare key fields for enrichment table lookups. \nAllowed enum values: equals
field [required]
\nstring
The items field.
path [required]
\nstring
Path to the CSV file.
schema [required]
\n[object]
Schema defining column names and their types.
column [required]
\nstring
The items column.
type [required]
\nenum
Declares allowed data types for enrichment table columns. \nAllowed enum values: string,boolean,integer,float,date,timestamp
geoip
\nobject
Uses a GeoIP database to enrich logs based on an IP field.
key_field [required]
\nstring
Path to the IP field in the log.
locale [required]
\nstring
Locale used to resolve geographical names.
path [required]
\nstring
Path to the GeoIP database file.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
target [required]
\nstring
Path where enrichment results should be stored in the log.
type [required]
\nenum
The processor type. The value should always be enrichment_table. \nAllowed enum values: enrichment_table
default: enrichment_table
Option 15
\nobject
The reduce processor aggregates and merges logs based on matching keys and merge strategies.
group_by [required]
\n[string]
A list of fields used to group log events for merging.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
merge_strategies [required]
\n[object]
List of merge strategies defining how values from grouped events should be combined.
path [required]
\nstring
The field path in the log event.
strategy [required]
\nenum
The merge strategy to apply. \nAllowed enum values: discard,retain,sum,max,min,array,concat,concat_newline,concat_raw,shortest_array,longest_array,flat_unique
type [required]
\nenum
The processor type. The value should always be reduce. \nAllowed enum values: reduce
default: reduce
Option 16
\nobject
The throttle processor limits the number of events that pass through over a given time window.
group_by
\n[string]
Optional list of fields used to group events before the threshold has been reached.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
threshold [required]
\nint64
the number of events allowed in a given time window. Events sent after the threshold has been reached, are dropped.
type [required]
\nenum
The processor type. The value should always be throttle. \nAllowed enum values: throttle
default: throttle
window [required]
\ndouble
The time window in seconds over which the threshold applies.
sources [required]
\n[ <oneOf>]
A list of configured data sources for the pipeline.
Option 1
\nobject
The kafka source ingests data from Apache Kafka topics.
group_id [required]
\nstring
Consumer group ID used by the Kafka client.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
librdkafka_options
\n[object]
Optional list of advanced Kafka client configuration options, defined as key-value pairs.
name [required]
\nstring
The name of the librdkafka configuration option to set.
value [required]
\nstring
The value assigned to the specified librdkafka configuration option.
sasl
\nobject
Specifies the SASL mechanism for authenticating with a Kafka cluster.
mechanism
\nenum
SASL mechanism used for Kafka authentication. \nAllowed enum values: PLAIN,SCRAM-SHA-256,SCRAM-SHA-512
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
topics [required]
\n[string]
A list of Kafka topic names to subscribe to. The source ingests messages from each topic specified.
type [required]
\nenum
The source type. The value should always be kafka. \nAllowed enum values: kafka
default: kafka
Option 2
\nobject
The datadog_agent source collects logs from the Datadog Agent.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be datadog_agent. \nAllowed enum values: datadog_agent
default: datadog_agent
Option 3
\nobject
The splunk_tcp source receives logs from a Splunk Universal Forwarder over TCP.\nTLS is supported for secure transmission.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always splunk_tcp. \nAllowed enum values: splunk_tcp
default: splunk_tcp
Option 4
\nobject
The splunk_hec source implements the Splunk HTTP Event Collector (HEC) API.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always splunk_hec. \nAllowed enum values: splunk_hec
default: splunk_hec
Option 5
\nobject
The amazon_s3 source ingests logs from an Amazon S3 bucket.\nIt supports AWS authentication and TLS encryption.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
region [required]
\nstring
AWS region where the S3 bucket resides.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always amazon_s3. \nAllowed enum values: amazon_s3
default: amazon_s3
Option 6
\nobject
The fluentd source ingests logs from a Fluentd-compatible service.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be `fluentd. \nAllowed enum values: fluentd
default: fluentd
Option 7
\nobject
The fluent_bit source ingests logs from Fluent Bit.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be fluent_bit. \nAllowed enum values: fluent_bit
default: fluent_bit
Option 8
\nobject
The http_server source collects logs over HTTP POST from external services.
auth_strategy [required]
\nenum
HTTP authentication method. \nAllowed enum values: none,plain
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
Unique ID for the HTTP server source.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be http_server. \nAllowed enum values: http_server
default: http_server
Option 9
\nobject
The sumo_logic source receives logs from Sumo Logic collectors.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
type [required]
\nenum
The source type. The value should always be sumo_logic. \nAllowed enum values: sumo_logic
default: sumo_logic
Option 10
\nobject
The rsyslog source listens for logs over TCP or UDP from an rsyslog server using the syslog protocol.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
mode [required]
\nenum
Protocol used by the syslog source to receive messages. \nAllowed enum values: tcp,udp
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be rsyslog. \nAllowed enum values: rsyslog
default: rsyslog
Option 11
\nobject
The syslog_ng source listens for logs over TCP or UDP from a syslog-ng server using the syslog protocol.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
mode [required]
\nenum
Protocol used by the syslog source to receive messages. \nAllowed enum values: tcp,udp
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be syslog_ng. \nAllowed enum values: syslog_ng
default: syslog_ng
Option 12
\nobject
The amazon_data_firehose source ingests logs from AWS Data Firehose.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be amazon_data_firehose. \nAllowed enum values: amazon_data_firehose
default: amazon_data_firehose
Option 13
\nobject
The google_pubsub source ingests logs from a Google Cloud Pub/Sub subscription.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
project [required]
\nstring
The GCP project ID that owns the Pub/Sub subscription.
subscription [required]
\nstring
The Pub/Sub subscription name from which messages are consumed.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be google_pubsub. \nAllowed enum values: google_pubsub
default: google_pubsub
Option 14
\nobject
The http_client source scrapes logs from HTTP endpoints at regular intervals.
auth_strategy
\nenum
Optional authentication strategy for HTTP requests. \nAllowed enum values: basic,bearer
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
scrape_interval_secs
\nint64
The interval (in seconds) between HTTP scrape requests.
scrape_timeout_secs
\nint64
The timeout (in seconds) for each scrape request.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be http_client. \nAllowed enum values: http_client
default: http_client
Option 15
\nobject
The logstash source ingests logs from a Logstash forwarder.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be logstash. \nAllowed enum values: logstash
default: logstash
name [required]
\nstring
Name of the pipeline.
id [required]
\nstring
Unique identifier for the pipeline.
type [required]
\nstring
The resource type identifier. For pipeline resources, this should always be set to pipelines.
default: pipelines
data [required]
\nobject
Contains the pipeline’s ID, type, and configuration attributes.
attributes [required]
\nobject
Defines the pipeline’s name and its components (sources, processors, and destinations).
config [required]
\nobject
Specifies the pipeline's configuration, including its sources, processors, and destinations.
destinations [required]
\n[ <oneOf>]
A list of destination components where processed logs are sent.
Option 1
\nobject
The datadog_logs destination forwards logs to Datadog Log Management.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be datadog_logs. \nAllowed enum values: datadog_logs
default: datadog_logs
Option 2
\nobject
The amazon_s3 destination sends your logs in Datadog-rehydratable format to an Amazon S3 bucket for archiving.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
bucket [required]
\nstring
S3 bucket name.
id [required]
\nstring
Unique identifier for the destination component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
key_prefix
\nstring
Optional prefix for object keys.
region [required]
\nstring
AWS region of the S3 bucket.
storage_class [required]
\nenum
S3 storage class. \nAllowed enum values: STANDARD,REDUCED_REDUNDANCY,INTELLIGENT_TIERING,STANDARD_IA,EXPRESS_ONEZONE,ONEZONE_IA,GLACIER,GLACIER_IR,DEEP_ARCHIVE
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. Always amazon_s3. \nAllowed enum values: amazon_s3
default: amazon_s3
Option 3
\nobject
The google_cloud_storage destination stores logs in a Google Cloud Storage (GCS) bucket.\nIt requires a bucket name, GCP authentication, and metadata fields.
acl [required]
\nenum
Access control list setting for objects written to the bucket. \nAllowed enum values: private,project-private,public-read,authenticated-read,bucket-owner-read,bucket-owner-full-control
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
bucket [required]
\nstring
Name of the GCS bucket.
id [required]
\nstring
Unique identifier for the destination component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
key_prefix
\nstring
Optional prefix for object keys within the GCS bucket.
metadata
\n[object]
Custom metadata to attach to each object uploaded to the GCS bucket.
name [required]
\nstring
The metadata key.
value [required]
\nstring
The metadata value.
storage_class [required]
\nenum
Storage class used for objects stored in GCS. \nAllowed enum values: STANDARD,NEARLINE,COLDLINE,ARCHIVE
type [required]
\nenum
The destination type. Always google_cloud_storage. \nAllowed enum values: google_cloud_storage
default: google_cloud_storage
Option 4
\nobject
The splunk_hec destination forwards logs to Splunk using the HTTP Event Collector (HEC).
auto_extract_timestamp
\nboolean
If true, Splunk tries to extract timestamps from incoming log events.\nIf false, Splunk assigns the time the event was received.
encoding
\nenum
Encoding format for log events. \nAllowed enum values: json,raw_message
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
index
\nstring
Optional name of the Splunk index where logs are written.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
sourcetype
\nstring
The Splunk sourcetype to assign to log events.
type [required]
\nenum
The destination type. Always splunk_hec. \nAllowed enum values: splunk_hec
default: splunk_hec
Option 5
\nobject
The sumo_logic destination forwards logs to Sumo Logic.
encoding
\nenum
The output encoding format. \nAllowed enum values: json,raw_message,logfmt
header_custom_fields
\n[object]
A list of custom headers to include in the request to Sumo Logic.
name [required]
\nstring
The header field name.
value [required]
\nstring
The header field value.
header_host_name
\nstring
Optional override for the host name header.
header_source_category
\nstring
Optional override for the source category header.
header_source_name
\nstring
Optional override for the source name header.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be sumo_logic. \nAllowed enum values: sumo_logic
default: sumo_logic
Option 6
\nobject
The elasticsearch destination writes logs to an Elasticsearch cluster.
api_version
\nenum
The Elasticsearch API version to use. Set to auto to auto-detect. \nAllowed enum values: auto,v6,v7,v8
bulk_index
\nstring
The index to write logs to in Elasticsearch.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be elasticsearch. \nAllowed enum values: elasticsearch
default: elasticsearch
Option 7
\nobject
The rsyslog destination forwards logs to an external rsyslog server over TCP or UDP using the syslog protocol.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
keepalive
\nint64
Optional socket keepalive duration in milliseconds.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. The value should always be rsyslog. \nAllowed enum values: rsyslog
default: rsyslog
Option 8
\nobject
The syslog_ng destination forwards logs to an external syslog-ng server over TCP or UDP using the syslog protocol.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
keepalive
\nint64
Optional socket keepalive duration in milliseconds.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The destination type. The value should always be syslog_ng. \nAllowed enum values: syslog_ng
default: syslog_ng
Option 9
\nobject
The azure_storage destination forwards logs to an Azure Blob Storage container.
blob_prefix
\nstring
Optional prefix for blobs written to the container.
container_name [required]
\nstring
The name of the Azure Blob Storage container to store logs in.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be azure_storage. \nAllowed enum values: azure_storage
default: azure_storage
Option 10
\nobject
The microsoft_sentinel destination forwards logs to Microsoft Sentinel.
client_id [required]
\nstring
Azure AD client ID used for authentication.
dcr_immutable_id [required]
\nstring
The immutable ID of the Data Collection Rule (DCR).
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
table [required]
\nstring
The name of the Log Analytics table where logs are sent.
tenant_id [required]
\nstring
Azure AD tenant ID.
type [required]
\nenum
The destination type. The value should always be microsoft_sentinel. \nAllowed enum values: microsoft_sentinel
default: microsoft_sentinel
Option 11
\nobject
The google_chronicle destination sends logs to Google Chronicle.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
customer_id [required]
\nstring
The Google Chronicle customer ID.
encoding
\nenum
The encoding format for the logs sent to Chronicle. \nAllowed enum values: json,raw_message
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
log_type
\nstring
The log type metadata associated with the Chronicle destination.
type [required]
\nenum
The destination type. The value should always be google_chronicle. \nAllowed enum values: google_chronicle
default: google_chronicle
Option 12
\nobject
The new_relic destination sends logs to the New Relic platform.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The New Relic region. \nAllowed enum values: us,eu
type [required]
\nenum
The destination type. The value should always be new_relic. \nAllowed enum values: new_relic
default: new_relic
Option 13
\nobject
The sentinel_one destination sends logs to SentinelOne.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The SentinelOne region to send logs to. \nAllowed enum values: us,eu,ca,data_set_us
type [required]
\nenum
The destination type. The value should always be sentinel_one. \nAllowed enum values: sentinel_one
default: sentinel_one
Option 14
\nobject
The opensearch destination writes logs to an OpenSearch cluster.
bulk_index
\nstring
The index to write logs to.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be opensearch. \nAllowed enum values: opensearch
default: opensearch
Option 15
\nobject
The amazon_opensearch destination writes logs to Amazon OpenSearch.
auth [required]
\nobject
Authentication settings for the Amazon OpenSearch destination.\nThe strategy field determines whether basic or AWS-based authentication is used.
assume_role
\nstring
The ARN of the role to assume (used with aws strategy).
aws_region
\nstring
AWS region
external_id
\nstring
External ID for the assumed role (used with aws strategy).
session_name
\nstring
Session name for the assumed role (used with aws strategy).
strategy [required]
\nenum
The authentication strategy to use. \nAllowed enum values: basic,aws
bulk_index
\nstring
The index to write logs to.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be amazon_opensearch. \nAllowed enum values: amazon_opensearch
default: amazon_opensearch
processors
\n[ <oneOf>]
A list of processors that transform or enrich log data.
Option 1
\nobject
The filter processor allows conditional processing of logs based on a Datadog search query. Logs that match the include query are passed through; others are discarded.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs should pass through the filter. Logs that match this query continue to downstream components; others are dropped.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be filter. \nAllowed enum values: filter
default: filter
Option 2
\nobject
The parse_json processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string.
field [required]
\nstring
The name of the log field that contains a JSON string.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be parse_json. \nAllowed enum values: parse_json
default: parse_json
Option 3
\nobject
The Quota Processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert.
drop_events [required]
\nboolean
If set to true, logs that matched the quota filter and sent after the quota has been met are dropped; only logs that did not match the filter query continue through the pipeline.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
ignore_when_missing_partitions
\nboolean
If true, the processor skips quota checks when partition fields are missing from the logs.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
name [required]
\nstring
Name of the quota.
overflow_action
\nenum
The action to take when the quota is exceeded. Options:
\ndrop: Drop the event.
no_action: Let the event pass through.
overflow_routing: Route to an overflow destination.
Allowed enum values: drop,no_action,overflow_routing
overrides
\n[object]
A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit.
fields [required]
\n[object]
A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
partition_fields
\n[string]
A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values.
type [required]
\nenum
The processor type. The value should always be quota. \nAllowed enum values: quota
default: quota
Option 4
\nobject
The add_fields processor adds static key-value fields to logs.
fields [required]
\n[object]
A list of static fields (key-value pairs) that is added to each log event processed by this component.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be add_fields. \nAllowed enum values: add_fields
default: add_fields
Option 5
\nobject
The remove_fields processor deletes specified fields from logs.
fields [required]
\n[string]
A list of field names to be removed from each log event.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
The PipelineRemoveFieldsProcessor inputs.
type [required]
\nenum
The processor type. The value should always be remove_fields. \nAllowed enum values: remove_fields
default: remove_fields
Option 6
\nobject
The rename_fields processor changes field names.
fields [required]
\n[object]
A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields.
destination [required]
\nstring
The field name to assign the renamed value to.
preserve_source [required]
\nboolean
Indicates whether the original field, that is received from the source, should be kept (true) or removed (false) after renaming.
source [required]
\nstring
The original field name in the log event that should be renamed.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be rename_fields. \nAllowed enum values: rename_fields
default: rename_fields
Option 7
\nobject
The generate_datadog_metrics processor creates custom metrics from logs and sends them to Datadog.\nMetrics can be counters, gauges, or distributions and optionally grouped by log fields.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
metrics [required]
\n[object]
Configuration for generating individual metrics.
group_by
\n[string]
Optional fields used to group the metric series.
include [required]
\nstring
Datadog filter query to match logs for metric generation.
metric_type [required]
\nenum
Type of metric to create. \nAllowed enum values: count,gauge,distribution
name [required]
\nstring
Name of the custom metric to be created.
value [required]
\n<oneOf>
Specifies how the value of the generated metric is computed.
Option 1
\nobject
Strategy that increments a generated metric by one for each matching event.
strategy [required]
\nenum
Increments the metric by 1 for each matching event. \nAllowed enum values: increment_by_one
Option 2
\nobject
Strategy that increments a generated metric based on the value of a log field.
field [required]
\nstring
Name of the log field containing the numeric value to increment the metric by.
strategy [required]
\nenum
Uses a numeric field in the log event as the metric increment. \nAllowed enum values: increment_by_field
type [required]
\nenum
The processor type. Always generate_datadog_metrics. \nAllowed enum values: generate_datadog_metrics
default: generate_datadog_metrics
Option 8
\nobject
The sample processor allows probabilistic sampling of logs at a fixed rate.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
percentage
\ndouble
The percentage of logs to sample.
rate
\nint64
Number of events to sample (1 in N).
type [required]
\nenum
The processor type. The value should always be sample. \nAllowed enum values: sample
default: sample
Option 9
\nobject
The parse_grok processor extracts structured fields from unstructured log messages using Grok patterns.
disable_library_rules
\nboolean
If set to true, disables the default Grok rules provided by Datadog.
id [required]
\nstring
A unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
rules [required]
\n[object]
The list of Grok parsing rules. If multiple matching rules are provided, they are evaluated in order. The first successful match is applied.
match_rules [required]
\n[object]
A list of Grok parsing rules that define how to extract fields from the source field.\nEach rule must contain a name and a valid Grok pattern.
name [required]
\nstring
The name of the rule.
rule [required]
\nstring
The definition of the Grok rule.
source [required]
\nstring
The name of the field in the log event to apply the Grok rules to.
support_rules [required]
\n[object]
A list of Grok helper rules that can be referenced by the parsing rules.
name [required]
\nstring
The name of the Grok helper rule.
rule [required]
\nstring
The definition of the Grok helper rule.
type [required]
\nenum
The processor type. The value should always be parse_grok. \nAllowed enum values: parse_grok
default: parse_grok
Option 10
\nobject
The sensitive_data_scanner processor detects and optionally redacts sensitive data in log events.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
rules [required]
\n[object]
A list of rules for identifying and acting on sensitive data patterns.
keyword_options
\nobject
Configuration for keywords used to reinforce sensitive data pattern detection.
keywords [required]
\n[string]
A list of keywords to match near the sensitive pattern.
proximity [required]
\nint64
Maximum number of tokens between a keyword and a sensitive value match.
name [required]
\nstring
A name identifying the rule.
on_match [required]
\n<oneOf>
Defines what action to take when sensitive data is matched.
Option 1
\nobject
Configuration for completely redacting matched sensitive data.
action [required]
\nenum
Action type that completely replaces the matched sensitive data with a fixed replacement string to remove all visibility. \nAllowed enum values: redact
options [required]
\nobject
Configuration for fully redacting sensitive data.
replace [required]
\nstring
The ObservabilityPipelineSensitiveDataScannerProcessorActionRedactOptions replace.
Option 2
\nobject
Configuration for hashing matched sensitive values.
action [required]
\nenum
Action type that replaces the matched sensitive data with a hashed representation, preserving structure while securing content. \nAllowed enum values: hash
options
\nobject
The ObservabilityPipelineSensitiveDataScannerProcessorActionHash options.
Option 3
\nobject
Configuration for partially redacting matched sensitive data.
action [required]
\nenum
Action type that redacts part of the sensitive data while preserving a configurable number of characters, typically used for masking purposes (e.g., show last 4 digits of a credit card). \nAllowed enum values: partial_redact
options [required]
\nobject
Controls how partial redaction is applied, including character count and direction.
characters [required]
\nint64
The ObservabilityPipelineSensitiveDataScannerProcessorActionPartialRedactOptions characters.
direction [required]
\nenum
Indicates whether to redact characters from the first or last part of the matched value. \nAllowed enum values: first,last
pattern [required]
\n<oneOf>
Pattern detection configuration for identifying sensitive data using either a custom regex or a library reference.
Option 1
\nobject
Defines a custom regex-based pattern for identifying sensitive data in logs.
options [required]
\nobject
Options for defining a custom regex pattern.
rule [required]
\nstring
A regular expression used to detect sensitive values. Must be a valid regex.
type [required]
\nenum
Indicates a custom regular expression is used for matching. \nAllowed enum values: custom
Option 2
\nobject
Specifies a pattern from Datadog’s sensitive data detection library to match known sensitive data types.
options [required]
\nobject
Options for selecting a predefined library pattern and enabling keyword support.
id [required]
\nstring
Identifier for a predefined pattern from the sensitive data scanner pattern library.
use_recommended_keywords
\nboolean
Whether to augment the pattern with recommended keywords (optional).
type [required]
\nenum
Indicates that a predefined library pattern is used. \nAllowed enum values: library
scope [required]
\n<oneOf>
Determines which parts of the log the pattern-matching rule should be applied to.
Option 1
\nobject
Includes only specific fields for sensitive data scanning.
options [required]
\nobject
Fields to which the scope rule applies.
fields [required]
\n[string]
The ObservabilityPipelineSensitiveDataScannerProcessorScopeOptions fields.
target [required]
\nenum
Applies the rule only to included fields. \nAllowed enum values: include
Option 2
\nobject
Excludes specific fields from sensitive data scanning.
options [required]
\nobject
Fields to which the scope rule applies.
fields [required]
\n[string]
The ObservabilityPipelineSensitiveDataScannerProcessorScopeOptions fields.
target [required]
\nenum
Excludes specific fields from processing. \nAllowed enum values: exclude
Option 3
\nobject
Applies scanning across all available fields.
target [required]
\nenum
Applies the rule to all fields. \nAllowed enum values: all
tags [required]
\n[string]
Tags assigned to this rule for filtering and classification.
type [required]
\nenum
The processor type. The value should always be sensitive_data_scanner. \nAllowed enum values: sensitive_data_scanner
default: sensitive_data_scanner
Option 11
\nobject
The ocsf_mapper processor transforms logs into the OCSF schema using a predefined mapping configuration.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mappings [required]
\n[object]
A list of mapping rules to convert events to the OCSF format.
include [required]
\nstring
A Datadog search query used to select the logs that this mapping should apply to.
mapping [required]
\n<oneOf>
Defines a single mapping rule for transforming logs into the OCSF schema.
Option 1
\nenum
Predefined library mappings for common log formats. \nAllowed enum values: CloudTrail Account Change,GCP Cloud Audit CreateBucket,GCP Cloud Audit CreateSink,GCP Cloud Audit SetIamPolicy,GCP Cloud Audit UpdateSink,Github Audit Log API Activity,Google Workspace Admin Audit addPrivilege,Microsoft 365 Defender Incident,Microsoft 365 Defender UserLoggedIn,Okta System Log Authentication,Palo Alto Networks Firewall Traffic
type [required]
\nenum
The processor type. The value should always be ocsf_mapper. \nAllowed enum values: ocsf_mapper
default: ocsf_mapper
Option 12
\nobject
The add_env_vars processor adds environment variable values to log events.
id [required]
\nstring
The unique identifier for this component. Used to reference this processor in the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
type [required]
\nenum
The processor type. The value should always be add_env_vars. \nAllowed enum values: add_env_vars
default: add_env_vars
variables [required]
\n[object]
A list of environment variable mappings to apply to log fields.
field [required]
\nstring
The target field in the log event.
name [required]
\nstring
The name of the environment variable to read.
Option 13
\nobject
The dedupe processor removes duplicate fields in log events.
fields [required]
\n[string]
A list of log field paths to check for duplicates.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mode [required]
\nenum
The deduplication mode to apply to the fields. \nAllowed enum values: match,ignore
type [required]
\nenum
The processor type. The value should always be dedupe. \nAllowed enum values: dedupe
default: dedupe
Option 14
\nobject
The enrichment_table processor enriches logs using a static CSV file or GeoIP database.
file
\nobject
Defines a static enrichment table loaded from a CSV file.
encoding [required]
\nobject
File encoding format.
delimiter [required]
\nstring
The encoding delimiter.
includes_headers [required]
\nboolean
The encoding includes_headers.
type [required]
\nenum
Specifies the encoding format (e.g., CSV) used for enrichment tables. \nAllowed enum values: csv
key [required]
\n[object]
Key fields used to look up enrichment values.
column [required]
\nstring
The items column.
comparison [required]
\nenum
Defines how to compare key fields for enrichment table lookups. \nAllowed enum values: equals
field [required]
\nstring
The items field.
path [required]
\nstring
Path to the CSV file.
schema [required]
\n[object]
Schema defining column names and their types.
column [required]
\nstring
The items column.
type [required]
\nenum
Declares allowed data types for enrichment table columns. \nAllowed enum values: string,boolean,integer,float,date,timestamp
geoip
\nobject
Uses a GeoIP database to enrich logs based on an IP field.
key_field [required]
\nstring
Path to the IP field in the log.
locale [required]
\nstring
Locale used to resolve geographical names.
path [required]
\nstring
Path to the GeoIP database file.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
target [required]
\nstring
Path where enrichment results should be stored in the log.
type [required]
\nenum
The processor type. The value should always be enrichment_table. \nAllowed enum values: enrichment_table
default: enrichment_table
Option 15
\nobject
The reduce processor aggregates and merges logs based on matching keys and merge strategies.
group_by [required]
\n[string]
A list of fields used to group log events for merging.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
merge_strategies [required]
\n[object]
List of merge strategies defining how values from grouped events should be combined.
path [required]
\nstring
The field path in the log event.
strategy [required]
\nenum
The merge strategy to apply. \nAllowed enum values: discard,retain,sum,max,min,array,concat,concat_newline,concat_raw,shortest_array,longest_array,flat_unique
type [required]
\nenum
The processor type. The value should always be reduce. \nAllowed enum values: reduce
default: reduce
Option 16
\nobject
The throttle processor limits the number of events that pass through over a given time window.
group_by
\n[string]
Optional list of fields used to group events before the threshold has been reached.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
threshold [required]
\nint64
the number of events allowed in a given time window. Events sent after the threshold has been reached, are dropped.
type [required]
\nenum
The processor type. The value should always be throttle. \nAllowed enum values: throttle
default: throttle
window [required]
\ndouble
The time window in seconds over which the threshold applies.
Option 17
\nobject
The datadog_tags processor includes or excludes specific Datadog tags in your logs.
action [required]
\nenum
The action to take on tags with matching keys. \nAllowed enum values: include,exclude
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
keys [required]
\n[string]
A list of tag keys.
mode [required]
\nenum
The processing mode. \nAllowed enum values: filter
type [required]
\nenum
The processor type. The value should always be datadog_tags. \nAllowed enum values: datadog_tags
default: datadog_tags
sources [required]
\n[ <oneOf>]
A list of configured data sources for the pipeline.
Option 1
\nobject
The kafka source ingests data from Apache Kafka topics.
group_id [required]
\nstring
Consumer group ID used by the Kafka client.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
librdkafka_options
\n[object]
Optional list of advanced Kafka client configuration options, defined as key-value pairs.
name [required]
\nstring
The name of the librdkafka configuration option to set.
value [required]
\nstring
The value assigned to the specified librdkafka configuration option.
sasl
\nobject
Specifies the SASL mechanism for authenticating with a Kafka cluster.
mechanism
\nenum
SASL mechanism used for Kafka authentication. \nAllowed enum values: PLAIN,SCRAM-SHA-256,SCRAM-SHA-512
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
topics [required]
\n[string]
A list of Kafka topic names to subscribe to. The source ingests messages from each topic specified.
type [required]
\nenum
The source type. The value should always be kafka. \nAllowed enum values: kafka
default: kafka
Option 2
\nobject
The datadog_agent source collects logs from the Datadog Agent.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be datadog_agent. \nAllowed enum values: datadog_agent
default: datadog_agent
Option 3
\nobject
The splunk_tcp source receives logs from a Splunk Universal Forwarder over TCP.\nTLS is supported for secure transmission.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always splunk_tcp. \nAllowed enum values: splunk_tcp
default: splunk_tcp
Option 4
\nobject
The splunk_hec source implements the Splunk HTTP Event Collector (HEC) API.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always splunk_hec. \nAllowed enum values: splunk_hec
default: splunk_hec
Option 5
\nobject
The amazon_s3 source ingests logs from an Amazon S3 bucket.\nIt supports AWS authentication and TLS encryption.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
region [required]
\nstring
AWS region where the S3 bucket resides.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. Always amazon_s3. \nAllowed enum values: amazon_s3
default: amazon_s3
Option 6
\nobject
The fluentd source ingests logs from a Fluentd-compatible service.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be `fluentd. \nAllowed enum values: fluentd
default: fluentd
Option 7
\nobject
The fluent_bit source ingests logs from Fluent Bit.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be fluent_bit. \nAllowed enum values: fluent_bit
default: fluent_bit
Option 8
\nobject
The http_server source collects logs over HTTP POST from external services.
auth_strategy [required]
\nenum
HTTP authentication method. \nAllowed enum values: none,plain
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
Unique ID for the HTTP server source.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be http_server. \nAllowed enum values: http_server
default: http_server
Option 9
\nobject
The sumo_logic source receives logs from Sumo Logic collectors.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
type [required]
\nenum
The source type. The value should always be sumo_logic. \nAllowed enum values: sumo_logic
default: sumo_logic
Option 10
\nobject
The rsyslog source listens for logs over TCP or UDP from an rsyslog server using the syslog protocol.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
mode [required]
\nenum
Protocol used by the syslog source to receive messages. \nAllowed enum values: tcp,udp
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be rsyslog. \nAllowed enum values: rsyslog
default: rsyslog
Option 11
\nobject
The syslog_ng source listens for logs over TCP or UDP from a syslog-ng server using the syslog protocol.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
mode [required]
\nenum
Protocol used by the syslog source to receive messages. \nAllowed enum values: tcp,udp
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be syslog_ng. \nAllowed enum values: syslog_ng
default: syslog_ng
Option 12
\nobject
The amazon_data_firehose source ingests logs from AWS Data Firehose.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be amazon_data_firehose. \nAllowed enum values: amazon_data_firehose
default: amazon_data_firehose
Option 13
\nobject
The google_pubsub source ingests logs from a Google Cloud Pub/Sub subscription.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
project [required]
\nstring
The GCP project ID that owns the Pub/Sub subscription.
subscription [required]
\nstring
The Pub/Sub subscription name from which messages are consumed.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be google_pubsub. \nAllowed enum values: google_pubsub
default: google_pubsub
Option 14
\nobject
The http_client source scrapes logs from HTTP endpoints at regular intervals.
auth_strategy
\nenum
Optional authentication strategy for HTTP requests. \nAllowed enum values: basic,bearer
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
scrape_interval_secs
\nint64
The interval (in seconds) between HTTP scrape requests.
scrape_timeout_secs
\nint64
The timeout (in seconds) for each scrape request.
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be http_client. \nAllowed enum values: http_client
default: http_client
Option 15
\nobject
The logstash source ingests logs from a Logstash forwarder.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption between the pipeline component and external services.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be logstash. \nAllowed enum values: logstash
default: logstash
name [required]
\nstring
Name of the pipeline.
id [required]
\nstring
Unique identifier for the pipeline.
type [required]
\nstring
The resource type identifier. For pipeline resources, this should always be set to pipelines.
default: pipelines