Update all dependencies

[renovate]

This PR contains the following updates:

Package	Type	Update	Change	Age	Adoption	Passing	Confidence
SonarSource/sonarqube-scan-action	action	major	v4 -> v5
github.com/Jeffail/benthos/v3	require	major	v3.65.0 -> v4.47.1
github.com/asecurityteam/settings	require	major	v1.0.0 -> v2.0.2
github.com/nats-io/nats-server/v2	replace	minor	v2.7.4 -> v2.10.25
go (source)	toolchain	minor	1.23.6 -> 1.24.0

---

Release Notes

SonarSource/sonarqube-scan-action (SonarSource/sonarqube-scan-action)

v5

Compare Source

Jeffail/benthos (github.com/Jeffail/benthos/v3)

v4.47.1

Compare Source

For installation instructions check out the getting started guide.

Fixed

• Fix an issue with left over staging files being left around in the snowflake_streaming output. (@​rockwotj)

The full change log can be found here.

v4.47.0

Compare Source

For installation instructions check out the getting started guide.

Added

• Field arguments added to the amqp_0_9 input and output. (@​calini)
• Field avro.mapping added to the schema_registry_decode processor to support converting custom avro types to standard avro types for legacy tooling. (@​rockwotj)
• (Benthos) A crash processor for FATAL logging. (@​rockwotj)
• (Benthos) A uuid_v7 bloblang function. (@​rockwotj)
• (Benthos) Field disable_http2 added to the http_client input and output and to the http processor. (@​mihaitodor)
• New elasticsearch_v8 output which supersedes the existing elasticsearch output that uses a deprecated Elasticsearch library. (@​ooesili)
• Field retry_on_conflict added to elasticsearch output to retry operations in case there are document version conflicts.

The full change log can be found here.

v4.46.0

Compare Source

For installation instructions check out the getting started guide.

Added

• New mysql_cdc input supporting change data capture (CDC) from MySQL. (@​rockwotj, @​le-vlad)
• Field instance_id added to kafka, kafka_franz, ockam_kafka, redpanda, redpanda_common, and redpanda_migrator inputs. (@​rockwotj)
• Fields rebalance_timeout, session_timeout and heartbeat_interval added to the kafka_franz, redpanda, redpanda_common, redpanda_migrator and ockam_kafka inputs. (@​rockwotj)
• Field avro.preserve_logical_types for processor schema_registry_decode was added to preserve logical types instead of decoding them as their primitive representation. (@​rockwotj)
• Processor schema_registry_decode now adds metadata schema_id for the schema's ID in the schema registry. (@​rockwotj)
• Field schema_evolution.processors added to snowpipe_streaming to support side effects or enrichment during schema evolution. (@​rockwotj)
• Field unchanged_toast_value added to postgres_cdc to control the value substituted for unchanged toast values when a table does not have full replica identity. (@​rockwotj)

Fixed

• Fix a snapshot stream consistency issue with postgres_cdc where data could be missed if writes were happening during the snapshot phase. (@​rockwotj)
• Fix an issue where @table metadata was quoted for the snapshot phase in postgres_cdc. (@​rockwotj)

Changed

• Field avro_raw_json was deprecated in favor of avro.raw_unions for processor schema_registry_decode. (@​rockwotj)
• The snowpipe_streaming output now has better error handling for authentication failures when uploading to cloud storage. (@​rockwotj)
• Field schema_evolution.new_column_type_mapping for snowpipe_streaming is deprecated and can be replaced with schema_evolution.processors. (@​rockwotj)
• Increased the default values for max_message_bytes and broker_write_max_bytes by using IEC units instead of SI units. This better matches defaults in Redpanda and Kafka. (@​rockwotj)
• Dropped support for postgres 10 and 11 in postgres_cdc. (@​rockwotj)

The full change log can be found here.

v4.45.1

Compare Source

For installation instructions check out the getting started guide.

Fixed

• Empty files read by input aws_s3 no longer cause spurious errors. (@​rockwotj)
• Fixes a SIGSEGV in postgres_cdc when using TOAST values with tables that don't have FULL replica identity. (@​rockwotj)

The full change log can be found here.

v4.45.0

Compare Source

For installation instructions check out the getting started guide.

Fixed

• The code and file fields on the javascript processor docs no longer erroneously mention interpolation support. (@​mihaitodor)
• The postgres_cdc now correctly handles null values. (@​rockwotj)
• The redpanda_migrator output no longer rejects messages if it can't perform schema ID translation. (@​mihaitodor)
• The redpanda_migrator input no longer converts the kafka key to string. (@​mihaitodor)

Added

• aws_sqs input now has a max_outstanding field to prevent unbounded memory usage. (@​rockwotj)
• avro scanner now emits metadata for the Avro schema it used along with the schema fingerprint. (@​rockwotj)
• Field content_type added to the amqp_1 output. (@​timo102)
• Field fetch_max_wait added to the kafka_franz, ockam_kafka, redpanda, redpanda_common and redpanda_migrator inputs. (@​birdayz)
• snowpipe_streaming output now supports interpolating table names. (@​rockwotj)
• snowpipe_streaming output now supports interpolating channel names. (@​rockwotj)
• snowpipe_streaming output now supports exactly once delivery using offset_token. (@​rockwotj)
• ollama_chat processor now supports tool calling. (@​rockwotj)
• New ollama_moderation processor which allows using LlamaGuard or ShieldGemma to check if LLM responses are safe. (@​rockwotj)
• Field queries added to sql_raw processor and output to support rummong multiple SQL statements transactionally. (@​rockwotj)
• New redpanda_migrator_offsets input. (@​mihaitodor)
• Fields offset_topic, offset_group, offset_partition, offset_commit_timestamp and offset_metadata added to the redpanda_migrator_offsets output. (@​mihaitodor)
• Field topic_lag_refresh_period added to the redpanda and redpanda_common inputs. (@​mihaitodor)
• Metric redpanda_lag now emitted by the redpanda and redpanda_common inputs. (@​mihaitodor)
• Metadata kafka_lag now emitted by the redpanda and redpanda_common inputs. (@​mihaitodor)
• The redpanda_migrator_bundle input and output now set labels for their subcomponents. (@​mihaitodor)
• (Benthos) Field label added to the template tests definitions. (@​mihaitodor)
• (Benthos) Metadata field label can now be utilized within a template's mapping field to access the label that is associated with the template instantiation in a config. (@​mihaitodor)
• (Benthos) bloblang scalar type added to template fields. (@​mihaitodor)
• (Benthos) Go API: Method SetOutputBrokerPattern added to the StreamBuilder type. (@​mihaitodor)
• (Benthos) New error_source_name, error_source_label and error_source_path bloblang functions. (@​mihaitodor)
• (Benthos) Flag --verbose added to the benthos lint and benthos template lint commands. (@​mihaitodor)

Changed

• Fix an issue in aws_sqs with refreshing in-flight message leases which could prevent acks from processed. (@​rockwotj)
• Fix an issue with postgres_cdc with TOAST values not being propagated with REPLICA IDENTITY FULL. (@​rockwotj)
• Fix a initial snapshot streaming consistency issue with postgres_cdc. (@​rockwotj)
• Fix bug in sftp input where the last file was not deleted when watcher and delete_on_finish were enabled. (@​ooesili)
• Fields batch_size, multi_header, replication_factor, replication_factor_override and output_resource for the redpanda_migrator input are now deprecated. (@​mihaitodor)
• Fields kafka_key and max_in_flight for the redpanda_migrator_offsets output are now deprecated. (@​mihaitodor)
• Field batching for the redpanda_migrator output is now deprecated. (@​mihaitodor)
• The redpanda_migrator input no longer emits tombstone messages. (@​mihaitodor)
• (Benthos) The branch processor no longer emits an entry in the log at error level when the child processors throw errors. (@​mihaitodor)
• (Benthos) Streams and the StreamBuilder API now use reject by default when no output is specified in the config and stdout isn't registered (for example when the io components are not imported). (@​mihaitodor)

The full change log can be found here.

v4.44.0

Compare Source

For installation instructions check out the getting started guide.

Added

• Go API: New public/license package added to allow custom programmatic instantiations of Redpanda Connect to run enterprise license components. (@​Jeffail)

Fixed

• gcp_bigquery output with parquet format no longer returns errors incorrectly. (@​rockwotj)
• postgres_cdc input now allows quoted identifiers for the table names. (@​mihaitodor, @​rockwotj)

The full change log can be found here.

v4.43.1

Compare Source

For installation instructions check out the getting started guide.

Fixed

• Trial Redpanda Enterprise licenses are now considered valid. (@​Jeffail)
• The redpanda_migrator_bundle output now skips schema ID translation when translate_schema_ids: false and schema_registry is configured. (@​mihaitodor)

The full change log can be found here.

v4.43.0

Compare Source

For installation instructions check out the getting started guide.

Changed

• The pg_stream input has been renamed to postgres_cdc. The old name will continue to function as an alias. (@​rockwotj)
• The postgres_cdc input no longer emits mode metadata and instead snapshot reads set operation metadata to be read instead of insert. (@​rockwotj)

Fixed

• The redpanda_migrator_bundle output no longer attempts to translate schema IDs when a schema registry is not configured. (@​mihaitodor)

The full change log can be found here.

v4.42.0

Compare Source

For installation instructions check out the getting started guide.

Added

• Add support for spanner driver to SQL plugins. (@​yufeng-deng)
• Add support for complex database types (JSONB, TEXT[], INET, TSVECTOR, TSRANGE, POINT, INTEGER[]) for pg_stream input. (@​le-vlad)
• Add support for Parquet files to bigquery output (@​rockwotj)
• (Benthos) New exists operator added to the cache processor. (@​mihaitodor)
• New CLI flag redpanda-license added as an alternative way to specify a Redpanda license. (@​Jeffail)

Fixed

• Fixed pg_stream issue with discrepancies between replication and snapshot streaming for UUID type. (@​le-vlad)
• Fixed avro scanner bug introduced in v4.25.0. (@​mihaitodor)

Changed

• The redpanda_migrator output now registers destination schemas with all the subjects associated with the source schema ID extracted from each message. (@​mihaitodor)
• Enterprise features will now only run when a valid Redpanda license is present. More information can be found at the licenses getting started guide. (@​Jeffail)

The full change log can be found here.

v4.41.0

Compare Source

For installation instructions check out the getting started guide.

Added

• Field max_records_per_request added to the aws_sqs output. (@​Jeffail)

Fixed

• (Benthos) Fixed an issue where running a CLI with a custom environment would cause imported templates to be rejected. (@​Jeffail)

Changed

• The -cgo suffixed docker images are no longer built and pushed along with the regular images. This decision was made due to low demand, and the unacceptable cadence with which the image base (Debian) receives security updates. It is still possible to create your own CGO builds with the command CGO_ENABLED=1 make TAGS=x_benthos_extra redpanda-connect. (@​Jeffail)

The full change log can be found here.

v4.40.0

Compare Source

For installation instructions check out the getting started guide.

Added

• New pg_stream input supporting change data capture (CDC) from PostgreSQL. (@​le-vlad)
• Field metadata_max_age added to the redpanda_migrator_offsets output. (@​mihaitodor)
• Field kafka_timestamp_ms added to the kafka, kafka_franz, redpanda, redpanda_common and redpanda_migrator outputs. (@​mihaitodor)
• (Benthos) New Bloblang method timestamp. (@​mihaitodor)
• (Benthos) New benchmark processor. (@​ooesili)

Fixed

• Addresses an issue where snowflake_streaming could create more channels than configured. (@​rockwotj)

Changed

• The snowflake_streaming output with schema_evolution.enabled set to true can now autocreate tables. (@​rockwotj)
• Fields translate_schema_ids and schema_registry_output_resource added to the redpanda_migrator output. (@​mihaitodor)
• Fields backfill_dependencies and input_resource added to the schema_registry output. (@​mihaitodor)
• The schema_registry input and output and the schema_registry_encode and schema_registry_decode processors now use the github.com/twmb/franz-go/pkg/sr SchemaRegistry client. (@​mihaitodor)
• Metadata field kafka_timestamp_ms added to the kafka, kafka_franz, redpanda, redpanda_common and redpanda_migrator inputs now contains a unix timestamp with millisecond precision. (@​mihaitodor)
• Metadata field kafka_timestamp removed from the kafka, kafka_franz, redpanda, redpanda_common and redpanda_migrator inputs. (@​mihaitodor)

The full change log can be found here.

v4.39.0

Compare Source

For installation instructions check out the getting started guide.

Added

• New timeplus input. (@​ye11ow)
• New snowflake_streaming output. (@​rockwotj)
• Redpanda Connect will now use an optional /etc/redpanda/connector_list.yaml config to determine which connectors are available to run. (@​Jeffail)
• (Benthos) Field follow_redirects added to the http processor. (@​ooesili)
• New CLI flag --secrets added. (@​Jeffail)
• New CLI flag --disable-telemetry added. (@​Jeffail)
• New experimental spicedb watch input. (@​simon0191)
• New redpanda_common input and output. (@​Jeffail)
• New redpanda input and output. (@​Jeffail)
• New snowflake_streaming output. (@​rockwotj)

Fixed

• The kafka, kafka_franz and redpanda_migrator outputs no longer waste CPU for large batches. (@​rockwotj)

Changed

• The aws_sqs output field url now supports interpolation functions. (@​rockwotj)
• (Benthos) CLI --set flags can now mutate array values indexed from the end via negative integers. E.g. --set 'foo.-1=meow' would set the last index of the array foo to the value of meow. (@​Jeffail)

The full change log can be found here.

v4.38.0

Compare Source

For installation instructions check out the getting started guide.

Added

• Anonymous telemetry data is now sent by Connect instances after running for >5 mins. Details about which data is sent, when it is sent, and how to disable it can be found in the telemetry README. (@​Jeffail)
• Field checksum_algorithm added to the aws_s3 output. (@​dom-lee-naimuri)
• Field nkey added to nats, nats_jetstream, nats_kv and nats_stream components. (@​ye11ow)
• Field private_key added to the snowflake_put output. (@​mihaitodor)
• New azure_data_lake_gen2 output. (@​ooesili)
• New timeplus output. (@​ye11ow)

Fixed

• The elasticsearch output now performs retries for HTTP status code 429 (Too Many Requests). (@​kahoowkh)
• The docs for the collection field of the mongodb output now specify support for interpolation functions. (@​mihaitodor)

Changed

• All components with a default path field value (such as the aws_s3 output) containing the deprecated function count have now been changed to use the new function counter. This could potentially change behaviour in cases where multiple components are executing a mapping with a count function sharing the same of the old default count, and these counters need to cascade. This is an extremely unlikely scenario, but for all users of these components it is recommended that your path is defined explicitly, and in a future major version we will be removing the defaults.

The full change log can be found here.

v4.37.0

Compare Source

For installation instructions check out the getting started guide.

Added

• New experimental gcp_vertex_ai_embeddings processor. (@​rockwotj)
• New experimental aws_bedrock_embeddings processor. (@​rockwotj)
• New experimental cohere_chat and cohere_embeddings processors. (@​rockwotj)
• New experimental questdb output. (@​sklarsa)
• Field metadata_max_age added to the kafka_franz input. (@​Scarjit)
• Field metadata_max_age added to the kafka_migrator input. (@​mihaitodor)
• New experimental cypher output. (@​rockwotj)
• New experimental couchbase output. (@​rockwotj)
• Field fetch_in_order added to the schema_registry input. (@​mihaitodor)

Fixed

• Fixed a bug with the input_resource field for the kafka_migrator output where new topics weren't created as expected. (@​mihaitodor)
• Fixed a bug in the kafka_migrator input which could lead to extra duplicate messages during a consumer group rebalance. (@​mihaitodor)
• kafka_migrator, kafka_migrator_offsets and kafka_migrator_bundle components renamed to redpanda_migrator, redpanda_migrator_offsets and redpanda_migrator_bundle (@​mihaitodor)

Fixed

• Fixes a panic in the parquet_encode processor (@​mihaitodor)

The full change log can be found here.

v4.36.0

Compare Source

For installation instructions check out the getting started guide.

Added

• Fields replication_factor and replication_factor_override added to the kafka_migrator input and output. (@​mihaitodor)

Fixed

• The schema_registry_encode and schema_registry_decode processors no longer unescape path separators in the schema name. (@​Mizaro)
• (Benthos) The switch output metrics now emit the case id as part of their labels. This is a regression introduced in v4.25.0. (@​mihaitodor)
• (Benthos) Fixed a bug where certain logs used the %w verb to print errors resulting in incorrect output. (@​mihaitodor)
• (Benthos) The logger no longer tries to replace Go fmt verbs in log messages. (@​mihaitodor)

The full change log can be found here.

v4.35.3

Compare Source

For installation instructions check out the getting started guide.

Added

• Azure and GCP components added to cloud builds. (@​Jeffail)

Fixed

• The kafka_migrator_bundle input and output no longer require schema registry to be configured. (@​mihaitodor)

The full change log can be found here.

v4.35.2

Compare Source

For installation instructions check out the getting started guide.

Added

• Azure and GCP components added to cloud builds. (@​Jeffail)

Fixed

• The kafka_migrator_bundle input and output no longer require schema registry to be configured. (@​mihaitodor)

The full change log can be found here.

v4.35.1

Compare Source

For installation instructions check out the getting started guide.

Added

• Azure and GCP components added to cloud builds. (@​Jeffail)

Fixed

• The kafka_migrator_bundle input and output no longer require schema registry to be configured. (@​mihaitodor)

The full change log can be found here.

v4.35.0

Compare Source

For installation instructions check out the getting started guide.

Added

• Auth fields added to the schema_registry input and output. (@​mihaitodor)
• New experimental kafka_migrator and kafka_migrator_bundle inputs and outputs. (@​mihaitodor)
• New experimental kafka_migrator_offsets output. (@​mihaitodor)
• Field job_project added to the gcp_bigquery output. (@​Roviluca)

The full change log can be found here.

v4.34.0

Compare Source

For installation instructions check out the getting started guide.

Fixed

• The schema_registry output now allows pushing schemas if the target Schema Registry instance is in IMPORT mode. (@​mihaitodor)
• Fixed an issue where the azure_blob_storage input would fail to delete blobs when using targets_input with delete_objects: true. (@​mihaitodor)
• New experimental gcp_vertex_ai_chat processor. (@​rockwotj)
• New experimental aws_bedrock_chat processor. (@​rockwotj)

The full change log can be found here.

v4.33.0

Compare Source

For installation instructions check out the getting started guide.

Added

• Field content_md5 added to the aws_s3 output. (@​dom-lee-naimuri)
• Field send_ack added to the nats input. (@​plejd-sebman)
• New Bloblang method vector. (@​rockwotj)
• New experimental ockam_kafka input and output. (@​mrinalwadhwa, @​davide-baldo)
• Field credentials_json added to all GCP components. (@​tomasz-sadura)
• (Benthos) The list subcommand now supports the format jsonschema. (@​Jeffail)
• New experimental schema_registry input and output. (@​mihaitodor)
• New experimental qdrant output. (@​Anush008)
• (Benthos) The --set run flag now supports structured values, e.g. --set input={}. (@​Jeffail)

The full change log can be found here.

v4.32.1

Compare Source

For installation instructions check out the getting started guide.

Added

• Field app_name added to the MongoDB components. (@​mihaitodor)
• New openai_chat_completion processor. (@​rockwotj)
• New openai_embeddings processor. (@​rockwotj)
• New openai_image_generation processor. (@​rockwotj)
• New openai_speech processor. (@​rockwotj)
• New openai_transcription processor. (@​rockwotj)
• New openai_translation processor. (@​rockwotj)
• New ollama_chat processor. (@​rockwotj)
• New ollama_embeddings processor. (@​rockwotj)

Changed

• The gcp_pubsub output now rejects messages with metadata values which contain invalid UTF-8-encoded runes. (@​AndreasBergmeier6176)
• The .goreleaser.yml configuration has been set back to version 1. (@​Jeffail)
• The number of release build artifacts for the community and cloud flavours have been reduced due to Github Action Runner disk space limitations.

The full change log can be found here.

v4.32.0

Compare Source

v4.31.0

Compare Source

For installation instructions check out the getting started guide.

4.31.0 - 2024-07-19

Added

• The splunk input and splunk_hec output now support custom tls configuration. (@​mihaitodor)
• Field timestamp added to the kafka and kafka_franz outputs. (@​mihaitodor)
• (Benthos) Field max_retries added to the retry processor. (@​mihaitodor)
• (Benthos) Metadata fields retry_count and backoff_duration added to the retry processor. (@​mihaitodor)
• (Benthos) Parameter escape_html added to the format_json() Bloblang method. (@​mihaitodor)
• (Benthos) New array bloblang method. (@​gramian)
• (Benthos) Algorithm fnv32 added to the hash bloblang method. (@​CallMeMhz)
• New experimental redpanda_data_transform. (@​rockwotj)
• New -community suffixed build included in release artifacts, containing only FOSS functionality. (@​Jeffail)
• New -cloud suffixed build included in release artifacts, containing components enabled in Redpanda Cloud. (@​Jeffail)
• Field status_topic added to the global redpanda config block. (@​Jeffail)
• New pinecone output. (@​rockwotj)
• (Benthos) The /ready endpoint in regular operation now provides a detailed summary of all inputs and outputs, including connection errors where applicable. (@​Jeffail)

Changed

• (Benthos) All cli subcommands that previously relied on root-level flags (streams, lint, test, echo) now explicitly define those flags such that they appear in help-text and can be specified after the subcommand itself. This means previous commands such as connect -r ./foo.yaml streams ./bar.yaml can now be more intuitively written as connect streams -r ./foo.yaml ./bar.yaml and so on. The old style will still work in order to preserve backwards compatibility, but the help-text for these root-level flags has been hidden. (@​Jeffail)

v4.30.1

Compare Source

For installation instructions check out the getting started guide.

Added

• (Benthos) Field omit_empty added to the lines scanner. (@​mihaitodor)
• (Benthos) New scheme gcm added to the encrypt_aes and decrypy_aes Bloblang methods. (@​abergmeier)
• (Benthos) New Bloblang method pow. (@​mfamador)
• (Benthos) New sin, cos, tan and pi bloblang methods. (@​mfamador)
• (Benthos) Field proxy_url added to the websocket input and output. (@​mihaitodor)
• New experimental splunk input. (@​mihaitodor)

Fixed

• The sql_insert and sql_raw components no longer fail when inserting large binary blobs into Oracle BLOB columns. (@​mihaitodor)
• (Benthos) The websocket input and output now obey the HTTP_PROXY, HTTPS_PROXY and NO_PROXY environment variables. (@​mihaitodor)
• AWS Lambda serverless build artifacts have been added back to official releases.

Changed

• The splunk_hec output is now implemented as a native Go component. (@​mihaitodor)

The full change log can be found here.

v4.30.0

Compare Source

For installation instructions check out the getting started guide.

Added

• (Benthos) Field omit_empty added to the lines scanner. (@​mihaitodor)
• (Benthos) New scheme gcm added to the encrypt_aes and decrypy_aes Bloblang methods. (@​abergmeier)
• (Benthos) New Bloblang method pow. (@​mfamador)
• (Benthos) New sin, cos, tan and pi bloblang methods. (@​mfamador)
• (Benthos) Field proxy_url added to the websocket input and output. (@​mihaitodor)
• New experimental splunk input. (@​mihaitodor)

Fixed

• The sql_insert and sql_raw components no longer fail when inserting large binary blobs into Oracle BLOB columns. (@​mihaitodor)
• (Benthos) The websocket input and output now obey the HTTP_PROXY, HTTPS_PROXY and NO_PROXY environment variables. (@​mihaitodor)

Changed

• The splunk_hec output is now implemented as a native Go component. (@​mihaitodor)

The full change log can be found here.

v4.29.0

Compare Source

For installation instructions check out the getting started guide.

Added

• Go API: New packages public/bundle/free and public/bundle/enterprise with explicit licensing for bundles of component imports.
• Field auth.oauth2.scope added to the pulsar input and output. (@​srenatus)
• Field subscription_initial_position added to the pulsar input. (@​srenatus)

Fixed

• The pulsar input and output should no longer ignore auth.oauth2 fields. (@​srenatus)
• Creating builds using make no longer prints warnings when the repository does not contain a tag. (@​mkysel)
• Messages resulting from the redis processor are no longer invalid when using hash commands. (@​mkysel)
• The nats_jetstream input no longer fails to initialise when a stream is specified and a subject is not. (@​maxarndt)

The full change log can be found here.

v4.28.0

Compare Source

For installation instructions check out the getting started guide.

Changed

• The repository has been moved to redpanda-data/connect and no longer contains the core Benthos engine, which is now broken out into redpanda-data/benthos.

The full change log can be found here.

v4.27.0

Compare Source

For installation instructions check out the getting started guide.

Added

• New nats_kv cache type.
• The nats_jetstream input now supports last_per_subject and new deliver fallbacks.
• Field error_patterns added to the drop_on output.
• New redis_scan input type.
• Field auto_replay_nacks added to all inputs that traditionally automatically retry nacked messages as a toggle for this behaviour.
• New retry processor.
• New noop cache.
• Field targets_input added to the azure_blob_storage input.
• New reject_errored output.
• New nats_request_reply processor.
• New json_documents scanner.

Fixed

• The unarchive processor no longer yields linting errors when the format csv:x is specified. This is a regression introduced in v4.25.0.
• The sftp input will no longer consume files when the watcher cache returns an error. Instead, it will reattempt the file upon the next poll.
• The aws_sqs input no longer logs error level logs for visibility timeout refreshing errors.
• The nats_kv processor now allows nats wildcards for the keys operation.
• The nats_kv processor keys operation now returns a single message with an array of found keys instead of a batch of messages.
• The nats_kv processor history operation now returns a single message with an array of objects containing the record fields instead of a batch of messages.
• Field timeout added to the nats_kv processor to specify the maximum period to wait on an operation before aborting and returning an error.
• Bloblang comparison operators (>, <, <=, >=) now match the precision of the compared integers when applicable.
• The parse_form_url_encoded Bloblang method no longer produces results with an unknown data type for repeated query parameters.
• The echo CLI command no longer fails to sanitise configs when encountering an empty password field.

Changed

• The log events from all inputs and outputs when they first connect have been made more consistent and no longer contain any information regarding the nature of their connections.
• Splitting message batches with a split processor (or custom plugins) no longer results in downstream error handling loops around nacks. This was previously implemented as a feature to ensure unbounded expanded and split batches don't flood downstream services in the event of a minority of errors. However, introducing more clever origin tracking of errored messages has eliminated the need for this undocumented behaviour.

The full change log can be found here.

v4.26.0

Compare Source

For installation instructions check out the getting started guide.

Added

• Field credit added to the amqp_1 input to specify the maximum number of unacknowledged messages the sender can transmit.
• Bloblang now supports root-level if statements.
• New experimental sql cache.
• Fields batch_size, sort and limit added to the mongodb input.
• Field idemponent_write added to the kafka output.

Changed

• The default value of the amqp_1.credit input has changed from 1 to 64.
• The mongodb processor and output now support extended JSON in canonical form for document, filter and hint mappings.
• The open_telemetry_collector tracer has had the url field of gRPC and HTTP collectors deprecated in favour of address, which more accurately describes the intended format of endpoints. The old style will continue to work, but eventually will have its default value removed and an explicit value will be required.

Fixed

• Resource config imports containing % characters were being incorrectly parsed during unit test execution. This was a regression introduced in v4.25.0.
• Dynamic input and output config updates containing % characters were being incorrectly parsed. This was a regression introduced in v4.25.0.

The full change log can be found here.

v4.25.1

Compare Source

For installation instructions check out the getting started guide.

Fixed

• Fixed a regression in v4.25.0 where template based components were not parsing correctly from configs.

The full change log can be found here.

v4.25.0

Compare Source

For installation instructions check out the getting started guide.

Added

• Field address_cache added to the socket_server input.
• Field read_header added to the amqp_1 input.
• All inputs with a codec field now support a new field scanner to replace it. Scanners are more powerful as they are configured in a structured way similar to other component types rather than via a single string field, for more information check out the scanners page.
• New diff and patch Bloblang methods.
• New processors processor.
• Field read_header added to the amqp_1 input.
• A debug endpoint /debug/pprof/allocs has been added for profiling allocations.
• New cockroachdb_changefeed input.
• The open_telemetry_collector tracer now supports sampling.
• The aws_kinesis input and output now support specifying ARNs as the stream target.
• New azure_cosmosdb input, processor and output.
• All sql_* components now support the gocosmos driver.
• New opensearch output.

Fixed

• The javascript processor now handles module imports correctly.
• Bloblang if statements now provide explicit errors when query expressions resolve to non-boolean values.
• Some metadata fields from the amqp_1 input were always empty due to type mismatch, this should no longer be the case.
• The zip Bloblang method no longer fails when executed without arguments.
• The amqp_0_9 output no longer prints bogus exchange name when connecting to the server.
• The generate input no longer adds an extra second to interval: '@​every x' syntax.
• The nats_jetstream input no longer fails to locate mirrored streams.
• Fixed a rare panic in batching mechanisms with a specified period, where data arrives in low volumes and is sporadic.
• Executing config unit tests should no longer fail due to output resources failing to connect.

Changed

• The parse_parquet Bloblang function, parquet_decode, parquet_encode processors and the parquet input have all been upgraded to the latest version of the underlying Parquet library. Since this underlying library is experimental it is likely that behaviour changes will result. One significant change is that encoding numerical values that are larger than the column type (float64 into FLOAT, int64 into INT32, etc) will no longer be automatically converted.
• The parse_log processor field codec is now deprecated.
• WARNING: Many components have had their underlying implementations moved onto newer internal APIs for defining and extracting their configuration fields. It's recommended that upgrades to this version are performed cautiously.
• WARNING: All AWS components have been upgraded to the latest client libraries. Although lots of testing has been done, these libraries have the potential to differ in discrete ways in terms of how credentials are evaluated, cross-account connections are performed, and so on. It's recommended that upgrades to this version are performed cautiously.

The full change log can be found here.

v4.24.0

Compare Source

For installation instructions check out the getting started guide.

Added

• Field idempotent_write added to the kafka_franz output.
• Field idle_timeout added to the read_until input.
• Field delay_seconds added to the aws_sqs output.
• Fields discard_unknown and use_proto_names added to the protobuf processors.

Fixed

• Bloblang error messages for bad function/method names or parameters should now be improved in mappings that use shorthand for root = ....
• All redis components now support usernames within the configured URL for authentication.
• The `protobuf

---

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.

👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

---

• If you want to rebase/retry this PR, check this box

---

This PR was generated by Mend Renovate. View the repository job log.

[renovate]

⚠ Artifact update problem

Renovate failed to update an artifact related to this branch. You probably do not want to merge this PR as-is.

♻ Renovate will retry this branch, including artifacts, only when one of the following happens:

• any of the package files in this branch needs updating, or
• the branch becomes conflicted, or
• you click the rebase/retry checkbox if found above, or
• you rename this PR's title to start with "rebase!" to trigger it manually

The artifact failure details are included below:

File name: go.sum

Command failed: go get -d -t ./...
go: github.com/Jeffail/benthos/[email protected]: parsing go.mod:
	module declares its path as: github.com/benthosdev/benthos/v4
	        but was required as: github.com/Jeffail/benthos/v4

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud