Skip to content

Latest commit

 

History

History
1813 lines (1100 loc) · 65.7 KB

data-prepper.change-log-2.9.0.md

File metadata and controls

1813 lines (1100 loc) · 65.7 KB

  • Change main branch version to 2.10-SNAPSHOT (#4851)

    Krishna Kondaka - Tue, 20 Aug 2024 13:46:09 -0700

    EAD -> refs/heads/main, refs/remotes/origin/main, refs/remotes/origin/HEAD Change main branch version to 2.10-SNAPSHOT Signed-off-by: Kondaka <[email protected]>

    Signed-off-by: Kondaka <[email protected]>

  • Add support for AWS security lake sink as a bucket selector mode in S3 sink (#4846)

    Krishna Kondaka - Tue, 20 Aug 2024 13:19:42 -0700

    Signed-off-by: Kondaka <[email protected]>

  • Add handle failed events option to parse json processors (#4844)

    Taylor Gray - Mon, 19 Aug 2024 17:12:29 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Fix bug where race condition on ack callback could cause S3 folder partition to not be given up (#4835)

    Taylor Gray - Mon, 19 Aug 2024 16:15:05 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Update the parse JSON/XML/ION processors to use EventKey. (#4842)

    David Venable - Mon, 19 Aug 2024 11:12:27 -0700

    Signed-off-by: David Venable <[email protected]>

  • Add support for dynamic rule detection for pipeline config transformation (#4601)

    Srikanth Govindarajan - Fri, 16 Aug 2024 18:10:52 -0700

    • Add support for dynamic rule detection for pipeline config transformation Signed-off-by: Srikanth Govindarajan <[email protected]>

    • Address comments Signed-off-by: Srikanth Govindarajan <[email protected]>

    • Move rules and templates to plugin level Signed-off-by: Srikanth Govindarajan <[email protected]>

    • Add dummy plugin for testing dynamic rule detection Signed-off-by: Srikanth Govindarajan <[email protected]>

    • Address comments Signed-off-by: Srikanth Govindarajan <[email protected]>

    Signed-off-by: Srikanth Govindarajan <[email protected]> Signed-off-by: Srikanth Govindarajan <[email protected]>

  • Add startsWith expression function (#4840)

    Taylor Gray - Fri, 16 Aug 2024 16:37:44 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Logging improvements when failing to parse JSON/XML/ION. Do not include the stack trace since it doesn't provide any value with these exceptions which are expected when the JSON is invalid. Log the input string rather than the Event which was not readable. (#4839)

    David Venable - Fri, 16 Aug 2024 05:52:46 -0700

    Signed-off-by: David Venable <[email protected]>

  • Translate proc optimizations (#4824)

    Krishna Kondaka - Thu, 15 Aug 2024 10:25:43 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Data Prepper expressions - Set operator fix (#4818)

    Krishna Kondaka - Wed, 14 Aug 2024 14:05:26 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Http chunking fixes (#4823)

    Krishna Kondaka - Wed, 14 Aug 2024 11:49:12 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • FIX: include schema cli into release (#4833)

    Qi Chen - Wed, 14 Aug 2024 09:38:44 -0700

    MAINT: include schema cli into release Signed-off-by: George Chen <[email protected]>

  • PersonalizeSink: add client and configuration classes (#4803)

    Ivan Tse - Wed, 14 Aug 2024 09:36:12 -0700

    PersonalizeSink: add client and configuration classes Signed-off-by: Ivan Tse <[email protected]>

  • Config description changes for aggregate and anomaly detector processors. (#4829)

    Krishna Kondaka - Tue, 13 Aug 2024 13:41:22 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Add delete_source parameter to the csv processor (#4828)

    Taylor Gray - Mon, 12 Aug 2024 16:21:05 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • FIX: build service map relationship even when trace group is missing (#4822)

    Qi Chen - Mon, 12 Aug 2024 13:24:50 -0500

    Signed-off-by: George Chen <[email protected]>

  • ENH: add folder path as output for schema generation (#4820)

    Qi Chen - Mon, 12 Aug 2024 11:51:51 -0500

  • Release notes for Data Prepper 2.8.1 (#4807)

    David Venable - Mon, 12 Aug 2024 09:25:22 -0700

    Signed-off-by: David Venable <[email protected]>

  • Create docker-compose-dataprepper.yaml (#4756)

    Jayesh Parmar - Mon, 12 Aug 2024 08:56:55 -0700

    • Create docker-compose-dataprepper.yaml Signed-off-by: Jayesh Parmar <[email protected]>

    Signed-off-by: jayeshjeh <[email protected]>

    Signed-off-by: Jayesh Parmar <[email protected]>

    Signed-off-by: jayeshjeh <[email protected]>

  • Fixes a regex expression bug. When the left-hand side of the operation is null, always return false rather than throwing an exception. Resolves #4763. (#4798)

    David Venable - Mon, 12 Aug 2024 08:20:48 -0700

    Signed-off-by: David Venable <[email protected]>

  • Fix null document in DLQ object (#4814)

    Krishna Kondaka - Sat, 10 Aug 2024 12:04:18 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Several improvements to RDS source (#4810)

    Hai Yan - Thu, 8 Aug 2024 16:51:12 -0500

    Signed-off-by: Hai Yan <[email protected]>

  • ADD: data prepper plugin schema generation (#4777)

    Qi Chen - Fri, 2 Aug 2024 14:44:32 -0500

  • Using Awaitility and mocks in the LogGeneratorSourceTest to attempt to improve reliability. (#4746)

    David Venable - Fri, 2 Aug 2024 09:28:55 -0700

    Signed-off-by: David Venable <[email protected]>

  • Removes @asifsmohammed from the CODEOWNERS to allow the release to proceed. (#4800)

    David Venable - Thu, 1 Aug 2024 14:11:53 -0700

    Signed-off-by: David Venable <[email protected]>

  • Increase timeout in Acknowledgement IT tests (#4774)

    Krishna Kondaka - Wed, 31 Jul 2024 18:45:39 -0700

    Increase timeout for acknowledgement IT tests Signed-off-by: Krishna Kondaka <[email protected]> Signed-off-by: Krishna Kondaka <[email protected]>

    Co-authored-by: Krishna Kondaka <[email protected]>

  • Cleanup resources properly when Opensearch sink fails to initialize (#4758)

    Krishna Kondaka - Tue, 30 Jul 2024 13:33:51 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Add rds source metrics (#4769)

    Hai Yan - Tue, 30 Jul 2024 12:09:20 -0500

    Signed-off-by: Hai Yan <[email protected]>

  • Reapply "Run tests on the current JVM for Java 17 & 21 / Gradle 8.8 (#4730)" (#4762) (#4771)

    David Venable - Tue, 30 Jul 2024 09:35:12 -0700

    This reverts commit 5c7d58c03059c7a753d882f5b74fa6ed32f45641. Signed-off-by: David Venable <[email protected]>

  • 4602 one way hash (#4750)

    mishavay-aws - Mon, 29 Jul 2024 18:41:20 -0700

    added capabilities for working with OneWay Hash Signed-off-by: mishavay-aws <[email protected]>

  • Corrects the TRIAGING.md with a video meeting since we currently use Chime. (#4743)

    David Venable - Mon, 29 Jul 2024 16:34:34 -0700

    Signed-off-by: David Venable <[email protected]>

  • Add json property description for list-to-map, map-to-list and user-agent processor (#4759)

    Hai Yan - Mon, 29 Jul 2024 11:32:31 -0500

    Signed-off-by: Hai Yan <[email protected]>

  • Add json property descriptions to dissect, flatten, copy_value and translate processor (#4760)

    Hai Yan - Mon, 29 Jul 2024 11:32:25 -0500

    Signed-off-by: Hai Yan <[email protected]>

  • Lambda sink refactor (#4766)

    Srikanth Govindarajan - Fri, 26 Jul 2024 22:56:36 -0500

    Signed-off-by: Srikanth Govindarajan <[email protected]>

  • A few improvements to rds source (#4765)

    Hai Yan - Fri, 26 Jul 2024 13:31:54 -0500

    Signed-off-by: Hai Yan <[email protected]>

  • Fixes a bug with HistogramAggregateAction where the startTime may be incorrect. This was discovered by a flaky test. (#4749)

    David Venable - Wed, 24 Jul 2024 23:01:21 -0700

    The HistogramAggregateAction was incorrectly using the current time as the start time for the aggregation when creating the group. The very first event's time was overridden by the current system time. If this event had the earliest time, then the overall histogram would never get the correct start time. This is fixed by removing an errant line. I also added a unit test to directly test this failure scenario. Signed-off-by: David Venable <[email protected]>

  • Add support for taking snapshots on RDS/Aurora Clusters (#4761)

    Hai Yan - Wed, 24 Jul 2024 22:52:38 -0500

    Signed-off-by: Hai Yan <[email protected]>

  • Revert "Run tests on the current JVM for Java 17 & 21 / Gradle 8.8 (#4730)" (#4762)

    Hai Yan - Tue, 23 Jul 2024 17:03:57 -0500

    This reverts commit 67f3595805f07442d8f05823c9959b50358aa4d9. Signed-off-by: Hai Yan <[email protected]>

  • Add stream processing for rds source (#4757)

    Hai Yan - Tue, 23 Jul 2024 16:30:35 -0500

    Signed-off-by: Hai Yan <[email protected]>

  • Add json description to AddEntry processor (#4752)

    Hai Yan - Tue, 23 Jul 2024 14:34:05 -0500

    Signed-off-by: Hai Yan <[email protected]>

  • doc strings for enhanced UI view auto-generation (#4755)

    Santhosh Gandhe - Mon, 22 Jul 2024 14:40:18 -0700

    Signed-off-by: Santhosh Gandhe <[email protected]>

  • MAINT: Add json property descriptions for csv processor (#4751)

    Katherine Shen - Mon, 22 Jul 2024 15:06:13 -0500

  • Improve the SQS shutdown process such that it does not prevent the pipeline from shutting down and no longer results in failures. Resolves #4575 (#4748)

    David Venable - Fri, 19 Jul 2024 11:40:57 -0700

    The previous approach to shutting down the SQS thread closed the SqsClient. However, with acknowledgments enabled, asynchronous callbacks would result in further attempts to either ChangeVisibilityTimeout or DeleteMessages. These were failing because the client was closed. Also, the threads would remain and prevent Data Prepper from correctly shutting down. With this change, we correctly stop each processing thread. Then we close the client. Additionally, the SqsWorker now checks that it is not stopped before attempting to change the message visibility or delete messages. Additionally, I found some missing test cases. Also, modifying this code and especially unit testing it is becoming more difficult, so I performed some refactoring to move message parsing out of the SqsWorker. Signed-off-by: David Venable <[email protected]>

  • MAINT: add json property descriptions for kv configs (#4747)

    Katherine Shen - Fri, 19 Jul 2024 10:04:50 -0700

    add json property descriptions for kv configs Signed-off-by: Katherine Shen <[email protected]>

  • Updates Jackson to 2.17.2. Related to #4729. (#4744)

    David Venable - Wed, 17 Jul 2024 15:49:55 -0700

    Signed-off-by: David Venable <[email protected]>

  • Updating the Python dependencies to resolve CVEs. Resolves #4715, #4713, 4714. (#4733)

    David Venable - Mon, 15 Jul 2024 11:55:11 -0700

    Signed-off-by: David Venable <[email protected]>

  • REF: service-map processor with the latest config model (#4734)

    Qi Chen - Mon, 15 Jul 2024 11:02:40 -0500

    • REF: service-map processor with the latest config model Signed-off-by: George Chen <[email protected]>

  • MAINT: add documentation in json property description for date processor (#4719)

    Qi Chen - Mon, 15 Jul 2024 10:30:14 -0500

    • MAINT: add documentation in json property description for date processor Signed-off-by: George Chen <[email protected]>

  • REF: grok processor with the latest config model (#4731)

    Qi Chen - Mon, 15 Jul 2024 10:05:10 -0500

    • REF: grok processor with the latest config model Signed-off-by: George Chen <[email protected]>

  • Load exported S3 files in RDS source (#4718)

    Hai Yan - Mon, 15 Jul 2024 09:57:24 -0500

    Signed-off-by: Hai Yan <[email protected]>

  • MAINT: backfill doc into json property for trim_string (#4728)

    Qi Chen - Sat, 13 Jul 2024 00:31:45 -0500

    • MAINT: backfill doc into json property for trim_string Signed-off-by: George Chen <[email protected]>

  • MAINT: backfill documentation in JsonPropertyDescription for split_string (#4720)

    Qi Chen - Sat, 13 Jul 2024 00:31:23 -0500

    • MAINT: add documentation in JsonPropertyDescription for split_string processor Signed-off-by: George Chen <[email protected]>

  • MAINT: backfill doc in json property description for otel_metrics (#4722)

    Qi Chen - Sat, 13 Jul 2024 00:30:56 -0500

    • MAINT: backfill doc in json property description for otel_metrics Signed-off-by: George Chen <[email protected]>

  • MAINT: add json property description into obfuscate processor (#4706)

    Qi Chen - Fri, 12 Jul 2024 22:28:10 -0500

  • Run tests on the current JVM for Java 17 & 21 / Gradle 8.8 (#4730)

    David Venable - Fri, 12 Jul 2024 15:35:02 -0700

    Run tests on the current JVM rather than always using Java 11 for the tests. This fixes a problem with our current GitHub tests where we are running against only Java 11 even though we want to run against different Java versions (11, 17, 21). Updates the Gradle version to 8.8. Fix Java 21 support in the AbstractSink by removing usage of Thread::stop which now always throws an UnsupportedOperationException. Use only microsecond precision time when comparing the times in the event_json codec. These tests are failing now on Java 17 and 21 with precision errors. Fixed a randomly failing test in BlockingBufferTests where a value 0 caused an IllegalArgumentException. Logging changes to avoid noise in the Gradle builds in GitHub. Signed-off-by: David Venable <[email protected]>

  • MAINT: backfill documentation into json description for truncate processor (#4726)

    Qi Chen - Fri, 12 Jul 2024 13:30:05 -0500

    • MAINT: backfill documentation into json description for truncate processor Signed-off-by: George Chen <[email protected]>

  • MAINT: backfill documentation into json property for substitute_string (#4727)

    Qi Chen - Fri, 12 Jul 2024 13:29:41 -0500

    • MAINT: backfill documentation into json property for substitute_string Signed-off-by: George Chen <[email protected]>

  • MAINT: backfill documentation into json description for delete_entries (#4721)

    Qi Chen - Fri, 12 Jul 2024 13:29:16 -0500

    • MAINT: backfill documentation into json description for delete_entries Signed-off-by: George Chen <[email protected]>

  • MAINT: backfill documentation in json description for otel_traces (#4724)

    Qi Chen - Fri, 12 Jul 2024 12:40:52 -0500

    • MAINT: backfill documentation in json property description for otel_traces Signed-off-by: George Chen <[email protected]>

  • MAINT: backfill documentation into json description for string_converter (#4725)

    Qi Chen - Fri, 12 Jul 2024 12:40:24 -0500

    • MAINT: backfill documentation into json description for string_converter Signed-off-by: George Chen <[email protected]>

  • Mockito 5 (#4712)

    David Venable - Thu, 11 Jul 2024 11:17:14 -0700

    Mockito 5

    • Synchronize the MetricsTestUtil methods to avoid test failures.
    • Create a copy of the collections to remove in MetricsTestUtil.
    • Updated two tests to JUnit 5 and to use mocks instead of actual metrics. Updates to MetricsTestUtil to provide clarity on NPEs. Signed-off-by: David Venable <[email protected]>

  • Updates to the AWS Lambda Sink tests to fix a flaky test. Also adds SLF4J logging for these tests. (#4723)

    David Venable - Thu, 11 Jul 2024 08:07:32 -0700

    Signed-off-by: David Venable <[email protected]>

  • Update the rename_keys and delete_entries processors to use the EventKey. (#4636)

    David Venable - Wed, 10 Jul 2024 08:50:39 -0700

    Signed-off-by: David Venable <[email protected]>

  • Update the mutate string processors to use the EventKey. #4646 (#4649)

    David Venable - Wed, 10 Jul 2024 08:50:21 -0700

    Change the source and keys properties for mutate string processors to use EventKey such that they are parsed by Data Prepper core. Also, use the TestEventFactory in the tests to avoid use of JacksonEvent directly. Removes an unused class. Signed-off-by: David Venable <[email protected]>

  • Removes Zookeeper from Data Prepper. This was a transitive dependency from Hadoop. (#4707)

    David Venable - Wed, 10 Jul 2024 08:49:20 -0700

    Signed-off-by: David Venable <[email protected]>

  • Updates the user_agent processor to use the EventKey. (#4628)

    David Venable - Tue, 9 Jul 2024 13:57:19 -0700

    Updates the user_agent processor to use the EventKey. Signed-off-by: David Venable <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]>

  • Introducing delete input configuration option for some parsers (#4702)

    Santhosh Gandhe - Thu, 4 Jul 2024 11:11:27 -0700

    • Introduced delete_source configuration option to give flexibility for the user to drop the raw source record if they don't want to propagate it downstream Signed-off-by: Santhosh Gandhe <[email protected]>

    • addressing review comments Signed-off-by: Santhosh Gandhe <[email protected]>

    • added delete_source option to other similar parser classes Signed-off-by: Santhosh Gandhe <[email protected]>

    Signed-off-by: Santhosh Gandhe <[email protected]>

  • Bump net.bytebuddy:byte-buddy in /data-prepper-plugins/opensearch (#4593)

    dependabot[bot] - Wed, 3 Jul 2024 13:49:02 -0700

    Bumps net.bytebuddy:byte-buddy from 1.14.12 to 1.14.17.

    Commits

    updated-dependencies:

    • dependency-name: net.bytebuddy:byte-buddy dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump net.bytebuddy:byte-buddy-agent in /data-prepper-plugins/opensearch (#4592)

    dependabot[bot] - Wed, 3 Jul 2024 12:56:23 -0700

    Bumps net.bytebuddy:byte-buddy-agent from 1.14.12 to 1.14.17.

    Commits

    updated-dependencies:

    • dependency-name: net.bytebuddy:byte-buddy-agent dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Key Value processor value grouping optimization (#4704)

    Krishna Kondaka - Wed, 3 Jul 2024 12:43:46 -0700

    Signed-off-by: Kondaka <[email protected]>

  • Updates to the CODE_OF_CONDUCT.md from the opensearch-project's official CODE_OF_CONDUCT.md. (#4665)

    David Venable - Wed, 3 Jul 2024 10:51:37 -0700

    Signed-off-by: David Venable <[email protected]>

  • Updates Parquet to 1.14.1 and Hadoop to 3.4.0. Make use of Gradle's version catalogue for Parquet. (#4705)

    David Venable - Wed, 3 Jul 2024 09:41:10 -0700

    Signed-off-by: David Venable <[email protected]>

  • Export to S3 in RDS source (#4664)

    Hai Yan - Wed, 3 Jul 2024 09:51:34 -0500

    Signed-off-by: Hai Yan <[email protected]>

  • Parquet codec tests fix (#4698)

    Krishna Kondaka - Tue, 2 Jul 2024 09:26:16 -0700

    Parquet codec tests fix Signed-off-by: Krishna Kondaka <[email protected]>

  • Bump org.skyscreamer:jsonassert from 1.5.1 to 1.5.3 in /data-prepper-api (#4678)

    dependabot[bot] - Tue, 2 Jul 2024 07:18:48 -0700

    Bumps org.skyscreamer:jsonassert from 1.5.1 to 1.5.3.

    Changelog

    Commits

    updated-dependencies:

    • dependency-name: org.skyscreamer:jsonassert dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump org.apache.maven:maven-artifact in /data-prepper-plugins/opensearch (#4692)

    dependabot[bot] - Mon, 1 Jul 2024 14:46:28 -0700

    Bumps org.apache.maven:maven-artifact from 3.9.6 to 3.9.8.

    updated-dependencies:

    • dependency-name: org.apache.maven:maven-artifact dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump software.amazon.awssdk:auth in /performance-test (#4685)

    dependabot[bot] - Mon, 1 Jul 2024 14:45:24 -0700

    Bumps software.amazon.awssdk:auth from 2.25.21 to 2.26.12.

    updated-dependencies:

    • dependency-name: software.amazon.awssdk:auth dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump braces from 3.0.2 to 3.0.3 in /testing/aws-testing-cdk (#4638)

    dependabot[bot] - Mon, 1 Jul 2024 14:44:18 -0700

    Bumps braces from 3.0.2 to 3.0.3.

    updated-dependencies:

    • dependency-name: braces dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump ws from 7.5.9 to 7.5.10 in /release/staging-resources-cdk (#4639)

    dependabot[bot] - Mon, 1 Jul 2024 14:43:47 -0700

    Bumps ws from 7.5.9 to 7.5.10.

    updated-dependencies:

    • dependency-name: ws dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump org.wiremock:wiremock in /data-prepper-plugins/s3-source (#4683)

    dependabot[bot] - Mon, 1 Jul 2024 14:43:01 -0700

    Bumps org.wiremock:wiremock from 3.4.2 to 3.8.0.

    updated-dependencies:

    • dependency-name: org.wiremock:wiremock dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump com.apptasticsoftware:rssreader in /data-prepper-plugins/rss-source (#4672)

    dependabot[bot] - Mon, 1 Jul 2024 14:42:31 -0700

    Bumps com.apptasticsoftware:rssreader from 3.6.0 to 3.7.0.

    updated-dependencies:

    • dependency-name: com.apptasticsoftware:rssreader dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Updates our usage of the Apache Parquet project to use their new interfaces over the old Hadoop ones. This allows use to be ready to extract Hadoop as other changes are made to the Parquet project. Remove some Hadoop transitive dependencies and make Hadoop runtime only where possible. Added a test for INT96, clean up some test files. Contributes toward #4612. (#4623)

    David Venable - Mon, 1 Jul 2024 13:54:53 -0700

    Signed-off-by: David Venable <[email protected]>

  • Adds the TRIAGING.md file to outline our triaging process (#4630)

    David Venable - Thu, 27 Jun 2024 17:47:18 -0700

    Adds the TRIAGING.md file, which outlines for the community the Data Prepper triaging process. Signed-off-by: David Venable <[email protected]>

  • Enhanced Kafka source logging through the use of MDC and better thread names for Kafka source threads. Resolves #4126. (#4663)

    David Venable - Wed, 26 Jun 2024 12:25:57 -0700

    Signed-off-by: David Venable <[email protected]>

  • Support default route option for Events that match no other route (#4662)

    Krishna Kondaka - Wed, 26 Jun 2024 07:15:04 -0700

    Support default route option for Events that match no other route Signed-off-by: Krishna Kondaka <[email protected]>

  • Updates the chunking algorithm for http source's JsonCodec to account for actual byte size. Test using Unicode characters to prove this was incorrectly chunking and verify against future changes. (#4656)

    David Venable - Tue, 25 Jun 2024 15:09:27 -0700

    Signed-off-by: David Venable <[email protected]>

  • Refactor lambda plugin (#4643)

    Srikanth Govindarajan - Tue, 25 Jun 2024 09:51:28 -0700

    Signed-off-by: Srikanth Govindarajan <[email protected]> Signed-off-by: Srikanth Govindarajan <[email protected]>

  • Fixes the loading of peer-forwarders when using multiple workers. This fixes a bug where the service_map processor would not load in a pipeline with multiple workers. Resolves #4660. (#4661)

    David Venable - Tue, 25 Jun 2024 08:07:40 -0700

    Signed-off-by: David Venable <[email protected]>

  • FIX: remove logging that includes credential info on kafka (#4659)

    Qi Chen - Tue, 25 Jun 2024 09:11:40 -0500

  • Add an option to count unique values of specified key(s) to CountAggregateAction (#4652)

    Krishna Kondaka - Mon, 24 Jun 2024 15:46:42 -0700

    Add an option to count unique values of specified key(s) to CountAggregateAction Signed-off-by: Krishna Kondaka <[email protected]>

  • MAINT: change log level for consumer properties in kafka source (#4658)

    Qi Chen - Mon, 24 Jun 2024 16:56:23 -0500

    Signed-off-by: George Chen <[email protected]>

  • Fixes performance regression with JacksonEvent put/delete operations. (#4650)

    David Venable - Fri, 21 Jun 2024 08:37:03 -0700

    With the addition of the EventKey, JacksonEvent always creates a JacksonEventKey in order to use the same code for all paths. However, when put/delete calls are made with a String key, JacksonEvent does not need the JSON Pointer. But, it is created anyway. This adds more work to the put/delete calls that have not yet migrated to the String version. This fixes regression by adding a lazy initialization option when used in JacksonEvent. We should not be lazy when used with the EventKeyFactory since we may lose some up-front validations. Signed-off-by: David Venable <[email protected]>

  • add remove_brackets option to flatten-processor (#4616) (#4653)

    timo-mue - Fri, 21 Jun 2024 10:04:04 -0500

    Signed-off-by: Timo Mueller <[email protected]>

  • Add support to configure metric name for count and histogram actions (#4642)

    Krishna Kondaka - Thu, 20 Jun 2024 11:05:40 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Support plugins defining the EventKey in the plugin configuration classes. Data Prepper will deserialize the EventKey from the pipeline configuration and validate @NotEmpty validations. Builds on the #1916. (#4635)

    David Venable - Wed, 19 Jun 2024 15:08:12 -0700

    Signed-off-by: David Venable <[email protected]>

  • DocumentDB Source improvements (#4645)

    Dinu John - Wed, 19 Jun 2024 16:09:54 -0500

    • Extend the export partition ownership during query partition creation Signed-off-by: Dinu John <[email protected]>

    • Add support to shutdown task refresher that starts export and stream scheduler/worker on data prepper shutdown Signed-off-by: Dinu John <[email protected]>

    • Add AcknowledgmentStatus enum and code refactor to fail negative ack right away Signed-off-by: Dinu John <[email protected]>

    Signed-off-by: Dinu John <[email protected]>

  • Track last scan time from before scan starts instead of based on last Modified of objects (#4493)

    Taylor Gray - Wed, 19 Jun 2024 14:55:10 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Bump urllib3 in /release/smoke-tests/otel-span-exporter (#4640)

    dependabot[bot] - Wed, 19 Jun 2024 07:23:44 -0700

    Bumps urllib3 from 1.26.18 to 1.26.19.

    updated-dependencies:

    • dependency-name: urllib3 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump urllib3 in /examples/trace-analytics-sample-app/sample-app (#4631)

    dependabot[bot] - Tue, 18 Jun 2024 16:50:42 -0700

    Bumps urllib3 from 2.0.7 to 2.2.2.

    updated-dependencies:

    • dependency-name: urllib3 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Implements equals, hashCode, and toString for JacksonEventKey. (#4633)

    David Venable - Tue, 18 Jun 2024 16:49:34 -0700

    Signed-off-by: David Venable <[email protected]>

  • Deprecates PluginSetting which should not be used for plugins anymore. (#4624)

    David Venable - Mon, 17 Jun 2024 12:55:38 -0700

    Signed-off-by: David Venable <[email protected]>

  • Add Aggregate event handle (#4625)

    Krishna Kondaka - Mon, 17 Jun 2024 12:34:33 -0700

    Aggregate event handle Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Adds the EventKey and EventKeyFactory. (#4627)

    David Venable - Mon, 17 Jun 2024 08:40:28 -0700

    Adds the EventKey and EventKeyFactory. Resolves #1916. Signed-off-by: David Venable <[email protected]>

  • Updates to Armeria 1.29.0 which fixes a bug that may help with #4080. (#4629)

    David Venable - Mon, 17 Jun 2024 07:26:05 -0700

    Signed-off-by: David Venable <[email protected]>

  • MAINT: bump io.confluent:* packages to match org.apache.kafka.* (#4626)

    Qi Chen - Fri, 14 Jun 2024 11:55:28 -0500

    Signed-off-by: George Chen <[email protected]>

  • Support multiple aggregate processors in local mode (#4574)

    Krishna Kondaka - Fri, 14 Jun 2024 09:27:17 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Tools to generate User Agent strings in the performance-test project (#4620)

    David Venable - Fri, 14 Jun 2024 09:00:52 -0700

    Changes to the performance-test project to generate User Agent strings. Used to help reproduce and test #4618. Signed-off-by: David Venable <[email protected]>

  • Rebased to latest (#4614)

    Krishna Kondaka - Fri, 14 Jun 2024 08:37:55 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • OpenSearch Sink add support for sending pipeline parameter in BulkRequest (#4609)

    Souvik Bose - Thu, 13 Jun 2024 23:36:00 -0700

    • Add support in OpenSearch sink to pass pipeline parameter in the index and create action operation for bulk api requests Signed-off-by: Souvik Bose <[email protected]>

    • Add more unit tests to increase code coverage Signed-off-by: Souvik Bose <[email protected]>

    • Update README Signed-off-by: Souvik Bose <[email protected]>

    • Fix the OpenSearch Integration tests Signed-off-by: Souvik Bose <[email protected]>

    Signed-off-by: Souvik Bose <[email protected]>

  • Caffeine-based caching parser for the user_agent processor (#4619)

    David Venable - Thu, 13 Jun 2024 08:22:15 -0700

    Adds and uses a Caffeine-based caching parser for the user_agent processor. Resolves #4618 Signed-off-by: David Venable <[email protected]>

  • FIX: decouple msk auth from glue auth in KafkaSource (#4613)

    Qi Chen - Wed, 12 Jun 2024 14:46:08 -0500

  • Fix missing closing parenthesis in CLOUDFRONT_ACCESS_LOG pattern (#4607)

    Joël Marty - Wed, 12 Jun 2024 10:45:37 -0700

    Signed-off-by: Joël Marty <[email protected]>

  • Aggrerate processor : add option to allow raw events (#4598)

    Krishna Kondaka - Mon, 10 Jun 2024 14:55:09 -0700

    • Aggregate Processor: Add support to allow raw events Signed-off-by: Krishna Kondaka <[email protected]>

    • Modify test to check for aggregated tag Signed-off-by: Krishna Kondaka <[email protected]>

    • Addressed review comments Signed-off-by: Krishna Kondaka <[email protected]>

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Updates Python requests to 2.32.3 in the smoke tests project to address CVE-2024-35195. (#4610)

    David Venable - Fri, 7 Jun 2024 14:07:03 -0700

    Signed-off-by: David Venable <[email protected]>

  • Add support for lambda sink (#4292)

    Srikanth Govindarajan - Thu, 6 Jun 2024 20:17:27 +0000

    Signed-off-by: srigovs <[email protected]> Signed-off-by: Srikanth Govindarajan <[email protected]>

  • Fix KeyValue Processor value grouping bug (#4606)

    Krishna Kondaka - Thu, 6 Jun 2024 12:25:36 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Refactor http source functionality for supporting a new OpenSearch API source in DataPrepper (#4570)

    Souvik Bose - Tue, 4 Jun 2024 16:49:07 -0700

    Refactor http source configuration to a separate http source common package. Signed-off-by: Souvik Bose <[email protected]>

  • Modify Key Value processor to support string literal grouping (#4599)

    Krishna Kondaka - Tue, 4 Jun 2024 14:19:48 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Introduced BigDecimalConverter (#4557)

    Santhosh Gandhe - Tue, 4 Jun 2024 13:40:46 -0700

    • Introduced BigDecimalConverter that users can use as part of convert_entry_type processor that currently exists. Optionally, users can also specify required scaling needed on the converted Signed-off-by: Santhosh Gandhe <[email protected]>

    • Added Test case for the newly introduced class. Removed * imports as per the review comment Signed-off-by: Santhosh Gandhe <[email protected]>

    • Avoiding using a deprecated method. Added additional test cases Signed-off-by: Santhosh Gandhe <[email protected]>

    • Additional tests to increase the coverage Signed-off-by: Santhosh Gandhe <[email protected]>

    • removed "scale" being the state of BigDecimal converter. We are now passing the scale while converting the instance only when the instance is BigDecimalConverter Signed-off-by: Santhosh Gandhe <[email protected]>

    • test case fix to be inline with the previous commit Signed-off-by: Santhosh Gandhe <[email protected]>

    • test case fix to be inline with the previous commit Signed-off-by: Santhosh Gandhe <[email protected]>

    • addressing review comments Signed-off-by: Santhosh Gandhe <[email protected]>

    • renaming bigdecimal to big_decimal Signed-off-by: Santhosh Gandhe <[email protected]>

    • Introduced ConverterArguments as a way to pass additional arguments to the converter and avoided conditional statement for calling converter methods Signed-off-by: Santhosh Gandhe <[email protected]>

    • Added additional override convert method to reduce the changes across the code Signed-off-by: Santhosh Gandhe <[email protected]>

    • additional Test cases to increase the coverage Signed-off-by: Santhosh Gandhe <[email protected]>

    • added additional tests for converter cases Signed-off-by: Santhosh Gandhe <[email protected]>

    Signed-off-by: Santhosh Gandhe <[email protected]> Signed-off-by: Santhosh Gandhe <[email protected]>

  • Add Rds source config (#4573)

    Hai Yan - Tue, 4 Jun 2024 15:19:35 -0500

    Signed-off-by: Hai Yan <[email protected]>

  • Track the source of request for Kafka server (#4572)

    Dinu John - Fri, 31 May 2024 09:46:28 -0500

    Signed-off-by: Dinu John <[email protected]>

  • Add support for Kafka headers and timestamp in the Kafka Source (#4566)

    Krishna Kondaka - Thu, 30 May 2024 09:31:47 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Add default role and region configuration to the data-prepper-config.yaml via extensions (#4559)

    Taylor Gray - Wed, 22 May 2024 12:50:07 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Corrected the release date for 2.8.0. (#4555)

    David Venable - Mon, 20 May 2024 13:30:25 -0700

    Signed-off-by: David Venable <[email protected]>

  • Release Notes for version 2.8 (#4538)

    Krishna Kondaka - Mon, 20 May 2024 12:46:53 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Addressed review comments (#4552)

    Krishna Kondaka - Mon, 20 May 2024 11:48:31 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Fix DocDB export and stream processing self recovery with invalid database or collection name (#4553)

    Dinu John - Mon, 20 May 2024 13:15:55 -0500

    • Fix DocDB export and stream processing self recovery with invalid database or collection name Signed-off-by: Dinu John <[email protected]>

    • Fix unit test Signed-off-by: Dinu John <[email protected]>

    Signed-off-by: Dinu John <[email protected]>

  • Change log for version 2.8 (#4539)

    Krishna Kondaka - Thu, 16 May 2024 10:03:13 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Adding 'single_word_only' option to obfuscate processor (#4476)

    Utkarsh Agarwal - Wed, 15 May 2024 15:38:36 -0700

    Adding 'single_word_only' option to obfuscate processor Signed-off-by: Utkarsh Agarwal <[email protected]>

  • Updates werkzeug to 3.0.3 in examples to fix CVE-2024-34069. Resolves #4515 (#4546)

    David Venable - Wed, 15 May 2024 15:04:53 -0700

    Signed-off-by: David Venable <[email protected]>

  • Logging update and config validation (#4541)

    Dinu John - Tue, 14 May 2024 17:57:24 -0500

    • Logging improvements for export and stream processing for DocumentDB source Signed-off-by: Dinu John <[email protected]>

    • Add validation for DocumentDB Collection Config Signed-off-by: Dinu John <[email protected]>

    Signed-off-by: Dinu John <[email protected]>

  • Add aggregate metrics (#4531)

    Hai Yan - Tue, 14 May 2024 16:30:13 -0500

    Signed-off-by: Hai Yan <[email protected]>

  • Updates the next Data Prepper version to 2.9 (#4532)

    Krishna Kondaka - Tue, 14 May 2024 14:19:56 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Adds an ndjson input codec. This reads JSON objects for ND-JSON and more lenient formats that do not have the newline. (#4533)

    David Venable - Tue, 14 May 2024 12:10:01 -0700

    Signed-off-by: David Venable <[email protected]>

  • Address route and subpipeline for pipeline tranformation (#4528)

    Srikanth Govindarajan - Mon, 13 May 2024 15:58:11 -0700

    Address route and subpipeline for pipeline tranformation Signed-off-by: srigovs <[email protected]>