Skip to content

Files

Latest commit

 

History

History
2568 lines (1585 loc) · 93.1 KB

data-prepper.change-log-2.1.0.md

File metadata and controls

2568 lines (1585 loc) · 93.1 KB

  • Add release notes for 2.1.0 (#2354)

    Hai Yan - Thu, 2 Mar 2023 13:52:16 -0600

    EAD -> refs/heads/change-log-2.1, refs/remotes/upstream/main, refs/remotes/origin/main, refs/remotes/origin/HEAD, refs/heads/main

  • Added backoff for SQS to reduce logging (#2326)

    Asif Sohail Mohammed - Thu, 2 Mar 2023 12:14:20 -0600

  • Removed default service endpoint for otel sources (#2346)

    Asif Sohail Mohammed - Thu, 2 Mar 2023 11:38:54 -0600

    • Removed default service endpoint for otel sources Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

  • Replace the java.util.* allowed pattern in the ObjectInputFilter with specific classes which are commonly used in Data Prepper. (#2351)

    David Venable - Thu, 2 Mar 2023 09:53:00 -0600

    Signed-off-by: David Venable <dlv@amazon.com>

  • Explicitly set the GitHub Actions thumbprint to resolve #2343. Updated the AWS CDK as well. (#2345)

    David Venable - Wed, 1 Mar 2023 15:40:05 -0600

    Signed-off-by: David Venable <dlv@amazon.com>

  • Updated version to 2.2 on main (#2342)

    Asif Sohail Mohammed - Wed, 1 Mar 2023 13:27:15 -0600

    Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

  • Generated THIRD-PARTY file for 062ae95 (#2344)

    opensearch-trigger-bot[bot] - Wed, 1 Mar 2023 13:26:20 -0600

    Signed-off-by: GitHub <noreply@github.com> Co-authored-by: asifsmohammed <asifsmohammed@users.noreply.github.com>

  • Use an ObjectInputFilter to serialize allow deserialization of only certain objects in peer-to-peer connections. Additionally, it refactors some application configurations to improve integration testing. Fixes #2310. (#2311)

    David Venable - Wed, 1 Mar 2023 10:06:52 -0600

    efs/heads/changelog-2.1 Signed-off-by: David Venable <dlv@amazon.com>

  • Bump org.apache.logging.log4j:log4j-bom in /data-prepper-core (#2333)

    dependabot[bot] - Wed, 1 Mar 2023 09:47:42 -0600

    Bumps org.apache.logging.log4j:log4j-bom from 2.19.0 to 2.20.0.

    updated-dependencies:

    • dependency-name: org.apache.logging.log4j:log4j-bom dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump org.assertj:assertj-core from 3.21.0 to 3.24.2 (#2331)

    dependabot[bot] - Wed, 1 Mar 2023 09:47:06 -0600

    Bumps org.assertj:assertj-core from 3.21.0 to 3.24.2.

    updated-dependencies:

    • dependency-name: org.assertj:assertj-core dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump org.junit.jupiter:junit-jupiter-api from 5.9.0 to 5.9.2 (#2332)

    dependabot[bot] - Wed, 1 Mar 2023 09:46:26 -0600

    Bumps org.junit.jupiter:junit-jupiter-api from 5.9.0 to 5.9.2.

    updated-dependencies:

    • dependency-name: org.junit.jupiter:junit-jupiter-api dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Support for path in OTel sources (#2297)

    Asif Sohail Mohammed - Tue, 28 Feb 2023 19:57:38 -0600

    Initial commit for OTel trace path changes Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com> Signed-off-by: Asif Sohail Mohammed <mdasifsohail7@gmail.com>

  • Fix grok processor to not create a new record (#2325)

    kkondaka - Tue, 28 Feb 2023 19:56:04 -0600

    Signed-off-by: Krishna Kondaka <krishkdk@amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@amazon.com>

  • Bump org.springframework:spring-context in /data-prepper-expression (#2223)

    dependabot[bot] - Mon, 27 Feb 2023 18:27:04 -0600

    efs/heads/otel-paths Bumps org.springframework:spring-context from 5.3.23 to 5.3.25.

    Commits

    updated-dependencies:

    • dependency-name: org.springframework:spring-context dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump org.springframework:spring-context in /data-prepper-core (#2214)

    dependabot[bot] - Mon, 27 Feb 2023 16:52:20 -0600

    Bumps org.springframework:spring-context from 5.3.23 to 5.3.25.

    Commits

    updated-dependencies:

    • dependency-name: org.springframework:spring-context dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump io.spring.dependency-management from 1.0.11.RELEASE to 1.1.0 (#2106)

    dependabot[bot] - Mon, 27 Feb 2023 16:48:25 -0600

    Bumps io.spring.dependency-management from 1.0.11.RELEASE to 1.1.0.

    updated-dependencies:

    • dependency-name: io.spring.dependency-management dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump com.fasterxml.jackson.datatype:jackson-datatype-jdk8 (#2217)

    dependabot[bot] - Mon, 27 Feb 2023 16:47:36 -0600

    Bumps com.fasterxml.jackson.datatype:jackson-datatype-jdk8 from 2.14.1 to 2.14.2.

    updated-dependencies:

    • dependency-name: com.fasterxml.jackson.datatype:jackson-datatype-jdk8 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Added path support for HTTP source (#2277)

    Asif Sohail Mohammed - Mon, 27 Feb 2023 15:26:58 -0600

    • Added path support for HTTP source Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

  • Updated Data Prepper base image for e2e tests (#2269)

    Asif Sohail Mohammed - Mon, 27 Feb 2023 15:08:13 -0600

    Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

  • Loop until interrupted in SqsWorker (#2306)

    Chase - Fri, 24 Feb 2023 00:10:26 -0600

    Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

  • Performance Improvement - Avoid event copying for the first sub-pipeline (#2290)

    kkondaka - Thu, 23 Feb 2023 22:08:47 -0600

    Avoid event copying for the first sub-pipeline Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

  • Bump slf4j-simple from 1.7.36 to 2.0.6 (#2112)

    dependabot[bot] - Wed, 22 Feb 2023 10:39:21 -0600

    Bumps slf4j-simple from 1.7.36 to 2.0.6.

    updated-dependencies:

    • dependency-name: org.slf4j:slf4j-simple dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Upgrades opentelemetry dependencies (#2288)

    Shivani Shukla - Tue, 21 Feb 2023 21:16:58 -0600

    Signed-off-by: Shivani Shukla <sshkamz@amazon.com>

  • Fix error message printed when OpenSearch Sink fails to initialize (#2296)

    kkondaka - Tue, 21 Feb 2023 14:38:36 -0600

    Fix error message printed when OpenSearch Sink fails to initialize Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    Signed-off-by: Krishna Kondaka <krishkdk@amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@amazon.com>

  • adding backwards compatibility support in versioning check (#2295)

    Christopher Manning - Tue, 21 Feb 2023 14:13:50 -0600

    Signed-off-by: Christopher Manning <cmanning09@users.noreply.github.com>

  • [2263] adding versioning property to pipeline yaml configuration (#2292)

    Christopher Manning - Tue, 21 Feb 2023 09:08:39 -0600

    Signed-off-by: Christopher Manning <cmanning09@users.noreply.github.com>

  • Updates to opensearch-java 2.2.0 (#2287)

    David Venable - Mon, 20 Feb 2023 15:18:55 -0600

    Signed-off-by: David Venable <dlv@amazon.com>

  • Replace an unnecessary builder pattern in trace raw processor with a factory method. This is a little simpler code and it reduces an object creation. (#2271)

    David Venable - Fri, 17 Feb 2023 14:34:21 -0600

    Signed-off-by: David Venable <dlv@amazon.com>

  • Create DataPrepper server after pipeline initialization (#2284)

    Chase - Fri, 17 Feb 2023 11:12:08 -0600

    • Create DataPrepper HttpServer after pipeline initialization Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

    • Remove injection provider implementation Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

    Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

  • Refactor PeerForwarderReceiveBuffer to extend AbstractBuffer to pick up metric logic (#2286)

    Chase - Fri, 17 Feb 2023 10:48:29 -0600

    Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

  • Add more metrics to OTEL metrics source (#2283)

    Chase - Fri, 17 Feb 2023 10:24:30 -0600

    Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

  • Make batching queue depth configurable, fill in some missing documentation around batching (#2278)

    Chase - Fri, 17 Feb 2023 10:24:06 -0600

    • Make batching queue depth configurable, fill in some missing documentation around batching Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

    • Fix validation message Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

    • Actually fix validation message Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

    Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

  • Update to opensearch 1.3.8. Resolves #2192, #2193 (#2285)

    David Venable - Thu, 16 Feb 2023 15:56:36 -0600

    Signed-off-by: David Venable <dlv@amazon.com>

  • OTel Trace Source: Logging and integration tests (#2262)

    David Venable - Thu, 16 Feb 2023 11:12:26 -0600

    Modified some of the logging for the OTel Trace source. Added integration tests which actually verify that gRPC requests operate as expected. Print out SLF4J logs from tests to stdout to help with debugging. Signed-off-by: David Venable <dlv@amazon.com>

  • Created untriaged issue workflow. (#2276)

    Daniel (dB.) Doubrovkine - Wed, 15 Feb 2023 13:15:53 -0600

    Signed-off-by: dblock <dblock@amazon.com>

  • Long int fix (#2265)

    kkondaka - Tue, 14 Feb 2023 16:13:20 -0600

    • OpenSearchSink should close open files before retrying initialization Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Minor fixes to data prepper plugins documentation Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Fix Long Integer comparisons Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Added more tests for long integer comparisons Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    Signed-off-by: Krishna Kondaka <krishkdk@amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@amazon.com>

  • CVE: updated guava version (#2254)

    Asif Sohail Mohammed - Tue, 14 Feb 2023 15:08:18 -0600

    Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com> Signed-off-by: Asif Sohail Mohammed <mdasifsohail7@gmail.com>

  • Update to Armeria 1.22.1 which fixes #2206. (#2274)

    David Venable - Tue, 14 Feb 2023 13:36:21 -0600

    Signed-off-by: David Venable <dlv@amazon.com>

  • Fix: e2e tests and added documentation (#2267)

    Asif Sohail Mohammed - Mon, 13 Feb 2023 14:40:27 -0600

    Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

  • Removes some unnecessary methods from ServiceMapRelationship and adds additional tests to verify behavior. (#2200)

    David Venable - Mon, 13 Feb 2023 13:45:31 -0600

    Signed-off-by: David Venable <dlv@amazon.com>

  • Created a metric for the overall JVM memory usage - both heap and non-heap. (#2266)

    David Venable - Mon, 13 Feb 2023 13:24:48 -0600

    Signed-off-by: David Venable <dlv@amazon.com>

  • Fixes index_type convention (#2261)

    Shivani Shukla - Fri, 10 Feb 2023 14:47:20 -0600

    Signed-off-by: Shivani Shukla <sshkamz@amazon.com>

  • Minor fixes to data prepper plugins documentation (#2260)

    kkondaka - Fri, 10 Feb 2023 13:39:12 -0600

    • OpenSearchSink should close open files before retrying initialization Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Minor fixes to data prepper plugins documentation Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    Signed-off-by: Krishna Kondaka <krishkdk@amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@amazon.com>

  • ENH: peer forwarding codec and model (#2256)

    Qi Chen - Fri, 10 Feb 2023 09:19:51 -0600

    Signed-off-by: George Chen <qchea@amazon.com>

  • Use the Gradle version catalog for common software versions to the extent that it can replace the versionMap. (#2253)

    David Venable - Thu, 9 Feb 2023 13:49:11 -0600

    Signed-off-by: David Venable <dlv@amazon.com>

  • OpenSearchSink should close open files before retrying initialization (#2255)

    kkondaka - Thu, 9 Feb 2023 11:15:19 -0600

    Signed-off-by: Krishna Kondaka <krishkdk@amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@amazon.com>

  • Added trace peer forwarder doc (#2245)

    Asif Sohail Mohammed - Wed, 8 Feb 2023 18:10:26 -0600

  • Refactor to remove stream in RemotePeerForwarder as micro optimization (#2250)

    Chase - Wed, 8 Feb 2023 18:09:17 -0600

    • Refactor to remove stream in RemotePeerForwarder as micro optimization Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

    • Remove unused import Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

    Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

  • Bump org.assertj:assertj-core (#2221)

    dependabot[bot] - Wed, 8 Feb 2023 16:01:58 -0600

    Bumps org.assertj:assertj-core from 3.23.1 to 3.24.2.

    updated-dependencies:

    • dependency-name: org.assertj:assertj-core dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump certifi in /release/smoke-tests/otel-span-exporter (#2063)

    dependabot[bot] - Wed, 8 Feb 2023 15:51:47 -0600

    Bumps certifi from 2021.10.8 to 2022.12.7.

    Commits

    updated-dependencies:

    • dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump com.google.guava:guava in /data-prepper-plugins/aggregate-processor (#2219)

    dependabot[bot] - Wed, 8 Feb 2023 15:50:29 -0600

    Bumps com.google.guava:guava from 10.0.1 to 23.0.

    updated-dependencies:

    • dependency-name: com.google.guava:guava dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump org.assertj:assertj-core (#2226)

    dependabot[bot] - Wed, 8 Feb 2023 15:42:11 -0600

    Bumps org.assertj:assertj-core from 3.23.1 to 3.24.2.

    updated-dependencies:

    • dependency-name: org.assertj:assertj-core dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump org.springframework:spring-test in /data-prepper-expression (#2224)

    dependabot[bot] - Wed, 8 Feb 2023 14:53:37 -0600

    Bumps org.springframework:spring-test from 5.3.23 to 5.3.25.

    Commits

    updated-dependencies:

    • dependency-name: org.springframework:spring-test dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Moved Random Cut Forest libraries to the Anomaly Detection processor. (#2252)

    David Venable - Wed, 8 Feb 2023 11:44:14 -0600

    Signed-off-by: David Venable <dlv@amazon.com>

  • Sink doInitialize() should be able throw exception to stop execution (#2249)

    kkondaka - Wed, 8 Feb 2023 09:53:30 -0600

    • Sink doInitialize() should be able throw exception to stop execution Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Sink doInitialize() should be able throw exception to stop execution - fixed code coverage failures Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    Signed-off-by: Krishna Kondaka <krishkdk@amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@amazon.com>

  • Consistent AWS Pipeline Configurations - #2184 (#2248)

    kkondaka - Tue, 7 Feb 2023 15:58:59 -0600

    • Consistent AWS Pipeline Configurations - #2184 Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Consistent AWS Pipeline Configurations - updated documentation Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Consistent AWS Pipeline Configurations - fixed documentation Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    Signed-off-by: Krishna Kondaka <krishkdk@amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@amazon.com>

  • Made some changes to the :release:archives:linux:linuxTar task to reduce dependencies and get a better build time when building multiple times. (#2240)

    David Venable - Tue, 7 Feb 2023 14:30:30 -0600

    Signed-off-by: David Venable <dlv@amazon.com>

  • Added SDK client metrics and metric filters (#2232)

    Asif Sohail Mohammed - Tue, 7 Feb 2023 14:04:10 -0600

    • Added SDK client metrics and metric filters Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

  • Improved how the PipelineConnector copies events by using a new static method to perform the copy. For JacksonEvent, this performs a deepCopy() which appears to be more efficient than the old process. (#2241)

    David Venable - Tue, 7 Feb 2023 10:06:18 -0600

    Signed-off-by: David Venable <dlv@amazon.com>

  • Use snake case for all configurations #2203 (#2243)

    kkondaka - Mon, 6 Feb 2023 17:30:14 -0600

    • Use snake case for all configurations #2203 Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

  • Add back LinkedBlockingQueue::poll to avoid busy wait (#2246)

    Chase - Mon, 6 Feb 2023 14:12:32 -0600

    • Add back LinkedBlockingQueue::poll to avoid busy wait Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

    • Drop poll timeout when delay=0 to 5 millis Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

    Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

  • Use Armeria's BlockingTaskExecutor to configure thread names (#2235)

    David Venable - Mon, 6 Feb 2023 09:32:47 -0600

    Use Armeria's BlockingTaskExecutor to configure thread name Signed-off-by: David Venable <dlv@amazon.com>

  • Update default values in buffer documentation (#2233)

    JannikBrand - Sat, 4 Feb 2023 14:04:56 -0600

  • The DefaultEventMetadata almost always creates an empty attributes. This uses the static ImmutableMap.of() which uses a shared instance underneath so that new objects are not created each time. (#2239)

    David Venable - Fri, 3 Feb 2023 13:21:31 -0600

    Signed-off-by: David Venable <dlv@amazon.com>

  • The RecordMetadata default metadata is used by most records and it creates a new instance each time. This is an immutable class, so just return the same metadata instance for all. (#2238)

    David Venable - Fri, 3 Feb 2023 12:47:25 -0600

    Signed-off-by: David Venable <dlv@amazon.com>

  • Abstract sink should not create a new retry thread everytime initialization fails (#2231)

    kkondaka - Thu, 2 Feb 2023 10:25:41 -0600

    Signed-off-by: Krishna Kondaka <krishkdk@amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@amazon.com>

  • Add Support for retry when Sink fails to initialize (#2198)

    kkondaka - Wed, 1 Feb 2023 16:39:50 -0600

    • Remove opensearch availability dependence - Issue #936 Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Add support for retry when Sink fails to initialize - issue #936 Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Add support for retry when Sink fails to initialize - added number of retries to dataprepper execute Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Add support for retry when Sink fails to initialize - Addressed review comments. Fixed test failures Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Add support for retry when Sink fails to initialize - Addressed review comments Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Add support for retry when Sink fails to initialize - Fixes for failing tests

    Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Add support for retry when Sink fails to initialize - Fixes for failing tests

    Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Add support for retry when Sink fails to initialize - Fixes for code coverage failures Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Add support for retry when Sink fails to initialize - Modified to check for retryable exception Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Add support for retry when Sink fails to initialize - addressed review comments Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    Signed-off-by: Krishna Kondaka <krishkdk@amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@amazon.com>

  • DynamicIndexTemplate can cause NPE that shuts down pipeline - Issue #2210 (#2211)

    kkondaka - Wed, 1 Feb 2023 15:15:01 -0600

    • DynamicIndexTemplate can cause NPE that shuts down pipeline - Issue #2210 Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • DynamicIndexTemplate can cause NPE that shuts down pipeline - addressed review comments Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • DynamicIndexTemplate can cause NPE that shuts down pipeline - addressed review comments Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • DynamicIndexTemplate can cause NPE that shuts down pipeline - fixed build failure Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    Signed-off-by: Krishna Kondaka <krishkdk@amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@amazon.com>

  • Add missing metrics for Opensearch Sink #2168 (#2205)

    kkondaka - Wed, 1 Feb 2023 15:10:21 -0600

    • Add missing metrics for Opensearch Sink #2168 Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Add missing metrics for Opensearch Sink -- updated the documentation Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Add missing metrics for Opensearch Sink -- addressed review comments Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Add missing metrics for Opensearch Sink -- addressed review comments Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Add missing metrics for Opensearch Sink -- modified documentation Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Add missing metrics for Opensearch Sink -- addressed review comments Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Add missing metrics for Opensearch Sink -- fixed build failure Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    Signed-off-by: Krishna Kondaka <krishkdk@amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@amazon.com>

  • Perform serialization sequentially, block for each batch of forwards (#2228)

    Chase - Wed, 1 Feb 2023 11:04:43 -0600

    • Perform serialization sequentially, block for each batch of forwards Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

    • Add test for mixed future results Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

    • Rename forwardRecords to forwardBatchedRecords to avoid naming duplication Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

    • Add missing log statement Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

    Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

  • Removed getObjectAttributes API call to check object size (#2179)

    Asif Sohail Mohammed - Wed, 1 Feb 2023 10:16:49 -0600

    • Removed getObjectAttributes API call to check object size Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

  • Bump org.springframework:spring-core in /data-prepper-expression (#2222)

    dependabot[bot] - Tue, 31 Jan 2023 21:00:33 -0600

    Bumps org.springframework:spring-core from 5.3.23 to 5.3.25.

    Commits

    updated-dependencies:

    • dependency-name: org.springframework:spring-core dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump org.assertj:assertj-core (#2225)

    dependabot[bot] - Tue, 31 Jan 2023 20:46:44 -0600

    Bumps org.assertj:assertj-core from 3.23.1 to 3.24.2.

    updated-dependencies:

    • dependency-name: org.assertj:assertj-core dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump org.assertj:assertj-core in /data-prepper-plugins/http-source (#2227)

    dependabot[bot] - Tue, 31 Jan 2023 20:46:18 -0600

    Bumps org.assertj:assertj-core from 3.23.1 to 3.24.2.

    updated-dependencies:

    • dependency-name: org.assertj:assertj-core dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump org.assertj:assertj-core in /data-prepper-plugins/otel-trace-source (#2212)

    dependabot[bot] - Tue, 31 Jan 2023 20:45:50 -0600

    Bumps org.assertj:assertj-core from 3.23.1 to 3.24.2.

    updated-dependencies:

    • dependency-name: org.assertj:assertj-core dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump org.springframework:spring-test in /data-prepper-core (#2215)

    dependabot[bot] - Tue, 31 Jan 2023 20:41:19 -0600

    Bumps org.springframework:spring-test from 5.3.23 to 5.3.25.

    Commits

    updated-dependencies:

    • dependency-name: org.springframework:spring-test dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump org.springframework:spring-core in /data-prepper-core (#2216)

    dependabot[bot] - Tue, 31 Jan 2023 20:40:24 -0600

    Bumps org.springframework:spring-core from 5.3.23 to 5.3.25.

    Commits

    updated-dependencies:

    • dependency-name: org.springframework:spring-core dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Implement batching for peer forwarder request documents (#2197)

    Chase - Tue, 31 Jan 2023 16:42:25 -0600

    • Implement batching for peer forwarder request documents Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

    • Add configurable forwarding_batch_timeout for low-traffic scenarios Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

    • Slight refactors and add unit tests for batching Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

    • Increase YAML deserialization size Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

    • Refactor for clarity and flush all available batches on each iteration Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

    • Fix typo in FORWARDING Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

    • Use getOrDefault when checking last flushed time Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

    Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

  • Integration test for a pipeline with multiple process workers (#2017)

    David Venable - Tue, 31 Jan 2023 14:29:38 -0600

    Created an integration test for a pipeline with multiple process workers. Signed-off-by: David Venable <dlv@amazon.com>

  • Add clientTimeout to peer forwarder configuration, optimize CPF seria… (#2190)

    Chase - Tue, 31 Jan 2023 11:53:15 -0600

    • Add clientTimeout to peer forwarder configuration, optimize CPF serialization

    Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

    • Fix miss in rename and use lower write timeout rather than higher request timeout Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

    • Add missing private Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

    Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

  • Bump hibernate-validator in /data-prepper-core (#1854)

    dependabot[bot] - Mon, 30 Jan 2023 10:29:58 -0600

    Bumps hibernate-validator from 7.0.5.Final to 8.0.0.Final.

    Changelog

    Commits

    updated-dependencies:

    • dependency-name: org.hibernate.validator:hibernate-validator dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump jackson-datatype-jdk8 from 2.13.3 to 2.14.1 in /data-prepper-api (#2101)

    dependabot[bot] - Mon, 30 Jan 2023 10:25:40 -0600

    Bumps jackson-datatype-jdk8 from 2.13.3 to 2.14.1.

    updated-dependencies:

    • dependency-name: com.fasterxml.jackson.datatype:jackson-datatype-jdk8 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Support OpenTelementry Logs (#1372)

    kmssap - Sat, 28 Jan 2023 09:21:54 -0600

    • Move mapping from processor to source
    • Move mapping logic to OtelProtoCodec
    • Enhance metrics of LogsGrpcService Signed-off-by: Kai Sternad <ksternad@sternad.de>

    Signed-off-by: Kai Sternad <kai.sternad@sap.com> Signed-off-by: Kai Sternad <ksternad@sternad.de> Co-authored-by: Kai Sternad <kai.sternad@sap.com>

    Co-authored-by: Tomas Longo <tomas.longo@sap.com> Co-authored-by: Karsten Schnitter <k.schnitter@sap.com> Co-authored-by: Kai Sternad <ksternad@sternad.de>

  • Optimize buffer reads, allow delay to be 0 (#2189)

    Chase - Fri, 27 Jan 2023 17:11:49 -0600

    Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

  • ENH: implement custom LogEventPatternConverter for desensitization on Data Prepper logging (#2188)

    Qi Chen - Fri, 27 Jan 2023 16:27:09 -0600

    • ADD: SensitiveArgumentMaskingConverter Signed-off-by: George Chen <qchea@amazon.com>

  • Generated THIRD-PARTY file for bd60dcc (#2201)

    opensearch-trigger-bot[bot] - Fri, 27 Jan 2023 14:30:58 -0600

    Signed-off-by: GitHub <noreply@github.com> Co-authored-by: asifsmohammed <asifsmohammed@users.noreply.github.com>

  • Updated OpenSearch version to 1.3.7 (#2191)

    Asif Sohail Mohammed - Fri, 27 Jan 2023 09:24:13 -0600

    Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com> Signed-off-by: Asif Sohail Mohammed <mdasifsohail7@gmail.com>

  • Fix Data Prepper to not terminate on invalid open telemetry metric/trace data (#2176)

    kkondaka - Thu, 26 Jan 2023 19:03:00 -0600

  • Add configurations and metrics to OTelTraceRawProcessor (#2164)

    David Venable - Thu, 26 Jan 2023 19:01:47 -0600

    Updated the OTelTraceRawProcessor to use the new plugin configuration model. Added two plugin metrics to tracking the usage of two of the collections used in this processor. Signed-off-by: David Venable <dlv@amazon.com>

  • Don't block on forwarding request response, populate records that failed to forward in local CPF buffer (#2175)

    Chase - Thu, 26 Jan 2023 17:26:14 -0600

    • Handle peer forwarding in the background Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

    • Rework tests to pass with new implementation Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

    • Small tweaks to fix the diff Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

    • Refactor to use Consumer<AggregateHttpResponse> and ExecutorService for background forwarding work Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

    • Add unit tests for processFailedRequestsLocally Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

    • Add more unit tests Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

    • Move requestsReceivedFromPeers to HttpService to avoid gauge usage Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

    • Coming full circle + an ExecutorService Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

    • Add missing private modifier Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

    • Reuse existing clientThreadCount parameter, cleanup unused code Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

    • Fix log message Signed-off-by: Chase Engelbrecht <engechas@amazon.com> Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

  • Add support to convert metrics to json strings without flattening attributes - issue #2146 (#2163)

    kkondaka - Thu, 26 Jan 2023 15:45:02 -0600

    Add support to convert metrics to json strings without flattening attributes field - issue #2146 Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

  • ENH: add and populate markers (#2180)

    Qi Chen - Thu, 26 Jan 2023 11:31:25 -0600

  • Fixes links to use org/opensearch from com/amazon now that this is the correct the package name. (#2186)

    David Venable - Wed, 25 Jan 2023 08:29:54 -0600

    Signed-off-by: David Venable <dlv@amazon.com>

  • Added AWS STS header override configurations for OpenSearch sink/S3 source (#1898)

    David Venable - Tue, 24 Jan 2023 17:17:13 -0600

    Added AWS STS header override configurations for the OpenSearch sink and the Amazon S3 source. Resolves #1888. Signed-off-by: David Venable <dlv@amazon.com>

  • ENH: add buffer records overflow metrics (#2170)

    Qi Chen - Tue, 24 Jan 2023 09:51:27 -0600

    • ENH: add buffer records overflow metrics Signed-off-by: George Chen <qchea@amazon.com>

  • Bump decode-uri-component in /release/staging-resources-cdk (#2064)

    dependabot[bot] - Mon, 23 Jan 2023 19:49:14 -0600

    Bumps decode-uri-component from 0.2.0 to 0.2.2.

    Commits

    updated-dependencies:

    • dependency-name: decode-uri-component dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • New aggregate action - percent sampler (#2096)

    kkondaka - Mon, 23 Jan 2023 17:36:31 -0600

    Percent Sampler aggregate action Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

  • Implemented a heap-based circuit breaker (#2155)

    David Venable - Mon, 23 Jan 2023 17:11:16 -0600

    Implemented a heap-based circuit breaker. This circuit breaker will prevent entry buffers from accepting events after the heap usage reaches a specified value. This checks for heap usage in a background thread and updates the state, which the buffer will then use to determine if the circuit breaker is open or closed. This also signals to the JVM to start a GC when the threshold is reached. Resolves #2150. Signed-off-by: David Venable <dlv@amazon.com>

  • Test against multiple OTel version - Issue #1963 (#2154)

    kkondaka - Mon, 23 Jan 2023 15:08:18 -0600

  • Bump byte-buddy in /data-prepper-plugins/opensearch (#2161)

    dependabot[bot] - Sat, 21 Jan 2023 14:48:03 -0600

    Bumps byte-buddy from 1.12.18 to 1.12.22.

    Commits

    updated-dependencies:

    • dependency-name: net.bytebuddy:byte-buddy dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump byte-buddy-agent in /data-prepper-plugins/opensearch (#2160)

    dependabot[bot] - Fri, 20 Jan 2023 11:11:23 -0600

    Bumps byte-buddy-agent from 1.12.20 to 1.12.22.

    Commits

    updated-dependencies:

    • dependency-name: net.bytebuddy:byte-buddy-agent dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • New Aggregate Action - Event Rate Limiter (#2090)

    kkondaka - Thu, 19 Jan 2023 17:28:12 -0600

    Signed-off-by: Krishna Kondaka <krishkdk@amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@amazon.com>

  • Bump byte-buddy-agent in /data-prepper-plugins/opensearch (#2100)

    dependabot[bot] - Thu, 19 Jan 2023 16:22:01 -0600

    Bumps byte-buddy-agent from 1.12.18 to 1.12.20.

    Commits

    updated-dependencies:

    • dependency-name: net.bytebuddy:byte-buddy-agent dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Added s3 support in Opensearch sink (#2121)

    Asif Sohail Mohammed - Thu, 19 Jan 2023 16:21:32 -0600

    • Added implementation of s3 support in Opensearch sink Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

  • Added batch delay to CPF configuration (#2159)

    Asif Sohail Mohammed - Thu, 19 Jan 2023 13:05:26 -0600

    Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com> Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

  • Fix for null pointer exception in remote peer forwarding (fix for issue 2123) (#2124)

    kkondaka - Wed, 18 Jan 2023 20:56:57 -0600

    • Fix for null pointer exception in remote peer forwarding (fix for issue #2123) Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Addressed review comments to add a counter and not skip when an identification key is missing Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Addressed review comments. Modified to increment the counter only when all identification keys are missing Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Added 'final' to the local variable Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Addressed review comments Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Added a test with all missing keys Signed-off-by: Krishna Kondaka <krishkdk@amazon.com> Signed-off-by: Krishna Kondaka <krishkdk@amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@amazon.com>

  • Adds ScheduledExecutorService for Polling the RSS feed (#2140)

    Shivani Shukla - Wed, 18 Jan 2023 15:10:08 -0600

    • Adds ScheduledExecutor Service and runnable task Signed-off-by: Shivani Shukla <sshkamz@amazon.com>

  • Combined two integration tests for conditional routes into one. Also fixed a bug in the tests where the data was not sent to the correct sink. (#2061)

    David Venable - Tue, 17 Jan 2023 21:04:00 -0600

    Signed-off-by: David Venable <dlv@amazon.com>

  • Add e2etest for testing log metrics (#2127)

    kkondaka - Fri, 13 Jan 2023 15:19:16 -0600

  • Adds count metrics to the service_map_stateful processor (#2130)

    David Venable - Thu, 12 Jan 2023 15:17:03 -0600

    Adds new metrics to the service_map_stateful processor. These count the number of items in the collections used by the service map. These do not have byte sizes, but use object counts. Signed-off-by: David Venable <dlv@amazon.com>

  • Bump guava from 10.0.1 to 23.0 in /data-prepper-plugins/opensearch (#2050)

    dependabot[bot] - Wed, 11 Jan 2023 12:07:17 -0600

    Bumps guava from 10.0.1 to 23.0.

    updated-dependencies:

    • dependency-name: com.google.guava:guava dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump json5 from 2.2.0 to 2.2.3 in /release/staging-resources-cdk (#2119)

    dependabot[bot] - Wed, 11 Jan 2023 12:06:05 -0600

    Bumps json5 from 2.2.0 to 2.2.3.

    updated-dependencies:

    • dependency-name: json5 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Updated MAINTAINERS.md to match recommended opensearch-project format. (#2117)

    Daniel (dB.) Doubrovkine - Thu, 5 Jan 2023 14:58:26 -0600

    Signed-off-by: dblock <dblock@amazon.com> Signed-off-by: dblock <dblock@amazon.com>

  • Update OTelProtoCodec for InstrumentationLibrary to InstrumentationScope rename (#2114)

    Chase - Wed, 4 Jan 2023 09:40:06 -0600

  • Update slf4j-simple; update log4j-slf4j-impl to log4j-slf4j2-impl (#2113)

    Hai Yan - Tue, 3 Jan 2023 17:16:03 -0600

    Signed-off-by: Hai Yan <oeyh@amazon.com>

  • MOD: fix typos (#2084)

    Shanelle Marasigan - Tue, 3 Jan 2023 15:51:49 -0600

    Signed-off-by: Russianhielle Marasigan <russianhielle@gmail.com>

  • Created a GitHub Action to generate the Third Party report. (#2033)

    David Venable - Thu, 22 Dec 2022 15:01:37 -0800

    Signed-off-by: David Venable <dlv@amazon.com>

  • updating S3 source documentation to include all codecs (#2091)

    Christopher Manning - Thu, 22 Dec 2022 16:04:08 -0600

    Signed-off-by: Christopher Manning <cmanning09@users.noreply.github.com>

  • Provide a type conversion / cast processor #2010 (#2020)

    kkondaka - Thu, 22 Dec 2022 15:50:01 -0600

  • Histogram Aggregate Action - Added duration and fixed end time in the aggregated output (#2085)

    kkondaka - Thu, 22 Dec 2022 12:20:08 -0600

    Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

  • Fix CVE-2022-41881, CVE-2021-21290 and CVE-2022-41915 (#2093)

    Asif Sohail Mohammed - Wed, 21 Dec 2022 16:36:04 -0800

    Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

  • Fix: CVE-2022-3509, CVE-2022-3510 (#2079)

    Asif Sohail Mohammed - Mon, 19 Dec 2022 10:14:22 -0800

    Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

  • Add robust retry strategy to AcmClients (#2082)

    Chase - Fri, 16 Dec 2022 19:29:07 -0600

    Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

  • Updated info logs to debug level (#2083)

    Asif Sohail Mohammed - Fri, 16 Dec 2022 16:35:28 -0800

    Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

  • Fix: CVE-2022-36944 (#2080)

    Asif Sohail Mohammed - Fri, 16 Dec 2022 15:59:07 -0800

    Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

  • Histogram aggregate action (#2078)

    kkondaka - Fri, 16 Dec 2022 11:22:23 -0800

  • Updated jackson bom dependency (#2068)

    Asif Sohail Mohammed - Thu, 15 Dec 2022 14:44:59 -0800

    Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

  • Anomaly detector (#2058)

    kkondaka - Thu, 15 Dec 2022 16:28:28 -0600

    • Add support for anomaly detection in the pipeline with new anomaly detector processor Signed-off-by: Krishna Kondaka <krishkdk@amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@amazon.com>

  • Add retry strategy to StsClient used for sigv4 auth against OpenSearch sinks (#2069)

    Chase - Thu, 15 Dec 2022 11:26:50 -0600

    Add retry strategy to STS client used for sigv4 auth Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

  • Parse RSS feed URL items and convert Item to Event (#2073)

    Shivani Shukla - Wed, 14 Dec 2022 13:16:46 -0600

    Signed-off-by: Shivani Shukla <sshkamz@amazon.com>

    • Adds Jackson Jdk8Module to enable usage of Optional in Rss Item model

    Signed-off-by: Shivani Shukla <sshkamz@amazon.com>

  • Fix: Updated parse json processor documentation (#2071)

    Asif Sohail Mohammed - Wed, 14 Dec 2022 08:47:12 -0800

    Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

  • Count Aggregate Action - fix aggregation temporality (#2067)

    kkondaka - Thu, 8 Dec 2022 17:21:41 -0600

    Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

  • Add support for count aggregate action (#2034)

    kkondaka - Wed, 7 Dec 2022 12:51:11 -0600

    Add support for count aggregate action with raw and otel_metrics output format Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

  • Setup boilerplate for RSS Source Plugin (#2062)

    Shivani Shukla - Wed, 7 Dec 2022 12:44:37 -0600

    • Adds boilerplate config and code for rss source Signed-off-by: Shivani Shukla <sshkamz@amazon.com>

    • Adds RSS Source plugin class Signed-off-by: Shivani Shukla <sshkamz@amazon.com>

    • Adds Document Event Type Signed-off-by: Shivani Shukla <sshkamz@amazon.com>

    • Adds Document interface and JacksonDocument class Signed-off-by: Shivani Shukla <sshkamz@amazon.com>

    • Adds a simple unit test checking default for pollingFrequency Signed-off-by: Shivani Shukla <sshkamz@amazon.com>

  • Bump slf4j-simple in /data-prepper-logstash-configuration (#2053)

    dependabot[bot] - Fri, 2 Dec 2022 14:04:27 -0600

    Bumps slf4j-simple from 1.7.36 to 2.0.5.

    updated-dependencies:

    • dependency-name: org.slf4j:slf4j-simple dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump slf4j-api from 1.7.36 to 2.0.5 (#2057)

    dependabot[bot] - Fri, 2 Dec 2022 14:00:34 -0600

    Bumps slf4j-api from 1.7.36 to 2.0.5.

    updated-dependencies:

    • dependency-name: org.slf4j:slf4j-api dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump jsonassert from 1.5.0 to 1.5.1 in /data-prepper-api (#2043)

    dependabot[bot] - Thu, 1 Dec 2022 17:51:03 -0600

    Bumps jsonassert from 1.5.0 to 1.5.1.

    Changelog

    Commits

    updated-dependencies:

    • dependency-name: org.skyscreamer:jsonassert dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump protobuf-java-util in /data-prepper-plugins/otel-trace-raw-prepper (#2042)

    dependabot[bot] - Thu, 1 Dec 2022 17:50:08 -0600

    Bumps protobuf-java-util from 3.21.9 to 3.21.10.

    updated-dependencies:

    • dependency-name: com.google.protobuf:protobuf-java-util dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Router integration tests to verify that conditional pipeline routes work as expected. (#1988)

    David Venable - Thu, 1 Dec 2022 15:51:52 -0600

    Signed-off-by: David Venable <dlv@amazon.com>

  • Updated the OpenSearch libraries to 1.3.6 and run the integration tests against 1.3.6 instead of 1.3.5. Fixes #2022 (#2030)

    David Venable - Wed, 30 Nov 2022 12:57:39 -0600

    Signed-off-by: David Venable <dlv@amazon.com>

  • Removes double brace initialization and includes this as a checkstyle rule to prevent future use. (#2035)

    David Venable - Tue, 22 Nov 2022 16:16:14 -0600

    Signed-off-by: David Venable <dlv@amazon.com>

  • Implemented additional metrics for the S3 source. (#2028)

    David Venable - Tue, 22 Nov 2022 14:25:20 -0600

    Implemented additional metrics for the S3 source. Resolves #2024 Signed-off-by: David Venable <dlv@amazon.com>

  • Use the same compression engines for GZIP and NONE in the AUTOMATIC compression engine. Fixes #2026. (#2027)

    David Venable - Mon, 21 Nov 2022 11:15:05 -0600

    Signed-off-by: David Venable <dlv@amazon.com>

  • Log the number of messages received from the SQS queue, including a count of the number of messages that will need to be processed. Also, include logging of deletes at the debug level. (#2011)

    David Venable - Sat, 19 Nov 2022 13:06:23 -0600

    Signed-off-by: David Venable <dlv@amazon.com>

  • Add PluginMetrics in the Auth Plugin for Http, OTel and Metrics Source (#2023)

    Dinu John - Fri, 18 Nov 2022 21:27:15 -0600

    Signed-off-by: Dinu John <dinujohn@amazon.com>

  • Add when condition to aggregate processor (#2018)

    kkondaka - Fri, 18 Nov 2022 15:31:54 -0600

  • Dynamic Index Name in OpenSearch sink - Resolves #1459 (#1999)

    kkondaka - Thu, 17 Nov 2022 20:36:47 -0600

    • Dynamic Index Name in OpenSearch sink #1459 Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Dynamic Index Name in OpenSearch sink #1459 -- updated documentation Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Dynamic Index Name in OpenSearch sink #1459 -- fixed a bug and increased test coverage Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Addressed review comments to add more tests and re-design index manager to accommodate dynamic indexes Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Added tests for DynamicIndexManager Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Dynamic Index Name in OpenSearch sink #1459 -- changed cache weigher to have constant value Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Dynamic Index Name in OpenSearch sink #1459 -- addressed review comments Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Dynamic Index Name in OpenSearch sink - Resolves Issue #1459 -- addressed review comments Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Dynamic Index Name in OpenSearch sink - Fixed checkSytleMain issues in opensearch Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Dynamic Index Name in OpenSearch sink - Fixed spotlessJavaCheck issues in opensearch Signed-off-by: Krishna Kondaka <krishkdk@amazon.com> Signed-off-by: Krishna Kondaka <krishkdk@amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@amazon.com>

  • updated samples for dataperpper 2 (#2019)

    Arunachalam Lakshmanan - Thu, 17 Nov 2022 10:02:31 -0600

  • Add IntelliJ's .iml file extension to the .gitignore. (#2016)

    David Venable - Wed, 16 Nov 2022 14:15:00 -0600

    Signed-off-by: David Venable <dlv@amazon.com>

  • Bump protobuf-java-util in /data-prepper-plugins/otel-trace-raw-prepper (#2001)

    dependabot[bot] - Fri, 11 Nov 2022 09:58:40 -0600

    Bumps protobuf-java-util from 3.19.4 to 3.21.9.

    updated-dependencies:

    • dependency-name: com.google.protobuf:protobuf-java-util dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Data Prepper Core integration tests (#1949)

    David Venable - Thu, 10 Nov 2022 09:50:03 -0600

    Creates an integration test source set for data-prepper-core and adds a small framework for running integration tests on data-prepper-core functionality. Signed-off-by: David Venable <dlv@amazon.com>

  • Adding Chase Engelbrecht to the MAINTAINERS.md. (#2005)

    David Venable - Wed, 9 Nov 2022 09:55:59 -0600

    Signed-off-by: David Venable <dlv@amazon.com>

  • Adding Hai Yan to the maintainers. (#2004)

    David Venable - Tue, 8 Nov 2022 16:13:31 -0600

    Signed-off-by: David Venable <dlv@amazon.com>

  • Bump opensearch-java from 1.0.0 to 2.1.0 (#1733)

    dependabot[bot] - Tue, 8 Nov 2022 08:17:20 -0600

    Bumps opensearch-java from 1.0.0 to 2.1.0.

    Commits

    updated-dependencies:

    • dependency-name: org.opensearch.client:opensearch-java dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump assertj-core in /data-prepper-plugins/otel-trace-raw-prepper (#2003)

    dependabot[bot] - Tue, 8 Nov 2022 08:15:44 -0600

    Bumps assertj-core from 3.22.0 to 3.23.1.

    Commits

    updated-dependencies:

    • dependency-name: org.assertj:assertj-core dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump awaitility in /data-prepper-plugins/otel-trace-raw-prepper (#2002)

    dependabot[bot] - Tue, 8 Nov 2022 08:15:12 -0600

    Bumps awaitility from 4.1.1 to 4.2.0.

    Changelog

    Commits

    updated-dependencies:

    • dependency-name: org.awaitility:awaitility dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Support remaining OpenTelemetry Metrics proto spec features (#1335)

    kmssap - Fri, 4 Nov 2022 10:58:14 -0500

    • Address review comments
    • Remove unused import, breaking Checkstyle
    • Change Exponential Histogram filter
    • Add lenient to some Mockito calls
    • Clarify metrics processor documentation Signed-off-by: Kai Sternad <kai.sternad@sap.com> Co-authored-by: Tomas Longo <tomas.longo@sap.com> Co-authored-by: Karsten Schnitter <k.schnitter@sap.com>

    Co-authored-by: Karsten Schnitter <k.schnitter@sap.com>

  • Bump byte-buddy in /data-prepper-plugins/opensearch (#1993)

    dependabot[bot] - Wed, 2 Nov 2022 20:22:12 -0500

    Bumps byte-buddy from 1.12.17 to 1.12.18.

    Commits

    updated-dependencies:

    • dependency-name: net.bytebuddy:byte-buddy dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump byte-buddy-agent in /data-prepper-plugins/opensearch (#1992)

    dependabot[bot] - Wed, 2 Nov 2022 19:20:41 -0500

    Bumps byte-buddy-agent from 1.12.17 to 1.12.18.

    Commits

    updated-dependencies:

    • dependency-name: net.bytebuddy:byte-buddy-agent dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Fixes for issues #1456 and #1458 - support for complex document ID and routing ID (#1966)

    kkondaka - Wed, 2 Nov 2022 18:57:46 -0500

    • Fixes for issues #1456 and #1458 - support for complex document ID and routing ID Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Fixes for issues #1456 and #1458 - updated README.md and added unit tests Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Fixes for issues #1456 and #1458 - addressed review comments Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Fixes for issues #1456 and #1458 - fixed check style build failure Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Add OpenSearch e2e test to github workflows Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Addressed review comments. Deleted e2e test for OpenSearch DocumentId/RoutingField testing Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Remove e2e test entry for removed e2e test Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Addressed review comments and added tests for fromStringAndOptionals method Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Fixed checkstyle failure Signed-off-by: Krishna Kondaka <krishkdk@amazon.com> Signed-off-by: Krishna Kondaka <krishkdk@amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@amazon.com>

  • Update a few logs that were dumping the actual customer logs and traces into the DataPrepper logs on failure. Logging the request sizes instead, to avoid customer data being logged (#1989)

    Deep Datta - Mon, 31 Oct 2022 14:30:12 -0500

    Signed-off-by: Deep Datta <deedatta@amazon.com>

  • Updated release notes and change log with spring change (#1981)

    Asif Sohail Mohammed - Thu, 27 Oct 2022 17:02:02 -0500

    Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

  • Created release notes for Data Prepper 2.0.1. (#1969)

    David Venable - Thu, 27 Oct 2022 13:28:47 -0500

    Signed-off-by: David Venable <dlv@amazon.com>

  • Added release notes for 1.5.2 (#1976)

    Asif Sohail Mohammed - Thu, 27 Oct 2022 12:44:57 -0500

    Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

  • Added change log for 1.5.2 (#1977)

    Asif Sohail Mohammed - Thu, 27 Oct 2022 12:28:10 -0500

    Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

  • Added the change log for 2.0.1. (#1968)

    David Venable - Thu, 27 Oct 2022 10:38:17 -0500

    Signed-off-by: David Venable <dlv@amazon.com>

  • Expressions with null (#1946)

    kkondaka - Tue, 25 Oct 2022 15:00:55 -0500

    • fix for issue #1136 Add null support to DataPrepperExpressions Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • fix for issue #1136 Add null support to DataPrepperExpressions - added more tests Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • fix for issue #1136 Add null support to DataPrepperExpressions - updated docs

    Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

  • Reduce smoke test timeout to 8 minutes from 30 minutes. These tests tend to pass within 3 minutes in my personal GitHub branch. So this leaves quite a bit of buffer time. It helps speed up retrying failures from flaky tests. (#1956)

    David Venable - Fri, 21 Oct 2022 09:48:44 -0500

    Signed-off-by: David Venable <dlv@amazon.com>

  • Attempt to reduce flakiness in RandomStringSourceTests by using awaitility. Split tests into two. JUnit 5. (#1921)

    David Venable - Fri, 21 Oct 2022 09:48:29 -0500

    Signed-off-by: David Venable <dlv@amazon.com>

  • Run smoke tests against OpenSearch 1.3.6. (#1955)

    David Venable - Fri, 21 Oct 2022 06:47:01 -0500

    Signed-off-by: David Venable <dlv@amazon.com>

  • Use Python grpcio 1.50.0 in smoke tests to reduce time to run. (#1954)

    David Venable - Fri, 21 Oct 2022 06:46:43 -0500

    Signed-off-by: David Venable <dlv@amazon.com>

  • Delete s3:TestEvent objects and log them when they are found in the SQS queue. Resolves #1924. (#1939)

    David Venable - Thu, 20 Oct 2022 12:56:56 -0500

    Signed-off-by: David Venable <dlv@amazon.com>

  • Add ExecutorService to DataPrepperServer (#1948)

    Chase - Thu, 20 Oct 2022 11:18:27 -0500

  • Updated k8s manifest to suit Data Prepper 2.0 (#1928)

    Rafael Gumiero - Wed, 19 Oct 2022 17:04:42 -0500

  • Require protobuf-java-util 3.21.7 to fix #1891 (#1938)

    David Venable - Wed, 19 Oct 2022 12:35:33 -0500

    Signed-off-by: David Venable <dlv@amazon.com>

  • Bug Fix: S3 source key (#1926)

    Asif Sohail Mohammed - Wed, 19 Oct 2022 11:39:56 -0500

  • Jackson 2.13.4.2 (#1925)

    David Venable - Tue, 18 Oct 2022 16:52:21 -0500

    Signed-off-by: David Venable <dlv@amazon.com>

  • Refactors the Data Prepper CLI argument parsing into data-prepper-main. Added an interface for the parts of DataPrepperArgs that client classes really need. (#1920)

    David Venable - Fri, 14 Oct 2022 10:03:06 -0500

    Signed-off-by: David Venable <dlv@amazon.com>

  • Fix PipelineConnector to duplicate the events (#1897)

    kkondaka - Wed, 12 Oct 2022 16:32:00 -0500

    • Fix string mutate processors to duplicate the events Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Fix string mutate processors to duplicate the events - made changes as per David's suggestions Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Removed unnecessary changes leftover from 1st commit Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Modified PipelineConnector to duplicate JacksonSpan type events too. Added testcases in PipelineConnectorTest Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Addressed review comments Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Addressed review comment and added a new testcase for JacksonSpan withData() Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

    • Addressed review comment and added parallel pipeline test to github/workflows

    Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

  • Simple duration regex did not allow for 0s or 0ms (#1910)

    Taylor Gray - Tue, 11 Oct 2022 10:58:40 -0500

    Signed-off-by: Taylor Gray <tylgry@amazon.com> Signed-off-by: Taylor Gray <tylgry@amazon.com>

  • Updated the release notes for 2.0.0 (#1911)

    David Venable - Mon, 10 Oct 2022 17:46:26 -0500

    Updated the release notes for 2.0.0 Signed-off-by: David Venable <dlv@amazon.com>

  • Updated the change log for 2.0.0 with most recent changes. (#1909)

    David Venable - Mon, 10 Oct 2022 17:33:31 -0500

    Signed-off-by: David Venable <dlv@amazon.com>

  • Update dev version to 2.1.0-SNAPSHOT (#1904)

    Hai Yan - Mon, 10 Oct 2022 13:34:46 -0500

    Signed-off-by: Hai Yan <oeyh@amazon.com>

  • Increase the default buffer configurations by 25. Capacity to 12,800 and batch size to 200. (#1906)

    David Venable - Mon, 10 Oct 2022 11:59:38 -0500

    Signed-off-by: David Venable <dlv@amazon.com>

  • Conditional routing documentation (#1894)

    David Venable - Mon, 10 Oct 2022 10:35:15 -0500

    Add documentation for conditional routing. Resolves #1890 Signed-off-by: David Venable <dlv@amazon.com>

  • Added change log (#1901)

    Asif Sohail Mohammed - Fri, 7 Oct 2022 20:52:38 -0500

    Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

  • Adds a stack-trace to failures from OpenSearch to help with debugging issues. (#1899)

    David Venable - Fri, 7 Oct 2022 18:32:26 -0500

    Signed-off-by: David Venable <dlv@amazon.com>