Skip to content

es_out: support Upstream Servers with configuration overriding #7608

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

mabrarov
Copy link
Contributor

@mabrarov mabrarov commented Jun 25, 2023

Implementation of Upstream feature for the Elasticsearch output plugin.

This pull request is based on pull request #1560 and Forward output plugin.

It was tested in a local setup with:

  1. Fluent Bit without Upstream feature connected to a single node of Elasticsearch cluster consisting of 3 master-eligible/data and 1 coordinating nodes.

    Refer to elastic-cluster directory of mabrarov/elastic-stack repository for Docker Compose project used to create target Elasticsearch cluster and Kibana.

    fluent-bit.conf Fluent Bit configuration file used for the test - refer to fluent-bit-es/fluent-bit.conf and (same in YAML format) fluent-bit-es/fluent-bit.yaml in mabrarov/elastic-stack repository.

    Debug log is available at flb_es.log.

  2. Fluent Bit with Upstream feature connected to all Elasticsearch data nodes of Elasticsearch cluster consisting of 3 master-eligible/data and 1 coordinating nodes.

    Refer to elastic-cluster directory of mabrarov/elastic-stack repository for Docker Compose project used to create target Elasticsearch cluster and Kibana.

    fluent-bit.conf Fluent Bit configuration file used for the test - refer to fluent-bit-es-cluster/fluent-bit.conf and (same in YAML format) fluent-bit-es-cluster/fluent-bit.yaml in mabrarov/elastic-stack repository.

    Debug log is available at flb_es_upstream.log.

Testing

  • Example configuration files for the change can be found in mabrarov/elastic-stack repository under fluent-bit-es-cluster directory.
  • Debug log output from testing the change - see above.
  • Attached Valgrind output that shows no leaks or memory corruption was found - refer to flb_run_code_analysis.log for the output of command
    TEST_PRESET=valgrind SKIP_TESTS='flb-rt-out_td flb-it-network' ./run_code_analysis.sh
  • [N/A] Run local packaging test showing all targets (including any new ones) build.
  • [N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

Backporting

  • [N/A] Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

return 0;
}

toks = flb_utils_split((const char *)cloud_auth, ':', -1);
Copy link
Contributor Author

@mabrarov mabrarov Jun 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not:

Suggested change
toks = flb_utils_split((const char *)cloud_auth, ':', -1);
toks = flb_utils_split((const char *) cloud_auth, ':', -1);

because https://github.com/fluent/fluent-bit/blob/master/CONTRIBUTING.md requires to follow https://httpd.apache.org/dev/styleguide.html which requires to not put a space in case of type cast:

The Guidelines
...
There is no whitespace between a cast and the item modified (e.g., "(int)j" and not "(int) j").

@mabrarov
Copy link
Contributor Author

Hi reviewers,

Is it possible to approve only workflow for this pull request, so that automated checks and build can start?

Thank you.

@mabrarov mabrarov temporarily deployed to pr June 28, 2023 17:55 — with GitHub Actions Inactive
@mabrarov mabrarov temporarily deployed to pr June 28, 2023 17:55 — with GitHub Actions Inactive
@mabrarov mabrarov temporarily deployed to pr June 28, 2023 17:55 — with GitHub Actions Inactive
@PettitWesley
Copy link
Contributor

@mabrarov sure

@mabrarov mabrarov temporarily deployed to pr June 28, 2023 18:22 — with GitHub Actions Inactive
@mabrarov
Copy link
Contributor Author

mabrarov commented Jun 29, 2023

Hi @PettitWesley,

It looks like all failed checks are around run-macos-unit-tests jobs and caused by the following failed unit tests:

  1. flb-rt-in_event_test
  2. flb-rt-out_tcp

I feel like other pull requests have the same issues, i.e. it doesn't seem that the failed checks are caused by this pull request changes.

Help of maintainers is appreciated.

Thank you.

}
tmp = flb_upstream_node_get_property(FLB_ES_CONFIG_PROPERTY_LOGSTASH_PREFIX, node);
if (tmp) {
ec->logstash_prefix = (char *)tmp;
Copy link
Contributor Author

@mabrarov mabrarov Jul 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Type cast is illegal. Should use flb_sds_create instead. Working on fix in feature/out_es_upstream_support_extended_test branch.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

@mabrarov mabrarov force-pushed the feature/out_es_upstream_support_extended branch from ba3382a to b7cd81b Compare July 8, 2023 10:22
@mabrarov
Copy link
Contributor Author

Hi @PettitWesley,

Is it possible to trigger automated workflow (build) for this pull request one more time? I found & fixed one issue and added tests for the new code since last build happened.

Thank you.

@mabrarov mabrarov temporarily deployed to pr July 10, 2023 12:05 — with GitHub Actions Inactive
@mabrarov mabrarov temporarily deployed to pr July 10, 2023 12:05 — with GitHub Actions Inactive
@mabrarov mabrarov temporarily deployed to pr July 10, 2023 12:05 — with GitHub Actions Inactive
@mabrarov mabrarov temporarily deployed to pr July 10, 2023 12:28 — with GitHub Actions Inactive
@mabrarov
Copy link
Contributor Author

Hi dear reviewers,

Is it possible to get this pull request reviewed / accepted sooner? Is there something pending / waiting from my side to start review?

Thank you.

@mabrarov mabrarov force-pushed the feature/out_es_upstream_support_extended branch from b7cd81b to b81d3f7 Compare July 20, 2023 19:38
@mabrarov
Copy link
Contributor Author

Hi @PettitWesley and @edsiper,

It feels like you are code owners for Elasticsearch output plugin. Is there something pending / waiting from my side to start review of this pull request? This new feature was requested 4 years ago and I feel it is something which multiple users of Fluent Bit (not just my team) would like to have.

Thank you.

@mabrarov mabrarov force-pushed the feature/out_es_upstream_support_extended branch from b81d3f7 to f6431c2 Compare September 30, 2023 13:32
@patrick-stephens patrick-stephens added the ok-package-test Run PR packaging tests label Apr 28, 2025
@patrick-stephens
Copy link
Contributor

@mabrarov / @cosmo0920 are the issues covered here? There's a lot of stuff in the conversation so not sure if the changes have been actioned.

@mabrarov
Copy link
Contributor Author

mabrarov commented Apr 28, 2025

Hi @patrick-stephens,

Answering your question about status of this pull request (my understanding): review of @cosmo0920 ended with conclusion that the ownership tracking (please note that it is not reference counting) implemented in this pull request doesn't follow Fluent Bit code style / architecture (which is true. IMHO, it is so, because existing Fluent Bit code doesn't have the case which this pull request has). Unfortunately, I failed to find better solution, because the only alternative - duplication of configuration per each upstream node - drastically complicates code without real benefit. I'd like Fluent Bit maintainers and community to review this pull request one more time. Maybe it's the case when it is acceptable to introduce changes into configuration data ownership handling.

Thank you.

mabrarov added 6 commits May 23, 2025 23:26
…wn to parser of Upstream node configuration section are implemented, e.g. "host" and "port"

Signed-off-by: Marat Abrarov <[email protected]>
…o the test callback based on configuration of Fluent Bit and based on configuration of plugin

Signed-off-by: Marat Abrarov <[email protected]>
…with Upstream node configuration

For Elastic cloud authentication these parameters are always taken from plugin configuration and never from Upstream node configuration: cloud_id.

For AWS authentication these parameters are always taken from plugin configuration and never from Upstream node configuration: http_proxy, no_proxy, tls*.

Signed-off-by: Marat Abrarov <[email protected]>
…o the test callback based on configuration of Fluent Bit and based on configuration of plugin

Signed-off-by: Marat Abrarov <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docs-required ok-package-test Run PR packaging tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants