feat: Update Elasticsearch json parser to use `LoggingReceiverMacro` #1987

dyl10s · 2025-07-21T18:25:07Z

Description

Refactored the Elasticsearch json parser to use the LoggingReceiverMacro

I did need to add some yaml tags to fix a parsing panic

Related issue

How has this been tested?

Checklist:

Unit tests
- Unit tests do not apply.
- Unit tests have been added/modified and passed for this PR.
Integration tests
- Integration tests do not apply.
- Integration tests have been added/modified and passed for this PR.
Documentation
- This PR introduces no user visible changes.
- This PR introduces user visible changes and the corresponding documentation change has been made.
Minor version bump
- This PR introduces no new features.
- This PR introduces new features, and there is a separate PR to bump the minor version since the last release already.
- This PR bumps the version.

google-cla · 2025-07-21T18:25:12Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

apps/elasticsearch.go

...erator/testdata/goldens/logging-receiver_elasticsearch/golden/linux-gpu/fluent_bit_main.conf

apps/elasticsearch.go

transformation_test/transformation_test.go

franciscovalentecastro · 2025-07-22T21:10:56Z

transformation_test/testdata/logging_processor-elasticsearch-json/output_fluentbit.yaml

Note: This is a general comment that i mentioned also in #1978 (comment). Another way of verifying that the refactor is correct is that the output_fluentbit.yaml should be the same before and after the update to elasticsearch.go.

franciscovalentecastro

LGTM!

…/dyl10s/ops-agent into feat/elasticsearch-json-refactor

franciscovalentecastro

Sorry for bringing this comments so late in the review process. Added some last suggested changes.

franciscovalentecastro · 2025-07-28T18:05:22Z

apps/elasticsearch.go

-	ReceiverMixin                     confgenerator.LoggingReceiverFilesMixin `yaml:",inline"`
+type LoggingReceiverMacroElasticsearchJson struct {
+	LoggingProcessorMacroElasticsearchJson `yaml:",inline"`
+	ReceiverMixin                          confgenerator.LoggingReceiverFilesMixin `yaml:",inline"`


This change should prevent the YAML un-marshalling to validate recursively all fields in ReceiverMixin, thus not causing the panic with missing yaml tags.

Suggested change :

Suggested change

ReceiverMixin confgenerator.LoggingReceiverFilesMixin `yaml:",inline"`

ReceiverMixin confgenerator.LoggingReceiverFilesMixin `yaml:",inline" validate:"structonly"`

franciscovalentecastro · 2025-07-28T18:06:30Z

confgenerator/logging_processors.go

+	StateName string `yaml:"state_name,omitempty"`
+	Regex     string `yaml:"regex,omitempty"`
+	NextState string `yaml:"next_state,omitempty"`


Sorry for bringing this comments so late in the review process. Now i'm realizing that since MultilineRule ¹ is a non-configurable feature (not exposed to users), these fields don't need and shouldn't have named yaml tags.

Note : I remember the updates in elasticsearch.go were causing some issues with a panic (missing tags) when running tests. I haven't been able to reproduce again, but i believe the other suggested change in elasticsearch.go should fix that.

Suggesting the following change :

Suggested change

StateName string `yaml:"state_name,omitempty"`

Regex string `yaml:"regex,omitempty"`

NextState string `yaml:"next_state,omitempty"`

StateName string

Regex string

NextState string

Footnotes

https://github.com/GoogleCloudPlatform/ops-agent/blob/master/confgenerator/logging_receivers.go#L74 ↩

dyl10s · 2025-07-28T18:23:08Z

Sorry for bringing this comments so late in the review process. Added some last suggested changes.

All good, got that fixed!

franciscovalentecastro · 2025-07-28T23:45:40Z

Previous commit passed most tests (see a483257). Merging.

franciscovalentecastro reviewed Jul 22, 2025

View reviewed changes

apps/elasticsearch.go Outdated Show resolved Hide resolved

apps/elasticsearch.go Outdated Show resolved Hide resolved

dyl10s requested a review from franciscovalentecastro July 22, 2025 17:23

franciscovalentecastro reviewed Jul 22, 2025

View reviewed changes

franciscovalentecastro added the review-focus label Jul 22, 2025

franciscovalentecastro reviewed Jul 22, 2025

View reviewed changes

dyl10s requested a review from franciscovalentecastro July 23, 2025 13:12

franciscovalentecastro approved these changes Jul 23, 2025

View reviewed changes

dyl10s added 6 commits July 24, 2025 17:40

generated test files

17cfa85

refactored elasticsearch json

5315299

removed double comments

b603b9a

move elasticsearch processor init to transform tests only

b3b6f00

updated to use ReceiverMixin

992497f

removed duplicate import

b16bece

dyl10s force-pushed the feat/elasticsearch-json-refactor branch from 65124b7 to b16bece Compare July 24, 2025 17:46

franciscovalentecastro and others added 3 commits July 24, 2025 16:32

Merge branch 'master' into feat/elasticsearch-json-refactor

6e92c31

Merge branch 'master' into feat/elasticsearch-json-refactor

79aeb34

Merge branch 'feat/elasticsearch-json-refactor' of https://github.com…

5d4df4e

…/dyl10s/ops-agent into feat/elasticsearch-json-refactor

franciscovalentecastro requested changes Jul 28, 2025

View reviewed changes

updated yaml tags

025d223

dyl10s requested a review from franciscovalentecastro July 28, 2025 18:23

Merge branch 'master' into feat/elasticsearch-json-refactor

a483257

franciscovalentecastro approved these changes Jul 28, 2025

View reviewed changes

Merge branch 'master' into feat/elasticsearch-json-refactor

d948e68

franciscovalentecastro merged commit 308261b into GoogleCloudPlatform:master Jul 28, 2025
34 of 53 checks passed

This was referenced Aug 18, 2025

feat: Update Rabbitmq parser to use LoggingReceiverMacro #2038

Merged

feat: Update Elasticsearch GC receiver to use LoggingReceiverMacro #2040

Merged

	ReceiverMixin confgenerator.LoggingReceiverFilesMixin `yaml:",inline"`
	ReceiverMixin confgenerator.LoggingReceiverFilesMixin `yaml:",inline" validate:"structonly"`

feat: Update Elasticsearch json parser to use LoggingReceiverMacro #1987

feat: Update Elasticsearch json parser to use LoggingReceiverMacro #1987

Uh oh!

Conversation

dyl10s commented Jul 21, 2025

Description

Related issue

How has this been tested?

Checklist:

Uh oh!

google-cla bot commented Jul 21, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

franciscovalentecastro Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

franciscovalentecastro left a comment

Choose a reason for hiding this comment

Uh oh!

franciscovalentecastro left a comment

Choose a reason for hiding this comment

Uh oh!

franciscovalentecastro Jul 28, 2025

Choose a reason for hiding this comment

Uh oh!

franciscovalentecastro Jul 28, 2025

Choose a reason for hiding this comment

Footnotes

Uh oh!

dyl10s commented Jul 28, 2025

Uh oh!

franciscovalentecastro commented Jul 28, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

feat: Update Elasticsearch json parser to use `LoggingReceiverMacro` #1987

feat: Update Elasticsearch json parser to use `LoggingReceiverMacro` #1987

franciscovalentecastro Jul 22, 2025 •

edited

Loading