appsec: Service Extension callout #2965

e-n-0 · 2024-11-06T12:40:33Z

Motivation

This is the part 2 PR to support Envoy's External Processing. This part is creating a package to be used for the GCP Services Extensions product.
You can find all related document for this implementation in Confluence ASM - GCP Services Extensions.
You can find the part 1 of this PR.

What does this PR do?

This PR creates a new binary project that implements a gRPC server using the Envoy External Processor protocol of Envoy and an HTTP server for the health check. The gRPC server use the gRPC Interceptor (StreamServerInterceptor) implemented from the part 1.

Release

The PR includes a new GitHub action to publish new docker images to the repo at each new release.
These images are push under the name ghcr.io/datadog/dd-trace-go/service-extensions-callout and exists with the following tags:

latest
version (e.g. v1.69.1)
commit sha

You can find the package released in the GitHub repo registry.

Tests

System tests

System tests have been implemented on this PR. A new external-processing scenario with the docker image given to customers and envoy infrastructure has been added in the golang stage.
APM and ASM test has been imported to be sure that all spans and tags are correctly set to validate the integrity of the created traces.

Macro Benchmarks

Benchmarks of the the whole extension is available on this benchmarking platform branch.
More information with dashboard can be found here.

Reviewer's Checklist

Changed code has unit tests for its functionality at or near 100% coverage.
System-Tests covering this feature have been added and enabled with the va.b.c-dev version tag.
There is a benchmark for any new code, or changes to existing code.
If this interacts with the agent in a new way, a system test has been added.
Add an appropriate team label so this PR gets put in the right place for the release notes.
Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild.

Unsure? Have a question? Request a review!

.github/workflows/service-extensions-publish.yml

pr-commenter · 2024-11-06T14:24:47Z

Benchmarks

Benchmark execution time: 2024-12-23 14:29:20

Comparing candidate commit 1613818 in PR branch flavien/service-extensions-part2 with baseline commit d393324 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 59 metrics, 0 unstable metrics.

.github/workflows/service-extensions-publish.yml

contrib/envoyproxy/envoy/cmd/serviceextensions/Dockerfile

contrib/envoyproxy/envoy/cmd/serviceextensions/localhost.crt

contrib/envoyproxy/envoy/cmd/serviceextensions/main.go

.github/workflows/service-extensions-publish.yml

contrib/envoyproxy/go-control-plane/cmd/serviceextensions/main.go

internal/env.go

contrib/envoyproxy/go-control-plane/cmd/serviceextensions/main.go

rarguelloF

Thanks for the changes, just a few extra comments.
Also please remember to run go mod tidy in the project root and commit the changes 🙏

contrib/envoyproxy/go-control-plane/cmd/serviceextensions/main.go

rarguelloF · 2024-12-19T12:37:57Z

contrib/envoyproxy/go-control-plane/cmd/serviceextensions/main.go

+	})
+
+	if err := g.Wait(); err != nil {
+		log.Error("service_extension: %v\n", err)


You might want to os.Exit(1) in this case to signal the process exit with an error.

However if you do this, please note any defer you might have won't run. If you want to use os.Exit and defer, the best approach would be to move the logic from lines 81-95 to a separate function func startService(cfg serviceExtensionConfig) error, and have these defer's in there.

Thanks for the comment, I applied the changes in e3cc085 😄
(I also added a Flush to the logs before exiting.)

Add Readme fix rebase update to register the service

ddtrace/tracer: initialize runtimeMetricsV2 with statsd "direct" client (#3006) Co-authored-by: Felix Geisendörfer <[email protected]> Co-authored-by: Nayef Ghattas <[email protected]> fix(.github/workflows): add tags-ignore to avoid running CI on pushing tags for contribs and other nested modules (#3005) Co-authored-by: Hannah Kim <[email protected]> contrib/envoyproxy: envoy external processing support (#2895) This PR adds a new gRPC Interceptor (StreamServerInterceptor) to support the interception of ext_proc v3 calls to gRPC server. When the interceptor is applied, all messages of the external processing protocol are instrumented without returning an handle to the original server code Co-authored-by: Eliott Bouhana <[email protected]> Co-authored-by: Flavien Darche <[email protected]> add go mod to workflows/apps (#3036) go.mod: module go.opentelemetry.io/collector/pdata@latest found (v1.21.0), but does not contain package go.opentelemetry.io/collector/pdata/internal/data/protogen/profiles/v1experimental (#3042) Signed-off-by: Eliott Bouhana <[email protected]> chore: update latest majors (#2993)

Applied comments appsec: stop storing span tags, directly call span.SetTag (#3044) Signed-off-by: Eliott Bouhana <[email protected]> ddtrace/tracer: Tracing as transport-only mode (APPSEC_STANDALONE) (#3033) Signed-off-by: Eliott Bouhana <[email protected]> fix: improving test logic for TestStreamSendsErrorCode to avoid flakiness (#3049) vuln: upgrade golang.org/x/{crypto,net} to non-vulnerable versions (#3050) contrib/miekg/dns: resolve flaky test in TestExchange* (#3045) ddtrace/tracer: report datadog.tracer.api.errors health metric (#3024) build(deps): bump google.golang.org/grpc from 1.64.0 to 1.64.1 (#3001) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Rodrigo Argüello <[email protected]> ddtrace/tracer: Report datadog.tracer.queue.enqueued.traces as health metric (#3019) ddtrace/tracer: Tracing as transport-only mode (APPSEC_STANDALONE) (#3033) Signed-off-by: Eliott Bouhana <[email protected]> fix: improving test logic for TestStreamSendsErrorCode to avoid flakiness (#3049) vuln: upgrade golang.org/x/{crypto,net} to non-vulnerable versions (#3050) contrib/miekg/dns: resolve flaky test in TestExchange* (#3045) ddtrace/tracer: report datadog.tracer.api.errors health metric (#3024) build(deps): bump google.golang.org/grpc from 1.64.0 to 1.64.1 (#3001) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Rodrigo Argüello <[email protected]> ddtrace/tracer: Report datadog.tracer.queue.enqueued.traces as health metric (#3019)

github-actions bot added the apm:ecosystem contrib/* related feature requests or bugs label Nov 6, 2024

datadog-datadog-prod-us1 bot reviewed Nov 6, 2024

View reviewed changes

e-n-0 force-pushed the flavien/service-extensions-part2 branch from a87535b to 0044a8f Compare November 6, 2024 12:42

datadog-datadog-prod-us1 bot reviewed Nov 6, 2024

View reviewed changes

.github/workflows/service-extensions-publish.yml Outdated Show resolved Hide resolved

datadog-datadog-prod-us1 bot reviewed Nov 6, 2024

View reviewed changes

.github/workflows/service-extensions-publish.yml Show resolved Hide resolved

e-n-0 force-pushed the flavien/service-extensions-part2 branch 2 times, most recently from 6c165d2 to 0607577 Compare November 6, 2024 13:41

e-n-0 force-pushed the flavien/service-extensions-part2 branch from 0607577 to 867a55b Compare November 6, 2024 14:44

datadog-datadog-prod-us1 bot reviewed Nov 6, 2024

View reviewed changes

.github/workflows/service-extensions-publish.yml Show resolved Hide resolved

e-n-0 mentioned this pull request Nov 6, 2024

contrib/envoyproxy: envoy external processing support #2895

Merged

6 tasks

e-n-0 changed the title ~~Service Extension Callout Standalone + Build~~ appsec: Service Extension callout Nov 6, 2024

e-n-0 force-pushed the flavien/service-extensions-part2 branch 2 times, most recently from 34f396f to 7d3e562 Compare November 6, 2024 16:12

e-n-0 force-pushed the flavien/service-extensions branch from 2eeb6b1 to 402f7d0 Compare November 6, 2024 16:14

e-n-0 force-pushed the flavien/service-extensions-part2 branch from 7d3e562 to ca1de07 Compare November 6, 2024 16:29

e-n-0 self-assigned this Nov 6, 2024

e-n-0 added the appsec label Nov 6, 2024

e-n-0 marked this pull request as ready for review November 6, 2024 16:50

e-n-0 requested review from a team as code owners November 6, 2024 16:50

e-n-0 force-pushed the flavien/service-extensions-part2 branch from 0614501 to 2b06ea5 Compare November 6, 2024 17:02

eliottness requested changes Nov 7, 2024

View reviewed changes

eliottness force-pushed the flavien/service-extensions branch from 1f2b791 to 9630788 Compare November 8, 2024 18:05

eliottness requested a review from a team as a code owner November 8, 2024 18:05

eliottness force-pushed the flavien/service-extensions branch 2 times, most recently from 0929de5 to e3bb1e5 Compare November 13, 2024 16:14

eliottness removed the request for review from a team November 20, 2024 08:57

e-n-0 force-pushed the flavien/service-extensions branch from c7063d8 to b0844b8 Compare November 28, 2024 14:59

e-n-0 force-pushed the flavien/service-extensions-part2 branch 9 times, most recently from 42d9365 to 673ec27 Compare December 16, 2024 19:50

DataDog deleted a comment from datadog-datadog-prod-us1 bot Dec 16, 2024

datadog-datadog-prod-us1 bot reviewed Dec 16, 2024

View reviewed changes

rarguelloF reviewed Dec 18, 2024

View reviewed changes

contrib/envoyproxy/go-control-plane/cmd/serviceextensions/main.go Outdated Show resolved Hide resolved

internal/env.go Outdated Show resolved Hide resolved

contrib/envoyproxy/go-control-plane/cmd/serviceextensions/main.go Outdated Show resolved Hide resolved

e-n-0 requested a review from rarguelloF December 18, 2024 16:43

rarguelloF reviewed Dec 19, 2024

View reviewed changes

e-n-0 requested a review from rarguelloF December 19, 2024 17:22

e-n-0 force-pushed the flavien/service-extensions-part2 branch from 265488f to b02036e Compare December 23, 2024 13:05

rarguelloF approved these changes Dec 23, 2024

View reviewed changes

e-n-0 added 5 commits December 23, 2024 14:41

Service Extension Callout (Envoy external processing)

96a6e3b

Generate self signed certificates instead of importing them

4ab3308

Add Readme fix rebase update to register the service

fix go mod tidy

1613818

e-n-0 force-pushed the flavien/service-extensions-part2 branch from 2fe8c9d to 1613818 Compare December 23, 2024 13:45

DataDog deleted a comment from datadog-datadog-prod-us1 bot Dec 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

appsec: Service Extension callout #2965

appsec: Service Extension callout #2965

e-n-0 commented Nov 6, 2024 •

edited

Loading

pr-commenter bot commented Nov 6, 2024 •

edited

Loading

rarguelloF left a comment

rarguelloF Dec 19, 2024

e-n-0 Dec 19, 2024

appsec: Service Extension callout #2965

Are you sure you want to change the base?

appsec: Service Extension callout #2965

Conversation

e-n-0 commented Nov 6, 2024 • edited Loading

Motivation

What does this PR do?

Release

Tests

System tests

Macro Benchmarks

Reviewer's Checklist

pr-commenter bot commented Nov 6, 2024 • edited Loading

Benchmarks

rarguelloF left a comment

Choose a reason for hiding this comment

rarguelloF Dec 19, 2024

Choose a reason for hiding this comment

e-n-0 Dec 19, 2024

Choose a reason for hiding this comment

e-n-0 commented Nov 6, 2024 •

edited

Loading

pr-commenter bot commented Nov 6, 2024 •

edited

Loading