DSM optimizations - major refactoring to get rid of LinkedHashMap #9151
+1,448
−1,321
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 48 metrics, 11 unstable metrics. Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.52.0-SNAPSHOT~a836ab6abf, baseline=1.52.0-SNAPSHOT~b6521fb91a
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.038 s) : 0, 1038144
Total [baseline] (8.649 s) : 0, 8649309
Agent [candidate] (1.039 s) : 0, 1039366
Total [candidate] (8.6 s) : 0, 8600395
section iast
Agent [baseline] (1.19 s) : 0, 1189864
Total [baseline] (9.328 s) : 0, 9328107
Agent [candidate] (1.18 s) : 0, 1179916
Total [candidate] (9.282 s) : 0, 9281994
gantt
title insecure-bank - break down per module: candidate=1.52.0-SNAPSHOT~a836ab6abf, baseline=1.52.0-SNAPSHOT~b6521fb91a
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.424 ms) : 0, 1424
crashtracking [candidate] (1.432 ms) : 0, 1432
BytebuddyAgent [baseline] (729.279 ms) : 0, 729279
BytebuddyAgent [candidate] (730.396 ms) : 0, 730396
GlobalTracer [baseline] (240.821 ms) : 0, 240821
GlobalTracer [candidate] (241.374 ms) : 0, 241374
AppSec [baseline] (30.666 ms) : 0, 30666
AppSec [candidate] (30.403 ms) : 0, 30403
Debugger [baseline] (6.051 ms) : 0, 6051
Debugger [candidate] (6.014 ms) : 0, 6014
Remote Config [baseline] (660.419 µs) : 0, 660
Remote Config [candidate] (650.194 µs) : 0, 650
Telemetry [baseline] (8.257 ms) : 0, 8257
Telemetry [candidate] (8.194 ms) : 0, 8194
section iast
crashtracking [baseline] (1.47 ms) : 0, 1470
crashtracking [candidate] (1.439 ms) : 0, 1439
BytebuddyAgent [baseline] (860.256 ms) : 0, 860256
BytebuddyAgent [candidate] (852.971 ms) : 0, 852971
GlobalTracer [baseline] (233.603 ms) : 0, 233603
GlobalTracer [candidate] (233.316 ms) : 0, 233316
IAST [baseline] (30.462 ms) : 0, 30462
IAST [candidate] (26.495 ms) : 0, 26495
AppSec [baseline] (27.028 ms) : 0, 27028
AppSec [candidate] (27.927 ms) : 0, 27927
Debugger [baseline] (7.385 ms) : 0, 7385
Debugger [candidate] (8.278 ms) : 0, 8278
Remote Config [baseline] (596.194 µs) : 0, 596
Remote Config [candidate] (575.767 µs) : 0, 576
Telemetry [baseline] (8.009 ms) : 0, 8009
Telemetry [candidate] (7.888 ms) : 0, 7888
Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.52.0-SNAPSHOT~a836ab6abf, baseline=1.52.0-SNAPSHOT~b6521fb91a
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.038 s) : 0, 1037852
Total [baseline] (10.678 s) : 0, 10677789
Agent [candidate] (1.039 s) : 0, 1038804
Total [candidate] (10.547 s) : 0, 10547083
section appsec
Agent [baseline] (1.226 s) : 0, 1226021
Total [baseline] (10.814 s) : 0, 10814221
Agent [candidate] (1.216 s) : 0, 1216424
Total [candidate] (10.711 s) : 0, 10711477
section iast
Agent [baseline] (1.171 s) : 0, 1171384
Total [baseline] (10.822 s) : 0, 10821777
Agent [candidate] (1.173 s) : 0, 1173251
Total [candidate] (10.914 s) : 0, 10914003
section profiling
Agent [baseline] (1.182 s) : 0, 1182300
Total [baseline] (10.863 s) : 0, 10863169
Agent [candidate] (1.184 s) : 0, 1183926
Total [candidate] (10.779 s) : 0, 10778525
gantt
title petclinic - break down per module: candidate=1.52.0-SNAPSHOT~a836ab6abf, baseline=1.52.0-SNAPSHOT~b6521fb91a
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.433 ms) : 0, 1433
crashtracking [candidate] (1.428 ms) : 0, 1428
BytebuddyAgent [baseline] (729.295 ms) : 0, 729295
BytebuddyAgent [candidate] (729.968 ms) : 0, 729968
GlobalTracer [baseline] (240.984 ms) : 0, 240984
GlobalTracer [candidate] (241.138 ms) : 0, 241138
AppSec [baseline] (30.369 ms) : 0, 30369
AppSec [candidate] (30.441 ms) : 0, 30441
Debugger [baseline] (5.978 ms) : 0, 5978
Debugger [candidate] (6.0 ms) : 0, 6000
Remote Config [baseline] (643.635 µs) : 0, 644
Remote Config [candidate] (645.435 µs) : 0, 645
Telemetry [baseline] (8.18 ms) : 0, 8180
Telemetry [candidate] (8.256 ms) : 0, 8256
section appsec
crashtracking [baseline] (1.433 ms) : 0, 1433
crashtracking [candidate] (1.429 ms) : 0, 1429
BytebuddyAgent [baseline] (757.849 ms) : 0, 757849
BytebuddyAgent [candidate] (750.796 ms) : 0, 750796
GlobalTracer [baseline] (235.868 ms) : 0, 235868
GlobalTracer [candidate] (234.189 ms) : 0, 234189
IAST [baseline] (23.804 ms) : 0, 23804
IAST [candidate] (23.477 ms) : 0, 23477
AppSec [baseline] (168.298 ms) : 0, 168298
AppSec [candidate] (167.931 ms) : 0, 167931
Debugger [baseline] (8.649 ms) : 0, 8649
Debugger [candidate] (7.079 ms) : 0, 7079
Remote Config [baseline] (622.724 µs) : 0, 623
Remote Config [candidate] (605.101 µs) : 0, 605
Telemetry [baseline] (8.16 ms) : 0, 8160
Telemetry [candidate] (9.586 ms) : 0, 9586
section iast
crashtracking [baseline] (1.425 ms) : 0, 1425
crashtracking [candidate] (1.436 ms) : 0, 1436
BytebuddyAgent [baseline] (845.99 ms) : 0, 845990
BytebuddyAgent [candidate] (847.224 ms) : 0, 847224
GlobalTracer [baseline] (230.947 ms) : 0, 230947
GlobalTracer [candidate] (231.285 ms) : 0, 231285
IAST [baseline] (27.407 ms) : 0, 27407
IAST [candidate] (30.07 ms) : 0, 30070
AppSec [baseline] (28.745 ms) : 0, 28745
AppSec [candidate] (26.484 ms) : 0, 26484
Debugger [baseline] (7.461 ms) : 0, 7461
Debugger [candidate] (7.405 ms) : 0, 7405
Remote Config [baseline] (572.256 µs) : 0, 572
Remote Config [candidate] (580.579 µs) : 0, 581
Telemetry [baseline] (7.875 ms) : 0, 7875
Telemetry [candidate] (7.862 ms) : 0, 7862
section profiling
crashtracking [baseline] (1.395 ms) : 0, 1395
crashtracking [candidate] (1.404 ms) : 0, 1404
BytebuddyAgent [baseline] (759.056 ms) : 0, 759056
BytebuddyAgent [candidate] (759.266 ms) : 0, 759266
GlobalTracer [baseline] (220.033 ms) : 0, 220033
GlobalTracer [candidate] (220.629 ms) : 0, 220629
AppSec [baseline] (30.16 ms) : 0, 30160
AppSec [candidate] (30.246 ms) : 0, 30246
Debugger [baseline] (6.231 ms) : 0, 6231
Debugger [candidate] (6.279 ms) : 0, 6279
Remote Config [baseline] (682.034 µs) : 0, 682
Remote Config [candidate] (653.839 µs) : 0, 654
Telemetry [baseline] (11.705 ms) : 0, 11705
Telemetry [candidate] (14.014 ms) : 0, 14014
ProfilingAgent [baseline] (104.581 ms) : 0, 104581
ProfilingAgent [candidate] (103.141 ms) : 0, 103141
Profiling [baseline] (104.604 ms) : 0, 104604
Profiling [candidate] (103.164 ms) : 0, 103164
LoadParameters
See matching parameters
SummaryFound 1 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 12 unstable metrics.
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.52.0-SNAPSHOT~a836ab6abf, baseline=1.52.0-SNAPSHOT~b6521fb91a
dateFormat X
axisFormat %s
section baseline
no_agent (4.442 ms) : 4388, 4496
. : milestone, 4442,
iast (9.27 ms) : 9120, 9420
. : milestone, 9270,
iast_FULL (14.047 ms) : 13768, 14326
. : milestone, 14047,
iast_GLOBAL (10.363 ms) : 10177, 10548
. : milestone, 10363,
profiling (8.686 ms) : 8543, 8829
. : milestone, 8686,
tracing (7.603 ms) : 7488, 7717
. : milestone, 7603,
section candidate
no_agent (4.336 ms) : 4283, 4390
. : milestone, 4336,
iast (9.379 ms) : 9225, 9532
. : milestone, 9379,
iast_FULL (14.337 ms) : 14046, 14627
. : milestone, 14337,
iast_GLOBAL (10.39 ms) : 10210, 10570
. : milestone, 10390,
profiling (9.004 ms) : 8849, 9159
. : milestone, 9004,
tracing (7.696 ms) : 7579, 7813
. : milestone, 7696,
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.52.0-SNAPSHOT~a836ab6abf, baseline=1.52.0-SNAPSHOT~b6521fb91a
dateFormat X
axisFormat %s
section baseline
no_agent (37.273 ms) : 36974, 37572
. : milestone, 37273,
appsec (47.243 ms) : 46817, 47669
. : milestone, 47243,
code_origins (45.927 ms) : 45527, 46327
. : milestone, 45927,
iast (44.567 ms) : 44185, 44950
. : milestone, 44567,
profiling (48.407 ms) : 47964, 48849
. : milestone, 48407,
tracing (44.978 ms) : 44597, 45358
. : milestone, 44978,
section candidate
no_agent (36.789 ms) : 36490, 37089
. : milestone, 36789,
appsec (48.05 ms) : 47614, 48486
. : milestone, 48050,
code_origins (44.324 ms) : 43951, 44696
. : milestone, 44324,
iast (44.459 ms) : 44079, 44839
. : milestone, 44459,
profiling (47.579 ms) : 47156, 48003
. : milestone, 47579,
tracing (45.276 ms) : 44900, 45652
. : milestone, 45276,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics. Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.52.0-SNAPSHOT~a836ab6abf, baseline=1.52.0-SNAPSHOT~b6521fb91a
dateFormat X
axisFormat %s
section baseline
no_agent (14.987 s) : 14987000, 14987000
. : milestone, 14987000,
appsec (14.845 s) : 14845000, 14845000
. : milestone, 14845000,
iast (18.455 s) : 18455000, 18455000
. : milestone, 18455000,
iast_GLOBAL (17.943 s) : 17943000, 17943000
. : milestone, 17943000,
profiling (15.408 s) : 15408000, 15408000
. : milestone, 15408000,
tracing (14.821 s) : 14821000, 14821000
. : milestone, 14821000,
section candidate
no_agent (14.911 s) : 14911000, 14911000
. : milestone, 14911000,
appsec (14.941 s) : 14941000, 14941000
. : milestone, 14941000,
iast (18.264 s) : 18264000, 18264000
. : milestone, 18264000,
iast_GLOBAL (17.878 s) : 17878000, 17878000
. : milestone, 17878000,
profiling (15.519 s) : 15519000, 15519000
. : milestone, 15519000,
tracing (14.979 s) : 14979000, 14979000
. : milestone, 14979000,
Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.52.0-SNAPSHOT~a836ab6abf, baseline=1.52.0-SNAPSHOT~b6521fb91a
dateFormat X
axisFormat %s
section baseline
no_agent (1.481 ms) : 1470, 1493
. : milestone, 1481,
appsec (3.649 ms) : 3431, 3868
. : milestone, 3649,
iast (2.194 ms) : 2131, 2257
. : milestone, 2194,
iast_GLOBAL (2.259 ms) : 2196, 2323
. : milestone, 2259,
profiling (2.054 ms) : 2003, 2104
. : milestone, 2054,
tracing (2.037 ms) : 1987, 2086
. : milestone, 2037,
section candidate
no_agent (1.483 ms) : 1471, 1495
. : milestone, 1483,
appsec (3.687 ms) : 3470, 3904
. : milestone, 3687,
iast (2.203 ms) : 2140, 2266
. : milestone, 2203,
iast_GLOBAL (2.242 ms) : 2180, 2305
. : milestone, 2242,
profiling (2.046 ms) : 1996, 2096
. : milestone, 2046,
tracing (2.021 ms) : 1973, 2070
. : milestone, 2021,
|
Kafka / producer-benchmarkParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics. See unchanged results
|
|
Hi! 👋 Thanks for your pull request! 🎉 To help us review it, please make sure to:
If you need help, please check our contributing guidelines. |
Kafka / consumer-benchmarkParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics. See unchanged results
|
dougqh
reviewed
Jul 14, 2025
| result.put(DIRECTION_TAG, DIRECTION_OUT); | ||
| result.put(TYPE_TAG, "grpc"); | ||
| return DataStreamsContext.fromTags(result); | ||
| return DataStreamsContext.fromTags( |
There was a problem hiding this comment.
I would avoid builder idioms in the critical path. That's still extra allocation that we really don't need.
dougqh
reviewed
Jul 14, 2025
|
|
||
| public DataStreamsTags(DataStreamsTagsBuilder builder) { | ||
| this.builder = builder; | ||
| this.size = |
There was a problem hiding this comment.
This is introducing a capturing lambda. I think a regular loop would probably be better here.
AlexeyKuznetsov-DD
requested changes
Jul 14, 2025
Comment on lines
93
to
102
| processor.process(BUS_TAG, this.builder.bus); | ||
| count += 1; | ||
| } | ||
|
|
||
| if (this.builder.direction == Direction.Inbound) { | ||
| count += 1; | ||
| processor.process(DIRECTION_TAG, DIRECTION_IN); | ||
| } else if (this.builder.direction == Direction.Outbound) { | ||
| count += 1; | ||
| processor.process(DIRECTION_TAG, DIRECTION_OUT); |
There was a problem hiding this comment.
Suggestion: Just a matter of taste, but probably count++; Would be a better option (here and other similar places)?
Comment on lines
286
to
299
| + "," | ||
| + ", kafkaClusterId='" | ||
| + this.builder.kafkaClusterId | ||
| + "," | ||
| + ", partition='" | ||
| + this.builder.partition | ||
| + "," | ||
| + ", hash=" | ||
| + hash | ||
| + "," | ||
| + ", aggregationHash=" | ||
| + aggregationHash | ||
| + "," | ||
| + ", size=" |
There was a problem hiding this comment.
I think this code will produce String with TWO commas. Is that expected?
There was a problem hiding this comment.
Yes, you are right... AI autosuggest :D
There was a problem hiding this comment.
Nice :), just FYI IDEA has toString() method generation too,
dougqh
reviewed
Jul 14, 2025
| result.put(TYPE_TAG, "grpc"); | ||
| return result; | ||
| private static final DataStreamsTags createServerPathwaySortedTags() { | ||
| return new DataStreamsTagsBuilder() |
There was a problem hiding this comment.
I think we can probably still keep these pretty clean without needing a builder. I've put suggestions throughout...
- createGrpcInbound() <-- bonus, that's cacheable
dougqh
reviewed
Jul 14, 2025
| DataStreamsContext dsmContext = DataStreamsContext.fromTags(getTags(eventBusName)); | ||
| DataStreamsTags tags = | ||
| new DataStreamsTagsBuilder() | ||
| .withDirection(DataStreamsTags.Direction.Outbound) |
There was a problem hiding this comment.
createBusOutbound(eventBusName)
dougqh
reviewed
Jul 14, 2025
| sortedTags.put(TagsProcessor.TOPIC_TAG, bucket); | ||
| sortedTags.put(TagsProcessor.TYPE_TAG, "s3"); | ||
| DataStreamsTags tags = | ||
| new DataStreamsTagsBuilder() |
There was a problem hiding this comment.
createS3Outbound(bucket, key)
dougqh
reviewed
Jul 14, 2025
| sortedTags.put(TYPE_TAG, "kinesis"); | ||
| DataStreamsTags tags = | ||
| new DataStreamsTagsBuilder() | ||
| .withType("kinesis") |
There was a problem hiding this comment.
createKinesisInbound(streamArn)
dougqh
reviewed
Jul 14, 2025
|
|
||
| return sortedTags; | ||
| private DataStreamsTags getTags(String snsTopicName) { | ||
| return new DataStreamsTagsBuilder() |
There was a problem hiding this comment.
createSnsOutbound(snsTopicName)
There was a problem hiding this comment.
If you're in refactoring mode, can you have a look (made in another PR if it's easier) to complete the context refactoring?
In short, PathwayContext should not be stored in span data model (DDSpanContext).
Instead, it must be stored with Context as DataStreamsContext and AgentSpan.
You may create a second Context entry if the DataStreamsContext holds different kind of data (or have a different lifecycle) than what PathwayContext should hold.
I already done some refactoring (adding DataStreamsContext, creating a dedicated DSM propagator) but it miss the last step of moving away PathwayContext from span.
Would that be something you could have a look at? Would you like us to pair on this task? The end goal would be to decouple Tracing from DSM. So you can save even more resources.
dougqh
reviewed
Jul 15, 2025
| @@ -89,7 +85,9 @@ private String getTraceContextToInject( | |||
| // Inject context | |||
| datadog.context.Context context = span; | |||
| if (traceConfig().isDataStreamsEnabled()) { | |||
| DataStreamsContext dsmContext = DataStreamsContext.fromTags(getTags(eventBusName)); | |||
| DataStreamsTags tags = | |||
| DataStreamsTags.createWithBus("bus", DataStreamsTags.Direction.Outbound, eventBusName); | |||
There was a problem hiding this comment.
Why pass "bus" as argument to withBus helper? Is there a case where the value isn't "bus"?
There was a problem hiding this comment.
Yes, the type is always bus, no need to pass it here.
dougqh
reviewed
Jul 15, 2025
| private long completeHash; | ||
| private int nonNullSize; | ||
|
|
||
| // hash tags |
There was a problem hiding this comment.
These should probably be final
Right now, strictly speaking this probably isn't thread safe
dougqh
reviewed
Jul 15, 2025
| kafkaClusterId != null ? KAFKA_CLUSTER_ID_TAG + ":" + kafkaClusterId : null; | ||
| this.partition = partition != null ? PARTITION_TAG + ":" + partition : null; | ||
|
|
||
| // hashable tags are 0-4 |
There was a problem hiding this comment.
Nice, that should unroll and inline nicely
I'm definitely for refactoring the context propagation logic, but I don't want to bundle the changes into a single PR. |
kr-igor
added
type: enhancement
dougqh
approved these changes
Jul 18, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change is aimed to reduce DSM overhead by using a lightweight object instead of linked maps.
Introduced
DataStreamsTagswhich are now used instead ofLinkedHashMap.Updated all integrations and tests.
My tests show ~20% throughput increase with a significant latency decrease without increasing CPU load. (left side is v1.51.1, right side is this branch)
