Skip to content

add fake Kafka change#8926

Closed
robcarlan-datadog wants to merge 3 commits intomasterfrom
rob.carlan/fake-kafka-change
Closed

add fake Kafka change#8926
robcarlan-datadog wants to merge 3 commits intomasterfrom
rob.carlan/fake-kafka-change

Conversation

@robcarlan-datadog
Copy link
Contributor

What Does This Do

Fake change to see benchmarking results

Motivation

Additional Notes

Contributor Checklist

Jira ticket: [PROJ-IDENT]

@pr-commenter
Copy link

pr-commenter bot commented Jun 4, 2025

Kafka / producer-benchmark

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master rob.carlan/fake-kafka-change
git_commit_date 1749044153 1749062317
git_commit_sha 7787af7 e5cd515
See matching parameters
Baseline Candidate
ci_job_date 1749063412 1749063412
ci_job_id 968473656 968473656
ci_pipeline_id 66981353 66981353
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
jdkVersion 11.0.25 11.0.25
jmhVersion 1.36 1.36
jvm /usr/lib/jvm/java-11-openjdk-amd64/bin/java /usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/producer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/producer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant
vmName OpenJDK 64-Bit Server VM OpenJDK 64-Bit Server VM
vmVersion 11.0.25+9-post-Ubuntu-1ubuntu122.04 11.0.25+9-post-Ubuntu-1ubuntu122.04

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

See unchanged results
scenario Δ mean throughput
scenario:not-instrumented/KafkaProduceBenchmark.benchProduce same
scenario:only-tracing-dsm-disabled-benchmarks/KafkaProduceBenchmark.benchProduce same
scenario:only-tracing-dsm-enabled-benchmarks/KafkaProduceBenchmark.benchProduce same

@pr-commenter
Copy link

pr-commenter bot commented Jun 4, 2025

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master rob.carlan/fake-kafka-change
git_commit_date 1749044153 1749062317
git_commit_sha 7787af7 e5cd515
release_version 1.50.0-SNAPSHOT~7787af738f 1.50.0-SNAPSHOT~e5cd515e88
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1749064810 1749064810
ci_job_id 968473649 968473649
ci_pipeline_id 66981353 66981353
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-sukzute8-project-304-concurrent-0-aug2qxnf 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-sukzute8-project-304-concurrent-0-aug2qxnf 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 59 metrics, 12 unstable metrics.

Startup time reports for petclinic
gantt
    title petclinic - global startup overhead: candidate=1.50.0-SNAPSHOT~e5cd515e88, baseline=1.50.0-SNAPSHOT~7787af738f

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.022 s) : 0, 1022231
Total [baseline] (11.13 s) : 0, 11129968
Agent [candidate] (1.031 s) : 0, 1030847
Total [candidate] (11.073 s) : 0, 11073258
section appsec
Agent [baseline] (1.165 s) : 0, 1165490
Total [baseline] (11.21 s) : 0, 11210103
Agent [candidate] (1.169 s) : 0, 1168811
Total [candidate] (11.291 s) : 0, 11291045
section iast
Agent [baseline] (1.157 s) : 0, 1157437
Total [baseline] (11.389 s) : 0, 11389252
Agent [candidate] (1.15 s) : 0, 1150408
Total [candidate] (11.356 s) : 0, 11356308
section profiling
Agent [baseline] (1.286 s) : 0, 1285992
Total [baseline] (11.551 s) : 0, 11550660
Agent [candidate] (1.265 s) : 0, 1264574
Total [candidate] (11.413 s) : 0, 11413059
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.022 s -
Agent appsec 1.165 s 143.259 ms (14.0%)
Agent iast 1.157 s 135.207 ms (13.2%)
Agent profiling 1.286 s 263.762 ms (25.8%)
Total tracing 11.13 s -
Total appsec 11.21 s 80.134 ms (0.7%)
Total iast 11.389 s 259.283 ms (2.3%)
Total profiling 11.551 s 420.692 ms (3.8%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.031 s -
Agent appsec 1.169 s 137.964 ms (13.4%)
Agent iast 1.15 s 119.561 ms (11.6%)
Agent profiling 1.265 s 233.728 ms (22.7%)
Total tracing 11.073 s -
Total appsec 11.291 s 217.787 ms (2.0%)
Total iast 11.356 s 283.05 ms (2.6%)
Total profiling 11.413 s 339.802 ms (3.1%)
gantt
    title petclinic - break down per module: candidate=1.50.0-SNAPSHOT~e5cd515e88, baseline=1.50.0-SNAPSHOT~7787af738f

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (682.193 ms) : 0, 682193
BytebuddyAgent [candidate] (688.169 ms) : 0, 688169
GlobalTracer [baseline] (240.374 ms) : 0, 240374
GlobalTracer [candidate] (242.17 ms) : 0, 242170
AppSec [baseline] (56.282 ms) : 0, 56282
AppSec [candidate] (56.071 ms) : 0, 56071
Debugger [baseline] (6.906 ms) : 0, 6906
Debugger [candidate] (6.221 ms) : 0, 6221
Remote Config [baseline] (730.311 µs) : 0, 730
Remote Config [candidate] (749.373 µs) : 0, 749
Telemetry [baseline] (12.144 ms) : 0, 12144
Telemetry [candidate] (13.686 ms) : 0, 13686
section appsec
BytebuddyAgent [baseline] (701.905 ms) : 0, 701905
BytebuddyAgent [candidate] (704.907 ms) : 0, 704907
GlobalTracer [baseline] (238.199 ms) : 0, 238199
GlobalTracer [candidate] (238.518 ms) : 0, 238518
IAST [baseline] (21.826 ms) : 0, 21826
IAST [candidate] (22.081 ms) : 0, 22081
AppSec [baseline] (176.876 ms) : 0, 176876
AppSec [candidate] (176.611 ms) : 0, 176611
Debugger [baseline] (5.974 ms) : 0, 5974
Debugger [candidate] (5.976 ms) : 0, 5976
Remote Config [baseline] (632.639 µs) : 0, 633
Remote Config [candidate] (639.477 µs) : 0, 639
Telemetry [baseline] (7.349 ms) : 0, 7349
Telemetry [candidate] (7.39 ms) : 0, 7390
section iast
BytebuddyAgent [baseline] (807.235 ms) : 0, 807235
BytebuddyAgent [candidate] (802.264 ms) : 0, 802264
GlobalTracer [baseline] (231.941 ms) : 0, 231941
GlobalTracer [candidate] (230.931 ms) : 0, 230931
IAST [baseline] (26.381 ms) : 0, 26381
IAST [candidate] (28.396 ms) : 0, 28396
AppSec [baseline] (53.689 ms) : 0, 53689
AppSec [candidate] (50.842 ms) : 0, 50842
Debugger [baseline] (5.98 ms) : 0, 5980
Debugger [candidate] (5.906 ms) : 0, 5906
Remote Config [baseline] (607.251 µs) : 0, 607
Remote Config [candidate] (586.147 µs) : 0, 586
Telemetry [baseline] (7.937 ms) : 0, 7937
Telemetry [candidate] (7.924 ms) : 0, 7924
section profiling
BytebuddyAgent [baseline] (685.076 ms) : 0, 685076
BytebuddyAgent [candidate] (674.108 ms) : 0, 674108
GlobalTracer [baseline] (365.49 ms) : 0, 365490
GlobalTracer [candidate] (359.906 ms) : 0, 359906
AppSec [baseline] (62.387 ms) : 0, 62387
AppSec [candidate] (62.486 ms) : 0, 62486
Debugger [baseline] (6.22 ms) : 0, 6220
Debugger [candidate] (6.084 ms) : 0, 6084
Remote Config [baseline] (661.592 µs) : 0, 662
Remote Config [candidate] (675.225 µs) : 0, 675
Telemetry [baseline] (8.266 ms) : 0, 8266
Telemetry [candidate] (8.052 ms) : 0, 8052
ProfilingAgent [baseline] (106.3 ms) : 0, 106300
ProfilingAgent [candidate] (102.471 ms) : 0, 102471
Profiling [baseline] (106.324 ms) : 0, 106324
Profiling [candidate] (102.495 ms) : 0, 102495
Loading
Startup time reports for insecure-bank
gantt
    title insecure-bank - global startup overhead: candidate=1.50.0-SNAPSHOT~e5cd515e88, baseline=1.50.0-SNAPSHOT~7787af738f

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.022 s) : 0, 1021933
Total [baseline] (8.543 s) : 0, 8543180
Agent [candidate] (1.023 s) : 0, 1023224
Total [candidate] (8.546 s) : 0, 8545548
section iast
Agent [baseline] (1.149 s) : 0, 1148918
Total [baseline] (9.16 s) : 0, 9159565
Agent [candidate] (1.154 s) : 0, 1153592
Total [candidate] (9.177 s) : 0, 9177343
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.15 s) : 0, 1149843
Total [baseline] (9.151 s) : 0, 9150678
Agent [candidate] (1.156 s) : 0, 1156127
Total [candidate] (9.179 s) : 0, 9179104
section iast_TELEMETRY_OFF
Agent [baseline] (1.145 s) : 0, 1144590
Total [baseline] (9.237 s) : 0, 9236735
Agent [candidate] (1.155 s) : 0, 1155203
Total [candidate] (9.237 s) : 0, 9237454
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.022 s -
Agent iast 1.149 s 126.985 ms (12.4%)
Agent iast_HARDCODED_SECRET_DISABLED 1.15 s 127.909 ms (12.5%)
Agent iast_TELEMETRY_OFF 1.145 s 122.657 ms (12.0%)
Total tracing 8.543 s -
Total iast 9.16 s 616.385 ms (7.2%)
Total iast_HARDCODED_SECRET_DISABLED 9.151 s 607.499 ms (7.1%)
Total iast_TELEMETRY_OFF 9.237 s 693.555 ms (8.1%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.023 s -
Agent iast 1.154 s 130.368 ms (12.7%)
Agent iast_HARDCODED_SECRET_DISABLED 1.156 s 132.903 ms (13.0%)
Agent iast_TELEMETRY_OFF 1.155 s 131.979 ms (12.9%)
Total tracing 8.546 s -
Total iast 9.177 s 631.794 ms (7.4%)
Total iast_HARDCODED_SECRET_DISABLED 9.179 s 633.555 ms (7.4%)
Total iast_TELEMETRY_OFF 9.237 s 691.906 ms (8.1%)
gantt
    title insecure-bank - break down per module: candidate=1.50.0-SNAPSHOT~e5cd515e88, baseline=1.50.0-SNAPSHOT~7787af738f

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (682.205 ms) : 0, 682205
BytebuddyAgent [candidate] (682.961 ms) : 0, 682961
GlobalTracer [baseline] (239.582 ms) : 0, 239582
GlobalTracer [candidate] (240.468 ms) : 0, 240468
AppSec [baseline] (56.456 ms) : 0, 56456
AppSec [candidate] (55.387 ms) : 0, 55387
Debugger [baseline] (6.251 ms) : 0, 6251
Debugger [candidate] (6.201 ms) : 0, 6201
Remote Config [baseline] (755.146 µs) : 0, 755
Remote Config [candidate] (727.512 µs) : 0, 728
Telemetry [baseline] (13.023 ms) : 0, 13023
Telemetry [candidate] (13.861 ms) : 0, 13861
section iast
BytebuddyAgent [baseline] (801.103 ms) : 0, 801103
BytebuddyAgent [candidate] (805.995 ms) : 0, 805995
GlobalTracer [baseline] (230.605 ms) : 0, 230605
GlobalTracer [candidate] (230.298 ms) : 0, 230298
IAST [baseline] (26.911 ms) : 0, 26911
IAST [candidate] (26.036 ms) : 0, 26036
AppSec [baseline] (52.256 ms) : 0, 52256
AppSec [candidate] (53.277 ms) : 0, 53277
Debugger [baseline] (5.948 ms) : 0, 5948
Debugger [candidate] (5.942 ms) : 0, 5942
Remote Config [baseline] (595.514 µs) : 0, 596
Remote Config [candidate] (594.68 µs) : 0, 595
Telemetry [baseline] (7.943 ms) : 0, 7943
Telemetry [candidate] (7.851 ms) : 0, 7851
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (801.098 ms) : 0, 801098
BytebuddyAgent [candidate] (806.174 ms) : 0, 806174
GlobalTracer [baseline] (230.623 ms) : 0, 230623
GlobalTracer [candidate] (231.663 ms) : 0, 231663
IAST [baseline] (27.924 ms) : 0, 27924
IAST [candidate] (26.314 ms) : 0, 26314
AppSec [baseline] (52.062 ms) : 0, 52062
AppSec [candidate] (53.744 ms) : 0, 53744
Debugger [baseline] (6.023 ms) : 0, 6023
Debugger [candidate] (6.0 ms) : 0, 6000
Remote Config [baseline] (594.953 µs) : 0, 595
Remote Config [candidate] (596.188 µs) : 0, 596
Telemetry [baseline] (7.942 ms) : 0, 7942
Telemetry [candidate] (8.0 ms) : 0, 8000
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (797.425 ms) : 0, 797425
BytebuddyAgent [candidate] (804.679 ms) : 0, 804679
GlobalTracer [baseline] (230.289 ms) : 0, 230289
GlobalTracer [candidate] (232.175 ms) : 0, 232175
IAST [baseline] (28.721 ms) : 0, 28721
IAST [candidate] (25.971 ms) : 0, 25971
AppSec [baseline] (50.25 ms) : 0, 50250
AppSec [candidate] (54.054 ms) : 0, 54054
Debugger [baseline] (6.002 ms) : 0, 6002
Debugger [candidate] (6.062 ms) : 0, 6062
Remote Config [baseline] (596.009 µs) : 0, 596
Remote Config [candidate] (609.468 µs) : 0, 609
Telemetry [baseline] (7.838 ms) : 0, 7838
Telemetry [candidate] (7.896 ms) : 0, 7896
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2025-06-04T18:59:44 2025-06-04T19:02:06
git_branch master rob.carlan/fake-kafka-change
git_commit_date 1749044153 1749062317
git_commit_sha 7787af7 e5cd515
release_version 1.50.0-SNAPSHOT~7787af738f 1.50.0-SNAPSHOT~e5cd515e88
start_time 2025-06-04T18:58:57 2025-06-04T19:01:19
See matching parameters
Baseline Candidate
application petclinic petclinic
ci_job_date 1749063726 1749063726
ci_job_id 968473651 968473651
ci_pipeline_id 66981353 66981353
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-sukzute8-project-304-concurrent-1-b9b4b67b 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-sukzute8-project-304-concurrent-1-b9b4b67b 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
thresholds_or_results results results
variant appsec appsec

Summary

Found 0 performance improvements and 2 performance regressions! Performance is the same for 0 metrics, 12 unstable metrics.

scenario Δ mean http_req_duration Δ mean throughput candidate mean http_req_duration candidate mean throughput baseline mean http_req_duration baseline mean throughput
scenario:load:petclinic:no_agent unstable
[+82.176ms; +85.165ms] or [+1073.676%; +1112.733%]
worse
[-603.623op/s; -578.626op/s] or [-93.562%; -89.687%]
91.325ms 54.037op/s 7.654ms 645.161op/s
scenario:load:petclinic:tracing unstable
[+87.701ms; +91.164ms] or [+1080.488%; +1123.155%]
worse
[-570.386op/s; -541.215op/s] or [-94.114%; -89.300%]
97.550ms 50.260op/s 8.117ms 606.061op/s
Request duration reports for petclinic
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.50.0-SNAPSHOT~e5cd515e88, baseline=1.50.0-SNAPSHOT~7787af738f
    dateFormat X
    axisFormat %s
section baseline
no_agent (7.654 ms) : 7632, 7676
.   : milestone, 7654,
appsec (43.826 ms) : 42867, 44786
.   : milestone, 43826,
appsec_no_iast (52.308 ms) : 50927, 53688
.   : milestone, 52308,
code_origins (67.902 ms) : 63784, 72021
.   : milestone, 67902,
iast (59.157 ms) : 57048, 61266
.   : milestone, 59157,
profiling (11.57 ms) : 11464, 11676
.   : milestone, 11570,
tracing (8.117 ms) : 8091, 8143
.   : milestone, 8117,
section candidate
no_agent (91.325 ms) : 89360, 93289
.   : milestone, 91325,
appsec (111.742 ms) : 108539, 114945
.   : milestone, 111742,
appsec_no_iast (114.889 ms) : 111017, 118761
.   : milestone, 114889,
code_origins (132.566 ms) : 124119, 141013
.   : milestone, 132566,
iast (125.689 ms) : 120385, 130993
.   : milestone, 125689,
profiling (103.382 ms) : 100766, 105998
.   : milestone, 103382,
tracing (97.55 ms) : 95274, 99825
.   : milestone, 97550,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 7.654 ms [7.632 ms, 7.676 ms] -
appsec 43.826 ms [42.867 ms, 44.786 ms] 36.173 ms (472.6%)
appsec_no_iast 52.308 ms [50.927 ms, 53.688 ms] 44.654 ms (583.4%)
code_origins 67.902 ms [63.784 ms, 72.021 ms] 60.249 ms (787.2%)
iast 59.157 ms [57.048 ms, 61.266 ms] 51.504 ms (672.9%)
profiling 11.57 ms [11.464 ms, 11.676 ms] 3.916 ms (51.2%)
tracing 8.117 ms [8.091 ms, 8.143 ms] 463.088 µs (6.1%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 91.325 ms [89.36 ms, 93.289 ms] -
appsec 111.742 ms [108.539 ms, 114.945 ms] 20.418 ms (22.4%)
appsec_no_iast 114.889 ms [111.017 ms, 118.761 ms] 23.565 ms (25.8%)
code_origins 132.566 ms [124.119 ms, 141.013 ms] 41.242 ms (45.2%)
iast 125.689 ms [120.385 ms, 130.993 ms] 34.365 ms (37.6%)
profiling 103.382 ms [100.766 ms, 105.998 ms] 12.058 ms (13.2%)
tracing 97.55 ms [95.274 ms, 99.825 ms] 6.225 ms (6.8%)

Dacapo

@pr-commenter
Copy link

pr-commenter bot commented Jun 4, 2025

Kafka / consumer-benchmark

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master rob.carlan/fake-kafka-change
git_commit_date 1749044153 1749059368
git_commit_sha 7787af7 fdde42c
See matching parameters
Baseline Candidate
ci_job_date 1749060512 1749060512
ci_job_id 968345528 968345528
ci_pipeline_id 66974087 66974087
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
jdkVersion 11.0.25 11.0.25
jmhVersion 1.36 1.36
jvm /usr/lib/jvm/java-11-openjdk-amd64/bin/java /usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/consumer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/consumer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant
vmName OpenJDK 64-Bit Server VM OpenJDK 64-Bit Server VM
vmVersion 11.0.25+9-post-Ubuntu-1ubuntu122.04 11.0.25+9-post-Ubuntu-1ubuntu122.04

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

See unchanged results
scenario Δ mean throughput
scenario:not-instrumented/KafkaConsumerBenchmark.benchConsume same
scenario:only-tracing-dsm-disabled-benchmarks/KafkaConsumerBenchmark.benchConsume same
scenario:only-tracing-dsm-enabled-benchmarks/KafkaConsumerBenchmark.benchConsume same

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant