xdsclient: fix unexpectedly large LoadReportInterval in initial load report request #8348

purnesh42H · 2025-05-22T07:40:12Z

Internal bug: b/416260484

The lastReportedAt field is uninitialized during reporter construction, defaulting to the Zero timestamp. This results in an excessively large LoadReportInterval (calculated as Now() - lastReportedAt) for the initial load report. Initializing lastReportedAt to Now() during construction will ensure the LoadReportInterval is accurately approximated for the first report.

RELEASE NOTES:

xdsclient: fix unexpectedly large LoadReportInterval in initial load report request.

codecov · 2025-05-22T07:45:13Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 82.29%. Comparing base (6995ef2) to head (6bdf940).
Report is 3 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8348      +/-   ##
==========================================
+ Coverage   82.16%   82.29%   +0.13%     
==========================================
  Files         419      419              
  Lines       42034    42052      +18     
==========================================
+ Hits        34537    34607      +70     
+ Misses       6023     5987      -36     
+ Partials     1474     1458      -16

Files with missing lines	Coverage Δ
xds/internal/clients/lrsclient/load_store.go	`93.19% <100.00%> (+0.10%)`	⬆️
xds/internal/xdsclient/load/store.go	`86.47% <100.00%> (+0.06%)`	⬆️

... and 29 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

arjan-bal

Can you please add a test for this to prevent regressions?

purnesh42H · 2025-05-22T09:12:38Z

Can you please add a test for this to prevent regressions?

we can only check if LoadReportInterval is positive but not excessively large. We can't compare exactly since calculation involves time.Now(). I have added test verifications in the existing e2e tests with some tolerance to ensure reportInterval of the first report and subsequent reports is a small, positive duration but greater than configured server LoadReportingInterval.

dfawley · 2025-05-22T23:32:44Z

we can only check if LoadReportInterval is positive but not excessively large. We can't compare exactly since calculation involves time.Now().

Without looking at the details here... Can we hook time.Now and make it return predictable values for our tests?

purnesh42H · 2025-05-23T09:00:38Z

we can only check if LoadReportInterval is positive but not excessively large. We can't compare exactly since calculation involves time.Now().

Without looking at the details here... Can we hook time.Now and make it return predictable values for our tests?

Added the hook. Thought that limits us to only verify with predictable report interval in the unit test of the load store and not in the e2e tests i.e. verifying the report interval that got reported to LRS server but that's probably fine.

dfawley · 2025-05-23T15:06:19Z

Thought that limits us to only verify with predictable report interval in the unit test of the load store and not in the e2e tests

If needed, you can hop through an internal package to be able to set from e2e tests.

dfawley

LGTM, just a couple very minor things.

dfawley · 2025-05-23T20:34:43Z

xds/internal/clients/lrsclient/load_store_test.go

+	wantInterval := 5 * time.Second
+	if stats1.reportInterval != wantInterval {


Optional:

I think the best way I've been recommended of writing this kind of test comparison is:

if got, want := stats1.reportInterval, 5 * time.Second; got != want { t.Fatalf("blah blah = %v; want %v", got, want) }

dfawley · 2025-05-23T20:35:57Z

xds/internal/clients/lrsclient/load_store.go

@@ -25,6 +25,9 @@ import (
 	"time"
 )

+// clockNow is used to get the current time. It can be overridden in tests.
+var clockNow = time.Now


Sorry for the nit, but would you mind renaming this to timeNow to match packagename.SymbolName but without the dot? That's the convention we generally use.

dfawley · 2025-05-23T20:36:28Z

xds/internal/xdsclient/load/store.go

@@ -25,6 +25,9 @@ import (

 const negativeOneUInt64 = ^uint64(0)

+// clockNow is used to get the current time. It can be overridden in tests.
+var clockNow = time.Now


dfawley · 2025-05-23T20:39:08Z

xds/internal/xdsclient/load/store_test.go

+	currentTime = currentTime.Add(5 * time.Second)
+	stats1 := store.Stats(nil)
+
+	if stats1 == nil {


It seems like you want even more from this check, since it's a slice that you dereference element 0 of immediately below.

if len(stats1) == 0 { // or != 1 ? t.Fatalf("store.Stats(nil) = %v; want len() >= 1", stats1) // or == 1? }

purnesh42H · 2025-05-26T04:02:25Z

Thought that limits us to only verify with predictable report interval in the unit test of the load store and not in the e2e tests

If needed, you can hop through an internal package to be able to set from e2e tests.

I did it for the generic lrs client tests but left it for the internal one since that will be external soon.

xds: fix unexpectedly large in initial load report request

76099c8

purnesh42H requested review from dfawley and easwars and removed request for dfawley May 22, 2025 07:40

purnesh42H assigned easwars and dfawley May 22, 2025

purnesh42H requested a review from dfawley May 22, 2025 07:40

purnesh42H changed the title ~~xdsclient: fix unexpectedly large in initial load report request~~ xdsclient: fix unexpectedly large LoadReportInterval in initial load report request May 22, 2025

purnesh42H added the Type: Bug label May 22, 2025

purnesh42H added this to the 1.73 Release milestone May 22, 2025

purnesh42H added the Area: xDS Includes everything xDS related, including LB policies used with xDS. label May 22, 2025

arjan-bal reviewed May 22, 2025

View reviewed changes

arjan-bal assigned purnesh42H May 22, 2025

add test verifications

7313692

purnesh42H requested a review from arjan-bal May 22, 2025 10:02

purnesh42H removed their assignment May 22, 2025

purnesh42H force-pushed the load-store-init-last-report-at branch from 012a5bc to 6a3ebaf Compare May 23, 2025 08:57

purnesh42H force-pushed the load-store-init-last-report-at branch from 6a3ebaf to 65fd12d Compare May 23, 2025 13:42

override time.Now() for predictable report interval calculations

b7c391d

purnesh42H force-pushed the load-store-init-last-report-at branch from 65fd12d to b7c391d Compare May 23, 2025 13:44

dfawley approved these changes May 23, 2025

View reviewed changes

dfawley assigned purnesh42H and unassigned easwars and dfawley May 23, 2025

review 2

6bdf940

purnesh42H force-pushed the load-store-init-last-report-at branch from ef2a9f6 to 6bdf940 Compare May 26, 2025 04:12

purnesh42H merged commit e3ca7f9 into grpc:master May 26, 2025
23 of 24 checks passed

		wantInterval := 5 * time.Second
		if stats1.reportInterval != wantInterval {

xdsclient: fix unexpectedly large LoadReportInterval in initial load report request #8348

xdsclient: fix unexpectedly large LoadReportInterval in initial load report request #8348

Uh oh!

Conversation

purnesh42H commented May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

arjan-bal left a comment

Choose a reason for hiding this comment

Uh oh!

purnesh42H commented May 22, 2025

Uh oh!

dfawley commented May 22, 2025

Uh oh!

purnesh42H commented May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dfawley commented May 23, 2025

Uh oh!

dfawley left a comment

Choose a reason for hiding this comment

Uh oh!

dfawley May 23, 2025

Choose a reason for hiding this comment

Uh oh!

purnesh42H May 26, 2025

Choose a reason for hiding this comment

Uh oh!

dfawley May 23, 2025

Choose a reason for hiding this comment

Uh oh!

purnesh42H May 26, 2025

Choose a reason for hiding this comment

Uh oh!

dfawley May 23, 2025

Choose a reason for hiding this comment

Uh oh!

dfawley May 23, 2025

Choose a reason for hiding this comment

Uh oh!

purnesh42H May 26, 2025

Choose a reason for hiding this comment

Uh oh!

purnesh42H commented May 26, 2025

Uh oh!

Uh oh!

Uh oh!

purnesh42H commented May 22, 2025 •

edited

Loading

codecov bot commented May 22, 2025 •

edited

Loading

purnesh42H commented May 23, 2025 •

edited

Loading