Add metric and logging for activator-autoscaler connectivity #16318

prashanthjos · 2025-12-24T02:32:29Z

Description

This PR adds observability for the websocket connection between the activator and autoscaler components. When the autoscaler is not reachable, operators currently have no easy way to identify this issue, which can lead to autoscaling failures.

Changes

New Metric

Name: kn.activator.autoscaler.reachable
Type: Int64Gauge
Values: 1 (reachable), 0 (not reachable)
Description: Whether the autoscaler is reachable from the activator

New Logging

ERROR level log when autoscaler is not reachable:
- "Autoscaler is not reachable from activator. Stats were not sent." (on send failure)
- "Autoscaler is not reachable from activator." (on connection check failure)

How It Works

The metric is recorded in two scenarios:

Periodic check (every 5s):
- Uses [conn.Status()] to check if connection is established
- Catches: Connection never established, DNS failures, autoscaler not running at startup
On stat send:
- Detects [SendRaw()] failures during actual stat message transmission
- Catches: Network timeouts, connection drops, autoscaler becoming unreachable

Testing

Unit tests pass (go test ./pkg/activator/... -v)
New test TestAutoscalerConnectionStatusMonitor added

Release Note

Include two new activator metrics (`kn.activator.stats.conn.reachable`, `kn.activator.stats.conn.errors`) that reflect the stats reporter connection status

knative-prow · 2025-12-24T02:32:38Z

Welcome @prashanthjos! It looks like this is your first PR to knative/serving 🎉

knative-prow · 2025-12-24T02:32:39Z

Hi @prashanthjos. Thanks for your PR.

I'm waiting for a github.com member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

This change adds observability for the websocket connection between the activator and autoscaler components: - Add `activator_autoscaler_reachable` gauge metric (1=reachable, 0=not reachable) - Log ERROR when autoscaler is not reachable during stat sending - Add periodic connection status monitor (every 5s) to detect connection establishment failures - Add unit tests for the new AutoscalerConnectionStatusMonitor function The metric is recorded in two scenarios: 1. When SendRaw fails/succeeds during stat message sending 2. When the periodic status check detects connection not established This helps operators identify connectivity issues between activator and autoscaler that could impact autoscaling decisions.

thiagomedina · 2025-12-24T13:07:15Z

/ok-to-test

codecov · 2025-12-24T13:13:20Z

Codecov Report

❌ Patch coverage is 69.04762% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.09%. Comparing base (a8803aa) to head (b8ef157).
⚠️ Report is 14 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/activator/stat_reporter.go	50.00%	8 Missing and 1 partial ⚠️
pkg/activator/metrics.go	83.33%	2 Missing and 2 partials ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #16318   +/-   ##
=======================================
  Coverage   80.09%   80.09%           
=======================================
  Files         215      216    +1     
  Lines       13391    13429   +38     
=======================================
+ Hits        10725    10756   +31     
- Misses       2304     2311    +7     
  Partials      362      362

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

prashanthjos · 2025-12-25T02:14:25Z

/retest

prashanthjos · 2025-12-29T19:12:42Z

Related docs PR:

knative/docs#6548

prashanthjos · 2026-01-02T19:14:38Z

@skonto @dsimansk can you please take a look at the PR when you get sometime

linkvt

Thanks for your PR! I understand the requirement but think we could simplify/change the current implementation a bit, see my comments.

I haven't worked that much with this code so far, a more experienced maintainer might have more thoughts about the changes, not sure if they are still in their winter break though.

linkvt · 2026-01-07T10:23:15Z

pkg/activator/stat_reporter.go

+
+// AutoscalerConnectionStatusMonitor periodically checks if the autoscaler is reachable
+// and emits metrics and logs accordingly.
+func AutoscalerConnectionStatusMonitor(ctx context.Context, logger *zap.SugaredLogger, conn StatusChecker, mp metric.MeterProvider) {


I don't think we need this monitor as the stats are already reported every second, see

serving/pkg/activator/handler/concurrency_reporter.go

Line 41 in 09e59ec

const reportInterval = time.Second

This means errors would be detected there already.

Good point! You're right that errors are already detected every second via ReportStats when stats are being sent. However, the monitor handles one edge case: when there's no traffic. If no requests are coming in, ConcurrencyReporter sends nothing (len(msgs) == 0), so ReportStats never calls SendRaw(), and we wouldn't detect a broken connection.

linkvt · 2026-01-07T10:33:01Z

pkg/activator/metrics.go

+
+	meter := provider.Meter(scopeName)
+
+	m.autoscalerReachable, err = meter.Int64Gauge(


I think we should use counters with labels here instead of a gauge. If the connection is flaky we might get the case where we always check when the gauge is 1.
If we have a counter with result=success or result=error we would:

not miss any errors anymore

could create an alert based on the success rate e.g. during the last 5 minutes if success rate is e.g. below 95%

This is a good question - @prashanthjos do you have an opinion here

I'm wondering if adding a second metric that tracks reconnects is sufficient to answer @linkvt's question

Thanks for the feedback! I've implemented a hybrid approach:

Keep the gauge (kn.activator.autoscaler.reachable), provides instant visibility into current connection state
Add a counter (kn.activator.autoscaler.connection_errors_total), monotonically increasing, tracks every error
This addresses @linkvt's concern:

Counter never misses errors (accumulates, doesn't sample)
Enables rate-based alerting: rate(connection_errors_total[5m]) > threshold
Gauge still answers "is it reachable right now?" for real-time dashboards
The counter increments on every error event (both periodic health checks and send failures), so flaky connections will be fully captured.

I understand the argument for keeping the gauge, sounds good to me 👍

With regards to the error counter, I did more research and found the following evaluation that gives a recommendation and also doesn't recommend the approach I proposed: https://promlabs.com/blog/2023/09/19/errors-successes-totals-which-metrics-should-i-expose-to-prometheus/#recommended-for-binary-outcomes-exposing-errors-and-totals

In summary:

Add a counter for total connection checks

Keep the counter for connection checks with errors (exists right now)

This allows you to track the absolute error rate but also the relative error ratio which is in my experience more important than an absolute number

The "errors + totals" pattern makes sense for variable-rate operations like HTTP requests.

However, for this connection health check, I'm not sure the totals counter adds value because:

Check frequency is fixed (every 5s) - the rate is predictable/constant

Any error is actionable, we don't need a ratio to decide if the connection is healthy

The error counter alone lets us alert on rate(connection_errors_total[5m]) > 0, which captures "autoscaler became unreachable.

Partially agree and also thought about that but:

there can be multiple activator pods so there is a difference if you have 1 error with 1 or 10 activators

there are outages/hickups in the environment on every level if your environment is sufficiently large, it's just a statistical certainty with network partitions, disk errors, memory errors, etc. happening at some point in time. Not every error is actionable as errors are expected in large environments, this is also why error budgets are common.

following common patterns (error + total metric) makes it easier for infra teams to write their alerts based on error budget burn rate.

This is at least my experience from working in some large environments handling PBs of data and working on the German railroads sales platform to give my statements some credibility 🙂 .

@linkvt thank you for the detailed feedback and context, this is really helpful! Your points about multiple activator pods and error budgets in large environments make a lot of sense. Following the error + total pattern for burn rate alerting is definitely the right approach for production observability.

Could I accommodate this in a follow-up PR? I'd like to get the current changes merged first and then add the connection_checks_total counter alongside the existing connection_errors_total in a subsequent PR.

pkg/activator/metrics.go

pkg/activator/stat_reporter_test.go

prashanthjos · 2026-01-07T21:41:00Z

Thank you @linkvt for dropping the comments, I will wait for @dsimansk and @skonto comments also, before I start addressing all comments together.

pkg/activator/metrics.go

dprotaso · 2026-01-12T15:07:44Z

pkg/activator/metrics.go

+
+	meter := provider.Meter(scopeName)
+
+	m.autoscalerReachable, err = meter.Int64Gauge(


This is a good question - @prashanthjos do you have an opinion here

I'm wondering if adding a second metric that tracks reconnects is sufficient to answer @linkvt's question

pkg/activator/stat_reporter.go

pkg/activator/stat_reporter_test.go

prashanthjos · 2026-01-13T07:14:07Z

/retest

prashanthjos · 2026-01-13T17:54:39Z

@dprotaso @linkvt I addressed the comments, can you please take a look.

dprotaso · 2026-01-13T20:56:54Z

Chatted on slack - I think we should do the pkg change - it'll make the diff here simpler. I explained how our automation works so once pkg changes is merged our automation will update the deps here.

dprotaso

Minor thing on my end (re: total in metric name).

dprotaso · 2026-01-15T03:12:17Z

pkg/activator/metrics.go

+	}
+
+	m.autoscalerConnectionErrors, err = meter.Int64Counter(
+		"kn.activator.autoscaler.connection_errors_total",


according to OTel documentation total shouldn't be in the name

https://opentelemetry.io/docs/specs/semconv/general/naming/#do-not-use-total

dprotaso · 2026-01-15T03:21:46Z

pkg/activator/metrics.go

+	meter := provider.Meter(scopeName)
+
+	m.autoscalerReachable, err = meter.Int64Gauge(
+		"kn.activator.autoscaler.reachable",


Not a huge deal in my mind but curious if we should make this generic.

For example kn.activator.open_connections and have a peer attribute of autoscaler

Thus if the activator were to ever connect to anything else we would just add a new peer attribute.

And likewise tweak the metric name below

prashanthjos · 2026-01-15T07:01:47Z

@dprotaso done addressed your comments!

dprotaso · 2026-01-15T17:58:10Z

thanks for the change @prashanthjos

/lgtm
/approve

knative-prow · 2026-01-15T17:58:25Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dprotaso, prashanthjos

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [dprotaso]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

knative-prow bot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Dec 24, 2025

knative-prow bot requested review from dsimansk and skonto December 24, 2025 02:33

prashanthjos force-pushed the main branch from 72d0280 to dce652d Compare December 24, 2025 02:38

knative-prow bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Dec 24, 2025

prashanthjos mentioned this pull request Dec 29, 2025

Add documentation for activator <> autoscaler connectivity metrics knative/docs#6548

Merged

prashanthjos added 2 commits December 30, 2025 08:58

Correcting the metric name inline with knative standards

1e54a49

Increasing the code coverage

0a216b5

linkvt reviewed Jan 7, 2026

View reviewed changes

dprotaso reviewed Jan 12, 2026

View reviewed changes

Addressing review comments

59d3df8

prashanthjos mentioned this pull request Jan 14, 2026

Add OnConnect and OnDisconnect callbacks to ManagedConnection knative/pkg#3309

Merged

Integrating the onConnect and onDisconnect calls

485a37a

dprotaso reviewed Jan 15, 2026

View reviewed changes

Adding the peer attribute into the metric

b8ef157

knative-prow bot assigned dprotaso Jan 15, 2026

knative-prow bot added the lgtm Indicates that a PR is ready to be merged. label Jan 15, 2026

knative-prow bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 15, 2026

knative-prow bot merged commit ca98904 into knative:main Jan 15, 2026
91 checks passed


		meter := provider.Meter(scopeName)

		m.autoscalerReachable, err = meter.Int64Gauge(

Add metric and logging for activator-autoscaler connectivity #16318

Add metric and logging for activator-autoscaler connectivity #16318

Conversation

prashanthjos commented Dec 24, 2025 • edited by dprotaso Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes

New Metric

New Logging

How It Works

Testing

Release Note

Uh oh!

knative-prow bot commented Dec 24, 2025

Uh oh!

knative-prow bot commented Dec 24, 2025

Uh oh!

thiagomedina commented Dec 24, 2025

Uh oh!

codecov bot commented Dec 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

prashanthjos commented Dec 25, 2025

Uh oh!

prashanthjos commented Dec 29, 2025

Uh oh!

prashanthjos commented Jan 2, 2026

Uh oh!

linkvt left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

prashanthjos Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

prashanthjos commented Jan 7, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

prashanthjos commented Jan 13, 2026

Uh oh!

prashanthjos commented Jan 13, 2026

Uh oh!

dprotaso commented Jan 13, 2026

Uh oh!

dprotaso left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

prashanthjos commented Jan 15, 2026

Uh oh!

dprotaso commented Jan 15, 2026

prashanthjos commented Dec 24, 2025 •

edited by dprotaso

Loading

codecov bot commented Dec 24, 2025 •

edited

Loading

prashanthjos Jan 15, 2026 •

edited

Loading

dprotaso left a comment •

edited

Loading