notifications: Opening substreams fails instantly with downgraded connections

[lexnv]

During the following incident, litep2p could not open substreams with collators:

• opening substreams fails instantly with error=ConnectionClosed
• attempts are made every ~5s, suggesting the request is initiated by the peerset of substrate (the peer being in the reserved state)

2025-11-27 14:20:43.927 TRACE tokio-runtime-worker litep2p::notification: [Parachain] open substream peer=PeerId("12D3KooWHai157nsmRK7g81q1erabs3tfCWL5FBqVSSsrG6ZQWDX") protocol=/77afd6190f1554ad45fd0d31aee62aacc33c6db0ea801129acb813f913e0764f/collation/2
2025-11-27 14:20:43.927 DEBUG tokio-runtime-worker litep2p::notification: [Parachain] failed to open substream peer=PeerId("12D3KooWHai157nsmRK7g81q1erabs3tfCWL5FBqVSSsrG6ZQWDX") protocol=/77afd6190f1554ad45fd0d31aee62aacc33c6db0ea801129acb813f913e0764f/collation/2 error=ConnectionClosed
2025-11-27 14:20:48.965 TRACE tokio-runtime-worker litep2p::notification: [Parachain] open substream peer=PeerId("12D3KooWHai157nsmRK7g81q1erabs3tfCWL5FBqVSSsrG6ZQWDX") protocol=/77afd6190f1554ad45fd0d31aee62aacc33c6db0ea801129acb813f913e0764f/collation/2
2025-11-27 14:20:48.965 DEBUG tokio-runtime-worker litep2p::notification: [Parachain] failed to open substream peer=PeerId("12D3KooWHai157nsmRK7g81q1erabs3tfCWL5FBqVSSsrG6ZQWDX") protocol=/77afd6190f1554ad45fd0d31aee62aacc33c6db0ea801129acb813f913e0764f/collation/2 error=ConnectionClosed
2025-11-27 14:20:53.979 TRACE tokio-runtime-worker litep2p::notification: [Parachain] open substream peer=PeerId("12D3KooWHai157nsmRK7g81q1erabs3tfCWL5FBqVSSsrG6ZQWDX") protocol=/77afd6190f1554ad45fd0d31aee62aacc33c6db0ea801129acb813f913e0764f/collation/2
2025-11-27 14:20:53.979 DEBUG tokio-runtime-worker litep2p::notification: [Parachain] failed to open substream peer=PeerId("12D3KooWHai157nsmRK7g81q1erabs3tfCWL5FBqVSSsrG6ZQWDX") protocol=/77afd6190f1554ad45fd0d31aee62aacc33c6db0ea801129acb813f913e0764f/collation/2 error=ConnectionClosed
2025-11-27 14:20:58.989 TRACE tokio-runtime-worker litep2p::notification: [Parachain] open substream peer=PeerId("12D3KooWHai157nsmRK7g81q1erabs3tfCWL5FBqVSSsrG6ZQWDX") protocol=/77afd6190f1554ad45fd0d31aee62aacc33c6db0ea801129acb813f913e0764f/collation/2
2025-11-27 14:20:58.989 DEBUG tokio-runtime-worker litep2p::notification: [Parachain] failed to open substream peer=PeerId("12D3KooWHai157nsmRK7g81q1erabs3tfCWL5FBqVSSsrG6ZQWDX") protocol=/77afd6190f1554ad45fd0d31aee62aacc33c6db0ea801129acb813f913e0764f/collation/2 error=ConnectionClosed
2025-11-27 14:21:04.001 TRACE tokio-runtime-worker litep2p::notification: [Parachain] open substream peer=PeerId("12D3KooWHai157nsmRK7g81q1erabs3tfCWL5FBqVSSsrG6ZQWDX") protocol=/77afd6190f1554ad45fd0d31aee62aacc33c6db0ea801129acb813f913e0764f/collation/2
2025-11-27 14:21:04.001 DEBUG tokio-runtime-worker litep2p::notification: [Parachain] failed to open substream peer=PeerId("12D3KooWHai157nsmRK7g81q1erabs3tfCWL5FBqVSSsrG6ZQWDX") protocol=/77afd6190f1554ad45fd0d31aee62aacc33c6db0ea801129acb813f913e0764f/collation/2 error=ConnectionClosed

Opening substreams are failing from:

litep2p/src/protocol/notification/mod.rs

Line 997 in b16c74c

PeerState::Closed { .. } => match self.service.open_substream(peer) {

The permit for the connection cannot be obtained:

litep2p/src/protocol/transport_service.rs

Line 550 in b16c74c

let permit = connection.try_get_permit().ok_or(SubstreamError::ConnectionClosed)?;

litep2p/src/protocol/connection.rs

Line 110 in b16c74c

ConnectionType::Inactive(inactive) => Some(Permit::new(inactive.upgrade()?)),

Timeline

The peer enters in an unrecoverable state due to:

• T0: Primary connection 830585
• T1: Secondary connection 831438 established
• T2: Secondary connection 831438 closed (WebSocket error)
• T3: Primary connection 830585 is still there but downgraded/inactive with dead WeakSender
• T4: /collator protocol tries to open a substream but the primary connection is not cleanedup

Discovered during:

• [ElasticScaling] Enable ElasticScaling on AHKusama and AHPolkadot polkadot-sdk#10425