Chain fault testing framework + Enable auto-rejoin #51
Conversation
There was a problem hiding this comment.
Pull request overview
This PR introduces a chain fault testing framework to test node behavior when blockchain connectivity is disrupted. The framework leverages Toxiproxy to simulate network faults between nodes and blockchain endpoints, enabling automated testing of node resilience and validator kick mechanisms.
Key changes:
- Adds new chain fault test infrastructure with proxy-based blockchain connection simulation
- Implements helper functions to disable/enable chain connections for individual nodes
- Extends testnet configuration to support signing round timeout customization
- Adds utility function in Actions to bulk update complaint configurations for testing
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| rust/lit-node/lit-node/tests/toxiproxy/chain_faults.rs | New test file implementing chain connection fault testing, specifically testing node kick behavior when a node loses blockchain connectivity |
| rust/lit-node/lit-node/tests/toxiproxy/mod.rs | Exports the new chain_faults module |
| rust/lit-node/lit-node/tests/common/faults.rs | Adds helper functions for chain-specific proxy setup and fault injection (enable/disable chain connections) |
| rust/lit-node/lit-node-testnet/src/testnet/node_config.rs | Extends node configuration builder to support custom signing round timeout settings |
| rust/lit-node/lit-node-testnet/src/testnet/actions.rs | Adds bulk complaint configuration update utility for testing |
| rust/lit-node/lit-node-testnet/src/lib.rs | Integrates signing round timeout configuration into TestSetupBuilder |
| rust/lit-core/lit-blockchain/src/contracts/mod.rs | Implements proxy port remapping for blockchain providers when proxy_chatter feature is enabled |
| rust/lit-node/lit-node/Cargo.toml | Updates feature flags to include blockchain testing support |
| rust/lit-node/lit-node-testnet/Cargo.toml | Removes duplicate proxy-chatter feature definition |
| rust/lit-core/lit-blockchain/Cargo.toml | Adds testing and proxy_chatter feature flags |
| rust/lit-node/shiva/log_levels.toml | Increases lit_blockchain log level to trace for debugging |
| rust/lit-node/lit-node/src/git_info.rs | Updates git commit hash (auto-generated) |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot <[email protected]>
|
PASS [ 43.194s] (3/3) lit_node::test toxiproxy::perf_tests::load_with_no_latency |
GTC6244
changed the title
DashKash54
left a comment
DashKash54
left a comment
There was a problem hiding this comment.
Looks good but left a few questions
There was a problem hiding this comment.
Is this the only place we need to update the provider?
There was a problem hiding this comment.
Yup - code is nicely centralized. ;-)
| } | ||
|
|
||
| // shortcut function to update all complaint configs to the same interval and tolerance for testing | ||
| pub async fn update_all_complaint_configs( |
There was a problem hiding this comment.
Which testing scenario is this used for?
There was a problem hiding this comment.
I used it for the auto-rejoin test ( not part of this PR ). Also used in the upgrade tests ( also not part of this PR! ).
It's basically a shortcut to make tests work faster ;-)
There was a problem hiding this comment.
Can remove the test: kick_node_who_loses_chain_connection since it's being tested in the auto-rejoin test?
There was a problem hiding this comment.
Yeah, it's probably redundant now. I might leave it and tag it as #[ignore]
…://github.com/LIT-Protocol/lit-peer into feature/node-4901-chain-fault-tests-framework
glitch003
left a comment
glitch003
left a comment
There was a problem hiding this comment.
yay excited to see this test!
Pull request overview
This PR introduces a chain fault testing framework to test node behavior when blockchain connectivity is disrupted. The framework leverages Toxiproxy to simulate network faults between nodes and blockchain endpoints, enabling automated testing of node resilience and validator kick mechanisms.
Key changes: