Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enhance(test): Add timeout and retry to test to make them more stable #5732

Merged
merged 21 commits into from
Mar 12, 2025
Merged
Changes from 1 commit
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
16597eb
fix(test): Use a timer in test_checkpoint_executor_crash_recovery
jkrvivian Feb 27, 2025
208299d
fix(test): Reduce number of synced checkpoints in test_checkpoint_exe…
jkrvivian Feb 27, 2025
492c877
fix(test): Increase timeout in test_checkpoint_executor_crash_recovery
jkrvivian Feb 27, 2025
e6f13ac
fix(network): Add timeout to recv() in test_byzantine_peer_handling
jkrvivian Mar 3, 2025
6d15e20
fix: Resolve comments
jkrvivian Mar 4, 2025
09a2989
fix: Check the first received result in receive_with_timeout
jkrvivian Mar 5, 2025
0d711ed
fix: Add err message to receive_with_timeout
jkrvivian Mar 5, 2025
9e44698
fix: Fix clippy errors
jkrvivian Mar 5, 2025
0334869
fix: increase client send_signatures request timeout
semenov-vladyslav Mar 6, 2025
c927c33
revert: Revert adding receive_with_timeout
jkrvivian Mar 7, 2025
21260a2
fix: add safety timeouts as test_byzantine_peer_handling can loop for…
semenov-vladyslav Mar 7, 2025
5dbe384
fix: ci-fmt
semenov-vladyslav Mar 7, 2025
81f71e6
fix: use timeout function
semenov-vladyslav Mar 7, 2025
1666d91
fix: increase timeout for tests only
semenov-vladyslav Mar 7, 2025
2d1f0b5
fix: revert test timeout overkill
semenov-vladyslav Mar 10, 2025
c859223
test(ignore): test_byzantine_peer_handling still fails on arm64, need…
semenov-vladyslav Mar 10, 2025
39237fe
fix: increate timeout value just of arm64 runners to check if it works
semenov-vladyslav Mar 10, 2025
d897114
Revert "test(ignore): test_byzantine_peer_handling still fails on arm…
semenov-vladyslav Mar 10, 2025
9b54c4a
ci: run rust tests on self-hosted-arm64 to test if randomness timeout…
semenov-vladyslav Mar 11, 2025
69163f2
Revert "ci: run rust tests on self-hosted-arm64 to test if randomness…
semenov-vladyslav Mar 11, 2025
c4f4882
fix: align SEND_PARTIAL_SIGNATURES_TIMEOUT to nextest timeout
muXxer Mar 12, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix: add safety timeouts as test_byzantine_peer_handling can loop for…
…ever
semenov-vladyslav authored and muXxer committed Mar 12, 2025
commit 21260a2db1c0f1669d369b7a3e2f15df131c42dd
49 changes: 33 additions & 16 deletions crates/iota-network/src/randomness/tests.rs
Original file line number Diff line number Diff line change
@@ -439,15 +439,25 @@ async fn test_byzantine_peer_handling() {
);
}

for rx in &mut randomness_rxs[2..] {
// This test can just deadlock, ie. run indefinitely without making any progress.
// We can control it by waiting on expected randomness for `timeout` secs.
// For some reason it takes so much time for honest peers to produce randomness in presence of byzantine peers.
let timeout = std::time::Duration::from_secs(60);
let (rx2_mut, rx3_mut) = randomness_rxs.split_at_mut(3);
let rnd2_fut = rx2_mut[2].recv();
let rnd3_fut = rx3_mut[0].recv();
tokio::select! {
// Validators (2, 3) can communicate normally.
let (epoch, round, bytes) = rx
.recv()
.await
.expect("Validators (2, 3) should receive randomness in epoch 0, round 0");
assert_eq!(0, epoch);
assert_eq!(0, round.0);
assert_ne!(0, bytes.len());
rnds = futures::future::join_all([rnd2_fut, rnd3_fut]) => {
for rnd in rnds {
let (epoch, round, bytes) = rnd
.expect("Validators (2, 3) should receive randomness in epoch 0, round 0");
assert_eq!(0, epoch);
assert_eq!(0, round.0);
assert_ne!(0, bytes.len());
}
},
_ = tokio::time::sleep(timeout) => panic!("Timeout expired to receive randomness"),
}
for rx in &mut randomness_rxs[..2] {
// Validators (0, 1) are byzantine.
@@ -475,15 +485,22 @@ async fn test_byzantine_peer_handling() {
None,
);
}
for rx in &mut randomness_rxs[..2] {

let (rx0_mut, rx1_mut) = randomness_rxs.split_at_mut(1);
let rnd0_fut = rx0_mut[0].recv();
let rnd1_fut = rx1_mut[0].recv();
tokio::select! {
// Validators (0, 1) can communicate normally in new epoch.
let (epoch, round, bytes) = rx
.recv()
.await
.expect("Validators (0, 1) should receive randomness in epoch 1, round 0");
assert_eq!(1, epoch);
assert_eq!(0, round.0);
assert_ne!(0, bytes.len());
rnds = futures::future::join_all([rnd0_fut, rnd1_fut]) => {
for rnd in rnds {
let (epoch, round, bytes) = rnd
.expect("Validators (0, 1) should receive randomness in epoch 1, round 0");
assert_eq!(1, epoch);
assert_eq!(0, round.0);
assert_ne!(0, bytes.len());
}
},
_ = tokio::time::sleep(timeout) => panic!("Timeout expired to receive randomness"),
}
for rx in &mut randomness_rxs[2..] {
// Validators (2, 3) are still on old epoch.