Skip to content

test: feature_asset_locks intermittent llmq_test_platform quorum timeout #7310

New issue
New issue
Closed
#7509
#7411
Closed
test: feature_asset_locks intermittent llmq_test_platform quorum timeout#7310
#7509
#7411

Description

@thepastaclaw
thepastaclaw
opened on May 7, 2026

Summary

feature_asset_locks.py intermittently times out while waiting for a freshly mined llmq_test_platform quorum to appear in quorum list.

This reappeared in PR #7309, but that PR only changes Qt file dialog code (src/qt/bitcoin.cpp, src/qt/guiutil.cpp, src/qt/guiutil.h, src/qt/intro.cpp). The failing test is unrelated to the PR diff. The latest develop CI for the PR base (5fd84aa23de0d79770bfc25c4c1393eb4eadbcf8) is green, so this looks intermittent rather than a deterministic base failure.

A previous closed issue, #6748, captured the same failure mode in feature_asset_locks.py at the same wait_for_quorum_list() point.

CI evidence

PR: #7309 (fix(qt): add non-native file dialog option)

Failing check:

Failure:

TestFramework (INFO): Mining next quorum to check tx 'asset_unlock_tx_late' is still valid...
TestFramework (INFO): Mining quorum: llmq_type_name=llmq_test_platform, llmq_type=106, expected_members=2, expected_connections=1, expected_contributions=2, expected_commitments=2
TestFramework (INFO): Waiting for quorum to appear in the list
...
AssertionError: Predicate ''''
        def wait_func():
            return quorum_hash in self.nodes[0].quorum('list')[llmq_type_name]
''' not true after 60.0 seconds

Stack:

feature_asset_locks.py:430 test_asset_unlocks()
feature_asset_locks.py:246 mine_quorum_2_nodes()
test_framework.py:2208 mine_quorum()
test_framework.py:2123 wait_for_quorum_list()

Scope check

PR #7309 touches only Qt GUI file dialog plumbing:

src/qt/bitcoin.cpp
src/qt/guiutil.cpp
src/qt/guiutil.h
src/qt/intro.cpp

No functional test, quorum, consensus, mempool, asset lock, or LLMQ code is changed by the PR.

Prior matching failure

Closed issue #6748 reported the same feature_asset_locks.py flake with the same wait_for_quorum_list() predicate timeout while mining an llmq_test_platform quorum for asset_unlock_tx_late.

Conclusion

This appears to be a pre-existing intermittent feature_asset_locks.py / LLMQ test timing flake, not a regression caused by #7309.

Activity

  1. thepastaclaw commented on May 18, 2026

    @thepastaclaw
    thepastaclaw
    on May 18, 2026
    CollaboratorAuthor

    Additional CI evidence from #7329:

    Scope check: #7329 only changes:

    test/functional/feature_governance.py
test/functional/test_framework/governance.py

    It does not touch feature_asset_locks.py, LLMQ, asset locks, mempool sync, or consensus code.

    Observed failure mode in feature_asset_locks.py after two earlier failed attempts and a final long-running attempt:

    feature_asset_locks.py:452 in test_asset_unlocks
  self.generate_batch(25)
feature_asset_locks.py:240 in generate_batch
  self.generate(self.nodes[1], batch)
test_framework.py:878 in sync_all
  self.sync_mempools(nodes)
test_framework.py:871 in sync_mempools
  raise AssertionError("Mempool sync timed out after {}s:{}".format(...))

AssertionError: Mempool sync timed out after 240s:
  {'580775b273851cb8a7704f1fb2111677eaa49133e2f0c872821baaa382179aa8',
   '8a92e18cf1dc7339ef1c48c6550402463804faf202c27c8e0f2cb772a3cfea28',
   'a2487ae374e2b1c870c091af481178f606e13b1c9dc781091b88fb2d5315496f'}
  set()
  set()
  set()

    The failed run was on PR head fefdd42fc87ec7cc51b4cf97349390a8dc1b3b96. This looks like another pre-existing intermittent feature_asset_locks.py timing/sync failure, not caused by #7329.

  2. thepastaclaw commented on Jun 9, 2026

    @thepastaclaw
    thepastaclaw
    on Jun 9, 2026
    CollaboratorAuthor

    Additional CI evidence from #7348:

    Scope check: #7348 only changes:

    src/net_processing.cpp
test/functional/p2p_tx_download.py

    It does not touch feature_asset_locks.py, LLMQ quorum mining/test framework code, asset locks, mempool sync, or Platform quorum code. The PR-specific test (p2p_tx_download.py) passed in the same failing job.

    Observed failure mode was another feature_asset_locks.py quorum-list timeout after retries:

    feature_asset_locks.py failed at attempt 1/3
feature_asset_locks.py failed at attempt 2/3
...
feature_asset_locks.py:328 in test_asset_locks
  self.mine_quorum(llmq_type_name='llmq_test_instantsend', expected_members=2,
                   expected_connections=1, expected_contributions=2,
                   expected_commitments=2, llmq_type=104)
test_framework.py:2208 in mine_quorum
  self.wait_for_quorum_list(q, nodes, llmq_type_name=llmq_type_name)
test_framework.py:2123 in wait_for_quorum_list
  self.wait_until(wait_func, timeout=timeout, sleep=0.05)

AssertionError: Predicate ''''
        def wait_func():
            return quorum_hash in self.nodes[0].quorum('list')[llmq_type_name]
''' not true after 60.0 seconds

    Recent develop CI evidence for the same base commit also supports this being intermittent/pre-existing rather than caused by #7348: run https://github.com/dashpay/dash/actions/runs/27096757128 on 317917a89e9a had linux64-test / Test source and linux64_sqlite-test / Test source both passing, while the overall run failed elsewhere.

    Conclusion: this is another pre-existing intermittent feature_asset_locks.py / LLMQ quorum timing failure, not a regression caused by #7348.

  3. thepastaclaw commented on Jun 18, 2026

    @thepastaclaw
    thepastaclaw
    on Jun 18, 2026
    CollaboratorAuthor

    Additional CI evidence from #7348:

    Scope check: GitHub's PR diff only changes:

    src/net_processing.cpp
test/functional/p2p_tx_download.py

    It does not touch feature_asset_locks.py, LLMQ quorum mining/test framework code, asset locks, mempool sync, or Platform quorum code. The targeted p2p_tx_download.py test passed in the same CI run.

    Observed failure mode:

    feature_asset_locks.py:680 in test_withdrawals_fork
  self.mine_quorum_2_nodes()
feature_asset_locks.py:246 in mine_quorum_2_nodes
  self.mine_quorum(... llmq_type_name='llmq_test_platform' ...)
test_framework.py:2123 in wait_for_quorum_list
  self.wait_until(wait_func, timeout=timeout, sleep=0.05)
AssertionError: quorum_hash in self.nodes[0].quorum('list')[llmq_type_name] not true after 60.0 seconds

    This is the same intermittent quorum-list timeout pattern tracked here, surfacing again on an unrelated oversized-notfound PR.

  4. thepastaclaw commented on Jun 25, 2026

    @thepastaclaw
    thepastaclaw
    on Jun 25, 2026
    CollaboratorAuthor

    Additional tracker scope check from thepastaclaw/tracker#1559 confirms the current #7348 red check is this same pre-existing flake, not an oversized-notfound regression.

    Failure root cause from the log is feature_asset_locks.py timing out in mine_quorum_2_nodes() / wait_for_quorum_list() while waiting for a new llmq_test_platform quorum hash to appear in quorum list:

    feature_asset_locks.py:680 in test_withdrawals_fork
  self.mine_quorum_2_nodes()
feature_asset_locks.py:246 in mine_quorum_2_nodes
  self.mine_quorum(... llmq_type_name='llmq_test_platform' ...)
test_framework.py:2123 in wait_for_quorum_list
  self.wait_until(wait_func, timeout=timeout, sleep=0.05)
AssertionError: quorum_hash in self.nodes[0].quorum('list')[llmq_type_name] not true after 60.0 seconds

    Scope check: GitHub's PR file list is only src/net_processing.cpp and test/functional/p2p_tx_download.py; the targeted p2p_tx_download.py test passed in the same CI run. No branch change, push, rebase, empty commit, or rerun was performed.

  5. thepastaclaw commented on Jun 27, 2026

    @thepastaclaw
    thepastaclaw
    on Jun 27, 2026
    CollaboratorAuthor

    Additional CI evidence from #7383:

    Scope check: #7383 only changes:

    src/wallet/test/walletdb_tests.cpp
src/wallet/walletdb.cpp

    It does not touch feature_asset_locks.py, LLMQ quorum mining/test framework code, asset locks, mempool sync, or Platform quorum code.

    Observed failure mode in the TSan job was another intermittent feature_asset_locks.py sync/propagation failure after three attempts. The final failure happened after mining the next llmq_test_platform quorum for asset_unlock_tx_late: testmempoolaccept returned the expected allowed result, but before the test checked the expected empty mempool size, the transaction had propagated from another node and node0's mempool size was already 1:

    feature_asset_locks.py:435 in test_asset_unlocks
  self.check_mempool_result(tx=asset_unlock_tx_late, result_expected={'allowed': True, ...})
feature_asset_locks.py:169 in check_mempool_result
  self.check_mempool_size()
feature_asset_locks.py:158 in check_mempool_size
  assert_equal(node.getmempoolinfo()['size'], self.mempool_size)
AssertionError: not(1 == 0)

node0 ... AcceptToMemoryPool: accepted ab22f9b5... (poolsz 1 txn, 0 kB)

    This is not a wallet DB regression from #7383; it is another pre-existing intermittent feature_asset_locks.py timing/sync failure in the LLMQ/asset-unlock part of the test.

  6. thepastaclaw commented on Jun 30, 2026

    @thepastaclaw
    thepastaclaw
    on Jun 30, 2026
    CollaboratorAuthor

    Additional CI evidence from #7395:

    Scope check: #7395 only changes GitHub workflow YAML files:

    .github/workflows/label-merge-conflicts.yml
.github/workflows/merge-check.yml
.github/workflows/release_docker_hub.yml
.github/workflows/semantic-pull-request.yml

    It does not touch feature_asset_locks.py, LLMQ quorum mining/test framework code, asset locks, mempool sync, or Platform quorum code. The same CI run had linux64-test, linux64_sqlite-test, linux64_ubsan-test, linux64_nowallet-test, and linux64_multiprocess-test pass; only the TSan functional test shard failed.

    Observed failure mode was another intermittent feature_asset_locks.py Platform quorum availability failure after retries:

    feature_asset_locks.py failed at attempt 1/3
feature_asset_locks.py failed at attempt 2/3
...
feature_asset_locks.py:350 in test_asset_unlocks
  asset_unlock_tx = self.create_assetunlock(101, COIN, pubkey)
feature_asset_locks.py:110 in create_assetunlock
  quorumHash = mninfo[0].get_node(self).quorum("selectquorum", llmq_type_test, request_id)["quorumHash"]
test_framework.authproxy.JSONRPCException: no quorums active (-1)

    The test log immediately before the failure shows the test had just mined/observed a platform quorum and then failed while selecting a quorum for the first asset-unlock transaction. This matches the existing intermittent LLMQ/platform-quorum timing class tracked here, not a regression from the workflow action-pin PR.

  7. thepastaclaw commented on Jul 1, 2026

    @thepastaclaw
    thepastaclaw
    on Jul 1, 2026
    CollaboratorAuthor

    Additional tracker confirmation from thepastaclaw/tracker#1676 for #7395:

    Scope re-check: the true PR diff against its merge base only changes GitHub workflow files:

    .github/workflows/label-merge-conflicts.yml
.github/workflows/merge-check.yml
.github/workflows/release_docker_hub.yml
.github/workflows/semantic-pull-request.yml

    It does not touch feature_asset_locks.py, LLMQ quorum mining/test framework code, asset locks, mempool sync, or Platform quorum code. A later develop CI run at dfe1e5d8b660e7d160f1f5076912b7c3cbd26257 passed linux64_tsan-test, linux64-test, and linux64_ubsan-test: https://github.com/dashpay/dash/actions/runs/28469578919

    No new branch change, push, rebase, empty commit, or rerun was performed. This remains consistent with the existing intermittent feature_asset_locks.py / LLMQ platform-quorum timing class tracked here, not a regression from #7395.

  8. thepastaclaw commented on Jul 3, 2026

    @thepastaclaw
    thepastaclaw
    on Jul 3, 2026
    CollaboratorAuthor

    Additional tracker re-check from thepastaclaw/tracker#1739 for the same #7383 CI run confirms this is still pre-existing flakiness, not a wallet DB regression.

    Scope: GitHub PR diff is only src/wallet/walletdb.cpp and src/wallet/test/walletdb_tests.cpp. The failing test is feature_asset_locks.py, which failed after asset_unlock_tx_late testmempoolaccept because node0's mempool size had already become 1 while the test still expected 0. No branch mutation was made.

  9. thepastaclaw commented on Jul 3, 2026

    @thepastaclaw
    thepastaclaw
    on Jul 3, 2026
    CollaboratorAuthor

    Additional tracker re-check from thepastaclaw/tracker#1761 for the same #7383 CI run confirms this is still pre-existing flakiness, not a wallet DB regression.

    Scope: GitHub PR diff is only src/wallet/walletdb.cpp and src/wallet/test/walletdb_tests.cpp. The failing test is feature_asset_locks.py, which failed after asset_unlock_tx_late testmempoolaccept because node0's mempool size had already become 1 while the test still expected 0. No branch mutation was made. No new findings beyond the prior tracker#1739 diagnosis.

  10. thepastaclaw commented on Jul 4, 2026

    @thepastaclaw
    thepastaclaw
    on Jul 4, 2026
    CollaboratorAuthor

    Additional CI evidence from #7405:

    Scope check: #7405 changes only:

    src/test/llmq_commitment_tests.cpp

    It does not touch feature_asset_locks.py, LLMQ quorum mining/test framework code used by that functional test, asset locks, mempool sync, or Platform quorum code. The PR-specific C++ test target was built successfully; the failing job failed only in the functional test shard.

    Observed failure mode is the same intermittent Platform-quorum availability class tracked here. feature_asset_locks.py had already logged Credit pool amount matched with '1000141421', then failed while creating the first asset unlock because quorum selection found no active quorums:

    feature_asset_locks.py:350 in test_asset_unlocks
  asset_unlock_tx = self.create_assetunlock(101, COIN, pubkey)
feature_asset_locks.py:110 in create_assetunlock
  quorumHash = mninfo[0].get_node(self).quorum("selectquorum", llmq_type_test, request_id)["quorumHash"]
test_framework.authproxy.JSONRPCException: no quorums active (-1)

    No branch mutation, push, rebase, empty commit, or CI rerun was performed. This remains consistent with pre-existing feature_asset_locks.py / LLMQ Platform-quorum timing flakiness, not a regression from #7405.

  11. thepastaclaw commented on Jul 4, 2026

    @thepastaclaw
    thepastaclaw
    on Jul 4, 2026
    CollaboratorAuthor

    Additional tracker re-check from thepastaclaw/tracker#1782 for the same #7383 CI run confirms this is still pre-existing flakiness, not a wallet DB regression.

    Scope: GitHub PR diff is only src/wallet/walletdb.cpp and src/wallet/test/walletdb_tests.cpp. The failing test is feature_asset_locks.py, which failed after asset_unlock_tx_late testmempoolaccept because node0's mempool size had already become 1 while the test still expected 0. No branch mutation was made. No new findings beyond the prior tracker#1761 diagnosis.

  12. thepastaclaw commented on Jul 6, 2026

    @thepastaclaw
    thepastaclaw
    on Jul 6, 2026
    CollaboratorAuthor

    Additional tracker re-check from thepastaclaw/tracker#1788 for the same #7383 CI run confirms this remains pre-existing flakiness, not a wallet DB regression.

    Scope: the mandatory two-dot diff was run, but upstream/develop has advanced past the PR base and includes unrelated develop-side drift. The actual merge-base PR diff is still only:

    src/wallet/test/walletdb_tests.cpp
src/wallet/walletdb.cpp

    The failing test is feature_asset_locks.py, which failed after asset_unlock_tx_late testmempoolaccept because node0's mempool size had already become 1 while the test still expected 0. No PR branch mutation was made. No new findings beyond the prior tracker#1782 diagnosis.

  13. thepastaclaw commented on Jul 7, 2026

    @thepastaclaw
    thepastaclaw
    on Jul 7, 2026
    CollaboratorAuthor

    Additional CI evidence from #7404:

    Observed failure mode is the same intermittent feature_asset_locks.py / Platform-quorum availability timeout tracked here. The test failed after retries while mining an llmq_test_platform quorum in test_v24_fork():

    feature_asset_locks.py:742 in test_v24_fork
  self.mine_quorum_2_nodes()
feature_asset_locks.py:247 in mine_quorum_2_nodes
  self.mine_quorum(... llmq_type_name='llmq_test_platform' ...)
test_framework.py:2125 in wait_for_quorum_list
  self.wait_until(wait_func, timeout=timeout, sleep=0.05)
AssertionError: quorum_hash in self.nodes[0].quorum('list')[llmq_type_name] not true after 60.0 seconds

    Scope check: the actual PR diff is limited to:

    src/llmq/utils.cpp
src/llmq/utils.h
src/rpc/quorums.cpp
test/functional/feature_llmq_evo.py
test/functional/feature_llmq_rotation.py

    The failing test file feature_asset_locks.py is not changed. The PR's LLMQ utility changes preserve the existing production quorum-member path for normal quorum mining; the new prediction helpers are used by quorum dkginfo, while all source-build jobs and the PR-touched functional-test areas passed. This failure also occurred only in the TSan functional shard, consistent with this issue's existing timing/availability class.

    No branch mutation, push, rebase, empty commit, or CI rerun was performed for this unrelated flake.

  14. PastaPastaPasta
    closed this as completedin cbab254on Jul 8, 2026

  15. thepastaclaw commented on Jul 8, 2026

    @thepastaclaw
    thepastaclaw
    on Jul 8, 2026
    CollaboratorAuthor

    Additional occurrence while triaging #7350 CI:

  16. thepastaclaw
    added a commit that references this issue on Jul 8, 2026

    Merge dashpay#7411: test: stabilize asset locks functional test

    cbab254

  17. thepastaclaw commented on Jul 8, 2026

    @thepastaclaw
    thepastaclaw
    on Jul 8, 2026
    CollaboratorAuthor

    Additional tracker re-check from thepastaclaw/tracker#1819 for the same #7383 CI run confirms this remains pre-existing flakiness, not a wallet DB regression.

    Scope: the required two-dot diff was run, but upstream/develop has advanced past the PR base and includes unrelated develop-side drift. The actual merge-base PR diff is still only:

    src/wallet/test/walletdb_tests.cpp
src/wallet/walletdb.cpp

    The failing test is feature_asset_locks.py, which failed after asset_unlock_tx_late testmempoolaccept because node0's mempool size had already become 1 while the test still expected 0. No PR branch mutation was made. No new findings beyond the prior #1788 diagnosis.

  18. thepastaclaw commented on Jul 8, 2026

    @thepastaclaw
    thepastaclaw
    on Jul 8, 2026
    CollaboratorAuthor

    Additional CI evidence from #7422:

    Observed failure mode is the same intermittent feature_asset_locks.py / Platform-quorum availability timeout tracked here. The final attempt failed in test_v24_fork() while mining an llmq_test_platform quorum:

    feature_asset_locks.py:742 in test_v24_fork
  self.mine_quorum_2_nodes()
feature_asset_locks.py:247 in mine_quorum_2_nodes
  self.mine_quorum(... llmq_type_name='llmq_test_platform' ...)
test_framework.py:2125 in wait_for_quorum_list
  self.wait_until(wait_func, timeout=timeout, sleep=0.05)
AssertionError: quorum_hash in self.nodes[0].quorum('list')[llmq_type_name] not true after 60.0 seconds

    Scope check: the actual merge-base PR diff only changes:

    .github/workflows/handle_potential_conflicts.py
.github/workflows/predict-conflicts.yml
.github/workflows/test_handle_potential_conflicts.py

    The mandatory two-dot diff against current upstream/develop includes unrelated develop-side drift because the branch base is older, but the GitHub PR diff / merge-base diff does not touch feature_asset_locks.py, LLMQ quorum mining, asset locks, or the functional-test framework. A local non-TSAN run of feature_asset_locks.py on the PR worktree passed with --timeout-factor=4. No PR branch mutation, push, rebase, empty commit, or CI rerun was performed.

  19. thepastaclaw commented on Jul 8, 2026

    @thepastaclaw
    thepastaclaw
    on Jul 8, 2026
    CollaboratorAuthor

    Additional CI evidence from #7418 after #7310 was closed:

    Observed failure mode is the same feature_asset_locks.py / Platform-quorum timeout class tracked here. The final retry failed in test_v24_fork() while mining the next llmq_test_platform quorum:

    feature_asset_locks.py:734 in test_v24_fork
  self.mine_quorum_2_nodes()
feature_asset_locks.py:247 in mine_quorum_2_nodes
  self.mine_quorum(... llmq_type_name='llmq_test_platform' ...)
test_framework.py:2125 in wait_for_quorum_list
  self.wait_until(wait_func, timeout=timeout, sleep=0.05)
AssertionError: quorum_hash in self.nodes[0].quorum('list')[llmq_type_name] not true after 60.0 seconds

    Scope check: the actual merge-base PR diff changes only:

    src/llmq/net_signing.cpp
src/llmq/signing_shares.h
src/test/llmq_utils_tests.cpp

    The failing feature_asset_locks.py file is not part of the PR diff. Current develop includes #7411 (test: stabilize asset locks functional test), while this PR head predates that unrelated stabilization change. No PR branch mutation, push, rebase, empty commit, or CI rerun was performed for this unrelated failure.

  20. PastaPastaPasta
    added 2 commits that reference this issue on Aug 7, 2026

    test: ignore MnEHF signal txs in asset lock mempool accounting

    03ff1ff

    Merge #7509: test: fix remaining feature_asset_locks flake from auton…

    e25f992
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions