Skip to content

Add timeouts to CI/CD pipeline to prevent hanging testsΒ #977

@acul71

Description

@acul71

Problem

Currently, the CI/CD pipeline has no timeouts configured, which can lead to:

  1. Tests hanging indefinitely - Some tests (especially interop tests) can hang for hours
  2. CI resources wasted - Jobs can run for up to 6 hours (GitHub Actions default) before being killed
  3. Developer experience issues - Long-running CI makes it difficult to get quick feedback

Current Situation

  • GitHub Actions: No job-level or step-level timeouts configured
  • Tox configuration: No pytest timeouts in tox.ini
  • Makefile: No timeouts in make test or make pr commands
  • pytest-timeout plugin: Installed but no global timeout configured

Evidence

Recent test runs show FloodSub interop tests timing out after 80+ seconds locally, but in CI these could run for the full 6-hour GitHub Actions timeout.

Proposed Solution

  1. Add job timeout in GitHub Actions workflow:

    jobs:
      tox:
        timeout-minutes: 60  # 1 hour timeout
  2. Add pytest timeouts in tox.ini:

    commands=
        core: pytest -n auto --timeout=300 {posargs:tests/core}
        interop: pytest -n auto --timeout=600 {posargs:tests/interop}
        demos: pytest -n auto --timeout=300 {posargs:tests/core/examples/test_examples.py}
  3. Update Makefile to include timeouts:

    test:
    	python -m pytest tests -n auto --timeout=300

Benefits

  • Faster feedback - Tests fail quickly instead of hanging
  • Resource efficiency - Prevents CI from running for hours
  • Better debugging - Clear timeout errors instead of indefinite hangs
  • Improved developer experience - Faster CI feedback loop

Related

This issue was discovered while working on PR #939 (flood publishing) where FloodSub interop tests are timing out.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesttestingimprovement to testing

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions