Skip to content

Conversation

mre
Copy link
Member

@mre mre commented Oct 1, 2025

  • Moved from sequential to parallel job processing using multiprocessing.Pool
  • Grouped jobs by source to prevent rate limiting (same source runs sequentially,
    different sources run in parallel)
  • Fixed multiprocessing pickling issues by moving worker functions from main.py
    to separate worker.py module
  • Introduced a PodcastJob dataclass for workers
  • Added if name == 'main': guard for proper multiprocessing behavior.
  • Each worker process gets its own database connection to avoid shared state issues

@mre mre requested a review from woolfg October 1, 2025 12:55
@mre mre added the enhancement New feature or request label Oct 1, 2025
@woolfg woolfg merged commit 30bb4af into main Oct 6, 2025
13 checks passed
@woolfg woolfg deleted the parallel-manager branch October 6, 2025 10:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants