[Bug] Microbatch can cause dbt runs to hang

[QMalcolm]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

You can cause microbatch models to hang the dbt runtime. This is most easily reproduced by providing the --fail-fast flag with a microbatch that has an error in it, full reproduction steps can be found https://github.com/dbt-labs/TMP_MICROBATCH_HANGING_REPRO. The hanging has been observed in other manners, but they are not as easily reproduced.

Expected Behavior

Microbatch models shouldn't cause the dbt runtime to hang.

Steps To Reproduce

https://github.com/dbt-labs/TMP_MICROBATCH_HANGING_REPRO

Relevant log output

Environment

- OS: MacOS 15.1.1
- Python: 3.11.9
- dbt: 1.9.2

Which database adapter are you using with dbt?

snowflake

Additional Context

No response

[john-rotenstein]

We experienced a similar issue in dbt Cloud when using Microbatching and Slim CI. dbt acknowledged the problem: "the root cause of this issue has been identified and a fix is being implemented"