BlockProviderExecutor bad state mechanism is not thread safe

Anywhere that an executor is checked for being in a bad state, and then the subsequent code relies on the fact that it is not, is race prone -- there is nothing stopping bad state being set by another thread after the state check but before relying behaviour.

This came up offline in review of #3994 but it is generally the case anywhere the bad state mechanism is used.

See also issue #3993 