People trying to commit to this project and create recipes are very likely to look up expected behavior of pipeline transformations (PTransforms) and they are very likely to get some nasty surprises as things are currently set up. It is extremely frustrating that Beam's pipeline definition DSL is so heavily dependent upon mutation of hidden state, but that appears to be a fundamental limitation of the pipeline API and a likely source for future bugs.
LET IT BE KNOWN: if you're ever writing a DSL for DAG construction, there's no good reason to avoid immutability. This doesn't need to be high performance! (I'm looking at you, beam devs...)
Anyway, we should probably torch anything that's going to introduce bugs and violate expectations of the underlying Beam libraries. The strategy I've proposed can be found here #168 and a compliant recipe can be found here:
People trying to commit to this project and create recipes are very likely to look up expected behavior of pipeline transformations (
PTransforms) and they are very likely to get some nasty surprises as things are currently set up. It is extremely frustrating that Beam's pipeline definition DSL is so heavily dependent upon mutation of hidden state, but that appears to be a fundamental limitation of the pipeline API and a likely source for future bugs.LET IT BE KNOWN: if you're ever writing a DSL for DAG construction, there's no good reason to avoid immutability. This doesn't need to be high performance! (I'm looking at you, beam devs...)
Anyway, we should probably torch anything that's going to introduce bugs and violate expectations of the underlying Beam libraries. The strategy I've proposed can be found here #168 and a compliant recipe can be found here: