Gracefully exit a pipeline or waterfall [discussion]

[djensen47]

I keep running into a situation where I don't need to continue a pipeline or a waterfall because a certain condition has been met.

One way to handle this is to return an error, which exits. The "problem" is that this is recorded as a failure, which really wasn't the case. Alternatively, I could pass a boolean var done along the pipeline and just skip the function if done.

It seems like there should be a better option to exit gracefully. Maybe a special exception class that causes the exit to not be reported as a failure?

[davepacheco]

Can you give a little more detail on your use case? It doesn't sound like what pipeline() and waterfall() were created for, but maybe there's another (similar) abstraction that would be useful here.

[djensen47]

Here's an example:

1. Check cache. Cache miss? Lookup some data from a 3rd party API
2. Check if the data meets certain conditions. (It's okay, and expected, if they don't)
   • Doesn't meet conditions, exit
   • Meets condition: cache and continue to next step
3. Do another lookup from another external API.
4. Check if the data meets certain conditions.
   • Doesn't meet conditions, exit
   • Meets condition: continue to next step

And so on.

I don't want to call every 3rd party API every time because it's both unnecessary and could potentially get $$ expensive.

[djensen47]

I considered node-workflow but it seems a little more than what I need in this particular instance.

[davepacheco]

That workflow makes sense (of course). Usually when I want to do this, I have steps 3-4 check whether the condition from step 2 is already met. In my experience, this is usually cleanest when the whole operation's state is encapsulated in an object, then step 2 updates that state, and subsequent steps check that state and potentially call their callback immediately. I like being explicit about what's being skipped and why.

Here's an example, though it's less clean and uses local variables instead of an object:
https://github.com/joyent/dragnet/blob/master/lib/datasource-manta.js#L582
https://github.com/joyent/dragnet/blob/master/lib/datasource-manta.js#L614

In that example, the optional "outstream" parameter determines whether several steps toward the end of the pipeline need to happen at all. I'm not saying this pattern is great, but on the plus side, the full-length path is pretty clear, and the code for bailing out early in some cases is explicit and clear too.

So there are a few options:

• Do the above: use state explicitly to skip steps. I obviously like this option because it's explicit, requires no new semantics, and keeps vasync simpler.
• Since it doesn't matter whether vasync considers something a failure or not, you could emit an Error, add a property to it that tells the caller not to consider this a real failure, and then just ignore that error in the pipeline callback.
• Create a new abstraction that's similar to pipeline() or waterfall(), but is more flexible. I could imagine something where each step gets a reference to the pipeline (or whatever we call it) and can push another function to call, rather than having the steps be fixed ahead of time. I've thought about this a little, but I think it very quickly gets too complex to be worth it (when you consider you might want pushFront() or pushBack(), and in my experience semantics like that don't layer well).
• Something else?