Triggering a DeployedFlow from a flow with a deploy-time Config broke in >=2.19.7

[chrschorn]

Hey there,

this is a bit of a weird one: when running a flow that uses a deploy-time Config that triggers another flow using DeployedFlow, we get this error:

<flow TriggerFoo step start> failed:
    Flow failed:
    Options were not properly set -- this is an internal error.

This does not happen on 2.19.6 and below. In 2.19.7 and above the issue is present.

Steps to reproduce below:

# trigger_foo.py
from metaflow import DeployedFlow, kubernetes, project, step, Config, FlowSpec

@project(name="test")
class TriggerFoo(FlowSpec):
    config = Config("config", default_value={"hello": "world"})

    @kubernetes()  # I needed an image that has requirements to run `from_argo_workflows` etc.
    @step
    def start(self):
        flow = DeployedFlow.from_argo_workflows(identifier="test.prod.foo")
        flow.trigger()

        self.next(self.end)

    @kubernetes()
    @step
    def end(self):
        pass


if __name__ == "__main__":
    TriggerFoo()

# foo.py
from metaflow import FlowSpec, kubernetes, project, step

@project(name="test")
class Foo(FlowSpec):

    @kubernetes()
    @step
    def start(self):
        self.next(self.end)

    @kubernetes()
    @step
    def end(self):
        pass


if __name__ == "__main__":
    Foo()

python foo.py --production argo-workflows create
python trigger_foo.py --production argo-workflows create
python trigger_foo.py --production argo-workflows trigger

[jpcm-enc]

I'm encountering the same issue.

The diff between 2.19.6 and 2.19.7 indicates that the default value of metaflow_config.CLICK_API_PROCESS_CONFIG was changed from False to True. Indeed, explicitly changing that default value back to False in metaflow_config.py let's me trigger deployed flows from other deployed flows as expected.

However, I have not been able to overwrite that default value through the intended mechanisms. Neither adding "METAFLOW_CLICK_API_PROCESS_CONFIG": "false" in my global metaflow config nor providing it as an environment variable seems to have an effect.

[jpcm-enc]

After looking into it more, I think the issue might lie with how the config values are parsed and lazily expanded. In particular, in the example above when this line is reached

metaflow/metaflow/runner/click_api.py

Line 535 in 7f60dc9

if (config_file is None) ^ (config_value is None):

config_value will have been parsed to config_value={'config': '@!c!@:kv.config'} (due to deferred resolution) and config_value=None. Therefore (config_file is None) ^ (config_value is None)=True and the error is raised. However, the flow Foo doesn't even have a config and in fact it happily gets triggered correctly, when this condition is skipped.

I'm still not sure how to properly solve this, since I'm sure there is some motivation for raising an error at this point.