Skip to content

exclude_tl_dirs drops same-named directories at every depth, not only at the top level #3408

Open

Description

@LemonCondor

What happens

MetaflowPackage documents exclude_tl_dirs as directories that are "only excluded at the top-level". In practice they are excluded at every depth. With data excluded, a nested package directory such as mypkg/data/ is also dropped from the code package, so mypkg/data/loader.py is silently missing at runtime.

Reproduction

import os, tempfile
from metaflow.packaging_sys.utils import walk

with tempfile.TemporaryDirectory() as root:
    for p in ["data/big.csv", "mypkg/data/loader.py", "mypkg/__init__.py"]:
        os.makedirs(os.path.join(root, os.path.dirname(p)), exist_ok=True)
        open(os.path.join(root, p), "w").close()
    print(sorted(a.split("/", 1)[1] for _, a in walk(root, exclude_tl_dirs=["data"])))

Output on master: ['mypkg/__init__.py']. Expected: ['mypkg/__init__.py', 'mypkg/data/loader.py'].

Root cause

In metaflow/util.py, walk_without_cycles resets skip_dirs = default_skip_dirs after the top level. But the reset changes a variable in the outer loop that the running os.walk iteration no longer reads, so the top-level exclusions keep applying below the top level.

Proposed fix

Reset the skip set inside the walk once the top-level directory has been processed. This is a two-line change plus a regression test in test/unit/test_packaging_utils.py that fails before and passes after. I have it ready and can open a PR if this approach is OK.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions