Skip to content

Extract tar members through the tarfile data filter - #5337

Open
inchang-ing wants to merge 2 commits into
pypa:mainfrom
inchang-ing:tarfile-data-filter
Open

inchang-ing wants to merge 2 commits into
pypa:mainfrom
inchang-ing:tarfile-data-filter

Conversation

@inchang-ing

Copy link
Copy Markdown

Fixes #5328

Problem

setuptools.archive_util.unpack_tarfile extracted through three private tarfile APIs, so none of the PEP 706 hardening ever ran:

  • _extract_member(member, final_dst) bypassed the extraction filter entirely, letting archive-supplied setuid/setgid/sticky bits and ownership reach the filesystem;
  • tar_obj.chown = lambda *args: None was a hand-rolled substitute for one piece of what the filter does properly;
  • tar_obj._getmember(linkpath) resolved link targets through an API with no stability guarantee.

The containment check from #5325 re-implemented one filter protection by hand; this change lets the stdlib own the rest.

Approach (option 1 from the issue)

  • Link resolution now scans the public getmembers() listing with the same semantics as the old private call: exact name match, last occurrence wins, None when absent. Notably the exact match is preserved deliberately — getmember() would not be equivalent, since it rstrips trailing slashes from the query (tarfile already strips them from parsed member names, so exact matching is what reproduces today's behavior).
  • Filtering: after link resolution, the member that is actually about to be written is passed through tarfile.data_filter(member, extract_dir), and its sanitized copy is extracted to the possibly-redirected destination from progress_filter. This keeps the documented progress_filter semantics (including redirection) while applying the stdlib's validation and sanitization (high mode bits stripped, ownership cleared, special files rejected).
  • The chown suppression is retained only for Pythons lacking the filter (early 3.10/3.11 micro releases without the security backport), where behavior is unchanged from before. requires-python >= 3.10 is satisfied for the filter itself by the 3.10.12+/3.11.4+ backports; the getattr guard covers anything older than those micros.

Testing

  • New test_iter_open_tar_applies_data_filter (cross-platform, at the TarInfo level): a setuid member comes out of _iter_open_tar with high mode bits stripped and ownership cleared.
  • New test_unpack_tarfile_strips_high_mode_bits (POSIX): end-to-end, the extracted file carries no setuid bit and stays executable.
  • New test_unpack_tarfile_resolves_links_to_targets: hardlink members are materialized as copies of their archive-relative targets through the public lookup (link resolution previously had no coverage).
  • Full test_archive_util.py (17 tests) and test_dist_info.py (which unpacks a real wheel through unpack_archive) pass: 53 passed, 2 skipped, 1 xpassed; the skip/xpass are pre-existing (symlink support on Windows, UnicodeEncodeError in archive_util #710/sandbox.run_setup incorrectly sets __file__ when setup_script is Unicode on Python 2 #712).

Disclosure

This PR was prepared with AI assistance (ZCode/GLM, orchestrated via WorkBuddy).

unpack_tarfile extracted through three private tarfile APIs
(_extract_member, _getmember and the chown override), so none of the
PEP 706 hardening ran: archive-supplied setuid/setgid/sticky bits and
ownership reached the filesystem, and link resolution depended on an
API with no stability guarantee.

Resolve links through the public getmembers() listing with the same
exact-name, last-occurrence-wins semantics, and run the resolved
member through tarfile.data_filter before it is written, picking up
the stdlib's sanitized copy (high mode bits stripped, ownership
cleared, special files rejected). The chown suppression is retained
only for Pythons without the filter backport.

Fixes pypa#5328
@mergify

mergify Bot commented Sep 28, 2026

Copy link
Copy Markdown
Contributor

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

  • Queue this pull request

Comment thread setuptools/archive_util.py Outdated
# and take its sanitized copy (high mode bits stripped,
# ownership cleared, special files rejected) before the
# member is written anywhere.
member = _DATA_FILTER(member, extract_dir)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With a hard link ok to a later member ../evil, this line raises tarfile.OutsideDestinationError where main raises UnsafeMember. Callers catching UnsafeMember or DistutilsError miss this abort.

The PEP 706 data filter raises tarfile.OutsideDestinationError for a
member whose (link-resolved) name escapes the destination, whereas the
pre-filter code raised UnsafeMember through _resolve_dest. Callers
catching UnsafeMember or DistutilsError would miss the abort entirely,
so the extraction driver's exception contract silently changed.

Translate the filter's outside-destination and absolute-path errors into
UnsafeMember, keeping the contract identical to the pre-filter behavior.
A parameterized regression test pins both hard links and symlinks.

Reported-by: jamalkamaladdin
@inchang-ing

Copy link
Copy Markdown
Author

Good catch - fixed in 4575b37.

Reproduced it: for a hard link ok pointing at a later ../evil member,
_resolve_tar_file_or_dir replaces the member with the escaping target, and
_resolve_dest has only ever been applied to the link name (ok), which is
inside the destination. The data filter was therefore the only thing left that
could reject the resolved target - and it raised
tarfile.OutsideDestinationError where main raised UnsafeMember via
_resolve_dest. So a caller catching UnsafeMember or DistutilsError would
have seen the abort silently disappear.

_iter_open_tar now translates the filter's outside-destination and
absolute-path errors back into UnsafeMember, so the exception contract
matches main again. I compared main and the branch over five archives -
hard link to an escaping member, symlink to an escaping member, link to an
absolute path, plain escaping member, and a fifo - and the two now agree in
every case (the fifo is still skipped through LookupError, not rejected).

Covered by test_iter_open_tar_reports_escaping_link_as_unsafe, parametrized
over LNKTYPE and SYMTYPE.

@inchang-ing

Copy link
Copy Markdown
Author

Hi maintainers, gentle ping on #5337 — Extract tar members through the tarfile data filter. It's a small, self-contained fix with passing tests. Would appreciate a review when you have a moment. Thanks for maintaining pypa/setuptools!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

archive_util bypasses tarfile's PEP 706 extraction filters by calling private APIs

2 participants