Skip to content

FD leak in compactor if any index file fails to load. #20685

@gunjambi

Description

@gunjambi

Describe the bug

In case a compaction fails due to a partially failed index fetching or loading, the successfully opened Index objects are not closed. The index *TSDBFile opens mmapped view to the file, which increases the reference count to the file description. This reference is never released.

This causes a file description leak. This also leaks disk space; even if the file is unlinked, the leaked reference keeps the backing disk storage alive.

The issue stems from a simple oversight in CompactTable() when handling error result of concurrency.ForEachJob(). We return before the cleanup defer handler is installed and the resources of the completed jobs are not closed. See PR #19408

To Reproduce
Steps to reproduce the behavior:

  1. Perform anything that causes compaction fail, such as: Backend deletes tsdb files before compacting them #12105 (comment)
  2. Run lsof to inspect open files, or observe cat /proc/sys/fs/file-nr
  3. Observe the repeated compaction failures increase the number of files

Expected behavior
Files are not leaked

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions