Skip to content

refactor: library scanning - #1495

Open
TrigamDev wants to merge 4 commits into
TagStudioDev:mainfrom
TrigamDev:refactor/library_scan
Open

refactor: library scanning#1495
TrigamDev wants to merge 4 commits into
TagStudioDev:mainfrom
TrigamDev:refactor/library_scan

Conversation

@TrigamDev

@TrigamDev TrigamDev commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Library scanning is reworked to get a list of all entry paths found in the library before scanning, rather than checking for each individual scanned file. This cuts down dramatically on the number of SQL queries created and improves scanning performance.
  • Previously, each scanner would be responsible for then also handling and running logic on the scanned files. Now, each scanner is only responsible for producing a list of discovered paths. The logic run on each file is instead placed in refresh_dir(), which consumes the files produced by which ever scanner is used.
  • The logic behind which scanner is used is moved into scan_for_files(). This way, any function that needs to scan for files can just blindly call scan_for_files() without needing to determine which scanner should be used, and it will receive an iterator of files.
  • Each scanner method has been renamed to be clearer.
    • To help with maintainability, I would recommend not shortening names arbitrarily. Good lord, the source code is filled with so many 2 or 3 character names.
  • Renamed the wcmatch scanner to simply "internal". This better reflects its purpose, and helps if the implementation of the internal scanner ever changes down the line.
  • Reworked the implementation of the ripgrep scanner to better process the discovered files as they're scanned, rather than scanning them all beforehand and only then processing them.
  • Changed how scanning is timed to include the time it takes for the ripgrep scanner to.. scan. Before, it would only track the time it took to process the files, excluding the time it took to scan them.
  • Made the scanner methods (and scan_for_files()) static, as they don't rely on any attributes of RefreshTracker.

Notes

  • This conflicts with perf: Merge library scanning with unlinked entry detection #1242. If perf: Merge library scanning with unlinked entry detection #1242 is merged first, I can rewrite this PR and rebase it on perf: Merge library scanning with unlinked entry detection #1242, since this is a fairly simple PR.
  • ignore_to_glob() produces a lot of redundant, recursive patterns that slow down the internal scanner so much that it's quite literally unusable on my primary library (~65k files, with a large ignore file). I was never able to get it to produce any output on my primary library. It may be worth looking into improving the translation from gitignore-style to glob, or replacing the internal scanner implementation altogether.
  • It could be worth having refresh_dir() emit more detailed progress, rather than just an int.
  • refresh_dir() is set to only emit output every 1/30th of a second.
    • Is there a reason refresh_dir() is the one responsible for enforcing this limit? At least to me personally, it'd make more sense to place the responsibility on the UI for limiting the rate of UI updates. Especially if another, rapidly-updating data source were connected to a progress UI.
    • It may be a good idea to create a general utility method for enforcing an update limit (say, it accepts an iterator, and returns an iterator limited to a certain update limit).
  • It could be worth having ripgrep use null separators, rather than new lines, to prevent issues with certain file names.

Tasks Completed

  • Platforms Tested:
    • Windows x86
    • Windows ARM
    • macOS x86
    • macOS ARM
    • Linux x86
    • Linux ARM
  • Tested For:
    • Basic functionality
    • PyInstaller executable

@TrigamDev TrigamDev added Type: Refactor Code that needs to be restructured or cleaned up TagStudio: Library Relating to the TagStudio library system Type: File System File system interactions Type: Performance An issue or change related to performance TagStudio: Tools/Macros Relating to macros or tool features for TagStudio labels Aug 28, 2026
@CyanVoxel CyanVoxel moved this to 🏓 Ready for Review in TagStudio Development Aug 28, 2026
@CyanVoxel CyanVoxel added this to the Alpha v9.6.4 milestone Aug 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

TagStudio: Library Relating to the TagStudio library system TagStudio: Tools/Macros Relating to macros or tool features for TagStudio Type: File System File system interactions Type: Performance An issue or change related to performance Type: Refactor Code that needs to be restructured or cleaned up

Projects

Status: 🏓 Ready for Review

Development

Successfully merging this pull request may close these issues.

2 participants