New commands: import, import --scan, fmt (#465)

# Changes

## New: Added `vcspull import`
- Unified command for importing repositories into vcspull configuration, following the infrastructure-as-code pattern similar to Terraform's `import` command.
- **Manual import**: Register a single repository with `vcspull import <name> <url>`
  - Optional `--dir`/`--path` helpers for base directory detection
  - Example: `uv run vcspull import myrepo [email protected]:user/myrepo.git --dir ~/code`
- **Filesystem scan**: Discover and import existing repositories with `vcspull import --scan <dir>`
  - Recursively scan with `--recursive`/`-r`
  - Interactive confirmation prompt or `--yes` for unattended runs
  - Custom base directory with `--base-dir-key`
  - Example: `uv run vcspull import --scan ~/study/typescript --recursive --yes`

## New: Added `vcspull fmt`
- Introduces a `fmt` subcommand to normalize configuration files (sort directories/repos, convert compact entries to verbose `{repo: ...}` form, and standardize keys).
- Example: `uv run vcspull fmt --config ~/.vcspull.yaml --write`

## Logging Improvements
- Enhanced logging system with better CLI module propagation and StreamHandler configuration.
- Improved output visibility in tests and CLI usage with proper handler management.
- Ensure handlers survive pytest stream rotation via safe flush/reset logic.
- Added regression test `test_import_scan_stream_output` to guarantee user-facing output remains visible.

## Config & CLI Enhancements
- `ConfigReader` now reads files using UTF-8 explicitly.
- Shared helper `save_config_yaml` centralizes YAML writes with UTF-8 encoding.
- CLI parser now wires in the new subcommands and routes to dedicated modules.
- Module naming: `_import.py` (leading underscore) avoids Python's `import` keyword conflict.

Fixes: #25, #333

# Testing
- `uv run ruff check . --fix --show-fixes`
- `uv run mypy`
- `uv run pytest` (132 tests passing)

# Additional Notes
- Extensive new CLI tests cover import/fmt behaviors, including duplicate detection, confirmation prompts, and formatting summaries.
- Logging test suite expanded to validate handler configuration, propagation, and formatter integration.
- Command naming aligns with IaC terminology: `import` clearly communicates bringing repositories under vcspull management, whether manually specified or discovered from the filesystem.

Author: tony

CHANGES

@@ -17,10 +17,29 @@ $ pipx install --suffix=@next 'vcspull' --pip-args '\--pre' --force
 
 ## vcspull v1.36.x (unreleased)
 
-- _Notes on upcoming releases will be added here_
-
 <!-- Maintainers, insert changes / features for the next release here -->
 
+_Notes on upcoming releases will be added here_
+
+### New features
+
+#### New command: `vcspull import` (#465)
+
+- **Manual import**: Register a single repository with `vcspull import <name> <url>`
+  - Optional `--dir`/`--path` helpers for base-directory detection
+- **Filesystem scan**: Discover and import existing repositories with `vcspull import --scan <dir>`
+  - Recursively scan with `--recursive`/`-r`
+  - Interactive confirmation prompt or `--yes` for unattended runs
+  - Custom base directory with `--base-dir-key`
+
+#### New command: `vcspull fmt` (#465)
+
+- Normalize configuration files by expanding compact entries to `{repo: ...}`, sorting directories/repos, and standardizing keys; pair with `--write` to persist the formatted output.
+
+### Improvements
+
+- Enhanced logging system with better CLI module propagation and StreamHandler configuration for improved output visibility in tests and CLI usage (#465).
+
 ### Development
 
 - Add Python 3.14 to test matrix, trove classifiers (#469)

docs/api/cli/fmt.md

@@ -0,0 +1,8 @@
+# vcspull fmt - `vcspull.cli.fmt`
+
+```{eval-rst}
+.. automodule:: vcspull.cli.fmt
+   :members:
+   :show-inheritance:
+   :undoc-members:
+```

docs/api/cli/import.md

@@ -0,0 +1,8 @@
+# vcspull import - `vcspull.cli._import`
+
+```{eval-rst}
+.. automodule:: vcspull.cli._import
+   :members:
+   :show-inheritance:
+   :undoc-members:
+```

docs/api/cli/index.md

@@ -9,6 +9,8 @@
 :maxdepth: 1
 
 sync
+import
+fmt
 ```
 
 ## vcspull CLI - `vcspull.cli`

docs/cli/fmt.md

@@ -0,0 +1,67 @@
+(cli-fmt)=
+
+# vcspull fmt
+
+`vcspull fmt` normalizes configuration files so directory keys and repository
+entries stay consistent. By default the formatter prints the proposed changes to
+stdout. Apply the updates in place with `--write`.
+
+## Command
+
+```{eval-rst}
+.. argparse::
+    :module: vcspull.cli
+    :func: create_parser
+    :prog: vcspull
+    :path: fmt
+    :nodescription:
+```
+
+## What gets formatted
+
+The formatter performs three main tasks:
+
+- Expands string-only entries into verbose dictionaries using the `repo` key.
+- Converts legacy `url` keys to `repo` for consistency with the rest of the
+  tooling.
+- Sorts directory keys and repository names alphabetically to minimize diffs.
+
+For example:
+
+```yaml
+~/code/:
+  libvcs: git+https://github.com/vcspull/libvcs.git
+  vcspull:
+    url: git+https://github.com/vcspull/vcspull.git
+```
+
+becomes:
+
+```yaml
+~/code/:
+  libvcs:
+    repo: git+https://github.com/vcspull/libvcs.git
+  vcspull:
+    repo: git+https://github.com/vcspull/vcspull.git
+```
+
+## Writing changes
+
+Run the formatter in dry-run mode first to preview the adjustments, then add
+`--write` (or `-w`) to persist them back to disk:
+
+```console
+$ vcspull fmt --config ~/.vcspull.yaml
+$ vcspull fmt --config ~/.vcspull.yaml --write
+```
+
+Use `--all` to iterate over the default search locations: the current working
+directory, `~/.vcspull.*`, and the XDG configuration directory. Each formatted
+file is reported individually.
+
+```console
+$ vcspull fmt --all --write
+```
+
+Pair the formatter with [`vcspull import`](cli-import) after scanning the file
+system to keep newly added repositories ordered and normalized.

docs/cli/import.md

@@ -0,0 +1,67 @@
+(cli-import)=
+
+# vcspull import
+
+The `vcspull import` command registers existing repositories with your vcspull
+configuration. You can either provide a single repository name and URL or scan
+directories for Git repositories that already live on disk.
+
+## Command
+
+```{eval-rst}
+.. argparse::
+    :module: vcspull.cli
+    :func: create_parser
+    :prog: vcspull
+    :path: import
+    :nodescription:
+```
+
+## Manual import
+
+Provide a repository name and remote URL to append an entry to your
+configuration. Use `--path` when you already have a working tree on disk so the
+configured base directory matches its location. Override the inferred base
+directory with `--dir` when you need a specific configuration key.
+
+```console
+$ vcspull import my-lib https://github.com/example/my-lib.git --path ~/code/my-lib
+```
+
+With no `-c/--config` flag vcspull looks for the first YAML configuration file
+under `~/.config/vcspull/` or the current working directory. When none exist a
+new `.vcspull.yaml` is created next to where you run the command.
+
+## Filesystem scanning
+
+`vcspull import --scan` discovers Git repositories that already exist on disk
+and writes them to your configuration. The command prompts before adding each
+repository, showing the inferred name, directory key, and origin URL (when
+available).
+
+```console
+$ vcspull import --scan ~/code --recursive
+? Add ~/code/vcspull (dir: ~/code/)? [y/N]: y
+? Add ~/code/libvcs (dir: ~/code/)? [y/N]: y
+```
+
+- `--recursive`/`-r` searches nested directories.
+- `--base-dir-key` forces all discovered repositories to use the same base
+  directory key, overriding the automatically expanded directory.
+- `--yes`/`-y` accepts every suggestion, which is useful for unattended
+  migrations.
+
+When vcspull detects a Git remote named `origin` it records the remote URL in
+the configuration. Repositories without a remote are still added, allowing you
+to fill the `repo` key later.
+
+## Choosing configuration files
+
+Pass `-c/--config` to import into a specific YAML file:
+
+```console
+$ vcspull import --scan ~/company --recursive --config ~/company/.vcspull.yaml
+```
+
+Use `--all` with the [`vcspull fmt`](cli-fmt) command after a large scan to keep
+configuration entries sorted and normalized.

docs/cli/index.md

@@ -7,6 +7,8 @@
 :maxdepth: 1
 
 sync
+import
+fmt
 ```
 
 ```{toctree}
@@ -31,5 +33,5 @@ completion
     :nodescription:
 
     subparser_name : @replace
-        See :ref:`cli-sync`
+        See :ref:`cli-sync`, :ref:`cli-import`, :ref:`cli-fmt`
 ```

docs/quickstart.md

@@ -111,6 +111,13 @@ YAML? Create a `~/.vcspull.yaml` file:
   "flask": "git+https://github.com/mitsuhiko/flask.git"
 ```
 
+Already have repositories cloned locally? Use
+`vcspull import --scan ~/code --recursive` to detect existing Git checkouts and
+append them to your configuration. See {ref}`cli-import` for more details and
+options such as `--base-dir-key` and `--yes` for unattended runs. After editing
+or importing, run `vcspull fmt --write` (documented in {ref}`cli-fmt`) to
+normalize keys and keep your configuration tidy.
+
 The `git+` in front of the repository URL. Mercurial repositories use
 `hg+` and Subversion will use `svn+`. Repo type and address is
 specified in [pip vcs url][pip vcs url] format.

src/vcspull/_internal/config_reader.py

@@ -104,7 +104,7 @@ def _from_file(cls, path: pathlib.Path) -> dict[str, t.Any]:
         {'session_name': 'my session'}
         """
         assert isinstance(path, pathlib.Path)
-        content = path.open().read()
+        content = path.open(encoding="utf-8").read()
 
         if path.suffix in {".yaml", ".yml"}:
             fmt: FormatLiteral = "yaml"

src/vcspull/cli/__init__.py

@@ -4,6 +4,7 @@
 
 import argparse
 import logging
+import pathlib
 import textwrap
 import typing as t
 from typing import overload
@@ -13,6 +14,12 @@
 from vcspull.__about__ import __version__
 from vcspull.log import setup_logger
 
+from ._import import (
+    create_import_subparser,
+    import_from_filesystem,
+    import_repo,
+)
+from .fmt import create_fmt_subparser, format_config_file
 from .sync import create_sync_subparser, sync
 
 log = logging.getLogger(__name__)
@@ -73,14 +80,36 @@ def create_parser(
     )
     create_sync_subparser(sync_parser)
 
+    import_parser = subparsers.add_parser(
+        "import",
+        help="import repository or scan filesystem for repositories",
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+        description="Import a repository to the vcspull configuration file. "
+        "Can import a single repository by name and URL, or scan a directory "
+        "to discover and import multiple repositories.",
+    )
+    create_import_subparser(import_parser)
+
+    fmt_parser = subparsers.add_parser(
+        "fmt",
+        help="format vcspull configuration files",
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+        description="Format vcspull configuration files for consistency. "
+        "Normalizes compact format to verbose format, standardizes on 'repo' key, "
+        "and sorts directories and repositories alphabetically.",
+    )
+    create_fmt_subparser(fmt_parser)
+
     if return_subparsers:
-        return parser, sync_parser
+        # Return all parsers needed by cli() function
+        return parser, (sync_parser, import_parser, fmt_parser)
     return parser
 
 
 def cli(_args: list[str] | None = None) -> None:
     """CLI entry point for vcspull."""
-    parser, sync_parser = create_parser(return_subparsers=True)
+    parser, subparsers = create_parser(return_subparsers=True)
+    sync_parser, _import_parser, _fmt_parser = subparsers
     args = parser.parse_args(_args)
 
     setup_logger(log=log, level=args.log_level.upper())
@@ -91,7 +120,33 @@ def cli(_args: list[str] | None = None) -> None:
     if args.subparser_name == "sync":
         sync(
             repo_patterns=args.repo_patterns,
-            config=args.config,
+            config=pathlib.Path(args.config) if args.config else None,
             exit_on_error=args.exit_on_error,
             parser=sync_parser,
         )
+    elif args.subparser_name == "import":
+        # Unified import command
+        if args.scan_dir:
+            # Filesystem scan mode
+            import_from_filesystem(
+                scan_dir_str=args.scan_dir,
+                config_file_path_str=args.config,
+                recursive=args.recursive,
+                base_dir_key_arg=args.base_dir_key,
+                yes=args.yes,
+            )
+        elif args.name and args.url:
+            # Manual import mode
+            import_repo(
+                name=args.name,
+                url=args.url,
+                config_file_path_str=args.config,
+                path=args.path,
+                base_dir=args.base_dir,
+            )
+        else:
+            # Error: need either name+url or --scan
+            log.error("Either provide NAME and URL, or use --scan DIR")
+            parser.exit(status=2)
+    elif args.subparser_name == "fmt":
+        format_config_file(args.config, args.write, args.all)