Skip to content

python-api.md

Latest commit

 

History

History
519 lines (451 loc) · 26.6 KB

python-api.md

File metadata and controls

519 lines (451 loc) · 26.6 KB
Raw
Created 2026-08-11
Last-Modified 2026-11-11
SPDX-FileCopyrightText 2026-present Arthit Suriyawongkul
SPDX-FileType DOCUMENTATION
SPDX-License-Identifier CC0-1.0

Python API

Use this when you're calling Pitloom from Python code you control -- a build script, a notebook, or a training/evaluation pipeline that wants to record its own provenance as it runs.

Three entry points:

  • Generator functions -- call generate() (or a target-specific function) to produce a full SBOM, the same output loom project/loom model/loom env produce on the CLI.
  • Standalone enrichment -- call enrich_model() to read a local README/model-card's YAML frontmatter (licence, datasets) into a mergeable fragment: datasets and their trainedOn link, with the licence recorded in its enrichment annotation -- no code annotation needed, the Python equivalent of loom enrich.
  • Tracking decorator -- annotate a training or evaluation script with @loom.run(...) to emit a small SPDX fragment describing what that run produced, to be merged into the SBOM later.

To embed an SBOM in a built wheel from code, see Wheel SBOMs: From Python.

See the API reference for exact call signatures, parameter types, and defaults, generated from the docstrings.

Installation

pip install pitloom
pip install "pitloom[ai]"            # AI model metadata extraction
pip install "pitloom[content-type]"  # content type detection (magika)

Generator functions

Quick guide

from pathlib import Path
from pitloom.assemble import generate

generate(Path("/path/to/project"), output_path=Path("sbom.spdx3.json"))

generate() always returns the SBOM as a JSON string; pass output_path to also write it to disk.

Usage details

from pathlib import Path
from pitloom.core.creation import CreationMetadata, Creator
from pitloom.assemble import generate, generate_project_sbom

# Smart auto-detection entrypoint
generate(
    target=Path("/path/to/project"),
    output_path=Path("sbom.spdx3.json"),
    creation_metadata=CreationMetadata(creators=[Creator(name="Your Name")]),
)

# Or target-specific generator
generate_project_sbom(
    project_target=Path("/path/to/project"),
    output_path=Path("sbom.spdx3.json"),
)

A project in a Git repository, or a release on PyPI, is read through its URL with generate() or a target-specific function (generate_git_sbom(), generate_pypi_sbom()); a URL can name an AI model too. The options, the errors and the safety rules are in Python API: remote targets.

When a supported lock file (pylock.toml, uv.lock, poetry.lock, pdm.lock, Pipfile.lock, or pinned requirements.txt) is present next to pyproject.toml, project generation automatically resolves and includes its exact transitive dependencies -- see Dependency sources and precedence. Pass use_lockfile=False to generate()/generate_project_sbom() to opt out (on by default; same as the CLI's --no-use-lockfile). Has no effect if pre-resolved project_metadata/pitloom_config are BOTH also passed in -- the cascade decision was already made when that metadata was produced. Passing project_metadata without pitloom_config is not supported: it is discarded and re-read from the target, with a WARNING: explaining why. pitloom_config alone is supported -- see below.

Explicit config (pitloom_config=)

Every generator function (generate(), generate_project_sbom(), generate_wheel_sbom(), generate_model_sbom(), generate_env_sbom(), enrich_model()) and embed_wheel_sbom() take pitloom_config=, the library equivalent of --config FILE: an already-built PitloomConfig. It replaces the target's own [tool.pitloom] outright: a field left at its default reverts to the default, not to what the target's config would set. On a target with no [tool.pitloom] of its own (a wheel, an installed environment, a model file, enrich_model() without project_target=) it is the only config that call can get: nothing is read from the current directory. See Where settings come from.

On generate_project_sbom(), pitloom_config= with project_metadata= skips re-reading the project (the caller asserts the two are consistent); pitloom_config= alone re-reads project_target's metadata with the project's own [tool.pitloom] replaced by the value given. The replaced config is not parsed, so an invalid one in the project (or inside an sdist) does not fail the call. An sdist's own config is read as its unpacked directory's is.

Other arguments that mirror a CLI flag or config key:

  • max_source_metadata_bytes= (generate(), generate_project_sbom(), generate_wheel_sbom(), generate_env_sbom(), generate_model_sbom()): --max-source-metadata-bytes; see Metadata provenance.
  • scan_model_usage= (generate(), generate_project_sbom(), generate_wheel_sbom(); ConfigOverrides.scan_model_usage for embed_wheel_sbom(), with or without project_dir=): --scan-model-usage. Other targets warn that it has no effect.
  • trust_model_files=True (generate(), generate_wheel_sbom(), generate_git_sbom(), generate_git_project_sbom(); ConfigOverrides.trust_model_files for embed_wheel_sbom() without project_dir=): --trust-model-files, for a wheel or Git repository you trust only. It has no config key, so a config cannot opt in. Never set it from a request a third party can make (a service that takes repository or wheel URLs): a hostile model file can crash or hang the process. Other targets warn that it has no effect.
  • max_model_extract_bytes= (generate(), generate_wheel_sbom(); ConfigOverrides.max_model_extract_bytes for embed_wheel_sbom() without project_dir=): --max-model-extract-bytes, the per-model ceiling of a wheel's model scan; it wins over pitloom_config='s max-model-extract-bytes. Other targets warn that it has no effect; a value that is not an integer of at least 1 (True included) raises ValueError on every target.
  • max_download_bytes= (generate(), generate_wheel_sbom(), generate_project_sbom(), generate_pypi_sbom(), generate_git_sbom(), generate_git_project_sbom(), generate_model_sbom()): --max-download-bytes, the cap on a file downloaded from a PyPI URL, on the disk a Git URL's checkout takes, and on each metadata file of a Hugging Face model (512 MiB by default); it wins over pitloom_config='s max-download-bytes. Other targets warn that it has no effect; a value that is not an integer of at least 1 (True included) raises ValueError on every target.
  • id_registry=: --id-registry; see Loom ID registry.
  • project_dir= (generate_wheel_sbom(), generate_project_sbom() on an sdist, generate() on either): --project-dir of wheel/project, the project directory the archive was built from, so its files are keyed below that directory's path from the registry's root (see Monorepo). A path that is not a directory raises FileNotFoundError; without a registry, or for any other target, it warns that it has no effect.
  • build_options=BuildOptions(...): --allow-build and its companions, with no config-file equivalent; see --allow-build.

What scan_model_usage records, and which wheel models are listed without metadata and why: Configuration and AI model scan limits. For a wheel, a bad max-model-extract-bytes in pitloom_config= raises ValueError too.

pitloom.assemble also exposes generate_wheel_sbom() (and generate_wheel_sbom_with_metadata(), which returns (json, metadata): the SBOM and the ProjectMetadata read from the wheel), generate_model_sbom(), and generate_env_sbom() -- the same target kinds the CLI's loom wheel / loom model / loom env subcommands cover. See AI model formats for what generate_model_sbom() accepts. None of the three read a [tool.pitloom] of their own -- not the current directory's, not one beside the wheel/model file -- either may belong to an unrelated project; pitloom_config= is the only config any of them can get (see above).

Config

Pass creation_metadata=CreationMetadata(...) to name creators, tools, a timestamp, or a comment on the record -- see Creation metadata for the full field reference. Without it, these functions fall back to [[tool.pitloom.creator]], [[tool.pitloom.creation-tool]] and [tool.pitloom.creation] from whichever config applies to the target -- the project's own [tool.pitloom] for generate_project_sbom() and embed_wheel_sbom(project_dir=...), or an explicit pitloom_config= for every other target (see Explicit config above) -- then the built-in default.

Standalone enrichment

The Python equivalent of loom enrich: parses a local README.md/MODEL_CARD.md's YAML frontmatter only (no prose, no reasoning) and writes a standalone fragment -- fast, free, and always safe to run before anything else.

from pathlib import Path
from pitloom.assemble import enrich_model

enrich_model(
    Path("path/to/model.safetensors"),
    output_path=Path("model.enrich.spdx3.json"),
)

Pass project_target= when merging into a project-level (not single-model) base SBOM -- see the equivalent --project-dir note on the Command line page: --project-dir's document identity is derived from the resolved file list, so it changes whenever that file list changes for the same project. The same applies to use_lockfile=: it must match whatever produced the base SBOM's identity, or the fragment references the wrong document -- omit it (the default, None) to auto-match project_target's own [tool.pitloom] use-lockfile config; pass it explicitly only when the base SBOM's generation used an explicit override that disagreed with that config. Pass id_registry= (a path, or an already-loaded IdRegistry) to reference a pinned entity id (a relative path resolves against project_target when it is a directory, else against the current directory) instead of one freshly computed from the model's own identity. Raises ValueError for a Hugging Face Hub source -- Hugging Face model cards are already parsed natively when generating the SBOM, so local enrichment doesn't apply there.

generate_model_sbom() and generate() also take a folder that holds exactly one model file and no project. A folder with neither raises ValueError: from generate() coded project-or-model-not-found (it raised FileNotFoundError before), from generate_model_sbom() coded model-wrong-format (not an AI model file of a supported format). generate() reads a file as a model by its header too (a GGUF file named x.dat), not only by suffix. generate_model_sbom(..., shown=) names the model file in messages as the caller knows it (a path in a repository) when it has been copied elsewhere; the SBOM names it by its own file name.

A local model is read as a project scan reads it: one whose read fails (truncated, over a bound, a missing extra) does not raise. enrich_model() and generate_model_sbom() log the scan's one WARNING:, and the model is a format-only entry. Both raise ValueError for a file that is not a model (empty, an unknown format, a header that contradicts the suffix), FileNotFoundError for an absent one and OSError for an unreadable one; see AI model scan limits.

The refusal of a file that is not a model carries the code model-wrong-format, and of a file that opens like a Git LFS pointer lfs-pointer or lfs-pointer-malformed (see Message codes): the ValueError carries the text loom model prints and the attribute pitloom_code (pitloom.messages.code_of(exc)). read_ai_model() in pitloom.extract.ai_model.reader raises the same errors. A Hugging Face model read without huggingface_hub raises an ImportError coded library-missing, naming the extra to install (library-unusable when it is installed but fails to import).

Tracking decorator

Annotate scripts or Jupyter notebooks to generate external SBOM fragments that Pitloom merges during the build process, as a function decorator or a context manager. Use set_model when generating a new model, and use_model when consuming one for inference or evaluation.

Quick guide

from pitloom import loom


@loom.run(output_file="fragments/train.json")
def train_model():
    loom.set_model("model-name")
    loom.add_dataset("dataset-name", dataset_type="text")
    # ... training logic ...

Usage details

from pitloom import loom


@loom.run(output_file="fragments/train.json")
def train_model():
    loom.set_model("model-name")  # <-- (A)
    loom.add_dataset("dataset-name", dataset_type="text")  # <-- (B)
    # ... training logic ...


@loom.run(output_file="fragments/eval.json")
def evaluate_model():
    loom.use_model("model-name")  # <-- (C)
    loom.add_dataset("dataset-name", dataset_type="text")  # <-- (B)
    # ... evaluation logic ...
  • (A) and (C) set the relationship between the code and the model.
  • (B) sets the relationship between the code and the dataset.

Pass path= (the model file, relative to the current directory at the call) to set_model() or use_model() to give the model a SHA-256 and a SHA-512 (verifiedUsing). The file is hashed when the run ends without an exception, so write it before the block exits. With a Loom ID registry, the SHA-256 gates the lookup: a training and an evaluation script declaring the same model name and file share one id, and a retrained file gets a new one. The model is keyed by that file's path from the registry's project root, the same key loom model and id generate use. When a project build merges the run's fragment, the model file's ai_AIPackage takes the name given to set_model()/use_model(), with one INFO: Merge: 'name' on ...; its id and registry key do not change. Runs naming one file two ways keep the project's name, with a WARNING: (see SBOM fragments). Without path= (one WARNING: when a registry is loaded), or when it is not a file when the run ends (one WARNING:), the model is matched by name only and gets no hash. The match hits an id held with no hash (a model with no file) and never one recorded with a SHA-256, which no content vouches for: a retrained model's id is not reused by a bare name. In the context-manager form, run.model is None until the block exits. A path= that cannot be read (a NUL in it included) gets the usual unreadable-model WARNING:. Leaving a block, by an exception too, restores the enclosing run, so a recursive @loom.run call is safe.

The run also records which script produced what: the calling script becomes a software_File (with SHA-256 and SHA-512 hashes) with generates relationships to the model it trained and/or the output datasets it wrote. A dataset that is a file gets SHA-256 and SHA-512 hashes and a software_File it contains; a directory gets a SHA-256 Merkle root over its files' paths and content, its comment saying so, and no file elements. Below a dataset directory, the directories id generate skips and symbolic links to directories are left out of the root, as is the declared registry file. A missing or remote dataset name gets no hash. Hashing one dataset reads at most max_dataset_hash_bytes= (else the config's max-dataset-hash-bytes, else 4 GiB); over it, the dataset gets no hash and one WARNING:. A dataset that is a file also gets its size in bytes (dataset_datasetSize): the bytes read, or, over the cap, its stat() size, with the hash's absence and the reason in the provenance. A directory, a Git LFS pointer and a file under 1,024 bytes that is over the cap get no size. The keyword is an integer of at least 1: 0 and a bool raise ValueError as the run begins. A dataset is hashed when it is declared, an output dataset too: call add_output_dataset() after the step has written the file, or it carries the hash of the file it replaces. One name holding one content is one package per run: declaring it again (as input and output, or twice) adds its roles to the same package, with no collision warning. A name that is no path is compared as a model card's datasets: entry is: whitespace runs count as one space and a Hugging Face repository id is case-insensitive (Org/Data is org/data), the package keeping the first spelling, which alone is looked up in the ID registry; an id pinned under a later spelling is not used, with one WARNING:.

A dataset can carry what its own metadata states. Pass croissant= (a local Croissant JSON-LD file) to any of the four dataset calls: its licence becomes the dataset's hasDeclaredLicense, and its version, description, download URL, creator (publishedBy) and dataset-profile fields (collection process, preprocessing, known bias, sensitive personal information) are written onto the package. A dataset that is a directory has its card read too: the YAML frontmatter of its README.md gives license, version and tags (the tags are read but have no SPDX 3.0.1 dataset property to go in). Pass declared_license= to state the licence yourself:

with loom.run("fragments/train.json"):
    loom.add_dataset("data/corpus", croissant="data/corpus.croissant.json")
    loom.add_dataset("data/extra.txt", declared_license="CC-BY-4.0")

Each field comes from the first source that states it: declared_license=, then the Croissant file, then the card. A source's unknown, N/A or NOASSERTION states nothing (a version, creator or licence of that text is left out), and its download URL must be a URL. A declared_license= on any call for the dataset wins over every Croissant file and card, whichever call came first; of two declared licences, the first call's wins. The call's own name, dataset_type and data_preprocessing= (even []) always win. A licence that is an SPDX expression is written as a LicenseExpression, any other text as a SimpleLicensingText reached through a LicenseExpression of one LicenseRef-loom-text-<hash>, which the relationship targets. Your own declared_license="NOASSERTION" (any case) is your statement: written as the NoAssertionLicense individual, and no source replaces it. Any other placeholder, yours or a source's (unknown, blank, a bare other), and a source's NOASSERTION, is no licence, written as nothing and logged as nothing, so the next source may state one. A declared_license= that is not a string raises TypeError. other with a license_name and/or license_link (a Hugging Face card's fields) is written as that name followed by the link, unless the name reads as an SPDX licence and the link does not read as a different one: then the name alone is written. This is the rule for dataset cards; a model card keeps license_name only as extra data. Each field the licence was read from is named in the package's provenance (license: Source: data/corpus/README.md | Field: license). A list of several licences gives each its own hasDeclaredLicense, in order and once each (letter case ignored), by the rules above (a bare other among them gives none), and the package comment says the source does not state whether they combine as AND or OR. A licence that holds no text (a number, a mapping), a version written as a decimal number (YAML and JSON read 1.10 as 1.1), a card version that is not quoted text (YAML 1.1 reads 1:1:1 as 3661 and 010 as 8: quote it, version: "1.0"), and any other value of the wrong shape (a description that is a list of several texts; a creator, URL or tag that is a number) is left out, with one WARNING:. A card that cannot be read at all (frontmatter YAML cannot parse: an impossible date such as 2024-02-30, a number of thousands of digits, nesting thousands of levels deep) is skipped with one WARNING:, and the declaration goes on with its other sources: the card is read because the dataset is a directory, not because you named it. A croissant= file you name is different: a URL, a missing file, one that is not a regular file (a FIFO), one that is not Croissant, one that nests more than 64 levels of arrays and objects or one that fails to be read in any other way raises ValueError naming it, at the call: loom reads no network source. A source file is opened as given and named in provenance by its path from the current directory, or by its bare name when it lies outside it. Hashing and the registry do not change: the sources add fields, not identity.

These generates edges are scoped build -- they describe a build-time step, not something that runs in the shipped artefact. Contrast with the hasDataFile relationship Pitloom emits when it detects a script using a model file at runtime -- that one is scoped runtime. Static detection of such scripts needs scan_model_usage (off by default); a loom.run that declares use_model emits it regardless.

loom.run can also be used as a context manager instead of a decorator, which lets a single run cover more than one independent output batch without their lineage bleeding into each other. Pass input_datasets= on add_output_dataset() to name exactly which add_input_dataset() calls a given output derives from:

with loom.run("fragments/preprocess.json") as run:
    for split in ("train", "valid", "test"):
        sources = [f"rawdata/{split}/{label}.txt" for label in labels]
        for source in sources:
            run.add_input_dataset(source, dataset_type="text")
        run.add_output_dataset(
            f"data/{split}.txt", dataset_type="text", input_datasets=sources
        )

Omit input_datasets (the default) when a run has exactly one output batch -- it then derives from every input the run declared. input_datasets= matches the string an input was declared with, not its resolved path: an input declared as ../data/t.txt is not found by data/t.txt (one WARNING:, that link skipped), even though a registry keys both to one package.

Config

Register the fragment file(s) so a later generate()/loom project/loom generate call merges them into the main SBOM:

[tool.pitloom.fragment]
files = ["fragments/train.json", "fragments/eval.json"]

loom.run/loom.Run calls configure_logging() as it begins, so its WARNING: and INFO: lines carry their tag as on the CLI. Each run resets the pitloom logger's level (INFO, or DEBUG with PITLOOM_DEBUG) and replaces its handlers with one stderr handler; propagation to the root logger is left alone, so an application's own root handler also prints them.

loom.run accepts the same creator/tool/timestamp overrides as the CLI and build hook, via creation_metadata=CreationMetadata(...). With none given, the fragment records the unattended-run default (Pitloom itself as both creator and tool). See Creation metadata.

Pass id_registry= (a path, or an already-loaded IdRegistry) to consult a Loom ID registry when minting ids for datasets, the model, and the generating script -- see Loom ID registry. A dataset package whose path exists below the registry's project root is looked up by that path, as the same package from any directory or spelling (../data/t.txt from a subdirectory is data/t.txt), and by its content digest when the entry records one; a remote, missing or outside-the-root dataset by its name. A file (a dataset's, or the script) is looked up by its path from the registry's project root and its hash, so it keeps its id from any directory (a path outside the root gets a new id and a WARNING:). A file reached through a symlinked directory is keyed by the path as written, which id generate does not index, so it finds no entry. See keys and content gates. The one entry loom writes is a model declared with path= (hashed when the run ends) whose file the registry does not know, saved after the fragment; update-id-registry = false in pitloom_config stops it. As on every other surface, loom.run/loom.Run uses a registry only when one is declared (id_registry=, or id-registry in pitloom_config= below): with none, no registry is used -- nothing is searched for or auto-discovered. A declared registry that's missing, unreadable or invalid (a format version 3 or 2 file included) raises ValueError when the with loom.run(...) block (or the decorated call) begins, before any fragment work happens.

loom.run reads no [tool.pitloom] of its own. Pass pitloom_config= (a PitloomConfig, or the path of a TOML file holding [tool.pitloom], relative to the current directory) to set id-registry, update-id-registry or max-dataset-hash-bytes; no other key has an effect there. id_registry= wins over the config's id-registry, and max_dataset_hash_bytes= over its max-dataset-hash-bytes. A config file that is missing, unreadable or invalid raises ValueError as the run begins.

The merge itself (pitloom.assemble.merge_fragments, run by generate()/generate_project_sbom() whenever [tool.pitloom.fragment] lists files) raises pitloom.assemble.FragmentMergeError if an element of the merged graph references an id that resolves to nothing -- most commonly a fragment recorded against a base SBOM whose ids have since changed: regenerate the base SBOM and re-run the fragment-producing script. generate_merged_sbom(fragments_dir) merges a directory of fragments into one document, as loom merge does. See SBOM fragments and API reference.

See also