Merge pull request #626 from simonchrz/feat/project-inventory

Add a project inventory built from uploaded SBOMs

Author: andrew

README.md

@@ -1343,6 +1343,17 @@ Registered when `hosted.enabled` is set. Pushing and deleting require `Authoriza
 | `GET /api/hosted/{ecosystem}/{name}` | List the archives pushed for a package |
 | `DELETE /api/hosted/{ecosystem}/{name}/{version}` | Delete a pushed version |
 
+### Projects API
+
+Registered when `projects.enabled` is set. Uploading and deleting require `Authorization: Bearer <projects.token>`. See [Project inventory](docs/configuration.md#project-inventory).
+
+| Endpoint | Description |
+|----------|-------------|
+| `PUT /api/projects/{project}/sbom` | Replace a project's recorded package versions with the ones the SBOM in the body lists |
+| `GET /api/projects` | List projects, or with `?purl=pkg:npm/lodash@4.17.20` the projects using a package or version |
+| `GET /api/projects/{project}` | A project's recorded package versions |
+| `DELETE /api/projects/{project}` | Delete a project |
+
 ### Cache API
 
 Registered when `cache_delete` is set. It has no authentication of its own.
@@ -1546,6 +1557,7 @@ The proxy serves a web UI under `/ui`. No separate frontend build is needed -- t
 - **Version detail** (`/ui/package/{ecosystem}/{name}/{version}`) -- per-version metadata, integrity hash, artifact cache status, and hit counts.
 - **Source browser** (`/ui/package/{ecosystem}/{name}/{version}/browse`) -- browse files inside cached archives with syntax highlighting for text files and image previews.
 - **Version diff** (`/ui/package/{ecosystem}/{name}/compare/{v1}...{v2}`) -- side-by-side diff of two cached versions showing added, removed, and changed files.
+- **Projects** (`/ui/projects`) -- with the [project inventory](docs/configuration.md#project-inventory) enabled, the package versions each repository's uploaded SBOM lists, marked when outdated. Package pages then show which projects use each version.
 
 ## Monitoring
 

cmd/proxy/main.go

@@ -8,7 +8,7 @@
 // @securityDefinitions.apikey BearerAuth
 // @in header
 // @name Authorization
-// @description "Bearer " followed by hosted.token
+// @description "Bearer " followed by hosted.token, or projects.token for /api/projects
 //
 // The proxy caches package artifacts from upstream registries (npm, cargo, etc.)
 // providing faster, more reliable access for development teams.
@@ -93,6 +93,9 @@
 //	PROXY_HOSTED_MAX_UPLOAD_SIZE             - Max size of one pushed archive (default "100MB")
 //	PROXY_HOSTED_NPM_SCOPES                  - npm scopes served from pushed packages only
 //	PROXY_HOSTED_COMPOSER_VENDORS            - Composer vendors served from pushed packages only
+//	PROXY_PROJECTS_ENABLED                   - Enable the project inventory (/api/projects, /ui/projects)
+//	PROXY_PROJECTS_TOKEN                     - Bearer token for uploading and deleting project SBOMs
+//	PROXY_PROJECTS_MAX_AGE                   - Remove projects not uploaded for this long (e.g. "90d")
 //	PROXY_URL                                - Proxy URL for proxy hosted
 //
 // Example:

config.example.yaml

@@ -451,6 +451,15 @@ gradle:
 #   npm_scopes: ["@acme"]
 #   composer_vendors: ["acme"]
 
+# Project inventory (disabled by default). Repositories upload an SBOM to
+# PUT /api/projects/{project}/sbom, and the UI shows which project uses
+# which package versions. Project names are visible on the unauthenticated UI.
+# projects:
+#   enabled: true
+#   token: "${PROXY_PROJECTS_TOKEN}"
+#   # Drop projects that have not uploaded for this long ("" keeps them)
+#   max_age: "90d"
+
 # Re-mirror container image tags on a schedule, so a moved tag is picked up
 # even when nobody pulls it. An unchanged tag costs one HEAD request.
 # mirror:

docs/architecture.md

@@ -367,6 +367,10 @@ Selective package mirroring for pre-populating the proxy cache. Supports multipl
 
 The package also provides a `MetadataCache` for storing raw upstream metadata blobs so the proxy can serve metadata responses offline. The `JobStore` manages async mirror jobs exposed via the `/api/mirror` endpoints.
 
+### `internal/inventory`
+
+Turns an SBOM uploaded to `/api/projects` into the package versions recorded for the project. Package URLs are mapped to the ecosystem names and unversioned PURLs the handlers record packages under, so a project's components join with `packages` by `purl`. Also compares versions for the outdated marker on the project pages.
+
 ### `internal/config`
 
 Configuration loading.

docs/configuration.md

@@ -1647,6 +1647,89 @@ Generic files never come from an upstream: anything not pushed answers 404. Whil
 
 Pushed archives are stored like cached ones but marked as local. The cache size limit and storage retention never remove them; only a delete does. Scanning, when enabled, checks pushed archives like fetched ones.
 
+## Project inventory
+
+The project inventory answers "which repository uses which version of a package?". Each repository uploads an SBOM of its default branch, and the proxy records the package versions it lists. The UI then shows:
+
+- `/ui/projects`: every project, and per project its package versions, marked when the proxy knows a newer version.
+- On a package page: which projects use each version, including versions the proxy hasn't cached.
+
+It is disabled by default. Uploads and deletes require a token:
+
+```yaml
+projects:
+  enabled: true
+  token: "${PROXY_PROJECTS_TOKEN}"
+  max_age: "90d"
+```
+
+| Config | Environment | Description |
+|--------|-------------|-------------|
+| `projects.enabled` | `PROXY_PROJECTS_ENABLED` | Register the `/api/projects` endpoints and the UI pages (default `false`) |
+| `projects.token` | `PROXY_PROJECTS_TOKEN` | Bearer token required to upload and delete; required when enabled, supports `${VAR}` |
+| `projects.max_age` | `PROXY_PROJECTS_MAX_AGE` | Remove projects that haven't uploaded for this long, e.g. `90d` or `720h`; empty or `0` keeps them (default) |
+
+Removing old projects runs hourly. With several instances on one database, one of them does it.
+
+### Uploading an SBOM
+
+`PUT /api/projects/{project}/sbom` takes a CycloneDX (JSON or XML) or SPDX (JSON or tag-value) document of up to 32 MiB and replaces everything recorded for the project with what it lists. The optional `ref` and `branch` query parameters are stored and shown with the project.
+
+The project name is a path such as `group/sub/repo`: segments of letters, digits, `.`, `_` and `-`, at most 255 characters. It can be sent as path segments (`/api/projects/group/sub/repo/sbom`) or URL-encoded as one segment (`/api/projects/group%2Fsub%2Frepo/sbom`).
+
+```bash
+syft dir:. -o cyclonedx-json > sbom.json
+curl -X PUT --data-binary @sbom.json \
+  -H "Authorization: Bearer $PROXY_PROJECTS_TOKEN" \
+  "https://proxy.example.com/api/projects/acme/shop/web/sbom?ref=$(git rev-parse HEAD)&branch=main"
+```
+
+The response counts the recorded package versions and the skipped entries:
+
+```json
+{"project":"acme/shop/web","components":9,"skipped":6}
+```
+
+Entries are recorded by their package URL, with the package named the way the proxy caches it: NuGet IDs lowercased, PyPI names normalized as in PEP 503 (`zope.interface` becomes `zope-interface`), the distribution dropped from `deb`, `rpm` and `apk` package URLs. Entries without a package URL (files, the operating system), package types the proxy doesn't serve (such as `pkg:github`), entries without a version, and conda entries without a `channel` qualifier (the proxy records conda packages as `channel/name`) are skipped. So is the project's own package: the root of a lock file such as `package-lock.json`, and with syft, the Go module its binaries were built from. Duplicates are recorded once.
+
+A GitLab CI job on the default branch could look like this:
+
+```yaml
+sbom:
+  image: alpine:3.20
+  rules:
+    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
+  script:
+    - apk add --no-cache curl
+    - curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh | sh -s -- -b /usr/local/bin
+    - syft dir:. -o cyclonedx-json > sbom.json
+    - >
+      curl -sSf -X PUT --data-binary @sbom.json
+      -H "Authorization: Bearer $PROXY_PROJECTS_TOKEN"
+      "https://proxy.example.com/api/projects/$CI_PROJECT_PATH/sbom?ref=$CI_COMMIT_SHA&branch=$CI_COMMIT_BRANCH"
+```
+
+### Querying
+
+```bash
+curl https://proxy.example.com/api/projects                                  # all projects
+curl https://proxy.example.com/api/projects/acme/shop/web                    # one project's package versions
+curl 'https://proxy.example.com/api/projects?purl=pkg:npm/lodash'             # projects using lodash
+curl 'https://proxy.example.com/api/projects?purl=pkg:npm/lodash@4.17.20'     # projects using lodash 4.17.20
+curl -X DELETE -H "Authorization: Bearer $PROXY_PROJECTS_TOKEN" \
+  https://proxy.example.com/api/projects/acme/shop/web
+```
+
+Reading needs no token, like the rest of `/api`. With `?purl=`, each project carries the `version` it uses and no `component_count`. A delete names the project itself: `DELETE /api/projects/acme/shop/web/sbom` is answered with 400 unless a project `acme/shop/web/sbom` exists.
+
+A project's page in the UI also has a delete button. It asks for the token and sends it to `DELETE /ui/api/projects/{project}`, which works like the API endpoint and also requires the token; the token is kept for the browser session. Like `/ui/api/cache`, it sits under `/ui`, so rules for the UI at an ingress cover it.
+
+### What it shows
+
+The inventory reflects what the projects' SBOMs list, not what was downloaded through the proxy: a project that installs from another registry still shows up, and the proxy may not have cached any of its packages. The outdated marker compares the version a project uses with the latest version the proxy has recorded for the package; a package the proxy has no record of has none. Versions are compared as the SBOM gives them: a version spelled differently from how the proxy caches it, such as Composer's `v6.4.1` for `6.4.1` or a Debian version with or without its epoch, shows as "not cached" on the package page.
+
+Project names and the packages they use are shown on `/ui` and `/api/projects` without authentication. If that is sensitive, restrict both at your ingress or reverse proxy.
+
 ## Mirror API
 
 The `/api/mirror` endpoints are disabled by default. Enable them to allow starting mirror jobs via HTTP: