[RFC 0197] package sets by-name #197
+537
−0
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,292 @@ | ||
| --- | ||
| feature: nixpkgs package set location | ||
| start-date: 2026-01-24 | ||
| author: quantenzitrone | ||
| co-authors: (find a buddy later to help out with the RFC) | ||
| shepherd-team: (names, to be nominated and accepted by RFC steering committee) | ||
| shepherd-leader: (name to be appointed by RFC steering committee) | ||
| related-issues: (will contain links to implementation PRs) | ||
| --- | ||
|
|
||
| # Summary | ||
| [summary]: #summary | ||
|
|
||
| Different ideas on how to handle package sets in nixpkgs. | ||
|
||
|
|
||
| # Motivation | ||
| [motivation]: #motivation | ||
|
|
||
| - get rid of the package categories as directories (decided in RFC 140 and 146) | ||
| - bring the benefits of by-name to package sets | ||
| - merge bot maintainer merging | ||
| - scaleability | ||
| - isolation | ||
| - vetting | ||
| - *your goal here* | ||
quantenzitrone marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| <details> | ||
| <summary> | ||
|
|
||
| # Idea 1: `pkgs/sets-by-name` | ||
|
|
||
| </summary> | ||
|
|
||
| Proof-Of-Concept implementation in https://github.com/NixOS/nixpkgs/pull/482538 | ||
|
|
||
| ## Detailed design | ||
|
|
||
| - Create a new directory under `pkgs/`, e.g. `pkgs/sets-by-name` that contains package sets. | ||
| - For example `pkgs/sets-by-name/fishPlugins`, `pkgs/sets-by-name/python3Packages`. | ||
| - Each package set is sharded like `pkgs/by-name`. | ||
| - The following additional have to exist for each package set. | ||
| - `functions.nix`: definitons of functions, like `buildFishPlugin`, `buildPythonPackage`. | ||
| - `overrides.nix`: overrides of packages, like `top-level/all-packages.nix` currently functions as | ||
| an overlay for `by-name` packages. | ||
| - This is something we try to keep empty. Most, maybe all, overrides can be inlined in the | ||
| package. | ||
| - `aliases.nix`: aliases for aliases in package sets behind `optionalAttrs config.allowAliases` | ||
| (like `top-level/aliases.nix`). | ||
| - All package sets with their sharded packages, overlayed with their functions, overrides and | ||
| aliases are automatically called by `top-level/package-sets-by-name.nix`. | ||
| - Versioned package sets like `python316Packages` are done in `all-packages.nix` by overriding the | ||
| default version package set. | ||
| - Versioned package sets without a default version will have to override the default version with | ||
| an error. | ||
| - e.g. `nextcloud*Packages` are in `sets-by-name/nextcloundPackages` and thus autocalled by | ||
quantenzitrone marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| `top-level/package-sets-by-name.nix`, however we will have an alias in `top-level/aliases.nix` | ||
| that says | ||
| ```nix | ||
| { | ||
| nextcloudPackages = throw "Please use nextcloudPackages for a specific nextcloud ersion e.g. nextcloud32Packages."; | ||
| } | ||
| ``` | ||
|
|
||
| ``` | ||
| pkgs | ||
| ├── by-name | ||
| │ └── ... | ||
| ├── sets-by-name | ||
| │ ├── fishPlugins | ||
| │ │ ├── as | ||
| │ │ │ ├── async-prompt | ||
| │ │ │ ... └── package.nix | ||
| │ │ ├── au | ||
| │ │ ... | ||
| │ │ ├── z_ | ||
| │ │ │ └── z | ||
| │ │ │ └── package.nix | ||
| │ │ ├── aliases.nix | ||
| │ │ ├── functions.nix | ||
| │ │ └── overrides.nix | ||
| │ │ | ||
| │ ├── python3Packages | ||
| │ │ ├── a2 | ||
| │ │ │ └── a2wsgi | ||
| │ │ │ └── package.nix | ||
| │ │ ├── aa | ||
| │ │ ... | ||
| │ │ ├── zx | ||
| │ │ │ ├── zxcvbn | ||
| │ │ │ │ └── package.nix | ||
| │ │ │ ├── zxcvbn-rs-py | ||
| │ │ │ │ └── package.nix | ||
| │ │ │ └── zxing-cpp | ||
| │ │ │ └── package.nix | ||
| │ │ ├── aliases.nix | ||
quantenzitrone marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| │ │ ├── functions.nix | ||
| │ │ └── overrides.nix | ||
| │ ... | ||
| └── top-level | ||
| ├── all-packages.nix <- calls all versioned package sets | ||
| ├── by-name-overlay.nix <- used to autocall sharded packages (no change required) | ||
| ... | ||
| └── package-sets-by-name.nix <- autocalls all sets by name | ||
| ``` | ||
|
|
||
| ## Advantages | ||
|
|
||
| - accessibility through github ui navigation (better than idea 2 and 3) | ||
| - clear package separation | ||
| - reuse of RFC 140 concepts | ||
| - making maintainer merging and nixpkgs-vet work for this is probably relatively simple | ||
| - implement tooling to work for multiple directories https://github.com/NixOS/nixpkgs-vet/pull/180 | ||
| - enable tooling on all subdirectories of `pkgs/sets-by-name` | ||
|
|
||
quantenzitrone marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| ## Drawbacks | ||
|
|
||
| - only allows top level package sets | ||
| - *your drawback here* | ||
|
|
||
| ## Unresolved Questions | ||
|
|
||
| - *your question here* | ||
quantenzitrone marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| </details> | ||
|
|
||
| <details> | ||
| <summary> | ||
|
|
||
| # Idea 2: nested by-name structure | ||
|
|
||
| </summary> | ||
|
|
||
| Proof-Of-Concept implementation in https://github.com/NixOS/nixpkgs/pull/483432 | ||
|
|
||
| ## Detailed design | ||
|
|
||
| - Package sets are done in a nested `by-name` structure under `pkgs/by-name`, e.g. | ||
| `fishPlugins.puffer` would be in `by-name/fi/fishPlugins/pu/puffer`. | ||
| - A marker is required in order to distinguish package sets from simple packages, such as using a `.packageset` file (example: `by-name/fi/fishPlugins/.packageset`) | ||
| if not the `package.nix` must exist and is called as a package. | ||
quantenzitrone marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| ``` | ||
| pkgs | ||
| └── by-name | ||
| ├── 0_ | ||
| ... | ||
| ├── fi | ||
| │ ├── fiano | ||
| │ ├── fiche | ||
| │ ├── ... | ||
| │ ├── fishnet | ||
| │ ├── fishPlugins | ||
quantenzitrone marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| │ │ ├── .packageset | ||
| │ │ ├── as | ||
| │ │ │ └── async-prompt | ||
| │ │ ├── au | ||
| │ │ ... | ||
| │ │ └── z_ | ||
| │ │ └── z | ||
| │ ├── fishy | ||
| │ ... | ||
| ... | ||
| ``` | ||
|
|
||
| ## Advantages | ||
|
|
||
| - clear package separation | ||
| - reuse of RFC 140 concepts | ||
| - making maintainer merging work for this is probably relatively simple | ||
| - allows nested package sets | ||
|
|
||
| ## Drawbacks | ||
|
|
||
| - unresolved questions | ||
| - `lixPackages` (behind all `lib*` packages) will not be accessible through GitHubs UI | ||
| - having package sets in `pkgs/by-name` may not fit the spirit of rfc 140 | ||
| - *your drawback here* | ||
|
|
||
| ## Unresolved Questions | ||
|
|
||
| - How do we handle functions like `fishPlugins.buildFishPlugin`? | ||
| - How do we handle aliases? | ||
quantenzitrone marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| - How do we handle versioned package sets? | ||
| - How do we move large package sets? | ||
|
|
||
| </details> | ||
|
|
||
| # Detailed design | ||
| [design]: #detailed-design | ||
|
|
||
| TODO: decide for one of the above ideas | ||
|
|
||
| # Examples and Interactions | ||
| [examples-and-interactions]: #examples-and-interactions | ||
|
|
||
| TODO | ||
|
|
||
| # Drawbacks | ||
| [drawbacks]: #drawbacks | ||
|
|
||
| TODO | ||
|
|
||
| # Alternatives | ||
| [alternatives]: #alternatives | ||
|
|
||
| TODO: move other designs here once decided | ||
|
|
||
| <details> | ||
| <summary> | ||
|
|
||
| # Idea 3: package sets in `pkgs/by-name` (scrapped) | ||
|
|
||
| </summary> | ||
|
|
||
| Proof-Of-Concept implementation in https://github.com/NixOS/nixpkgs/pull/483128 | ||
|
|
||
| ## Detailed design | ||
|
|
||
| - Instead of `by-name/<shard>/<pname>` we have `by-name/<shard>/<attrpath>`, so e.g. | ||
| `fishPlugins.puffer` would go in `by-name/fi/fishPlugins.puffer`. | ||
| - The `top-level/by-name-overlay.nix` will call all folders in a `<shard>` that contain a dot as a | ||
| package set. | ||
|
|
||
| ``` | ||
| pkgs | ||
| └── by-name | ||
| ├── 0_ | ||
| ... | ||
| ├── fi | ||
| │ ├── fiano | ||
| │ ├── fiche | ||
| │ ├── ... | ||
| │ ├── fishnet | ||
| │ ├── fishPlugins.async-prompt | ||
| │ ├── fishPlugins.autopair | ||
| │ ├── fishPlugins.z | ||
| │ ├── fishy | ||
| │ ... | ||
| ... | ||
| ``` | ||
|
|
||
| ## Advantages | ||
|
|
||
| - clear package separation | ||
| - reuse of RFC 140 concepts | ||
| - making maintainer merging work for this is probably easy | ||
| - allows nested package sets | ||
|
|
||
| ## Drawbacks | ||
|
|
||
| - huge shards due to package sets | ||
| - currently only few shards like `li` are too large for GitHubs UI, but with this idea more shards | ||
| will be huge as well | ||
| - specifically 12 more shards `em`, `gn`, `ha`, `ho`, `oc`, `pe`, `py`, `rp`, `sb`, | ||
| `te`, `ty`, `vi` and `vi` (for `emacsPackages`, `gnomeExtensions`, `haskellPackages`, | ||
| `home-assistant-component-tests`, `ocamlPackages`, `pearlPackages`, `python3Packages`, | ||
| `rPackages`, `sbclPackages`, `texlivePackages`, `typstPackages` and `vimPlugins`) could become | ||
| inaccessible. | ||
| - some package sets like `lixPackages` (behind all `lib*` packages) will not be accessible through | ||
| GitHub UI | ||
| - having package sets in `pkgs/by-name` may not fit the spirit of rfc 140 | ||
| - it's called pkgs/by-**name** and not pkgs/by-**attrpath** | ||
| - directory names as attrpaths is sketchy | ||
| - unresolved questions | ||
|
|
||
| ## Unresolved Questions | ||
|
|
||
| - How do we handle functions like `fishPlugins.buildFishPlugin`? | ||
|
There was a problem hiding this comment. (Answering because this applies to idea 1) I never thought it made sense for functions and derivation builders to be inside package sets. I think our Go infrastructure has the right idea: move them to the top-level, and version them in the attribute name.
Member
There was a problem hiding this comment. This proposal is awkward with Python where you have CPython and PyPy and then language version-dialects within them, and even worse for Common Lisp where you have same base language but multiple compatible implementations with their completely own versioning.
Author
There was a problem hiding this comment. i also think keeping functions in package sets is fine |
||
| - How do we handle aliases? | ||
quantenzitrone marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - How do we handle versioned package sets? | ||
|
There was a problem hiding this comment. (Answering because this applies to idea 1) I generally think it's a good idea to think of Nixpkgs as having three types of package infrastructure systems:
Author
There was a problem hiding this comment.
Author
There was a problem hiding this comment.
|
||
| - How do we move large package sets? | ||
quantenzitrone marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| </details> | ||
|
|
||
| # Prior art | ||
| [prior-art]: #prior-art | ||
|
|
||
| - `by-name` stucture for `python3Packages` https://github.com/NixOS/nixpkgs/pull/449896 https://github.com/NixOS/nixpkgs-vet/pull/180 | ||
| - https://github.com/NixOS/nixpkgs/issues/482537 | ||
| - https://github.com/NixOS/nixpkgs/issues/432625 | ||
| - `tclPackages` has their own `by-name` structure https://github.com/NixOS/nixpkgs/pull/344716 | ||
| - attempt to move `nushellPlugins` to `by-name` https://github.com/NixOS/nixpkgs/pull/482961 | ||
|
|
||
| # Unresolved questions | ||
| [unresolved]: #unresolved-questions | ||
|
|
||
| What parts of the design are still TBD or unknowns? | ||
|
|
||
| # Future work | ||
| [future]: #future-work | ||
|
|
||
| What future work, if any, would be implied or impacted by this feature without being directly part of the work? | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we don't incorporate a system for nested package sets/package sets nested in attribute sets, how do we deal with them?
emacsPackages.*Packagesocaml-ng.ocamlPackages*haskell.packages.{,native-bignum.}*chickenPackages.chickenEggsvscode-extensionscould probably be flattenedocamlPackages*.janeStreetcould maybe be merged intoocamlPackages*kdePackages.{sources,gear,framework,plasma}?vimPlugins.nvim-treesitter-{,legacy-}parsers?linuxKernels.**?luaPackages.luaLib?androidenv.**?nginxModules.*?