Add a special set of tag to filter out packages based on user preferences

[jorisgio]

Overview

All packages in the repos are not equal : some are really old, some are known to be broken, some might have security vulnerability.
In the current state, this information is not readily available to the user from the package manager. It is possible to add a message: which will warn about security or obsolete packages, but most of the time you have to go down looking at the issue tracker to find what's what.

Previous discussions

This idea has been discussed in various places, and is based on a @paurkedal suggestions

• Deprecating packages #2046
• Big new year cleanup opam-repository#11194
• https://discuss.ocaml.org/t/introducing-a-yanked-mechanism-in-opam/1707/14

Masks

A special tag/mask fields containing a list of flags could be added to the tag namespaces/in another fields. The user could configure the solver has they wish to treat various state

Obsolete flag

Old minor version could be marked obsolete for some time before being removed. Default configuration would warn about installation of obsolete package versions.

Insecure/Broken

Packages versions with known vulnerabilities are marked insecure. Default configuration requires a force flag to install insecure packages, with big red warning.
Packages versions which are known to be broken are marked broken, broken version should not be installed unless the dependencies constraints make them the only possible choice. Should probably warn too.

Insecure/broken tags could contains a link to the issue, but that's sugar

Testing

testing/beta/rc tag mark unstable versions. They should probably be installed by default. User can opt out of them. A good usecase would be to depend on testing version in some test CI/environnement but still use stable on prod

[Drup]

@AltGr While discussing/reading this, I wondered if it could also help #3161 : obsolete/hidden packages would get a very high penalty, or would simply be omited from the query unless explicitly requested.

[hcarty]

I like the idea of omission by default for these penalized tags. A command line argument could be added to enable certain disabled-by-default tagged packages:

opam install mypackage --allow-tags=beta,rc

A matching --disallow-tags could be provided as well to filter out cases like deprecated packages.

[paurkedal]

Excluding masked packages from the solver seems like a good idea, since it allows us to put a cap on how the the solver complexity develops over time by obsoleting packages. Speeding up resolving will add up to save developer time and improve end user experience.

For testing and obsolete, excluding masked packages would hardly make a difference with the following practise:

• We don't lift testing status of a package version before the same mask is removed from all dependencies.
• We don't obsolete a package version before all dependent package versions are obsoleted.

Presumably, broken packages will be discovered quite fast, making it unlikely that a new version of a dependent package is released which requires the broken version. I am assuming here that broken is used for regressions in new package, not for mere bugs discovered in old packages.

On the other hand, it is very likely that we discover a package to be insecure long after other package versions depending on their presence has been released. We must then assume, at least without further inspection, that the dependent packages are also insecure, so it seems reasonable that dependent packages are downgraded if possible, or needs an explicit unmask by the user. However, this is not an issue if security issues are communicated privately and only announced when providing an upgrade path to users. That is, one would send a single PR to the opam repository with fixed minor version(s) and insecure-masking affected versions. Any dependent packages with sensible version constraints (no "=") would still be installable.

So, I think we have some candidate tags with cover different kinds of usage, yet opam can be treat them equally; even the default opam configuration does not need to list any of them, as none are subject to default unmasking. In other words, opam can be agnostic to mask names and leave masking policies to repository maintainers.

[AltGr]

Note that using specific package flags (tags also exist at the moment but are only informative) could be more user-friendly, but a generic feature for filtering packages before solving is already present in opam, as the avaiable: field.

So you could just add e.g.:

available: obsolete-packages

to your package definition. By default, the variable obsolete-packages is undefined, and available: defaults to false in this case. To enable, just use opam config set obsolete-packages true, and the package will become discoverable in the current switch (there is also set-global).

I still think, though, that from a user point of view, multiple repositories are a cleaner interface for this, as they are already present and clearly correspond to the opt-in opt-out scheme. I am afraid that multiple filtering mechanisms will make people wonder why they can see the package in the online repository, but not locally.

Also, when a package is removed from the repository, opam will attempt to remove it at the next opam upgrade, but ignore it on unrelated requests. If, on the other hand, it is made unavailable, the solver will attempt to fix this by removing it, whatever the request. The former seems like a better behaviour to me.

[Drup]

I disagree on the fact that multiple repositories are better for this. Repositories are fairly heavyweight. We want something lighter here. It doesn't solve most of the issues regarding pushing beta versions in opam-respository (which people will always try to do. Creating a custom repo is only for rather big orgs) or flagging versions as obsoletes. Also, users precisely would be able to see them, and opam could give feedback ("This query will require installing an obsolete package, would you like to proceed ? [y/N]").

The fact that all of this can be implemented with available means that it should be easy to do. :p

[AltGr]

Repositories are fairly heavyweight. We want something lighter here. It doesn't solve most of the issues regarding pushing beta versions in opam-respository (which people will always try to do. Creating a custom repo is only for rather big orgs) or flagging versions as obsoletes

Do you mean from the user's point of view, or the packager's / repository maintainer's ? It doesn't seem more heavyweight than the other option for the user, and has the benefit of fitting within existing mechanisms (and is similar to e.g. how debian/ubuntu repositories work).

This query will require installing an obsolete package, would you like to proceed ? [y/N]

Displaying this would require passing the flags along to the solver, with an additional criterion to minimise obsolete packages, putting more stress on it and defeating the original purpose. Using available: would either succeed without any obsolete packages, or fail with the usual message (stg like foo is unavailable because 'obsolete-packages' is false). In which case you could re-add obsolete packages and retry.

[Drup]

I meant for both users and devs. Users will have to look for the repo, add it for the things they are interested in, enable it on the right switches. It's not very flexible either, since it's per-switch, not per-request. For devs, setting up a repo and keeping it up to date is quite a bit of work (that's why both mirage and ocsigen abandoned it).

Displaying this would require passing the flags along to the solver, with an additional criterion to minimise obsolete packages, putting more stress on it and defeating the original purpose

Fair enough! The error message for available sounds sufficient anyway. In this case, we have to set up the convention to be used in the opam-repository. We can't just say "well, you can encode it like that" and close the chapter.

[kit-ty-kate]

was this partially fixed by the introduction of the avoid-version and deprecated flags in opam 2.1.0? This is used by pre-releases is opam-repository, making them picked by the solver only if there is no other choice.