Add support for x-maintenance-intent in dune-project

[art-w]

With progress being made on the opam packages archiving policy, a lot of published packages are going to add an x-maintenance-intent: ["..."] to their opam files. When using dune to generate the opam files, adding this new field requires an extra .opam.template file. Right now the custom opam templates escape hatch are fairly exceptional (~60 published projects uses them), so it would be nice for all the others to stay on the happy path if possible.

In this PR, the opam x-maintenance-intent: [...] is only generated if the user specified (maintenance_intent ...) in their dune-project. Perhaps it should produce ["(latest)"] by default?

See #11272 (cc @hannesm)

[maiste]

Thanks for the contribution! 👍
You need to update the tests to reflect your changes. I would split this in two PRs: the part that updates the code and the part that updates Dune itself (in case it breaks for the former).

Also, could you add an entry into the changes directory?

[art-w]

Thanks for the quick review! I've added tests, an entry in changes and split the opam files update into #11275

[rgrinberg]

Perhaps it should produce ["(latest)"] by default?

That sounds like a useful thing to do.

Is there a need to validate this field at all?

[Leonidas-from-XIV]

Nice feature, however due to how we handle compatibility an upgrade from Dune 3.17 to 3.18 should not trigger user visible changes (that is only allowed via dune-lang), so my suggestion is to gate the generation to dune-lang 3.18+. That will also remove lots of test changes, since these wouldn't change anymore.

[polytypic]

This is perhaps not the right place to comment, because this is more about opam rather than dune, but if you generate latest by default, which also makes sense to me, then why generate it at all? Why couldn't that be the default assumption made by opam when the field is absent?

Personally I'd like to see a much more explicitly elaborated story on how the x-maintenance-intent field is actually supposed to work and help opam as opposed to not having that field.

How are the intent values to be interpreted by users and by opam-repository maintainers? How does it affect archiving? What happens if the maintainers just vanish? What sort of automation is planned based on that field? ...

Like discussed in this PR, I would expect most people to just add latest and essentially forget about this. If it is done implicitly by dune, a lot of people probably won't even be aware of it later on. How will it help opam to have tons of packages with maintenance intent set to latest (as opposed to not having that field (by default))?

Seriously, If I (and most other package authors) just set the field as ”latest” and be done with it, then how does that help?

I'm sceptical that a field like this really helps. Intent is subjective. Actual behaviour is objective. People are optimistic and tend to lie about their own intentions. The expectation seems to be that package authors are good citizen and will carefully maintain the x-maintenance-intent field. I question that assumption. Circumstances change. People move on. I would not expect such a field to be maintained any better than the rest of the package. In fact, I would expect such metadata fields to be maintained significantly less carefully than the actual content of the package. That is just how people work in my experience.

To me this x-maintenance-intent field seems like yet another hurdle to publishing packages on opam, which I already think has been made too slow and laborious compared to other ecosystems. With x-maintenance-intent you are now (exaggerating a bit) basically asked to do the equivalent of swearing on the Bible that you have the best of intentions in order to publish a package.

If the idea is to use x-maintenance-intent to automatically deprecate or archive packages or make choices on which versions to install and so on, then this could make sense (mostly for a small number of actively maintained packages), but in that case the semantics should be very carefully defined so that a package author can set the field correctly.

Or perhaps something like x-maintenance-objectively field which could be set by users and opam-repository maintainers after witnessing how a package has actually been maintained might be more useful? 😆

FYI: Much of the above is duplicated from what I posted on OCaml Discord on the opam-repository channel, before noticing this PR.

[maiste]

Thanks, @polytypic, for reporting this!

This is perhaps not the right place to comment, because this is more about opam rather than dune, but if you generate latest by default, which also makes sense to me, then why generate it at all? Why couldn't that be the default assumption made by opam when the field is absent?

FYI: Much of the above is duplicated from what I posted on OCaml Discord on the opam-repository channel, before noticing this PR.

Unfortunately, considering Dune is only implementing opam policies, this is not the right place to talk about it. My suggestion is to raise the issue on discuss.ocaml.org to make the discussion more public (Discord requires an account to be accessed). I would suggest this thread as it is the last one about the maintenance.

Regarding your concerns, you raised interesting questions, and it is worth discussing them in the right place 😸

[Leonidas-from-XIV]

Nice. I have one small concern about the changes in basic-generate.t where I don't think the changes are required and a small documentation improvement but otherwise this looks very good.

[Leonidas-from-XIV]

Could you add a link to the global field? The global field has the links to the documentation which is nice, so being able to go there directly without having to search for it would be useful.

[art-w]

Is there any way to link to an anchor with rst?

[maiste]

@Leonidas-from-XIV for uniformity, I don't think this is what we want. If you look above, there is no link to global.

[Leonidas-from-XIV]

@art-w No, I don't think so, at least not easily. I think the other file would need to define a named reference at the location where the field is defined and we would need to link to that. Or use a plugin which automatically creates these anchors. If you find an easier way, I'd love to know about it, it does feel like an odd omission in reST/RTD.

@maiste I understand your uniformity argument but I think it should not prevent us from making the documentation easier to navigate. I think it is always better to add a link (and we could do that for other fields that allow overriding as well, or say "all the fields are overrides of the global, link here", though that's not really true as you can't have global depends)

[maiste]

Indeed, my argument makes more sense regarding the code than the documentation. I'm fine with both in the end 😄

[Leonidas-from-XIV]

I've removed the unrelated test changes and rebased. Seems to work fine and the CI failure is unrelated and worked-around by #11293. I think it is fine to merge this before that one is merged.

[art-w]

Thanks @Leonidas-from-XIV !

@rgrinberg > Regarding validation, could this be addressed in a follow-up PR if the lack of validation turns out to be an issue? Otherwise I'm happy to add the validation now.

[rgrinberg]

Regarding validation, could this be addressed in a follow-up PR if the lack of validation turns out to be an issue? Otherwise I'm happy to add the validation now.

I think we should add validation as soon as possible as doing it post the release would mean the validation would have to be guarded behind the new version.

@Leonidas-from-XIV could you revert this if validation isn't ready before we release 3.18

[Leonidas-from-XIV]

@Leonidas-from-XIV could you revert this if validation isn't ready before we release 3.18

Sure, will keep that in mind and make sure not to release 3.18 without 👍 (or revert before).