Provide structured data (docstring, arglist, etc.) for completionItem

[lynaghk]

Would y'all be open to a PR that populates the optional data field of the LSP completionItem with a map of more structured information?
E.g., for a function it would include the symbol, fully qualified symbol, docstring, arglist, etc.

This would allow clients to display more structured information.

I started investigating this when I noticed that the Emacs completions were quite bare for lsp-mode Clojure:

compared to the built-in Emacs describe function:

(These screenshots are using the Vertico and Marginalia Emacs packages.)

I would love for the completion to actually search for matches in docstrings too, though that would likely be a more involved change.
(I'd probably build an n-gram index using roaring bitmaps.)

Anyway, just wanted to float the idea before I started working on anything.
I totally understand if this would be out of scope for the project and more appropriate for a fork / standalone thing.

Clojure LSP is really cool and I will continue to use it either way --- thanks for all the great work!

[ericdallo]

@lynaghk the data field is used to trafic information server -> client so when client resolves the item in a next request (completionItem/resolve), send again that field to resolve extra things like docs.
Concerns I see:

• I think we already include docs like hover information in the completionItem/resolve
• if we include more data to data field, this may affect performance, as the data field is returned for all items, if you are completing and receiving 100 items, you are saying you are adding a lot more info for each item, that's why completionItem/resolve exists, to resolve a single item after user hovered over that item

[lynaghk]

Thanks @ericdallo for the feedback and pointers. I took a closer look at completionItem/resolve and it does seem like lsp-mode in emacs is calling that automatically when populating the completion list and it's getting the documentation back formatted as markdown:

[Trace - 01:29:01 PM] Received response 'completionItem/resolve - (878)' in 32ms.
Result: {
  "label": "foo",
  "kind": 3,
  "score": 14,
  "documentation": {
    "kind": "markdown",
    "value": "```clojure\nclj.scratch/foo [x]\n```\n\nMy docstring\n\n----\n\n*[/Users/dev/work/incubator/scratch/src/clj/scratch.clj](file:///Users/dev/work/incubator/scratch/src/clj/scratch.clj)*"
  }
}

Would it be appropriate to add a machine-parsable data field on this response? Then the editor could format additional information in the popup (docstring, arglists, etc.)

[ericdallo]

@kiennq any idea why lsp-mode doesn't call resolveItem automatically like other editors?

[kiennq]

@kiennq any idea why lsp-mode doesn't call resolveItem automatically like other editors?

The resolveItem is called when:

• The document for an item is requested (via hover or auto doc)
• The item is selected, we will resolve it for extra information (commands, extra edits) before inserting

I've tried to preemptively call the resolveItem on item annotation before but unfortunately it flooded the language server with multiple requests (around a page of candidates) and make thing slow down for the language server so @wyuenho reverted it in emacs-lsp/lsp-mode#4625

[wyuenho]

@kiennq any idea why lsp-mode doesn't call resolveItem automatically like other editors?

@ericdallo
What do you mean by calling resolveItem automatically and which editor does this?

[ericdallo]

@wyuenho, vscode and zed that I'm aware of, they resolve the item as soon it's hovered in the completion popup (which makes sense to me as they show more info in the completion for the hovered item)

[wyuenho]

That's the Show More/Less feature from VSCode that requires completion frontend cooperation in Emacs. No completion frontend in Emacs supports that at the moment but emacs-lsp/lsp-mode#4625 laid the groundwork for that. @dgutov will need to take advantage of :company-docsig in the completion menu, and corfu needs to do the same.

For now, you can give https://github.com/wyuenho/emacs-corfu-pixel-perfect a try. The README is very much outdated but the package works quite well, I'm using it daily.

[wyuenho]

As a side node, I think clojure-lsp's response to textDocument/completion is very much lacking. It doesn't support labelDetails, detail is only used for FQN. Completion is also wrong or incomplete, or return too much or different things when backspacing. There's no support for sortText, or filterText, textDocument/documentSymbol is also incredibly slow. If you are looking for opportunities to improve performance and the experience for clojure-lsp users, I think the server leaves a lot to be desired.

If clojure-lsp supports detail or labelDetails in textDocument/completion properly, this issue will be moot.

[ericdallo]

I don't think completion is bad currently. We have already iterated multiple times to improve performance, and there are no complaints about performance at the moment.
I need to check what labelDetails is different than detail or the other fields, but I'm not aware of a bad UX in any editor regarding completion besides this issue