fix(rag): Retry ingestion for documents whose materialization failed

[joelbarmettlerUZH]

A document whose ingestion fails is never re-attempted, and there is no way to ask for one. It stays visible in the UI as parsed-but-not-ingested — indistinguishable from a document still in flight — until someone notices and re-uploads the same file by hand.

Materialization is driven by the source file's DataVersion ({updated}-{hash}). A failed run does not change that, so the next observation sees an unchanged file, the automation condition does not re-fire, and the partition keeps its failed state indefinitely. The nightly observe schedule does not help either: it reconciles partition membership, not the materialization of a partition that already exists.

Seen for real: a document failed at chunk_ref_doc_into_nodes during a bug, and after the bug was fixed it stayed stuck. Re-uploading the identical bytes recovered it, purely because that changed the file's modification time.

In scope

• Re-attempt a partition whose latest materialization failed or is missing, rather than requiring the source file to change.
• Bound the retries. A document that fails deterministically — an unparseable file, a model that rejects every chunk — must not re-run every tick forever. Some combination of an attempt ceiling and backoff, in the shape of the run-record-derived attempt counter the knowledge-teardown sensor already uses.
• Distinguish the states the UI shows. "Ingesting", "failed after N attempts" and "queued" are three different things and currently look the same.
• Decide whether a manual "reprocess" affordance is needed once automatic retry exists, or whether it becomes redundant.

Out of scope

• Op-level retry for transient errors, which RetryPolicy already covers on the ops that need it. This is about a run that failed and is never revisited.
• Surfacing pipeline run status per knowledge database, see Per-knowledge-database pipeline status view #1783 (sibling) — that shows what happened, this changes what happens next.

Accepted when

• A document whose ingestion failed is retried without its source file changing.
• A document that fails deterministically stops being retried rather than re-running on every tick.
• A failed document is distinguishable in the UI from one still being processed.
• Fixing a pipeline bug is enough to recover previously-failed documents, with no re-upload.
• make test passes in the affected scopes.