feat(platform)!: listElement references into a list of a referenced document (PV14)

book/src/data-model/documents.md

@@ -270,7 +270,7 @@ Revision 0 is never used for active documents. This allows `0` to serve as a sen
 
 ## Document References (`refersTo`)
 
-From protocol version 14 a property of a document type can declare what it points at, and consensus refuses a create or replace whose target does not exist when the document is written (the reference is a write-time constraint only; nothing resolves it for a reader). The keyword is `refersTo` on the property, its `type` one of `identity`, `contract` (optionally with `contractRequirements`, see [Contract Moderation](contract-moderation.md)), `token`, `permanentDocument`, `deletableDocument` and `identityPublicKey`. Every form sits on an identifier property, with one exception below. The parsed shape is `DocumentPropertyType::IdentifierWithReference(target)`, and any change to a declaration on contract update is an incompatible schema change.
+From protocol version 14 a property of a document type can declare what it points at, and consensus refuses a create or replace whose target does not exist when the document is written (the reference is a write-time constraint only; nothing resolves it for a reader). The keyword is `refersTo` on the property, its `type` one of `identity`, `contract` (optionally with `contractRequirements`, see [Contract Moderation](contract-moderation.md)), `token`, `permanentDocument`, `deletableDocument`, `identityPublicKey` and `listElement` (see [An element of a list](#an-element-of-a-list-listelement)). Every form sits on an identifier property, with one exception below. The parsed shape is `DocumentPropertyType::IdentifierWithReference(target)`, and any change to a declaration on contract update is an incompatible schema change.
 
 An `identityPublicKey` reference names one key of one identity, and comes in two forms that differ in which property carries what:
 
@@ -335,6 +335,60 @@ When the referring document is created or replaced, the document reference valid
 
 Joins cannot go through a lookup reference: a chained query or a composite by-id join needs the join property's values to be the outer documents' ids, so both refuse such a property, and a `preallocated` index cannot be bound through one. In Rust the declaration is its own variant, `DocumentPropertyReferenceTarget::PermanentDocumentLookup`, appended to the enum rather than a field of `PermanentDocument`: the enum is embedded in the reference errors, so an id reference keeps its encoding, and code matching `PermanentDocument` as "the value is a document id" cannot mistake a lookup for one. The rules are on `DocumentReferenceLookup`. `as_document_reference` returns only references whose value is a document id, the accessor for joins; the validators use `as_any_document_reference`, whose declaration carries the lookup.
 
+### An element of a list (`listElement`)
+
+A `listElement` reference says the value must be one of the identifiers a list of another document holds, the document another property of the same document refers to:
+
+```json
+"resignation": {
+  "type": "object",
+  "properties": {
+    "electedCharterId": {
+      "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
+      "contentMediaType": "application/x.dash.dpp.identifier",
+      "refersTo": { "type": "permanentDocument", "documentType": "electedCharter" },
+      "position": 0
+    },
+    "memberId": {
+      "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
+      "contentMediaType": "application/x.dash.dpp.identifier",
+      "refersTo": {
+        "type": "listElement",
+        "documentType": "electedCharter",
+        "documentProperty": "electedCharterId",
+        "list": "members"
+      },
+      "position": 1
+    }
+  },
+  "required": ["electedCharterId", "memberId"],
+  "additionalProperties": false
+}
+```
+
+reads: `memberId` must be one of the `members` of the `electedCharter` document `electedCharterId` refers to. The moderation charters, whose elected charter holds its `members`, are the first users. The declaration sits on an identifier property or on the `items` of a typed array of identifiers, where every element must be listed (see [References on the Elements](#references-on-the-elements)).
+
+`documentProperty` names an identifier property of the same referring type that carries a `permanentDocument` reference to `documentType`, by id or through a `lookup`; the list's document is the one that reference finds, in whichever contract it names, so a `listElement` takes no `contractId` of its own. `list` names a typed array of identifiers of `documentType`. What is checked when the contract enters the chain:
+
+- `documentProperty` exists, is a stored (not transient) identifier property, not a typed array, and carries a `permanentDocument` reference naming `documentType`: a reader can then tell from the stored document which list the value was checked against. It may be optional. Checked by the parse under full validation (generation 3).
+- The list's document type forbids deletion, and `list` is a stored typed array of identifiers on it that never changes once a document is written: the type is immutable (`documentsMutable: false`), or the list's top-level property is listed under `immutable`. An `immutableAllowSetting` entry can only be set on a document that has no value for it, against which no value was ever accepted, so it does not weaken the rule. For a type of the same contract the contract parse checks this (`create_document_types_from_document_schemas` 1); for a type of another contract, registration checks it against that contract in state and refuses a list that does not qualify with `ReferencedDocumentListInvalidError` (state code 40138). A missing or deletable document type is refused by `documentProperty`'s own reference (40121, 40122).
+- Each value counts against `SystemLimits::max_references_per_document`, one for a property, `maxItems` for a typed array, like every other reference.
+- A changed, added or removed `listElement` is an incompatible schema change on update.
+
+When the referring document is created or replaced, the document reference validation checks the list elements after every other reference. `documentProperty`'s reference fetches the list's document for its own existence check, and the list check reads that document, so it adds no read: it scans at most the list's `maxItems` identifiers. A value the list does not hold, or one set while `documentProperty` is not, refuses the write, paid, with `ReferencedEntityNotFoundError` (40120) naming the property, or the element by its list path (`witnesses[1]`); its target reads "list element (`<list>` of the `<documentType>` document `<documentProperty>` refers to)". A replace re-validates a list element when its value or `documentProperty` changed (a typed array: every element when `documentProperty` changed, otherwise the elements the stored list did not hold), and then validates `documentProperty`'s reference with it, even if that property is unchanged, which is how the document is in hand. Nothing else can make a validated value unlisted: the list's document can never be deleted and its list never changes.
+
+For example:
+
+```text
+charter 7kX...: members [Alice, Bob]
+resignation { electedCharterId: 7kX..., memberId: Alice }  -> accepted
+resignation { electedCharterId: 7kX..., memberId: Carol }  -> refused, 40120:
+  referenced list element (members of the electedCharter document electedCharterId refers to)
+  <Carol> not found for path memberId
+```
+
+In Rust the declaration is the appended variant `DocumentPropertyReferenceTarget::ListElement(ListElementReference)`, so every earlier variant keeps its encoding in the reference errors; the rules are on `ListElementReference` (`referring_side_error`, `referenced_side_error`, `is_listed_in`). It is not a document reference: `as_document_reference` and `as_any_document_reference` return `None` for it, so joins and `preallocated` indexes never go through it.
+
 ## Immutable Properties on Mutable Document Types
 
 A document type either allows replaces (`documentsMutable: true`, the default) or freezes its documents entirely. Protocol version 14 adds a middle ground: the doctype-level `immutable` keyword lists top-level properties that are frozen at creation while the rest of the document stays replaceable.

packages/rs-dpp/schema/meta_schemas/document/v3/document-meta.json

@@ -138,7 +138,8 @@
                 "token",
                 "permanentDocument",
                 "identityPublicKey",
-                "deletableDocument"
+                "deletableDocument",
+                "listElement"
               ]
             },
             "contractId": {
@@ -163,7 +164,7 @@
               ]
             },
             "documentType": {
-              "description": "The name of the referenced document type; for permanentDocument it must forbid deletion (canBeDeleted: false), for deletableDocument it must allow it; a deletableDocument reference is re-validated on every replace of the referring document, so one whose target was deleted has to be repointed at an existing document or cleared",
+              "description": "The name of the referenced document type; for permanentDocument it must forbid deletion (canBeDeleted: false), for deletableDocument it must allow it; a deletableDocument reference is re-validated on every replace of the referring document, so one whose target was deleted has to be repointed at an existing document or cleared. For listElement, the document type holding the list: the one documentProperty refers to",
               "type": "string",
               "minLength": 1,
               "maxLength": 64,
@@ -288,6 +289,20 @@
                 ]
               }
             },
+            "documentProperty": {
+              "description": "listElement references only: the property of the same referring document type (a dotted path when nested) whose permanentDocument reference, by id or through a lookup, finds the document holding the list; it must be a stored (not transient) identifier property whose refersTo names documentType, checked at contract registration, and it may be optional. When the referring document is created or replaced, consensus validates documentProperty's own reference, which fetches that document, and requires the value (on a typed array, every element) to be an element of its list, reading the document already in hand, so the check adds no read; a value that is not, or one set while documentProperty is not, refuses the write with ReferencedEntityNotFoundError (40120). A replace re-validates it when the value or documentProperty changed",
+              "type": "string",
+              "minLength": 1,
+              "maxLength": 256,
+              "pattern": "^[a-zA-Z0-9_]{1,64}(\\.[a-zA-Z0-9_]{1,64})*$"
+            },
+            "list": {
+              "description": "listElement references only: the typed array of identifiers of documentType (a dotted path when nested) the value must be an element of. At contract registration documentType must forbid deletion, and the list must be stored (not transient) and fixed once a document is written: documentType is immutable (documentsMutable: false) or lists the list's top-level property under immutable, so a value accepted as an element stays one",
+              "type": "string",
+              "minLength": 1,
+              "maxLength": 256,
+              "pattern": "^[a-zA-Z0-9_]{1,64}(\\.[a-zA-Z0-9_]{1,64})*$"
+            },
             "lookup": {
               "description": "permanentDocument references only (a key into a deletable type could find a new document once the one it found is deleted, where an id is produced at most once): the property's value is not the referenced document's id. The referenced document is the one the named unique index of the referenced document type finds for a key assembled from the referring document, and the reference holds if that document exists. index names an index of the referenced document type that is unique, carries no timeRange and does not belong to an indexOnly type; its key must stay with the document it found (every schema property of the index fixed by an immutable document type or the immutable list, $ownerId only on a type whose documents can be neither transferred nor traded), so the reference never dangles; keys maps every property of that index (by its name on the referenced side, system ones such as $ownerId included), in any order, to where its value comes from: a property path of the referring document type, \"$ownerId\" for the referring document's owner, or \".\" for the value of the property carrying the reference (on an array element, the element), which must appear exactly once. A property a key reads must be required (and so must every object around it), not transient, and hold the same kind of value as its index property, so a lookup never runs with a missing key part; all of this is validated at contract registration. When the referring document is created or replaced, consensus queries the index for the assembled key, billed as a document fetch, and refuses the write with ReferencedEntityNotFoundError (40120) if it finds no document; propertyAgreement pairs are checked against the document found. A replace re-validates the reference when a property a key reads changed, and on every replace when a key reads $ownerId",
               "type": "object",
@@ -338,7 +353,7 @@
             {
               "if": {
                 "properties": {
-                  "type": { "enum": ["permanentDocument", "deletableDocument"] }
+                  "type": { "enum": ["permanentDocument", "deletableDocument", "listElement"] }
                 },
                 "required": ["type"]
               },
@@ -352,6 +367,25 @@
                 }
               }
             },
+            {
+              "if": {
+                "properties": { "type": { "const": "listElement" } },
+                "required": ["type"]
+              },
+              "then": {
+                "required": ["type", "documentType", "documentProperty", "list"],
+                "properties": {
+                  "contractId": false,
+                  "propertyAgreement": false
+                }
+              },
+              "else": {
+                "properties": {
+                  "documentProperty": false,
+                  "list": false
+                }
+              }
+            },
             {
               "if": {
                 "properties": { "type": { "const": "identityPublicKey" } },

packages/rs-dpp/src/data_contract/document_type/class_methods/create_document_types_from_document_schemas/v1/mod.rs

@@ -208,6 +208,52 @@ impl DocumentType {
             }
         }
 
+        // Protocol version 14 and later: a `refersTo: listElement` whose list lives in a
+        // document type of this contract must find a list there that holds identifiers and
+        // never changes: its documents cannot be deleted, `list` is a stored typed array of
+        // identifiers of it, and the list is fixed once a document is written (see
+        // `ListElementReference::referenced_side_error`). The list's document type is the one
+        // `documentProperty`'s `permanentDocument` reference names, which the document type
+        // parse checked under full validation; one in another contract is checked against
+        // that contract's state at registration, and one this contract does not have is left
+        // to the reference validation of `documentProperty`, which reports it.
+        //
+        // Inert for every protocol version before 14 for the same reason as the checks above:
+        // a parsed reference is a `listElement` only where the tables carry
+        // `apply_property_reference: Some(_)`, so the loop below finds none there.
+        for (name, document_type) in &contract_document_types {
+            for (path, property) in document_type.as_ref().flattened_properties() {
+                // On an identifier property or on the elements of a typed array
+                let Some(reference) = property
+                    .property_type
+                    .reference()
+                    .and_then(|reference| reference.target())
+                    .and_then(|target| target.as_list_element_reference())
+                else {
+                    continue;
+                };
+                if reference.list_contract_id(document_type.as_ref(), data_contract_id)
+                    != Some(data_contract_id)
+                {
+                    continue;
+                }
+                let Some(referenced_document_type) =
+                    contract_document_types.get(&reference.document_type_name)
+                else {
+                    continue;
+                };
+                if let Some(reason) =
+                    reference.referenced_side_error(referenced_document_type.as_ref())
+                {
+                    return Err(consensus_or_protocol_data_contract_error(
+                        DataContractError::InvalidContractStructure(format!(
+                            "document type \"{name}\" property \"{path}\" refersTo listElement: {reason}"
+                        )),
+                    ));
+                }
+            }
+        }
+
         Ok(contract_document_types)
     }
 }