matrix-org · richvdh · Jan 22, 2025 · May 1, 2023 · Aug 9, 2023 · Aug 9, 2023
@@ -0,0 +1 @@
+Clarify that arbitrary unicode is allowed in user/room IDs and room aliases.
@@ -611,10 +611,18 @@ characters permitted in user ID localparts. There are currently active
 users whose user IDs do not conform to the permitted character set, and
 a number of rooms whose history includes events with a `sender` which
 does not conform. In order to handle these rooms successfully, clients
-and servers MUST accept user IDs with localparts from the expanded
-character set:
+and servers MUST accept user IDs with localparts consisting of any legal
+non-surrogate Unicode code points except for `:` and `NUL` (U+0000), including other control
+characters and the empty string.
 
-    extended_user_id_char = %x21-39 / %x3B-7E  ; all ASCII printing chars except :
+User IDs with localparts containing characters outside the range U+0021 to U+007E, or with
+an empty localpart, are considered non-compliant. For current room versions, servers must
+still accept events using such user IDs over federation; however they SHOULD NOT forward
+such user IDs to clients when referenced outside the context of an event. For example,
+device list updates from non-compliant user IDs would be dropped by the receiving server.
+
+A future room version may prevent users using a historical character set
+from participating. Use of the historical character set is *deprecated*.
 
 ##### Mapping from other character sets
 
@@ -663,6 +671,11 @@ Room IDs are case-sensitive. They are not meant to be
 human-readable. They are intended to be treated as fully opaque strings
 by clients.
 
+The localpart of a room ID (`opaque_id` above) may contain any valid
+non-surrogate Unicode code points, including control characters, except `:` and `NUL`
+(U+0000), but it is recommended to only include ASCII letters and
+digits (`A-Z`, `a-z`, `0-9`) when generating them.
+
 The length of a room ID, including the `!` sigil and the domain, MUST
 NOT exceed 255 bytes.
 
@@ -676,6 +689,9 @@ The `domain` of a room alias is the [server name](#server-name) of the
 homeserver which created the alias. Other servers may contact this
 homeserver to look up the alias.
 
+The localpart of a room alias may contain any valid non-surrogate Unicode codepoints
+except `:` and `NUL`.
+
 The length of a room alias, including the `#` sigil and the domain, MUST
 NOT exceed 255 bytes.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		Clarify that arbitrary unicode is allowed in user/room IDs and room aliases.
Copy link Member kegsay Aug 15, 2023 Choose a reason for hiding this comment The reason will be displayed to describe this comment to others. Learn more. Strongly disagree on this. The original vintage 2014 Synapse implementation only allowed non URL quotable characters in user localparts when registering for an account. Unfortunately, federation did not apply the same check, alluded to by other comments here: Arbitrary unicode ids exist in the wild too, but it has never been possible to create them without modifying the server. If you are playing silly games (removing validation checks on your server) we should NOT set the precedent that we are just going to accept your games imo, otherwise where do you draw the line? Some comments in this proposal mention about excluding the null byte, but why? Sure, postgres cannot represent it when used with TEXT columns, but hey I am using a modified Synapse which doesn't have this problem, so why are we allowing some silly games but not others? The result is an inconsistent mess, and it was never designed to be that way. This isn't like other cases where "hey this is what synapse does, in the spec it goes" because in order to get the failure mode of unicode characters you need a malicious and/or buggy actor. The consequences of making this the rule in the specification, and hence removing these checks in Dendrite, Conduit, et al is an increased risk of homograph attacks. I cannot and will not support increasing the attack surface of Matrix just because a few people back in 2014 removed validation and sent unicode user IDs into a room, which synapse accepted. It is worth emphasising that room versions ARE NOT a get-out-of-jail-free card here, as user IDs are outside the scope of rooms. For example, the sliding sync proxy recently had an issue with unicode user IDs in device list changes. It's not hard to see how this can also be an issue with the user directory and to-device msgs, both of which sit outside of rooms. Counter proposal: sorry folks with smiley poos as user localparts, you're going to be broken in the next release of Synapse, and we remove / subsequently ignore events with malformed user IDs aka what we should have done in 2014. Copy link Contributor neilalexander Aug 15, 2023 Choose a reason for hiding this comment The reason will be displayed to describe this comment to others. Learn more. While I agree that this only really happens because of people modifying their servers, the result of enforcing those checks in one implementation and not in another can end up being outright disastrous for innocent users. For example, if Dendrite enforces these checks but Synapse doesn't, then all it takes is for a Synapse user with an emoji localpart to make a power level change or perform any kind of power action to break the room for Dendrite users probably irreversibly. Hell, they might not even have to perform a power action, because we might drop auth events (such as the user joining to begin with) or refuse to pull in prev events (from normal timeline events), which in turn causes us to run state resolution with a different set of input events (which can result in a different output state set or a complete state reset) or to accidentally propagate broken state to other servers when they ask us for `/state_ids` or `/state`. This situation 100% sucks but Matrix only functions if implementations agree to handle these things in the same way, otherwise different implementations will never arrive at a consistent state. We already see this happen due to other corner cases and it just ends up feeling terrible for all users involved. As for room versions, it is true that it's not a great solution to the problem due to the fact that it still leaks into device updates or other areas, but at least it's possible in a future room version to make sure that users with invalid localparts can't join the rooms to begin with. That is a huge step towards stamping out invalid localparts. turt2live, richvdh, and clokep reacted with thumbs up emoji