Unexpected Illegal surrogate character when parsing field names

Reference: Discussion in #1494

Affects: 2.21.0/2.21.1 at least, possibly 3.0.x (but not 3.1.0+)

Seems like something in broken/missing in field name decoding with JSON escapes.

In the minimum repro unit tests below, acceptJsonEscapedSurrogatePairInFieldName is failing but acceptJsonEscapedSurrogatePairInStringValue passes. I don't know nearly enough about unicode but seems like either both should fail or both should pass (as evidenced by similar tests in UTF8SurrogateValidation363Test). Seems like the field name codepath is doing something different.

    @Test
    void acceptJsonEscapedSurrogatePairInFieldName() throws Exception
    {
        // JSON: {"\ud83d\udc4d":"value"}
        byte[] doc = new byte[] {
            '{', '"',
            '\\', 'u', 'd', '8', '3', 'd',  // JSON escape: \ud83d (high surrogate)
            '\\', 'u', 'd', 'c', '4', 'd',  // JSON escape: \udc4d (low surrogate)
            '"', ':', '"', 'v', 'a', 'l', 'u', 'e', '"',
            '}'
        };

        try (JsonParser p = FACTORY.createParser(doc)) {
            assertToken(JsonToken.START_OBJECT, p.nextToken());
            assertToken(JsonToken.FIELD_NAME, p.nextToken());
            // The escaped surrogate pair should decode to U+1F44D (thumbs up emoji)
            assertEquals("\uD83D\uDC4D", p.currentName());
            assertToken(JsonToken.VALUE_STRING, p.nextToken());
            assertEquals("value", p.getText());
            assertToken(JsonToken.END_OBJECT, p.nextToken());
        }
    }

    /**
     * Test that JSON escape sequence \ud83d\udc4d in string value is accepted.
     *
     * JSON: {"key":"\ud83d\udc4d"}
     */
    @Test
    void acceptJsonEscapedSurrogatePairInStringValue() throws Exception
    {
        // JSON: {"key":"\ud83d\udc4d"}
        byte[] doc = new byte[] {
            '{', '"', 'k', 'e', 'y', '"', ':', '"',
            '\\', 'u', 'd', '8', '3', 'd',  // JSON escape: \ud83d (high surrogate)
            '\\', 'u', 'd', 'c', '4', 'd',  // JSON escape: \udc4d (low surrogate)
            '"',
            '}'
        };

        try (JsonParser p = FACTORY.createParser(doc)) {
            assertToken(JsonToken.START_OBJECT, p.nextToken());
            assertToken(JsonToken.FIELD_NAME, p.nextToken());
            assertEquals("key", p.currentName());
            assertToken(JsonToken.VALUE_STRING, p.nextToken());
            // The escaped surrogate pair should decode to U+1F44D (thumbs up emoji)
            assertEquals("\uD83D\uDC4D", p.getText());
            assertToken(JsonToken.END_OBJECT, p.nextToken());
        }
    }

Here's what Claude is saying about this fwiw:

When parsing field names, the code at lines 2025-2045 re-encodes the decoded escape sequence value (e.g., 0xD83D from \ud83d) as a 3-byte UTF-8 sequence (0xED 0xA0 0xBD) into the quads buffer, which later gets rejected by addName() as an illegal surrogate—whereas string value parsing avoids this entirely by storing the decoded value directly into a char[] buffer where Java natively handles surrogate pairs.

Fix #1541: inconsistent handling of UTF-8 surrogates in names

Fix FasterXML#1541: inconsistent handling of UTF-8 surrogates in names (

@cowtowncoder

@cowtowncoder thanks for the fix! Will this be part of 2.x as well please?

No immediate plans to backport, seems bit too risky.

@cowtowncoder

@cowtowncoder I think I've encountered this issue as a regression when upgrading from 2.20.1 to 2.21.1. This is my test:

import com.fasterxml.jackson.core.*;
import java.io.*;

public class JacksonSurrogateRepro {
    public static void main(String[] args) throws Exception {
        JsonFactory factory = new JsonFactory();

        // U+1F44D (👍) is a supplementary character, stored in Java as surrogate pair \uD83D\uDC4D
        String fieldName = "👍";

        ByteArrayOutputStream out = new ByteArrayOutputStream();
        try (JsonGenerator gen = factory.createGenerator(out, JsonEncoding.UTF8)) {
            gen.writeStartObject();
            gen.writeStringField(fieldName, "value");
            gen.writeEndObject();
        }

        byte[] json = out.toByteArray();
        System.out.println("Generated JSON: " + new String(json, "UTF-8"));
        // Prints: {"\\uD83D\\uDC4D":"value"}

        try (JsonParser parser = factory.createParser(new ByteArrayInputStream(json))) {
            while (parser.nextToken() != null) {
                // no-op, just parse
            }
            System.out.println("Parse succeeded");
        } catch (JsonParseException e) {
            System.out.println("Parse FAILED: " + e.getOriginalMessage());
        }
    }
}

$ javac -cp './jackson-core-2.20.1.jar:.' JacksonSurrogateRepro.java && java -cp './jackson-core-2.20.1.jar:.' JacksonSurrogateRepro
Generated JSON: {"\uD83D\uDC4D":"value"}
Parse succeeded

$ javac -cp './jackson-core-2.21.1.jar:.' JacksonSurrogateRepro.java && java -cp './jackson-core-2.21.1.jar:.' JacksonSurrogateRepro
Generated JSON: {"\uD83D\uDC4D":"value"}
Parse FAILED: Invalid UTF-8: Illegal surrogate character 0xd83d

Does the effect the decision whether to fix this in the 2.x line?

@andrross

@andrross Unfortunately fix itself is rather risky. But I will see if I can create a more limited fix for 2.21 branch since this is a gnarly problem, and we'd want all 2.x users able to upgrade to the latest LTS version.

Partial backport of #1542 to fix #1541 in 2.21 for 2.21.2

Partial backport of #1542 to fix #1541 in 2.21 (for 2.21.2 patch) (#1567

Ok. So, fix is in:

3.1.0 (for 3.x)

and will be in

2.21.2, 2.22.0 (for 2.x)

versions before 2.21.0 not affected.

[SPARK-56156][BUILD] Upgrade jackson to 2.21.2

OWASP dependency check

Validate JSON serialization round-trip to reject invalid surrogate ch…

Validate JSON serialization round-trip to reject invalid surrogate ch… (

Navigation Menu

Unexpected Illegal surrogate character when parsing field names #1541

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Unexpected Illegal surrogate character when parsing field names #1541

Description

Activity

AfzalivE commented on Feb 9, 2026

cowtowncoder commented on Feb 9, 2026

andrross commented on Mar 4, 2026

cowtowncoder commented on Mar 4, 2026

cowtowncoder commented on Mar 4, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions