Default polymorphic type handler for deserialization not working for Protobuf

[ShreckYe]

Describe the bug

Default polymorphic type handler for deserialization is not working for Protobuf. Originally found while working on #2587 here.

To Reproduce
Attach a code snippet or test data if possible.

Code (adpated from example-poly-19.kt):

@Serializable
abstract class Project {
    abstract val name: String
}

@Serializable
data class NamedProject(@ProtoNumber(1) override val name: String) : Project()

@Serializable
data class BasicProject(@ProtoNumber(1) val type: String, @ProtoNumber(1) override val name: String) : Project()

val module = SerializersModule {
    polymorphic(Project::class) {
        subclass(NamedProject::class)
        defaultDeserializer { BasicProject.serializer() }
    }
}
val json = Json { serializersModule = module }
val protoBuf = ProtoBuf { serializersModule = module }

@OptIn(ExperimentalStdlibApi::class)
fun main() {
    fun decodeProjectTest(serialName: String) {
        println(json.decodeFromString<Project>("""{"type":"$serialName","name":"test"}"""))

        val serializedNameMessageString = "12" + "06" + "0a" + "04" + "test".encodeToByteArray().toHexString()
        println(
            protoBuf.decodeFromHexString<Project>(
                "0a" +
                    serialName.length.toByte().toHexString() +
                    serialName.encodeToByteArray().toHexString() +
                    serializedNameMessageString
            )
        )
    }
    decodeProjectTest(NamedProject.serializer().descriptor.serialName)
    decodeProjectTest("NotRegisteredProject")
}

Output:

NamedProject(name=test)
NamedProject(name=test)
BasicProject(type=NotRegisteredProject, name=test)
Exception in thread "main" kotlinx.serialization.MissingFieldException: Field 'type' is required for type with serial name 'example.examplePoly19.BasicProject', but it was missing

Expected behavior

Environment

Commit 1f7372a of this project.

• Kotlin version: [e.g. 1.9.22]
• Library version: [e.g. 1.6.4-SNAPSHOT]
• Kotlin platforms: [e.g. JVM]
• Gradle version: [e.g. 7.6.1]
• IDE version (if bug is related to the IDE) [e.g. IntelliJ IDEA 2023.3.4 (Community Edition)]
• Other relevant context [OS version Ubuntu 22.04.4 LTS, JRE version 17]

[pdvrieze]

There is an error in your class: BasicProject(@ProtoNumber(1) val type: String, @ProtoNumber(1) override val name: String) You reuse the ProtoNumber. Also the type member for polymorphic serialization is not part of the actual serialized type.

The polymorphic serializer works by injecting a synthetics structure with 2 members: type and value. Formats can then hide this by having a special type member/attribute (as such the json deserialization works by accident - it can't distinguish between the generated (and inlined) type attribute and the explicit type attribute that is part of BasicProject).

[sandwwraith]

Yes, I think that the problem is that type and name have the same proto numbers. As we build Int -> class property map on this data, only name got deserialized.

Also, pay attention that only Json supports polymorphism with an embedded 'type' key. Other formats use default polymorphism, which is array-based (see PolymorphicSerializer.descriptor structure)

[ShreckYe]

There is an error in your class: BasicProject(@ProtoNumber(1) val type: String, @ProtoNumber(1) override val name: String) You reuse the ProtoNumber.

There are 2 things I'd like to explain for this. Firstly, after either I remove all the @ProtoNumber annotations, or I give all names a different @ProtoNumber such as @ProtoNumber(2), it's still the same exception with a slightly different message in that now it's the field name missing instead of type:

Exception in thread "main" kotlinx.serialization.MissingFieldException: Field 'name' is required for type with serial name 'example.examplePoly19.BasicProject', but it was missing

And secondly, I was actually doing this intentionally. As you both explained:

The polymorphic serializer works by injecting a synthetics structure with 2 members: type and value.

Also, pay attention that only Json supports polymorphism with an embedded 'type' key. Other formats use default polymorphism, which is array-based (see PolymorphicSerializer.descriptor structure)

So type would be a member in the top-level structure, and name would be in the nested value structure, and then they shouldn't conflict with each other while using the same proto number. I was trying to test this.

[sandwwraith]

I see. So, the input is:

0a144e6f745265676973746572656450726f6a65637412060a0474657374

Field #1: 0A String Length = 20, Hex = 14, UTF8 = "NotRegisteredProject"
Field #2: 12 String Length = 6, Hex = 06
    As sub-object :
    Field #1: 0A String Length = 4, Hex = 04, UTF8 = "test"

(you can use https://protogen.marcgravell.com/decode to get this kind of info)

And you're trying protoBuf.decodeFromHexString<Project>, expecting it to call defaultDeserializer { BasicProject.serializer() }. Did I get it right?

In that case, I do not see how name can be deserialized because it is missing from the input. The error Field 'name' is required for type with serial name '...', but it was missing is correct — provided that name has proto number 2. If it has proto number 1, then it is a clash of numbers, as I explained earlier. In case name is 1 and type is 2, you get Field 'type' is required for type with serial name 'BasicProject', but it was missing, which is still true.

[ShreckYe]

(you can use https://protogen.marcgravell.com/decode to get this kind of info)

This is quite helpful. Thank you.

And you're trying protoBuf.decodeFromHexString<Project>, expecting it to call defaultDeserializer { BasicProject.serializer() }. Did I get it right?

Yes.

In that case, I do not see how name can be deserialized because it is missing from the input. The error Field 'name' is required for type with serial name '...', but it was missing is correct — provided that name has proto number 2.

You are right. Sorry about this. I didn't update the serialized string and made a mistake here. To avoid such mistakes, I adapted the code to avoid any manual representations of encoded strings and inconsistencies in proto numbers like this:

(also moved to the root package and renamed to DefaultDeserializerIssue.kt to avoid too long serial names)

import kotlinx.serialization.*
import kotlinx.serialization.json.*
import kotlinx.serialization.modules.*
import kotlinx.serialization.protobuf.*

@Serializable
abstract class Project {
    @ProtoNumber(2)
    abstract val name: String
}

@Serializable
data class NamedProject(@ProtoNumber(2) override val name: String) : Project()

@Serializable
data class BasicProject(@ProtoNumber(1) val type: String, @ProtoNumber(2) override val name: String) : Project()

@Serializable
data class NotRegisteredForDeserializationProject(@ProtoNumber(2) override val name: String) : Project()

val serializingModule = SerializersModule {
    polymorphic(Project::class) {
        subclass(NamedProject::class)
        subclass(NotRegisteredForDeserializationProject::class)
    }
}
val sJson = Json { serializersModule = serializingModule }
val sProtoBuf = ProtoBuf { serializersModule = serializingModule }

val deserializingModule = SerializersModule {
    polymorphic(Project::class) {
        subclass(NamedProject::class)
        defaultDeserializer { BasicProject.serializer() }
    }
}
val dJson = Json { serializersModule = deserializingModule }
val dProtoBuf = ProtoBuf { serializersModule = deserializingModule }

fun main() {
    fun decodeProjectTest(project: Project) {
        println(dJson.decodeFromString<Project>(sJson.encodeToString<Project>(project).also(::println)))
        println()
        println(dProtoBuf.decodeFromHexString<Project>(sProtoBuf.encodeToHexString<Project>(project).also(::println)))
        println()
        println()
    }
    decodeProjectTest(NamedProject("test"))
    decodeProjectTest(NotRegisteredForDeserializationProject("test"))
}

And the output is still similar:

{"type":"NamedProject","name":"test"}
NamedProject(name=test)

0a0c4e616d656450726f6a6563741206120474657374
NamedProject(name=test)


{"type":"NotRegisteredForDeserializationProject","name":"test"}
BasicProject(type=NotRegisteredForDeserializationProject, name=test)

0a264e6f7452656769737465726564466f72446573657269616c697a6174696f6e50726f6a6563741206120474657374
Exception in thread "main" kotlinx.serialization.MissingFieldException: Field 'type' is required for type with serial name 'BasicProject', but it was missing
	at kotlinx.serialization.internal.PluginExceptionsKt.throwMissingFieldException(PluginExceptions.kt:20)
	at BasicProject.<init>(DefaultDeserializerIssue.kt:15)
	at BasicProject$$serializer.deserialize(DefaultDeserializerIssue.kt:15)
	at BasicProject$$serializer.deserialize(DefaultDeserializerIssue.kt:15)
	at kotlinx.serialization.protobuf.internal.ProtobufDecoder.decodeSerializableValue(ProtobufDecoding.kt:196)
	at kotlinx.serialization.protobuf.internal.ProtobufTaggedDecoder.decodeSerializableElement(ProtobufTaggedDecoder.kt:82)
	at kotlinx.serialization.encoding.CompositeDecoder$DefaultImpls.decodeSerializableElement$default(Decoding.kt:538)
	at kotlinx.serialization.internal.AbstractPolymorphicSerializer.deserialize(AbstractPolymorphicSerializer.kt:57)
	at kotlinx.serialization.protobuf.internal.ProtobufDecoder.decodeSerializableValue(ProtobufDecoding.kt:196)
	at kotlinx.serialization.protobuf.internal.ProtobufDecoder.decodeSerializableValue(ProtobufDecoding.kt:186)
	at kotlinx.serialization.protobuf.ProtoBuf.decodeFromByteArray(ProtoBuf.kt:144)
	at kotlinx.serialization.SerialFormatKt.decodeFromHexString(SerialFormat.kt:150)
	at DefaultDeserializerIssueKt.main$decodeProjectTest(DefaultDeserializerIssue.kt:55)
	at DefaultDeserializerIssueKt.main(DefaultDeserializerIssue.kt:48)
	at DefaultDeserializerIssueKt.main(DefaultDeserializerIssue.kt)

Process finished with exit code 1

From the serialized message and with the Online Protobuf Decoder you told me about, we can see that type is serialized with proto number 1 by default. However, we can see from the code that after adding @ProtoNumber(1) to type the exception is still there.

[ShreckYe]

Oh I think I get what went wrong (for details see the hidden comment above). After I remove type from BasicProject, the code works:

@Serializable
data class BasicProject(@ProtoNumber(2) override val name: String) : Project()

So does this mean that with Protobuf I can't deserialize type to a string property with a default deserializer in the way we do with JSON as instructed in the guide (I know there is a className parameter in the lambda parameter of defaultDeserializer that I can use, but it's a different approach then)? And decoding the type in default serialization is currently somewhat format-dependent?

[sandwwraith]

Yes, in the case of default (array) polymorphic serialization you cannot access type afterward, as the deserializer gets the inner object directly, which doesn't exist there unless you specifically duplicate it in input.

And decoding the type in default serialization is currently somewhat format-dependent?

It is for Json. If you enable deprecated useArrayPolymorphism in Json, which essentially disables all specific handling of type inside the object, you get similar behavior to protobuf and other formats. For array-polymorphic Json, if input would look like ["NotRegisteredProject", {"name": "foo"}], the error would be the same — there's no type key inside curly braces of the inner object.

[ShreckYe]

@sandwwraith Thank you very much for clarifying. I shall close this issue then. I think it might be helpful to point this out in the guide here, however.

[pdvrieze]

@sandwwraith Thank you very much for clarifying. I shall close this issue then. I think it might be helpful to point this out in the guide here, however.

I had a look at the guide. The use of type there in BasicProject is not great:

• It is misleading in that it appears needed for polymorphism (but actually is only valid because of the use as a default serializer
• It is format specific (it captures the polymorphic type attribute by accident) - it only works for Json. Instead a serializer with the type should be created to be format agnostic.