Passing serializers dynamically

[arkivanov]

It would be really useful if we could specify serializers for non-serializable properties at runtime, similarly to how we specify serializers for generic properties. An imaginatory API could look like this.

The code from a library:

interface Args

@Serializable
class ArgsHolder(@DynamicallySerializable val args: Args)

Then the library consumer's (client) code:

@Serializable
class ArgsA(val data: String) : Args

@Serializable
class ArgsB(val data: String) : Args

val argsSerializer : KSerializer<Args> = ... // Created dynamically at runtime

val serializer = ArgsHolder.serializer(ArgsSerializer)

[pdvrieze]

Do you mean that you want the ability to specify a serializer factory instead of a serializer? In any case it would require a good amount of work, not only to implement, but especially to determine what the right semantics of such would be.

[arkivanov]

An example of I want would be as follows.

In a library I have this:

interface Args

@Serializable
class Data(val Args: Args)

Also in the same library I want to serialize and deserialize Data using an internal format, which is implementation details of the library. It could be Json or CBOR or any other.

E.g. I want to expose:

fun save(data: Data, serializer: KSerializer<Data>)

So the consumers of the library would create their own implementations of Args, create a polymorphic serializer of Args and pass it to the library.

[pdvrieze]

As to formats, that is always up to the code that calls the (de)serialization. By all means create your own format.

On Data this should work already if annotated with @Polymorphic (as it is an interface that is the default) with the condition that you must provide the serializersModule. If serialization is initiated inside your library you have to provide the serializersModule to library somehow (so the format knows it). You can do this in any way you want (e.g. ArgsRegistry.registerSubType<MyArgSubtype>())

In the alternate, the library also supports generated serializers with parameters when given a type parameter. The serializer for the type parameter is resolved on creation of the relevant serializer. That can be in a consuming library (you need an ArgsHolder<T>(data: T).

[arkivanov]

Thanks for the suggestion. I'm aware of the @Polymorphic annotation and the serializersModule think. In reality my use case is a bit more complex, I just wanted to keep the example as simple as possible. But in general, I don't want to deal with serializer modules as it would make the library's API surface more complex. Also I don't want to re-instantiate Json every time. Another reason is that that the case with serializer modules is less compile-time safe, as the user will have to not forget to properly create and supply the module. That's why I submitted this feature request.

The generic type suggestion is also not suitable in my particular case. It also makes the API surface an the usage more complex than I would like it to be. E.g. ArgsHolder may contain multiple polymorphic properties, so having <T1, T2, T3, ...> is a no go for me.

That's why I created this feature request.

[pdvrieze]

@arkivanov I'm still not 100% of what you want. Another option can be the "self-serializable" data is where you store the serializer in the object. E.g:

@Serializable(MyContainer.Companion::class)
class MyContainer<T>(private val serializer: KSerializer<T>, val data: T) {

    companion object: KSerializer<MyContainer<*>> {
        override val serialDescriptor = SerialDescriptor("MyContainer<${serializer.serialDescriptor.serialName}>", serializer.descriptor)

        override fun encode(encoder: Encoder, value: MyContainer<*>) {
            serializer.encode(value.data as T)
        }

        override fun decode(decoder: Decoder): MyContainer<*> {
            return MyContainer(serializer.decode(decoder))
        }
    }
}

This certainly works (although it needs a custom serializer). It comes into place where you have code that has dynamic return types that need (de)serialization based upon the dynamic type (without polymorphism).

[arkivanov]

I want to be able to serialize polymorphic types without using serializer modules.

Another example would be as follows. Consider a library that we don't control and can't change with the following API.

fun <T> saveData(data: T, serializer: SerializationStrategy<T>)

Now I have this code in my project consuming the library.

// Module :base
interface Args

@Serializable
class ArgsHolder(val args: Args)

// Module :foo depends on :base
@Serializable
class FooArgs(val data: String) : Args

// Module :bar depends on :base
@Serializable
class BarArgs(val data: String) : Args

Finally, somewhere in my code I want to pass ArgsHolder to the library.

fun saveArgs(args: Args, serializer: KSerializer<Args>) {
    saveData(data = ArgsHolder(args), serializer = ...) // Somehow I need to pass the ArgsHolder serializer
}

I would like to do something like this, but it's not possible currently.

// Module :base
@Serializable
class ArgsHolder(@DynamicallySerializable val args: Args)

// Somewhere in my code
fun saveArgs(args: Args, serializer: KSerializer<Args>) {
    saveData(data = ArgsHolder(args), serializer = ArgsHolder.serializer(serializer))
}

Hope this explains it better. Btw, your code snippet doesn't compile.

[pdvrieze]

Btw, your code snippet doesn't compile.
Thanks, it also doesn't work even with the correct names (deserialization is an issue as somehow the deserializer must be determined).

I've had another shot at it (now with the compiler/testing involved). But notice that using a generic parameter just makes things easier. And you can always pass an explicit serializer that is constructed however you want. If you want the serializers to be transparently provided for you, you have to follow the rules though:

    @Test
    fun testSerializeArgsHolder() {
        val data = ArgsHolder(FooArgs("Foo"))
        val ser = ArgsHolderSerializer(FooArgs.serializer())
        val actual = Json.encodeToString(ser,data)
        val expected = "{\"args\":{\"data\":\"Foo\"}}"
        assertEquals(expected, actual)
    }

    @Test
    fun testSerializeTypedArgsHolder() {
        val data = TypedArgsHolder(FooArgs("Foo"))
        val actual = Json.encodeToString(data)
        val expected = "{\"args\":{\"data\":\"Foo\"}}"
        assertEquals(expected, actual)
    }

    @Serializable
    data class TypedArgsHolder<T: Args>(val args: T)

    class ArgsHolderSerializer<T: Args>(private val elemSerializer: KSerializer<T>) : KSerializer<ArgsHolder> {
        override val descriptor: SerialDescriptor = buildClassSerialDescriptor("ArgsHolder") {
            element("args", elemSerializer.descriptor)
        }

        override fun serialize(
            encoder: Encoder,
            value: ArgsHolder,
        ) {
            encoder.encodeStructure(descriptor) {
                @Suppress("UNCHECKED_CAST")
                encodeSerializableElement(descriptor, 0, (elemSerializer as SerializationStrategy<Args>), value.args)
            }
        }

        override fun deserialize(decoder: Decoder): ArgsHolder {
            return decoder.decodeStructure(descriptor) {
                var args: Args? = null
                while(true) {
                    when(val i = decodeElementIndex(descriptor)) {
                        0 -> args = decodeSerializableElement(descriptor, 0, elemSerializer)
                        CompositeDecoder.DECODE_DONE -> break
                        else -> error("Unexpected index $i")
                    }
                    break;
                }
                requireNotNull(args) { "No args in data" }
                ArgsHolder(args)
            }
        }
    }

You could store the element serializer in the value for encoding purposes (a good technique for the returns in web frameworks like ktor), but deserialization doesn't have a value, so can't actually determine the deserializer to use from the data (unless you have an alternative way that determines the appropriate child deserializer from the context).

[arkivanov]

Thanks for looking at this!

Your ArgsHolderSerializer works, however it implies that I have to provide KSerializer<T : Args> as a parameter. If you look at the use case I described in my previous message, there are two implementations of Args in different modules: FooArgs and BarArgs. So I will also need to create the polymorphic serializer of Args in my app module. But this is probably out of scope of this feature request.

I think the main point of this feature request is to allow automatic generation of ArgsHolderSerializer<T> for easier usage, as writing ArgsHolderSerializer manually is not trivial. Talking about that example egain, the use case might be quite common for a particular library, meaning that many library consumers will have to write the same serializer over and over again for their custom ArgsHolder class.

The suggestion is to allow the following API, that can be used in the library consumer's code:

interface Args

@Serializable
class ArgsHolder(
    @DynamicallySerializable val args: Args,
    // Potentially more @DynamicallySerializable parameters
)

It should automatically generate something similar to ArgsHolderSerializer<T> from your last comment.

[sandwwraith]

Am I right that the only difference between @Contextual and @DynamicallySerializable is that you want the serializer to be passed statically as ArgsHolder.serializer(...) instead of creating a serializersModule?

Another example would be as follows. Consider a library that we don't control and can't change with the following API.
fun saveData(data: T, serializer: SerializationStrategy)

Usually formats should have SerializersModule as one of their parameters (to create Encoder and Decoder). In that case, your usage can look like this:

interface Args

@Serializable
class ArgsHolder(@Contextual val args: Args)

// Module :foo depends on :base
@Serializable
class FooArgs(val data: String) : Args

// Module :bar depends on :base
@Serializable
class BarArgs(val data: String) : Args

fun main() {
  val myModule = serializersModuleOf<BarArgs>(MyBarSerializer())
  val mySaver = DataSaver(myModule)

  mySaver.saveData(ArgsHolder(BarArgs(...)) // automatically picks MyBarSerializer from the module
 
}

[arkivanov]

Am I right that the only difference between @contextual and @DynamicallySerializable is that you want the serializer to be passed statically as ArgsHolder.serializer(...) instead of creating a serializersModule?

Yes, correct. There are a few reasons that I described above. In particular, I want it to be as much compile time safe as possible (i.e. you won't forget to pass the serializer). Also, it feels logical and consistent with generic serializers. Plus, DataSaver could use one single instance of e.g. Json. But the main reason is because of the specifics of the library API, it's pretty inconvenient for the client to pass an additional SerializersModule.

[pdvrieze]

@arkivanov I guess you don't want the generic parameter either as in TypedArgsHolder in my example (as its serializer is automatically initialized with the serializer for the generic argument).

[arkivanov]

@pdvrieze, correct. There are a few reasons. I don't like that the serialization requirement affects the API (I wouldn't use generics in this case if the serialization is not needed). There can be multiple arguments in one class, so having multiple type parameters is inconvenient. In theory, the property can be mutable, so with T instead of out T it will be a bit harder to work with.

[pdvrieze]

@arkivanov I find that a good way to deal with API / serialization discrepancies is through a serial delegate. What I mean with this is a serializable private nested (not inner) class that is only used for serialization. It can use defaults and duplication to allow for increased flexibility. Then a "custom" serializer is used to just convert the values and otherwise delegate to the generated serializer for the delegate.

The delegate would be able to be automatically generated, although the delegate has two requirements: it must contain a single argument constructor accepting the value; it must also have a function to get an instance of the actual type (this could be done using an interface that marks the type as delegate).