JsonGenerator writeRaw causes 'IllegalArgumentException: Called operation not supported for TokenBuffer'

[elwaxoro]

I'm still running into issues with java.lang.IllegalArgumentException: Called operation not supported for TokenBuffer exception using objectMapper.valueToTree<JsonNode>(...) with a serializer that invokes JsonGenerator.writeRaw. Jackson databind version 2.9.6

According to 348 and 743 this should be working?

Exception is as follows:

java.lang.IllegalArgumentException: Called operation not supported for TokenBuffer

	at com.fasterxml.jackson.databind.ObjectMapper.valueToTree(ObjectMapper.java:2802)
	at foo.JacksonSerializerTest.testJsonNode(JacksonSerializerTest.kt:26)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
	at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Called operation not supported for TokenBuffer
	at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._wrapAsIOE(DefaultSerializerProvider.java:509)
	at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:482)
	at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
	at com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:2655)
	at com.fasterxml.jackson.databind.ObjectMapper.valueToTree(ObjectMapper.java:2797)
	... 23 more
Caused by: java.lang.UnsupportedOperationException: Called operation not supported for TokenBuffer
	at com.fasterxml.jackson.databind.util.TokenBuffer._reportUnsupportedOperation(TokenBuffer.java:1217)
	at com.fasterxml.jackson.databind.util.TokenBuffer.writeRaw(TokenBuffer.java:775)
	at fooTestSerializer.serialize(JacksonSerializerTest.kt:40)
	at foo.TestSerializer.serialize(JacksonSerializerTest.kt:38)
	at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
	... 26 more

Self-contained unit test showing issue (kotlin):

import com.fasterxml.jackson.core.JsonGenerator
import com.fasterxml.jackson.databind.JsonNode
import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.databind.SerializerProvider
import com.fasterxml.jackson.databind.annotation.JsonSerialize
import com.fasterxml.jackson.databind.ser.std.StdSerializer
import org.junit.Test

class JacksonSerializerTest {

    private val objectMapper: ObjectMapper = ObjectMapper()

    @Test
    fun testSerialize() {
        val foo = Foo("JSON!")
        // This works great
        val json: String = objectMapper.writeValueAsString(foo)
    }

    @Test
    fun testJsonNode() {
        val foo = Foo("JSON!")
        // UnsupportedOperationException is thrown here
        val node: JsonNode = objectMapper.valueToTree<JsonNode>(foo)
    }

    @Test
    fun testJsonNode2() {
        val foo = Foo("JSON!")
        val json = objectMapper.writeValueAsString(foo)
        // This works great
        val node: JsonNode = objectMapper.readTree(json)
    }
}

class TestSerializer : StdSerializer<Foo>(Foo::class.java) {
    override fun serialize(value: Foo, gen: JsonGenerator, provider: SerializerProvider) =
            gen.writeRaw("""
                { "Bar" : "${value.Bar}", "Bonus": "Foo" }
            """.trimIndent())
}

@JsonSerialize(using = TestSerializer::class)
data class Foo(val Bar: String)

[cowtowncoder]

No, basic "writeRaw()" does not and can not work: "writeValue()", however, does.
The difference is in that writeRaw() will literally by-pass all verifications and append its contents to format output -- nothing is known of its structure, if any, and it will not change output state, so no separators are added and so on.
It may be used, for example, for adding white-space for indentation.

writeRawValue(), on the other hand, is bit more limited in that it will be assumed to be a syntactic value, like JSON String, or Boolean, and generator will add necessary separators.
Output must occur in place where value is expected: this is the case unless output is expecting field name for JSON Object.
As such it can be stored in TokenBuffer; contents themselves could be structured value, no assumption is made it would be a single token, but it should be valid JSON value construct (or construct of whatever underlying format is).

So: your serializer probably should use writeRawValue() instead, and if so, things should work.

I hope this helps.

[elwaxoro]

@cowtowncoder, thanks for the detail - writeRawValue() works exactly as advertised! The only side effect appears to be that it requires an additional check on deserialization? Possibly I'm putting this deserializer together incorrectly:

class JacksonSerializerTest {

    private val objectMapper: ObjectMapper = ObjectMapper()
    
    @Test
    fun testJsonNode() {
        val foo = Foo("JSON!")
        // UnsupportedOperationException was being thrown here
        val node = objectMapper.valueToTree<JsonNode>(foo)
        val foo2 = objectMapper.treeToValue(node, Foo::class.java)
    }

    @Test
    fun testJsonNode2() {
        val foo = Foo("JSON!")
        val json = objectMapper.writeValueAsString(foo)
        val node: JsonNode = objectMapper.readTree(json)
        val foo2 = objectMapper.treeToValue(node, Foo::class.java)
    }
}

class TestSerializer : StdSerializer<Foo>(Foo::class.java) {
    override fun serialize(value: Foo, gen: JsonGenerator, provider: SerializerProvider) =
            gen.writeRawValue("""
                { "Bar" : "${value.Bar}", "Bonus": "Foo" }
            """.trimIndent())
}

class TestDeserializer : StdDeserializer<Foo>(Foo::class.java) {
    override fun deserialize(p: JsonParser, ctxt: DeserializationContext): Foo {
        val json: String =
                if (p.embeddedObject != null && p.embeddedObject is RawValue) {
                    // handles valueToTree -> treeToValue
                    (p.embeddedObject as RawValue).rawValue() as String
                } else {
                    // handles json string -> readTree -> treeToValue
                    p.readValueAsTree<JsonNode>().toString()
                }
        println(json)
        // just return junk here for example, actual deserializer wants plain json string
        return Foo("asdf")
    }
}

@JsonSerialize(using = TestSerializer::class)
@JsonDeserialize(using = TestDeserializer::class)
data class Foo(val Bar: String)

Thanks again very much for the clarification. Closing this issue...

[cowtowncoder]

@elwaxoro Unfortunately I think that's as clean as it gets, since raw value / EMBEDDED_OBJECT is bit of a work-around, and handling has to essentially differ.

But just one more aspect that might be helpful is that instead of writeRawValue() you could also try writeEmbeddedObject(). That should retain whatever opaque thing you want, through processing.
Thing with "raw [value]" is that it really is meant as by-pass to raw physical format; and TokenBuffer not having such has to somehow fake it, leading to virtual non-format.
So what you see is that with JSON, there is the physical JSON that gets generated, and then re-parsed; with buffer not so much.
With embedded objects thing would be carried through; although if it is ever written as JSON, it would not retain its identity either -- embedded objects are also meant for format-specific things, but instead of physical constructs, as containers for logical values (like, say, UUIDs of BSON; or binary data from any source [since JSON has no native binary value type]).

[elwaxoro]

@cowtowncoder , no worries. This will work just fine as-is. Thanks again!