[cbor] Support writing Map (Object) with finite length

[cowtowncoder]

(from https://github.com/FasterXML/jackson-dataformat-cbor/issues/11)

By @mbaril:

It would be great to add the possibility to encode array and map with finite length.

[cowtowncoder]

(modified title as #1 already covers case of known-length arrays)

[martiwi]

I'm facing the same issue, which lead to have some unnecessary additional bytes to the serialized data.
Example and data in SOF:
https://stackoverflow.com/questions/48991682/serialize-fixed-size-map-to-cbor

[cowtowncoder]

As per example, usage would be via Streaming API (generator), and supporting that should be a bit easier than general-purpose databind. At databind, Maps could perhaps be supported as well.

But one big problem is that there is no applicable method use: writeStartObject() would require explicit length indicator: otherwise buffering would need to be used to calculate entry count, then encode (since indicator is variable-length).
And this can not be done in a patch (API change).
So most likely this is 3.0 only at this point.

[martiwi]

Thanks @cowtowncoder. Do you see any possible workaround at this point?

[cowtowncoder]

Not really. You can by-pass encoder with writeRaw / writeRawValue but that pretty much defeats the purpose.

[martiwi]

Thanks @cowtowncoder , actually, I've found out by reading the source code that the method to write a Map with a fixed size exists: public final void writeStartObject(int elementsToWrite)
So, I only have to cast jgen into a CBORGenerator and it works like a charm!
((CBORGenerator) jgen).writeStartObject(1);

[cowtowncoder]

Ah. Well spotted. The reason I missed that was that I had thought of adding method in JsonGenerator, but hadn't because of being unsure if and how to combine with possible argument of forObject. Will probably add combined in 3.0.
But it is good this is available for CBOR already, and that it actually works as expected -- keeping track of elements actually written is implemented to safeguard against invalid usage.

[quannh0308]

I'm having a problem with this feature.
The case is that I have a complicated/nested String. First, I can convert it to JsonNode, and later, use an ObjectMapper to convert JsonNode to CBOR byte[].
In the results byte[], the array size is finite, but the map size is not.
I haven't found any workaround by now. But for general purpose, I think you should put a flag in SerializationFeature like WRITE_FIXED_SIZE_OBJECT, so that it is posible to write a complicated json string to CBOR with sized map to remove extra '-1' bytes

[cowtowncoder]

@quannh0308 I don't think this can work as general SerializationFeature as it is CBOR-specific, unfortunately: tracking of sizes for all uses complicates code and adds overhead, primarily because then potentially full output hierarchy would need to be buffered.

But I think it will be possible to add necessary method in JsonGenerator for 2.10:

https://github.com/FasterXML/jackson-core/issues/517

and then eventually make ObjectNode and Map serializers use it: in these case we do know actual count of entries without buffering, and can encode things efficiently.

What will not be easy or perhaps even possible, unfortunately, will be doing the same for POJOs: problem is that things like @JsonAnyGetter and various filtering methods will affect number of properties actually written.

But I think that ability to at least force this via Tree Model (it is, after all, always possible to convert from POJO to Tree Model, then serialize that) would make this possible in general case and by users who really need to force length prefix.

I do think however that requiring use of length-prefix is a bad idea and implementations should not require it.

[rangaataws]

Hi cowtowncoder, I wanted to use finite length Object serialization through ObjectMapper. I m doing

        final CBORFactory cborFactory = new CBORFactory();
        mapper = new ObjectMapper(cborFactory);

What I understood from your conversations is, the support is added since 3.0. But it looks like it hasn't been made in Jackson-databind package which means I cannot use ObjectMapper to get the finite length support. Can you comment on it.

And second question is, do you think there should be a feature or flag to enable disable this functionality instead of making own assumption? If flag needs to be added, where will be added. Would like to know next steps on supporting finite length maps. Thanks for your hardwork on supporting the feature in 3.0.

[cowtowncoder]

@rangaataws 3.0 has not been released, and will not yet for quite some time. Current releases are 2.x, latest being 2.10.0.
But since underlying write method has been added, it should actually be possible to add support before 3.x, perhaps in 2.11, as 2.10 added this method in JsonGenerator API:

public void writeStartObject(Object forValue, int size) throws IOException;

so that databind could start calling it in 2.11.

Now... as to feature to add... perhaps. I think the way it would work is simply that databind would call method that passes length, when known -- it is not always practical to try to calculate it, for example when entries are filtered -- and then dataformat would decide to use it (or not).
So that would be CBORGenerator.Feature.ADD_MAP_LENGTH_PREFIX or something like that.

As to next steps: to be completely honest, I don't think I will have time to work on this in near future.
If anyone else wants to work on it that would be great. Changes should go in 2.11 branch.