Binary value being cut in case of zero byte at the end

[sam0delkin]

Hi, i have a table where i storing IDs as binary UUIDs:

CREATE TABLE `test` (`id` binary(16) NOT NULL, PRIMARY KEY (`id`));

When i insert a new row with binary UUID which has a zero at the end:

INSERT INTO test (id) VALUES (UUID_TO_BIN('1f0727b7-2cc9-6910-920c-b12f2d100600'));

Maxwell provides a following JSON:

{"database":"app","table":"test","type":"insert","ts":1756363904,"xid":97707,"commit":true,"data":{"id":"HwcntyzJaRCSDLEvLRAG"}}

Notice the value HwcntyzJaRCSDLEvLRAG. Now let's try to base64 encode the binary UUID on the MySQL side:

SELECT BIN_TO_UUID(id), to_base64(id) FROM test;

->

BIN_TO_UUID(id)	to_base64(id)
1f0727b7-2cc9-6910-920c-b12f2d100600	HwcntyzJaRCSDLEvLRAGAA==

After some investigation i found that HwcntyzJaRCSDLEvLRAG corresponds to 1f0727b7-2cc9-6910-920c-b12f2d1006 (without last zero byte).

So, Maxwell is cut the last byte in case of it's value is zero. I suppose Maxwell treat the last zero byte as the "null char".

I'm using maxwell v1.44.0

[osheroff]

yeah, this is a real bug, and it's not all that easy to fix.

as part of the field in the replication stream, mysql outputs a length N stating (more/less) "how much of this field is used". Deep under the hood, maxwell uses this to read a byte[] of length N. In this case, mysql's length is 15, causing us to erroneously chop off the data. At this deep-under-hood level I think there's no difference between a BINARY and a VARBINARY (or VARCHAR for that matter), so I can't simply ignore the value of 15 and use 16.

Over the hood, now, one could fix this by special casing it: if it's a BINARY column, ignore the fact that we're given a length 15 byte[]; initialize a length 16 byte[] with zeros, copy over the relevant bits, everything's fine. But the problem here is that this requires knowing that the column was defined as BINARY with length 16. It's not a huge arduous change but it requires:

• mucking about with the sql parser to train it that binary columns are, in fact, not the same as strings
• doing the same with the schema capture code
• and now, all the installations will still have to recapture their schema.

Were I a younger, more ambitious man, I'd certainly be tilting at this windmill, but I am that proverbial thankless man in Nebraska on this one and I'm not sure where more it's going, at least for now.