Add binary stream support for XREAD and XREADGROUP (#3566) #4152

YoHanKi · 2025-04-27T12:03:58Z

Created StreamEntryBinary class to support binary data with Map<byte[], byte[]>
Added xreadBinary, xreadBinaryAsMap, xreadGroupBinary, and xreadGroupBinaryAsMap methods to StreamBinaryCommands
Implemented binary stream builders in BuilderFactory
Added implementation in Jedis and UnifiedJedis classes
Created BinaryStreamEntryTest to verify binary stream functionality

- Created StreamEntryBinary class to support binary data with Map<byte[], byte[]> - Added xreadBinary, xreadBinaryAsMap, xreadGroupBinary, and xreadGroupBinaryAsMap methods to StreamBinaryCommands - Implemented binary stream builders in BuilderFactory - Added implementation in Jedis and UnifiedJedis classes - Created BinaryStreamEntryTest to verify binary stream functionality

ggivo

Took a brief look at the PR and it looks good, one think that popped up is to extend the tests and make them follow existing testing pattern

For example use parameterized test to cover both RESP2/RESP3 and also cover UnifiedJedis, JedisCluster and JedisPooled

For Jedis client streams related test we already have redis.clients.jedis.commands.jedis.StreamsCommandsTest
For UnifiedJedis you can take a look as example at src/test/java/redis/clients/jedis/commands/unified/BinaryValuesCommandsTestBase.java and related ClusterBinaryValuesCommandsTest, PooledBinaryValuesCommandsTest. Following similar structure for stream commands will test also in cluster&pooled setup

src/test/java/redis/clients/jedis/commands/binary/BinaryStreamEntryTest.java

…er, and Pooled clients TEST (redis#3566) - Introduce `BinaryStreamEntryTest` and `BinaryStreamsCommandsTestBase` following the unified test pattern - Add `ClusterBinaryStreamsCommandsTest` and `PooledBinaryStreamsCommandsTest` to cover JedisCluster and JedisPooled - Parameterize tests to run under both RESP2 and RESP3 protocols - Format BuilderFactory to conform to project formatter rules

src/main/java/redis/clients/jedis/commands/StreamBinaryCommands.java

ggivo

One more thing I forgot: we are planning a mass reformat at some point and will enforce common formatting rules. But for now, please only format the code that has changed, since formatting the entire file makes code reviews harder.

…sequence test (redis#3566) - Refactor StreamsBinaryCommandsTest: replace Map.Entry<byte[], byte[]> with Map.Entry<byte[], StreamEntryID> for xreadBinary, xreadBinaryAsMap, xreadGroupBinary and related methods - Update tests to assert on StreamEntryBinary IDs and byte-array payloads accordingly - Add a new test case covering an illegal UTF-8 sequence (0xc3 0x28) to validate correct binary handling (issue redis#3566)

…#3566) - Introduce a Map<byte[], StreamEntryID> parameter for binary stream commands - Duplicate all existing test cases for xread, xreadAsMap, xreadGroup and xreadGroupAsMap from StreamsCommandsTest.java and replace them with xreadBinary, xreadBinaryAsMap, xreadGroupBinary and xreadGroupBinaryAsMap in StreamsBinaryCommandsTest

…dis#3566) - Duplicate all existing test cases for xread, xreadAsMap, xreadGroup and xreadGroupAsMap from StreamsCommandsTest.java and replace them with xreadBinary, xreadBinaryAsMap, xreadGroupBinary and xreadGroupBinaryAsMap in StreamsBinaryCommandsTest

YoHanKi · 2025-05-10T07:10:40Z

Hello @ggivo,

As you suggested, during the process of writing the duplicated tests, changing the existing Map<String, StreamEntryID> to Map.Entry<byte[], StreamEntryID> caused widespread code modifications. To preserve consistency, I added a method that accepts Map<byte[], StreamEntryID> as its parameter:

public final CommandObject<List<Map.Entry<byte[], List<StreamEntryBinary>>>> xreadBinary(
    XReadParams xReadParams,
    Map<byte[], StreamEntryID> streams
)

(in StreamsBinaryCommandsTest.java).

Apart from replacing calls to xread with xreadBinary, I endeavored to keep the tests as close to the originals as possible. Due to the nature of byte[] arrays, I only added the necessary private equals helper method for proper comparison.

Thank you for your valuable guidance and review.

- Implemented a new JedisByteMap<T> class similar to the existing JedisByteHashMap to properly handle byte array keys in Map<byte[], T> operations - This implementation ensures that `get(byte[])` operations work correctly by using proper byte array equality comparison instead of reference comparison - Fixes issues when using byte arrays as keys in stream-related operations

- Added tests for JedisByteMap similar to existing JedisByteHashMap tests

ggivo · 2025-05-20T10:50:14Z

@YoHanKi Hey, thanks for the update. Will take a look at it later today/tomorrow!

Copilot

Pull Request Overview

This PR adds binary stream support for XREAD and XREADGROUP by introducing new response types, commands, builders, and tests.

Defined StreamEntryBinary and added corresponding builders in BuilderFactory
Extended command interfaces and implementations (xreadBinary* and xreadGroupBinary*) in StreamBinaryCommands, UnifiedJedis, Jedis, and CommandObjects
Introduced JedisByteMap and expanded test coverage with BinaryStreamEntryTest, pooled/cluster tests, and JedisByteHashMapTest updates

Reviewed Changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
StreamBinaryCommands.java	Added `xreadBinary` and `xreadGroupBinary` method signatures
BuilderFactory.java	Added binary stream builders (`STREAM_ENTRY_BINARY`, etc.)
CommandObjects.java	Implemented command objects for binary stream commands
JedisByteMap.java	New `Map<byte[],T>` implementation
JedisByteHashMapTest.java	Added tests for `JedisByteMap`

Comments suppressed due to low confidence (1)

src/test/java/redis/clients/jedis/collections/JedisByteHashMapTest.java:25

[nitpick] The test class JedisByteHashMapTest now tests JedisByteMap as well; its name may mislead. Consider renaming to cover both implementations or splitting tests into separate classes.

private static JedisByteMap<byte[]> map2 = new JedisByteMap<>();

src/main/java/redis/clients/jedis/commands/StreamBinaryCommands.java

src/main/java/redis/clients/jedis/CommandObjects.java

src/main/java/redis/clients/jedis/BuilderFactory.java

src/main/java/redis/clients/jedis/util/JedisByteMap.java

…ashMap (redis#3566) - Changed Map<byte[], byte[]> in STREAM_ENTRY_BINARY_LIST to JedisByteHashMap.

YoHanKi · 2025-05-20T15:23:28Z

Hello, @ggivo
I ran the full test suite locally using Redis version 7.2, which matches the version where the CI failures occurred. However, all tests passed successfully on my end.
Could you please advise on what else I should check to identify the cause of the CI failures?

ggivo · 2025-05-21T04:42:03Z

Hello, @ggivo I ran the full test suite locally using Redis version 7.2, which matches the version where the CI failures occurred. However, all tests passed successfully on my end. Could you please advise on what else I should check to identify the cause of the CI failures?

Looks like flaky tests unrelated to your change. I have opened a dedicated issue to investigate the flaky tests & rerun 7.2 job and now it passed. Let me check the other one

There is already discrepancy in API between `xread` binary variant & `xread` string one. Binary one accepts varargs for requested streams, while the string variant expects them to be provided as `Map`. Going forward xread binary methods should use Map. This commit also marks existing `List<Object> xread` one as deprecated in favor of newly introduced methods for type safety and better stream entry parsing.

ggivo · 2025-05-21T08:04:19Z

@YoHanKi
Seems there is a regression in the Redis server. After rebuilding the server from the latest unstable branch, I am able to reproduce setAndKeepttl failures locally.

YoHanKi · 2025-05-21T10:37:17Z

Hello @ggivo

Thank you for taking the time to review this despite your busy schedule. Is there anything I need to verify or address regarding this issue?

ggivo · 2025-05-21T11:19:21Z

@YoHanKi
Since tests are passing against 7.x & 8.0 stable builds, I think failures are not related to this change, and it is safe. I have also reported the server issue. For your local dev, you can use the containerized test environment against some of the stable versions.

Meantime started reviewing your latest change. Unfortunately have some other tasks to complete today, and will continue tomorrow.

Thanks for spending time on this!

* Deprecate List<Object> xreadXXX in favor of newly introduced typed methods

* Java docs * STREAM_ENTRY_BINARY not used

Added tests to cover Pipeline command executions. UnifiedJedis BinaryStreamsCommand test clean up and attempt to simplify focused on testing xreadBinary and xreadGroupBinary methods .

* Merged & refactored a bit Jedis BinaryStreamEntryTest.java & StreamsBinaryCommandsTest to match UnifiedJedis StreamsBinaryCommandsTestBase.

ggivo · 2025-05-27T08:57:59Z

@YoHanKi
Nice work on this, @ — thanks for the contribution! 🙌
I made a few small tweaks to the test, and it’s all set now. Merging it in.
Appreciate the effort!

ggivo

LGTM

YoHanKi · 2025-05-27T09:06:58Z

Hello @ggivo !
It was a great opportunity to contribute, and reviewing the changes was a valuable learning experience. Thank you for giving me this opportunity.

ggivo added the enhancement label Apr 29, 2025

ggivo self-requested a review April 29, 2025 06:56

ggivo requested changes May 8, 2025

View reviewed changes

ggivo reviewed May 8, 2025

View reviewed changes

src/test/java/redis/clients/jedis/commands/binary/BinaryStreamEntryTest.java Outdated Show resolved Hide resolved

YoHanKi and others added 2 commits May 9, 2025 01:14

Merge branch 'master' into feature/XREAD-support-binary

308db55

ggivo reviewed May 9, 2025

View reviewed changes

src/main/java/redis/clients/jedis/commands/StreamBinaryCommands.java Outdated Show resolved Hide resolved

ggivo reviewed May 9, 2025

View reviewed changes

YoHanKi added 3 commits May 10, 2025 13:20

YoHanKi requested a review from ggivo May 10, 2025 11:05

YoHanKi added 2 commits May 17, 2025 12:54

Add JedisByteMapTest (redis#3566)

7ddf39c

- Added tests for JedisByteMap similar to existing JedisByteHashMap tests

ggivo requested a review from Copilot May 20, 2025 10:52

Copilot AI reviewed May 20, 2025

View reviewed changes

Changed Map<byte[], byte[]> in STREAM_ENTRY_BINARY_LIST to JedisByteH…

246123d

…ashMap (redis#3566) - Changed Map<byte[], byte[]> in STREAM_ENTRY_BINARY_LIST to JedisByteHashMap.

ggivo added 4 commits May 23, 2025 15:26

Add xreadBinary and xreadGroupBinary methods to PipelineBase

a2cf07e

* Deprecate List<Object> xreadXXX in favor of newly introduced typed methods

Clean up

03f0d8b

* Java docs * STREAM_ENTRY_BINARY not used

Merge branch 'master' into feature/XREAD-support-binary

15142da

Fix BinaryStreamEntryTest test

dc37d97

ggivo added 2 commits May 24, 2025 11:29

Add BinaryStreamsPipeline tests

663c0e1

Added tests to cover Pipeline command executions. UnifiedJedis BinaryStreamsCommand test clean up and attempt to simplify focused on testing xreadBinary and xreadGroupBinary methods .

Test cleanup & formating

a05d95d

* Merged & refactored a bit Jedis BinaryStreamEntryTest.java & StreamsBinaryCommandsTest to match UnifiedJedis StreamsBinaryCommandsTestBase.

ggivo added the feature label May 27, 2025

Merge branch 'master' into feature/XREAD-support-binary

e7ab9e0

ggivo approved these changes May 27, 2025

View reviewed changes

ggivo merged commit 33dd489 into redis:master May 27, 2025
7 of 9 checks passed

ggivo linked an issue May 27, 2025 that may be closed by this pull request

StreamEntry (XREAD) does not support binary because it uses Map<String, String> #3566

Closed

ggivo mentioned this pull request May 27, 2025

StreamEntry (XREAD) does not support binary because it uses Map<String, String> #3566

Closed

Add binary stream support for XREAD and XREADGROUP (#3566) #4152

Add binary stream support for XREAD and XREADGROUP (#3566) #4152

Uh oh!

Conversation

YoHanKi commented Apr 27, 2025

Uh oh!

ggivo left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ggivo left a comment

Choose a reason for hiding this comment

Uh oh!

YoHanKi commented May 10, 2025

Uh oh!

ggivo commented May 20, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

YoHanKi commented May 20, 2025

Uh oh!

ggivo commented May 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ggivo commented May 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

YoHanKi commented May 21, 2025

Uh oh!

ggivo commented May 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ggivo commented May 27, 2025

Uh oh!

ggivo left a comment

Choose a reason for hiding this comment

Uh oh!

YoHanKi commented May 27, 2025

Uh oh!

Uh oh!

Uh oh!

ggivo left a comment •

edited

Loading

ggivo commented May 21, 2025 •

edited

Loading

ggivo commented May 21, 2025 •

edited

Loading

ggivo commented May 21, 2025 •

edited

Loading