Skip to content

Add binary stream support for XREAD and XREADGROUP (#3566) #4152

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 17 commits into from
May 27, 2025

Conversation

YoHanKi
Copy link
Contributor

@YoHanKi YoHanKi commented Apr 27, 2025

  • Created StreamEntryBinary class to support binary data with Map<byte[], byte[]>
  • Added xreadBinary, xreadBinaryAsMap, xreadGroupBinary, and xreadGroupBinaryAsMap methods to StreamBinaryCommands
  • Implemented binary stream builders in BuilderFactory
  • Added implementation in Jedis and UnifiedJedis classes
  • Created BinaryStreamEntryTest to verify binary stream functionality

- Created StreamEntryBinary class to support binary data with Map<byte[], byte[]>
- Added xreadBinary, xreadBinaryAsMap, xreadGroupBinary, and xreadGroupBinaryAsMap methods to StreamBinaryCommands
- Implemented binary stream builders in BuilderFactory
- Added implementation in Jedis and UnifiedJedis classes
- Created BinaryStreamEntryTest to verify binary stream functionality
@ggivo ggivo self-requested a review April 29, 2025 06:56
Copy link
Collaborator

@ggivo ggivo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Took a brief look at the PR and it looks good, one think that popped up is to extend the tests and make them follow existing testing pattern

For example use parameterized test to cover both RESP2/RESP3 and also cover UnifiedJedis, JedisCluster and JedisPooled

  • For Jedis client streams related test we already have redis.clients.jedis.commands.jedis.StreamsCommandsTest
  • For UnifiedJedis you can take a look as example at src/test/java/redis/clients/jedis/commands/unified/BinaryValuesCommandsTestBase.java and related ClusterBinaryValuesCommandsTest, PooledBinaryValuesCommandsTest. Following similar structure for stream commands will test also in cluster&pooled setup

YoHanKi and others added 2 commits May 9, 2025 01:14
…er, and Pooled clients TEST (redis#3566)

- Introduce `BinaryStreamEntryTest` and `BinaryStreamsCommandsTestBase` following the unified test pattern
- Add `ClusterBinaryStreamsCommandsTest` and `PooledBinaryStreamsCommandsTest` to cover JedisCluster and JedisPooled
- Parameterize tests to run under both RESP2 and RESP3 protocols
- Format BuilderFactory to conform to project formatter rules
Copy link
Collaborator

@ggivo ggivo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One more thing I forgot: we are planning a mass reformat at some point and will enforce common formatting rules. But for now, please only format the code that has changed, since formatting the entire file makes code reviews harder.

YoHanKi added 3 commits May 10, 2025 13:20
…sequence test (redis#3566)

- Refactor StreamsBinaryCommandsTest: replace Map.Entry<byte[], byte[]> with Map.Entry<byte[], StreamEntryID> for xreadBinary, xreadBinaryAsMap, xreadGroupBinary and related methods

- Update tests to assert on StreamEntryBinary IDs and byte-array payloads accordingly

- Add a new test case covering an illegal UTF-8 sequence (0xc3 0x28) to validate correct binary handling (issue redis#3566)
…#3566)

- Introduce a Map<byte[], StreamEntryID> parameter for binary stream commands

- Duplicate all existing test cases for xread, xreadAsMap, xreadGroup and xreadGroupAsMap from StreamsCommandsTest.java and replace them with xreadBinary, xreadBinaryAsMap, xreadGroupBinary and xreadGroupBinaryAsMap in StreamsBinaryCommandsTest
…dis#3566)

- Duplicate all existing test cases for xread, xreadAsMap, xreadGroup and xreadGroupAsMap from StreamsCommandsTest.java and replace them with xreadBinary, xreadBinaryAsMap, xreadGroupBinary and xreadGroupBinaryAsMap in StreamsBinaryCommandsTest
@YoHanKi
Copy link
Contributor Author

YoHanKi commented May 10, 2025

Hello @ggivo,

As you suggested, during the process of writing the duplicated tests, changing the existing Map<String, StreamEntryID> to Map.Entry<byte[], StreamEntryID> caused widespread code modifications. To preserve consistency, I added a method that accepts Map<byte[], StreamEntryID> as its parameter:

public final CommandObject<List<Map.Entry<byte[], List<StreamEntryBinary>>>> xreadBinary(
    XReadParams xReadParams,
    Map<byte[], StreamEntryID> streams
)

(in StreamsBinaryCommandsTest.java).

Apart from replacing calls to xread with xreadBinary, I endeavored to keep the tests as close to the originals as possible. Due to the nature of byte[] arrays, I only added the necessary private equals helper method for proper comparison.

Thank you for your valuable guidance and review.

@YoHanKi YoHanKi requested a review from ggivo May 10, 2025 11:05
YoHanKi added 2 commits May 17, 2025 12:54
- Implemented a new JedisByteMap<T> class similar to the existing JedisByteHashMap to properly handle byte array keys in Map<byte[], T> operations
- This implementation ensures that `get(byte[])` operations work correctly by using proper byte array equality comparison instead of reference comparison
- Fixes issues when using byte arrays as keys in stream-related operations
- Added tests for JedisByteMap similar to existing JedisByteHashMap tests
@ggivo
Copy link
Collaborator

ggivo commented May 20, 2025

@YoHanKi Hey, thanks for the update. Will take a look at it later today/tomorrow!

@ggivo ggivo requested a review from Copilot May 20, 2025 10:52
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds binary stream support for XREAD and XREADGROUP by introducing new response types, commands, builders, and tests.

  • Defined StreamEntryBinary and added corresponding builders in BuilderFactory
  • Extended command interfaces and implementations (xreadBinary* and xreadGroupBinary*) in StreamBinaryCommands, UnifiedJedis, Jedis, and CommandObjects
  • Introduced JedisByteMap and expanded test coverage with BinaryStreamEntryTest, pooled/cluster tests, and JedisByteHashMapTest updates

Reviewed Changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
StreamBinaryCommands.java Added xreadBinary* and xreadGroupBinary* method signatures
BuilderFactory.java Added binary stream builders (STREAM_ENTRY_BINARY, etc.)
CommandObjects.java Implemented command objects for binary stream commands
JedisByteMap.java New Map<byte[],T> implementation
JedisByteHashMapTest.java Added tests for JedisByteMap
Comments suppressed due to low confidence (1)

src/test/java/redis/clients/jedis/collections/JedisByteHashMapTest.java:25

  • [nitpick] The test class JedisByteHashMapTest now tests JedisByteMap as well; its name may mislead. Consider renaming to cover both implementations or splitting tests into separate classes.
private static JedisByteMap<byte[]> map2 = new JedisByteMap<>();

…ashMap (redis#3566)

- Changed Map<byte[], byte[]> in STREAM_ENTRY_BINARY_LIST to JedisByteHashMap.
@YoHanKi
Copy link
Contributor Author

YoHanKi commented May 20, 2025

Hello, @ggivo
I ran the full test suite locally using Redis version 7.2, which matches the version where the CI failures occurred. However, all tests passed successfully on my end.
Could you please advise on what else I should check to identify the cause of the CI failures?

@ggivo
Copy link
Collaborator

ggivo commented May 21, 2025

Hello, @ggivo I ran the full test suite locally using Redis version 7.2, which matches the version where the CI failures occurred. However, all tests passed successfully on my end. Could you please advise on what else I should check to identify the cause of the CI failures?

Looks like flaky tests unrelated to your change. I have opened a dedicated issue to investigate the flaky tests & rerun 7.2 job and now it passed. Let me check the other one

 There is already discrepancy in API between
 `xread` binary variant & `xread` string one.
 Binary one accepts varargs for requested streams, while the string variant expects them to be provided as `Map`.

  Going forward xread binary methods should use Map. This commit also marks existing `List<Object> xread`  one as deprecated in favor of newly introduced methods
  for type safety and better stream entry parsing.
@ggivo
Copy link
Collaborator

ggivo commented May 21, 2025

@YoHanKi
Seems there is a regression in the Redis server. After rebuilding the server from the latest unstable branch, I am able to reproduce setAndKeepttl failures locally.

@YoHanKi
Copy link
Contributor Author

YoHanKi commented May 21, 2025

Hello @ggivo

Thank you for taking the time to review this despite your busy schedule. Is there anything I need to verify or address regarding this issue?

@ggivo
Copy link
Collaborator

ggivo commented May 21, 2025

@YoHanKi
Since tests are passing against 7.x & 8.0 stable builds, I think failures are not related to this change, and it is safe. I have also reported the server issue. For your local dev, you can use the containerized test environment against some of the stable versions.

Meantime started reviewing your latest change. Unfortunately have some other tasks to complete today, and will continue tomorrow.

Thanks for spending time on this!

ggivo added 4 commits May 23, 2025 15:26
   * Deprecate List<Object> xreadXXX in favor of newly introduced typed methods
   * Java docs
   * STREAM_ENTRY_BINARY not used
ggivo added 2 commits May 24, 2025 11:29
  Added tests to cover Pipeline command executions.
  UnifiedJedis BinaryStreamsCommand test clean up and attempt to simplify focused on testing xreadBinary and xreadGroupBinary methods .
   * Merged & refactored a bit Jedis BinaryStreamEntryTest.java & StreamsBinaryCommandsTest to match UnifiedJedis StreamsBinaryCommandsTestBase.
@ggivo ggivo added the feature label May 27, 2025
@ggivo
Copy link
Collaborator

ggivo commented May 27, 2025

@YoHanKi
Nice work on this, @ — thanks for the contribution! 🙌
I made a few small tweaks to the test, and it’s all set now. Merging it in.
Appreciate the effort!

Copy link
Collaborator

@ggivo ggivo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@YoHanKi
Copy link
Contributor Author

YoHanKi commented May 27, 2025

Hello @ggivo !
It was a great opportunity to contribute, and reviewing the changes was a valuable learning experience. Thank you for giving me this opportunity.

@ggivo ggivo merged commit 33dd489 into redis:master May 27, 2025
7 of 9 checks passed
@ggivo ggivo added this to the 6.1.0 milestone May 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

StreamEntry (XREAD) does not support binary because it uses Map<String, String>
2 participants