Silent exception when consuming a compressed topic Kafka

[ldrogou]

Hello,

I have issue with consumer kafka

Environment:

• Quarkus : 3.27.0
• Image: eclipse-temurin:21-jre-alpine
• Compression: compression.lz4.level = 9

Description:

I am attempting to consume a compressed Kafka topic. Connection to the broker's established successfully, but consumer lag increasing and zero messages are consumed.

Observation:

• Standard logs show no errors.
• Even with DEBUG logging enabled, I saw no anomalies (no timeouts or disconnections....).
• The failure is silent.

Debugging:
After inspecting the Kubernetes pod in debug mode (step by step), I found that exception is thrown during every poll in io.smallrye.reactive.messaging.kafka.impl.ReactiveKafkaConsumer:

java.lang.UnsatisfiedLinkError: /tmp/snappy-1.1.10-8e353f29-2657-4e24-9ab8-4b9010e69232-libsnappyjava.so: Error loading shared library ld-linux-x86-64.so.2: No such file or directory (needed by /tmp/snappy-1.1.10-8e353f29-2657-4e24-9ab8-4b9010e69232-libsnappyjava.so)
Could not initialize class org.xerial.snappy.Snappy
Exception java.lang.UnsatisfiedLinkError: /tmp/snappy-1.1.10-8e353f29-2657-4e24-9ab8-4b9010e69232-libsnappyjava.so: Error loading shared library ld-linux-x86-64.so.2: No such file or directory (needed by /tmp/snappy-1.1.10-8e353f29-2657-4e24-9ab8-4b9010e69232-libsnappyjava.so) [in thread "smallrye-kafka-consumer-thread-1"]

If I explicitly add snappy-java dependency in pom.xml it works well.
It also works correctly with the eclipse-temurin:21-jre image, even without the explicit dependency

Do you known why I don't have anything in log ?
I expected this type of error to propagate up the stack and be logged explicitly. What do you think ?

[quarkus-bot[bot]]

/cc @alesj (kafka), @cescoffier (kafka), @geoand (kubernetes), @ozangunalp (kafka)

[cescoffier]

It’s very likely related to the base image: Alpine is MUSL-based and doesn’t provide the glibc loader (ld-linux-x86-64.so) that the native snappy-java library expects. That’s why it works fine with eclipse-temurin:21-jre but fails with eclipse-temurin:21-jre-alpine. In general, we don’t recommend Alpine images for Quarkus because of exactly these kinds of native library incompatibilities.

What is clearly on us, though, is the lack of visibility: this exception should not be effectively “silent”. We need to improve error reporting in this area so that such failures are surfaced in the logs rather than only visible when you attach a debugger.

@ozangunalp this would be in SmallRye Reactive Messaging right?

[ozangunalp]

Yes, it gets stuck in the polling retry.
I am not sure if we can come up with a list of "retryable errors" but we should at least log them.

[ldrogou]

Version 1.1.10.8 snappy-java fixes my specific issue for Alpine image : xerial/snappy-java#632
I use Quarkus 3.27.0 / kafka-client 4.0.0 => snappy-java 1.1.10.5

This is where I put my breakpoint to see the error (in kubernetes environment) : https://github.com/smallrye/smallrye-reactive-messaging/blob/main/smallrye-reactive-messaging-kafka/src/main/java/io/smallrye/reactive/messaging/kafka/impl/ReactiveKafkaConsumer.java#L199

[cescoffier]

We should be able to bump the snappy-java version.