Skip to content

Commit 94afcb9

Browse files
authored
[FLINK-38547] [flink-protobuf] Upgrade protobuf to 4.32.1 (#27137)
1 parent 0ffacdc commit 94afcb9

File tree

11 files changed

+1206
-18
lines changed

11 files changed

+1206
-18
lines changed

docs/content/docs/connectors/table/formats/protobuf.md

Lines changed: 37 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -151,10 +151,8 @@ Format Options
151151
<td>
152152
If this value is set to true, the format will read empty values as the default values defined in the proto file.
153153
If the value is set to false, the format will generate null values if the data element does not exist in the binary protobuf message.
154-
If proto syntax is proto3, users need to set this to true when using protobuf versions lower than 3.15 as older versions do not support
155-
checking for field presence which can cause runtime compilation issues. Additionally, primtive types will be set to default values
156-
instead of null as field presence cannot be checked for them. Please be aware that setting this to true will cause the deserialization
157-
performance to be much slower depending on schema complexity and message size.
154+
With Flink's current protobuf version (4.32.1), field presence is properly supported for proto3, allowing null handling for non-primitive types.
155+
Please be aware that setting this to true will cause the deserialization performance to be much slower depending on schema complexity and message size.
158156
</td>
159157
</tr>
160158
<tr>
@@ -291,4 +289,38 @@ OneOf field
291289
In the serialization process, there's no guarantee that the Flink fields of the same one-of group only contain at most one valid value.
292290
When serializing, each field is set in the order of Flink schema, so the field in the higher position will override the field in lower position in the same one-of group.
293291

294-
You can refer to [Language Guide (proto2)](https://developers.google.com/protocol-buffers/docs/proto) or [Language Guide (proto3)](https://developers.google.com/protocol-buffers/docs/proto3) for more information about Protobuf types.
292+
Supported Protobuf Versions
293+
------------
294+
295+
Flink uses protobuf-java 4.32.1 (corresponding to Protocol Buffers version 32), which includes support for:
296+
297+
- **Proto2 and Proto3 syntax**: Traditional `syntax = "proto2"` and `syntax = "proto3"` definitions
298+
- **Protobuf Editions**: The new `edition = "2023"` and `edition = "2024"` syntax introduced in Protocol Buffers v27+
299+
- **Improved proto3 field presence detection**: Better handling of optional fields without the limitations of older protobuf versions
300+
301+
### Using Protobuf Editions
302+
303+
Protobuf Editions provide a unified syntax that combines proto2 and proto3 functionality. If you're using Editions in your `.proto` files, Flink fully supports them:
304+
305+
```
306+
edition = "2023";
307+
package com.example;
308+
option java_package = "com.example";
309+
option java_multiple_files = true;
310+
311+
message SimpleTest {
312+
int64 uid = 1;
313+
string name = 2 [features.field_presence = EXPLICIT];
314+
// ... rest of your message definition
315+
}
316+
```
317+
318+
Editions allow fine-grained control over feature behavior at the file, message, or field level, while maintaining backward compatibility with proto2 and proto3. For more information, see the [Protobuf Editions documentation](https://protobuf.dev/editions/overview/).
319+
320+
Additional Resources
321+
----------------
322+
For more information about Protocol Buffers, refer to:
323+
- [Language Guide (proto2)](https://developers.google.com/protocol-buffers/docs/proto)
324+
- [Language Guide (proto3)](https://developers.google.com/protocol-buffers/docs/proto3)
325+
- [Language Guide (Editions)](https://protobuf.dev/programming-guides/editions/) - for the new Editions syntax
326+
- [Protobuf Editions Overview](https://protobuf.dev/editions/overview/) - understand the motivation and benefits of Editions

docs/content/release-notes/flink-2.1.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -182,4 +182,4 @@ Bump flink-shaded version to 20.0 to support Smile format.
182182
##### [FLINK-37760](https://issues.apache.org/jira/browse/FLINK-37760)
183183

184184
Bump parquet version to 1.15.3 to resolve parquet-avro module
185-
vulnerability found in [CVE-2025-30065](https://nvd.nist.gov/vuln/detail/CVE-2025-30065).
185+
vulnerability found in [CVE-2025-30065](https://nvd.nist.gov/vuln/detail/CVE-2025-30065).

flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/protobuf/ParquetProtoWriters.java

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,6 @@
2626
import org.apache.parquet.hadoop.ParquetWriter;
2727
import org.apache.parquet.hadoop.api.WriteSupport;
2828
import org.apache.parquet.io.OutputFile;
29-
import org.apache.parquet.proto.ProtoWriteSupport;
3029

3130
/** Convenience builder for creating {@link ParquetWriterFactory} instances for Protobuf classes. */
3231
public class ParquetProtoWriters {
@@ -62,7 +61,8 @@ protected ParquetProtoWriterBuilder<T> self() {
6261

6362
@Override
6463
protected WriteSupport<T> getWriteSupport(Configuration conf) {
65-
return new ProtoWriteSupport<>(clazz);
64+
// Use patched implementation compatible with protobuf 4.x
65+
return new PatchedProtoWriteSupport<>(clazz);
6666
}
6767
}
6868

0 commit comments

Comments
 (0)