Skip to content

Commit e64a205

Browse files
committed
update docs for protobuf upgrade
1 parent 9798a75 commit e64a205

File tree

2 files changed

+61
-6
lines changed

2 files changed

+61
-6
lines changed

docs/content/docs/connectors/table/formats/protobuf.md

Lines changed: 37 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -151,10 +151,8 @@ Format Options
151151
<td>
152152
If this value is set to true, the format will read empty values as the default values defined in the proto file.
153153
If the value is set to false, the format will generate null values if the data element does not exist in the binary protobuf message.
154-
If proto syntax is proto3, users need to set this to true when using protobuf versions lower than 3.15 as older versions do not support
155-
checking for field presence which can cause runtime compilation issues. Additionally, primtive types will be set to default values
156-
instead of null as field presence cannot be checked for them. Please be aware that setting this to true will cause the deserialization
157-
performance to be much slower depending on schema complexity and message size.
154+
With Flink's current protobuf version (4.32.1), field presence is properly supported for proto3, allowing null handling for non-primitive types.
155+
Please be aware that setting this to true will cause the deserialization performance to be much slower depending on schema complexity and message size.
158156
</td>
159157
</tr>
160158
<tr>
@@ -291,4 +289,38 @@ OneOf field
291289
In the serialization process, there's no guarantee that the Flink fields of the same one-of group only contain at most one valid value.
292290
When serializing, each field is set in the order of Flink schema, so the field in the higher position will override the field in lower position in the same one-of group.
293291

294-
You can refer to [Language Guide (proto2)](https://developers.google.com/protocol-buffers/docs/proto) or [Language Guide (proto3)](https://developers.google.com/protocol-buffers/docs/proto3) for more information about Protobuf types.
292+
Supported Protobuf Versions
293+
------------
294+
295+
Flink uses protobuf-java 4.32.1 (corresponding to Protocol Buffers version 32), which includes support for:
296+
297+
- **Proto2 and Proto3 syntax**: Traditional `syntax = "proto2"` and `syntax = "proto3"` definitions
298+
- **Protobuf Editions**: The new `edition = "2023"` and `edition = "2024"` syntax introduced in Protocol Buffers v27+
299+
- **Improved proto3 field presence detection**: Better handling of optional fields without the limitations of older protobuf versions
300+
301+
### Using Protobuf Editions
302+
303+
Protobuf Editions provide a unified syntax that combines proto2 and proto3 functionality. If you're using Editions in your `.proto` files, Flink fully supports them:
304+
305+
```
306+
edition = "2023";
307+
package com.example;
308+
option java_package = "com.example";
309+
option java_multiple_files = true;
310+
311+
message SimpleTest {
312+
int64 uid = 1;
313+
string name = 2 [features.field_presence = EXPLICIT];
314+
// ... rest of your message definition
315+
}
316+
```
317+
318+
Editions allow fine-grained control over feature behavior at the file, message, or field level, while maintaining backward compatibility with proto2 and proto3. For more information, see the [Protobuf Editions documentation](https://protobuf.dev/editions/overview/).
319+
320+
Additional Resources
321+
----------------
322+
For more information about Protocol Buffers, refer to:
323+
- [Language Guide (proto2)](https://developers.google.com/protocol-buffers/docs/proto)
324+
- [Language Guide (proto3)](https://developers.google.com/protocol-buffers/docs/proto3)
325+
- [Language Guide (Editions)](https://protobuf.dev/programming-guides/editions/) - for the new Editions syntax
326+
- [Protobuf Editions Overview](https://protobuf.dev/editions/overview/) - understand the motivation and benefits of Editions

docs/content/release-notes/flink-2.1.md

Lines changed: 24 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -182,4 +182,27 @@ Bump flink-shaded version to 20.0 to support Smile format.
182182
##### [FLINK-37760](https://issues.apache.org/jira/browse/FLINK-37760)
183183

184184
Bump parquet version to 1.15.3 to resolve parquet-avro module
185-
vulnerability found in [CVE-2025-30065](https://nvd.nist.gov/vuln/detail/CVE-2025-30065).
185+
vulnerability found in [CVE-2025-30065](https://nvd.nist.gov/vuln/detail/CVE-2025-30065).
186+
187+
#### Upgrade Protocol Buffers to 4.32.1
188+
189+
##### [FLINK-38547](https://issues.apache.org/jira/browse/FLINK-38547)
190+
191+
Flink now uses protobuf-java 4.32.1 (corresponding to Protocol Buffers version 32), upgrading from
192+
protobuf-java 3.21.7 (Protocol Buffers version 21). This major upgrade enables:
193+
194+
- **Protobuf Editions Support**: Full support for the new `edition = "2023"` and `edition = "2024"`
195+
syntax introduced in Protocol Buffers v27+. Editions provide a unified approach that combines
196+
proto2 and proto3 functionality with fine-grained feature control.
197+
- **Improved Proto3 Field Presence**: Better handling of optional fields in proto3 without the
198+
limitations of older protobuf versions, eliminating the need to set `protobuf.read-default-values`
199+
to `true` for field presence checking.
200+
- **Enhanced Performance**: Leverages performance improvements and bug fixes from 11 Protocol
201+
Buffers releases (versions 22-32).
202+
- **Modern Protobuf Features**: Access to newer protobuf capabilities including Edition 2024
203+
features and improved runtime behavior.
204+
205+
Users with existing proto2 and proto3 `.proto` files will continue to work without changes. For
206+
those interested in adopting Protobuf Editions, see the updated
207+
[Protobuf format documentation](https://nightlies.apache.org/flink/flink-docs-release-2.1/docs/connectors/table/formats/protobuf/)
208+
for examples and guidance.

0 commit comments

Comments
 (0)