You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/content/docs/connectors/table/formats/protobuf.md
+37-5Lines changed: 37 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -151,10 +151,8 @@ Format Options
151
151
<td>
152
152
If this value is set to true, the format will read empty values as the default values defined in the proto file.
153
153
If the value is set to false, the format will generate null values if the data element does not exist in the binary protobuf message.
154
-
If proto syntax is proto3, users need to set this to true when using protobuf versions lower than 3.15 as older versions do not support
155
-
checking for field presence which can cause runtime compilation issues. Additionally, primtive types will be set to default values
156
-
instead of null as field presence cannot be checked for them. Please be aware that setting this to true will cause the deserialization
157
-
performance to be much slower depending on schema complexity and message size.
154
+
With Flink's current protobuf version (4.32.1), field presence is properly supported for proto3, allowing null handling for non-primitive types.
155
+
Please be aware that setting this to true will cause the deserialization performance to be much slower depending on schema complexity and message size.
158
156
</td>
159
157
</tr>
160
158
<tr>
@@ -291,4 +289,38 @@ OneOf field
291
289
In the serialization process, there's no guarantee that the Flink fields of the same one-of group only contain at most one valid value.
292
290
When serializing, each field is set in the order of Flink schema, so the field in the higher position will override the field in lower position in the same one-of group.
293
291
294
-
You can refer to [Language Guide (proto2)](https://developers.google.com/protocol-buffers/docs/proto) or [Language Guide (proto3)](https://developers.google.com/protocol-buffers/docs/proto3) for more information about Protobuf types.
292
+
Supported Protobuf Versions
293
+
------------
294
+
295
+
Flink uses protobuf-java 4.32.1 (corresponding to Protocol Buffers version 32), which includes support for:
296
+
297
+
-**Proto2 and Proto3 syntax**: Traditional `syntax = "proto2"` and `syntax = "proto3"` definitions
298
+
-**Protobuf Editions**: The new `edition = "2023"` and `edition = "2024"` syntax introduced in Protocol Buffers v27+
299
+
-**Improved proto3 field presence detection**: Better handling of optional fields without the limitations of older protobuf versions
300
+
301
+
### Using Protobuf Editions
302
+
303
+
Protobuf Editions provide a unified syntax that combines proto2 and proto3 functionality. If you're using Editions in your `.proto` files, Flink fully supports them:
304
+
305
+
```
306
+
edition = "2023";
307
+
package com.example;
308
+
option java_package = "com.example";
309
+
option java_multiple_files = true;
310
+
311
+
message SimpleTest {
312
+
int64 uid = 1;
313
+
string name = 2 [features.field_presence = EXPLICIT];
314
+
// ... rest of your message definition
315
+
}
316
+
```
317
+
318
+
Editions allow fine-grained control over feature behavior at the file, message, or field level, while maintaining backward compatibility with proto2 and proto3. For more information, see the [Protobuf Editions documentation](https://protobuf.dev/editions/overview/).
319
+
320
+
Additional Resources
321
+
----------------
322
+
For more information about Protocol Buffers, refer to:
0 commit comments