-
Notifications
You must be signed in to change notification settings - Fork 3
Add protobuf text format support #13
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 3 commits
fbaa52e
c70d3c3
88f0845
61d601b
ec3fdde
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,6 @@ | ||
| # API Reference | ||
|
|
||
| - [`protobuf`](./protobuf.md) — Core primitives: messages, enums, descriptors, extensions | ||
| - [`protobuf.txtpb`](./txtpb.md) — Protobuf text format (`.txtpb`) serialization | ||
| - [`protobuf.wkt`](./wkt.md) — Well-Known Types | ||
| - [`protobuf.plugin`](./plugin.md) — protoc plugin framework |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| # `protobuf.txtpb` | ||
|
|
||
| ::: protobuf.txtpb | ||
| options: | ||
| show_root_heading: false |
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -1,14 +1,14 @@ | ||||||
| # Serializing Messages | ||||||
|
|
||||||
| Messages can be serialized to and from two formats: **binary** and **JSON**. | ||||||
| Messages can be serialized to and from three formats: **binary**, **JSON**, and the **text format**. | ||||||
|
|
||||||
| As a general guide: use JSON when you need human-readable output or interoperability with non-Protobuf consumers. | ||||||
| As a general guide: use JSON when you need human-readable output or interoperability with non-Protobuf consumers, and the text format for debugging, tests, and config files (`.txtpb`). | ||||||
| Use binary for everything else; it is more compact, faster to parse, and more resilient to schema changes. | ||||||
| For example, you can rename a field in your `.proto` file and still parse binary data serialized with the previous version, because binary encoding uses field numbers rather than names. | ||||||
| JSON output uses field names, so a rename will break consumers unless you use the `json_name` option. | ||||||
| JSON and text output use field names, so a rename will break consumers (JSON consumers can be shielded with the `json_name` option). | ||||||
|
|
||||||
| JSON output follows the [Protobuf JSON specification](https://protobuf.dev/programming-guides/json/). | ||||||
| Both formats pass the conformance test suite, ensuring interoperability with implementations in other languages. | ||||||
| JSON output follows the [Protobuf JSON specification](https://protobuf.dev/programming-guides/json/), and text output follows the [text format specification](https://protobuf.dev/reference/protobuf/textformat-spec/). | ||||||
| All three formats pass the conformance test suite, ensuring interoperability with implementations in other languages. | ||||||
|
|
||||||
| ## Binary | ||||||
|
|
||||||
|
|
@@ -94,20 +94,78 @@ Set this to `True` to silently skip them: | |||||
| user = User.from_json(text, ignore_unknown_fields=True) | ||||||
| ``` | ||||||
|
|
||||||
| ## Text Format | ||||||
|
|
||||||
| The [text format](https://protobuf.dev/reference/protobuf/textformat-spec/) is a plain-text syntax mainly used for debugging, tests, and config files. | ||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
This follows on https://github.com/bufbuild/protobuf-py/pull/13/changes#r3548113471 and notice removing "mainly" quite intentionally. BTW what do you think about suggesting JSON instead of text format unless working with an existing system?
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Also how about moving this past Merging to be the last section |
||||||
| You can use it to read and write `.txtpb` files. | ||||||
|
|
||||||
| Unlike binary and JSON, text format serialization is exposed as free functions from `protobuf.txtpb`, not as methods on `Message`. | ||||||
| This keeps the less commonly used text format from adding new reserved attribute names to every generated message class. | ||||||
|
|
||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
Careful when Claude repeats prompts as docs ;) |
||||||
| ```python | ||||||
| from protobuf.txtpb import to_text, from_text | ||||||
|
|
||||||
| # Serialize | ||||||
| text: str = to_text(user) | ||||||
|
|
||||||
| # Deserialize | ||||||
| user = from_text(User, text) | ||||||
| ``` | ||||||
|
|
||||||
| Serializing the example message from the [tutorial](./tutorial.md) prints: | ||||||
|
|
||||||
| ```txtpb | ||||||
| first_name: "Alice" | ||||||
| last_name: "Smith" | ||||||
| active: true | ||||||
| locations: "NYC" | ||||||
| locations: "LDN" | ||||||
| projects { | ||||||
| key: "atlas" | ||||||
| value: "infra" | ||||||
| } | ||||||
| ``` | ||||||
|
|
||||||
| The output matches the canonical writer used by `google.protobuf.text_format` (the same style `protoc --decode` and other Protobuf implementations produce): two-space indentation, one field per line, and no colon before a message value's `{`. | ||||||
| Text written by `to_text` can be read by the buf CLI, `protoc`, and other implementations, and text they produce can be read by `from_text`. | ||||||
|
|
||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
This is true of at least binary too and isn't something we usually doc here I think |
||||||
| Unlike binary and JSON serialization, `to_text` does not validate legacy required fields: unset fields are simply omitted, so a partially initialized message can always be dumped for debugging. | ||||||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. calling this out as a difference for this library, but matching the upstream protobuf-es behavior (IIUC).
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I was thinking about the above diff and this seems to seal the deal that it's probably best to leave text format out from the top-level details, i.e. revert the diff on lines 3-11. I don't want users to think this is a real format that they need to consider alongside the others, it's really a "if you don't know you need it you don't even need to read this section" type of feature. Also of note is that JSON is also "a text format" and arguably much better and we need to make sure that's reflected in these docs |
||||||
|
|
||||||
| ### Options | ||||||
|
|
||||||
| `registry`: required to write and read [`google.protobuf.Any`](./well-known-types.md#any) fields in their expanded `[type.url] {...}` form, and extension fields (`[pkg.extension_name]`). | ||||||
| Without it, an Any is written as its raw `type_url`/`value` fields, and extensions are omitted from output and rejected on parse. | ||||||
|
|
||||||
| ```python | ||||||
| to_text(user, registry=registry) | ||||||
| user = from_text(User, text, registry=registry) | ||||||
| ``` | ||||||
|
|
||||||
| `print_unknown_fields`: set to `True` to print unknown fields by their field number. | ||||||
| This is a debugging aid only: `from_text` rejects fields addressed by number, so output that includes them cannot be parsed back. | ||||||
|
|
||||||
| ```python | ||||||
| to_text(user, print_unknown_fields=True) | ||||||
| ``` | ||||||
|
|
||||||
| ## Merging | ||||||
|
|
||||||
| Instead of creating a new message, you can parse data into an existing one. | ||||||
| This is useful for applying partial updates or combining data from multiple sources. | ||||||
|
|
||||||
| ```python | ||||||
| from protobuf import merge_from_binary, merge_from_json, merge_from | ||||||
| from protobuf.txtpb import merge_from_text | ||||||
|
|
||||||
| # Merge binary data into an existing message | ||||||
| merge_from_binary(user, data) | ||||||
|
|
||||||
| # Merge JSON into an existing message | ||||||
| merge_from_json(user, text) | ||||||
|
|
||||||
| # Merge text format into an existing message | ||||||
| merge_from_text(user, text) | ||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If moving the text format docs to the last section, I think we can remove these from this one and keep them all together under |
||||||
|
|
||||||
| # Merge one message into another of the same type | ||||||
| merge_from(target, source) | ||||||
| ``` | ||||||
|
|
||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -105,6 +105,11 @@ def is_zero_value(member: DescFieldValue | DescOneof, value: Any) -> bool: | |
| """ | ||
| if isinstance(member, DescFieldValueEnum): | ||
| return value == member.enum.values[0].number | ||
| if isinstance(member, DescFieldValueScalar) and member.scalar == ScalarType.FLOAT: | ||
| if isinstance(member, DescFieldValueScalar) and member.scalar in ( | ||
| ScalarType.FLOAT, | ||
| ScalarType.DOUBLE, | ||
| ): | ||
| # Negative zero is not the zero value: it is distinguishable from 0.0 | ||
| # and must survive serialization, matching C++ and protobuf-go. | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. can split this out if you'd rather land it separately.
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Doh, thanks! Can you add a test in test_float.py?
stefanvanburen marked this conversation as resolved.
Outdated
|
||
| return value == 0.0 and copysign(1, value) == 1 | ||
| return not bool(value) | ||
| Original file line number | Diff line number | Diff line change | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,106 @@ | ||||||||||||||
| # Copyright (c) 2025-2026 Buf Technologies, Inc. | ||||||||||||||
| # | ||||||||||||||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||||||||||||||
| # you may not use this file except in compliance with the License. | ||||||||||||||
| # You may obtain a copy of the License at | ||||||||||||||
| # | ||||||||||||||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||||||||||||||
| # | ||||||||||||||
| # Unless required by applicable law or agreed to in writing, software | ||||||||||||||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||||||||||||||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||||||||||||||
| # See the License for the specific language governing permissions and | ||||||||||||||
| # limitations under the License. | ||||||||||||||
| """Protobuf text format (`.txtpb`) serialization. | ||||||||||||||
|
|
||||||||||||||
| The [text format](https://protobuf.dev/reference/protobuf/textformat-spec/) | ||||||||||||||
| is a plain-text syntax mainly used for debugging, tests, and config files. | ||||||||||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||
|
|
||||||||||||||
| This functionality is kept in its own module, separate from `protobuf`, so | ||||||||||||||
| that using it never adds new reserved attribute names to `Message` subclasses. | ||||||||||||||
|
|
||||||||||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||
| Examples: | ||||||||||||||
| ```python | ||||||||||||||
| from protobuf.txtpb import from_text, to_text | ||||||||||||||
|
stefanvanburen marked this conversation as resolved.
Outdated
|
||||||||||||||
|
|
||||||||||||||
| text = to_text(user) | ||||||||||||||
| user = from_text(User, text) | ||||||||||||||
| ``` | ||||||||||||||
| """ | ||||||||||||||
|
|
||||||||||||||
| from __future__ import annotations | ||||||||||||||
|
|
||||||||||||||
| from typing import TYPE_CHECKING, TypeVar | ||||||||||||||
|
|
||||||||||||||
| from ._from_text import merge_from_text | ||||||||||||||
| from ._to_text import ToTextOptions, to_text as _to_text_impl | ||||||||||||||
|
|
||||||||||||||
| if TYPE_CHECKING: | ||||||||||||||
| from protobuf._message import Message | ||||||||||||||
| from protobuf._registry import Registry | ||||||||||||||
|
|
||||||||||||||
| T = TypeVar("T", bound="Message") | ||||||||||||||
|
|
||||||||||||||
| __all__ = ["from_text", "merge_from_text", "to_text"] | ||||||||||||||
|
|
||||||||||||||
|
|
||||||||||||||
| def to_text( | ||||||||||||||
| message: Message, | ||||||||||||||
| /, | ||||||||||||||
| *, | ||||||||||||||
| registry: Registry | None = None, | ||||||||||||||
| print_unknown_fields: bool = False, | ||||||||||||||
| ) -> str: | ||||||||||||||
| """Serialize a message to the protobuf text format. | ||||||||||||||
|
|
||||||||||||||
| The output matches the canonical `google.protobuf.text_format` writer: | ||||||||||||||
| two-space indentation, one field per line, no colon before a message | ||||||||||||||
| value's `{`, and a trailing newline. | ||||||||||||||
|
|
||||||||||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||
| Unset fields are omitted, including unset required fields; unlike | ||||||||||||||
| [`Message.to_binary`][] and [`Message.to_json`][], legacy required fields | ||||||||||||||
| are not validated when serializing to text. | ||||||||||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
Or such |
||||||||||||||
|
|
||||||||||||||
| Args: | ||||||||||||||
| message: The message to serialize. | ||||||||||||||
| registry: A registry for resolving google.protobuf.Any messages | ||||||||||||||
| and extensions. Without it, an Any is written as its raw | ||||||||||||||
| `type_url`/`value` fields and extensions are omitted. | ||||||||||||||
| print_unknown_fields: If `True`, unknown fields are printed by | ||||||||||||||
| field number. This is a debugging aid only: `from_text` | ||||||||||||||
| rejects fields named by number, so output that includes them | ||||||||||||||
| cannot be parsed back. | ||||||||||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||
|
|
||||||||||||||
| Returns: | ||||||||||||||
| The message in protobuf text format. | ||||||||||||||
| """ | ||||||||||||||
| return _to_text_impl( | ||||||||||||||
| message, | ||||||||||||||
| ToTextOptions(print_unknown_fields=print_unknown_fields, registry=registry), | ||||||||||||||
| ) | ||||||||||||||
|
|
||||||||||||||
|
|
||||||||||||||
| def from_text( | ||||||||||||||
| message_type: type[T], | ||||||||||||||
| text: str | bytes | bytearray, | ||||||||||||||
| *, | ||||||||||||||
| registry: Registry | None = None, | ||||||||||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. While it deviates from protobuf-es a bit, I feel it is worth having the |
||||||||||||||
| ) -> T: | ||||||||||||||
| """Create a new message by parsing the protobuf text format. | ||||||||||||||
|
|
||||||||||||||
| To merge into an existing message, use [`merge_from_text`][]. | ||||||||||||||
|
|
||||||||||||||
| Args: | ||||||||||||||
| message_type: The type of message to create. | ||||||||||||||
| text: A str, bytes, or bytearray instance containing the text | ||||||||||||||
| format. | ||||||||||||||
|
stefanvanburen marked this conversation as resolved.
Outdated
|
||||||||||||||
| registry: Required to read `google.protobuf.Any` in its expanded | ||||||||||||||
| `[type.url] {...}` form, and extension fields, from text | ||||||||||||||
| format. | ||||||||||||||
|
|
||||||||||||||
| Raises: | ||||||||||||||
| ValueError: If the text cannot be parsed into the message. | ||||||||||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "This includes when encountering an unknown field" ("unless ignore_unknown_fields is set" if applying the above) |
||||||||||||||
| """ | ||||||||||||||
| message = message_type() | ||||||||||||||
| merge_from_text(message, text, registry=registry) | ||||||||||||||
| return message | ||||||||||||||
Uh oh!
There was an error while loading. Please reload this page.