Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,8 @@ Deserialization uses merge semantics: parsing into an existing message overwrite

ProtoJSON serialization and deserialization are complete, including the special encodings required for well-known types like `Timestamp`, `Any`, `Struct`, `Duration`, `FieldMask`, `Value`, and `ListValue`.

The [text format](https://protobuf.dev/reference/protobuf/textformat-spec/) is supported for debugging, tests, and config files (`.txtpb`), via free functions in `protobuf.txtpb` (`to_text`/`from_text`/`merge_from_text`) rather than `Message` methods — this is the one serialization format that is *not* exposed as a method, so that using it never expands the set of reserved attribute names on every generated message class. Output matches the canonical writer used by `google.protobuf.text_format` (two-space indentation, no colon before a message value's `{`); where the text-format spec otherwise leaves details open, behavior matches protobuf-go's `prototext` package, like protobuf-es. All three formats pass the upstream conformance suite.
Comment thread
stefanvanburen marked this conversation as resolved.
Outdated

## Type System

**Oneofs** are represented as `Oneof` dataclasses with a `field` and `value`, which works naturally with Python's `match` statement for type-safe dispatch:
Expand Down Expand Up @@ -216,7 +218,7 @@ in `validate.proto`, `message`, `field`, and `oneof`. As these names provide lit
**Messages support equality comparison.** `==` compares messages by value: two messages are equal when they have the same type, the same fields set, and the same field values. Extensions and unknown fields are not considered.

**Legacy required fields are validated on serialization, not on parsing.**
Fields with `features.field_presence = LEGACY_REQUIRED` (proto2 `required`) are checked when serializing to binary or ProtoJSON — an error is raised if the field is not set. When parsing from binary or ProtoJSON, missing required fields are silently accepted. This matches protobuf-go, protobuf-es, and the C++ implementation.
Fields with `features.field_presence = LEGACY_REQUIRED` (proto2 `required`) are checked when serializing to binary or ProtoJSON — an error is raised if the field is not set. When parsing from binary or ProtoJSON, missing required fields are silently accepted. This matches protobuf-go, protobuf-es, and the C++ implementation. The text format is the deliberate exception: `to_text` does not validate required fields and simply omits unset ones, like protobuf-es, so a partially initialized message can always be dumped for debugging.
Comment thread
stefanvanburen marked this conversation as resolved.
Outdated

### Escaping Python code

Expand Down
18 changes: 10 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -196,11 +196,13 @@ for field in user:

## Migration guide

| `google-protobuf` | `protobuf-py` |
| ------------------------------------------ | ------------------------------------- |
| `msg.SerializeToString()` | `msg.to_binary()` |
| `MessageType.FromString(data)` | `MessageType.from_binary(data)` |
| `msg.HasField("nickname")` | `msg.has_field("nickname")` |
| `msg.WhichOneof("result")` + string checks | `match msg.result` with typed `Oneof` |
| `msg.Extensions[ext]` | `msg[ext]` |
| `msg.child.CopyFrom(other)` | `msg.child = other` |
| `google-protobuf` | `protobuf-py` |
| ------------------------------------------ | --------------------------------------------- |
| `msg.SerializeToString()` | `msg.to_binary()` |
| `MessageType.FromString(data)` | `MessageType.from_binary(data)` |
| `text_format.MessageToString(msg)` | `protobuf.txtpb.to_text(msg)` |
| `text_format.Parse(text, msg)` | `protobuf.txtpb.from_text(MessageType, text)` |
Comment thread
stefanvanburen marked this conversation as resolved.
Outdated
| `msg.HasField("nickname")` | `msg.has_field("nickname")` |
| `msg.WhichOneof("result")` + string checks | `match msg.result` with typed `Oneof` |
| `msg.Extensions[ext]` | `msg[ext]` |
| `msg.child.CopyFrom(other)` | `msg.child = other` |
1 change: 1 addition & 0 deletions docs/api/index.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# API Reference

- [`protobuf`](./protobuf.md) — Core primitives: messages, enums, descriptors, extensions
- [`protobuf.txtpb`](./txtpb.md) — Protobuf text format (`.txtpb`) serialization
- [`protobuf.wkt`](./wkt.md) — Well-Known Types
- [`protobuf.plugin`](./plugin.md) — protoc plugin framework
5 changes: 5 additions & 0 deletions docs/api/txtpb.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# `protobuf.txtpb`

::: protobuf.txtpb
options:
show_root_heading: false
68 changes: 63 additions & 5 deletions docs/serialization.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
# Serializing Messages

Messages can be serialized to and from two formats: **binary** and **JSON**.
Messages can be serialized to and from three formats: **binary**, **JSON**, and the **text format**.

As a general guide: use JSON when you need human-readable output or interoperability with non-Protobuf consumers.
As a general guide: use JSON when you need human-readable output or interoperability with non-Protobuf consumers, and the text format for debugging, tests, and config files (`.txtpb`).
Use binary for everything else; it is more compact, faster to parse, and more resilient to schema changes.
For example, you can rename a field in your `.proto` file and still parse binary data serialized with the previous version, because binary encoding uses field numbers rather than names.
JSON output uses field names, so a rename will break consumers unless you use the `json_name` option.
JSON and text output use field names, so a rename will break consumers (JSON consumers can be shielded with the `json_name` option).

JSON output follows the [Protobuf JSON specification](https://protobuf.dev/programming-guides/json/).
Both formats pass the conformance test suite, ensuring interoperability with implementations in other languages.
JSON output follows the [Protobuf JSON specification](https://protobuf.dev/programming-guides/json/), and text output follows the [text format specification](https://protobuf.dev/reference/protobuf/textformat-spec/).
All three formats pass the conformance test suite, ensuring interoperability with implementations in other languages.

## Binary

Expand Down Expand Up @@ -94,20 +94,78 @@ Set this to `True` to silently skip them:
user = User.from_json(text, ignore_unknown_fields=True)
```

## Text Format

The [text format](https://protobuf.dev/reference/protobuf/textformat-spec/) is a plain-text syntax mainly used for debugging, tests, and config files.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The [text format](https://protobuf.dev/reference/protobuf/textformat-spec/) is a plain-text syntax mainly used for debugging, tests, and config files.
In addition to binary, and JSON, Protobuf also provides, [text format](https://protobuf.dev/reference/protobuf/textformat-spec/), a plain-text syntax used for debugging, tests, and config files.

This follows on https://github.com/bufbuild/protobuf-py/pull/13/changes#r3548113471 and notice removing "mainly" quite intentionally.

BTW what do you think about suggesting JSON instead of text format unless working with an existing system?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also how about moving this past Merging to be the last section

You can use it to read and write `.txtpb` files.

Unlike binary and JSON, text format serialization is exposed as free functions from `protobuf.txtpb`, not as methods on `Message`.
This keeps the less commonly used text format from adding new reserved attribute names to every generated message class.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Unlike binary and JSON, text format serialization is exposed as free functions from `protobuf.txtpb`, not as methods on `Message`.
This keeps the less commonly used text format from adding new reserved attribute names to every generated message class.

Careful when Claude repeats prompts as docs ;)

```python
from protobuf.txtpb import to_text, from_text

# Serialize
text: str = to_text(user)

# Deserialize
user = from_text(User, text)
```

Serializing the example message from the [tutorial](./tutorial.md) prints:

```txtpb
first_name: "Alice"
last_name: "Smith"
active: true
locations: "NYC"
locations: "LDN"
projects {
key: "atlas"
value: "infra"
}
```

The output matches the canonical writer used by `google.protobuf.text_format` (the same style `protoc --decode` and other Protobuf implementations produce): two-space indentation, one field per line, and no colon before a message value's `{`.
Text written by `to_text` can be read by the buf CLI, `protoc`, and other implementations, and text they produce can be read by `from_text`.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The output matches the canonical writer used by `google.protobuf.text_format` (the same style `protoc --decode` and other Protobuf implementations produce): two-space indentation, one field per line, and no colon before a message value's `{`.
Text written by `to_text` can be read by the buf CLI, `protoc`, and other implementations, and text they produce can be read by `from_text`.

This is true of at least binary too and isn't something we usually doc here I think

Unlike binary and JSON serialization, `to_text` does not validate legacy required fields: unset fields are simply omitted, so a partially initialized message can always be dumped for debugging.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

calling this out as a difference for this library, but matching the upstream protobuf-es behavior (IIUC).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking about the above diff and this seems to seal the deal that it's probably best to leave text format out from the top-level details, i.e. revert the diff on lines 3-11. I don't want users to think this is a real format that they need to consider alongside the others, it's really a "if you don't know you need it you don't even need to read this section" type of feature. Also of note is that JSON is also "a text format" and arguably much better and we need to make sure that's reflected in these docs


### Options

`registry`: required to write and read [`google.protobuf.Any`](./well-known-types.md#any) fields in their expanded `[type.url] {...}` form, and extension fields (`[pkg.extension_name]`).
Without it, an Any is written as its raw `type_url`/`value` fields, and extensions are omitted from output and rejected on parse.

```python
to_text(user, registry=registry)
user = from_text(User, text, registry=registry)
```

`print_unknown_fields`: set to `True` to print unknown fields by their field number.
This is a debugging aid only: `from_text` rejects fields addressed by number, so output that includes them cannot be parsed back.

```python
to_text(user, print_unknown_fields=True)
```

## Merging

Instead of creating a new message, you can parse data into an existing one.
This is useful for applying partial updates or combining data from multiple sources.

```python
from protobuf import merge_from_binary, merge_from_json, merge_from
from protobuf.txtpb import merge_from_text

# Merge binary data into an existing message
merge_from_binary(user, data)

# Merge JSON into an existing message
merge_from_json(user, text)

# Merge text format into an existing message
merge_from_text(user, text)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If moving the text format docs to the last section, I think we can remove these from this one and keep them all together under Text Format


# Merge one message into another of the same type
merge_from(target, source)
```
Expand Down
7 changes: 6 additions & 1 deletion src/protobuf/_field_values.py
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,11 @@ def is_zero_value(member: DescFieldValue | DescOneof, value: Any) -> bool:
"""
if isinstance(member, DescFieldValueEnum):
return value == member.enum.values[0].number
if isinstance(member, DescFieldValueScalar) and member.scalar == ScalarType.FLOAT:
if isinstance(member, DescFieldValueScalar) and member.scalar in (
ScalarType.FLOAT,
ScalarType.DOUBLE,
):
# Negative zero is not the zero value: it is distinguishable from 0.0
# and must survive serialization, matching C++ and protobuf-go.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can split this out if you'd rather land it separately.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doh, thanks! Can you add a test in test_float.py?

Comment thread
stefanvanburen marked this conversation as resolved.
Outdated
return value == 0.0 and copysign(1, value) == 1
return not bool(value)
106 changes: 106 additions & 0 deletions src/protobuf/txtpb/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
# Copyright (c) 2025-2026 Buf Technologies, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Protobuf text format (`.txtpb`) serialization.

The [text format](https://protobuf.dev/reference/protobuf/textformat-spec/)
is a plain-text syntax mainly used for debugging, tests, and config files.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
is a plain-text syntax mainly used for debugging, tests, and config files.
is a plain-text syntax used for debugging, tests, and config files.


This functionality is kept in its own module, separate from `protobuf`, so
that using it never adds new reserved attribute names to `Message` subclasses.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
This functionality is kept in its own module, separate from `protobuf`, so
that using it never adds new reserved attribute names to `Message` subclasses.

Examples:
```python
from protobuf.txtpb import from_text, to_text
Comment thread
stefanvanburen marked this conversation as resolved.
Outdated

text = to_text(user)
user = from_text(User, text)
```
"""

from __future__ import annotations

from typing import TYPE_CHECKING, TypeVar

from ._from_text import merge_from_text
from ._to_text import ToTextOptions, to_text as _to_text_impl

if TYPE_CHECKING:
from protobuf._message import Message
from protobuf._registry import Registry

T = TypeVar("T", bound="Message")

__all__ = ["from_text", "merge_from_text", "to_text"]


def to_text(
message: Message,
/,
*,
registry: Registry | None = None,
print_unknown_fields: bool = False,
) -> str:
"""Serialize a message to the protobuf text format.

The output matches the canonical `google.protobuf.text_format` writer:
two-space indentation, one field per line, no colon before a message
value's `{`, and a trailing newline.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The output matches the canonical `google.protobuf.text_format` writer:
two-space indentation, one field per line, no colon before a message
value's `{`, and a trailing newline.

Unset fields are omitted, including unset required fields; unlike
[`Message.to_binary`][] and [`Message.to_json`][], legacy required fields
are not validated when serializing to text.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Unset fields are omitted, including unset required fields; unlike
[`Message.to_binary`][] and [`Message.to_json`][], legacy required fields
are not validated when serializing to text.
Unlike standard serialization, unset required fields will not raise an error.

Or such


Args:
message: The message to serialize.
registry: A registry for resolving google.protobuf.Any messages
and extensions. Without it, an Any is written as its raw
`type_url`/`value` fields and extensions are omitted.
print_unknown_fields: If `True`, unknown fields are printed by
field number. This is a debugging aid only: `from_text`
rejects fields named by number, so output that includes them
cannot be parsed back.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
print_unknown_fields: If `True`, unknown fields are printed by
field number. This is a debugging aid only: `from_text`
rejects fields named by number, so output that includes them
cannot be parsed back.
print_unknown_fields: If `True`, unknown fields are printed by
field number.


Returns:
The message in protobuf text format.
"""
return _to_text_impl(
message,
ToTextOptions(print_unknown_fields=print_unknown_fields, registry=registry),
)


def from_text(
message_type: type[T],
text: str | bytes | bytearray,
*,
registry: Registry | None = None,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While it deviates from protobuf-es a bit, I feel it is worth having the ignore_unknown_fields option here (default to False) to have symmetry between the two APIs, and it looks like upstream Python and Go both have the option.

) -> T:
"""Create a new message by parsing the protobuf text format.

To merge into an existing message, use [`merge_from_text`][].

Args:
message_type: The type of message to create.
text: A str, bytes, or bytearray instance containing the text
format.
Comment thread
stefanvanburen marked this conversation as resolved.
Outdated
registry: Required to read `google.protobuf.Any` in its expanded
`[type.url] {...}` form, and extension fields, from text
format.

Raises:
ValueError: If the text cannot be parsed into the message.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"This includes when encountering an unknown field" ("unless ignore_unknown_fields is set" if applying the above)

"""
message = message_type()
merge_from_text(message, text, registry=registry)
return message
Loading
Loading