Skip to content

Decode bytes to strings in Python Object API #8551

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Decode bytes to strings in Python Object API #8551

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e6f3a6d
Decode bytes to strings in Python Object API
sethraymond Mar 5, 2025
f47dce8
Merge branch 'master' into feature/python-obj-api-decode-strings
aardappel Apr 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions include/flatbuffers/idl.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -710,6 +710,7 @@ struct IDLOptions {
/********************************** Python **********************************/
bool python_no_type_prefix_suffix;
bool python_typing;
bool python_decode_obj_api_strings=false;

// The target Python version. Can be one of the following:
// - "0"
Expand Down
2 changes: 1 addition & 1 deletion scripts/generate_code.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ def glob(path, pattern):
KOTLIN_OPTS = ["--kotlin"]
PHP_OPTS = ["--php"]
DART_OPTS = ["--dart"]
PYTHON_OPTS = ["--python", "--python-typing"]
PYTHON_OPTS = ["--python", "--python-typing", "--python-decode-obj-api-string"]
BINARY_OPTS = ["-b", "--schema", "--bfbs-comments", "--bfbs-builtins"]
PROTO_OPTS = ["--proto"]

Expand Down
3 changes: 3 additions & 0 deletions src/flatc.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -256,6 +256,7 @@ const static FlatCOption flatc_options[] = {
"Skip emission of Python functions that are prefixed with typenames" },
{ "", "python-typing", "", "Generate Python type annotations" },
{ "", "python-version", "", "Generate code for the given Python version." },
{ "", "python-decode-obj-api-strings", "", "Decode bytes to strings for the Python Object API"},
{ "", "python-gen-numpy", "", "Whether to generate numpy helpers." },
{ "", "ts-omit-entrypoint", "",
"Omit emission of namespace entrypoint file" },
Expand Down Expand Up @@ -682,6 +683,8 @@ FlatCOptions FlatCompiler::ParseFromCommandLineArguments(int argc,
} else if (arg == "--python-version") {
if (++argi >= argc) Error("missing value following: " + arg, true);
opts.python_version = argv[argi];
} else if (arg == "--python-decode-obj-api-strings") {
opts.python_decode_obj_api_strings = true;
} else if (arg == "--python-gen-numpy" ||
arg == "--python-gen-numpy=true") {
opts.python_gen_numpy = true;
Expand Down
22 changes: 22 additions & 0 deletions src/idl_gen_python.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2028,6 +2028,20 @@ class PythonGenerator : public BaseGenerator {
}
}

void GenUnPackForString(const StructDef &struct_def, const FieldDef &field,
std::string *code_ptr) const {
auto &code = *code_ptr;
const auto field_field = namer_.Field(field);
const auto field_method = namer_.Method(field);
const auto struct_var = namer_.Variable(struct_def);

code += GenIndents(2) + "self." + field_field + " = " + struct_var + "." +
field_method + "()";
code += GenIndents(2) + "if self." + field_field + " is not None:";
code += GenIndents(3) + "self." + field_field + " = self." + field_field +
".decode('utf-8')";
}

void GenUnPackForScalar(const StructDef &struct_def, const FieldDef &field,
std::string *code_ptr) const {
auto &code = *code_ptr;
Expand Down Expand Up @@ -2070,6 +2084,14 @@ class PythonGenerator : public BaseGenerator {
}
break;
}
case BASE_TYPE_STRING: {
if (parser_.opts.python_decode_obj_api_strings) {
GenUnPackForString(struct_def, field, &code);
} else {
GenUnPackForScalar(struct_def, field, &code);
}
break;
}
default: GenUnPackForScalar(struct_def, field, &code);
}
}
Expand Down
2 changes: 2 additions & 0 deletions tests/MyGame/Example/NestedUnion/NestedUnionTest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,8 @@ def _UnPack(self, nestedUnionTest):
if nestedUnionTest is None:
return
self.name = nestedUnionTest.Name()
if self.name is not None:
self.name = self.name.decode('utf-8')
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this meant to be self.name = nestedUnionTest.name.decode('utf-8') ?

Copy link
Author

@sethraymond sethraymond Apr 23, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nestedUnionTest is not an object API type, so the only way to access the name from that object is via nestedUnionTest.Name().

This change is intentional, but if you're asking for a modification to the auto-generated code, I could change the autogeneration here to be

code += GetIndents(3) + "self." + field_field + " = " + struct_var + "." + field_method + "().decode('utf-8')";

Then this would read

self.name = nestedUnionTest.Name()
if self.name is not None:
    self.name = nestedUnionTest.Name().decode('utf-8')

It's functionally the same, though, since self.name is set to nestedUnionTest.name() a couple lines beforehand. I personally think the current implementation is a bit more readable, but I'm open to suggestions.

self.dataType = nestedUnionTest.DataType()
self.data = MyGame.Example.NestedUnion.Any.AnyCreator(self.dataType, nestedUnionTest.Data())
self.id = nestedUnionTest.Id()
Expand Down
Loading