Skip to content

More bugs with Avro format support in BigQueryType writes #5642

@clairemcginty

Description

@clairemcginty

The new GenericRecord default format still has a few bugs:

  • Doesn't support doubly-nested optional record fields
  • Doesn't support Elitzur validation types -- the schema converter doesn't take the OverrideTypeProvider into account and tries to encode the record type instead of the underlying primitive
  • Timestamp representation works for Storage write API but not FileLoads API

tbh, there are enough bugs found so far that I think we should try to restore TableRow format at the default and make GenericRecord opt-in; the schema converter needs more flexibility. Maybe we can eventually switch to Magnolify for schema conversion or write our own.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions