Skip to content

[Feature Request]: Add table information to BigQueryStorageApiInsertError #36831

@hekk-kaori-maeda

Description

@hekk-kaori-maeda

What would you like to happen?

What is the problem?

When using BigQuery Storage Write API (STORAGE_WRITE_API method) and write operations fail, the BigQueryStorageApiInsertError object does not contain information about which
table the error occurred on. This makes it difficult for users to identify and troubleshoot errors, especially when writing to multiple tables.

Current behavior:

  • BigQueryStorageApiInsertError only contains the row data and error message
  • Users cannot determine which table caused the error without additional logging

Expected behavior:

  • BigQueryStorageApiInsertError should include table identification (project, dataset, table)
  • API should be consistent with BigQueryInsertError (used by STREAMING_INSERTS method), which provides table information via TableReference

Proposed solution:

Add the following to BigQueryStorageApiInsertError:

  • tableUrn field (format: projects/{project}/datasets/{dataset}/tables/{table})
  • Convenience methods: getProjectId(), getDatasetId(), getTableId()

Use case:
This is particularly useful for:

  • Error monitoring and alerting systems
  • Debugging write failures in multi-table pipelines
  • Logging and auditing

Related:

  • Consistent with existing BigQueryInsertError API
  • Uses standard TableDestination.getTableUrn() format

Additional context:
I'm willing to contribute a PR for this enhancement.

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam YAML
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Infrastructure
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner

Metadata

Metadata

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions