feat: auto alter table during trace ingestion from int to float

[shuiyisong]

I hereby agree to the terms of the GreptimeDB CLA.

Refer to a related PR or issue link (optional)

#7860

What's changed and what's your intention?

Note: merge #7870 first

This is the second PR for solving issue 7860:

• When the existing table's datatype is Int64 and the incoming datatype is Float64, issue an alter table request to change the datatype from Int64 to Float64

Note: concurrent update should work but have not yet been tested

PR Checklist

Please convert it to a draft if some of the following conditions are not met.

• I have written the necessary rustdoc comments.
• I have added the necessary unit tests and integration tests.
• This PR requires documentation updates.
• API changes are backward compatible.
• Schema or data changes are backward compatible.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly improves the robustness of OTLP trace ingestion by introducing dynamic type handling. It enables the system to automatically adapt table schemas to accommodate varying data types from incoming traces, specifically by widening integer columns to floating-point when necessary and performing other compatible type coercions. This prevents ingestion failures and ensures data integrity, making the trace ingestion process more resilient to schema mismatches.

Highlights

• Automatic Type Widening (Int64 to Float64): Implemented automatic schema evolution during OTLP trace ingestion. If an existing table column is Int64 and incoming data for that column is Float64, the system will automatically issue an ALTER TABLE request to change the column's type to Float64 to prevent data loss.
• In-Batch Type Coercion: Added support for various type coercions within a single batch of OTLP trace data. This includes Int64 to Float64, String to Int64, String to Float64, and String to Boolean conversions, ensuring compatibility with existing column types.
• Robust Error Handling for Coercion: Introduced error handling for unparsable string values during type coercion, which will now result in a BAD_REQUEST status, preventing corrupted data from being ingested.
• Enhanced TableData Utilities: Extended the TableData structure with new methods (column_datatype and upgrade_column_to_f64) to facilitate type checking and retroactive column type upgrades, particularly for Int64 to Float64 conversions.
• Comprehensive Integration Tests: Added new integration tests to thoroughly validate the type coercion and automatic table alteration functionalities for OTLP trace ingestion, covering both in-batch and cross-batch scenarios.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces comprehensive type coercion and widening capabilities for OTLP trace data ingestion. It enables in-batch type conversions for compatible types (e.g., Int64 to Float64, String to numeric/boolean) and supports cross-batch column widening from Int64 to Float64 by issuing ALTER TABLE statements. New utility functions and integration tests are included to support these features. A review comment highlights a potential data loss or inconsistency issue in the coerce_trace_numeric_columns function due to incorrect handling of missing values and short rows, suggesting a fix to ensure all rows are processed and padded correctly.

[discord9]

/gemini review

[gemini-code-assist]

Code Review

This pull request implements automatic column widening for OTLP traces, specifically allowing existing Int64 columns to be widened to Float64 when incoming data contains floating-point values. It introduces a reconciliation decision mechanism to determine whether to use existing types, request-local types, or perform an alter operation. The ingestion logic was updated to collect and execute these schema changes before processing row data. Feedback was provided regarding the error handling in the new alter_trace_table_columns_to_float64 method, suggesting that the original error's status code and context should be preserved rather than being wrapped in a generic InternalSnafu.

[fengys1996]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8121d784aa

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".