Skip to content

Parquet Converter should convert 2h block with no compact marker #6831

Open
@yeya24

Description

@yeya24

Is your feature request related to a problem? Please describe.
Currently, Parquet Converter will only convert blocks with certain size to optimize cost. For example, if the block ranges are configured to be [2h, 12h, 24h], we will not convert 2h block but start converting 12h blocks. The idea is that 2h block will be compacted to 12h so it is unnecessary to convert 2h blocks. User is expected to set query store after config properly to avoid querying 2h blocks in most recent time.

However, if a 2h block is marked as no compact then it will never be compacted to 12h, which means it never gets converted to Parquet and we are unable to query it via Parquet.

Describe the solution you'd like
We can change parquet converter to try to convert 2h block if no compact marker exists or convert 2h block if it is older than certain time (like 1 week).

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions