feat: new clickhouse operation table rollups

[n1ru4l]

Description

Introduces 9 new aggregation tables (3 for each original aggregation: minutely, daily, hourly).

Based on the business logic filters we can route q query to a dedicated source table and massively reduce the amount of data that must be read in order to satisfy the query.

This will also allow us to filter by graph_id and graph_version_id in the future.

For the daily granularity tables we now use UInt64 instead of UInt32 for total and total_ok values to give us more headroom as we are getting closer to hitting that.

New Table Overview

flowchart LR
  source[(default.operations)]

  subgraph minute[Minutely - bucket with toStartOfMinute - TTL 24 hours]
    minDetailMV[operations_v01_minutely_mv]
    minDetail[(operations_v01_minutely)]
    minTimestampMV[operations_v01_minutely_by_timestamp_mv]
    minTimestamp[(operations_v01_minutely_by_timestamp)]
    minClientMV[operations_v01_minutely_by_client_mv]
    minClient[(operations_v01_minutely_by_client)]
    minDetailMV --> minDetail
    minTimestampMV --> minTimestamp
    minClientMV --> minClient
  end

  subgraph hour[Hourly - bucket with toStartOfHour - TTL 30 days]
    hourDetailMV[operations_v01_hourly_mv]
    hourDetail[(operations_v01_hourly)]
    hourTimestampMV[operations_v01_hourly_by_timestamp_mv]
    hourTimestamp[(operations_v01_hourly_by_timestamp)]
    hourClientMV[operations_v01_hourly_by_client_mv]
    hourClient[(operations_v01_hourly_by_client)]
    hourDetailMV --> hourDetail
    hourTimestampMV --> hourTimestamp
    hourClientMV --> hourClient
  end

  subgraph day[Daily - bucket with toStartOfDay - TTL 1 year]
    dayDetailMV[operations_v01_daily_mv]
    dayDetail[(operations_v01_daily)]
    dayTimestampMV[operations_v01_daily_by_timestamp_mv]
    dayTimestamp[(operations_v01_daily_by_timestamp)]
    dayClientMV[operations_v01_daily_by_client_mv]
    dayClient[(operations_v01_daily_by_client)]
    dayDetailMV --> dayDetail
    dayTimestampMV --> dayTimestamp
    dayClientMV --> dayClient
  end

  source --> minDetailMV
  source --> minTimestampMV
  source --> minClientMV
  source --> hourDetailMV
  source --> hourTimestampMV
  source --> hourClientMV
  source --> dayDetailMV
  source --> dayTimestampMV
  source --> dayClientMV

Loading

[theguild-bot]

🚀 Snapshot Release (alpha)

The latest changes of this PR are available as alpha on npm (based on the declared changesets):

Package	Version	Info
hive	11.14.0-alpha-20260914170059-8b02cdb78613686524c8939307709579cca7e856	npm ↗︎ unpkg ↗︎

[github-actions]

🐋 This PR was built and pushed to the following Docker images:

Targets: build

Platforms: linux/amd64

Image Tags: 11.14.0-alpha-8b02cdb, 8b02cdb, 8b02cdb78613686524c8939307709579cca7e856