Skip to content

feat: new clickhouse operation table rollups #8479

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
n1ru4l wants to merge 3 commits into from
Open

feat: new clickhouse operation table rollups #8479

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6b6915f
feat: new operation rollups using `quantilesTDigest` and dedicated vi…
n1ru4l Sep 14, 2026
fbf8947
use uint64 for daily total count to avoid future overflows; adjust gr…
n1ru4l Sep 14, 2026
3349470
opt-in migration
n1ru4l Sep 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions integration-tests/docker-compose.integration.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,10 @@ networks:
name: hive

services:
migrations:
environment:
CLICKHOUSE_OPERATIONS_V01_ROLLUPS: '1'

local_cdn:
image: node${NODE_VERSION_TAG}
working_dir: /app
Expand Down
4 changes: 3 additions & 1 deletion packages/migrations/.env.template
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,6 @@ POSTGRES_USER=postgres
POSTGRES_PASSWORD=postgres
POSTGRES_HOST=localhost
POSTGRES_PORT=5432
POSTGRES_DB=registry
POSTGRES_DB=registry

CLICKHOUSE_OPERATIONS_V01_ROLLUPS=1
144 changes: 144 additions & 0 deletions packages/migrations/src/clickhouse-actions/022-operation-v01-rollups.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,144 @@
import type { Action } from '../clickhouse';

type Granularity = 'daily' | 'hourly' | 'minutely';

const pickUInt = (aggregation: Granularity) => (aggregation === 'daily' ? 'UInt64' : 'UInt32');

const aggregateColumns = (granularity: Granularity) => `
total ${pickUInt(granularity)} CODEC(T64, ZSTD(1)),
total_ok ${pickUInt(granularity)} CODEC(T64, ZSTD(1)),
duration_avg AggregateFunction(avg, UInt64) CODEC(ZSTD(1)),
duration_quantiles AggregateFunction(quantilesTDigest(0.75, 0.9, 0.95, 0.99), UInt64) CODEC(ZSTD(1))
`;

const aggregateStates = (granularity: Granularity) => `
CAST(count() AS ${pickUInt(granularity)}) AS total,
CAST(sum(ok) AS ${pickUInt(granularity)}) AS total_ok,
avgState(duration) AS duration_avg,
quantilesTDigestState(0.75, 0.9, 0.95, 0.99)(duration) AS duration_quantiles
`;

const createRollups = async (
exec: (query: string) => Promise<void>,
granularity: Granularity,
bucket: 'toStartOfMinute' | 'toStartOfHour' | 'toStartOfDay',
partitionBy: string,
ttlInterval: string,
) => {
const table = `operations_v01_${granularity}`;

await exec(`
CREATE TABLE IF NOT EXISTS default.${table}

@jdolle jdolle Sep 15, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this be called _by_hash for consistency?

(
target LowCardinality(String) CODEC(ZSTD(1)),
graph_id LowCardinality(String) CODEC(ZSTD(1)),
timestamp DateTime('UTC') CODEC(DoubleDelta, LZ4),
hash String CODEC(ZSTD(1)),
client_name String CODEC(ZSTD(1)),
client_version String CODEC(ZSTD(1)),
graph_version_id String CODEC(ZSTD(1)),
${aggregateColumns(granularity)}
)
ENGINE = SummingMergeTree
PARTITION BY ${partitionBy}
PRIMARY KEY (target, graph_id, timestamp, hash)
ORDER BY (target, graph_id, timestamp, hash, client_name, client_version, graph_version_id)
TTL timestamp + INTERVAL ${ttlInterval}
SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1
`);

await exec(`
CREATE MATERIALIZED VIEW IF NOT EXISTS default.${table}_mv TO default.${table}
AS (
SELECT
target,
graph_id,
${bucket}(timestamp) AS timestamp,
hash,
client_name,
client_version,
graph_version_id,
${aggregateStates(granularity)}
FROM default.operations
GROUP BY target, graph_id, timestamp, hash, client_name, client_version, graph_version_id
)
`);

await exec(`
CREATE TABLE IF NOT EXISTS default.${table}_by_timestamp
(
target LowCardinality(String) CODEC(ZSTD(1)),
graph_id LowCardinality(String) CODEC(ZSTD(1)),
timestamp DateTime('UTC') CODEC(DoubleDelta, LZ4),
graph_version_id String CODEC(ZSTD(1)),
${aggregateColumns(granularity)}
)
ENGINE = SummingMergeTree
PARTITION BY ${partitionBy}
PRIMARY KEY (target, graph_id, timestamp)
ORDER BY (target, graph_id, timestamp, graph_version_id)
TTL timestamp + INTERVAL ${ttlInterval}
SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1
`);

await exec(`
CREATE MATERIALIZED VIEW IF NOT EXISTS default.${table}_by_timestamp_mv TO default.${table}_by_timestamp
AS (
SELECT
target,
graph_id,
graph_version_id,
${bucket}(timestamp) AS timestamp,
${aggregateStates(granularity)}
FROM default.operations
GROUP BY target, graph_id, timestamp, graph_version_id
)
`);

await exec(`
CREATE TABLE IF NOT EXISTS default.${table}_by_client
(
target LowCardinality(String) CODEC(ZSTD(1)),
graph_id LowCardinality(String) CODEC(ZSTD(1)),
timestamp DateTime('UTC') CODEC(DoubleDelta, LZ4),
client_name String CODEC(ZSTD(1)),
client_version String CODEC(ZSTD(1)),
graph_version_id String CODEC(ZSTD(1)),
${aggregateColumns(granularity)}
)
ENGINE = SummingMergeTree
PARTITION BY ${partitionBy}
PRIMARY KEY (target, graph_id, timestamp, client_name, client_version)

@jdolle jdolle Sep 15, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This table was under utilized in your benchmarks. I still think it's worth adding a hash after client_version for the insights page.

ORDER BY (target, graph_id, timestamp, client_name, client_version, graph_version_id)
TTL timestamp + INTERVAL ${ttlInterval}
SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1
`);

await exec(`
CREATE MATERIALIZED VIEW IF NOT EXISTS default.${table}_by_client_mv TO default.${table}_by_client
AS (
SELECT
target,
graph_id,
${bucket}(timestamp) AS timestamp,
client_name,
client_version,
graph_version_id,
${aggregateStates(granularity)}
FROM default.operations
GROUP BY target, graph_id, timestamp, client_name, client_version, graph_version_id
)
`);
};

export const action: Action = async exec => {
await exec(`
ALTER TABLE default.operations
ADD COLUMN IF NOT EXISTS graph_id LowCardinality(String) DEFAULT '' CODEC(ZSTD(1)) AFTER target,
ADD COLUMN IF NOT EXISTS graph_version_id String DEFAULT '' CODEC(ZSTD(1)) AFTER graph_id
`);

await createRollups(exec, 'minutely', 'toStartOfMinute', 'toStartOfHour(timestamp)', '24 HOUR');

@jdolle jdolle Sep 15, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even though it means storing a bit more (b/c ttl_only_drop_parts), I'd prefer the partition to be bigger here also. Maybe at least every 3 or 4 hours?

await createRollups(exec, 'hourly', 'toStartOfHour', 'toYYYYMMDD(timestamp)', '30 DAY');

@jdolle jdolle Sep 15, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

did we not want to do larger partitions?

await createRollups(exec, 'daily', 'toStartOfDay', 'toYYYYMM(timestamp)', '1 YEAR');
};
8 changes: 8 additions & 0 deletions packages/migrations/src/clickhouse.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ export type Action = (

export async function migrateClickHouse(
isClickHouseMigrator: boolean,
enableOperationsV01Rollups: boolean,
isHiveCloud: boolean,
hiveCloudEnvironment: 'prod' | 'staging' | 'dev' | null,
clickhouse: {
Expand Down Expand Up @@ -185,6 +186,7 @@ export async function migrateClickHouse(
import('./clickhouse-actions/019-metric-alert-target-daily-rollup'),
import('./clickhouse-actions/020-usage-coordinate-counts'),
import('./clickhouse-actions/021-usage-coordinate-errors'),
import('./clickhouse-actions/022-operation-v01-rollups'),
]);

async function actionRunner(action: Action, index: number) {
Expand All @@ -196,6 +198,12 @@ export async function migrateClickHouse(
return;
}

// Keep the v01 rollup tables opt-in without recording the migration as completed.
if (index === 21 && !enableOperationsV01Rollups) {
console.log(' Skipping because CLICKHOUSE_OPERATIONS_V01_ROLLUPS_START is not set');
return;
}

try {
await action(
async (query, settings) => {
Expand Down
4 changes: 4 additions & 0 deletions packages/migrations/src/environment.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,9 @@ const EnvironmentModel = zod.object({
RELEASE: emptyString(zod.string().optional()),
MIGRATOR: emptyString(zod.string().optional()),
CLICKHOUSE_MIGRATOR: emptyString(zod.string().optional()),
CLICKHOUSE_OPERATIONS_V01_ROLLUPS: emptyString(
zod.union([zod.literal('1'), zod.literal('0')]).optional(),
),
CLICKHOUSE_MIGRATOR_GRAPHQL_HIVE_CLOUD: zod
.union([zod.literal('1'), zod.literal('0')])
.optional(),
Expand Down Expand Up @@ -111,6 +114,7 @@ export const env = {
: null,
isMigrator: base.MIGRATOR === 'up',
isClickHouseMigrator: base.CLICKHOUSE_MIGRATOR === 'up',
enableOperationsV01Rollups: base.CLICKHOUSE_OPERATIONS_V01_ROLLUPS === '1',
isHiveCloud: base.CLICKHOUSE_MIGRATOR_GRAPHQL_HIVE_CLOUD === '1',
hiveCloudEnvironment: base.GRAPHQL_HIVE_ENVIRONMENT ?? null,
} as const;
1 change: 1 addition & 0 deletions packages/migrations/src/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ try {
if (env.clickhouse) {
await migrateClickHouse(
env.isClickHouseMigrator,
env.enableOperationsV01Rollups,
env.isHiveCloud,
env.hiveCloudEnvironment,
env.clickhouse,
Expand Down
Loading