Skip to content

[ENHANCEMENT] Clickhouse: support label and time access in table view #571

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
fraser-qs wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[ENHANCEMENT] Clickhouse: support label and time access in table view #571

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 16 additions & 18 deletions clickhouse/src/components/constants.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,22 +11,20 @@
// See the License for the specific language governing permissions and
// limitations under the License.

export const queryExample = `-- Time Series Query
SELECT
toStartOfMinute(timestamp) as time,
avg(cpu_usage) as avg_cpu,
max(memory_usage) as max_memory
FROM system_metrics
WHERE timestamp BETWEEN '{start}' AND '{end}'
export const queryExample = `-- Time Series Query (must alias to 'time' and 'value')
-- Available macros: $__timeFrom, $__timeTo, $__interval, $__interval_ms
SELECT
toStartOfInterval(timestamp, INTERVAL $__interval second) as time,
avg(cpu_usage) as value
FROM system_metrics
WHERE timestamp BETWEEN '$__timeFrom' AND '$__timeTo'
GROUP BY time ORDER BY time
-- Logs Query
SELECT
Timestamp as log_time,
Body,
ServiceName,
ResourceAttributes,
SeverityNumber,
SeverityText
FROM application_logs
WHERE timestamp >= '{start}'
ORDER BY time DESC LIMIT 1000`;

-- With labels (extra columns become labels)
SELECT
toStartOfInterval(timestamp, INTERVAL $__interval second) as time,
count(*) as value,
service_name
FROM logs
WHERE timestamp BETWEEN '$__timeFrom' AND '$__timeTo'
GROUP BY time, service_name ORDER BY time`;
13 changes: 4 additions & 9 deletions clickhouse/src/datasources/click-house-datasource/click-house-datasource-types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,18 +13,13 @@

import { HTTPProxy, RequestHeaders } from '@perses-dev/core';
import { DatasourceClient } from '@perses-dev/plugin-system';
import { ClickHouseQueryParams } from '../../model/click-house-client';

export interface ClickHouseDatasourceSpec {
directUrl?: string;
proxy?: HTTPProxy;
}

interface QueryRequestParameters extends Record<string, string> {
query: string;
start: string;
end: string;
}

interface ClickHouseDatasourceClientOptions {
datasourceUrl: string;
headers?: RequestHeaders;
Expand All @@ -33,11 +28,11 @@ interface ClickHouseDatasourceClientOptions {
export interface ClickHouseDatasourceResponse {
status: string;
warnings?: string[];
// TODO: adjust this type to match your datasource response shape
data: unknown;
// eslint-disable-next-line @typescript-eslint/no-explicit-any
data: any;
}

export interface ClickHouseDatasourceClient extends DatasourceClient {
options: ClickHouseDatasourceClientOptions;
query(params: QueryRequestParameters, headers?: RequestHeaders): Promise<ClickHouseDatasourceResponse>;
query(params: ClickHouseQueryParams, headers?: RequestHeaders): Promise<ClickHouseDatasourceResponse>;
}
5 changes: 3 additions & 2 deletions clickhouse/src/model/click-house-client.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,11 +25,12 @@ export interface ClickHouseQueryOptions {

export interface ClickHouseQueryResponse {
status: 'success' | 'error';
data: unknown;
// eslint-disable-next-line @typescript-eslint/no-explicit-any
data: any;
}

export interface ClickHouseClient {
query: (params: { start: string; end: string; query: string }) => Promise<ClickHouseQueryResponse>;
query: (params: ClickHouseQueryParams) => Promise<ClickHouseQueryResponse>;
}

export async function query(
Expand Down
11 changes: 0 additions & 11 deletions clickhouse/src/model/click-house-data-types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,14 +10,3 @@
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

import { LogData, TimeSeriesData } from '@perses-dev/core';

export interface ClickHouseTimeSeriesData extends TimeSeriesData {
logs?: LogData;
}

export interface TimeSeriesEntry {
time: string;
log_count: number | string;
}
14 changes: 9 additions & 5 deletions clickhouse/src/queries/click-house-log-query/get-click-house-log-data.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
import { DEFAULT_DATASOURCE } from '../constants';
import { ClickHouseLogQuerySpec } from './click-house-log-query-types';
import { LogQueryPlugin } from './log-query-plugin-interface';
import { replaceClickHouseBuiltinVariables } from '../replace-click-house-builtin-variables';

Check failure on line 20 in clickhouse/src/queries/click-house-log-query/get-click-house-log-data.ts

View workflow job for this annotation

GitHub Actions / lint-npm 

`../replace-click-house-builtin-variables` import should occur before import of `./click-house-log-query-types`

function flattenObject(
obj: Record<string, unknown>,
Expand Down Expand Up @@ -76,17 +77,20 @@
};
}

const query = replaceVariables(spec.query, context.variableState);
const { start, end } = context.timeRange;

// Default step for log queries (60 seconds)
const stepMs = 60 * 1000;
Comment on lines +82 to +83
Copy link
Contributor

@shahrokni shahrokni Feb 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Name convention?
STEP_MS


// Replace built-in variables first, then user-defined variables
let query = replaceClickHouseBuiltinVariables(spec.query, start, end, stepMs);
query = replaceVariables(query, context.variableState);

const client = (await context.datasourceStore.getDatasourceClient(
spec.datasource ?? DEFAULT_DATASOURCE
)) as ClickHouseClient;

const { start, end } = context.timeRange;

const response: ClickHouseQueryResponse = await client.query({
start: start.getTime().toString(),
end: end.getTime().toString(),
query,
});

Expand Down
116 changes: 94 additions & 22 deletions clickhouse/src/queries/click-house-time-series-query/get-click-house-data.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,31 +11,102 @@
// See the License for the specific language governing permissions and
// limitations under the License.

import { TimeSeries } from '@perses-dev/core';
import { Labels, TimeSeries } from '@perses-dev/core';
import { TimeSeriesQueryPlugin, replaceVariables } from '@perses-dev/plugin-system';
import { ClickHouseTimeSeriesQuerySpec } from './click-house-query-types';

Check failure on line 16 in clickhouse/src/queries/click-house-time-series-query/get-click-house-data.ts

View workflow job for this annotation

GitHub Actions / lint-npm 

`./click-house-query-types` import should occur after import of `../replace-click-house-builtin-variables`
import { DEFAULT_DATASOURCE } from '../constants';
import { TimeSeriesEntry } from '../../model/click-house-data-types';
import { ClickHouseClient, ClickHouseQueryResponse } from '../../model/click-house-client';
import { ClickHouseTimeSeriesQuerySpec, DatasourceQueryResponse } from './click-house-query-types';
import { replaceClickHouseBuiltinVariables } from '../replace-click-house-builtin-variables';

function buildTimeSeries(response?: DatasourceQueryResponse): TimeSeries[] {
const data = response?.data as TimeSeriesEntry[];
if (!response || !data || data.length === 0) {
// Default minimum step in milliseconds (15 seconds)
const DEFAULT_MIN_STEP_MS = 15 * 1000;

// Calculate step based on time range, aiming for ~1000 data points
function calculateStep(start: Date, end: Date, suggestedStepMs?: number): number {
const rangeMs = end.getTime() - start.getTime();
const calculatedStep = Math.ceil(rangeMs / 1000);
const step = Math.max(calculatedStep, DEFAULT_MIN_STEP_MS);
return suggestedStepMs ? Math.max(step, suggestedStepMs) : step;
}

// Fixed column names - queries must alias their columns to these names
const TIMESTAMP_COLUMN = 'time';
const VALUE_COLUMN = 'value';

// Build labels from all columns except timestamp and value
function buildLabels(row: Record<string, unknown>, timestampCol: string, valueCol: string): Labels {
const labels: Labels = {};
for (const [key, value] of Object.entries(row)) {
if (key !== timestampCol && key !== valueCol) {
labels[key] = value === null || value === undefined ? '' : String(value);
}
}
return labels;
}
Comment on lines +37 to +45
Copy link
Contributor

@shahrokni shahrokni Feb 18, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Object.entries(row) creates a new intermediate array of [key, value] pairs for every single row. For a dashboard with 1,000+ points and multiple series, this adds unnecessary memory pressure and garbage collection. Iterate over keys directly to avoid creating extra arrays.

ClickHouse queries often return internal columns (like _part or _sample_factor) or extra metadata that you might not want as labels in your Perses dashboard.

The current code converts everything to a string. While safe, it can be slightly optimized by checking if the value is already a string before calling String(value)

function buildLabels(row: Record<string, unknown>, timestampCol: string, valueCol: string): Labels {
  const labels: Labels = {};
  

  const keys = Object.keys(row);
  
  for (let i = 0; i < keys.length; i++) {
    const key = keys[i];
    

    if (key === timestampCol || key === valueCol || key.startsWith('_')) {
      continue;
    }

    const val = row[key];
    

    if (val === null || val === undefined) {
      labels[key] = '';
    } else {
      labels[key] = typeof val === 'string' ? val : String(val);
    }
  }
  
  return labels;
}


// Create a unique key from labels for grouping
function labelsToKey(labels: Labels): string {
return Object.entries(labels)
.sort(([a], [b]) => a.localeCompare(b))
.map(([k, v]) => `${k}=${v}`)
.join(',');
}
Comment on lines +48 to +53
Copy link
Contributor

@shahrokni shahrokni Feb 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Each time this function runs, it creates multiple intermediate arrays (Object.entries, .sort, .map) and several temporary strings before producing the final key. If your ClickHouse query returns thousands of rows, this can lead to significant garbage collection (GC) pressure and slower dashboard rendering.

function labelsToKey(labels: Labels): string {
  const keys = Object.keys(labels).sort();
  let keyString = '';

  for (let i = 0; i < keys.length; i++) {
    const k = keys[i];
    const v = labels[k];
    
   
    if (i > 0) keyString += ',';
    keyString += k + '=' + v;
  }
  
  return keyString;
}


// Create a display name from labels
function labelsToName(labels: Labels): string {
const entries = Object.entries(labels);
if (entries.length === 0) return 'series';
if (entries.length === 1) return entries[0]?.[1] ?? 'series';
return entries.map(([k, v]) => `${k}="${v}"`).join(', ');
Comment on lines +59 to +60
Copy link
Contributor

@shahrokni shahrokni Feb 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The function does not sort the labels. If ClickHouse returns columns in a different order for different rows, your legend items might look inconsistent.

The entries[0]?.[1] is a bit redundant since the code already checked that length === 1

}

function buildTimeSeries(response: ClickHouseQueryResponse | undefined): TimeSeries[] {
if (!response || !response.data || response.data.length === 0) {
return [];
}

const values: Array<[number, number]> = data.map((row: TimeSeriesEntry) => {
const timestamp = new Date(row.time).getTime();
const value = Number(row.log_count);
return [timestamp, value];
});
const data = response.data as Record<string, unknown>[];

Check failure on line 68 in clickhouse/src/queries/click-house-time-series-query/get-click-house-data.ts

View workflow job for this annotation

GitHub Actions / lint-npm 

Array type using 'T[]' is forbidden for non-simple types. Use 'Array<T>' instead

// Group rows by their label combination
const seriesMap = new Map<string, { labels: Labels; values: Array<[number, number]> }>();

return [
{
name: 'log_count',
for (const row of data) {
const labels = buildLabels(row, TIMESTAMP_COLUMN, VALUE_COLUMN);
const key = labelsToKey(labels);

// Parse timestamp
const rawTime = row[TIMESTAMP_COLUMN];
let timestamp: number;
if (typeof rawTime === 'number') {
timestamp = rawTime > 1e12 ? rawTime : rawTime * 1000; // Handle seconds vs ms
} else {
timestamp = new Date(String(rawTime)).getTime();
}

// Parse value
const rawValue = row[VALUE_COLUMN];
const value = typeof rawValue === 'number' ? rawValue : Number(rawValue) || 0;

if (!seriesMap.has(key)) {
seriesMap.set(key, { labels, values: [] });
}
seriesMap.get(key)!.values.push([timestamp, value]);
}

// Convert map to array of TimeSeries
const result: TimeSeries[] = [];
for (const { labels, values } of seriesMap.values()) {
// Sort values by timestamp
values.sort((a, b) => a[0]! - b[0]!);

result.push({
name: labelsToName(labels),
labels,
values,
},
];
});
}

return result;
}

export const getTimeSeriesData: TimeSeriesQueryPlugin<ClickHouseTimeSeriesQuerySpec>['getTimeSeriesData'] = async (
Expand All @@ -46,24 +117,25 @@
return { series: [] };
}

const query = replaceVariables(spec.query, context.variableState);
const { start, end } = context.timeRange;
const stepMs = calculateStep(start, end, context.suggestedStepMs);

// Replace built-in variables first, then user-defined variables
let query = replaceClickHouseBuiltinVariables(spec.query, start, end, stepMs);
query = replaceVariables(query, context.variableState);

const client = (await context.datasourceStore.getDatasourceClient(
spec.datasource ?? DEFAULT_DATASOURCE
)) as ClickHouseClient;

const { start, end } = context.timeRange;

const response: ClickHouseQueryResponse = await client.query({
start: start.getTime().toString(),
end: end.getTime().toString(),
query,
});

return {
series: buildTimeSeries(response),
timeRange: { start, end },
stepMs: 30 * 1000,
stepMs,
metadata: {
executedQueryString: query,
},
Expand Down
61 changes: 61 additions & 0 deletions clickhouse/src/queries/replace-click-house-builtin-variables.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
// Copyright The Perses Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

import { replaceVariable } from '@perses-dev/plugin-system';

/**
* Format a Date as a ClickHouse DateTime string (YYYY-MM-DD HH:MM:SS)
*/
function formatClickHouseDateTime(date: Date): string {
const pad = (n: number) => n.toString().padStart(2, '0');
return `${date.getUTCFullYear()}-${pad(date.getUTCMonth() + 1)}-${pad(date.getUTCDate())} ${pad(date.getUTCHours())}:${pad(date.getUTCMinutes())}:${pad(date.getUTCSeconds())}`;
}

/**
* Replace ClickHouse built-in variable placeholders in a query
*
* Supported variables:
* - $__timeFrom / $__timeTo - Time range as ClickHouse DateTime (YYYY-MM-DD HH:MM:SS)
* - $__timeFrom_ms / $__timeTo_ms - Time range as Unix milliseconds
* - $__interval - Step interval in seconds (for toStartOfInterval)
* - $__interval_ms - Step interval in milliseconds
*
* @param query The SQL query containing variable placeholders
* @param start The start time of the query range
* @param end The end time of the query range
* @param intervalMs The step interval in milliseconds
* @returns The query with variables replaced
*/
export function replaceClickHouseBuiltinVariables(

Check failure on line 39 in clickhouse/src/queries/replace-click-house-builtin-variables.ts

View workflow job for this annotation

GitHub Actions / lint-npm 

Replace `⏎··query:·string,⏎··start:·Date,⏎··end:·Date,⏎··intervalMs:·number⏎` with `query:·string,·start:·Date,·end:·Date,·intervalMs:·number`
query: string,
start: Date,
end: Date,
intervalMs: number
): string {
let updatedQuery = query;

// Time range as ClickHouse DateTime format
updatedQuery = replaceVariable(updatedQuery, '__timeFrom', formatClickHouseDateTime(start));
updatedQuery = replaceVariable(updatedQuery, '__timeTo', formatClickHouseDateTime(end));

// Time range as Unix milliseconds
updatedQuery = replaceVariable(updatedQuery, '__timeFrom_ms', start.getTime().toString());
updatedQuery = replaceVariable(updatedQuery, '__timeTo_ms', end.getTime().toString());

// Interval
const intervalSeconds = Math.floor(intervalMs / 1000);
updatedQuery = replaceVariable(updatedQuery, '__interval', intervalSeconds.toString());
updatedQuery = replaceVariable(updatedQuery, '__interval_ms', intervalMs.toString());

return updatedQuery;
}
Loading
Loading