Skip to content

v4.5.0 ClickHouse writers require CH ≥25.8, but the Helm chart still bundles ClickHouse 25.7.5 → every insert fails with UNKNOWN_SETTING #4197

Description

@brentshulman-silkline

Summary

v4.5.0's ClickHouse writers (internal-packages/clickhouse/src/{taskRuns,taskEvents,metrics,sessions}.ts) unconditionally set input_format_json_infer_array_of_dynamic_from_array_of_different_types: 1 on every insert. That setting was introduced in ClickHouse 25.8. The Trigger Helm chart (oci://ghcr.io/triggerdotdev/charts/trigger) still bundles ClickHouse via the Bitnami subchart at 25.7.5 (older chart versions: 25.6.1). On any server < 25.8, every insert is rejected:

Code: 115 … input_format_json_infer_array_of_dynamic_from_array_of_different_types
is neither a builtin setting nor started with the prefix 'SQL_' (UNKNOWN_SETTING)

Result: run execution is unaffected (Postgres is the execution source of truth), but every ClickHouse-backed read path — the dashboard run list, metrics, and the list-runs / run-status APIs — silently freezes, and failed insert batches are dropped after retries (permanent run-history gaps). The setting is clearly intentional (there's a regression test in taskRuns.test.ts guarding mixed-type JSON arrays from nested-Tuple inference / CH Code 117 during merges), so this is a version-floor / packaging issue, not a request to remove the setting.

Environment

  • Self-hosted v4 on Kubernetes (EKS)
  • Upgraded chart + webapp/supervisor images 4.4.6 → 4.5.0
  • ClickHouse 25.7.5 (the chart's bundled Bitnami subchart)
  • Had to roll back to 4.4.6 to recover. v4.5.1 does not fix it (its only change is WorkOS Directory Sync / SCIM).

Requests

  1. Bump the chart's bundled ClickHouse subchart to ≥25.8, or gate the setting on the server version so it degrades gracefully on older servers.
  2. Document the minimum ClickHouse version for v4.5.x in the self-hosting docs. This is the second time the bundled-CH version has bitten self-hosters — see Helm chart pins ClickHouse 25.6.1, which has a memory-tracker overflow that breaks the dashboard until pod restart #3520, where the 25.6.1 memory-tracker overflow led to the bump to 25.7.5.
  3. Guidance: now that Bitnami's free image catalog is EOL (bitnamilegacy is a frozen archive with no ≥25.8 tag), what is the supported ClickHouse image path for the chart going forward — a Bitnami Secure Images tag, the official clickhouse/clickhouse-server image, or something else?

Happy to test a chart bump and report back, as with #3520.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions