Image upgrade with version-specific settings causes CrashLoopBackOff due to ConfigMap update before StatefulSet reconciliation

[dashashutosh24]

Describe the bug

When upgrading ClickHouse from one version to another (e.g., 25.3 → 25.8) while simultaneously adding version-specific settings, the pods enter CrashLoopBackOff and never recover without manual intervention.

The root cause is the reconciliation order introduced in v0.24.3:

1. ConfigMaps are updated first with new version-specific settings
2. Operator attempts SYSTEM SHUTDOWN (software restart) on the existing pod
3. Pod restarts with the OLD image but mounts the NEW ConfigMap
4. ClickHouse crashes because the old version doesn't recognize the new settings
5. Operator gets stuck in waitHostIsReady() waiting for the unhealthy host
6. StatefulSet update (which would apply the new image) is never reached

This is a regression from v0.24.3 where the restart behavior was changed:

"Changed a way ClickHouse is restarted in order to pickup server configuration change. Instead of pod re-creation it tries SYSTEM SHUTDOWN first."

To Reproduce

1. Deploy a ClickHouse cluster with operator v0.25.6 (or any version >= 0.24.3)
2. Use ClickHouse version 25.3
3. Update the CHI to:
   • Change image to ClickHouse 25.8
   • Add a setting that only exists in 25.5+ (e.g., write_marks_for_substreams_in_compact_parts)
4. Apply the updated CHI
5. Observe pods entering CrashLoopBackOff

Example CHI change:

spec:
  configuration:
    settings:
      merge_tree:
        # This MergeTree setting is only valid in ClickHouse 25.5+
        write_marks_for_substreams_in_compact_parts: 0
  templates:
    podTemplates:
      - name: clickhouse-pod
        spec:
          containers:
            - name: clickhouse
              image: clickhouse/clickhouse-server:25.8.16.34  # Changed from 25.3

Expected behavior

When an image change is detected, the operator should:

1. Update the StatefulSet with the new image first (or skip software restart for image changes)
2. Wait for the new pod to be running with the new image
3. Then apply ConfigMap changes

OR

The operator should detect that an image change requires a full pod replacement and skip the SYSTEM SHUTDOWN software restart attempt.

Actual behavior

1. Operator detects the image change correctly (visible in logs)
2. ConfigMaps are updated with new settings (including version-specific ones)
3. Operator sends SYSTEM SHUTDOWN to ClickHouse
4. Kubernetes restarts the container with the same old image (StatefulSet not yet updated)
5. ClickHouse 25.3 fails to start due to unrecognized MergeTree setting:

   Code: 137. DB::Exception: Unknown setting 'write_marks_for_substreams_in_compact_parts'.
   

6. Pod enters CrashLoopBackOff
7. Operator stuck in waitHostIsReady() indefinitely
8. StatefulSet reconciliation (with new image) is never executed

Workaround

Manual recovery:

kubectl rollout restart deployment clickhouse-operator -n <namespace>

This triggers a fresh reconciliation that properly updates the StatefulSet.

Prevention:
Perform upgrades in two phases:

1. First: Update image only (no new settings)
2. Second: Add version-specific settings after pods are running new image

Operator logs

Change detection working correctly:

I0217 09:59:32.115253  1 worker-reconciler-chi.go:182] logSWVersion():Host:0-0[0/0]:
  default/clickhouse:Host software version: 0-0 25.8.16[25.8.16.34/parsed from the tag: '25.8.16.34']

diff item [14]:'.Templates.PodTemplates[0].Spec.Containers[0].Image' = '"clickhouse/clickhouse-server:25.8.16.34"'

Stuck waiting for unhealthy host:

I0217 09:59:XX.XXXXXX  1 worker.go:XXX] waitHostIsReady():
  Waiting for host to be ready...
  [Repeated indefinitely - host never becomes ready due to CrashLoopBackOff]

Environment

Component	Version
Operator	0.25.6
Previous Operator	0.24.2 (worked correctly)
ClickHouse (before)	25.3
ClickHouse (after)	25.8
Kubernetes	1.34
Installation method	Helm

Additional context

• This issue was introduced in v0.24.3 with the SYSTEM SHUTDOWN optimization
• v0.24.2 and earlier versions did not have this issue because they always did full pod replacement
• The issue only manifests when upgrading images AND adding version-specific settings simultaneously

[sunsingerus]

Thanks for the detailed report. The root cause is correct, but the failure mechanism description needs some clarification based on the code.

What actually happens

Reconciliation order (confirmed)

reconcileHostMain runs in this order:

1. reconcileConfigMapHost — ConfigMap written first with new settings
2. reconcileHostStatefulSet — STS updated after

So yes, the new config is on-disk before the pod restarts.

What triggers SYSTEM SHUTDOWN

SYSTEM SHUTDOWN is triggered by the settings change, not the image change. shouldForceRestartHost calls IsConfigurationChangeRequiresReboot, which fires when settings differ between old and new CR. Adding write_marks_for_substreams_in_compact_parts satisfies this. An image-only change would NOT trigger a software restart — the STS reconciler would handle it directly.

The scenario: settings change → SYSTEM SHUTDOWN → kubernetes restarts container with the same STS (old image) → container picks up the already-updated ConfigMap volume → old CH 25.3 fails to parse the unknown setting → CrashLoopBackOff. This part is accurate.

Where the description is inaccurate

"Operator stuck in waitHostIsReady() indefinitely" — not quite right.

All wait functions (waitHostIsReady, waitHostIsRunning, waitHostIsStarted, waitHostRestart) use domain.PollHost with a hard Timeout sourced from config.Reconcile.StatefulSet.Update.Timeout (default 300 seconds). They are not infinite.

"StatefulSet reconciliation is never executed" — also not right.

The actual fallback path in the code:

func (w *worker) hostForceRestart(...) error {
    if host.IsStopped() || (w.hostSoftwareRestart(ctx, host) != nil) {
        _ = w.hostScaleDown(ctx, host, opts)  // scale to 0 if software restart fails
    }
    return nil
}

// in reconcileHostStatefulSet:
_ = w.hostForceRestart(ctx, host, opts)  // error discarded
// ...
err := w.stsReconciler.ReconcileStatefulSet(ctx, host, true, opts)  // ALWAYS runs

When hostSoftwareRestart fails, hostForceRestart calls hostScaleDown (scale STS to 0), then returns. ReconcileStatefulSet is always called after, applying the new image and scaling back up.

Where the actual blocking happens

The block is almost certainly in waitHostIsStarted, not waitHostIsReady. isPodStarted checks containerStatus.Started, which is set true only after the container passes its postStart lifecycle hook. In CrashLoopBackOff, CH exits at config-parse time — before postStart — so Started stays false on every attempt. This polls for the full 5-minute timeout before returning "host is not started" and triggering the fallback.

(waitHostIsRunning wouldn't block here because PodPhaseIsRunning checks pod.Status.Phase, which remains Running during CrashLoopBackOff — it's the container that crashes, not the pod.)

Summary

The operator does self-recover, but only after ~5 minutes of unnecessary CrashLoopBackOff. It's not stuck permanently. Restarting the operator deployment skips the wait, but the automatic timeout + fallback path would reach the same result on its own.

Proposed fix direction

The fix in shouldForceRestartHost is the right place: if the pod template image differs between ancestor and new STS, skip SYSTEM SHUTDOWN and let ReconcileStatefulSet handle the pod replacement with the correct image from the start:

case isImageChanged(host):
    w.a.V(1).M(host).F().Info("Image change detected, no software restart applicable. Host: %s", host.GetName())
    return false

This avoids the entire intermediate restart-with-wrong-image cycle.

[dashashutosh24]

Thanks @sunsingerus , yes the wait indefinitely part was a miss on my end. We have a wait time of 15 mins as opposed to default of 5 mins, hence the wait till timeout option is not a valid resolution for us. Hence we had to restart operator manually to skip the wait and trigger reconciliation again.

[CaptTofu]

Greetings - I have confirmed this as well:

(after adding

        <clickhouse>
            <merge_tree>
                <write_marks_for_substreams_in_compact_parts>0</write_marks_for_substreams_in_compact_parts>
            </merge_tree>
        </clickhouse>

to my example CHI and bumping the version to 25.8.16.34, reconcile starts and pod fails to CrashLoopBackOff

$ kubectl apply -f example-zk-chi.yaml -n test-namespace 
clickhouseinstallation.clickhouse.altinity.com/example configured

$ kubectl get po -n test-namespace
NAME                        READY   STATUS    RESTARTS   AGE
chi-example-example-0-0-0   1/1     Running   0          14m
chi-example-example-0-1-0   1/1     Running   0          13m
chi-example-example-1-0-0   1/1     Running   0          14m
chi-example-example-1-1-0   1/1     Running   0          13m
zookeeper-0                 1/1     Running   0          2d22h
zookeeper-1                 1/1     Running   0          2d22h
zookeeper-2                 1/1     Running   0          2d21h
MacBookPro:k8s patrickgalbraith$ kubectl get po -n test-namespace 
NAME                        READY   STATUS             RESTARTS     AGE
chi-example-example-0-0-0   0/1     CrashLoopBackOff   1 (3s ago)   14m
chi-example-example-0-1-0   1/1     Running            0            13m
chi-example-example-1-0-0   1/1     Running            0            14m
chi-example-example-1-1-0   1/1     Running            0            13m
zookeeper-0                 1/1     Running            0            2d22h
zookeeper-1                 1/1     Running            0            2d22h
zookeeper-2                 1/1     Running            0            2d21h

$ kubectl logs -f chi-example-example-0-0-0 -n test-namespace 
/entrypoint.sh: explicitly skip changing user 'default'
Processing configuration file '/etc/clickhouse-server/config.xml'.
Merging configuration file '/etc/clickhouse-server/conf.d/chop-generated-hostname-ports.xml'.
Merging configuration file '/etc/clickhouse-server/conf.d/chop-generated-macros.xml'.
Merging configuration file '/etc/clickhouse-server/conf.d/chop-generated-zookeeper.xml'.
Merging configuration file '/etc/clickhouse-server/conf.d/mergetree.xml'.
Merging configuration file '/etc/clickhouse-server/conf.d/s3.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/01-clickhouse-01-listen.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/01-clickhouse-02-logger.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/01-clickhouse-03-query_log.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/01-clickhouse-04-part_log.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/01-clickhouse-05-trace_log.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/chop-generated-remote_servers.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/chop-generated-settings.xml'.
Logging debug to /var/log/clickhouse-server/clickhouse-server.log
Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
<snip>
2026.02.20 15:09:01.074291 [ 1 ] {} <Error> Application: Code: 115. DB::Exception: Unknown setting 'write_marks_for_substreams_in_compact_parts': in MergeTree config. (UNKNOWN_SETTING), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000d812b28
1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x00000000095441bc
2. DB::Exception::Exception<String>(int, FormatStringHelperImpl<std::type_identity<String>::type>, String&&) @ 0x0000000009543f18
3. DB::BaseSettingsHelpers::throwSettingNotFound(std::basic_string_view<char, std::char_traits<char>>) @ 0x000000001043b488
4. DB::BaseSettings<DB::MergeTreeSettingsTraits>::set(std::basic_string_view<char, std::char_traits<char>>, DB::Field const&) @ 0x00000000121dd8a4
5. DB::MergeTreeSettings::loadFromConfig(String const&, Poco::Util::AbstractConfiguration const&) @ 0x00000000121f8b90
6. DB::Context::getMergeTreeSettings() const @ 0x0000000010e22a30
7. DB::(anonymous namespace)::sanityChecks(DB::Server&) @ 0x000000000daf3670
8. DB::Server::main(std::vector<String, std::allocator<String>> const&) @ 0x000000000dae1434
9. Poco::Util::Application::run() @ 0x00000000156ab53c
10. DB::Server::run() @ 0x000000000dad36fc
11. mainEntryClickHouseServer(int, char**) @ 0x000000000dad0dc8
12. main @ 0x0000000009540054
13. ? @ 0x0000000000027400
14. ? @ 0x00000000000274d8
 (version 25.3.14.14 (official build))
2026.02.20 15:09:01.074329 [ 1 ] {} <Information> Application: shutting down
2026.02.20 15:09:01.074333 [ 1 ] {} <Debug> Application: Uninitializing subsystem: Logging Subsystem
2026.02.20 15:09:01.074669 [ 79 ] {} <Information> BaseDaemon: Stop SignalListener thread

$ kubectl describe po -n test-namespace chi-example-example-0-0-0
Name:             chi-example-example-0-0-0
Namespace:        altinity-cloud-managed-clickhouse
Priority:         0
Service Account:  default
Node:             desktop-worker2/172.19.0.4
Start Time:       Fri, 20 Feb 2026 09:54:34 -0500
Labels:           apps.kubernetes.io/pod-index=0
                  clickhouse.altinity.com/app=chop
                  clickhouse.altinity.com/chi=example
                  clickhouse.altinity.com/cluster=example
                  clickhouse.altinity.com/namespace=altinity-cloud-managed-clickhouse
                  clickhouse.altinity.com/ready=no
                  clickhouse.altinity.com/replica=0
                  clickhouse.altinity.com/shard=0
                  controller-revision-hash=chi-example-example-0-0-57dcd5dc7b
                  statefulset.kubernetes.io/pod-name=chi-example-example-0-0-0
Annotations:      <none>
Status:           Running
IP:               10.244.2.32
IPs:
  IP:           10.244.2.32
Controlled By:  StatefulSet/chi-example-example-0-0
Containers:
  clickhouse:
    Container ID:   containerd://956ab4febcc3447e0618559cc13aeffe32ca95c081e6c131401a79efacd6d8c5
    Image:          docker.io/clickhouse/clickhouse-server:25.3
    Image ID:       docker.io/clickhouse/clickhouse-server@sha256:b627d7a9bc0e0c1bac26cdbe9d2fc6316faa29c5d8a174f28f5abd57d0fa6ba2
    Ports:          9000/TCP, 8123/TCP, 9009/TCP
    Host Ports:     0/TCP, 0/TCP, 0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    115
      Started:      Fri, 20 Feb 2026 10:09:28 -0500
      Finished:     Fri, 20 Feb 2026 10:09:29 -0500
    Ready:          False
    Restart Count:  3
    Liveness:       http-get http://:http/ping delay=60s timeout=1s period=3s #success=1 #failure=10
    Readiness:      http-get http://:http/ping delay=10s timeout=1s period=3s #success=1 #failure=3
    Environment:
      CLICKHOUSE_SKIP_USER_SETUP:  1
    Mounts:
      /etc/clickhouse-server/conf.d/ from chi-example-deploy-confd-example-0-0 (rw)
      /etc/clickhouse-server/config.d/ from chi-example-common-configd (rw)
      /etc/clickhouse-server/users.d/ from chi-example-common-usersd (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-sdtmj (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  chi-example-common-configd:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      chi-example-common-configd
    Optional:  false
  chi-example-common-usersd:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      chi-example-common-usersd
    Optional:  false
  chi-example-deploy-confd-example-0-0:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      chi-example-deploy-confd-example-0-0
    Optional:  false
  kube-api-access-sdtmj:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  15m                default-scheduler  Successfully assigned altinity-cloud-managed-clickhouse/chi-example-example-0-0-0 to desktop-worker2
  Normal   Pulled     15m                kubelet            Successfully pulled image "docker.io/clickhouse/clickhouse-server:25.3" in 806ms (806ms including waiting). Image size: 193821838 bytes.
  Warning  Unhealthy  62s (x2 over 65s)  kubelet            Readiness probe failed: Get "http://10.244.2.32:8123/ping": dial tcp 10.244.2.32:8123: connect: connection refused
  Warning  Unhealthy  62s (x2 over 65s)  kubelet            Liveness probe failed: Get "http://10.244.2.32:8123/ping": dial tcp 10.244.2.32:8123: connect: connection refused
  Normal   Pulled     60s                kubelet            Successfully pulled image "docker.io/clickhouse/clickhouse-server:25.3" in 764ms (764ms including waiting). Image size: 193821838 bytes.
  Normal   Pulled     45s                kubelet            Successfully pulled image "docker.io/clickhouse/clickhouse-server:25.3" in 574ms (574ms including waiting). Image size: 193821838 bytes.
  Normal   Pulling    18s (x4 over 15m)  kubelet            Pulling image "docker.io/clickhouse/clickhouse-server:25.3"
  Normal   Created    17s (x4 over 15m)  kubelet            Created container clickhouse
  Normal   Started    17s (x4 over 15m)  kubelet            Started container clickhouse
  Normal   Pulled     17s                kubelet            Successfully pulled image "docker.io/clickhouse/clickhouse-server:25.3" in 707ms (707ms including waiting). Image size: 193821838 bytes.
  Warning  BackOff    1s (x11 over 59s)  kubelet            Back-off restarting failed container clickhouse in pod chi-example-example-0-0-0_altinity-cloud-managed-clickhouse(449ee738-e8c8-480e-aa80-3235f907c72e)

$ kubectl describe chi -n test-namespace example |grep 25
          Image:              docker.io/clickhouse/clickhouse-server:25.8.16.34
  Chop - Date:     2025-12-11T21:11:23
  Chop - Version:  0.25.6
    auto-b32591de-7c78-420e-8b46-dba106af2127
    auto-e27b5c55-d254-4d6d-a7bf-f8c07d6734c1
  Info   ReconcileStarted        15m   clickhouse-operator  reconcile started, task id: auto-e27b5c55-d254-4d6d-a7bf-f8c07d6734c1
  Info   ReconcileCompleted      15m   clickhouse-operator  Reconcile Host completed. Host: 0-0 ClickHouse version running: 25.3.0[25.3/parsed from the tag: '25.3']
  Info   ReconcileCompleted      14m   clickhouse-operator  Reconcile Host completed. Host: 1-0 ClickHouse version running: 25.3.0[25.3/parsed from the tag: '25.3']
  Info   ReconcileCompleted      14m   clickhouse-operator  Reconcile Host completed. Host: 0-1 ClickHouse version running: 25.3.0[25.3/parsed from the tag: '25.3']
  Info   ReconcileCompleted      14m   clickhouse-operator  Reconcile Host completed. Host: 1-1 ClickHouse version running: 25.3.0[25.3/parsed from the tag: '25.3']
  Info   ReconcileCompleted      14m   clickhouse-operator  reconcile completed successfully, task id: auto-e27b5c55-d254-4d6d-a7bf-f8c07d6734c1
$