MaruOpsInfrastructure and AI notes

About (Korean)Contact (Korean)

[Git and GitOps 22] Review Auto-Sync, selfHeal, prune and Git Revert

Equipment aisle at Virginia Tech data center

Reference photo · Equipment aisle at Virginia Tech data center · Photo: Christopher Bowns / Source / CC BY-SA 2.0 · Resized for listings; illustrative photo, not the tested environment

If Git declares 1 replica while the cluster has been changed to 2, GitOps observes a difference between desired and actual state, or drift. The synchronization policy determines whether the controller corrects it automatically or waits for a reviewed action. Check that policy before testing selfHeal or prune.

It uses the dedicated practice cluster of Part 21 and the gitops-web Application. This is not an order intended to reproduce a failure in operations. First, check that manual synchronization was successful and Deployment is in a healthy state.

Git · GitOps All Series · 22 / 24

On this page

The three options do not mean the same thing

settingsmeaningrisk to check
automatedAutomatic synchronization of Git’s new desired stateUnapproved changes to main may also be reflected.
selfHealCluster-side changes are also rebalanced to the desired state.May conflict with emergency manual fixes
pruneDelete missing managed resources from GitIncorrectly deleted YAML can lead to actual deletion

Turning on automatic synchronization alone does not always recover all manual changes to the cluster. Distinguish between selfHeal status and change detection cycle. If HPA or another controller manages the same fields, you also need to design who owns the replicas.

1. Send replica changes to Git

In the distribution repository, change the replica of environments/dev/deployment.yaml from 1 to 2 and review it as a PR. In a manual policy, even if a Git change is detected and becomes OutOfSync, the actual replica may remain the same until Sync. Read the desired difference from the UI and sync.

kubectl -n gitops-demo get deployment gitops-web \
  -o jsonpath='{.spec.replicas}{"\n"}'

Depending on the interval at which Argo CD checks the storage and the webhook settings, it may not be reflected immediately on the screen. Refresh is an action that re-compares the desired state and distinguishes it from Sync. Check the commit ID and comparison time.

2. Turn on auto-sync and selfHeal

In the bootstrap you manage, application.yaml, replace syncPolicy with the following: Do not turn on automated deletion yet. If bootstrap is managed separately, the application settings can be changed only after recording it in Git and having the administrator apply the file.

syncPolicy:
  automated:
    enabled: true
    selfHeal: true
    prune: false
kubectl apply -f application.yaml
# Intentional drift test in the dedicated lab only:
kubectl -n gitops-demo scale deployment/gitops-web --replicas=3
kubectl -n gitops-demo get deployment gitops-web --watch

If Git’s value is 2, a controller with selfHeal enabled will try to set the actual state back to 2. The scale success returned immediately may be different from the value observed a few seconds later. Check events and Argo CD synchronization history together. This response is for the reader to verify on a real cluster and recovery times measured in this article are not presented.

prune even automates deletion

When you delete the Service YAML in the lab, you can turn on prune to automatically delete the actual Service. However, before turning on automatic deletion, check the target with Diff and manual prune. We do not experiment with PVCs or external resources where our data resides.

Invalid directories, empty rendering results, and resource name changes can lead to deletion. Also review Argo CD’s allowEmpty and deletion protection policies. The reason for using automatic synchronization is not to skip deletion judgment.

Restore the previous state with new changes in Git

cd ~/git-lab/gitops-config
git switch main
git pull --ff-only
git log --oneline -5
git switch -c revert/bad-deployment
# Replace BAD_CONFIG_COMMIT with the specific non-merge configuration commit.
git revert BAD_CONFIG_COMMIT
git push -u origin revert/bad-deployment

After reviewing and merging the PRs, the desired state itself will revert to its previous settings. Even if an operator temporarily deploys an old revision from the UI, they can move forward again if Git’s main still points to the new state. Argo CD’s rollback functionality with auto-sync enabled also has limitations, so check the official policy.

If emergency manual action is needed, record who and what the automation is pausing, what will be reflected in Git, and the conditions for resuming. If the controller keeps reverting and repeating the same kubectl edit, it becomes difficult to determine the fault. Rolling back code and restoring data are separate procedures.

For detailed operation, refer to Argo CD Automated Sync Policy, and for delete/apply options, refer to Sync Options.

Previous · [Git and GitOps 21] Deploy with Argo CD: Installation, Projects and Git Access · Next · [Git and GitOps 23] Manage Secrets with SOPS and age · Series list and example files