Locate the failing boundary before repeating a command. A failed push points toward Git connectivity and authorization; a failed CI job needs its execution environment and tests inspected; a deployment failure requires comparing declared and actual state. Bring those checks together across the workflow built in the series.
The previous two exercises have different roles. The Python repository is an example for learning Git and CI, and the nginx distribution repository is an example for learning Kubernetes orchestration. When expanding to an actual app, store the image created by CI in the registry and add a connection to send the verified digest to the PR of the distribution repository.
Git · GitOps All Series · 24 / 24
On this page
Paths changed by humans and inspected by automation
개발자 브랜치 → PR 리뷰 → main
↓
CI 테스트·빌드
↓
레지스트리의 고정 digest
↓
배포 설정 PR → 환경 승인
↓
Argo CD 비교·동기화 → 상태 관찰When CI proposes changes to the deployment repository, we also use a separate bot account and minimal privileges. It is easier to check the reasons for changes with limited branches and review paths than with tokens that are unconditionally written directly to main. We also separate the standards for trusting CI results from the standards for judging the health of the app.
Narrow down the area of responsibility starting from the symptoms.
| symptoms | Section to check | first check |
|---|---|---|
| reject push authentication | PC → Git server | URL·User·Key·Storage permissions |
| CI checkout failed | runner → Git server | Execution node·known_hosts·read key |
| test failed | Code and build environment | Corresponding commit, tool version, and failure test |
| ComparisonError | Argo CD → Storage/Rendering | Authentication·Branch·Route·YAML |
| Maintain OutOfSync | Sync Policy | Automation/project permissions/events |
| ImagePullBackOff | Node → Registry | Image path·digest·authentication·network |
OutOfSync is the difference state and Healthy is the resource health state. Even if it is Synced, you cannot be sure that the app is normal, and even if it is Healthy, there may be differences from Git. View both states together with the actual user request.
Things that cannot be recovered if you only store Git
You can use git bundle when moving local history or making additional copies. Git objects and references can be bundled and delivered offline, but they do not replace Gitea DB and LFS objects, issue attachments, or secret repositories.
cd ~/git-lab/gitops-demo
git bundle create ../gitops-demo.bundle --all
git bundle verify ../gitops-demo.bundle
git clone ../gitops-demo.bundle ../gitops-demo-recovery
git -C ../gitops-demo-recovery fsck --fullCheck the restored bundle repository for the branches and tags you need. Preserve untracked files and uncommitted work separately. Test a full server recovery with the database, file and configuration restore procedure in Part 16. Cluster data requires its own storage and database backup procedures.
Emergency changes are recorded up to the resumption conditions.
If manual modifications are required during operation, check the affected applications and automatic scaling policies. If automation is paused within the approved scope, we leave the stop time, actual actions, commits reflected in Git, and who is responsible for normal confirmation and resumption. You should be able to describe which state will be in effect when the automation is turned back on.
Code problems are responded to with PR and revert, output problems are responded to with a verified previous digest, and data corruption is responded to with a data recovery plan. If you unconditionally perform reset --hard, force push, or complete reinstallation, you may lose the cause and recovery data.
Four identifiers are recorded in operational records.
For each deployment, we connect the source commit, CI execution, result digest, and deployment configuration commit. If you add the approval and distribution results here, it is good to track whether the code or settings were changed in case of a failure. This information does not include your actual password or token.
If you are a team using it for the first time, start with one small service and one development namespace. It is easier to leave a basis for judgment by reviewing changes, failing tests, reverting incorrect settings, and even restoring servers once and then expanding the scope of operation. Progress is based on the speed with which we understand the recovery path rather than the speed with which our tools grow.
Offline history storage can be found at git bundle, Argo CD status concepts can be found at Core Concepts, and service recovery can be found at Gitea Backup and Restore.
Previous · [Git and GitOps 23] Manage Secrets with SOPS and age · Series list and example files
