Primers › Deployment

GitOps, Argo CD and Flux

A team runs a small web shop on Kubernetes. One afternoon the site is slow, and someone runs helm upgrade by hand to raise the number of copies of the app from 2 to 3. It works. Three months later nobody remembers that the cluster no longer matches what’s in the team’s repository, and the next deploy quietly puts it back to 2.

GitOps is the practice that stops that from happening. The repository says what should run, and a program inside the cluster keeps the cluster matching it. The deploy step becomes a commit: instead of someone running a command against the cluster, a pipeline or a person pushes a change to git and the cluster picks it up [1]. Argo CD and Flux are the two common tools that run that program. This primer follows one change, raising the shop from 2 copies to 3, through both of them.

A few terms

• Kubernetes manifest: a YAML file describing one object Kubernetes should run, such as a Deployment (copies of an app) or a Service (a stable address in front of them).
• Controller: a program running in the cluster that watches some kind of object and keeps changing the cluster until it matches. Kubernetes itself is mostly controllers, and so are the GitOps tools.
• CRD (custom resource definition): a way to add your own object types to Kubernetes. Argo CD and Flux are configured through their own object types, like Application or HelmRelease.
• Helm chart: a package of Kubernetes manifests with placeholders. A values file fills in the placeholders, for example replicaCount: 2. Running helm install or helm upgrade records a release in the cluster, which helm list shows.
• Kustomize: another way to vary manifests. A base holds the common files, and an overlay per environment patches them (more copies in production, fewer in staging).
• Desired state: what should be running, as written in git. Argo CD calls it the target state [2].
• Live state: what’s running in the cluster right now [2].
• Drift: any difference between the two, like the hand-run helm upgrade above.
• Reconcile: compare live with desired, and change live to match.

The four principles

The OpenGitOps project defines GitOps with four principles about the desired state [3]. Git is the usual place to keep it, but the principles only ask for something versioned and immutable.

Push and pull

Pulling changes which side opens the connection:

Arrows point from the side that opens the connection. With push, something outside the cluster needs its credentials. With pull, the cluster only reaches out to git.

With push, a pipeline outside the cluster runs helm or kubectl against it, so the pipeline needs credentials for the cluster and a network path to its API server, and nothing notices drift until the next pipeline run. With pull, the agent runs inside the cluster and fetches from git on its own, so the pipeline only needs to push a commit [1], and the agent keeps checking between commits.

One Argo CD can also manage other clusters. It stores their addresses and credentials as Secrets in its own namespace and connects to their API servers [4], [5]. For those clusters it’s pull from git but push to the cluster, which is the main architectural difference from Flux (below).

One change, step by step

The shop’s chart lives in the team’s repository at charts/shop, with production settings in charts/shop/values-production.yaml. To go from 2 copies to 3:

1. Someone changes replicaCount: 2 to replicaCount: 3 in values-production.yaml and opens a pull request.
2. The change is reviewed and merged. The commit is the record of who changed what, and reverting it is the rollback.
3. The agent in the cluster fetches the repository on its next check, or immediately if git sends it a webhook.
4. It renders the chart with the new values and compares the result with the live state. The Deployment’s replica count differs, so the app shows as out of sync.
5. It applies the difference, and Kubernetes starts a third copy.

The numbers match the steps above. Steps 3 to 6 repeat on a schedule, not only when someone commits.

Neither tool knows about the repository until something in the cluster tells it where to look. That link is itself a Kubernetes object, usually kept in git too:

Argo CD

Argo CD works with two separate sets of YAML, and they live in different places:

What an Application is

An Application is a Kubernetes object, stored by the cluster like a Deployment or a Service is. A Deployment describes containers to run. An Application describes a link between a place in git and a place in a cluster. Argo CD’s docs define it by two pieces of information: a source, the repository, revision, and path of the desired state, and a destination, the cluster and namespace to deploy into [6]. On its own it runs nothing. Argo CD’s application controller reads it, gets the manifests from the source, compares them with what’s running at the destination, and syncs the difference [5].

Here’s the shop’s Application, with what each part means [6], [7], [8]:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: shop               # the name Argo CD shows in its UI and CLI
  namespace: argocd        # where this object itself is stored
spec:
  project: default         # the Argo CD project it belongs to
  source:                  # where the desired state is
    repoURL: https://git.example.com/platform/deploy.git
    targetRevision: main   # branch, tag, or commit to follow
    path: charts/shop      # folder in that repository
    helm:
      valueFiles:
        - values-production.yaml
  destination:             # where to deploy it
    server: https://kubernetes.default.svc   # the cluster Argo CD runs in
    namespace: shop
  syncPolicy:              # optional: apply changes without anyone clicking Sync
    automated:
      prune: true
      selfHeal: true

Nothing about the shop’s containers is in there. Those are in the chart, in git. Putting the two together:

The chart stays in git. The `Application` lives in Argo CD's namespace and only points at it. Argo CD does the fetching, and creates the real objects in the destination namespace.

What the folder in git has to contain

The Application can point at any folder, but Argo CD has to be able to turn what’s in it into Kubernetes objects. Its repo server knows how to build from a few formats [9]:

Argo CD checks in that order unless the Application names a tool, so a folder with a Chart.yaml is always treated as a Helm chart [12]. Whichever way it’s built, the result has to be valid Kubernetes objects. A folder of plain manifests is read only at its top level unless the Application sets directory.recurse: true. Plain-manifest mode is only for plain files: if the Application forces it and Argo CD finds Helm, Kustomize, or Jsonnet files there, rendering fails [10].

Application files follow the same rule when they’re in git. They only need to be there if Argo CD reads them from git, which is what app of apps does: the parent Application’s folder holds the child Applications, and it’s just another source, so it can be plain Application YAML files or a Helm chart that produces them [13]. An ApplicationSet keeps one file in git, and the Applications it generates never exist as files at all.

Three ways to create an Application

Each of these creates the same kind of object in the same place. The web UI and the CLI send it through Argo CD’s API server, and kubectl writes it straight to the cluster [5], [6].

• The web UI. Click + New App and fill in the same fields as the YAML: name, project, sync policy, repository URL, revision, path, and the destination cluster and namespace. Then click Create [8].

• The argocd CLI [14]:

  argocd app create shop \
    --repo https://git.example.com/platform/deploy.git \
    --revision main \
    --path charts/shop \
    --values values-production.yaml \
    --dest-server https://kubernetes.default.svc \
    --dest-namespace shop

• kubectl, with the YAML above saved as a file. Argo CD supports managing Applications this way without touching the argocd CLI [6]:

  kubectl apply -f shop-application.yaml

None of these write anything to git. The Application only points at the repository.

Keeping the Application in git too

An Application made in the web UI or with the CLI exists only in the cluster. The shop’s chart is versioned, but the pointer to it isn’t, which is the opening problem one level up: if someone changes or deletes the Application by hand, git has no record of it.

Argo CD’s docs say you can create Applications by hand or with a script, but recommend generating them with an ApplicationSet, with a pattern called app of apps as the alternative [13]. Both work the same way at heart: one object you write once and keep in git produces the Applications, so nobody makes them by hand. In app of apps, one parent Application is created by hand, once, and its source is a folder in git full of other Application files. From then on, adding an app to the cluster means committing an Application file. Argo CD warns that this is an admin-level tool, because whoever can push to that folder can create Applications in any project, so only admins should have push access to it and changes to it should be reviewed [13].

Generating Applications with an ApplicationSet

An ApplicationSet is a recipe for Applications. It’s another Argo CD object with two parts [15], [16]:

• a generator, which produces a list of parameter sets, such as one per cluster, one per folder in a repository, or one per item in a list you write;
• a template, an Application with blanks in it. For each parameter set, Argo CD’s ApplicationSet controller fills in the blanks and creates that Application.

Say the shop now runs on two clusters, staging and production, and the team wants it on any future cluster that should have it. Each cluster Argo CD manages is registered as a Secret in Argo CD’s namespace, and those Secrets can carry labels [4]. The team labels each cluster with its environment, and labels the ones that should run the shop with shop: "enabled". Then, instead of one Application per cluster, they write one ApplicationSet [4]:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: shop
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - clusters:                  # one parameter set per registered cluster...
        selector:
          matchLabels:
            shop: "enabled"      # ...but only clusters with this label
  template:                      # an Application, with blanks
    metadata:
      name: 'shop-{{.name}}'     # the cluster's name
    spec:
      project: default
      source:
        repoURL: https://git.example.com/platform/deploy.git
        targetRevision: main
        path: charts/shop
        helm:
          valueFiles:
            - 'values-{{.metadata.labels.env}}.yaml'   # the cluster's env label
      destination:
        server: '{{.server}}'      # the cluster's API address
        namespace: shop
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

For every cluster with shop: "enabled", the cluster generator hands the template that cluster’s name, its server address, and its labels [4]. With three clusters registered, the result looks like this:

The generator decides how many Applications there are, one per selected cluster. The template decides what each one looks like.

Each generated Application is an ordinary Application, the same kind of object as the one in the previous section. It shows up in the web UI and syncs the same way. The difference is who made it.

From then on, the controller keeps the Applications matching the clusters:

• A new cluster gets the shop by being labeled. Register a fourth cluster with env=staging and shop=enabled, and the controller creates shop-<name> for it. Nobody writes an Application. The cluster generator responds to clusters being added and removed [16].
• Removing the label removes the shop. Under the controller’s default policy, sync, it creates, updates, and deletes Applications to match what the generator produces [17]. Generated Applications carry a finalizer that, by default, makes Argo CD delete what they deployed when they go away [18].
• Hand edits to a generated Application don’t stick. The same sync policy updates them back to what the template says. A policy such as create-only or create-update limits what the controller may change [17].
• Deleting the ApplicationSet deletes everything it made: the Applications, and the Deployments and Services they deployed, unless syncPolicy.preserveResourcesOnDeletion is true [18]. Be careful with kubectl delete on one.

Argo CD’s bootstrapping guide recommends exactly this pattern, cluster labels plus ApplicationSets, for getting apps onto new clusters [13]. The ApplicationSet itself still has to be created once, for example with kubectl apply, but it’s one file to keep in git instead of one per app per cluster, and every Application it makes follows from that file.

The cluster generator is one of nine. A few others that come up often [16]:

How Argo CD’s parts fit together

Argo CD runs as three main parts: an API server behind its web UI and CLI, a repo server that keeps a copy of the git repository and renders manifests from it, and an application controller that keeps comparing live state with desired state [5]. The web UI and the CLI only talk to the API server, which also handles logins and permissions [5], [19]. The API server never calls the application controller. Both work through the Kubernetes API of the cluster Argo CD runs in: the API server creates and updates Application objects there, and the controller watches those objects and does the syncing [19]. Clicking Sync in the UI writes a sync request onto the Application, which is also how you can sync with plain kubectl [20]. For other clusters, the controller connects with the credentials Argo CD stores for them [4].

The UI and CLI never reach the controller directly. The API server writes `Application` objects, the controller watches them, and the Kubernetes API sits between the two.

Watching the shop change

Each Application has a sync status, whether the live state matches git, and a separate health status, whether the app is running and able to serve requests [2]. Just after the replica change is merged, argocd app get shop would show something like this (the format follows the Argo CD getting-started guide [8]):

Name:               shop
Server:             https://kubernetes.default.svc
Namespace:          shop
Repo:               https://git.example.com/platform/deploy.git
Target:             main
Path:               charts/shop
Sync Status:        OutOfSync from main (a1b2c3d)
Health Status:      Healthy

GROUP  KIND        NAMESPACE  NAME  STATUS     HEALTH
       Service     shop       shop  Synced     Healthy
apps   Deployment  shop       shop  OutOfSync  Healthy

OutOfSync on the Deployment is the replica change waiting to be applied. The app is still Healthy, because the two existing copies are running fine.

Some behavior to know before relying on it:

• Syncing automatically is opt-in. Without a syncPolicy.automated block, Argo CD only reports OutOfSync and waits for someone to sync. Deleting what was removed from git (prune) and undoing hand edits to the cluster (selfHeal) are separate switches, and both are off by default [1]. Without selfHeal, the hand-run helm upgrade from the opening would just show as OutOfSync.
• It checks git about every three minutes by default: a 120-second interval plus up to 60 seconds of random delay [1]. A webhook from the git server makes it immediate.
• Helm is only used to render. Argo CD runs helm template and applies the result itself, instead of helm install. So helm list doesn’t show the release, and Helm’s own commands can’t inspect or roll it back. Argo CD keeps its own history and rollback instead [21], [7].
• Values come from several places, in a fixed order. From lowest to highest priority: the chart’s own values.yaml, then valueFiles, then inline values, then valuesObject, then individual parameters [7]. Since version 2.6, values files can live in a different repository from the chart [7].

Flux

Flux works with the same two sets of YAML as Argo CD, the app’s manifests and the objects that tell the tool what to do with them. The difference is where the second set lives:

Argo CD’s Application normally starts life in the cluster, from the web UI or the CLI. Flux has no web UI of its own [24], and its objects normally start life as files in git that Flux applies to itself.

Flux’s three objects

Each object is a Kubernetes object, like Argo CD’s Application, and each one is handled by a different Flux controller [25]:

• GitRepository: where a repository is, which branch to follow, and how often to check it. The source controller fetches it and packages the files as an artifact for the other controllers [26], [27].
• Kustomization: a folder in a source to apply to the cluster, how often, and whether to delete what’s been removed from git [28]. This isn’t Kustomize’s own kustomization.yaml file, despite the name. Flux’s FAQ confirms there are two different types called Kustomization: Flux’s is a Kubernetes object that points at a folder, and Kustomize’s is the file inside that folder that lists what to build [24].
• HelmRelease: a Helm chart from a source, with its values. The helm controller turns it into a real Helm release [27].

Here’s the shop’s HelmRelease, saved as clusters/production/shop.yaml in the repository [26], [28]:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: shop
  namespace: shop
spec:
  interval: 10m              # how often to check the release against git
  chart:
    spec:
      chart: ./charts/shop   # path to the chart inside the source
      reconcileStrategy: Revision   # upgrade on any new commit (see below)
      sourceRef:             # which source the chart comes from
        kind: GitRepository
        name: flux-system    # the repository bootstrap already set up
        namespace: flux-system
      valuesFiles:
        - ./charts/shop/values-production.yaml

Like an Application, it says nothing about the shop’s containers. It points at the chart, and the chart describes the containers.

The reconcileStrategy line matters for a chart kept in git. By default Flux only builds a new copy of the chart when the version in Chart.yaml goes up, so a commit that only changes values-production.yaml wouldn’t reach the cluster. Revision makes every new commit count [29], [30].

The folders Flux reads have rules too. A HelmRelease needs a Helm chart. A Kustomization reads a folder of Kubernetes manifests: if there’s no kustomization.yaml, Flux generates one covering every YAML file under the path, so every one of them has to be a valid Kubernetes manifest unless it’s excluded with a .sourceignore file or the source’s ignore field. Flux recommends committing your own kustomization.yaml so CI can check it [28].

How Flux finds the shop

Neither the HelmRelease nor the chart does anything until Flux is reading the folder they’re in. That starts with bootstrap, and the rest follows in a chain:

1. flux bootstrap, run once per cluster, installs Flux’s controllers, commits Flux’s own manifests to the repository under clusters/production/flux-system, and sets Flux to keep the cluster in sync with that folder of the repository. From then on, any change to the cluster, Flux upgrades included, can be made with a git push [31], [22].
2. Bootstrap leaves a GitRepository for the repository and a Kustomization for the cluster’s folder, both called flux-system. Flux’s docs build on that GitRepository in their own examples [22], [28].
3. The team commits shop.yaml to clusters/production. The flux-system Kustomization applies everything in that folder on its next check, so the HelmRelease appears in the cluster [28].
4. The helm controller sees the new HelmRelease, gets the chart from the source’s artifact, and runs a Helm install. The Deployment and Service appear in the shop namespace [27].

Dashed lines show which files in git each object comes from. Bootstrap writes the first two, the team commits shop.yaml, and everything below the first object is applied by Flux itself.

Raising the shop to 3 copies is then the change from earlier: edit values-production.yaml and merge. Because of reconcileStrategy: Revision, the new commit produces a new chart artifact, and the helm controller upgrades the release [29], [27].

If the chart lived in a separate repository, the team would commit a second GitRepository next to shop.yaml and point sourceRef at it instead [26]:

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: shop-chart
  namespace: flux-system
spec:
  interval: 1m               # how often to check for new commits
  url: https://git.example.com/team/shop-chart.git
  ref:
    branch: main

Creating Flux’s objects

• Commit the YAML to the cluster’s folder. That’s the normal way, and the only one that keeps git a full record.

• Generate it with flux create ... --export, which prints the YAML instead of applying it, then commit the file [32], [22]. The generated HelmRelease still needs the valuesFiles and reconcileStrategy lines added by hand:

  flux create helmrelease shop \
    --namespace=shop \
    --source=GitRepository/flux-system.flux-system \
    --chart=./charts/shop \
    --export > clusters/production/shop.yaml

• flux create without --export, or kubectl apply, creates the object straight in the cluster [33]. That works, but then the cluster has something git doesn’t, the same problem as an Argo CD Application made in the web UI.

How Flux’s controllers fit together

Flux splits the work across controllers that pass an artifact along. There's no central server, and each cluster usually runs its own copy.

Watching the shop change

Some behavior to know:

• Every object has an interval. It sets how often Flux checks that object. For a Kustomization, each check fetches the source, rebuilds, applies, and corrects any drift, with no separate switch for undoing hand edits [28].
• A HelmRelease doesn’t undo hand edits unless asked. Flux only compares the cluster with the Helm release, and corrects what differs, when driftDetection.mode is set to enabled. Set to warn, it only reports the drift [34].
• prune is a required field on a Kustomization. With prune: true, objects that were removed from git get deleted from the cluster [28].
• HelmRelease runs real Helm. The helm controller performs Helm installs, upgrades, tests, and rollbacks itself, and can roll back automatically when an upgrade fails [27]. Because these are ordinary Helm releases, helm list shows them.

flux get kustomizations shows which commit each Kustomization has applied [22]:

NAME          REVISION              SUSPENDED  READY  MESSAGE
flux-system   main@sha1:a1b2c3d4    False      True   Applied revision: main@sha1:a1b2c3d4

For many clusters, the Flux guide suggests a folder layout instead of a central server: an apps/base folder with the common manifests, an overlay per environment such as apps/production, and one clusters/<name> folder per cluster that pulls in the right overlays [23]. Usually each cluster runs its own Flux and pulls for itself. A Kustomization can apply to a remote cluster through a kubeconfig, but that’s an option, not the default design [28].

Side by side

What GitOps leaves open

Neither tool decides how to handle secrets, since passwords and keys can’t sit in plain text in git, or how to lay out values for many environments. Both are design choices a team makes on top of either tool.

References

Every source cited across the primers is collected on one page.