Primers › Kubernetes

Kubernetes object size limits

A Helm upgrade fails with this message:

Error: UPGRADE FAILED: cannot patch "app-settings" with kind ConfigMap:
ConfigMap "app-settings" is invalid: []: Too long: may not be more than 1048576 bytes

1,048,576 is 1024 × 1024, so the limit is 1 MiB. Kubernetes refused to store the object because it’s too big. Nothing is wrong with Helm, the pod, or the node, and retrying won’t help. The object has to get smaller.

A few terms

• Object: a record stored in a Kubernetes cluster that describes something that should exist, such as a running app, a group of settings, or a password. Objects are usually written as YAML files.
• API server: the part of Kubernetes that every change goes through [1]. It checks each object before saving it, and the error above comes from that check.
• etcd: the database where the API server keeps every object [1].
• ConfigMap: an object that holds settings as keys and values, for an app to read as environment variables or files. It isn’t meant for large data, and its data can’t exceed 1 MiB [2].
• Secret: the same idea for sensitive values like passwords and keys, with the same 1 MiB limit [3].
• Helm: a tool that installs apps into Kubernetes from a chart, a package of templates for all the objects an app needs [4]. A values file (usually values.yaml) fills in the templates. A release is one installed copy of a chart, and every install, upgrade, or rollback adds a new revision of it [4], [5].
• MiB and MB: a mebibyte (MiB) is 1,048,576 bytes and a megabyte (MB) is 1,000,000. Kubernetes counts in bytes, and 1 MB of data fits under the limit while 1.1 MB doesn’t.

Where the limit is checked

Every object passes the same check on its way into the cluster. During a Helm upgrade, two kinds of object go through it: the ones your chart defines, and a record Helm keeps about the release itself.

Both kinds of object pass the same check. Helm saves its release record before it applies the chart's objects, so a release record that's too big fails the upgrade before any of them change.

The check adds up the values under data (and binaryData, for a ConfigMap), not the keys or the YAML around them, and compares the total with 1,048,576 bytes [6]. etcd has limits of its own, which the 1 MiB check stays safely under: by default it accepts requests up to 1.5 MiB and stores up to 2 GiB in total, with 8 GiB as the suggested maximum [7]. That total is why lots of objects that each fit can still cause trouble together.

The rendered size is the size that counts

The object Kubernetes checks is the one Helm produces from the templates, not any file you edited. Templates can repeat their input, so a modest values file can render into an object far bigger than anything in the repo:

Nobody edited a file anywhere near 1 MiB. The loop did the multiplying, and only the rendered object shows it.

To see the real size, render just the suspect template and count its bytes:

helm template ./chart -f values.yaml --show-only templates/settings-configmap.yaml | wc -c

$out = helm template ./chart -f values.yaml --show-only templates/settings-configmap.yaml | Out-String
[Text.Encoding]::UTF8.GetByteCount($out)

helm template renders the chart locally without touching the cluster, and --show-only keeps just the one file [8]. wc -c counts bytes. In PowerShell, Out-String joins the output into one string [9] and GetByteCount counts the bytes it takes as UTF-8 [10]. Either way the count includes the YAML around the data, so it reads a little high, and a number under 1,048,576 is safely under the limit.

Helm’s release record hits the same limit

Helm keeps a record of every release revision, and by default that record is a Secret in the release’s namespace, holding the contents of the chart and the values [11]. Its name follows the pattern sh.helm.release.v1.<release>.v<revision> [12]. Being a Secret, it gets the same 1 MiB check, and the error names that Secret instead of your ConfigMap.

Two details change how this one behaves:

• It fails first. During an upgrade, Helm saves the new release record before it applies any of the chart’s objects [13]. A record that’s too big stops the upgrade before anything in the cluster changes.
• It’s compressed. Helm gzips the record and then base64-encodes it [14]. Compression usually makes the record much smaller than the chart, but base64 adds about a third back, so a chart whose contents are large even after compression hits the limit this way.

For release records that really are that large, Helm documents a beta sql storage backend that keeps them in a database instead of the cluster [11].

Ways to get under the limit

Moving the data to a Secret doesn’t help. Kubernetes doesn’t compress Secrets, and the base64 you see in a Secret’s YAML is only how the bytes are written down: the check counts the decoded bytes under data, against the same 1,048,576 [15]. Helm’s release record is smaller than its contents only because Helm compresses it before saving it.

The Kubernetes docs point the same way for settings larger than the limit: a mounted volume, a database, or a file service [2]. Start with the template, though. If the rendered output holds the same block many times, include it once and reference it, and the object may drop well under the limit with no change to how the app works.

References

Every source cited across the primers is collected on one page.