Custom Resource Definitions
This section of the Best Practices Guide deals with creating and using Custom Resource Definition objects.
When working with Custom Resource Definitions (CRDs), it is important to distinguish two different pieces:
- There is a declaration of a CRD. This is the YAML file that has the kind
- Then there are resources that use the CRD. Say a CRD defines
foo.example.com/v1. Any resource that has
apiVersion: example.com/v1and kind
Foois a resource that uses the CRD.
Install a CRD Declaration Before Using the Resource
Helm is optimized to load as many resources into Kubernetes as fast as possible. By design, Kubernetes can take an entire set of manifests and bring them all online (this is called the reconciliation loop).
But there’s a difference with CRDs.
For a CRD, the declaration must be registered before any resources of that CRDs kind(s) can be used. And the registration process sometimes takes a few seconds.
Method 1: Let
helm Do It For You
With the arrival of Helm 3, we removed the old
crd-install hooks for a more
simple methodology. There is now a special directory called
crds that you can
create in your chart to hold your CRDs. These CRDs are not templated, but will
be installed by default when running a
helm install for the chart. If the CRD
already exists, it will be skipped with a warning. If you wish to skip the CRD
installation step, you can pass the
Some caveats (and explanations)
There is not support at this time for upgrading or deleting CRDs using Helm. This was an explicit decision after much community discussion due to the danger for unintentional data loss. Furthermore, there is currently no community consensus around how to handle CRDs and their lifecycle. As this evolves, Helm will add support for those use cases.
--dry-run flag of
helm install and
helm upgrade is not currently
supported for CRDs. The purpose of “Dry Run” is to validate that the output of
the chart will actually work if sent to the server. But CRDs are a modification
of the server’s behavior. Helm cannot install the CRD on a dry run, so the
discovery client will not know about that Custom Resource (CR), and validation
will fail. You can alternatively move the CRDs to their own chart or use
helm template instead.
Another important point to consider in the discussion around CRD support is how
the rendering of templates is handled. One of the distinct disadvantages of the
crd-install method used in Helm 2 was the inability to properly validate
charts due to changing API availability (a CRD is actually adding another
available API to your Kubernetes cluster). If a chart installed a CRD,
longer had a valid set of API versions to work against. This is also the reason
behind removing templating support from CRDs. With the new
crds method of CRD
installation, we now ensure that
helm has completely valid information about
the current state of the cluster.
Method 2: Separate Charts
Another way to do this is to put the CRD definition in one chart, and then put any resources that use that CRD in another chart.
In this method, each chart must be installed separately. However, this workflow may be more useful for cluster operators who have admin access to a cluster