Kubernetes (K8S) comes with a set of objects out of the box to help manage your containers, but what if you need to manage something outside the usual Kubernetes scope? That’s where Custom Resource Definitions (CRDs) come into play.
CRDs are a powerful feature in Kubernetes that lets you extend its native API, enabling you to create your own resource types. Think of them like custom tools that you can add to Kubernetes to manage anything you need, whether it’s a new application or a unique service that Kubernetes wasn’t designed to handle by default.
In this article, we’ll dive into what CRDs are, where you can use them, and how you can start creating your own. We’ll also go over how to manage them and share some best practices for using CRDs effectively.
What is a Custom Resource Definition (CRD) in Kubernetes?
A Custom Resource Definition (CRD) in Kubernetes allows you to define your own custom objects, which work just like the default Kubernetes resources (like Pods or Deployments). You defined these custom resources, and they allow Kubernetes to manage a wider range of use cases beyond the built-in objects.
For example, you could define a CRD to manage database instances, where you can automate the provisioning and scaling of databases within Kubernetes itself. CRDs are especially useful for building Kubernetes Operators, which are controllers that automate the lifecycle of complex applications.
Difference between an API and CRDs in Kubernetes
In Kubernetes, the API (Application Programming Interface) defines the programmatic interactions between system components.It provides predefined endpoints for managing resources such as Pods, Services, and Deployments.
A Custom Resource Definition (CRD) is an extension of the Kubernetes API which enables you to define custom resource types beyond the built-in ones. CRDs allow developers to introduce application-specific abstractions while maintaining compatibility with Kubernetes’ declarative management model.
Kubernetes API extensions provide deeper integration by modifying or extending the Kubernetes API server itself. This includes mechanisms such as Aggregated API Servers and Admission Webhooks, which allow for custom logic to be applied at the API level, such as validation, authentication, and admission control.
While CRDs are simpler to manage and implement, API extensions offer more flexibility and control, making them suitable for advanced use cases where custom logic or specialized rules are required.
Difference between ConfigMaps and CRDs in Kubernetes
ConfigMaps hold non-sensitive configuration data in key-value pairs which enables applications to separate their configuration from the code. This allows pods to access external settings without the need to alter container images.
In essence, ConfigMaps are used to store external configuration data for applications, while CRDs extend the Kubernetes API to support custom resources. This difference makes ConfigMaps well-suited for storing environment-specific settings, whereas CRDs are more appropriate for creating reusable and extendable Kubernetes objects.
Read Also: Data Management in Kubernetes with Portworx
What are the Common Use Cases for CRDs in Kubernetes?
Here are some common use cases:
Build Custom Operators and Controllers
CRDs help create operators that automate complex tasks such as installation, upgrades, and scaling. They manage the application lifecycle without manual effort.
Manage Application Workloads
CRDs define custom resources according to specific applications like KafkaCluster or ElasticSearchCluster. This simplifies deployment, configuration, and scaling.
Automate Infrastructure Management
Tools like Crossplane use CRDs to represent cloud infrastructure such as databases, storage, and networking. This allows teams to manage infrastructure declaratively within Kubernetes.
Enforce Policies and Compliance
CRDs work with controllers to enforce security policies, compliance standards, and resource constraints. They apply these rules automatically across the cluster.
Support CI/CD Pipelines
Kubernetes-native CI/CD tools define CRDs to manage deployment workflows. This ensures automated, repeatable, and consistent release processes.
Also Read: Optimizing CI/CD Pipelines with DevOps Best Practices
How can View the List of all Existing CRDs in Kubernetes?
You might already have CRDs installed on your cluster, especially if you’re using specific tools. To view them, just run:
kubectl get crdIf you want more details about a specific CRD, you can use:
kubectl describe crd Some Kubernetes tools will automatically install CRDs when you set them up. This allows the tools to offer features beyond what Kubernetes provides by default. Here are a few examples:
- Helm Charts: Many Helm charts come with CRDs to enable additional functionality (e.g., cert-manager, Istio, ArgoCD).
- Operators: Kubernetes Operators typically use CRDs to manage their custom resources (e.g., Prometheus Operator, FluxCD).
- Kustomize: A tool that can be used to install CRDs along with other Kubernetes resources.
For example, if you install Prometheus on your cluster, the corresponding CRDs for Prometheus will be installed automatically, allowing you to use Prometheus-specific resources.
Also Read: Top DevOps Tools for Seamless Salesforce CI/CD Integration
How to Create a Kubernetes CRD?
To extend Kubernetes with your custom resources, you can define a Custom Resource Definition (CRD). This allows Kubernetes to recognize and manage new resource types beyond the built-in ones, like Pods or Deployments. Once you define and apply the CRD, you can create instances of your custom resource (CR) to manage them.
Here’s an example of creating a CRD in Kubernetes for managing switches in a data center. Most use cases can leverage existing CRDs available online, so making one from scratch should be done when necessary.
Define the CRD YAML File
The first step is to define a YAML file for your CRD. This file specifies the custom resource’s API group, version, and schema.
Example: switchcrd.yaml (CRD definition for switches)
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: switches.datacenter.com # Name must match the format: .
spec:
group: datacenter.com # Group name for the REST API (e.g., /apis//)
versions:
- name: v1
served: true # Whether this version is available
storage: true # The storage version, only one version can be marked as storage
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
dataCenter:
type: string
rack:
type: integer
type:
type: string
scope: Cluster # Can be either Namespaced or Cluster
names:
plural: switches # Plural name used in the URL (/apis///)
singular: switch # Singular name used in the CLI and for display
kind: Switch # The kind field, used in the resource manifests
shortNames:
- sw # Short name for use in the CLI
Apply the CRD to Kubernetes
Once the CRD YAML is ready, apply it to your Kubernetes cluster using the following command:
kubectl apply -f switchcrd.yaml
This registers the new CRD with Kubernetes, allowing you to create instances of your custom resource.
Verify the CRD
To ensure the CRD was successfully applied, list the CRDs in your cluster:
kubectl get crd
Now that the CRD is in place, you can check if your custom resource is available:
kubectl get switches
At this point, you will see an empty list because you have defined the CRD, but no custom resources (switches) have been created yet.
Create a Custom Resource (CR)
Now, create a custom resource based on the CRD. In this case, let’s define a switch called Switch1.
Example: switch1.yaml (Custom resource definition for a switch)
apiVersion: datacenter.com/v1
kind: Switch
metadata:
name: switch1
spec:
dataCenter: uksouth
rack: 280
type: physical
Apply the custom resource to Kubernetes:
kubectl apply -f switch1.yaml
Verify the Custom Resource
Now that the custom resource is created, you can list all the switches:
kubectl get switches
At this point, the switch is now a part of the Kubernetes API. However, this resource doesn’t perform any actions on its own. It’s simply a record with the specifications defined.
Automating the Custom Resource with a Controller
To make the custom resource functional, you can create a Custom Controller. A custom controller is an application or script that listens for changes to your custom resources and takes action accordingly. For example, you can use tools like Kubebuilder, Operator SDK, or custom Go code to create these controllers.
A custom controller will monitor the Kubernetes API for changes to the switch objects and take appropriate actions based on the defined logic. This introduces automation, allowing you to manage the state of your custom resources and integrate them with your Kubernetes cluster in a seamless way.
By following these steps, you’ve created a CRD, deployed a custom resource, and can automate its management through a custom controller. This approach provides a flexible and scalable way to manage custom resources in Kubernetes, allowing you to extend Kubernetes to fit your unique requirements.
Read Also: How to Manage and Protect Virtualized and Containerized Environments with Scale Computing and Rubrik?
How can Manage Kubernetes CRDs?
Effectively managing Custom Resource Definitions (CRDs) in Kubernetes is critical to maintaining the stability and integrity of your resources. This involves taking care when updating CRDs, handling versioning properly, and ensuring that deletion is done safely. Without proper management, you could face issues like schema conflicts, orphaned resources, or unexpected failures in your applications.
Update CRDs
When updating a CRD, it’s important to avoid breaking existing resources. If the changes are non-disruptive, such as adding new optional fields, updating the CRD is usually safe. However, Kubernetes enforces schema validation, so even minor changes like altering constraints or default values can cause issues if not tested properly.
You can use the kubectl apply or kubectl replace commands to apply the update. Before making the change, it’s best to preview it with kubectl diff or use kubectl apply --dry-run=client to ensure backward compatibility. It’s also a good idea to validate the schema with OpenAPI v3.
Example:
Here’s an example of updating a CRD:
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: widgets.example.com
spec:
group: example.com
names:
kind: Widget
listKind: WidgetList
plural: widgets
singular: widget
scope: Namespaced
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
size:
type: string
To apply this update, use:
kubectl apply -f widget-crd.yaml
If you need to make structural schema changes (e.g., removing fields), Kubernetes will not allow these changes in the current version unless x-kubernetes-preserve-unknown-fields was enabled beforehand.
Version CRDs
Versioning your CRDs is crucial for ensuring API stability while allowing you to make incremental improvements. Kubernetes supports versioning by defining multiple versions in the versions field.
For instance, if you want to add a new version v2 while keeping the v1 version, you would define the versions field like this:
versions:
- name: v1
served: true
storage: false
- name: v2
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
size:
type: string
color: # New field in v2
type: string
To apply the version update, run:
kubectl apply -f widget-crd.yaml
Before deprecating a version, it’s important to check which versions are in use:
kubectl get crd widgets.example.com -o yaml | grep versions -A 5
When you deprecate a version, set storage: false for that version but keep served: true temporarily to allow a smooth migration. Once all resources have migrated to the new version, you can safely remove the deprecated version to reduce overhead.
Delete CRDs
Deleting a CRD removes all associated resources, which could result in data loss. Therefore, it’s essential to ensure no critical workloads depend on the CRD before removing it.
Since Kubernetes doesn’t automatically back up CRDs, you should export the custom resource data before deleting:
kubectl get myresources.example.com
If resources are still present, consider backing them up first. You can also use finalizers to prevent the CRD from being deleted before necessary cleanup is done.
Example of adding a finalizer to prevent deletion:
apiVersion: example.com/v1
kind: MyResource
metadata:
name: example-resource
finalizers:
- prevent-deletion
To delete the CRD safely, use:
kubectl delete crd myresources.example.com
If the deletion hangs, check the reasons for the block:
kubectl describe crd myresources.example.com
kubectl get events --field-selector involvedObject.name=myresources.example.com
kubectl logs
In case of an issue, you can forcefully remove the finalizers:
kubectl patch myresource example-resource -p '{"metadata":{"finalizers":[]}}' --type=merge
By following these steps for managing updates, versioning, and deletion, you can ensure that your CRDs are handled correctly without disrupting your resources or causing unforeseen issues in your Kubernetes environment.
Best Practices When Using Kubernetes CRDs
Here are some best practices to ensure you manage CRDs effectively:
- Implement Controllers: Use controllers to automate the management and reconciliation of custom resources.
- Secure Access: Define proper RBAC policies to restrict access to sensitive CRDs.
- Track Resource Status: Use the status subresource to track the state of your resources separately from the spec.
- Design for Stability: Use versioning to safely introduce changes to your resources.
- Validate with OpenAPI Schema: Use OpenAPI schema validation to ensure your resources are correctly configured.
- Follow Kubernetes Conventions: Stick to standard naming conventions and resource structures for consistency.
Wrapping Up
Custom Resource Definitions (CRDs) are a game-changer for Kubernetes, which allows you to define and manage custom resources that meet your needs. Whether you’re building automation, defining custom workloads, or managing infrastructure, CRDs offer the flexibility to extend Kubernetes beyond its built-in resources.
By following best practices and using the right tools, you can leverage CRDs to streamline your Kubernetes management and unlock new possibilities for your applications.
