Making Operator Reconciliation Transparent

When working with Kubernetes operators, you've likely heard phrases like "We have an in-house operator for that" or "Just use this custom resource (CR)." While operators simplify complex tasks through declarative definitions, they often fall short in providing clear status information, especially when things go wrong.

This is where Kubernetes Conditions come in. They're not just a nice-to-have feature—they're essential for providing transparency and reducing operational overhead. While they won't prevent operations entirely, they give application developers the information they need to diagnose and fix issues independently.

What are Conditions?

Conditions are a standardized way to report the current state of a Kubernetes object. They provide a clear picture of what's happening with your custom resources, similar to how native Kubernetes objects report their status. Each object can have multiple conditions, represented as a list of condition objects.

Implementing Conditions

Conditions should be included as a top-level element in the status field of your custom resource. The schema follows the standard defined in k8s.io/apimachinery/pkg/apis/meta/v1/types.go.

Here's how to define conditions in your custom resource status:

import (
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

type MyCRStatus struct {
    Conditions []metav1.Condition `json:"conditions"`
}

The metav1.Condition structure includes these key fields:

type Condition struct {
    Type               string
    Status             ConditionStatus
    ObservedGeneration int64
    LastTransitionTime Time
    Reason             string
    Message            string
}

Managing Conditions

While the structure might look complex, the Kubernetes apimachinery package provides helpful utilities to manage conditions. The main focus should be on determining the appropriate status to report.

A typical workflow starts with setting the Unknown status when reconciliation begins:

import (
    "k8s.io/apimachinery/pkg/api/meta"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

const (
    MyObjectAvailableType    string = "ObjectAvailable"
    MyObjectReconcilingReason string = "Reconciling"
    MyObjectFailedReason     string = "Failed"
    MyObjectCreatedReason    string = "Created"
)

meta.SetStatusCondition(
    &moObject.Status.Conditions,
    metav1.Condition{
        Status:  metav1.ConditionUnknown,
        Reason:  MyObjectReconcilingReason,
        Type:    MyObjectAvailableType,
        Message: "Object is in reconciliation phase.",
    }
)

Condition Fields Explained

• Type: A unique identifier for the condition. For example, a database operator might have types like DatabaseCreated, DatabaseHealthy, etc.
• Status: The current state (True, False, or Unknown)
• Reason: A concise explanation of why the condition is in its current state
• Message: Detailed human-readable information about the condition

Example Implementation

Here's how you might update conditions based on the outcome of an operation:

err := thirdPartyClient.create(myObject)
if err != nil {
    meta.SetStatusCondition(
        &moObject.Status.Conditions,
        metav1.Condition{
            Status:  metav1.ConditionFalse,
            Reason:  MyObjectFailedReason,
            Type:    MyObjectAvailableType,
            Message: fmt.Sprintf("Could not create MyObject: %v", err),
        }
    )
    return ctrl.Result{}, err
} else {
    meta.SetStatusCondition(
        &moObject.Status.Conditions,
        metav1.Condition{
            Status:  metav1.ConditionTrue,
            Reason:  MyObjectCreatedReason,
            Type:    MyObjectAvailableType,
            Message: "Object was successfully created",
        }
    )
}

User Experience

From the user's perspective, conditions provide clear visibility into the state of their resources:

$ kubectl describe myobject example
Name: example
Kind: MyObject
<redacted>
Status:
    Conditions:
        Message: "Could not create MyObject: user did something wrong"
        Reason: Failed
        Status: False
        Type: ObjectAvailable

Conclusion

Implementing conditions in your Kubernetes operator is straightforward and significantly improves the user experience. With just a few lines of code, you can provide valuable debugging information and reduce the operational burden on your team. Conditions are a powerful tool that every operator developer should leverage to make their operators more user-friendly and maintainable.