Lesson 4.2: Finalizers and Cleanup

Navigation: ← Previous: Conditions and Status

Introduction

When a user deletes a Custom Resource, you often need to perform cleanup before the resource is actually removed. Finalizers allow you to intercept deletion and perform necessary cleanup operations like deleting external resources, backing up data, or notifying external systems.

What are Finalizers?

Finalizers are keys in metadata.finalizers that prevent resource deletion until they’re removed:

graph TB
    DELETE[Delete Request] --> FINALIZERS{Has<br/>Finalizers?}
    FINALIZERS -->|No| REMOVE[Remove Immediately]
    FINALIZERS -->|Yes| MARK[Mark as DeletionTimestamp]
    MARK --> RECONCILE[Reconcile Called]
    RECONCILE --> CLEANUP[Perform Cleanup]
    CLEANUP --> REMOVE_FINALIZER[Remove Finalizer]
    REMOVE_FINALIZER --> REMOVE
    
    style FINALIZERS fill:#90EE90
    style CLEANUP fill:#FFB6C1

Deletion Flow with Finalizers

Here’s what happens when a resource with finalizers is deleted:

sequenceDiagram
    participant User
    participant API as API Server
    participant Controller as Controller
    participant External as External System
    
    User->>API: Delete CustomResource
    API->>API: Set DeletionTimestamp
    API->>API: Keep Resource (has finalizers)
    API->>Controller: Watch Event: UPDATE
    Controller->>Controller: Check DeletionTimestamp
    Controller->>External: Cleanup external resources
    External-->>Controller: Cleanup complete
    Controller->>API: Remove finalizer
    API->>API: Delete resource
    API-->>User: Resource deleted

Implementing Finalizers

Step 1: Add Finalizer on Creation

func (r *DatabaseReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    db := &databasev1.Database{}
    if err := r.Get(ctx, req.NamespacedName, db); err != nil {
        return ctrl.Result{}, err
    }
    
    // Add finalizer if not present
    finalizerName := "database.example.com/finalizer"
    if !controllerutil.ContainsFinalizer(db, finalizerName) {
        controllerutil.AddFinalizer(db, finalizerName)
        if err := r.Update(ctx, db); err != nil {
            return ctrl.Result{}, err
        }
    }
    
    // ... rest of reconciliation ...
}

Step 2: Handle Deletion

// Check if resource is being deleted
if !db.DeletionTimestamp.IsZero() {
    // Resource is being deleted
    return r.handleDeletion(ctx, db)
}

// Normal reconciliation
// ...

Step 3: Implement Cleanup

func (r *DatabaseReconciler) handleDeletion(ctx context.Context, db *databasev1.Database) (ctrl.Result, error) {
    logger := log.FromContext(ctx)
    finalizerName := "database.example.com/finalizer"
    
    // Check if finalizer exists
    if !controllerutil.ContainsFinalizer(db, finalizerName) {
        // Finalizer already removed, nothing to do
        return ctrl.Result{}, nil
    }
    
    logger.Info("Handling deletion", "name", db.Name)
    
    // Perform cleanup operations
    if err := r.cleanupExternalResources(ctx, db); err != nil {
        logger.Error(err, "Failed to cleanup external resources")
        // Return error to retry
        return ctrl.Result{RequeueAfter: 10 * time.Second}, err
    }
    
    // Cleanup successful, remove finalizer
    controllerutil.RemoveFinalizer(db, finalizerName)
    if err := r.Update(ctx, db); err != nil {
        return ctrl.Result{}, err
    }
    
    logger.Info("Finalizer removed, resource will be deleted")
    return ctrl.Result{}, nil
}

Cleanup Patterns

Pattern 1: Delete Owned Resources

func (r *DatabaseReconciler) cleanupExternalResources(ctx context.Context, db *databasev1.Database) error {
    // Owner references handle most cleanup automatically
    // But you might need to delete external resources
    
    // Delete backup in external system
    if err := r.deleteBackup(ctx, db); err != nil {
        return err
    }
    
    return nil
}

Pattern 2: Delete Child Resources Explicitly

Critical: When using finalizers, you must explicitly delete child resources. Owner references only cascade deletes when the parent is deleted, but finalizers prevent the parent from being deleted until cleanup completes - creating a deadlock if you only wait for resources to disappear.

func (r *DatabaseReconciler) cleanupExternalResources(ctx context.Context, db *databasev1.Database) error {
    log := log.FromContext(ctx)
    
    // Delete StatefulSet if it exists
    statefulSet := &appsv1.StatefulSet{}
    err := r.Get(ctx, client.ObjectKey{
        Name:      db.Name,
        Namespace: db.Namespace,
    }, statefulSet)
    
    if err == nil {
        // StatefulSet exists, delete it explicitly
        log.Info("Deleting StatefulSet", "name", statefulSet.Name)
        if err := r.Delete(ctx, statefulSet); err != nil && !errors.IsNotFound(err) {
            return fmt.Errorf("failed to delete StatefulSet: %w", err)
        }
        // Requeue to wait for deletion to complete
        return fmt.Errorf("waiting for StatefulSet to be deleted")
    } else if !errors.IsNotFound(err) {
        return fmt.Errorf("failed to get StatefulSet: %w", err)
    }
    
    // StatefulSet deleted, continue cleanup
    return nil
}

Pattern 3: External API Cleanup

func (r *DatabaseReconciler) cleanupExternalResources(ctx context.Context, db *databasev1.Database) error {
    // Call external API to delete resource
    if err := r.externalAPIClient.DeleteDatabase(db.Name); err != nil {
        return err
    }
    
    return nil
}

Avoiding Finalizer Deadlocks

Finalizer deadlocks can occur when:

graph TB
    DEADLOCK[Finalizer Deadlock]
    
    DEADLOCK --> CAUSE1[Controller not running]
    DEADLOCK --> CAUSE2[Cleanup always fails]
    DEADLOCK --> CAUSE3[Circular dependencies]
    DEADLOCK --> CAUSE4[External system down]
    DEADLOCK --> CAUSE5[Waiting for owner-ref cascade]
    
    style DEADLOCK fill:#FFB6C1

Common Pitfall: Owner Reference + Finalizer Deadlock

A very common deadlock occurs when:

Parent resource has a finalizer
Cleanup code waits for child resources to be deleted via owner references
Owner reference cascade only works when the parent is deleted
Parent can’t be deleted because finalizer is waiting for children to disappear

Solution: Always explicitly delete child resources during cleanup - don’t rely on owner reference cascade.

Prevention Strategies

Explicit Deletion: Delete child resources explicitly, don’t wait for owner reference cascade
Idempotent Cleanup: Cleanup should be safe to retry
Timeout: Set maximum time for cleanup
Force Removal: Allow manual finalizer removal in emergencies
Health Checks: Ensure controller is running before cleanup

Example: Timeout Protection

func (r *DatabaseReconciler) handleDeletion(ctx context.Context, db *databasev1.Database) (ctrl.Result, error) {
    // Check if deletion is taking too long
    if time.Since(db.DeletionTimestamp.Time) > 5*time.Minute {
        log.Info("Deletion timeout, forcing cleanup")
        // Force cleanup or remove finalizer
    }
    
    // ... cleanup ...
}

Multiple Finalizers

Resources can have multiple finalizers:

// Add multiple finalizers
controllerutil.AddFinalizer(db, "database.example.com/finalizer")
controllerutil.AddFinalizer(db, "backup.example.com/finalizer")

// Each controller removes its own finalizer
// Resource is deleted when all finalizers are removed

Key Takeaways

Finalizers prevent deletion until cleanup is complete
Add finalizer on resource creation
Check DeletionTimestamp to detect deletion
Explicitly delete child resources - don’t rely on owner reference cascade (causes deadlock)
Perform cleanup operations before removing finalizer
Remove finalizer only after cleanup succeeds
Make cleanup idempotent (safe to retry)
Avoid finalizer deadlocks with timeouts and health checks

Understanding for Building Operators

When implementing finalizers:

Add finalizer early in reconciliation
Check DeletionTimestamp for deletion
Perform all cleanup before removing finalizer
Handle cleanup failures gracefully
Make cleanup idempotent
Set timeouts to prevent deadlocks

Lab 4.2: Implementing Finalizers - Hands-on exercises for this lesson

References

Official Documentation

Next Steps

Now that you understand finalizers, let’s learn about watching and indexing for efficient controllers.