Requeue a kubernetes event in a non-blocking reconcile loop

Question

We have a kubernetes cluster, where a reconcile is triggered in response to a custom event. Implemented in Golang using the following format:

type reconciler struct {}

func (reconciler) Reconcile(ctx context.Context, o reconcile.Request) (reconcile.Result, error) {
    // Implement business logic of reading and writing objects here
    return reconcile.Result{}, nil
}

We have identified possible bottlenecks in the reconcile logic when the custom events are too many. So, the code has been updated to have a non-blocking reconcile logic. Example:

type reconciler struct {}

func (reconciler) Reconcile(ctx context.Context, o reconcile.Request) (reconcile.Result, error) {
    go func() {
        // Implement business logic of reading and writing objects here
    }()
    return reconcile.Result{}, nil
}

However, there are some places where the non-blocking go routine may return return ctrl.Result{Requeue: true}, nil or return ctrl.Result{RequeueAfter: someTime}, nil

How could we requeue such events to the reconcile loop in such scenarios, since the return would not return to the caller Reconcile()

Answer 1

I think a better approach would be to use concurrent reconciles allowing other requests to be handled while a single reconcile request is slow to handle. This way the reconcile request queue won't be blocked, and idle go routines are fairly cheap so it should not affect your performance all that much.

Check out the MaxConcurrentReconciles in controller-runtime: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/controller