Summary
Make the Mendix Operator responsible for cleaning up the custom resources it owns (`StorageInstance`, `Build`, `Runtime`, `Endpoint`) when the operator itself is being removed, so that namespaces don't get stuck in `Terminating` state during IaC-driven teardown.
Problem / Motivation
When a Mendix app is created on a self-managed Kubernetes cluster (connected cloud), the operator creates several custom resources in the namespace, each carrying finalizers like `privatecloud.mendix.com/storage-provisioner`. These finalizers can only be removed by the operator itself during normal reconciliation.
When the surrounding infrastructure is torn down via Terraform, the operator deployment is removed before or in parallel with the CRs it owns. Once the operator is gone, nothing can remove the finalizers, the CRs are stuck, and the namespace stays in `Terminating` indefinitely. The entire destroy operation blocks on a single namespace.
The only manual workaround today is to patch the finalizers off the stuck CRs directly:
kubectl patch storageinstance ... --type=merge -p '{"metadata":{"finalizers":[]}}'
...which is exactly the kind of thing an operator is supposed to spare its users from.
Proposed Solution
This can be solved entirely inside the operator, following the standard Kubebuilder / Operator SDK pattern:
1. The operator (or its `MendixApp` CRD / Helm release) gets a finalizer on itself.
2. On delete, the operator runs a graceful drain:
- Iterate over all owned CRs (`StorageInstance`, `Build`, `Runtime`, `Endpoint`) in its watched namespaces.
- Run the normal cleanup logic for each (deprovision storage, clean up resources, etc.).
- Remove the per-CR finalizers.
3. Only after all owned CRs are gone does the operator remove its own finalizer and allow itself to be deleted.
This is the same pattern used by e.g. cert-manager, the AWS Load Balancer Controller, and most mature Operator-SDK-based operators. It is also documented as best practice in the Kubebuilder book and the OLM guidelines.
Benefit
- `terraform destroy` / `helm uninstall` / `kubectl delete ns ...` just work, no stuck namespaces
- No manual `kubectl patch` workarounds needed in CI/CD pipelines
- Brings the Mendix Operator in line with standard Kubernetes operator best practices
Best regards,
Marvin