We were finally able to figure out what was causing this problem and address it. Below is the answer we found for the solution if anyone else runs into something like this in the future. Mind you, we’re working in a private cloud solution.
Here's a small breakdown of what seems to be causing the 502s:
- Mendix uses a keepalive connection during which it barrages Nginx with 100+ requests
- Our Nginx has version 1.19.9 setup on ALL affected clusters
- Nginx has keepalive_requests parameter, which sets the maximum number of requests that can be served through one keep-alive connection
- Version 1.19.9 has keepalive_requests set to 100(refer back to point 1 to figure out where i'm going with)
- So, whenever you start working on mendix app, it creates a keepalive connection and after reaching 100 requests the connection is closed
- That leads to the error developers are seeing in the UI and dev tools
How to fix -
- Run kubectl edit cm nginx-ingress-ingress-nginx-controller -n kube-system
- Add keep-alive-requests: "1000" under data
- Run kubectl rollout restart deploy nginx-ingress-ingress-nginx-controller -n kube-system to have ingress restart and pick up updated configmap
- Verify that 502 error is gone.
Links - https://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_requests
Best of luck!