Hi Harry,
We are currently experiencing some challenges with this and are trying various solutions. We saw that on a small node, 200.000 objects caused our server to crash and 50.000 objects made the experience pretty horrible for our users.
A straight-forward solution is to delete cache objects that are not needed. For example, when you create a filter object for a user, delete the previous one for that user.
However, with many users and complex filters, this could still lead to the 50.000 (as it did for us). We are currently building solutions that directly query the database (using OQL) and render it on the front end (using custom widgets). This brings down the cached objects to a minimum and works really well for us so far.
Hope this helps!