Insertions and deletions are indeed difficult to grasp in the event listeners. Best is to listen to class refreshes as well (you can trigger those easily with communitycommons) or the make your object somehow related to your current context.
In the latter case you can than refresh the 'parent' object to know that you should reload the children. (if the parent object refreshes often, you might even add an dedicated timestamp attribute to parent for this notification)