If you render the reference selector as a drop down, you can use a custom data source for this.
Otherwise, I don't know an easy way to achieve the same. You'd need the following xpath constraint on the reference selector:
[id != '[%CurrentObject%]']
Mendix will automatically constrain the possible options using the association however, resulting in an empty list. Normally you can prevent this by setting the context to private, but I don't think this will work in situations where you have a self-reference (because you need the object in the context for the xpath constraint to work, but also need to drop it to prevent the automatic association constraint).