Hi Andalam,
In order to use reference selector, a user role must have read access to the attribute used to make the selection (assuming dropdown) and read/write access to the reference i.e. association that they are trying to set.
If they do not have read access to the selecting attribute, they will see all the objects but with blank values and they will not be able to distinguish them.
Hope this helps,
Andrej