When changing a reference set you have 3 options (radiobutton in change object):
When using Set the reference set will be set to the object or list passed.
When using Add the reference set will be adjusted and the object or list is added to the list of already existing values.
When using Remove the reference set will be adjusted and the object or list is removed from the list of already existing values.
When setting the manager you should use Add instead of Set.