The error is there for technical reasons. The client does not retrieve all elements of a (multi-page) grid at once and cannot send the IDs of all elements to the runtime when you click Select. For now.
A workaround is to offer 'Clear All' and 'Add All' microflow buttons in the reference set selector that is being edited.