Hi Jordan,
You could indeed add an XPath constraint to your reference selector.
Open your reference selector and go to the 'Selectable objects' tab, select 'XPath' and add the following XPath constraint:
[not(IsAnonymous)]
This should filter out all the Anonymous users for you.
Hope this helps!