Hi Hartley,
Once you upload a document with the filemanager the Filedocument object will be committed.
So you can create a Before Commit event handler on your entity that will check if the file name should be rejected or not.
You can also replace the spaces in the file name with this event handler instead of rejecting it.
https://docs.mendix.com/refguide/event-handlers/