Hi Nasser,
The commit is essential for Mendix to be able to work with documents, that won't go away. What you could do is have the user upload to a temporary filedocument entity, then scan the file and decide to either delete it or duplicate it into the target filedocument using the relevant communitycommons java action.
I hope this helps.
Michiel
Do your check in a before commit event handler. You can't give proper feedback to the user if it fails, but if people are trying to upload infected files that's the least of your concerns.