Hi Nasser,
The commit is essential for Mendix to be able to work with documents, that won't go away. What you could do is have the user upload to a temporary filedocument entity, then scan the file and decide to either delete it or duplicate it into the target filedocument using the relevant communitycommons java action.
I hope this helps.
Michiel
Do your check in a before commit event handler. You can't give proper feedback to the user if it fails, but if people are trying to upload infected files that's the least of your concerns.
What Nasser Khasawneh said before; once you have selected a file, the BCo runs when cancelling or trying to leave the file manager pop up as well, making the BCo option practically useless..