It's not about searching in Mendix 'filedocuments'. It's about searching in PDFs, DOCs, XLS, or whatever is stored in the filedocuments. And that is not a simple thing.
We used the Apache Lucene to index and search just PDFs in the past. But is has to be done in Java. I guess you'll need a Java expert to help you with this, there's not just a simple answer to this question. Anyway, I hope that this helps you a bit further.
I think you need to create a searchindex on those filedocuments after a commit. You can easily search in Mendix on this 'searchindex' object.
Difficult part is how to create the search index. Probably you need a java-action to read the contents of a filedocument and save this in an attribute in the search index object.
Searching on google and stackoverflow results in a couple of java-libraries which are already supporting fulltext searches in documents. Good luck!