This can be done by creating your own microflow or java action.
To expand on Stephan's answer: create a custom microflow for your save action. In this microflow, you can retrieve an object which has duplicate attributes (using the retrieve action). The XPath constraint would look like:
[id != $ObjectYouWantToCommit]
[Attribute1 = $ObjectYouWantToCommit/Attribute1]
[Attribute2 = $ObjectYouWantToCommit/Attribute2]
If you retrieve such an object, you know you should not commit the current object to the database, because it is a duplicate. You can then give feedback to the user about what is wrong. If you find no such object, you can then commit the current object to the database.
Mendix has a Unique constraint you can add to the Attribute of the Entity in your Domain Model.
Go to Domain Model -> Open Entity -> Validation Rules tab -> Click New -> Select Attribute -> Select the Unique radio button.
If you want to do this on multiple fields, I would suggest maybe a checksum (http://en.wikipedia.org/wiki/Checksum) field with this validation.