As you correctly concluded from the Java documentation, to create a single quote in the output of SimpleDateFormat, you'll need to escape the single quote with another single quote (otherwise it would be interpreted as a quoted text). However, because in a Mendix microflow a single quote is the string delimiter, you'll need to escape them with another single quote again (see Mendix documentation). So the correct microflow expression is:
formatDateTime([%CurrentDateTime%],'MMM ''''yy')
The issue is you trying to insert a quote before the 2 digit year. If you remove this it should work OK - 'MMM yy'
Even simpler, in the datagrid column or in the dataview of the entity, select the column/attribute and set the Date Format to Custom. You can then add MMM yy and it will display like that. (Note the custom date format in the dataview cannot be editable)
Remember, this is just how the date is displayed - you still need to store a full date/time in the database
Edit I did a bit more testing. It looks like the problem is only with the single quote. If you try a double quote or some other characters it works OK. There is one other character that you might use instead if the normal single quote - on my UK keyboard this character (`) is on the top left keyboard row to the left of the number 1. It is a left-single tick. This works and gives the output you want.
Try this, worked for me: formatDateTime([%CurrentDateTime%],'MMM-yy')
And if you really needed the extra quote you could use formatDateTime([%CurrentDateTime%],'MMM') + ' '''+ formatDateTime([%CurrentDateTime%],'yy')
Remember that if you want to put a single quote in a string you have to add two single quotes. So to add a space and a single quote you type single quote space single quote single quote singele quote ( ' ''')