You can add microflow on button which you going to click after selecting those items. In microflow add condition element in which you can try logic for selected items should be <=5. After that add validation feedback message if condition get false or continue with microflow.
Hope this will help you.
Open your list box property. Go to data source section. In the X-path select the entity of your item then in X-path constraint add square bracket. It will be display the attribute of entity. Then select id or name of your item which is less than or equal to the number of item that you want to display.