Its only possible to configure a custom login page as default for anonymous users, when anonymous users are enabled.
If anonymous users isn't enabled, then the authentication tab disappears and the default login page is used.
To add on to what Rene said, you can create a user role for anonymous users, enable the anonymous user setting, and add a role based home page in the navigation profile.
There login form widget can be used on a page to log anonymous users in.