For anyone seeing this post.
The issue turned out to be some specific SCSS. Because we used a header image in the layout (which is then only applied to the first page), the requirement was for the other pages to have some spacing on the top of the page. Therefore custom SCSS was applied that was targeting the @page selector and applying margin to all pages except the first one, which made the page numbers of all other pages getting pushed just outside the page.
Turns out the widget created by Aiden did exactly what we needed, and made it possible to add a header image to each page, or leveraging the header to apply spacing to each page.
https://marketplace.mendix.com/link/component/212764