Upgrading to Mendix 8 I seem to have lost my robots.txt again. Thus getting errors in my logs.
I would like Mendix to place a robots.txt where it should be. Having these errors and searching for information how to solve them is a waste of every users time.
I also want Mendix to give documentation of where it is located and a few different use cases. My use case is that no robots should crawl anything. That is maybe not the standard one so some more examples are probably good.
Kind regards
Johan
Nice. Thank you.
Hi Johan,
I would like to inform you that we decided to add the robots.txt to the `theme` folder. The file will allow all crawling by default. If you want to change this to disallow all crawling you should use the text:
User-agent: *
Disallow: /
The slash here will disallow any crawling starting from the root folder.
The robots.txt will be part of the Atlas module from v2.4.2 on (not released yet).