Include robots.txt by default - Mendix Forum

Include robots.txt by default

8

Upgrading to Mendix 8 I seem to have lost my robots.txt again. Thus getting errors in my logs.

 

 

I would like Mendix to place a robots.txt where it should be. Having these errors and searching for information how to solve them is a waste of every users time.

I also want Mendix to give documentation of where it is located and a few different use cases. My use case is that no robots should crawl anything. That is maybe not the standard one so some more examples are probably good.

Kind regards

Johan

asked
2 answers

Nice. Thank you.

Created

Hi Johan,

I would like to inform you that we decided to add the robots.txt to the `theme` folder. The file will allow all crawling by default. If you want to change this to disallow all crawling you should use the text:

User-agent: *
Disallow: /

The slash here will disallow any crawling starting from the root folder.

 

The robots.txt will be part of the Atlas module from v2.4.2 on (not released yet).

Created