Looking in my system log I noticed an error that appears directly after starting up the environment. It says 'robots.txt not found'. I can't remember having seen this during previous log-inspections. What is this about?
1 answers

Robots.txt is a file where you can specify what webcrawlers can/can't do on your webpages. Preventing them from indexing your site/app. 

Heres's some more information on it.

There are some older questions related to robots.txt on the forum. Where it states that robot.txt can be, or should be found in the Theme. But a quick search in my own projects didn't show any robots.txt file.