Robots.txt file not crawled by Google

Question

We use a force site, with a second level domain like http://mycompany.force.com/site1

The salesforce configuration allows a robots.txt file at this address

http://mycompany.force.com/site1/robots.txt

But in the Google robots.txt specification the robots.txt file in the location is not allowed and Google will not crawl it.

As Google said "The robots.txt file must be in the top-level directory of the host, accessible though the appropriate protocol and port number".

How can we resolve this problem?

Is possible to configure a robots.txt file on the root address http://mycompany.force.com ?

Thanks in advance.

Answer 1

After some googling I found the solution.

You need to create a new Site and leave empty the web site suffix. With this solution you can also configure the favicon for your website.