How about sites on a dev org or partner edition orgs?
2 Answers
Yes, a web crawler could reach them. If someone can access the Force.com site by typing the URL into the address bar of their browser then a web crawler (computer program) could access it as well.
You can set a robots.txt file to specify what you want or don't want indexed and all web crawlers that are written to honor robots.txt directives will follow it. However, there is nothing technically stopping a computer program from accessing your site if it is written to do so.
You can view the robots.txt file for your Force.com Site at http://thedomain.com/sitename/robots.txt (e.g., http://abc-developer-edition.na.15.force.com/testsite/robots.txt. If you don't set up a robots.txt file on your Force.com Site in a dev org you get the default. The default is to disallow everything:
#
# robots.txt for sfdc appservers.
#
# For use by salesforce.com
#
User-agent: * # applies to all robots
Disallow: / # disallow indexing of all pages
The Salesforce help has information about setting up a robots.txt file for your Force.com Site. From the help page:
A file that determines which parts of your public site Web spiders and other Web robots can access. Web robots are often used by search engines to categorize and archive websites. HTML tags are not allowed, as they are not compatible with robots.txt spiders. For Visualforce pages, add
contentType="text/plain"
to the<apex:page>
tag.Example disallowing all robots from indexing all pages:
<apex:page contentType="text/plain"> User-agent: * # Applies to all robots Disallow: / # Disallow indexing of all pages </apex:page>
Example allowing all robots to index all pages:
<apex:page contentType="text/plain"> User-agent: * # Applies to all robots Disallow: # Allow indexing of all pages </apex:page>
Yes. If you've got requirements to not have that happen, you can also ip-restrict the site to only allow certain ranges to view the site.