Production has a robots.txt file setup and associate to the public facing site. Now if a sandbox is refreshed or created is the live robots.txt associated to the site in the sandbox or is the default Salesforce robots.txt file used?
Unfortunately I am unable to test this myself and I have not been able to find any documentation answering this.
I am using force.com sites.
Update from Salesforce Support:
As per your query " if our Production environment force.com site has a robots.txt file setup and then a sandbox is refreshed or created does the sandbox use the production version of the robots.txt file or the default Salesforce robots.txt" on consultation with the project manager and the senior developer support the answer to this would be Yes the robot.txt file gets updated automatically to the one that is present in the production upon sandbox refresh/create.
Currently we do not have a documentation to support this but I believe that the default robot.txt will get updated to the one in production
I'll add it as an answer once I can confirm that's true.