Is there a way to modify or change the robots.txt for our public facing KB articles, so I can change which search engines can crawl them, etc?
On force.com sites you can create a custom robots.txt file and modify .
Here are some the links to assist
In nut shell to summarize from these docs ,you will need a visualforce page with below lines of code
<apex:page contentType="text/plain"> User-agent: * Disallow: / # hides everything from ALL bots Allow: /<path-prefix-1>/s # add path you want to open to bots Allow: /<path-prefix-2>/s # add path you want to open to bots Sitemap: http://<community_URL>/s/sitemap.xml Sitemap: http://<community_URL>/<sub_path>/s/sitemap.xml </apex:page>
And then attach this visualforce to site settings using sites record .