3

I run support.telect.com. We use the Public Knowledge Base app. As part of this, from a report, we can see what terms people search, and then if they don't find anything we can create results articles specifically for them. However when looking at the report of terms searched, it seems like bots are making searches. In the month of November we had 321 unique users to the site, according to Google, and many of those were search users. However we had over 2500 searches made on the site during the month according to the report, many of them duplicates from days before. However all of the duplicates are valid searches.

It seems like bots are making these searches to me, and I would like to stop them. Right now it is difficult to see what are actual searches by users in need of help, and what are bots. Please let me know if you have any questions.

Improve this question

1 Answer 1

Reset to default
3

Have you tried modifying your robot.txt? This only works if the bot respects the robot.txt file..

Could web crawlers reach my sandbox force.com sites?

Improve this answer
3
  • Wouldn't that affect how I show up in Google searches, etc.? Sorry, I'm just a mechanical engineer in charge of the website, so not really smart with stuff like this. Also, these bots can fill in and make searches?
    – Brad
    Dec 2, 2014 at 19:25
  • Using robots.txt is a great way to block well-behaved bots, like Google. And yes, they can full out forms. Bots are capable of navigating almost any type of site. Use the robots file to limit then just to the parts you expect them to index.
    – sfdcfox
    Dec 3, 2014 at 2:24
  • Great, I'll start making the edits. thanks for the help.
    – Brad
    Dec 4, 2014 at 16:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .