3

We are using Salesforce's Visualforce to run one of our websites.

We use Moz to monitor SEO issues.

Moz has identified more issues pages than we have - see the attached image. We have only a few thousand pages at best. However Google and Moz are taking keywords and adding it to the end of the domain and creating a URL that doesn't exist. Recently I made all of these URLs redirect to the home page to see if it resolved the issue - it didn't and the URLs that don't exist are still being crawled.

What do I do - this is effecting my rank. Please help?

enter image description here

3

I've seen similar issues before. There are a few ways you can tackle this

A) Add a robots.txt file where you can specify which all pages/ prefixes you want to disallow for crawling.

See this link for more details.

e.g.

A file that determines which parts of your public site Web spiders and other Web robots can access. Web robots are often used by search engines to categorize and archive websites. HTML tags are not allowed, as they are not compatible with robots.txt spiders. For Visualforce pages, add contentType="text/plain" to the <apex:page> tag.

Example disallowing all robots from indexing all pages:

<apex:page contentType="text/plain">
    User-agent: *  # Applies to all robots
    Disallow: /    # Disallow indexing of all pages
    </apex:page>

Example allowing all robots to index all pages:

<apex:page contentType="text/plain">
            User-agent: *  # Applies to all robots
            Disallow:      # Allow indexing of all pages
            </apex:page>

B) Make sure your sitemap includes only the URLs for the pages that you want the crawlers to index. Refer to this link for more details.

  • Thank you for your assistance - but does this mean I will have to disallow access to all the URLs that shouldn't be crawled because that is a huge list. I've tried updating the sitemap but it is not helping. – Trent Jan 5 '16 at 0:06
  • No, you can use wildcards to create your robots.txt file to specifically exclude the URLs that you do want to exclude from crawling – Gaurav Kheterpal Jan 5 '16 at 2:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.