Why is the Salesforce developer documentation NOT indexed on Google anymore?

Question

For years now I'm working frequently with some Salesforce developer documentations, e.g.:

• Apex Developer Guide
• Visualforce Developer Guide

The best and easiest way in my workflow to find stuff (in general) was to use some reasonable keywords on Google, like "apex map" oder "apex list" or "visualforce repeat" and so on. This worked very well!

I was used to get high quality deep links (at least within TOP 3) as results for searches like:

• searching with Google for apex map class ==> TOP 1 was https://developer.salesforce.com/docs/atlas.en-us.apexcode.meta/apexcode/apex_methods_system_map.htm
• searching with Google for visualfore apex repeat ==> TOP 1 was https://developer.salesforce.com/docs/atlas.en-us.pages.meta/pages/pages_compref_repeat.htm?search_text=repeat

Why I did this?

I did intentionally NOT use the buildin-search (inside the documentation provided by Salesforce), because the results were usually way worse than Google, it takes more keystrokes and clicks to use it and the way the resultset is displayed is simply very inconvenient (tiny listview, severe need of scrolling, painful handling) and before you can search you have to open the documentation itself (way slower than goolge showing results)

Issue:

Now a short time ago (can't say exactly when it started) searching on Google is not bringing the expected hits as stated above within the TOP 10 results. It looks to me, as if the documentation not longer indexed correctly or not index at all by the Google bot.

This situation is bad for learners and beginners. They should find the official documentation using the search engine the will most likely use.

Questions:

• Any tips and tricks on tweaking my search terms in order to get back the expected results? I've tried already a lot of combinations (e.g. "salesforce visualforce apex:repeat" which really should bring this result close to top 1) but the official documentation is pretty resistant against being found via Google.
• Are there others with the same experience?
• Does someone know to whom at Salesforce and over which channels we could reach out and address this issue? (I have no premier support, and my questions to anything close to development will be shut down when I open a case)

What I checked so far

The URLs of the deep-content have not been changed (as it happened some time ago causing a lot of trouble). I could check this, because I've lots of deep-link-bookmarks collected - all of them working correctly with no redirect noticed. If the URLs had changed, it could be just a question of time the google bot needs to re-index the documentation again, but since the links are the same I'm expecting no improvement just by further waiting.

Workaround

As @JamesSullivan stated in his comment the search engine https://duckduckgo.com is doing better than Google. I can confirm that searching for apex map class salesforce brings up the right match on TOP 1. But it feels slightly less powerful as Google felt before and I need to feed one or two keywords more to get such precise matches as I was used to.

Extra note

Google for apex map class salesforce (like duckduckgo above) brings at TOP 1 a poor match to the root https://developer.salesforce.com/docs/ (not to the deep content) and it shows up in Japanese or Chinese for me by default. I think this might have been the link I'm after, but got crippled somehow... (but not sure)

Conclusion

I think only Salesforce can help.

It seems to me, that maybe the google bot is just no longer crawling the documentation - maybe because of a bad robots.txt or something similar, which might be easy for Salesforce to change back as it was a short while ago...

Answer 1

Thanks to everyone in the community for calling this to our attention.

We've noticed that some of our documentation has fallen out of Google's index, or fallen below first-page results. We're currently investigating what may have caused this. We did release our Summer 17 (API v40.0) documentation on Monday 22 May, and are trying to determine if anything in the metadata could've affected our placement in the results.

As many of you know, predicting how or why Google may change their results algorithms is a dark art, but we'll do our best to understand what happened, and how we might be able to solve it.

I will keep this thread up-to-date with information as I have it.

Update 1 2017-05-26: We've requested a re-index of the /docs/ pages through Google Webmaster tools. We have no information on when that crawl will occur, or how long it will take.

Update 2 2017-05-30: We can't confirm status of the re-index, although credible sources inform us that it is likely complete. Unfortunately, we are still seeing pages being dropped from Google's index. Our current recommendation is to use an alternative search engine to find the documentation content you're looking for (As mentioned in the Question, DuckDuckGo seems to have a good index of our content, as does Bing.com)

Our plan is to inspect for similar behavior on the Salesforce Help & Training site, which is driven by similar metadata, and see if they are also observing pages being de-indexed. If anyone has noticed help.salesforce.com pages being de-indexed, please let me know. Keyword searches that no longer come up with the expected content would be greatly appreciated.

We are also engaged with our Web Search and Analytics teams to try and identify anything on the affected pages that could be causing the problem. Thank you for your patience, and please let us know if you discover any other anomalies that may help us get this fixed.

Update 3 2017-05-31 Small update, we're currently investigating removing all occurrences of the "noindex" robots meta directive, and revising our canonical linking strategy. We're also looking into a non-dynamic content loading strategy, although this is currently not as high on the priority list.