We are considering developing a new version of our website (where our customers can purchase and manage certain types of content) on the force.com platform. Looking into the impact of governor limits we analyzed some visitor and page request data of our current website. One of the conclusions was that there is a high likelihood that some of our customers are web scraping our website for information.

If we would build our website on the force.com platform using sites, we would have the sites limitations (40gb bandwith per rolling 24h, 60h processing page requests per rolling 24h and 1m of unauthorized page requests). Even though these limits are fairly high, robots scraping our pages will definitely increase the rate in which we reach limits (and some governor limits like callouts).

How can we analyze our webtraffic to know what customers are doing this (authenticated users over customer portal) and how can we block it ?

7 Answers 7


Use a custom controller for each of the Visualforce pages and have it track usage either directly against the contacts logged in, or in a new custom object.

I'd suggest TrackingController extends StandardController,

And then for each page specific controller extend the TrackingController.

  • This is exactly what I would do as well. And regardless of extending StandardController, you could create a TrackingController base class and extend it with other Apex controller classes that aren't tied to a specific object (like contact).
    – Adam
    Aug 3, 2012 at 14:32
  • wouldn't this block legitimate users from viewing the site if the scrapers used up the tracked limit?
    – LVS
    Aug 5, 2012 at 16:51
  • All right. I've kept it open for a while, so more input would be given, and your answer is the most supported. Aug 7, 2012 at 5:07

Look in particular at the user agent HTTP header:

String userAgent = System.currentPageReference().getHeaders().get('User-Agent');

Some scrapers might set this to emulate a real browser, but you may get some insight here.

  • Downvoted as this is not really a solution. Web scrapers that do not already set this will easily adapt to this "solution".
    – GuiSim
    Aug 6, 2012 at 13:43
  • As someone who has written web scrapers in an ajax world, I often use Selenium or another browser automater to do the scraping. There's no way to use the User Agent to tell. Rate and record limiting is the only real solution. Aug 6, 2012 at 14:47
  • 2
    Agreed - this will only catch the most naive scrapers.
    – metadaddy
    Aug 6, 2012 at 17:58

Another way to think about the underlying issue:

Is the concern just limits, or do you have another reason to prevent you customers from scraping your data? If it's just limits, could you provide a the data customers want in another fashion? For example, a csv (or even XML) file with just the data a customer wants would represent far less bandwidth than a full html page render, and potentially fewer page requests if the customers must currently scrape multiple pages to get all the data needed. Is the information is at all cachable? Could you precompute a download file, hourly or daily, and host it on AWS or Heroku for example?

  • I was thinking along this line as well. Assuming those customers doing scrapping are actually customers you want, give them an easier and more efficient way of getting the data they want. Aug 4, 2012 at 8:54
  • Giving some type of API support is on the roadmap, but not a reason to skip scraping detection. Aug 7, 2012 at 5:08

Most screen scrapers do not support javascript, so having the page load content via javascript may be an option. However if you go this route I'd avoid the rerender ing built into visualforce as it counts as an additional page view against your quota as far as sites is concerned.

You could also potentially use this by setting a cookie saying "this client is a bot" on every request from a client that does not have a cookie and then on all "this client is a bot" or uncookied requests load an intermediate page where javascript on the page changes this to a "this client is not a bot" cookie and then redirects them to your data.

All but the most persistent and well-coded bots should be put off by this approach.



  • scrapers tend to only follow links, create an image take that points to a visualforce page with an ID in the parameter. track hits to your visualforce page (custom object), if the corresponding image tag is not found repeatedly you've either got a person using lynx or a scraper.(this doubles your bandwidth)
  • rate checking, scrapers can be built to act like browsers but tend to follow standard execution patterns, ie: polling at standard intervals. by tracking every time your visualforce page is read you can determine patters and associate probable scraping to an IP.

    String ip = ApexPages.currentPage().getHeaders().get('X-Salesforce-SIP');


  • Rotate your output template. Scrapes tend to be setup using anchor points and xpaths to find the data. By restructuring your data every now and again you break their scripts. This could be done automatically by generating html in Apex and using .
  • Another method is to render content as images
  • rate limit per ip

If these users are authenticated like you say (customer portal users) then most of those limits shouldn't apply.


Doing this in Apex isn't going to address the unauthorized request / month limit. It also will incur processing costs, although if you do it right they could be minimal.

If your robot traffic is so high that you're in danger of crossing 1m hits/month, you should look at commercial CDN's (unless your app absolutely must be dynamic, but presumably not if you're getting routinely scraped). They reduce the load on your site, and they also offer services that could explicitly block your scrapers.

  • 1
    Force.com Sites already has a CDN built in, you would probably want to investigate using that and caching your page before looking elsewhere.
    – Ryan Guest
    Dec 4, 2012 at 4:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .