7

We are considering developing a new version of our website (where our customers can purchase and manage certain types of content) on the force.com platform. Looking into the impact of governor limits we analyzed some visitor and page request data of our current website. One of the conclusions was that there is a high likelihood that some of our customers are web scraping our website for information.

If we would build our website on the force.com platform using sites, we would have the sites limitations (40gb bandwith per rolling 24h, 60h processing page requests per rolling 24h and 1m of unauthorized page requests). Even though these limits are fairly high, robots scraping our pages will definitely increase the rate in which we reach limits (and some governor limits like callouts).

How can we analyze our webtraffic to know what customers are doing this (authenticated users over customer portal) and how can we block it ?

6

Use a custom controller for each of the Visualforce pages and have it track usage either directly against the contacts logged in, or in a new custom object.

I'd suggest TrackingController extends StandardController,

And then for each page specific controller extend the TrackingController.

| improve this answer | |
  • This is exactly what I would do as well. And regardless of extending StandardController, you could create a TrackingController base class and extend it with other Apex controller classes that aren't tied to a specific object (like contact). – Adam Aug 3 '12 at 14:32
  • wouldn't this block legitimate users from viewing the site if the scrapers used up the tracked limit? – Vid L Aug 5 '12 at 16:51
  • All right. I've kept it open for a while, so more input would be given, and your answer is the most supported. – Samuel De Rycke Aug 7 '12 at 5:07
2

Look in particular at the user agent HTTP header:

String userAgent = System.currentPageReference().getHeaders().get('User-Agent');

Some scrapers might set this to emulate a real browser, but you may get some insight here.

| improve this answer | |
  • Downvoted as this is not really a solution. Web scrapers that do not already set this will easily adapt to this "solution". – GuiSim Aug 6 '12 at 13:43
  • As someone who has written web scrapers in an ajax world, I often use Selenium or another browser automater to do the scraping. There's no way to use the User Agent to tell. Rate and record limiting is the only real solution. – RickMeasham Aug 6 '12 at 14:47
  • 2
    Agreed - this will only catch the most naive scrapers. – metadaddy Aug 6 '12 at 17:58
1

Another way to think about the underlying issue:

Is the concern just limits, or do you have another reason to prevent you customers from scraping your data? If it's just limits, could you provide a the data customers want in another fashion? For example, a csv (or even XML) file with just the data a customer wants would represent far less bandwidth than a full html page render, and potentially fewer page requests if the customers must currently scrape multiple pages to get all the data needed. Is the information is at all cachable? Could you precompute a download file, hourly or daily, and host it on AWS or Heroku for example?

| improve this answer | |
  • I was thinking along this line as well. Assuming those customers doing scrapping are actually customers you want, give them an easier and more efficient way of getting the data they want. – Daniel Ballinger Aug 4 '12 at 8:54
  • Giving some type of API support is on the roadmap, but not a reason to skip scraping detection. – Samuel De Rycke Aug 7 '12 at 5:08
1

Most screen scrapers do not support javascript, so having the page load content via javascript may be an option. However if you go this route I'd avoid the rerender ing built into visualforce as it counts as an additional page view against your quota as far as sites is concerned.

You could also potentially use this by setting a cookie saying "this client is a bot" on every request from a client that does not have a cookie and then on all "this client is a bot" or uncookied requests load an intermediate page where javascript on the page changes this to a "this client is not a bot" cookie and then redirects them to your data.

All but the most persistent and well-coded bots should be put off by this approach.

| improve this answer | |
1

determination

  • scrapers tend to only follow links, create an image take that points to a visualforce page with an ID in the parameter. track hits to your visualforce page (custom object), if the corresponding image tag is not found repeatedly you've either got a person using lynx or a scraper.(this doubles your bandwidth)
  • rate checking, scrapers can be built to act like browsers but tend to follow standard execution patterns, ie: polling at standard intervals. by tracking every time your visualforce page is read you can determine patters and associate probable scraping to an IP.

    String ip = ApexPages.currentPage().getHeaders().get('X-Salesforce-SIP');
    

action

  • Rotate your output template. Scrapes tend to be setup using anchor points and xpaths to find the data. By restructuring your data every now and again you break their scripts. This could be done automatically by generating html in Apex and using .
  • Another method is to render content as images
  • rate limit per ip
| improve this answer | |
1

If these users are authenticated like you say (customer portal users) then most of those limits shouldn't apply.

| improve this answer | |
0

Doing this in Apex isn't going to address the unauthorized request / month limit. It also will incur processing costs, although if you do it right they could be minimal.

If your robot traffic is so high that you're in danger of crossing 1m hits/month, you should look at commercial CDN's (unless your app absolutely must be dynamic, but presumably not if you're getting routinely scraped). They reduce the load on your site, and they also offer services that could explicitly block your scrapers.

| improve this answer | |
  • 1
    Force.com Sites already has a CDN built in, you would probably want to investigate using that and caching your page before looking elsewhere. – Ryan Guest Dec 4 '12 at 4:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.