0

We have created an app that includes a Visualforce page with an Angular dev inside. We use Javascript Remoting / RemoteActions to connect the Javascript code to the Salesforce Server. We have faced two main problems: the total number of records retrieved by SOQL queries limit and the maximum MB limit. For the first one, we have solved it using the @ReadOnly annotation that allows as to do unrestricted queries against the database. But for the second one, we don’t have a solution yet. The response is bigger than 40MB because we are managing more than 200K records. Somebody know how to solve this problem or maybe a different approach?

2
  • 2
    I think a different approach will be keyed on why you are accessing 200,000 records on a page view. Are you using aggregate information? Doing some kind of filtering in JavaScript?
    – David Reed
    Mar 8, 2018 at 1:24
  • Hi David Reed. The Visualforce page has an Angular app inside really optimize to work with big amounts of data and we store the information in IndexedDB in the browser. The user can interact really fast with a map inside (zooming, scrolling, applying filters...) without waiting every time for data from the server. this is why we want to collect all the data first and store it in the Indexed DB. We have implemented some kind of pagination to get the data with waterfall requests. The problem is that the total response size is really big (15-20 for 250k records). Mar 8, 2018 at 10:16

1 Answer 1

3

Depending on your use case, you should be able to just query all of the ID values (<4.5MB worth of data) in a single query, then in subsequent callbacks, pass in about 10,000 Id values at once and collect all the results into a client-side cache. This will allow you to have a coherent initial query (because you'll have all the ID values), and then lazy load additional data from the server as you need to.

Alternatively, you can do pagination as myself and others have described in various answers on this site, other sites, blogs, and so on. There's a ridiculous amount of information you can find on the subject if you just take a moment to use a search engine. Your use case is not unique, people have done it before, and, despite warnings from others, will continue to do so in the future.

Browsers are amazing pieces of software, but asking it to load 200,000+ records in memory and render it all is just asking for a performance nightmare. You'll get much better performance if you load the data on demand, and unload it when you're done with it. Taking my ID suggestion above, you can have a small cache of ID values, then go up to the server and get just the data you want to show when the time is right.

3
  • Hi sfdcfox. Thank you for your detailed answer. We already have some kind of pagination implemented. We want the all the data first to create a map with all the Accounts geolocated. The performance, in that case, is really good in the browser. This is not the problem. The problem is the aggregate response time. We have already some pagination (request in a waterfall) but we are thinking about creating parallel request to reduce the waiting time. Mar 8, 2018 at 10:26
  • But maybe there is a way to overcome MB limit for every response. Our pagination makes a number of requests in a waterfall. The first idea is to change that and create parallel requests, but we wanted to ask first the community to see if there was a much better approach. Mar 8, 2018 at 10:33
  • @RobertoSolís You can't overcome the limits, which is why they're called limits. You should be able to make the data requests parallel, but the initial ID grab should be in serial, since it's all one query with a single cursor. Depending on your browser, I'd guess you can probably make 2-4 parallel calls at once. Since you know there's 40mb of data, I would split the data into 8 chunks and get 5mb responses. You'll get better performance that way.
    – sfdcfox
    Mar 8, 2018 at 15:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.