How to resolve Webcallout error System.LimitException: Too many callouts: 101 when displaying data on Lightning component

Question

I'm making a webservice callout to an external webservice and displaying data on a Lightning component. Data is returned in JSON format in multiple pages. Each page has 50 records. In my apex controller class, I make multiple callouts in a for loop by specifying page number and populate the data into a list which is then displayed on the lightning page. Everything was working fine as long as the maximum nuber of pages was 100.

But in some cases, data returned from JSON has more than 100 pages. Then I get this

Error System.LimitException: Too many callouts: 101.

I saw solutions where using Batch class was suggested but in my case, I have to display all the data on a UI page. I'm not making any updates. It is just displayed on a UI page. I'm not sure how to use batch class to make more than 100 callouts and display that data on a UI page. Can someone please suggest a solution or point me in a direction on how to handle this situation.

Answer 1

You need to move some or all of your pagination to the front end, rather than trying to load the entire data set in a single Apex transaction.

Your Lightning component might, for example, fire a server action to request a load of the first 10 pages' worth of data. Once the user consumes this data (whether that means scrolling through or paging in your UI, or whatever metaphor you are using), your component can then fire another server action to request the next 10 pages of data.

In this fashion, you avoid ever firing more than 10 (or whatever cutoff you choose) callouts in a single Apex transaction. Additionally, you will probably increase the responsiveness of your component substantially, because you're not trying to load more than 5,000 records in a single server-side operation via callouts.

While it's not impossible that a batch class could be a solution here in some form or another - you haven't given us a lot of information about your use case - I don't think it's likely to be the route you need to take.

Answer 2

You can also think of using Continuation call from Lightning Component through Visualforce (use ContinuationProxy component). Refer Invoking Apex Continuations from Lightning Components

This way, you can get a chunk of data from asynchronous webservice call and store them on UI layer in JSON format and use array.slice(start, end) to display the data in a paginated way.

Just to note, the limit of response size from this each Continuation call is 1 MB.