Metadata API exception: IO Exception: Exceeded max size limit of 3000000

Question

I am using Metadata API to get information about org. I want to retrieve all custom fields present in my org. I am using listMetadata() method which returns list of components. Below is my code:

MetadataService.MetadataPort service = createService();            
        List<MetadataService.ListMetadataQuery> queries = new List<MetadataService.ListMetadataQuery>();       
        MetadataService.ListMetadataQuery queryLayout = new MetadataService.ListMetadataQuery();
        //queryLayout.folder = MetaDataFolder;
        queryLayout.type_x ='CustomField';
        queries.add(queryLayout);      
        MetadataService.FileProperties[] fileProperties; 
        try{    
         fileProperties= service.listMetadata(queries, 29); //here 29 is api version
        }catch(exception e){
            ApexPages.addMessage(new ApexPages.Message(ApexPages.Severity.Info,e.getMessage()));
        }

I am get exception as "IO Exception: Exceeded max size limit of 3000000". I know that number of custom fields present in my org is to much and the response that I am getting exceeding the allowed limit of 3MB. I want to list all custom fields present in my org. Same exception is coming when I am retrieving few other metadata(like all profile). Above mentioned code works fine in my developer org but in sandbox, its not working as number of metadata components is too much in my sandbox.

I tried retrieving the components using Workbench which also uses metadata api. Workbench retrieved all components successfully. So there will be some mechanism in metadata API through which I can retrieve large number of components. I saw few post where people says to retrieve info in different chunks by spliting single request in 2 or 3 different request. But in my case, I am not able to split my request as I want all custom fields.

Copying the question listed here:https://developer.salesforce.com/forums/ForumsMain?id=906F0000000AW48IAG

Answer 1

Try breaking your request into a series of smaller requests that you can compose into a single list on the receiving end. For instance,

Ask the MdAPI for a list of objects, then iteratively ask for the fields on 10 of them at a time.

Answer 2

Is there a reason you need to use the metadata API for this? Apex has the ability to inspect custom fields without the need for callouts using the standard library. You can call Schema.getGlobalDescribe() to get a Map<String,SObjectType> for all types in the org (standard and custom). From there you can describe the fields on each and access those. The docs have a quite good section on how to do this in more depth.

This will be dramatically more scalable than the metadata API, faster, and useful in areas that callouts aren't allowed (like triggers).