Apex Blob data chunking

Question

I am able to upload the Blob data to the local web server. However, the I have run into the error

"Exceeded max size limit of 12000000 with request size xxxxx"

This means that the file size is huge and that I will not be able to perform a full file upload. I have to get a way to configure my server does chunk uploading feature. That part is done and I'd like to convert it to a chunk based upload.

This is the full-upload code:

public with sharing class QueueableUploader implements Queueable, Database.AllowsCallouts{

    // Properties and Constructor setting the required properties for the document. Get the body and store it here.

    public void execute(QueueableContext context)
    {
        HttpRequest req = new HttpRequest();
        HttpResponse res = new HttpResponse();
        Http http = new Http();

        req.setMethod('POST');
        // Set properties here

        req.setHeader('FileEOF', '0');
        req.setEndpoint(url);

        req.setBodyAsBlob(body);
        req.setTimeout(120000);

        res = http.send(req);
        Integer statusCode = res.getStatusCode();
        String statusMessage = res.getStatus();

        if(statusCode == 200)
        {
            DOM.Document doc = new DOM.Document();      
            doc.load(res.getBody());    
            DOM.XMLNode rootNode = doc.getRootElement();
            String errorCode = rootNode.getAttribute('errorCode', null);
            if(errorCode == '200')
            {

                String fileId = rootNode.getAttribute('fileId', null);
                System.debug('Upload success for ' + fileName + '. FileId: ' + fileId);
                //fileIdMap.put(fileName, rootNode.getAttribute('errorString', null));
            }
            else
            {
                String errorMessage = rootNode.getAttribute('errorString', null);
                System.debug('Error under upload for ' + fileName +'. Error code: ' + errorCode + '. Error Message: ' + errorMessage);
            }
        }
        else
            System.debug('Error under upload for ' + fileName + '. Status code: ' + statusCode + '. Reason: ' + statusMessage);
    }

}

This is the intended modification for chunk upload.

public with sharing class QueueableUploader implements Queueable, Database.AllowsCallouts{

    // Properties and Constructor setting the required properties for the document. Get the body and store it here.

    **// HOW TO? Constructor should chunk the blob into smaller blob units to upload.**


    public void execute(QueueableContext context)
    {
        HttpRequest req = new HttpRequest();
        HttpResponse res = new HttpResponse();
        Http http = new Http();

       // Set properties here

        // FileEOF = 1 when last chunk during chunked upload
        req.setHeader('FileEOF', '0');
        req.setEndpoint(url);

        req.setBodyAsBlob(body);
        req.setTimeout(120000);

        res = http.send(req);
        Integer statusCode = res.getStatusCode();
        String statusMessage = res.getStatus();

        if(statusCode == 200)
        {
            DOM.Document doc = new DOM.Document();      
            doc.load(res.getBody());    
            DOM.XMLNode rootNode = doc.getRootElement();
            String errorCode = rootNode.getAttribute('errorCode', null);
            if(errorCode == '200')
            {
                // Call the method chunkUpload(params)
            }
            else
            {
                String errorMessage = rootNode.getAttribute('errorString', null);
                System.debug('Error under upload for ' + fileName +'. Error code: ' + errorCode + '. Error Message: ' + errorMessage);
            }
        }
        else
            System.debug('Error under upload for ' + fileName + '. Status code: ' + statusCode + '. Reason: ' + statusMessage);
    }


    public void chunkUpload(Params params)
    {

        // Send HttpRequest using params
        // Wait for response and get errorCode
        if(errorCode == '200')
        {
            // Okay, continue to the next chunk by calling chunkUpload modifying the params
        }
        else
        {
            throw exception;
        }

    }

}

From the above modification, I have a few questions:

1. I'm missing on how to divide a Blob into smaller Blob units. How can I do that? The docs doesn't have it but in java, I can divide into smaller byte array units.

2. From a design perspective, this is a queueable instance and I am not sure whether this can be chained (may require rewording; continue this queueable job instance until all chunk upload is complete). Is this approach correct or will I run into heap limits? I am running 10 parallel jobs at a time.

Answer 1

A few issues here:

1. No APEX support to split a blob
2. If you converted the Blob to base64 with Apex, given the size of your Blob you would likely run out of heap. Even if that didn't happen, you'd have to send each base64 chunk to remote system that would have to know how to reconstruct

One alternative to consider is to send the ID of the Document (or ContentVersion) to the remote system and have the remote system use the SOAP API to fetch the document/contentVersion body.

See also