Bulk API - Upload documents (files and URL) through CSV

Question

I have tried uploading files through CSV using Bulk API's createBatchWithFileattachment() call. It basically goes like this:

createBatchWithFileAttachments(JobInfo jobInfo, InputStream batchContent, File rootDirectory,
        String... files)

Documents can be either files or the URLs. This particular call is used for uploading files through CSVs. If there is a document that is of the type 'URL', the body field is empty and the call returns FileNotFoundException(). Is there a way that I can create a job that encompasses both files and non-files through Bulk API?

This is with reference for Bulk API class mentioned here:

To import binary attachments, use the following methods. Specify the CSV, XML, or JSON content for the batch in the batchContent parameter, or include request.txt in the attached files and pass null to the batchContent parameter. These methods are contained within the com.async.BulkConnection class:

createBatchFromDir()
createBatchWithFileAttachments()
createBatchWithInputStreamAttachments()
createBatchFromZipStream()

EDIT: Solution is to use the existing method mentioned (Thanks to Daniel). I am able to write a separate utility that does this:

private BatchInfo createBatchFromFiles(JobInfo job, Path fileDir, List<String> paths, Path newCsv)
            throws AsyncApiException, IOException
    {

        Map<String, InputStream> attachments = new HashMap<>();
        for (String path : paths)
        {
            Path filePath = fileDir.resolve(path);
            if (Files.exists(filePath))
                attachments.put(path, Files.newInputStream(filePath));
            else
                attachments.put(path, new ByteArrayInputStream("".getBytes()));
        }

        return ConnectionManager.getBulkConnection().createBatchWithInputStreamAttachments(job,
                Files.newInputStream(newCsv), attachments);
    }

Note that the CSV under the body field should contain the data as "#" for a document and "" (empty string) for a URL.

Answer 1

I can see in the source for BulkConnection the version of createBatchWithFileAttachments you are calling.

That method attempts to use the string file as a path relative to the rootDirectory. That wouldn't make so much sense for a URL.

Instead have a look at the alternative method:

  public BatchInfo createBatchWithInputStreamAttachments(JobInfo jobInfo, InputStream batchContent,
        Map<String, InputStream> attachments)

You will need to create your own InputStream for each file or URL to pull them in appropriately.