Python: Object of type bytes is not JSON serializable

Question

I am attempting to move ContentVersion records from one Salesforce org to another using the simple-salesforce python library.

I am querying the VersionData and returning it in r.content here:

import json
import requests
import base64
import chardet


headers = {
        "Authorization": "OAuth " + source_sf.session_id,
        "Content-Type": "application/octet-stream"
    }


for index, row in content_doc_source_data_df.iterrows():
  r = requests.request('get', source_sf.base_url+'sobjects/ContentVersion/'+row['Id']+'/VersionData', headers=headers)
  content_doc_source_data_df.at[index, 'File Body'] = r.content
  break

content_doc_source_data_df.drop('Id', axis=1, inplace=True)
content_doc_source_data_df.drop('ContentDocumentId', axis=1, inplace=True)
content_doc_source_data_df.drop('ContentLocation', axis=1, inplace=True)
content_doc_source_data_df.drop('ContentUrl', axis=1, inplace=True)
content_doc_source_data_df.drop('IsLatest', axis=1, inplace=True)
content_doc_source_data_df.drop('VersionData', axis=1, inplace=True)

content_doc_source_data_df = content_doc_source_data_df.rename(columns={'File Body': 'VersionData'})

display(content_doc_source_data_df)

Which I can tell is returning the bites data, which you can see here:

However, whenever I then attempt to resend it to the new org using the API I get the following error: "TypeError: Object of type bytes is not JSON serializable"

Here's the code for sending:

#INSERT FILE
import ast
content_doc_source_data = content_doc_source_data_df.to_dict('records')

results = source_sf.bulk.ContentVersion.insert(content_doc_source_data, batch_size=10000,use_serial=True)
display(results)

Answer 1

You should be following the steps found in Inserting a Document with Blob Data. That is, your payload must be a multipart/form-data content, with one element called entity_document with the appropriate JSON, and a second element that has the name VersionData with binary data.

You can't upload files using the Bulk API. Even if you could, note that the limits are pretty strict:

A batch can contain a maximum of 10,000,000 characters for all the data in a batch.

A field can contain a maximum of 32,000 characters.

Are your PDF files smaller than about 24KB? If not, they won't fit in the field limit.

To be honest, I don't think you can use simple-salesforce here (I think that's what you're using?). You'll want to just manually construct a multipart/form-data payload and use the appropriate endpoint (POST /services/data/v55.0/ContentVersion).