2

I am using simple-salesforce python library using REST to call bulk services which is working fine till calling but for large dataset library continuously poll the batch status and spit all result (in my case it is 1mn) but I am getting only 7lac records in python codebase, I double checked the load jobs which displayed the total record in processed state.

In other case when I tried the same with csv it stream out whole data with xml content type. But with rest I am getting incomplete data.

What should I suppose to do for whole data, I checked with library and found it should stream data or chunk data through loop instead giving directly.

Improve this question

1 Answer 1

Reset to default
2

simple_salesforce has a bug in its Bulk API implementation. It only retrieves the first batch of results from a Bulk Query job. This is why you do not retrieve complete results.

Consider using salesforce-bulk instead, which uses simple_salesforce for authentication but has its own, correct, implementation of the Bulk API and supports multiple result batches.

Improve this answer
4
  • Trying same but getting result through bulk.get_all_results_for_query_batch taking indefinite time for 1mn record, which is not so huge.
    – Shashank
    Mar 10, 2019 at 14:05
  • I don't know why that might be, @Shashank - it works fine for me. Feel free to open another question if you can't work it out, with some more debugging details.
    – David Reed ♦
    Mar 10, 2019 at 14:07
  • I got stuck in snippet are as: for result in bulk.get_all_results_for_query_batch(batch): result = json.load(IteratorBytesIO(result)) for row in result: print(row) It goes in indefinite loop. I am following this codebase: github.com/heroku/salesforce-bulk#bulk-query
    – Shashank
    Mar 10, 2019 at 14:32
  • @Shashank I know that that library works because I use it in my own project. I can't tell you from what you're showing here why it's not working for you. I would suggest you open a new question since this is a separate issue.
    – David Reed ♦
    Mar 10, 2019 at 19:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .