0

I am trying to write a function in Python that can query all Salesforce Object data. I'm utilizing the salesforce_bulk Python Library. This is an overview of what I've written:

def fetch_salesforce_object_data(bulk, object_name):

    job = bulk.create_queryall_job(object_name=object_name, contentType='JSON')
    batch = bulk.query(job, query)
    
    # Wait for batch completion
    while not bulk.is_batch_done(batch):
        time.sleep(10)
    
    bulk.close_job(job)
    
    final_result_list = []
    
    for result in bulk.get_all_results_for_query_batch(batch, job):
        result = json.load(IteratorBytesIO(result))
        df = pd.DataFrame(result)
        final_result_list.append(df)
    
    df = pd.concat(final_result_list)
    return df

This is giving me the results I want but it is running very slow. Upon debugging, I found that in the "Bulk Data Job Loads", Salesforce is returning the query results very quickly. But the code is taking way too long to execute at this line result = json.load(IteratorBytesIO(result)). Is there something we can replace IteratorBytesIO with? Is there any other way to fetch query result?

1 Answer 1

0

I went with the simple-salesforce library and did it this way and saw that it was significantly faster for me:

def getData(sf, describe_result, object_name, where_clause, limit):
    # Extract field names
    field_names_arr = [field['name'] for field in describe_result['fields']]
    field_names = ','.join(field_names_arr)
    if limit:
        data_obj = sf.query_all_iter(f"SELECT {field_names} FROM {object_name} {where_clause} LIMIT {limit}")
    else:
        data_obj = sf.query_all_iter(f"SELECT {field_names} FROM {object_name} {where_clause}")

    # Convert records to DataFrame
    df = pd.DataFrame([convert_to_dict(row) for row in tqdm(data_obj)]).dropna(axis=1, how='all')

    return df


def getTaskData(sf, where_clause, limit):
    # Specify the object you want to explore (e.g., Account, Contact)
    object_name = 'Task'  
    describe_result = sf.Task.describe() 

    # Convert records to DataFrame
    data_df = getData(sf, describe_result, object_name, where_clause, limit)
    data_df = data_df[data_df['Description'].notna()]

    return data_df

Hope this helps!

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .