15

For my app I want to build a Archive and Restore functionality. For a given Custom Object MyObject__c and a Big Object ArchivedMyObject__b those requirements need to be fulfilled:

  • MyObject__c records are archived (original is deleted) using a scheduled batch.
  • ArchivedMyObject__b can be restored back into Custom Objects

While building this I ran into some conceptual/technical challenges:

  1. Should restored records be deleted as Big Objects?
  2. As the CreatedDate changes after a Restore should there be a custom OriginalCreatedDate__c?
  3. If restored Big Objects are not deleted how can I make sure I don't archive duplicates over and over again?
  4. I've read about Big Object upsert when doing Async Soql Write. Can I use this for duplicate prevention?
  5. I also read about idempotent Big Object Storage. What does it mean? Must the Index match or ALL field values?
  6. Does the index play a role at all in duplicate handling?
  7. Not every org has AsyncSOQL. Can I also batch-restore millions of records? Or what is the real reason for AsyncSOQL? Data volume, query flexibility? I don't get it.
  8. What happens if the customer reaches the 1 Mio. Big Object limit? Will it fail immediately.
1
  • 1
    for #2 - if you use the API and running user has Create Audit Fields permission wouldn't you be able to preserve CreatedDate ? Also, custom object field history would be unrestorable
    – cropredy
    Jun 18, 2021 at 21:44

1 Answer 1

6
+25

Thank you for posting such an interesting question.

Btw, Big Objects API name ends with __b, So archived Big Object API name could be ArchivedMyObject__b.

  1. Should restored records be deleted as Big Objects?

Based on your requirement you can delete restore records from Big Object. This is not automatic, for this you need to write code to delete those archived records from Big Object.

  1. As the CreatedDate changes after a Restore should there be a custom OriginalCreatedDate__c?

Yes, better to have separate custom field to hold the original value. Also if you have the autonumber field for record then that will also get changed after restoration.

  1. If restored Big Objects are not deleted how can I make sure I don't archive duplicates over and over again?

I have checked that, existing records are always be updated with same indexed values. So, there will not be any duplicates.

Reinserting a record with the same index but different data results in behavior similar to an upsert operation. If a record with the index exists, the insert overwrites the index values with the new data. Insertion is idempotent, so inserting data that exists doesn’t result in duplicates. Reinserting is helpful when uploading millions of records. If an error occurs, the reinsertion reuploads the failed uploads without duplicate data. During the reinsertion, if no record exists for the provided index, a new record is inserted.

Refer: Populate a Custom Big Object with Apex

  1. I've read about Big Object upsert when doing Async Soql Write. Can I use this for duplicate prevention?

Refer the answer above.

  1. I also read about idempotent Big Object Storage. What does it mean? Must the Index match or ALL field values?

It is compared with all the indexed values. If indexed values are same then it will be treated as upsert. You can archive same records multiple times without any fear of insertion of duplicate records.

  1. Does the index play a role at all in duplicate handling?

Yes it is and every Big Object should have an index.

  1. Not every org has AsyncSOQL. Can I also batch-restore millions of records? Or what is the real reason for AsyncSOQL? Data volume, query flexibility? I don't get it.

Async SOQL is a method for running SOQL queries when you can’t wait for immediate results. These queries are run in the background over Salesforce big object data. Async SOQL provides a convenient way to query large amounts of data stored in Salesforce. Because of its asynchronous operation, you can subset, join, and create more complex queries and not be subject to timeout limits. This situation is ideal when you have millions or billions of records and need more performant processing than is possible using synchronous SOQL. The results of each query are deposited into an object you specify, which can be a standard object, custom object, or big object.

The limit for Async SOQL queries is one concurrent query at a time.

Refer: Async SOQL

  1. What happens if the customer reaches the 1 Mio. Big Object limit? Will it fail immediately.

I never tried this. I think it will follow the same principles of limit exceed error.

3

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.