How to prevent duplicate File (ContentDocument) if the same file is uploaded at multiple records?

Question

This is a day-to-day use case:

Assume there are many Opportunities on an Org. Now Sales-Reps are uploading some auxiliary information as Files in Lightning using the standard related list and there the Files section (under the hood: ContentDocument and ContentDocumentLink are created). Usually this is large stuff, like Images, Hi-Res PDFs or even Videos...

The tend to re-use these auxiliary files on many Opportunities (since they are selling the same products or services over and over again).

Now using the standard UI, identical files will be redundantly uploaded that means the ContentDocument will be created n-times with the same physical file consuming n-times the server storage. That means on each upload a new ContentDocument will be created.

What I would have expected is, that if an MD5-Hash of two files is equal, Salesforce should re-use the existing ContentDocument and just create a new ContentDocumentLink. It looks to me, that the ContentDocumentLink is just there to make multiple usages of the same file possible. But the standard UI is not using it - at least not by default.

Is there a way to upload files without these redundancies?

Answer 1

Well, Apex does have that lovely heap limit of only a few MB so you can't compare the entire content of two large files.

But the ContentVersion object does have a Checksum field. So in principle you could have a trigger or Batchable process that searches for checksum and size matches upon insertion of a ContentVersion, and if found, shares the existing file to those records instead and deletes the copy. As long as you are confident that it makes sense to pool the sharing of those two uploads together.

In general though you could also set up a library of commonly used files and have reps look up & attach the existing copy in Salesforce rather than re-upload.