1

If I export a file of Custom_Object__c with a Custom_Field__c (developer console and salesforce page layout displayed) value of:

XX – XYZ

I receive an exported CSV with: 

XX � XYZ

If I open that CSV with Excel, it renders as –. If I open with VS Code it shows as �. Same with certain ' symbols.

If I import via data loader, it results in the –, however with CSV parsing it inserts as �.

I'm aware this is an encoding issue, these are not UTF-8 supported characters, but I'm looking to find out the difference between CSV parsing (in apex) and data loader that allows for this conversion and support of non-UTF 8 characters. Is it the use of Bulk API vs. Apex? Java vs. Apex?

Is there any way I can utilize the same conversion process that is within dataloader in apex?

Improve this question

1 Answer 1

Reset to default
2

All strings in Apex and on the Salesforce platform are encoded in UTF-8, full stop. All Salesforce APIs accept input in UTF-8.

If you upload a text file that is not UTF-8 into a File object and attempt to process that file's content in Apex, you may encounter multiple different negative outcomes, or no change at all, depending on exactly what your code does. Text that cannot be interpreted as UTF-8 will result in a StringException. Text that can be interpreted wrongly as UTF-8 may simply be mangled.

Every other application you might use to open a text file has its own semantics around how encodings are handled. Excel is notorious for making things difficult when working with CSV files that contain non-ASCII characters. I generally recommend the free and open source LibreOffice, which offers an encoding selector window every time you open a CSV so that you can ensure its content is interpreted and saved correctly. That said, it sounds like Excel is probably working for you here, unless the issues with unknown characters are manifesting after you've opened and resaved the file in Excel.

In Visual Studio Code, an encoding selector is available in the status bar at the base of the editor UI. Make sure it's set to UTF-8 when working with files from Salesforce.

For files destined to go into Salesforce, you may need to convert the encoding to UTF-8 from whatever is used by the software that generated the file, either in that source software or using some other tool to perform the conversion. The details will be dependent on what tools you are using.

In summary: use UTF-8 everywhere, and use only tools that fully support UTF-8. And if at all possible, don't parse CSVs in Apex.

Improve this answer
4
  • This makes sense and I appreciate the through answer, what I’m struggling to piece together is how I’m exporting a file with data loader, opening it in VS with encoding set to UTF-8 (not UTF-8 BOM or any other flavor), and after converting this exported CSV (no modifications to the original CSV from data loader/ no re-save) to a static resource, grabbing the resources body, and converting this body to a string I get the symbol mentioned above. Since everything is stored UTF-8, I don’t understand why it exports and re-imports differently whether it’s through data loader vs. apex.
    – S.B.
    Commented May 13, 2020 at 1:45
  • It certainly sounds like you're doing. everything right. But there's a lot of moving parts in what you describe. If you can pin down the mutation to a specific step and contrive a minimal reproduction, it will be a lot easier to find the cause.
    – David Reed
    Commented May 13, 2020 at 3:37
  • I came across an interesting setting within data loader that I believe may be the root cause, it provides the settings to read / write all CSVs with UTF-8 encoding. Since everything in Salesforce is stored in UTF-8, why even provide these options? If it's stored in SF as UTF-8 how would it export as any other encoding than UTF-8?
    – S.B.
    Commented May 19, 2020 at 15:45
  • I'm not sure about the exact behavior of that Data Loader option.
    – David Reed
    Commented May 19, 2020 at 18:04

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .