2

I created a CSV parser that parses CSV's from static resources.

I am returned with a Blob is not a Valid UTF-8 String error when parsing the character: "–" different from "-". The "–" is apparently still a unicode character: https://unicodelookup.com/#%E2%80%93/1.

The sheet is being saved as UTF-8 CSV and the data was exporting from Salesforce using data loader 38.0.

Can someone explain why this character is returning the Blob is not a Valid UTF-8 String even though the character appears to be a UTF-8 character?

CSV:

enter image description here

HEX with bad character: 

00:00:00:XXX USER_DEBUG [XX]|DEBUG|4f626a6563742c437573746f6d5f446563696d616c5f5f632c437573746f6d5f5f632c42726f6b656e5f4669656c645f5f630d0a4d794f746865724f626a6563745f5f632c312c412c5858209620546869732068797068656e206973206275737465640d0a

HEX without bad character: 

00:00:00:XXX USER_DEBUG [XX]|DEBUG|4f626a6563742c437573746f6d5f446563696d616c5f5f632c437573746f6d5f5f632c42726f6b656e5f4669656c645f5f630d0a4d794f746865724f626a6563745f5f632c312c412c58582020546869732068797068656e206973206275737465640d0a

||||||Update:

Upon opening the file in Notepad, it appears the file was encoded in ANSI, even though excel states the file is encoding in UTF-8 when I open it. After saving at UTF-8 in Notepad, that resolved the symbol issue.

This did not work:

enter image description here

This did:

enter image description here

With this being said are there any capabilities within the blob class or another class to convert the file from ANSI to UTF-8 within Salesforce as opposed to relying on customers to ensure the file is in the correct encoding?

Improve this question
2
  • 3
    We would need to see a hex dump of the file. Maybe there's a BOM (Byte Order Mark), or maybe it's an invalid UTF-8 encoding (i.e. it's actually encoded in a different code page/encoding). Perhaps you could encode an example with uuencode? This would help us narrow it down. Or even a simple script: System.debug(EncodingUtil.convertTohex(resource.body));
    – sfdcfox ♦
    Commented Apr 3, 2020 at 20:50
  • @sfdcfox I have added the Hex values and the CSV table data above.
    – S.B.
    Commented Apr 3, 2020 at 21:05

1 Answer 1

Reset to default
2

So, yes, the offending byte was 0x96, instead of whatever it should have been in UTF-8. There's no good way to fix that, as most methods only support UTF-8. You may try a combination of EncodingUtil.convertToHex/String.replace/EncodingUtil.urlDecode to try and get it from ASCII/ANSI to UTF-8, but you'll have to guess the encoding, which would be non-trivial. It's probably just better to catch the exception and advise the user to obtain help with getting the correct encoding.

Improve this answer

Not the answer you're looking for? Browse other questions tagged .