Encoding and Decoding BLOB due to error 'BLOB is not a valid UTF-8 string'

Question

I try to encode and decode a BLOB in order to avoid the 'BLOB is not valid UTF-8 string' error, but when I encode the BLOB in UTF-8, decode it again and use the .toString-method, I stil get the error. Here is my code:

global Messaging.InboundEmailResult handleInboundEmail(Messaging.InboundEmail email, Messaging.InboundEnvelope env){


   Messaging.InboundEmailResult result = new Messaging.InboundEmailResult();

   for(Messaging.InboundEmail.BinaryAttachment att : email.binaryAttachments){
      Blob csvBlob = att.body;
      String csvBody = EncodingUtil.base64Encode(csvBlob);
      Blob csvBlobAfter = EncodingUtil.base64Decode(csvBody);
      String csvString = csvBlobAfter.toString();

      system.debug(logginglevel.debug, 'Attachment Body '+csvString);
   }

}

The csv file was a test file and looks like this:

foo,bar
Hello,World

It's properties are "Microsoft Excel-CSV-File (.csv)"

What am I missing here?

As a further step, considering the answer to my question, and I do have a signature in my email, I tried:

  for(Messaging.InboundEmail.BinaryAttachment att : email.binaryAttachments){
    if(att.mimeTypeSubType != 'text/csv') {
      System.debug(logginglevel.debug, 'Attachment is not text');
      System.debug(logginglevel.debug, 'Attachment Body '+att.body);
      continue; // not a text/csv file! //
    }
    // Remove BOM //
    System.debug(logginglevel.debug, 'Attachment Body after text check'+att.body);
    String encoded = EncodingUtil.base64encode(att.body);
    System.debug(logginglevel.debug, 'encoded '+encoded);
    Blob decoded = EncodingUtil.base64Decode(encoded.removeStart('feff').removeStart('fffe'));
    String csvText = decoded.toString();

    system.debug(logginglevel.debug, 'csvText '+csvText);

  }

This gives me the following debug log:

The debug log is the same also when I uncheck 'Convert Text Attachments to Binary Attachments' in the Email Services Settings.

This code however gave me the error 'BLOB is not a valid UTF-8 string' and a different debug log:

for(Messaging.InboundEmail.BinaryAttachment att : email.binaryAttachments){
   if(att.mimeTypeSubType != 'text/csv') {
      System.debug(logginglevel.debug, 'Attachment is not text');
      System.debug(logginglevel.debug, 'Attachment Body '+att.body);

      String encoded = EncodingUtil.base64encode(att.body);
      system.debug(logginglevel.debug, 'encoded '+encoded);
      Blob decoded = EncodingUtil.base64Decode(encoded.removeStart('feff').removeStart('fffe'));
      String csvText = decoded.toString();

      system.debug(logginglevel.debug, 'csvText '+csvText);

   }

}

After the suggestion to use hex instead of base64 I used this code:

  for(Messaging.InboundEmail.BinaryAttachment att : email.binaryAttachments){

    System.debug(logginglevel.debug, 'Attachment Body '+att.body);
    String encoded = EncodingUtil.convertToHex(att.body);
    System.debug(logginglevel.debug, 'encoded '+encoded);
    Blob decoded = EncodingUtil.convertFromHex(encoded.removeStart('feff').removeStart('fffe'));
    String csvText = decoded.toString();
    system.debug(logginglevel.debug, 'csvText '+csvText);

  }

I got the same error 'BLOB is not a valid UTF-8 string' with this debug log:

Then I tried: encoded.removeStart('ffd8ffe') and got the error: System.InvalidParameterValueException:

input string must be an even number of characters long, but was 80501

Answer 1

Encoding and decoding is just wasting time; the transformation is lossless and won't wipe out invalid characters. It's possible that you have an extra attachment incoming (e.g. if you have an inline image in your signature, it would appear as an attachment). Make sure you're dealing with a text file:

for(Messaging.InboundEmail.BinaryAttachment att : email.binaryAttachments){
    if(!att.mimeTypeSubType.startsWith('text/')) {
        continue; // not a text file! //
    }

}

There may also be a BOM (Byte Order Mark), if so, we need to take this out, too:

for(Messaging.InboundEmail.BinaryAttachment att : email.binaryAttachments){
    if(att.mimeTypeSubType != 'text/csv') {
        continue; // not a text/csv file! //
    }
    // Remove BOM //
    String encoded = EncodingUtil.convertToHex(att.body);
    Blob decoded = EncodingUtil.convertFromHex(encoded.removeStart('feff').removeStart('fffe'));
    String csvText = decoded.toString();
}

Answer 2

Same as previous answer, but added other BOM:

Blob csvBlob = [...];

// Remove BOM //
String encoded = EncodingUtil.convertToHex(csvBlob);
Blob decoded = EncodingUtil.convertFromHex(
encoded
    .removeStart('0000feff') // UTF-32, big-endian
    .removeStart('fffe0000') // UTF-32, little-endian
    .removeStart('feff')     // UTF-16, big-endian
    .removeStart('fffe')     // UTF-16, little-endian
    .removeStart('efbbbf')   // UTF-8
);