1

We've included Chinese characters on our Account.Name records. These characters display correctly as expected however when they are being retrieved using Apex, their encoding becomes UTF-8. This is an issue because we are currently trying to retrieve these record names and put them in Text Files inside ZIPs.

When these Chinese characters are hardcoded as a String, they will be shown correctly. The problem happens when the String is retrieved dynamically using Apex. For example:

Object: Account

Account.Id: 0012w000005nwmtAAA

Account.Name: SFDC Test 公司

Expected Output:

Account name:
SFDC Test 公司

Actual Output:

A c c o u n t n a m e :

S F D C T e s t lQøS

I'm trying to show them as Unicode/UTF-16, so Chinese characters become legible.

Is there a way to retrieve string data of our records using Apex without their encoding transforming to UTF-8?

Appreciate any suggestions on this.

Thank you!

UPDATED: Added code and example.

Everything starts once the onclickDownload() method fires.

JS

// ------------------------------------------------------

function encodeUTF16LE(str) {

  var byteArray = new Uint8Array(str.length * 2);
  for (var i = 0; i < str.length; i++) {
      byteArray[i*2] = str.charCodeAt(i) // & 0xff;
      byteArray[i*2+1] = str.charCodeAt(i) >> 8 // & 0xff;
  }

  return String.fromCharCode.apply( String, byteArray );
}

// ------------------------------------------------------

onclickDownload() {

  // ----------------------------------------------------

  downloadFile({
    argv: argv
  })
    .then((result) => {

      // ------------------------

      if (resultData.downloadDataMap) {
        const downloadDataMap = resultData.downloadDataMap;
        this.logger.debug(funcName, "downloadDataMap:", downloadDataMap);

        Object.keys(downloadDataMap).forEach(function(downloadDataFilename) {
          downloadDataMap[downloadDataFilename] = encodeUTF16LE(downloadDataMap[downloadDataFilename]);
        });
        this.logger.debug(funcName, "encodeUTF16LE(downloadDataMap):", downloadDataMap);

        this.downloadDataMap = downloadDataMap;
      }
      
      // ------------------------

      this.onArchiveFile();

    });
}
// ----------------------------------------------------------------

onArchiveFile() {

  // ----------------------------------------------------

  archiveFile({
    argv: argv
  })
    .then((result) => {

       // ------------------------

        if (resultData.downloadData) {
          const downloadData = resultData.downloadData;

          const filename = downloadData.filename;
          const content = downloadData.content;
          var link = document.createElement("a");
          link.href = "data:application/zip;base64," + content;
          link.download = filename;
          link.click();
        }

        // ------------------------

      this.onclickClose();
    });
  }

}

Apex class

// --------------------------------------------------------------

@AuraEnabled(cacheable=false)
public static Object downloadFile(String argv)
{
    // ------------------------------------------------------------------
    
    Map<String, Object> downloadDataMap = new Map<String, Object>();
        
    // ------------------------------------------

    String accountId = '0012w000005nwmtAAA';
    String filename = 'TEST.txt';
    String filedata = '';

    filedata += 'Account Name: ' + '\r\n';
    filedata += getAccountChineseName.retrieveAccountName(accountId) + '\r\n';
        
    context.put('downloadDataStr', filedata);
        
    downloadDataMap.put(filename, filedata);
        
    // ------------------------------------------
        
    context.put('downloadDataMap', downloadDataMap);
        
    // ------------------------------------------
        
    return res.responseJson(context);
    
    // ------------------------------------------------------------------
}

// ----------------------------------------------------------------

@AuraEnabled(cacheable=false)
public static Object archiveFile(String argv)
{
    // ------------------------------------------------------------------
    
    Map<String, Object> fileMap = req.getAsMap('downloadDataMap');

    if(fileMap == null || fileMap.size() == 0)
    {
      throw new IllegalArgumentException('Missing downloadDataMap');
    }
        
    // ------------------------------------------
    
    Zippex resZip = new Zippex();
        
    // ------------------------------------------
        
    for(String filename: fileMap.keySet())
    {
      String filevalue = (String) fileMap.get(filename);
      resZip.addFile(filename, Blob.valueOf(filevalue), null);
     }
        
    // ------------------------------------------
    
    Map<String, Object> data = new Map<String, Object>();
        
    data.put('filename', 'SF-' + DateTime.now().format('yyyyMMdd-hhmmss') + '.zip');
    data.put('content', EncodingUtil.base64Encode(resZip.getZipArchive()));
        
    context.put('downloadData', data);
        
    // ------------------------------------------
        
    return res.responseJson(context);
}

// ----------------------------------------------------------------------------

public with sharing class getAccountChineseName {

  // RETRIEVE ACCOUNT NAME
  @AuraEnabled(cacheable=true)
  public static String retrieveAccountName(String accountId){

      String accountName = [SELECT Id, Name FROM Account 
      WHERE Id = :accountId LIMIT 1].Name;

      return accountName;
  }

}

// -----------------------------------------------------------

On Debug, this is what is shown:

enter image description here

3
  • I'm not sure I understand the problem. Is the software reading the text files not UTF-8 compliant? Note that hard-coded strings in Apex are also UTF-8.
    – David Reed
    Jul 9, 2020 at 18:21
  • 1
    Could you share a minimal code sample we can reproduce in another org with copy-pastable Chinese text?
    – David Reed
    Jul 9, 2020 at 18:26
  • Interesting, for some reason when strings with Chinese text are retrieved using Apex, they get messed up at the final output. I've updated the post with code and more context. I wish I could explain it better but I hope it helps. Thanks! @DavidReed Jul 11, 2020 at 7:23

1 Answer 1

0

This algorithm for converting UTF-8 text into UTF-16LE is not correct.

function encodeUTF16LE(str) {

  var byteArray = new Uint8Array(str.length * 2);
  for (var i = 0; i < str.length; i++) {
      byteArray[i*2] = str.charCodeAt(i) // & 0xff;
      byteArray[i*2+1] = str.charCodeAt(i) >> 8 // & 0xff;
  }

  return String.fromCharCode.apply( String, byteArray );
}

UTF-8 represents Unicode characters using up to four bytes. Most Chinese characters (disclaimer: I do not read Chinese and am not an expert on working with text in that language) fall in the Basic Multilingual Plane, which is encoded with three bytes, not two. In particular, the two-character string you're testing with, 公司, is encoded as a total of six bytes in UTF-8.

The UTF-8 encoding uses at most the lower seven bits of each byte, with one or two high bits used as padding or byte-length indicators. Wikipedia has a detailed article that describes the exact bit layout of a UTF-8 encoded string.

Since your debug logs show that you already have a valid, printable string value containing Chinese characters in JavaScript, it's not clear to me why you need to do this conversion at all. However, I think we can conclusively say that retrieving the data via Apex is not the culprit.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .