6

I'm using the REST API for a texting app (Lightning components). When I send a messages that includes special characters 妮 暗示 or 😀, the characters go out correctly and get saved to my custom object Text field correctly. However, if receiving a text, the special characters are gibberish -- not Unicode or UTF-8 encodings, but ððð or ç±è°è°.

Here's my code where I'm pulling the incoming parameters:

global class ReceiveSMS {
    @HttpPost
    global static void saveSMS() {
        // Store the request received
        RestRequest req = RestContext.request;
        System.debug(req);

        // Store the HTTP parameters received in a Map
        Map<String, String> smsParams = req.params ;

        String fromMobile ;
        String msgBody;
        Contact contactRecord;

        // Extract SMS Sender's phone number and store it in a variable
        if (smsParams.containsKey('From')){
            fromMobile = smsParams.get('From') ;
        }

        // Extract the body of SMS and store it in a variable
        if (smsParams.containsKey('Body')){
            msgBody = smsParams.get('Body');

        }

       // code goes on to construct and insert the message, etc.

I'm not sure what I'm missing... I know the field on the custom object can display the characters (when I send instead of receive).

The HttpPost raw code (with my access codes obscured, and I've inserted line returns here to make it easier to read). Note the Body parameter is passing F0 9F 8D 8E (with % in place of spaces). That's UTF-8 for Red Apple, but when it's saved in my custom object field, I see this: ð instead of 🍎.

ApiVersion=2010-04-01&SmsSid=SM9f774axxxxxf48a3a145&SmsStatus=received& SmsMessageSid=SM9f774xxxxxxxxx4b34f48a3a145&NumSegments=1& From=%2B14151234567&ToState=CA&MessageSid=SM9fxxxxx4f48a3a145& AccountSid=AC3xxxxx7a9f58&ToZip=&FromCountry=US&ToCity=& FromCity=SAN+FRANCISCO&To=%2B1987654321&FromZip=95550& Body=%F0%9F%8D%8E&ToCountry=US&FromState=CA&NumMedia=0

Suggestions?


@IllusiveBrian, I'm not sure, but I think it's missing something when my APEX ReceiveSMS class extracts the Body parameter and inserts it into the custom object field. I have other code that saves outgoing messages into that same field, and putting an emoji in is no problem -- it displays as emoji, not as UTF-8 other encryption.


@DavidReed, thanks for that bit of info about that character. That would suggest that it's picking up the first part (FO) to show ð, but not the whole thing? Not sure what to make of that.


@Sumaga, thanks for your insights. I can't figure out how to dump the entire raw incoming request where I can see/share it. I tried adding System.debug, but the logs don't seem to run on an @RestResource class static method. I also tried dumping the full request into a text field, but the RestRequest class isn't a string, and string.valueOf() doesn't work. Open to suggestions on this.


@DavidReed, here's the header:

Accept: */*, CipherSuite: ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 256-bits, Referer: http://[my instance].na73.force.com/services/apexrest/ReceiveSMS, User-Agent: TwilioProxy/1.1, Host: [my instance].na73.force.com, X-Salesforce-VIP: FORCE, X-Salesforce-SIP: 34.227.11.139, Cache-Control: max-age=259200, X-Twilio-Signature: af9fTva9ra2SUagUSxCNvQTztig=, X-Cnection: close, X-Salesforce-Forwarded-To: na73.salesforce.com, Content-Type: application/x-www-form-urlencoded

Same header, line returns inserted and blockquoted for easier reading:

Accept: /, CipherSuite: ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 256-bits, Referer: http://[my instance].na73.force.com/services/apexrest/ReceiveSMS, User-Agent: TwilioProxy/1.1, Host: [my instance].na73.force.com, X Salesforce-VIP: FORCE, X-Salesforce-SIP: 34.227.11.139, Cache-Control: max- age=259200, X-Twilio-Signature: af9fTva9ra2SUagUSxCNvQTztig=, X-Cnection: close, X-Salesforce-Forwarded-To: na73.salesforce.com, Content-Type: application/x-www-form-urlencoded

I think the key element is that last line, Content-Type, but that's what I was expecting to get.


MORE STUFF I'VE TRIED... I tried grabbing the incoming request as a blob, then decoding that -- no luck. The base64Encode is supposed to: Converts a Blob to an unencoded String representing its normal form.

RestRequest req = RestContext.request;
Blob reqBlob = req.requestBody;
String reqString = EncodingUtil.base64Encode(reqBlob);

For a message sent as 'Cherries 🍒🍒', I still get: Body: Cherries ðð

I also tried converting the blob with a simple String.valueOf(reqBlob), but same result.

I was hopeful about the EncodingUtil.urlDecode, which decodes a string and lets you specify the format, but it only takes a string, and the string I have access to has already corrupted the emoji. I tried this, with the same result as before.

    RestRequest req = RestContext.request;
    Blob reqBlob = req.requestBody;
    String reqString = EncodingUtil.urlDecode(String.valueOf(reqBlob), 'UTF-8');

QUESTION: is there a way to grab the incoming rest request without using RestContext?

ANOTHER QUESTION: I notice in the httpRequest, it starts with: ApiVersion=2010-04-01. Could it be calling to an older version of Salesforce that doesn't support utf-8?

  • 6
    I have no idea how to fix this problem, but I just wanted to say thanks for taking the time to post a good question. The content is well formatted, and you include your code as well as clear expected and observed behavior. +1 – Adrian Larson Jan 1 '18 at 1:26
  • 1
    Where exactly is the conversion going wrong? Is the value in the field getting messed up when it is saved to the database, or sometime before that? There is a string method called codePointAt which might be helpful to verify whether the raw value of the string is correct. – IllusiveBrian Jan 1 '18 at 4:48
  • 1
    @Pat McClellan: Is there any chance that you can dump the entire incoming request or the header parameters. For a test i tried sending "%F0%9F%8D%8E%20%F0%9F%98%80" which is an 🍎 & 😀 to a sample REST resource class, I was able to see the UTF-8 values for RTA fields and Unicode values for text fields getting converted as expected. The only thing i think which could have caused issues are the mult-byte characters which unicode might not have support. But these two chars seems to get converted just fine. – StormBr34ker Jan 1 '18 at 12:06
  • 1
    Like @Sumuga, I tried this in a developer org and wasn't able to reproduce the issue. I noticed, though, that 0x00F0 is the actual Unicode codepoint (not the UTF-8 representation) for the ð character. That suggests that there may be an text encoding issue in play, which the HTTP headers might shed light on. – David Reed Jan 2 '18 at 16:18
  • @AdrianLarson, thx. – PatMcClellan__c Jan 2 '18 at 17:21
7

I've been able to reproduce this in a test environment using Workbench. It appears that Salesforce by default interprets text in the body of a request with Content-Type: application/x-www-form-urlencoded as either ISO Latin-1 or UTF-16, which is why that specific piece of text yields an eth character (Unicode/Latin-1 F0).

If you can get Twilio to send the header Content-Type: application/x-www-form-urlencoded; charset=utf-8, I've confirmed that this fixes the issue.

Unfortunately, it looks like Apex grabs the request body and deserializes it to params before we actually ask for params, and the original binary request content is no longer available by the time it gets to us. Further, setting the Content-Type header on the RestRequest object once it gets to Salesforce has no effect.

  • THANK YOU! I think I can write an app on my Twilio account to customize the header. At least now I know what's causing the issue. – PatMcClellan__c Jan 2 '18 at 18:55
  • I was wrong... haven't been able to alter the Twilio-sent header to include the UTF-8 charset. Still hacking at my code, trying to see if EncodingUtil class or Blob are useful. I also tried dropping the Body text into a RichText field -- still no luck. – PatMcClellan__c Jan 2 '18 at 23:19
  • It appears (based on my correspondence with Twilio) that I can't change the request header. – PatMcClellan__c Jan 3 '18 at 16:34
  • @PatMcClellan, can you change the request content? Can you get delivery in JSON format, for example, or can you prepend a few bytes to the text they send? I'm curious as to whether adding a byte order mark would make a difference – David Reed Jan 3 '18 at 16:36
  • no, I don't think I can alter what they send. I can change whether it's httpGet or httpPost, but I've tried both with no difference in what is sent into the REST API. See my question about the ApiVersion above. Is that THEIR API or Salesforce's API? I assume that's the Twilio API, as Salesforce doesn't number theirs by date. – PatMcClellan__c Jan 3 '18 at 17:06
0

Good news: it works now! Hooray. All I did was change from HttpPost to HttpGet. That seems to be necessary.

The header is different on a HttpGet -- it does not include the content type.

KUDOS to @DavidReed for all the help and moral support.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.