Encoding problems from Python to Salesforce

Question

we are working with a third-party CPQ tool. A lot of information between both systems are transferred via Python. IronPython 2.7.5 is the used Python version.

In one part of the script I´m creating Opportunity Line Items based on what was quoted in CPQ. I´ve created a custom field "Comment_Long__c" with is Long Text Area(5000) field. This field is also filled by script. However I´m facing problems in transferring the text correctly to Salesforce especially with German umlauts like ö,ä,ü or Chinese letters. So those letters are replaced by a ? in SF. As far as I know Salesforce is expecting UTF-8 in order to display the characters correctly.

String in Python should be in UTF-8 already, isn´it In addition i tried the following encodings to get the correct text in SF, but nothing worked for me.

"Comment_Long__c":item.UserDescription.encode('utf-8', 'ignore') --> Tried also the following codecs: mbcs, cp850, latin_1

What could I change, so that SF will recognize all characters?

Used snippet from code:

def CreateOpportunityLineItem(item, quote, sFEnvironment):
    opportunityPricebookId = GetFieldFromObjectById("Pricebook2Id", "Opportunity", sFEnvironment.Opportunity.Id, sFEnvironment)
    opportunityCurrency = GetFieldFromObjectById("CurrencyIsoCode", "Opportunity", sFEnvironment.Opportunity.Id, sFEnvironment)
    productId = GetCreateProduct(item, sFEnvironment)
    pricebookEntryId = GetCreatePricebookEntry(item, opportunityPricebookId, productId, opportunityCurrency, sFEnvironment)

        newOppLineItem = {
            "Quantity":item.Quantity,
            "Product2Id":productId,
            "Comment_Long__c": item.UserDescription,
                        }

Answer 1

You are correct that Salesforce requires inbound text data to be encoded in UTF-8.

Unfortunately, you're using a very obsolete and no longer supported version of Python (2.7), where the handling of Unicode text is not nearly as advanced as in current-generation Python (3.9). While the final operation you're performing looks like it's correct, working in Python 2.7, it's very easy to accidentally treat data as the wrong encoding (like your system encoding, which might be Codepage 1252).

String in Python should be in UTF-8 already

In Python 2.7, strings and Unicode strings are distinct types. In Python 3, strings are stored internally in Unicode and must be explicitly encoded and decoded when transformed to bytes types. It's easy to mess this up in 2.7.

I don't have a clear answer for you, but my suspicion is that your data is being mangled earlier in your pipeline. While you may be able to fix this in code, I'd strongly recommend you consider migrating to a modern Python that will make your life easier working with Unicode.