UI-testing Salesforce apps with Selenium

Question

Some aspects of our Salesforce apps can only be poorly tested or not tested at all with Apex Unit tests. Therefore we are currently investigating Java-based Selenium UI-tests.

Question 1: Are you using Selenium in your Salesforce developement? Is it working? Can you share some experiences?

The part of Selenium tests we are struggling the most, is the setup and cleanup of test data. Other than with native Apex tests this is quite complicated. We are currently looking to two options which both have major drawbacks.

1. Use REST API: Create and teardown test data use the Force.com REST Toolkit for Java.

   • (+) Test setup in same context (server, language) as rest of tests
   • (-) Complicated setup with Spring, JPA, ....
   • (-) Basically we need to create and synchronise a cloned data model as Java beans..

2. DELEGATE SETUP TO APEX: Java test code calls Apex code that creates and cleans testdata

   • (+) No separate model needs to be held on the Java side
   • (-) Test code is spread between Java and Apex classes in different environments
   • (-) We basically build our own API which will likely have to change very often

Question 2: Which option would you choose? And why?

Answer 1

Selenium has some drawbacks in regards to automated testing that limit its usefulness. Notably, it doesn't handle drag-and-drop interfaces properly, and some items in salesforce.com have ID values (some cryptic) in their URL and so can't be reliably tested without extensive discovery code in the Selenium tests.

That said, Selenium can help automate the mundane process of testing Visualforce pages and other "untestable" development. The primary concern, of course, is that any data modified during a test run will remain modified after the test completes, since it is operating in a live context as far as salesforce.com can perceive (as opposed to its native test methods, which create an automatic rollback point). This is especially problematic in certain environments where data cannot be safely deleted because of potential interactions with live data that was not created through testing.

In general, the system performing the testing should be responsible for the clean up of any data it produces upon completion. Additionally, specific to salesforce.com, code space on the server is limited, while the code space on an individual computer is relatively limitless. Furthermore, since Selenium operates in a "live mode", any Apex Code framework for creating objects will also have to occupy the "live mode" code space, in turn using up the limited code space and requiring additional unit testing to cover the framework that itself is used only for testing.

While creating objects in the REST API is exceptionally trivial (even more so than the SOAP API), it is not without its drawbacks. However, since the testing is external to the native platform, the external system must assume responsibility for any data that it creates, modifies, or deletes during the course of the test run. It would make sense to develop a set of classes designed to create and delete test data so that the production system isn't polluted with this data after the Selenium tests complete.

It may be required that you will need to create "clean up tests" in case a Selenium test fails in a way that leaves test data in the production system. You would need to do this anyways with a delegated solution (again increasing code usage without any non-testing benefit), so the apparent answer would be to keep all of the testing in Selenium.

On the other hand, delegating the creation and deletion of test data to a set of salesforce.com utility classes has a distinct benefit. Tests will initialize faster since less data has to run over the wire to get things started, and the Java code would be simplified by a fair margin. This is especially important for people like me who code primarily in salesforce.com, and use Java and other languages only as auxiliary languages to perform tasks that speed up Apex Code development.

Furthermore, since primary development happens in Apex Code, making sure that the utility classes meet the requirements of the Apex Code that it supports can be tested through automation (unit tests in salesforce.com), reducing the probability that something will go wrong without noticing it earlier on. In the majority of cases, this will leave you with two types of tests that are independent of each other: Apex Code that tests controllers and utility logic, and Selenium tests that strictly test UI changes; this allows each system of testing to handle a core area of responsibility with minimal interaction or conflict.

Answer 2

My approach to this issue is similar to your choice #2, doing this in Apex:

1. Write a test data library class (or multiple classes) in Apex that can generate realistic test data. If applicable, write tear-down routines as well.*
2. In your Apex unit tests, always use those library classes to generate your test data rather than hard-coding the sample data in your unit tests.
3. In your non-Apex unit tests, call the appropriate test data builders before executing your tests, by exposing them as web services or rest resources.

*My standard format for test libraries looks something like this:

// Build a single object
public Object__c createObject (String name, String etc., Boolean insertIt) {
    Object__c obj = new Object__c(name=name);
    // Set other fields as needed

    if (insertIt) {
        insert obj;
    }
    return obj;
}

// Build several objects
public Object__c[] createObjects (Integer num, String baseName, String etc., Boolean insertIt) {
    Object__c[] objects = new List<Object__c>();
    for (Integer i=0; i<num; i++) {  
        String name = baseName + String.valueOf(i);      
        Object__c obj = createObject(name, etc., false);
        objects.add(obj);
    }

    if (insertIt) {
        insert objects;
    }
    return objects;
}

// Build a typical scenario
public void createRecipeXYZ ()) {
    Object__c[] objects = createObjects(10, 'base', true);
    Other__c other = createOther('Name', true);
    // And so on.
}

Answer 3

With AutoRABIT - a release management suite for Salesforce Applications, we have built a curing engine for Selenium that can let business users seamlessly record and playback the selenium tests . This is where real power of Selenium can be levaraged. There were many instances where the default playback would fail with detecting look-up fields , the dynamic IDs of service clouds etc., . There is also data extraction step , that would extract the data from the test script and provide the capabilities to add more data tests or change the release environment details so that with a test case recorded once- it can seamlessly playback on multiple release environments with no additional effort. Hope this helps.

-Niranjan

Answer 4

Please checkout ATTEST which would help you organize your testing in terms of business flows. It is is available on AppExchange. ATTEST integrates with not only Selenium for browser-based application testing like Salesforce, but also with HP UFT (formerly HP QTP) and Oracle Openscript (OATS). Thus you can do complete end-to-end testing of your enterprise applications (including Oracle/SAP/Peoplesoft) with ATTEST. Please let me know if you need any help to get started.