2

I am running CI using Circle CI. Here is my config.yml:

version: 2.1
general:
jobs:
  build:
    machine:
      image: ubuntu-2004:202201-02
    resource_class: large
    working_directory: ~/ci_app
    steps:
      - checkout
      - run:
          name: Install CLI
          command: |
            node --version
            npm install sfdx-cli --global
            sfdx --version
      - run:
          name: Create hub key
          command: |
            echo 'make hub key'
            openssl enc -nosalt -aes-256-cbc -d -in .circleci/server.key.enc -out .circleci/server.key -base64 -K $DECRYPTION_KEY -iv $DECRYPTION_IV
      - run:
          name: Setup Org
          command: |
            echo 'Create/Push'
            ls -al
            pwd
            echo 'Here we go!'
            sfdx force:auth:jwt:grant --clientid $HUB_CONSUMER_KEY --jwtkeyfile .circleci/server.key --username $HUB_SFDC_USER --setdefaultdevhubusername -a hub            
            sfdx force:org:create -s -f .circleci/project-scratch-def.json -a circle_build_$CIRCLE_BUILD_NUM --wait 4 -d 1
            sfdx force:source:push -u circle_build_$CIRCLE_BUILD_NUM 
      - run:
          name: Run Apex Tests
          no_output_timeout: 30m
          command: |
            mkdir -p ~/junit
            sfdx force:apex:test:run -c -d ~/junit -r junit --wait 30
      - store_test_results:
          # if: always()
          path: ~/junit
      - run:
          name: Delete Useless Scratch Org
          # if: always()
          command: |
            sfdx force:org:delete -u circle_build_$CIRCLE_BUILD_NUM -p

Here is my question: The Apex unit tests run TWICE for some reason. Why? Thanks.

Output for running ONE unit test class. Again double display of tests:

#!/bin/bash -eo pipefail
mkdir -p ~/junit
# sfdx force:apex:test:run -c -d ~/junit -r junit --wait 30
sfdx force:apex:test:run -c -d ~/junit -r junit --wait 30 -n "UnitTestsBusinessLogic7"

WARNING: WARNING: In the Summer ’21 release, we'll rename or remove some of the output files that are created when you run force:apex:test:run or force:apex:test:report with the --outputdir (-d) parameter.  This breaking change could impact your CI/CD scripts if they reference specific filenames. See https://sfdc.co/apex-test-updates for details

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
    <testsuite name="force.apex" timestamp="2022-04-19T19:00:26.000Z" hostname="https://innovation-site-6782-dev-ed.cs78.my.salesforce.com" tests="31" failures="7"  errors="0"  time="40.16">
        <properties>
            <property name="outcome" value="Failed"/>
            <property name="testsRan" value="31"/>
            <property name="passing" value="24"/>
            <property name="failing" value="7"/>
            <property name="skipped" value="0"/>
            <property name="passRate" value="77%"/>
            <property name="failRate" value="23%"/>
            <property name="testStartTime" value="Tue Apr 19 2022 7:00:26 PM"/>
            <property name="testExecutionTime" value="40.16 s"/>
            <property name="testTotalTime" value="40.16 s"/>
            <property name="commandTime" value="0.14 s"/>
            <property name="hostname" value="https://innovation-site-6782-dev-ed.cs78.my.salesforce.com"/>
            <property name="orgId" value="00D1k0000007IRtEAM"/>
            <property name="username" value="[email protected]"/>
            <property name="testRunId" value="7071k0000267UQu"/>
            <property name="userId" value="0051k000008F0ZrAAK"/>
            <property name="coveredLines" value="4539"/>
            <property name="testRunCoverage" value="25%"/>
            <property name="orgWideCoverage" value="6%"/>
        </properties>
        <testcase name="testCustomerInvoiceDocument1" classname="UnitTestsBusinessLogic7" time="0.18">
        </testcase>
        <testcase name="testCustomerInvoiceDocument10" classname="UnitTestsBusinessLogic7" time="3.60">
        </testcase>
        <testcase name="testCustomerInvoiceDocument11" classname="UnitTestsBusinessLogic7" time="1.98">
        </testcase>
        <testcase name="testCustomerInvoiceDocument2" classname="UnitTestsBusinessLogic7" time="2.09">
        </testcase>
        <testcase name="testCustomerInvoiceDocument3" classname="UnitTestsBusinessLogic7" time="1.84">
        </testcase>
        <testcase name="testCustomerInvoiceDocument4" classname="UnitTestsBusinessLogic7" time="2.09">
        </testcase>
        <testcase name="testCustomerInvoiceDocument5" classname="UnitTestsBusinessLogic7" time="2.03">
        </testcase>
        <testcase name="testCustomerInvoiceDocument6" classname="UnitTestsBusinessLogic7" time="2.66">
        </testcase>
        <testcase name="testCustomerInvoiceDocument7" classname="UnitTestsBusinessLogic7" time="2.22">
        </testcase>
        <testcase name="testCustomerInvoiceDocument8" classname="UnitTestsBusinessLogic7" time="2.10">
        </testcase>
        <testcase name="testCustomerInvoiceDocument9" classname="UnitTestsBusinessLogic7" time="2.12">
        </testcase>
        <testcase name="testLoadImporter" classname="UnitTestsBusinessLogic7" time="0.38">
            <failure message="System.AssertException: Assertion Failed: testLoadImporter failed: Expected: 15, Actual: 0"><![CDATA[Class.UnitTestsBusinessLogic7.testLoadImporter: line 726, column 1]]></failure>
        </testcase>
        <testcase name="testLoadPricingWorkbench1" classname="UnitTestsBusinessLogic7" time="0.78">
            <failure message="System.ListException: List index out of bounds: 12"><![CDATA[Class.UnitTestsBusinessLogic7.testLoadPricingWorkbench1: line 775, column 1]]></failure>
        </testcase>
        <testcase name="testLoadPricingWorkbench4" classname="UnitTestsBusinessLogic7" time="0.61">
            <failure message="System.ListException: List index out of bounds: 12"><![CDATA[Class.UnitTestsBusinessLogic7.testLoadPricingWorkbench4: line 902, column 1]]></failure>
        </testcase>
        <testcase name="testLoadPricingWorkbench5" classname="UnitTestsBusinessLogic7" time="0.53">
            <failure message="System.ListException: List index out of bounds: 12"><![CDATA[Class.UnitTestsBusinessLogic7.testLoadPricingWorkbench5: line 945, column 1]]></failure>
        </testcase>
        <testcase name="testLoadPricingWorkbench6" classname="UnitTestsBusinessLogic7" time="0.51">
            <failure message="System.ListException: List index out of bounds: 12"><![CDATA[Class.UnitTestsBusinessLogic7.testLoadPricingWorkbench6: line 987, column 1]]></failure>
        </testcase>
        <testcase name="testRateCalculation1" classname="UnitTestsBusinessLogic7" time="0.82">
        </testcase>
        <testcase name="testRateCalculation2" classname="UnitTestsBusinessLogic7" time="1.92">
        </testcase>
        <testcase name="testRateCalculation3" classname="UnitTestsBusinessLogic7" time="0.94">
        </testcase>
        <testcase name="testRateCalculation4" classname="UnitTestsBusinessLogic7" time="1.16">
        </testcase>
        <testcase name="testRateCalculation5" classname="UnitTestsBusinessLogic7" time="1.35">
        </testcase>
        <testcase name="testRateCalculation6" classname="UnitTestsBusinessLogic7" time="1.00">
        </testcase>
        <testcase name="testRateCalculation7" classname="UnitTestsBusinessLogic7" time="1.08">
        </testcase>
        <testcase name="testRateCalculation8" classname="UnitTestsBusinessLogic7" time="0.99">
        </testcase>
        <testcase name="testRateCalculationResult1" classname="UnitTestsBusinessLogic7" time="0.12">
        </testcase>
        <testcase name="testTMSDocumentOverrides1" classname="UnitTestsBusinessLogic7" time="0.72">
        </testcase>
        <testcase name="testTMSRateIQService1" classname="UnitTestsBusinessLogic7" time="1.11">
        </testcase>
        <testcase name="testTMSRateIQService2" classname="UnitTestsBusinessLogic7" time="1.01">
        </testcase>
        <testcase name="testLoadPricingWorkbench2" classname="UnitTestsBusinessLogic7" time="0.53">
            <failure message="System.ListException: List index out of bounds: 12"><![CDATA[Class.UnitTestsBusinessLogic7.testLoadPricingWorkbench2: line 816, column 1]]></failure>
        </testcase>
        <testcase name="testLoadPricingWorkbench3" classname="UnitTestsBusinessLogic7" time="0.62">
            <failure message="System.ListException: List index out of bounds: 12"><![CDATA[Class.UnitTestsBusinessLogic7.testLoadPricingWorkbench3: line 857, column 1]]></failure>
        </testcase>
        <testcase name="testTMSRateIQService3" classname="UnitTestsBusinessLogic7" time="1.08">
        </testcase>
    </testsuite>
</testsuites>


Exited with code exit status 100
CircleCI received exit code 100

enter image description here

9
  • Can you upload the console output?
    – mario ruiz
    Commented Apr 20, 2022 at 16:26
  • Sure. I have added a run where only one test class is executed. The tests in that one class appear to execute twice. There are 31 tests but the app is saying there are 62. There are 7 failures but the app is saying 14.
    – Dave C
    Commented Apr 20, 2022 at 17:26
  • Hello Dave - could you please explain what do you mean by double display when running ONE unit test class? I could not find duplicates test classes being executed on the file you shared. Commented Apr 20, 2022 at 20:38
  • 2
    What happens if you comment out the Store Test Results step? See if that makes a difference Commented Apr 21, 2022 at 12:28
  • 1
    Have you verified that there are two separate XML files (or sets thereof) in your /junit folder after CircleCI runs?
    – Moonpie
    Commented Apr 21, 2022 at 13:50

1 Answer 1

4
+50

From the documentation (https://circleci.com/docs/2.0/collect-test-data/):

Using the store_test_results step allows you to not only upload and store test results, but also provides a view of your passing/failing tests in the CircleCI web app.

Not knowing the inner workings of CircleCI, I am surmising that the view that comes with the store_test_results step essentially just parses whatever it finds in that path folder.


To test my hypothesis, on my local machine in VS Code I created a scratch org, pushed code, and then ran the following command:

sfdx force:apex:test:run -c -r junit -d ./junit --wait 30

Here is what is in that junit folder after the run: enter image description here

Notice that there are two XML files.

I did a diff of the two and they are exactly the same.


Here is what I suspect is happening in your case:

  • Your sfdx force:apex:test:run command run is automatically saving in your -d ~/junit folder two XML files that have the same data
  • The view parser that comes with using store_test_results in CircleCI is showing everything from all XML files in that folder

So I think that your tests are not running twice, you are just viewing duplicate outputs.


Again, not knowing CircleCI, I'm not certain of all your options; but the first two potential ones that pop into my mind are:

  • See if you can add any parameter/option/config to your store_test_results step that would have it look at only one file named test-result.xml.
  • Add a step after your test run and before store_test_results that EITHER
    • deletes or moves all other files except test-result.xml; OR
    • moves only test-result.xml to a separate folder, and then point store_test_results to that folder

UPDATE

"On a hunch" I changed the result format of the sfdx test run from junit to json. After running, my hunch was confirmed as here are the contents of the output folder:

enter image description here

Notice that the XML file which includes the test-run-id in the filename is still being generated (along with a few other of the same non-XML files) regardless of the result format chosen.

I compared that file to both XMLs that I mentioned earlier, and it looks like the same contents.

So if my hypothesis is correct about CircleCI parsing whatever XML files it finds in the path folder, then you could quickly and simply test it by changing your test run line to:

sfdx force:apex:test:run -c -r json -d ./junit --wait 30

(Or for that matter, you could use human instead of json and get a (mostly) nicely formatted more readable text file to go along with that one XML that always seems to get generated.)

Now if it works, you would then have to decide whether to permanently leave it like that in your script. Sure it would be an easy fix, but would that be confusing to someone else (or a future you) trying to maintain/update this script? It might be OK to leave it if you document it well. That would be your call.

2
  • 1
    Wow, thanks for all the excellent help! Especially thanks to Moonpie. I will try this and get back to you.
    – Dave C
    Commented Apr 21, 2022 at 19:19
  • 1
    Updated with additional findings.
    – Moonpie
    Commented Apr 21, 2022 at 21:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .