locationsNotCovered gets empty objects

[Bartheleway]

Summary

I'm trying to get uncovered lines of code from a "validate" job result.

Running the same commands on my machine works like a charm but running them from the docker runner on the server doesn't.

Steps To Reproduce

• Make a package.xml with at least one meta to deploy (doesn't matter which one)

<?xml version="1.0" encoding="UTF-8"?>
<Package xmlns="http://soap.sforce.com/2006/04/metadata">
    <types>
        <members>*</members>
        <name>ApexPage</name>
    </types>
    <version>63.0</version>
</Package>

• Run a validation with sf project deploy validate -x ./package.xml --json -g --async -l RunLocalTests > ./dry-run-id.json
• Run sf project deploy resume -i JOB_ID --json > ./validate-resume.json (replace JOB_ID by the deployment id)
• Get the result with sf project deploy report -i JOB_ID --json > ./dry-run-result.json (replace JOB_ID by the deployment id)

Check validate-resume.json and dry-run-result.json file. They should both contains test results with uncovered lines.

Expected result

Example part of the file about codeCoverage, especially the locationsNotCovered section.

{
    "id": "some_sf_id",
    "locationsNotCovered": [
        {
            "column": 0,
            "line": 11,
            "numExecutions": 0,
            "time": -1
        },
        {
            "column": 0,
            "line": 26,
            "numExecutions": 0,
            "time": -1
        }
    ],
    "name": "some_trigger",
    "namespace": null,
    "numLocations": 15,
    "numLocationsNotCovered": 2,
    "type": "Trigger"
},

Actual result

In the dry-run-result.json file all locationsNotCovered sections are broken like that (validate-resume.json is fine)

{
    "id": "some_sf_id",
    "locationsNotCovered": [
        {},
        {}
    ],
    "name": "some_trigger",
    "namespace": null,
    "numLocations": 15,
    "numLocationsNotCovered": 2,
    "type": "Trigger"
},

Additional information

I tried to reproduce in several ways without success. It only happens inside my gitlab docker runner.

I tried :

• locally by executing directly the commands (windows with powershell)
• in a local docker runner using the exact same docker image (windows with docker desktop)

Running the report command locally when my gitlab job fails and using the exact same JOB_ID is giving me the correct result.

I tried to look in the source code to see if I was missing some flags but it doesn't seems to exist any that would kind of limit the depth of the JSON file.

System Information

sf doctor command result on the local docker runner (should be identical with the gitlab docker runner).

{
  "status": 0,
  "result": {
    "versionDetail": {
      "architecture": "linux-x64",
      "cliVersion": "@salesforce/cli/2.98.6",
      "nodeVersion": "node-v22.18.0",
      "osVersion": "Linux 6.6.87.2-microsoft-standard-WSL2",
      "rootPath": "/usr/lib/node_modules/@salesforce/cli",
      "shell": "bash",
      "pluginVersions": [
        "@oclif/plugin-autocomplete 3.2.33 (core)",
        "@oclif/plugin-commands 4.1.29 (core)",
        "@oclif/plugin-help 6.2.31 (core)",
        "@oclif/plugin-not-found 3.2.60 (core)",
        "@oclif/plugin-plugins 5.4.45 (core)",
        "@oclif/plugin-search 1.2.27 (core)",
        "@oclif/plugin-update 4.7.1 (core)",
        "@oclif/plugin-version 2.2.31 (core)",
        "@oclif/plugin-warn-if-update-available 3.1.44 (core)",
        "@oclif/plugin-which 3.2.39 (core)",
        "@salesforce/cli 2.98.6 (core)",
        "agent 1.23.3 (core)",
        "apex 3.6.19 (core)",
        "api 1.3.3 (core)",
        "auth 3.7.9 (core)",
        "data 4.0.46 (core)",
        "deploy-retrieve 3.22.33 (core)",
        "info 3.4.73 (core)",
        "limits 3.3.60 (core)",
        "marketplace 1.3.8 (core)",
        "org 5.9.9 (core)",
        "packaging 2.18.0 (core)",
        "schema 3.3.73 (core)",
        "settings 2.4.36 (core)",
        "sobject 1.4.65 (core)",
        "telemetry 3.6.48 (core)",
        "templates 56.3.54 (core)",
        "trust 3.7.109 (core)",
        "user 3.6.29 (core)",
        "sfdx-git-delta 6.16.0 (user)"
      ]
    },
    "sfdxEnvVars": [],
    "sfEnvVars": [
      [
        "SF_DISABLE_AUTOUPDATE",
        "true"
      ],
      [
        "SF_IMPROVED_CODE_COVERAGE",
        "true"
      ],
      [
        "SF_ORG_METADATA_REST_DEPLOY",
        "true"
      ],
      [
        "SF_DOMAIN_RETRY",
        "300"
      ],
      [
        "SF_DISABLE_TELEMETRY",
        "true"
      ],
      [
        "SF_CONTAINER_MODE",
        "true"
      ],
      [
        "SF_SKIP_NEW_VERSION_CHECK",
        "true"
      ],
      [
        "SF_CONTENT_TYPE",
        "JSON"
      ],
      [
        "SF_UPDATE_INSTRUCTIONS",
        "Use \"npm update --global @salesforce/cli\" to update npm-based installations."
      ]
    ],
    "proxyEnvVars": [],
    "cliConfig": {
      "arch": "x64",
      "bin": "sf",
      "binAliases": [
        "sfdx"
      ],
      "cacheDir": "/root/.cache/sf",
      "channel": "stable",
      "configDir": "/root/.config/sf",
      "dataDir": "/root/.local/share/sf",
      "dirname": "sf",
      "flexibleTaxonomy": true,
      "home": "/root",
      "isSingleCommandCLI": false,
      "name": "@salesforce/cli",
      "nsisCustomization": "scripts/nsis.nsi",
      "platform": "linux",
      "root": "/usr/lib/node_modules/@salesforce/cli",
      "shell": "bash",
      "theme": {
        "bin": "blueBright",
        "command": "blueBright",
        "dollarSign": "green",
        "flag": "green",
        "flagDefaultValue": "blueBright",
        "flagOptions": "blueBright",
        "flagRequired": "red",
        "sectionHeader": "blue",
        "topic": "blueBright",
        "json": {
          "brace": "whiteBright",
          "bracket": "whiteBright",
          "comma": "green",
          "key": "blueBright",
          "string": "greenBright",
          "number": "blue",
          "boolean": "redBright",
          "null": "blackBright"
        }
      },
      "topicSeparator": " ",
      "updateConfig": {
        "node": {
          "options": [
            "--no-deprecation"
          ]
        },
        "s3": {
          "bucket": "dfc-data-production",
          "host": "https://developer.salesforce.com",
          "templates": {
            "target": {
              "baseDir": "<%- bin %>",
              "manifest": "<%- channel === 'stable' ? '' : 'channels/' + channel + '/' %><%- platform %>-<%- arch %>",
              "unversioned": "<%- channel === 'stable' ? '' : 'channels/' + channel + '/' %><%- bin %>-<%- platform %>-<%- arch %><%- ext %>",
              "versioned": "<%- channel === 'stable' ? '' : 'channels/' + channel + '/' %><%- bin %>-v<%- version %>/<%- bin %>-v<%- version %>-<%- platform %>-<%- arch %><%- ext %>"
            },
            "vanilla": {
              "baseDir": "<%- bin %>",
              "manifest": "<%- channel === 'stable' ? '' : 'channels/' + channel + '/' %>version",
              "unversioned": "<%- channel === 'stable' ? '' : 'channels/' + channel + '/' %><%- bin %><%- ext %>",
              "versioned": "<%- channel === 'stable' ? '' : 'channels/' + channel + '/' %><%- bin %>-v<%- version %>/<%- bin %>-v<%- version %><%- ext %>"
            }
          }
        }
      },
      "userAgent": "@salesforce/cli/2.98.6 linux-x64 node-v22.18.0",
      "valid": true,
      "version": "2.98.6",
      "warned": false,
      "windows": false,
      "_base": "@oclif/core@4.5.1",
      "_commands": {},
      "_topics": {},
      "commandPermutations": {
        "validPermutations": {}
      },
      "pluginLoader": {
        "options": {
          "root": "/usr/lib/node_modules/@salesforce/cli/dist"
        },
        "errors": [],
        "plugins": {},
        "pluginsProvided": false
      },
      "topicPermutations": {
        "validPermutations": {}
      },
      "nodeEngine": ">=18.6.0"
    },
    "pluginSpecificData": {
      "@salesforce/plugin-deploy-retrieve": [
        {}
      ],
      "@salesforce/plugin-auth": [
        {
          "isUsingGenericKeychain": false,
          "sfCryptoV2Support": true,
          "cryptoVersion": "unknown"
        }
      ]
    },
    "diagnosticResults": [
      {
        "testName": "salesforcedx plugin isn’t installed",
        "status": "pass"
      },
      {
        "testName": "you don't have any linked plugins",
        "status": "pass"
      },
      {
        "testName": "[@salesforce/plugin-deploy-retrieve] sourceApiVersion matches apiVersion",
        "status": "warn"
      },
      {
        "testName": "[@salesforce/plugin-trust] can ping: https://registry.npmjs.org",
        "status": "pass"
      },
      {
        "testName": "[@salesforce/plugin-trust] can ping: https://registry.yarnpkg.com",
        "status": "pass"
      },
      {
        "testName": "[@salesforce/plugin-trust] can ping: https://registry.npmjs.org/",
        "status": "pass"
      },
      {
        "testName": "using latest or latest-rc CLI version",
        "status": "pass"
      },
      {
        "testName": "can access: https://test.salesforce.com",
        "status": "pass"
      },
      {
        "testName": "can access: https://appexchange.salesforce.com/services/data",
        "status": "pass"
      },
      {
        "testName": "can access: https://developer.salesforce.com/media/salesforce-cli/sf/channels/stable/sf-win32-x64-buildmanifest",
        "status": "pass"
      },
      {
        "testName": "[@salesforce/plugin-auth] CLI supports v2 crypto",
        "status": "pass"
      }
    ],
    "suggestions": [
      "Check https://github.com/forcedotcom/cli/issues for CLI issues posted by the community.",
      "Check http://status.salesforce.com for general Salesforce availability and performance.",
      "Neither sourceApiVersion nor apiVersion are defined. The commands that deploy and retrieve source use the max apiVersion of the target org in this case. The issue isn't a problem, as long as it's the behavior you actually want.",
      "using npm registry https://registry.npmjs.org/ from npm config"
    ],
    "logFilePaths": [
      "/opt/ci-scripts/1754984540366-diagnosis.json",
      "/opt/ci-scripts/1754984540366-diagnosis.json"
    ],
    "commandExitCode": 0
  },
  "warnings": []
}

[github-actions]

Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.

[github-actions]

Hello @Bartheleway 👋 None of the versions of sf you shared match the latest release.

Shared: 2.98.6, 2.98.6
Latest: 2.100.4

Update to the latest version of Salesforce CLI (docs) and confirm that you're still seeing your issue.
You can also try the rc and nightly releases! (docs)

After updating, share the full output of sf version --verbose --json

[iowillhoit]

Hey @Bartheleway, unfortunately these "only happens in my CI" issues are notoriously difficult to track down. What is the name of the Docker container you are using in Gitlab?

If the same Docker container works from your local machine, something must be up with the Gitlab one. Could you please provide us with the following information executed from inside your Gitlab CI runner.

• The output of sf doctor
• Add the --dev-debug flag to your sf project deploy report -i JOB_ID --json > ./dry-run-result.json command and provide the full output

[github-actions]

Hello @Bartheleway 👋 None of the versions of sf you shared match the latest release.

Shared: 2.98.6, 2.98.6
Latest: 2.101.5

Update to the latest version of Salesforce CLI (docs) and confirm that you're still seeing your issue.
You can also try the rc and nightly releases! (docs)

After updating, share the full output of sf version --verbose --json

[github-actions]

This issue has not received a response in 7 days. It will auto-close in 7 days unless a response is posted.

[Bartheleway]

@iowillhoit, yes I agree. I would not be here if it was easier 😄

I did run the sf project deploy report -i JOB_ID --json > ./dry-run-result.json in a debug job and got the expected output 😮

There is one additional factor to check: Command is executed right after it finishes to make the validation on SF side, so maybe I just need to wait 1min or so?

The updated sf doctor from Gitlab runner

{
  "status": 0,
  "result": {
    "versionDetail": {
      "architecture": "linux-x64",
      "cliVersion": "@salesforce/cli/2.103.7",
      "nodeVersion": "node-v22.18.0",
      "osVersion": "Linux 5.14.0-570.32.1.el9_6.x86_64",
      "rootPath": "/usr/lib/node_modules/@salesforce/cli",
      "shell": "bash",
      "pluginVersions": [
        "@oclif/plugin-autocomplete 3.2.34 (core)",
        "@oclif/plugin-commands 4.1.32 (core)",
        "@oclif/plugin-help 6.2.32 (core)",
        "@oclif/plugin-not-found 3.2.65 (core)",
        "@oclif/plugin-plugins 5.4.46 (core)",
        "@oclif/plugin-search 1.2.28 (core)",
        "@oclif/plugin-update 4.7.4 (core)",
        "@oclif/plugin-version 2.2.32 (core)",
        "@oclif/plugin-warn-if-update-available 3.1.46 (core)",
        "@oclif/plugin-which 3.2.39 (core)",
        "@salesforce/cli 2.103.7 (core)",
        "agent 1.24.4 (core)",
        "apex 3.6.19 (core)",
        "api 1.3.3 (core)",
        "auth 3.9.0 (core)",
        "data 4.0.53 (core)",
        "deploy-retrieve 3.22.41 (core)",
        "info 3.4.81 (core)",
        "limits 3.3.64 (core)",
        "marketplace 1.3.8 (core)",
        "org 5.9.23 (core)",
        "packaging 2.19.5 (core)",
        "schema 3.3.78 (core)",
        "settings 2.4.42 (core)",
        "sobject 1.4.68 (core)",
        "telemetry 3.6.55 (core)",
        "templates 56.3.60 (core)",
        "trust 3.7.113 (core)",
        "user 3.6.34 (core)",
        "sfdx-git-delta 6.18.2 (user)"
      ]
    },
    "sfdxEnvVars": [],
    "sfEnvVars": [
      [
        "SF_CONTAINER_MODE",
        "true"
      ],
      [
        "SF_DISABLE_TELEMETRY",
        "true"
      ],
      [
        "SF_DISABLE_AUTOUPDATE",
        "true"
      ],
      [
        "SF_DOMAIN_RETRY",
        "300"
      ],
      [
        "SF_SKIP_NEW_VERSION_CHECK",
        "true"
      ],
      [
        "SF_CONTENT_TYPE",
        "JSON"
      ],
      [
        "SF_ORG_METADATA_REST_DEPLOY",
        "true"
      ],
      [
        "SF_IMPROVED_CODE_COVERAGE",
        "true"
      ],
      [
        "SF_UPDATE_INSTRUCTIONS",
        "Use \"npm update --global @salesforce/cli\" to update npm-based installations."
      ]
    ],
    "proxyEnvVars": [],
    "cliConfig": {
      "arch": "x64",
      "bin": "sf",
      "binAliases": [
        "sfdx"
      ],
      "cacheDir": "/root/.cache/sf",
      "channel": "stable",
      "configDir": "/root/.config/sf",
      "dataDir": "/root/.local/share/sf",
      "dirname": "sf",
      "flexibleTaxonomy": true,
      "home": "/root",
      "isSingleCommandCLI": false,
      "name": "@salesforce/cli",
      "nsisCustomization": "scripts/nsis.nsi",
      "platform": "linux",
      "root": "/usr/lib/node_modules/@salesforce/cli",
      "shell": "bash",
      "theme": {
        "bin": "blueBright",
        "command": "blueBright",
        "dollarSign": "green",
        "flag": "green",
        "flagDefaultValue": "blueBright",
        "flagOptions": "blueBright",
        "flagRequired": "red",
        "sectionHeader": "blue",
        "topic": "blueBright",
        "json": {
          "brace": "whiteBright",
          "bracket": "whiteBright",
          "comma": "green",
          "key": "blueBright",
          "string": "greenBright",
          "number": "blue",
          "boolean": "redBright",
          "null": "blackBright"
        }
      },
      "topicSeparator": " ",
      "updateConfig": {
        "node": {
          "options": [
            "--no-deprecation"
          ]
        },
        "s3": {
          "bucket": "dfc-data-production",
          "host": "https://developer.salesforce.com/",
          "templates": {
            "target": {
              "baseDir": "<%- bin %>",
              "manifest": "<%- channel === 'stable' ? '' : 'channels/' + channel + '/' %><%- platform %>-<%- arch %>",
              "unversioned": "<%- channel === 'stable' ? '' : 'channels/' + channel + '/' %><%- bin %>-<%- platform %>-<%- arch %><%- ext %>",
              "versioned": "<%- channel === 'stable' ? '' : 'channels/' + channel + '/' %><%- bin %>-v<%- version %>/<%- bin %>-v<%- version %>-<%- platform %>-<%- arch %><%- ext %>"
            },
            "vanilla": {
              "baseDir": "<%- bin %>",
              "manifest": "<%- channel === 'stable' ? '' : 'channels/' + channel + '/' %>version",
              "unversioned": "<%- channel === 'stable' ? '' : 'channels/' + channel + '/' %><%- bin %><%- ext %>",
              "versioned": "<%- channel === 'stable' ? '' : 'channels/' + channel + '/' %><%- bin %>-v<%- version %>/<%- bin %>-v<%- version %><%- ext %>"
            }
          }
        }
      },
      "userAgent": "@salesforce/cli/2.103.7 linux-x64 node-v22.18.0",
      "valid": true,
      "version": "2.103.7",
      "warned": false,
      "windows": false,
      "_base": "@oclif/core@4.5.2",
      "_commands": {},
      "_topics": {},
      "commandPermutations": {
        "validPermutations": {}
      },
      "pluginLoader": {
        "options": {
          "root": "/usr/lib/node_modules/@salesforce/cli/dist"
        },
        "errors": [],
        "plugins": {},
        "pluginsProvided": false
      },
      "topicPermutations": {
        "validPermutations": {}
      },
      "nodeEngine": ">=18.6.0"
    },
    "pluginSpecificData": {
      "@salesforce/plugin-deploy-retrieve": [
        {
          "sourceApiVersion": "63.0"
        }
      ],
      "@salesforce/plugin-auth": [
        {
          "isUsingGenericKeychain": false,
          "sfCryptoV2Support": true,
          "cryptoVersion": "unknown"
        }
      ]
    },
    "diagnosticResults": [
      {
        "testName": "salesforcedx plugin isn’t installed",
        "status": "pass"
      },
      {
        "testName": "you don't have any linked plugins",
        "status": "pass"
      },
      {
        "testName": "[@salesforce/plugin-trust] can ping: https://registry.npmjs.org/",
        "status": "pass"
      },
      {
        "testName": "[@salesforce/plugin-trust] can ping: https://registry.yarnpkg.com/",
        "status": "pass"
      },
      {
        "testName": "[@salesforce/plugin-trust] can ping: https://registry.npmjs.org/",
        "status": "pass"
      },
      {
        "testName": "using latest or latest-rc CLI version",
        "status": "pass"
      },
      {
        "testName": "[@salesforce/plugin-deploy-retrieve] sourceApiVersion matches apiVersion",
        "status": "pass"
      },
      {
        "testName": "can access: https://test.salesforce.com/",
        "status": "pass"
      },
      {
        "testName": "can access: https://appexchange.salesforce.com/services/data",
        "status": "pass"
      },
      {
        "testName": "can access: https://developer.salesforce.com/media/salesforce-cli/sf/channels/stable/sf-win32-x64-buildmanifest",
        "status": "pass"
      },
      {
        "testName": "[@salesforce/plugin-auth] CLI supports v2 crypto",
        "status": "pass"
      }
    ],
    "suggestions": [
      "Check https://github.com/forcedotcom/cli/issues for CLI issues posted by the community.",
      "Check http://status.salesforce.com/ for general Salesforce availability and performance.",
      "using npm registry https://registry.npmjs.org/ from npm config"
    ],
    "commandExitCode": 0
  },
  "warnings": []
}

[iowillhoit]

yes I agree. I would not be here if it was easier 😄

lol, fair point 😆 Sorry for the delayed response @Bartheleway. It is entirely possible that there is a race condition here. Did you try adding a sleep before calling the sf project deploy report?

[Bartheleway]

As said initially, my command flow is like this

sf project deploy validate -x ./package.xml --json -g --async -l RunLocalTests > ./dry-run-id.json
sf project deploy resume -i JOB_ID --json > ./validate-resume.json (replace JOB_ID by the deployment id)
sf project deploy report -i JOB_ID --json > ./dry-run-result.json

Which means:

• First I send package to org and get deploy id in dry-run-id.json file
• Second I wait until org validation is done and put result in validate-resume.json file
• Third (which is just a workaround and should not be necessary), I get again the result of the validation in dry-run-result.json

Second step is getting result right after validation ends on org. Second and third step are run just "seconds" apart but offer different result where they should not. How can it be that result from two different calls differ especially if 1st one succeed, 2nd failed and later 3rd succeed again?