testMate.cpp.test.advancedExecutables

[Jump to README]

(TypeScript interface)

The location of your test executables (relative to the workspace folder or absolute path) and with a lot of other setting.

he extension is *pre-configured and it should find executables inside the working directory which match the following glob pattern:

"testMate.cpp.test.advancedExecutables": [ "{build,Build,BUILD,out,Out,OUT}/**/*{test,Test,TEST}*" ]

This basically means executables inside the build and out directories (recursive /**/) which contain the test word in their name (including extensions).

(Examples here)

See vscode's documentation for syntax.

The first example (.vscode/settings.json or hit Ctr/Cmd + ,):

"testMate.cpp.test.advancedExecutables": [
  {
    "pattern": "{build,Build,BUILD,out,Out,OUT}/**/*{test,Test,TEST}*",
    "cwd": "${absDirpath}",
    "env": {
      "ExampleENV1": "You can use variables here too, like ${relPath}",
      "PATH": "${os_env:PATH}${osPathEnvSep}/adding/new/item/to/PATH/env"
    }
  }
]

More examples.

This variable can be

• a string (ex.: "out/**/*test.exe") or
• an array of objects (ex.: [ { "pattern": "release/*test.exe" }, ... ]).

If it is an object it can contains the following properties:

The pattern (or the executables used as string or an array of strings) can contain search-pattern. Also it can contain variables related to the process's environment variables (Ex.: ${os_env:PATH}).

Test executables and patterns are being watched. In case of one recompiles it will try to preserve the test states. If compilation reaches timeout it will drop the suite (testMate.cpp.discovery.defaultWatchTimeoutSec).

Note that there is a mechanism which will filter out every possible executable which:

• on windows: NOT ends with .exe, .cmd or .bat.
• on other platforms: ends with one of the following: '.a', '.bat', '.c', '.cc', '.cmake', '.cpp', '.cxx', '.deb', '.dir', '.gz', '.h', '.hpp', '.hxx', '.in', '.input', '.ko', '.log', '.md', '.mm', '.o', '.pc', '.php', '.pyc', '.rpm', '.so', '.tar', '.txt', '.vcxproj.user', '.xml'.
• inside a sub-folder called: _deps. Except when it is explicitly contained by the pattern.

It won't filter out '.sh', '.py' (etc.) files, so that could be used for wrappers.

If the pattern is too general like out/**/*test*, it could cause unexpected executable or script execution (with --help argument) which would not just increase the test-loading duration but also could have other unexpected effects. I suggest to have a stricter file-name convention and a corresponding pattern like out/**/*.test.* or out/**/Test.*

Applies a simple search and replace for the file paths. It resolves variables

Examples:

  "testMate.test.advancedExecutables": [{
    "pattern": "<default pattern>",
    "sourceFileMap": {
      "/path/to/be/replaced/": "/path/I/want/to/use/",
      "/other-path-to-resolve/": "${workspaceFolder}/"
    }
  }]

• ℹ️Executables found by pattern are automatically watched, don't need to add them to dependsOn.
• If "Enable autorun" is enabled in "..." menu (next to the play button), it will trigger the related test suites by detecting the recompilation of the executable.
• It accumulates events with the following strategy: waiting for 2 seconds after the last event.
• Works flawlessly with paths/patterns inside of the workspace directory (Usually there is no reason to keep your executables outside of the workspace. See.),
• but have some issue/limitation with paths/patterns outside of the workspace directory:
  • Theoretically is should support glob patterns, but it seem there is an issue with double star (**).
  • Paths on different drive in the same dependsOn array maybe won't work.
  • (If you find another corner case, feel free to open an issue. It could be handy once in the future.)

Probably your best option if your executables need some initinalization of enviroment. Use it with dependsOn and then in case the envFile changes the tests will be retired.

Example envFile content:

{
  "MYVAR": "my value"
}

Simple example:

  "testMate.test.advancedExecutables": [{
    "pattern": "<default pattern>",
    "envFile": "out/env.json", // or "out/my.env"
    "dependsOn": "out/env.json"
  }]

Advanced example:

  "testMate.test.advancedExecutables": [{
    "pattern": "<default pattern>",
    "envFile": "${relPath}.env.json",
    "dependsOn": "${relPath}.env.json"
  }]

Example:

  "testMate.test.advancedExecutables": [{
    "pattern": "<default pattern>",
    "darwin": { "pattern":"<value on mac>" },
    "win32": { "pattern":"<value on win>" },
    "linux": { "pattern":"<value on linux>" }
  }]

"testMate.cpp.test.advancedExecutables": ["dir/test1.exe", "dir/test2.exe"]

"testMate.cpp.test.advancedExecutables": {
  "name": "${filename}",
  "description": "${relDirpath}/",
  "pattern": "{build,Build,BUILD,out,Out,OUT}/**/*{test,Test,TEST}*",
  "cwd": "${absDirpath}",
  "env": {
    "ExampleENV1": "You can use variables here too, like ${absPath}",
    "PATH": "${os_env:PATH}${osPathEnvSep}/adding/new/item/to/PATH/env"
  }
}

"testMate.cpp.test.advancedExecutables": [
  {
    "name": "Test1 suite",
    "pattern": "dir/test.exe"
  },
  "canBeMixed.exe",
  {
    "pattern": "${os_env:HOME}/dir2/{t,T}est",
    "cwd": "out/tmp",
    "env": {}
  }
]

which can be used in cwd and env of executables:

Array indexing: (?:\[(-?[0-9]+)?:(-?[0-9]+)?\])?. Example: ${relPath[:-2]}: 'a/b/c/d' -> 'a/b'

Under property: catch2, gtest, doctest.

One can fine-tune framework related behaviour.

If the regex is too general it will mach all the executables❗️ One should avoid that❗️

Note .* matches \n

In case of gtest the helpRegex' the first capture should be the value of GTEST_FLAG_PREFIX. For example:

• --gtest_list_tests + --(\w+)list_tests -> gtest_

In case if catch2 and doctest the first, second and third captures are the version number's major, minor and patch values. For example:

• Catch v2.11.1 + /Catch v(\d+)\.(\d+)\.(\d+)\s?/ -> [2, 11, 1]

As the name says test enumeraton will ignore std::err output. Having content from std::err in when the tests are enumerated is usually indicates some error but sometimes just some basic output from one of the components. The usage of this feature is discouraged because it can hide some real issue. (Usually logging libraries allows custom sinks and with a custom main this can be solved.)

Ex.:

{
  "testMate.cpp.test.advancedExecutables": [
    {
      "pattern": "{build,Build,BUILD,out,Out,OUT}/**/*{test,Test,TEST}*",
      "gtest":   { "ignoreTestEnumerationStdErr": true }
    }
   ]
}

It is undocumented. Contact me by opening an issue or read the code a bit.

• interface
• code

{
  "testMate.cpp.test.advancedExecutables": [
    {
      "pattern": "{build,Build,BUILD,out,Out,OUT}/**/*{test,Test,TEST}*",
      "catch2": {
        "testGrouping": {
          "groupByExecutable": {
            /*...*/
          }
        }
      }
    }
  ]
}

Note: This example overused it.

{
  "testMate.cpp.test.advancedExecutables": [
    {
      "pattern": "{build,Build,BUILD,out,Out,OUT}/**/*{test,Test,TEST}*",
      "catch2": {
        "testGrouping": {
          "groupByExecutable": {
            "label": "${filename}",
            "description": "${relDirpath}/",
            "groupBySource": {
              "label": "Source: ${sourceRelPath[2:]}",
              "groupByTags": {
                "tags": [],
                "label": "[${tag}]"
              }
            }
          }
        }
      },
      "gtest": {
        "testGrouping": {
          "groupByExecutable": {
            "label": "${filename}",
            "description": "${relDirpath}/",
            "groupByTags": {
              "tags": [],
              "label": "[${tag}]",
              "groupBySource": {
                "label": "📝${sourceRelPath[2:]}",
                "groupUngroupedTo": "📝unknown source file"
              }
            }
          }
        }
      },
      "doctest": {
        "testGrouping": {
          "groupByExecutable": {
            "label": "${filename}",
            "description": "${relDirpath}/",
            "groupByRegex": {
              "regexes": ["(suite..)"],
              "label": "just ${match}"
            }
          }
        }
      }
    }
  ]
}

"testMate.cpp.test.advancedExecutables": [
  {
    "pattern": "{build,Build,BUILD,out,Out,OUT}/**/*{test,Test,TEST}*",
    "catch2": {
      "testGrouping": {
        "groupBySource": "[-3:]",
        "groupByTags": true, // or [["1","2"],["2"],["3"]],
        "groupByRegex": [ "(apple|peach)" ],
        "groupUngroupedTo": "ungrouped"
      }
    },
    "gtest": {
      "testGrouping": {
        "groupBySource": "[-3:-1]",
        "groupByRegex": [ "(?:good|bad) (apple|peach)" ]
      }
    },
    "doctest": {
      "testGrouping": {
        "groupBySource": "[-1]",
        "groupByRegex": ["apple", "peach"]
      }
    }
  }
]

• ${cmd}
• ${argsFlat}: Example: ["pre", "${argsFlat}", "post"] -> ["pre", "$1", "$2", ..., "$x", "post"]
• ${argsStr}: Example: ["pre", "pre ${argsStr} post", "post"] -> ["pre", " pre \"$1\" \"$2\" ... \"$x\" post", "post"]

Examples:

This is useful if stderr/std::cerr is missing:

"executionWrapper": {
  "path": "/bin/sh",
  "args": [ "-c", "\"${cmd}\" ${argsStr} 2>&1" ]
}

"executionWrapper": {
  "path": "emulator.exe",
  "args": [ "${cmd}", "${argsFlat}" ]
}

• before
• beforeEach
• after
• afterEach

"testMate.cpp.test.advancedExecutables": [
  {
    "pattern": "{build,Build,BUILD,out,Out,OUT}/**/*{test,Test,TEST}*",
    "runTask": {
      "before": [ "taskName in tasks.json" ]
    }
  }
]

"testMate.cpp.test.advancedExecutables": [
  {
    "name": "${filename} [APPLE=1]",
    "pattern": "{build,Build,BUILD,out,Out,OUT}/**/*{test,Test,TEST}*",
    "env":{ "APPLE": "1" }
  },
  {
    "name": "${filename} [APPLE=2]",
    "pattern": "{build,Build,BUILD,out,Out,OUT}/**/*{test,Test,TEST}*",
    "env":{ "APPLE": "2" }
  }
]