Navigation :

N2PTF Test

Overview

The n2ptf_test program is provided to aid in integrating CI/CD scripts against the N2PTF middleware API.

Supports:

Secure CI/CD OAuth access for confidential machine accounts.
Output of XML driven test formats for display of test outcomes in CI/CD platforms.
- Google Test Output.
Ability to run singular or a suite of tests.
Configuration of library files, directories or substitution of imports.
Configuration of initial arguments to pass to test executions.

Running ./n2ptf_test.pl --help will show the program usage:

usage: n2ptf_test.pl <lua-file> [... <lua-file>]

        --connection <connection-string>    Connection string to utilize when connecting to the N2PTF middleware.
                                            Default = 'http://localhost:18080'.

        --environment <env>                 Define the environment to execute against using the middleware layer.

        --output-format <format>            Output test output as a formatted file that can be used by invoking applications to run statistics.
                                            The currently supported formats are:

                                                google = Google XML test file format.
                                                junit  = JUnit XML test file format.

        --output-file <out-file>            If an output format is specified an output file location may be specified
                                            otherwise output will be provided via STDOUT.

        --output-trace <out-dir>            Directory (will be created if it does not exist) to output trace logging for each
                                            test Lua script that is being executed.

        --debug                             First level of tracing. Generally includes debugging logs.
        --dump                              Second level of tracing. Includes all debug logs and request/response payloads.
        --spam                              Third level of tracing. Includes all debug and dump logs as well as internal logs.

        --lib <directory | file | mapping>  One or more instances of --lib may be defined. These imports are used to load library files.
                                            When loading library files there are 3 supported configuration types:

                                                A) A directory.
                                                B) A singular file.
                                                C) A singular file mapped to a specific import name.

                                            When specifying a directory all directories are recursed looking for Lua files which will be
                                            available as imports using the directory structure itself followed by the filename.

                                            When specifying a file the filename will be available for the import.

                                            Files may be attributed to specific import names delimited by a = character.

                                                For example                   : config=/some/path/to/config_override.lua
                                                Is usable in the Lua script as: import * from config

        --initial-args      <args>          A JSON string containing arbitrary arguments to be passed to the Lua script
        --initial-args-file <file>          A filename for a file containing arbitrary JSON arguments to be passed to the Lua script.

        If the PTF test utility is expected to be run in a secure environment; the OAuth2 secure client credential flow is supported.
        In this context the script is being expected to be run in a CI/CD pipeline which will export environment variables containing
        secure information such as secret keys for authentication. The following environment variables are supported:

            N2PTF_AUTH_URI           - The URI to the authentication servers Open ID token endpoint.
            N2PTF_AUTH_CLIENT_ID     - The client to authenticate against.
            N2PTF_AUTH_CLIENT_SECRET - The secret key for the client to authenticate against.

Common Options

The following common options may be specified when executing n2ptf_test.

Option	Type	Default	Description
`--connection`	String	`http://localhost:18080`	IP address or FQDN of the N2PTF middleware to connect to.
`--environment`	String		[Optional] The environment name to utilize when performing testing against the N2PTF middleware layer. Note: This is optional if the CI/CD environment has access to only one singular execution environment. Singular environments will automatically be used. If more than one or no environments are present the script will exit with an error code.
`--output-format`	String		[Optional] Required if `--output-file` is set. Specifies additional output formatting to generate along side standard test and trace output. If `--output-file` is not set output will be included on `STDOUT`.
`--output-file`	String		[Optional] Specifies the output file location of the specified output format to generate to.
`--output-trace`	String		[Optional] Specifies the output directory to utilize when generating trace output for individual test scripts. Note: If the directory does not exist it will be created.
`--debug`	Switch	`no`	Enables level one tracing.
`--dump`	Switch	`no`	Enables level two tracing.
`--spam`	Switch	`no`	Enables level three tracing.
`--lib`	String		[Optional] Allows for the specification of library files and directories that will be bundled with test script executions. These may be specified as: A) A single directory in which all Lua files will be recursively included. B) A singular Lua file. C) A singular Lua file mapped to an overridden require name.
`--initial-args`	String		[Optional] Allows for the specification of a JSON string that will be provided to each test script at run time. Note: Only one of `--initial-args` and `--initial-args-file` may be specified.
`--initial-args-file`	String		[Optional] Allows for the specification of a JSON file that will be loaded and will be provided to each test script at run time. Note: Only one of `--initial-args` and `--initial-args-file` may be specified.

Environment Variables

The n2ptf_test scripts also supports a number of environment variables that may be used to define secrets or properties to pass to it from a secure CI/CD platform.

This allows n2ptf_test to run in a secure client credential OAuth2 flow while being executed by a CI/CD platform. The following environment variables are supported:

Environment Variable	Type	Description
`N2PTF_AUTH_URI`	String	URI to the authentication servers Open ID token endpoint to which `n2ptf_test` will perform machine to machine OAuth against and retrieve a token to pass to the Jarvis middleware layer.;
`N2PTF_AUTH_CLIENT_ID`	String	The client ID of the application to authenticate against.
`N2PTF_AUTH_CLIENT_SECRET`	String	The secret key of the application to authenticate against.

Example Usage

Overview

The following example sets options that create the following outcomes:

Runs a single test script test.lua.
- The test script is a simple script that uses a number of basic match stages.
Outputs a single trace output file for test.lua in ./ptf-trace.
Bundles the following library dependencies:
- A singular library file single_library.lua.
- A directory of library files stored in ./ptf-example/lib.
- A singular library file mapped_library.lua which is mapped and accessible via require 'mapped' in the Lua.
Additional output in a Google XML Test Output format is enabled and written to STDOUT.

Example Command

./n2ptf_test.pl 
    --output-trace  ./ptf-trace/
    --lib           ./ptf-example/lib 
    --lib           ./ptf-example/single_library.lua 
    --lib           mapped=./ptf-example/mapped_library.lua 
    --output-format google
    ./ptf-example/test.lua

Example Output

Have libraries configuration. Parsing for library files.
Checking ./ptf-example/lib for Lua library.
Bundling ./ptf-example/lib/n2/library.lua as import signature 'n2.library' library dependency.
Bundling ./ptf-example/lib/n2/secondary_library.lua as import signature 'n2.secondary_library' library dependency.
Checking ./ptf-example/single_library.lua for Lua library.
Bundling ./ptf-example/single_library.lua as library dependency keyed on import 'single_library'
Checking mapped=./ptf-example/mapped_library.lua for Lua library.
Detected variable delimited library mapping overriding ./ptf-example/mapped_library.lua to override import 'mapped'.
Bundling ./ptf-example/mapped_library.lua as library dependency keyed on import 'mapped'

[1/1][START ./ptf-example/test.lua]
[STARTED n2ptf_095270_66149]
[RUNNING... ACTION IDX = 11, ACTION = +wait]
[RUNNING... ACTION IDX = 11, ACTION = +wait]
[RUNNING... ACTION IDX = 14, ACTION = +wait]
[RUNNING... ACTION IDX = 15, ACTION = +wait]
[COMPLETED 1(1) of 1 instances in 3.042 seconds. Aborted = 0, Failed = 1, Cancelled = 0]
>>> FIRST CALL CHECKS
>>> ... [04|14:27:50.870290] [PASS] Testing Match Pass
>>> *** [05|14:27:50.870349] [FAIL] Actual string [Testing String Match] should = 'test no match', but = 'test'.
>>> ... [06|14:27:50.870389] [PASS] Testing Match Pass
>>> ... [07|14:27:50.870419] [PASS] Testing Match Pass
>>> ... [08|14:27:50.870500] [PASS] Testing Match Pass
>>> ... [09|14:27:50.870539] [PASS] Testing Match Pass
>>> *** [10|14:27:50.870589] [FAIL] Actual string [Testing String Match] should = 'test no match', but = 'test'.
>>> *** [12|14:27:51.873990] [FAIL] Actual string [Testing String Match] should = 'test no match', but = 'test'.
>>> ... [16|14:27:53.909600] [PASS] Actual string [Testing String Match] = test, as expected.
>>> *** [18|14:27:53.909729] [FAIL] Actual number [Testing Elapsed Match] should be range 1 - 100, but = 1781095273.9096.
>>> First Instance 4 FAIL/ABORT test result(s) out of 10 tests.


Generated Google Test XML:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites tests="10" time="3.04203009605408" failures="4" errors="0">
  <testsuite name="./ptf-example/test.lua" tests="10" time="3.04203009605408" failures="4" errors="0">
    <testcase status="run" time="0" classname="" name="Testing Match Pass"/>
    <testcase status="run" time="0" classname="" name="Test expected string [Testing String Match].">
      <failure message="Actual string [Testing String Match] should = 'test no match', but = 'test'." type=""/>
    </testcase>
    <testcase status="run" time="0" classname="" name="Testing Match Pass"/>
    <testcase status="run" time="0" classname="" name="Testing Match Pass"/>
    <testcase status="run" time="0" classname="" name="Testing Match Pass"/>
    <testcase status="run" time="0" classname="" name="Testing Match Pass"/>
    <testcase status="run" time="0" classname="" name="Test expected string [Testing String Match].">
      <failure message="Actual string [Testing String Match] should = 'test no match', but = 'test'." type=""/>
    </testcase>
    <testcase status="run" time="0" classname="" name="Test expected string [Testing String Match].">
      <failure message="Actual string [Testing String Match] should = 'test no match', but = 'test'." type=""/>
    </testcase>
    <testcase status="run" time="0" classname="" name="Test expected string [Testing String Match]."/>
    <testcase status="run" time="0" classname="" name="Test expected number [Testing Elapsed Match].">
      <failure message="Actual number [Testing Elapsed Match] should be range 1 - 100, but = 1781095273.9096." type=""/>
    </testcase>
  </testsuite>
</testsuites>