logtalk_tester - Logtalk script for automating running unit tests
logtalk_tester -v
logtalk_tester -p prolog [-o output] [-m mode] [-f format] [-d results] [-t timeout] [-n driver] [-s prefix] [-b tracker] [-u url] [-c report] [-l level] [-e exclude] [-i options] [-g goal] [-r seed] [-w] [-- arguments]
logtalk_tester -h
logtalk_tester is a script for automating running Logtalk unit tests. This script runs unit tests found on the current directory and recursively in its sub-directories by scanning by default for tester.lgt and tester.logtalk source files. The script returns a non-zero exit status in case of failed tests or test set errors. When a tester.sh file exists in the tests directory, the file is sourced before running the tests. The tester.sh file is sourced with all the parameters passed to the script. The script prints by default a full report of test compilation errors and warnings, skipped test sets, broken test sets, crashed test sets, skipped tests, failed tests, and test set and test stats. Note that the existence of broken or crashed test sets may result in some inconsistency on test stats, notably in the number of passed and failed tests.
-v Print script version.
-p Backend Prolog compiler. Required. The following identifiers are used: B-Prolog - b, Ciao Prolog - ciao, CxProlog - cx, ECLiPSe - eclipse, GNU Prolog - gnu, GNU Prolog (native code) - gnunc, JIProlog - ji, LVM - lvm, SICStus Prolog - sicstus, SWI-Prolog - swi, SWI-Prolog with Logtalk pack - swipack, Tau Prolog - tau, Trealla Prolog - trealla, XSB - xsb, YAP - yap.
-o Output verbosity. Default is verbose. Valid values are verbose and minimal.
-m Compilation mode. Default is normal. Valid values are optimal, normal, debug, and all.
-f Format for writing the test results. Default is default. Valid values are default, tap, xunit, and xunit_net_v2.
-d Directory to store the test results. Default is the local ./logtalk_tester_logs directory.
-t Timeout in seconds for running each test set. Default is 0 (i.e. disabled).
-n Name of the tests driver and sourced files (minus file name extensions). Default is tester.
-s Path prefix to suppress when printing test suite and test result paths. Default is the user home directory.
-b Bug report server and optionally the bug report label(s). Valid values are github and gitlab. (no default). Use of this option requires the script to be called from a git repo directory or from one its sub-directories and the use of the -s and -u options. To set the label(s), use the syntax -b server:labels where labels are a comma separated list.
-u Base URL for generating links to test files (no default). Must end with a slash and usually includes a git hash so that the link remains valid if the test files are later updated.
-c Code coverage report. Default is none. Valid values are xml and none.
-l Directory depth level to look for test sets. Level 1 means current directory only. Default is to recurse into all sub-directories.
-e Regular expression for excluding directories (using the find command support for POSIX-extended regular expressions).
-i Integration script command-line options (no default).
-g Initialization goal. Default is true. Called after loading the Logtalk compiler and runtime.
-r Random generator starting seed (no default).
-w Wipe test set default scratch directories (./.lgt_tmp and ./lgt_tmp) before running it. This option should not be used when running parallel processes that use the same test sets.
-- Arguments to be passed to the tests (no default).
-h Provide help on script usage.
0 All non-skipped tests pass (flaky test failures are ignored) and there are no broken, timedout, or crashed test sets.
1 Some (non-flaky) tests fail but there are no broken, timedout, or crashed test sets.
3 One or more test sets timedout while executing but there are no broken or crashed test sets.
5 One or more test sets are broken and thus could not be executed but there are no crashed test sets.
7 One or more test sets crashed the process running them.
9 The sourced tester.sh file returned a non-zero code.
11 The find command used internally to enumerate test sets returned a non-zero code (most likely due to a non-supported option).
Run all tests on the current directory and sub-directories using the GNU Prolog backend, with results in TAP format, code coverage reports in XML format, and passing arguments foo, bar, and baz to the tests: logtalk_tester -p gnu -tap -c xml -- foo bar baz
Run all tests on the current directory only using the ECLiPSe backend, with a timeout of 4 minutes per test set: logtalk_tester -p eclipse -l 1 -t 240
Run all tests on the current directory and sub-directories using the SICStus Prolog backend and a random generator starting seed: logtalk_tester -p sicstus -r "seed(3172,9814,20125)"
Run all tests on the current directory and sub-directories using the SWI-Prolog backend and looking for tester_debug driver and sourced files: logtalk_tester -p swi -n tester_debug
Run all tests on the current directory and sub-directories using the Tau Prolog backend, caching the compilation of the testing tool for better performance, but wiping the test set default scratch directories before running it: logtalk_tester -p tau -g "set_logtalk_flag(clean,off)" -w
Run the Logtalk distribution tests using the Trealla Prolog backend, passing a base URL that can be used for creating links to the test files: logtalk_tester -p trealla -f xunit_net_v2 -s "$LOGTALKUSER/" -u https://github.com/LogtalkDotOrg/logtalk3/tree/55aa900775befa135e0d5b48ea63098df8b97f5c/
Run the Logtalk distribution tests using the GNU Prolog backend, passing a base URL to automatically create bug reports: logtalk_tester -p gnu -b github:bug,auto -s "$HOME/logtalk3/" -u https://github.com/LogtalkDotOrg/logtalk3/tree/$(git rev-parse HEAD)/
Run all tests on the current directory and sub-directories using the YAP backend with its disable signal handling option: logtalk_tester -p yap -i "-nosignals"
Run all Prolog standards compliance tests with the exception of the catch_3 and cut_0 test sets: logtalk_tester -p swi -e ".*/(catch_3|cut_0)/.*"
When the script detects either a timeout or a gtimeout command (provided by the GNU coreutils package), it will use it to run each test set if the timeout option is set to a value greater than zero. This package can be easily installed in POSIX operating-systems (including Linux and macOS) if not provided by default. On Windows operating-systems, the native timeout command is not usable for this purpose. But the Git for Windows installer includes the GNU coreutils package, which can be used from the bundled Bash shell.
Installation of the GNU sed command, when not available by default, is strongly recommended. The Git for Windows installer includes this command, which can be used from the bundled Bash shell.
The version of the find command provided by some operating-systems doesn't support (extended) regular-expressions, thus preventing using the -e option.
The gnunc backend requires creating a GNU Prolog executable that includes Logtalk named gplgtnc and making it available in the system PATH. This executable can be created using the embedding script provided in the Logtalk distribution. For example: gprolog_embed_logtalk.sh -c -d ~/collect -s none -- --new-top-level && mv ~/collect/logtalk ~/collect/gplgtnc
LOGTALKHOME Specifies the location of the Logtalk installation.
LOGTALKUSER Specifies the location of the Logtalk user directory.
$LOGTALKUSER/library/tester_versions.lgt Logtalk source file used for printing the Logtalk and backend Prolog compiler versions.
$LOGTALKUSER/library/parallel_logtalk_processes_setup.pl Logtalk source file used for parallel Logtalk processes with selected backend Prolog compilers.
$LOGTALKUSER/tools/lgtunit/automation_report.lgt Logtalk source file that intercepts unit test execution messages and generates report files that are parsed by this script.
The Logtalk website at https://logtalk.org/
The Logtalk User and Reference Manuals at $LOGTALKHOME/manuals/index.html
Paulo Moura <pmoura@logtalk.org>
Copyright (c) 1998-2023 Paulo Moura.
This HTML page was made with roffit.