Regress Tool Specification
Newsgroup: mozilla.dev.tech.cryptoTechnical contact: Bob Relyea
Yell at the manager: Bob Lord
Last updated: 6/30/1998
Contents
- Overview
- Command Line Interface
- Regress Return Codes
- Regress Specification File
- Java Mode
- Platform Strings
- Individual Test Return Codes
- Regress Test Report format
- Reporter Specfile output format
- Fail Code Descriptions
- Miscellaneous Notes
Overview
Regress is a test case execution and reporting tool that operates as follows. It accepts a specification file [that is intended to be platform and location independent] that defines a number of test cases (programs) and a description for each case. Regress runs the series of programs, in the order of the specfile, and captures into an html output file a table of results: the date/time of run, program run, and results (PASS or FAIL). Regress also generates a summary output file suitable for collecting overall results from a number of test suites. Regress generates results that are utilized by reporter. Additionally, tools like replacer generate test suites that integrate well with regress.NOTE: When specifying directories in this document, it is not necessary to worry about the "/" vs. the "\" conventions. The regression tool will figure out which slash to use.
Command line interface
regress specfile=filename [htmldir=dir] [progdir=dir] [lang=java|c] [style=table|javascript] [jvm=<java interpreter>] [classpath=<java CLASSPATH setting>] [args=<global arguments>] [debug]
[commands in brackets are optional]| Command line option | Explanation |
| specfile | specifies the input specfile (defined below) that regress uses to find out what to execute. The path to the file can be relative or absolute. |
| debug | If debug is in the command line, information will be output to STDOUT so developers of regress can see what is going on. |
| progdir | specifies the base directory where all the executables are. It could be a relative or absolute path. |
| htmldir | specifies the base directory where all the html results and reporter specs are output to. It could be a relative or absolute path. |
| lang=[java|c] | specifies whether regress is to expect a java class or c executable.
By specifying the option "lang=java" at the command line will put regress
in JAVA_MODE, similarly "lang=c" places regress in C_MODE.
Again, command line options override specfile setting. |
| style=[table|javascript] | specifies the display style of the output HTML, whether it be the table format or Java Script format. If not specified, the default is the javascript format (unless, otherwise specified in the regress specfile) |
| jvm=<java interpreter path> | specifies the path of the java virtual machines used to interpret java .class files. (This option overrides the jvm specified in the specfile). If lang=c then the jvm field is ignored. |
| classpath=<java CLASSPATH setting> | This option allows the added flexibility of letting the user specify the java environment variable CLASSPATH for running their testsuites. This option will overide a classpath setting in the [General] section of the regress specfile. If classpath is not set both at the command line or in the specfile, then the CLASSPATH of the user's environment will be used. (If environment CLASSPATH is not set, then, the default "." (current directory) is used.) |
| args=<global arguments> | This option allows on the fly specification of globalArgs. This field overrides the globalArgs field in the specfile. (This feature may be currently only available on Solaris 5.5.1 versions) |
Regress Return Codes
| Error Codes | Description |
| 0 | All Tests are PASS or Known failures |
| 1 | One or more tests FAIL or are MALFORMED |
| 2 | One or more tests returned the predefined abortCode (see abortCode definition below) |
| 3 | Did not include specfile= directive on command line |
| 4 | Specfile is malformed or does not exist |
| 5 | No [General] section in specfile |
| 6 | No mut= line in [General] section |
| 7 | No mutversion= line in [General] section |
| 8 | No htmlout= line in [General] section |
| 9 | No reporterSpec= line in [General] section |
| 10 | No mainClassName= line in [General] while in Java mode |
| 11 | Usage message displayed; Undefined command line options used. |
Regress Specfile
It is a convention to use a .reg extension for any regress specfiles. This is not checked, but is for purely readability purposes. Please check the convention document if you don't know what extension to put on your files for other people's sake.The following is a specification of the regress tools Test Case Specification
file:
| Specification File Element | Description of Element |
| [General] | Starts the General information block that describes the collection of unit tests in this directory. |
| mut=description | Defines a description for the module under test (e.g., ns/security/lib/crypto module could be described as "Security Library: Cryptographic Module (lib/crypto)". |
| mutversion=versionstring | Defines a version string for the collection of tests. It is assumed that each test will execute against this higher level versionstring version, and that the results will be reported against this version string. |
| htmlout=<filename> | Defines the location and filename relative to htmldir of the html test output that is generated by the regress tool. If the <filename> is NOW, then a yyyymmdd.htm string will be substituted for the filename (eg. 19970801). This has to be a relative path to the htmldir specified above. If htmldir is not set, it is automatically default to "." |
| reporterSpec=<filename> | Defines the location and filename of the specfile that is generated by the regress tool and read in by the reporter tool). See specification of reporter specfile. If the <filename> is NOW, then a yyyymmdd.rpt string will be substituted for the filename (eg. 19970801). This has to be a relative path to the htmldir specified above. If htmldir is not set, it is automatically default to "." |
| program=<filename> | Program name placed in general section is used with single source testing method only, which also requires to have test IDs in the individual test sections. For additional information see replacer. |
| globalArgs=<arg list string> | Defines a list of arguments tacked on to all of the programs when the regression suite is run (this will simply concatenate the <program><space><globalArgs> together). This header is optional. |
| bugQueryUrl=<URL with metavariable> | Defines a URL that has a metavariable $[BUG-NUMBER] that is replaced by the bug number when a test is a known failure. This header is optional. |
| firstKnownFailure=<number between 0 and 256> | Specifies the first return code which regress should consider as a known failure. When specified with first Error code, defines a range of executable return codes that are considered known failures. When not specified or incorrectly specified (i.e. bigger than firstErrorCode) the default value is 0. |
| firstErrorCode=<number between 0 and 256> | Defines the upper bound of known failure codes. [Hence known failure
codes are from
firstKnownFailure to firstErrorCode - 1]. If firstErrorCode is smaller than firstKnownFailure or otherwise undefined, then default value of 0 is used. |
| timeout=<an positive integer| FOREVER> | Specifies the global timeout for the program. The timeout value can be a number from 1 to 2^32 -1, which will be in the units of seconds. By setting timeout to FOREVER means that there won't be a timeout. (regress will let the process run literally forever). The global timeout can be overridden by the timeout attribute for each test. If neither the global timeout or testspecific timeout is given, the default value of 1000 seconds is used. [A Note of warning: a timeout of 0 is interpreted the same way as not setting this field, so the default 1000 seconds would be used in that case.] |
| localExecutable=[TRUE|FALSE] | Normally, if progdir is not specified at the command line, regress
assumes that the executable is in the currently working directory; Regress
will pre-append to the executable: "./<executable>".
If localExecutable = FALSE, this default behavior is turned off, in which
case, regress will prepend nothing in front of the executable (unless progdir
is specified). This option is useful since it allows one to
manipulate the environment PATH variable to select the executable to test.
By default localExecutable is TRUE. |
| autoAppendExtension=[TRUE|FALSE] | By default, regress will append the extension .exe to all executables in the Windows/PC versions. By specfying this field as FALSE will turn off this behavior. By default autoAppendExtension is set to TRUE. |
| abortCode=<number between 0 and 256> | If a test returns a code identical to the abortCode as defined here. That particular test will be marked as "FAIL: ABORT". And regress will return 2 as its return code (after finish running all the remaining tests). If abortCode is not defined, the default value of 255 will be used. |
| style=[table|javascript] | Specifies whether the output report HTML will be in table format or JavaScript format. |
| language=[java|c] | If language=java, puts Regress in JAVA_MODE, and hence regress expects
a java class file.
By default language=c if language is not specified. |
| mainClassName = <name of your class containing main() > | Required if in JAVA_MODE. mainClassName is used to qualify
the names of injected arrays in single file mode, and during given as argument
to the java interpreter as in:
<jvm> <packageName>.<mainClassName> |
| jvm = <path to your Java Interpreter> | Used to specify which java interpreter to use. It's best that full path is specified. If not specified, regress will use "java". |
| packageName=<package name from package declaration> | Required if a package declaration appears in the source file.
Required by Regress to construct the test program command:
<jvm> <packageName>.<mainClassName> |
| classpath=<java CLASSPATH setting> | Required if the test suite uses a CLASSPATH different from that specified in the user environment. If specified, regress will create a CLASSPATH environment variable in its test environment and clobber any previous settings. If this field is not specified, then the classpath from the environment will be used. If no environment CLASSPATH is set, then the default working directory (progdir) will be used. |
| In JAVA_MODE,
If Single Source Mode is specified "program=" field in [General] is not necessary. |
When Single Source Mode is TRUE (on). The user no longer need
to specify
the program field if JAVA_MODE is on. Regress will simple construct the program to run from the jvm, packageName, and mainClassName provided. As a result, Regress will run: <jvm> <packageName>.<mainClassName> <Args> |
| Test Header | |
| [Test-X] | Begins an information block for a specific test. The X value must be replaced with an integer >= 1. The following name value pairs are related to each [Test-X] block. |
| program=<filename> | This is the relative path (relative to the progdir) to the executable.
This is the program to execute that calls PR_ProcessExit or return
and returns a 0 for success, or 1 for failure. If progdir is not
set, progdir is defaulted to "."
It is not necessary to add a ".exe" extension at the end of the program name when specifying a Windows / DOS executable. Program won’t be specified in this section when using single source testing method. |
| In JAVA_MODE If Single Source Mode is NOT specified (Multiple
source mode)
program =<test directory> |
For each test specified, the user must set the program metavariable to the name of the directory in which the Java source code or (source Tree) resides. (for a better explanation see Regress in Java Mode below) |
| testId=<test ID> | testId is used with single source testing method only. It is transferred to the test as a first argument. |
| testname=<description> | This is the description of what the test program is testing. |
| timeout=<positive integer|FOREVER> | This is the timeout in seconds to run a test. If a test runs longer than the timeout, it is killed and returns fail. This field is optional. If this field is not set, the global timeout specified in the timeout= field of the [General] section will be used. The key word FOREVER will cause the test to be run without any timeout. By default, if neither the local timeout nor global timeout is set, 1000 seconds will be used as the timeout limit. |
| bug=<positive integer> | This is an optional field. This is set whenever there is a known test failure. The test will NOT be run when this field is set. |
| include=<platforms> | This is an optional field. This is set whenever a test is specific
to a certain platform or group of platforms. The test will be run only
on the platforms (separated by commas or whitespace) in the include list.Check
the list for what string to use for your
platform. If your platform is not on the list, check the output platform
header in the output HTML to see the string you need to use.
This field is case insensitive (eg. you can use solaris2.5 if you want). The test will result in a PASS if the platform is not in the include list (The test didn't really fail, it just wasn't run). Note: having both an include and exclude tag is invalid and will
result in an error.
|
| exclude=<platforms> | This is an optional field. This is set whenever a test is not
supposed to be run on a certain platform or group of platforms. The test
will not be run on the platforms (separated by commas or whitespace) in
the exclude list. Check the list for what
string to use for your platform. If your platform is not on the list, check
the output platform header in the output HTML to see the string you need
to use.
This field is case insensitive (eg. you can use solaris2.5 if you want). The test will result in a PASS if the platform is in the exclude list (The test didn't really fail, it just wasn't run). Note: having both an include and exclude tag is invalid and will
result in an error.
|
Regress in Java Mode
When Regress is in Java Mode (by setting lang=java at the command line
or language=java in the specfile). Regress expects the following
fields to appear in the specfile under the [General Header]:
mainClassName, jvm, classpath, and packageName.
jvm, classpath, and packageName are optional. jvm is set
to a default of "java" if not specified any where. And classpath
inherits the environment CLASSPATH, if not specified anywhere.
The user is responsible for making sure that the directory tree in which
the .class files are stored exist and the .class files themselves also
exist. Because regress do not read environmental variables,
the value of classpath set in the specfile must be a real path not a variable.
However, if classpath is set at the command line, the argument can be a
variable.
e.g.
A correct usage:
regress specfile=mytest.reg classpath=$CLASSPATH:/usr/bin
But
[General]
...
classpath=$CLASSPATH #WRONG!
..
The classpath can be further appended for each test run. This
is accomplished by adding the attribute
program=<testdir> in each test heading.
[test-1]
program=t0
....
Regress will append the value set in the program field to the CLASSPATH it already has.
When regress runs, it will do the following:
setenv CLASSPATH <value set in classpath=>:<testdir>
<jvm> <fully qualified class name> # i.e. combination
of package name and mainClassName.
Following is an example that illustrates what processes must occur for regress to work in Java mode (multiple source mode).
Example:
A user has 5 tests (.java files). These 5 tests sources are place in the following directories:
t1/foo/bar/t1.java
t2/foo/bar/t2.java
t3/foo/bar/t3.java
t4/foo/bar/t4.java
t5/foo/bar/t5.java
Each .java file has the same package declaration:
package foo.bar;
and has a mainClassName of "test"
After compilation the following 5 .class files were created:
t1/foo/bar/test.class
t2/foo/bar/test.class
t3/foo/bar/test.class
t4/foo/bar/test.class
t5/foo/bar/test.class
In order to run regress for the above example, the user must write a specfile that is something like the following:
[General]
...
language=java
mainClassName=test
packageName=foo.bar
jvm=/tools/ns/bin/java
# no classpath, inherit from environment
....
[Test-1]
program=t1
.... other fields
[Test-2]
program=t2
.... other fields
[Test-n]
program=<test directory n>
.... other fields
Regress takes the above specfile and runs each test in a manner equivalent to the following:
setenv CLASSPATH <$CLASSPATH>:t0 // or
which ever test that is running
/tools/ns/bin/java foo.bar.test
Platform strings
The following is a specification of sample platform strings you can use for the include and exclude headers:SOLARIS2.5
SOLARIS2.4
SOLARIS2.3
WINNT
HPUXB.10.10
OSFV4.0
AIX1
IRIX5.3
IRIX6.2
Individual Test Output
The following is a specification of what the return value should be for each Individual Test case executable.
Each of these test cases makes up one singular test within the overall
regression test suite. The regression test suite is defined by the Test
Specification File.
| Output Description | Format of Output |
| Test pass fail status indicator. The program must call PR_ProcessExit(0) if the test passes. If any other value is returned, that means that the test fails. | 0 -- PASS
non-zero value -- FAIL marked non-zero value(set using firstKnownFailure) -- FAIL -- KNOWN FAIL |
Note: If the Individual Test case requires known answers to benchmark the results of the execution against, then it is the responsibility of the individual test case to manage the creation / deletion of startup / shutdown known answers.
Regress Test Results Report
Table styleThe following is a specification of the html output format for the regress tool in TABLE mode:
| Section Description | HTML format |
| Document Title
The title is generated from the mut= specifier. The version
is
generated from mutversion=. The date and time is relative
to the beginning point of the execution of the regression test suite.
|
<HTML><HEAD><TITLE>mut description: Version: mutversion Platform: platform Run date mm/dd/yy</TITLE></HEAD><BODY>\n |
| H1 Header for the report. | <H1>mut description: Version: mutversion Platform: platform Run date mm/dd/yy</H1><HR>\n |
| Table header | <TABLE BORDER=1><TR>\n
<TD><B>Test Case Number</B></TD>\n <TD><B>Program</B></TD>\n <TD><B>Description of Test Case</B></TD>\n <TD><B>Start date/time<B></TD>\n <TD><B>End date/time<B></TD>\n <TD><B>PASS/FAIL</B></TD>\n </TR>\n |
| Table Detail
Open the output file defined by htmlout= specifier in the [General] section of the specification file. For each test case header [Test-X] (X >= 1) in the Test Specification
file,
(2) Execute the program (program=filename) capturing the return code. If program runs past timeout, kill the program. (3) Capture the date/time after execution (this is the end date/time). (4) Generate the HTML ouptut -- if the specfile is missing some headers, then the result is "MALFORMED". Otherwise, the result string is either PASS or FAIL. If the test returns a pass, the test row is greyed out. If the test fails, the result will be in red. |
<TR [if pass then BG COLOR="#C0C0C0">]\n
<TD><B>X</B></TD>\n <TD>filename</TD>\n <TD>description</TD>\n <TD>mm/dd/yy hh:mm:ss</TD>\n <TD>mm/dd/yy hh:mm:ss</TD>\n <TD><B> PASS | PASS : This test was excluded or not included for this platform | <FONT COLOR="#FF0000"> FAIL: return code = errnum. </FONT> | <FONT COLOR="#FF0000"> FAIL: CreateProcess </FONT> | <FONT COLOR="#FF0000"> FAIL: WaitProcess </FONT> | <FONT COLOR="#FF0000"> FAIL: Known bug. Bug number = 51325 </FONT> | <FONT COLOR="#FF0000"> FAIL: Timeout after 10 seconds </FONT> | MALFORMED : include= and exclude= tags BOTH defined | MALFORMED : program= not defined in [header] | MALFORMED : testname= not defined in [header] | MALFORMED : test number=header-part not valid positive integer | MALFORMED : test number header invalid </B></TD>\n </TR>\n |
| Table trailer | </TABLE>\n |
| Summary Line | <TABLE>
<TR><TD><B>Summary Results</B></TD><TD>Count</TD></TR> <TR><TD>Tests Failed</TD><TD><number failed></TD></TR> <TR><TD>Test Passed</TD><TD><number passed></TD></TR> <TR><TD>Known Failures</TD><TD><known failures></TD><TR> <TR><TD>Malformed Specification</TD><TD><number malformed></TD></TR> </TABLE> |
| End of HTML | </BODY></HTML>\n |
Overview
The strictly tabular format produced by Regress has the following disadvantages:
- When entries get above 8000, the Browser will often crash due to difficulties in drawing the table for such a large number of entries.
- When there are too many entries, it is difficult to go to a specific test and look at the results.
- There is no mechanism in which to look at only fail entries or only pass entries given a page containing a mix of both types
through the data.
The new javascript format has a self contained paging mechanism and allows display of FAILS only or PASS only. Buttons on the bottom of the page adds page up, page down, table top, table bottom, goto page functionalities.
Here is an example of the JavaScript output
The output is separated into 3 frames. The top frame contains
static information such as summary statistics, platform name, test name,
and test description.
The center frame contains the paged result table. Currently,
the number of entries per page is set to 12. But this can be
changed by changing the javascript variable entriesPerPage in the source
file genjs_template.html. (see extra building
instructions). The bottom frame is basically a control panel
that has the paging buttons.
Behavior
When viewing a page in FAIL only or PASS only mode. The javascript
is designed to find as many entries as possible that fit the PASS or FAIL
category starting from the current entry number. For example, in
the case where we are viewing FAIL entries only, if the current entry is
entry 47, and there are only 3 fail entries after 47, the javascript program
will search forward until either it hits the beginning or 12 entries are
found; Then displaying all 12 or less entries found.
Reporter Specfile Output Format:
| Specification File Element | Description of Element |
| [Status] | Is the only block in this specification file |
| mut=description | Defines a description for the module under test (e.g., ns/security/lib/crypto module could be described as "Security Library: Cryptographic Module (lib/crypto)". |
| mutversion=versionstring | Defines a version string for the collection of tests. It is assumed that each test will execute against this higher level versionstring version, and that the results will be reported against this versionstring. |
| platform=<string> | Defines the platform the tests were run under |
| pass=<number> | Number of tests that passed. |
| fail=<number> | Number of tests that failed due to non-zero return codes, timeouts, OR any other abnormality when running the test. |
| knownFail=<number> | Number of tests that were known to fail because a bug number was specified in its regress specfile. |
| malformed=<number> | Number of tests with bad headers (see regress specfile specifications). |
Note: check the Reporter Spec for the latest reporter specification.
Implementation Notes for regress
FAIL code descriptions
| Output | Description |
| FAIL: return code = 3 | When the test program exits with non zero return code. |
| FAIL: Create Process | Regress could not run the program or create the process. This usually means a misspelling of the program name, or that you are running on a machine that does not support processes (possibly MAC) |
| FAIL: Wait Process | Regress could not get the return code of the process for some reason. |
| FAIL: Timeout after X seconds | The program ran longer than the alloted X seconds and was killed. |
| FAIL: Known bug. Bug number = 53416 | The bug = tag in the specfile is set, so the test is a known bug. |
| FAIL: Known hang | A kludge in an older version of regress that recognizes special program names that block forever and outputs this error instead of running them. |
Test Specification File for Regress Tool Testing
Test Specification for regressHTML Output of Regression test on regress
Specification of Directory Structure / Naming Conventions
Regress Special Building Instructions:
Any Javascript related changes can be made to genjs_template.html. (see also javascript changes documentation). Once changes are made, run the perl script: makeGenJS.pl which rebuilds the two files: genjs.c and genjs.h. After which follow normal sectools build instructions.