Regress: A Test Suite Automation and Reporting Tool
Newsgroup: mozilla.dev.tech.cryptoTechnical contact: Bob Relyea
Yell at the manager: Bob Lord
Overview
Regress is a test case execution and reporting tool that operates as follows. It accepts a specification file [that is intended to be platform and location independent] that defines a number of test cases (programs) and a description for each case. Regress runs the series of programs, in the order of the specfile, and captures into an html output file a table of results: the date/time of run, program run, and results (PASS or FAIL). Regress also generates a summary output file suitable for collecting overall results from a number of test suites. Regress generates results that are utilized by reporter.Command line interface
regress specfile=filename [htmldir=dir] [progdir=dir] [debug][commands in brackets are optional]
specfile specifies the input specfile (defined below) that regress
uses to find out what to execute. The path to the file can be relative
or absolute.
If debug is in the command line, information will be output
to STDOUT so developers of regress can see what is going on.
progdir specifies the base directory where all the executables
are. It could be a relative or absolute path.
htmldir specifies the base directory where all the html results
and reporter specs are output to. It could be a relative or absolute path.
When specifying directories in this document, it is not necessary to
worry about the "/" vs. the "\" conventions. The regression tool will figure
out which slash to use.
| Error Codes | Description |
| 0 | No error |
| 1 | Did not include specfile= directive on command line |
| 2 | Specfile is malformed or does not exist |
| 3 | No [General] section in specfile |
| 4 | No mut= line in [General] section |
| 5 | No mutversion= line in [General] section |
| 6 | No htmlout= line in [General] section |
| 7 | No platform= line in [General] section |
| 8 | No reporterSpec= line in [General] section |
Input
The following is a specification of the regress tools Test Case Specification file:| Specification File Element | Description of Element |
| [General] | Starts the General information block that describes the collection of unit tests in this directory. |
| mut=description | Defines a description for the module under test (e.g., ns/security/lib/crypto module could be described as "Security Library: Cryptographic Module (lib/crypto)". |
| mutversion=versionstring | Defines a version string for the collection of tests. It is assumed that each test will execute against this higher level versionstring version, and that the results will be reported against this version string. |
| htmlout=<filename> | Defines the location and filename relative to htmldir of the html test output that is generated by the regress tool. If the <filename> is NOW, then a yyyymmdd.html string will be substituted for the filename (eg. 19970801). This has to be a relative path to the htmldir specified above. If htmldir is not set, it is automatically default to "." |
| reporterSpec=<filename> | Defines the location and filename of the specfile that is generated by the regress tool and read in by the reporter tool). (See also specification of reporter specfile.) If the <filename> is NOW, then a yyyymmdd.rpt string will be substituted for the filename (eg. 19970801). This has to be a relative path to the htmldir specified above. If htmldir is not set, it is automatically default to "." |
| program=<filename> | Program name placed in general section is used with single source testing method only, which also requires to have test IDs in the individual test sections. |
| globalArgs=<arg list string> | Defines a list of arguments tacked on to all of the programs when the regression suite is run (this will simply concatenate the <program><space><globalArgs> together). This header is optional. |
| bugQueryUrl=<URL with metavariable> | Defines a URL that has a metavariable $[BUG-NUMBER] that is replaced by the bug number when a test is a known failure. This header is optional. |
| [Test-X] | Begins an information block for a specific test. The X value must be replaced with an integer >= 1. The following name value pairs are related to each [Test-X] block. |
| program=<filename> | This is the relative path (relative to the progdir) to the executable.
This is the program to execute that calls PR_ProcessExit or return
and returns a 0 for success, or 1 for failure. If progdir is not
set, progdir is defauted to "." It is not necessary to add a ".exe"
extension at the end of the program name when specifying a Windows / DOS
executable.
Program won’t be specified in this section with single source testing method. |
| testId=<test ID> | testId is used with single source testing method only. It is transferred to the test as a first argument. |
| testname=<description> | This is the description of what the test program is testing. |
| timeout=<integer> | This is the timeout in seconds to run a test. If a test runs longer than the timeout, it is killed and returns fail. This field is optional. If it is not set, timeout is automatically set to 10 seconds. |
| bug=<integer> | This is an optional field. This is set whenever there is a known test failure. The test will NOT be run when this field is set. |
| include=<platforms> | This is an optional field. This is set whenever a test is specific
to a certain platform or group of platforms. The test will be run only
on the platforms (separated by commas or whitespace) in the include list.Check
the list for what string to use for your
platform. If your platform is not on the list, check the output platform
header in the output HTML to see the string you need to use.
This field is case insensitive (eg. you can use solaris2.5 if you want). The test will result in a PASS if the platform is not in the include list (The test didn't really fail, it just wasn't run). Note: having both an include and exclude tag is invalid and will result in an error. |
| exclude=<platforms> | This is an optional field. This is set whenever a test is not
supposed to be run on a certain platform or group of platforms. The test
will not be run on the platforms (separated by commas or whitespace) in
the exclude list. Check the list for what
string to use for your platform. If your platform is not on the list, check
the output platform header in the output HTML to see the string you need
to use.
This field is case insensitive (eg. you can use solaris2.5 if you want). The test will result in a PASS if the platform is in the exclude list (The test didn't really fail, it just wasn't run). Note: having both an include and exclude tag is invalid and will result in an error. |
Platform strings
The following is a specification of sample platform strings you can use for the include and exclude headers:SOLARIS2.5
SOLARIS2.4
SOLARIS2.3
WINNT
HPUXB.10.10
OSFV4.0
AIX1
IRIX5.3
IRIX6.2
The following is a specification of what the return value should be for each Individual Test case executable.
Each of these test cases makes up one singular test within the overall regression test suite. The regression test suite is defined by the Test Specification File.
| Output Description | Format of Output |
| Test pass fail status indicator. The program must call PR_ProcessExit(0) if the test passes. If any other value is returned, that means that the test fails. | 0 -- PASS
non-zero value -- FAIL |
Note: If the Individual Test case requires known answers to benchmark
the results of the execution against, then it is the responsibility of
the individual test case to manage the creation / deletion of startup /
shutdown known answers.
Output
The following is a specification of the html output format for the
regress tool:
| Section Description | HTML format |
| Document Title
The title is generated from the mut= specifier. The version is generated from mutversion=. The date and time is relative to the beginning point of the execution of the regression test suite. |
<HTML><HEAD><TITLE>mut description: Version: mutversion Platform: platform Run date mm/dd/yy</TITLE></HEAD><BODY>\n |
| H1 Header for the report. | <H1>mut description: Version: mutversion
Platform: platform Run date mm/dd/yy</H1><HR>\n
|
| Table header | <TABLE BORDER=1><TR>\n
<TD><B>Test Case Number</B></TD>\n <TD><B>Program</B></TD>\n <TD><B>Description of Test Case</B></TD>\n <TD><B>Start date/time<B></TD>\n <TD><B>End date/time<B></TD>\n <TD><B>PASS/FAIL</B></TD>\n </TR>\n |
| Table Detail
Open the output file defined by htmlout= specifier in the [General] section of the specification file. For each test case header [Test-X] (X >= 1) in the Test Specification file,
(2) Execute the program (program=filename) capturing the return code. If program runs past timeout, kill the program. (3) Capture the date/time after execution (this is the end date/time). (4) Generate the HTML ouptut -- if the specfile is missing some headers, then the result is "MALFORMED". Otherwise, the result string is either PASS or FAIL. If the test returns a pass, the test row is greyed out. If the test fails, the result will be in red. |
<TR [if pass then BG COLOR="#C0C0C0">]\n
<TD><B>X</B></TD>\n <TD>filename</TD>\n <TD>description</TD>\n <TD>mm/dd/yy hh:mm:ss</TD>\n <TD>mm/dd/yy hh:mm:ss</TD>\n <TD><B> PASS | PASS : This test was excluded or not included for this platform | <FONT COLOR="#FF0000"> FAIL: return code = errnum. </FONT> | <FONT COLOR="#FF0000"> FAIL: CreateProcess </FONT> | <FONT COLOR="#FF0000"> FAIL: WaitProcess </FONT> | <FONT COLOR="#FF0000"> FAIL: Known bug. Bug number = 51325 </FONT> | <FONT COLOR="#FF0000"> FAIL: Timeout after 10 seconds </FONT> | MALFORMED : include= and exclude= tags BOTH defined | MALFORMED : program= not defined in [header] | MALFORMED : testname= not defined in [header] | MALFORMED : test number=header-part not valid positive integer | MALFORMED : test number header invalid </B></TD>\n </TR>\n |
| Table trailer | </TABLE>\n |
| Summary Line | <TABLE>
<TR><TD><B>Summary Results</B></TD><TD>Count</TD></TR> <TR><TD>Tests Failed</TD><TD><number failed></TD></TR> <TR><TD>Test Passed</TD><TD><number passed></TD></TR> <TR><TD>Known Failures</TD><TD><known failures></TD><TR> <TR><TD>Malformed Specification</TD><TD><number malformed></TD></TR> </TABLE> |
| End of HTML | </BODY></HTML>\n |
| Specification File Element | Description of Element |
| [Status] | Is the only block in this specification file |
| mut=description | Defines a description for the module under test (e.g., ns/security/lib/crypto module could be described as "Security Library: Cryptographic Module (lib/crypto)". |
| mutversion=versionstring | Defines a version string for the collection of tests. It is assumed that each test will execute against this higher level versionstring version, and that the results will be reported against this versionstring. |
| platform=<string> | Defines the platform the tests were run under |
| pass=<number> | Number of tests that passed. |
| fail=<number> | Number of tests that failed due to non-zero return codes, timeouts, OR any other abnormality when running the test. |
| knownFail=<number> | Number of tests that were known to fail because a bug number was specified in its regress specfile. |
| malformed=<number> | Number of tests with bad headers. |
FAIL Code Descriptions
| Output | Description |
| FAIL: return code = 3 | When the test program exits with non zero return code. |
| FAIL: Create Process | Regress could not run the program or create the process. This usually means a misspelling of the program name, or that you are running on a machine that does not support processes (possibly MAC) |
| FAIL: Wait Process | Regress could not get the return code of the process for some reason. | ug
| FAIL: Timeout after X seconds | The program ran longer than the alloted X seconds and was killed. |
| FAIL: Known bug. Bug number = 53416 | The bug = tag in the specfile is set, so the test is a known bug. |
| FAIL: Known hang | A kludge in an older version of regress that recognizes special program names that block forever and outputs this error instead of running them. |