Reporter Tool Specification
Newsgroup: mozilla.dev.tech.cryptoTechnical contact: Bob Relyea
Yell at the manager: Bob Lord
Contents
Overview
Reporter is a productivity tool that automates the web indexing of regression results generated by the regress tool. The tool takes a specfile (defined below) as input, which defines the Component/Test Suite/Platform topology of the report directory structure. The tool then parses all the individual regress summary files found at the lowest level of the reporting hierarcy and updates all the intermediate index.html files. The reporter tool is expected to automate the routine maintenance of updating web content to form an orderly and predictable testing results web, that can be reindexed automatically.
|
Command Line Arguments
reporter specfile=<filename> [rptdir=<dirname>]
[debug] [novrml] [createdirs]
Note: The arguments above are order independent. |
Error Reporting
The following table defines hard error will will prevent reporter from completing its task.Error Code | Description |
NO_ERROR (0) | Program ran to completion and completed its indexing job. |
ERR_NO_RPTDIR (1) | Program could not open the directory rptdir (as defined in [General] section or passed in on command line). |
ERR_NO_SPECFILE (2) | Program could not open the specfile passed in on the command line. |
ERR_NO_GENERAL (3) | Could not locate a [General] section in the specfile. |
ERR_NO_RPTDIR (4) | Could not locate a rptdir= line in the [General] section. Only if rptdir not passed in on the command line. |
ERR_NO_COMPONENT (5) | Specification file does not contain a [Component] section. |
ERR_NO_TESTSUITE (6) | Specification file does not contain a [TestSuite] section. |
ERR_NO_PLATFORM (7) | Specification file does not contain a [Platform] section. |
ERR_IDX_NOWRITE (8) | Failed to write (or over-write) one of the index.html files. |
ERR_USAGE (9) | Provided an invalid or undefined argument. See usage. |
ERR_BADSECTIONS (10) | One of the required fields in one of the sections was missing. |
Error Message - reported to standard output | Description |
Fatal error: One of the required fields (field) in <section name> was not present. | One of the fields required in provide section was missing, see specification., corresponds to the hard error ERR_BADSECTIONS. |
reporter: Malformed summary file <filename> : could not locate <field> field. | The summary file, identified by <filename> did not conform to the specifiction (see below). |
reporter: Could not open directory <dirname> defined in section <section name>. | Attempted to open a directory which did not exist. The dirname
variable as composed as follows:
|
Fatal error: error code = <number> : message | Additionally, all hard errors will be reported, as defined in the table above, to standard output., use for all ERR_ codes defined above that don't have a more specific message defined in this table. |
reporter: Usage: reporter specfile=<filename> [rptdir=<dirname>] [debug] | Corresponds to ERR_USAGE defined above. |
Format of Summary Specification files
As a regression suite completes (driven by the regress tool), it writes out an HTML output file, and a regression summary specification file. Both of these files are utilized in the creation of the higher level summary index files.The following defines the contents of the reporter summary
specification file, as generated by regress:
Specification File Element | Description of Element |
[Status] | Is the only block in this specification file |
mut=description | Defines a description for the module under test (e.g., ns/security/lib/crypto module could be described as "Security Library: Cryptographic Module (lib/crypto)". |
mutversion=versionstring | Defines a version string for the collection of tests. It is assumed that each test will execute against this higher level versionstring version, and that the results will be reported against this versionstring. |
platform=<string> | Defines the platform the tests were run under (e.g., HP UX 1.0, Solaris 2.4.1, etc.) |
pass=<number> | Number of tests that passed. |
fail=<number> | Number of tests that failed due to non-zero return codes, timeouts, OR any other abnormality when running the test. |
knownFail=<number> | Number of tests that were known to fail because a bug number was specified in its regress specfile. |
malformed=<number> | Number of tests with bad headers (see regress specfile specifications). |
Format of Reporter Input Specification File
The following defines the contents of the reporter input specification file:Section / Field names | Description |
[General]
rptdir=<directory> [index=filename] [vrml=[TRUE|FALSE]] |
|
[Component-name]
desc=<description> version=<version> dirname=<directory name> |
|
[TestSuite-name]
desc=description component=name dirname=<directory name> |
|
[Platform-name]
desc=description dirname=<directory name> testsuite-name=yes|no |
|
Pseudo-code of reporter program flow
Data structures:enum DirNodeType{ DIRROOT, COMPONENT, TSUITE, PLATFORM, FILE} ;
typedef struct _StatusNode {
typedef struct _DirectoryNode {
|
/* 1.0 Process specification file */
Parse command line arguments, on error generate ERR_USAGE, otherwise set debug, rptdir, specfile variables. Open specfile, if not then ERR_NO_SPECFILE. if index entry in [General Section] set index to filename, otherwise index = index.html. if no [General Section] then ERR_NO_GENERAL. if rptdir=NULL and no rptdir in [General Section] then ERR_NO_RPTDIR. /* 1.1 Add all component names to DirectoryNode tree */
/* 1.2 Add all test suite names to DirectoryNode tree */
/* 1.3 Add all test suite names to DirectoryNode tree */
/* 2.0 Open each of the status nodes, and augment tree with HTML
file objects */
/* 2.1 Select most recent report at the PLATFORM level and store
in tree */
/* 2.2 Calculate suite results */
/* 2.3 Calculate component results */
/* TAT.6 create VRML display */
By setting the tree height limit to 2 (i.e the tallest CoordNode tree created will have at most 3 levels), only the root and each test suite node will be represented by a .wrl file. The name of the .wrl file at the root is "Root.wrl" and the at each Testsuite is <testsuitename>.wrl. /* 3.0 Write out all index files at ROOT, COMPONENT, SUITE, and
PLATFORM level */
|
Format of generated index.htm files
Level | HTML Format |
ROOT | <HTML>
<HEAD></TITLE></TITLE></HEAD> <BODY> <TABLE> For each component in ROOT { <TR><TD><A HREF="<filename>/<index>">desc</A></TD></TR> } </TABLE> </BODY> </HTML> |
COMPONENT | <HTML>
<HEAD></TITLE></TITLE></HEAD> <BODY> <TABLE> For each suite in component node { <TR><TD><A HREF="<filename>/<index>">desc</A></TD></TR> } </TABLE> </BODY> </HTML> |
SUITE | <HTML>
<HEAD></TITLE></TITLE></HEAD> <BODY> <TABLE> <TR><TD></TD></TR> </TABLE> </BODY> </HTML> |
PLATFORM | <HTML>
<HEAD></TITLE></TITLE></HEAD> <BODY> <TABLE> <TR><TD></TD></TR> </TABLE> </BODY> </HTML> |
Sample Reporter input specification file
The following defines a sample input specification file to reporter (which is highly domain specific):[General]
dirname=/u/jhines/saw/projects/hardcore/prj-ttools/testing/rptroot [Component-seclib]
[TestSuite-pkcs5]
[TestSuite-crypto]
[Platform=hp10_2]
[Platform=sol2_4]
|
Notes on the VRML display:
All the code necessary to implement VRML display is packaged in
the files:
genvrml.h genvrml.c
genvrml.h contains c preprocessor CONSTANTS that may be changed
to alter the VRML display. Below is a list of constants and their
purpose:
Constant | Function |
LINE_SHININESS | defines the brightness with which the lines interconnecting nodes appear. Currently set to 1.0 (the maximum). Value can range from 0.0 to 1.0 |
X_ORIGIN, Y_ORIGIN, Z_ORIGIN | defines the default origin. |
NODE_RADIUS | defines the radius of each sphere/node |
CAPTION_X_OFFSET, CAPTION_Y_OFFSET | defines the offset in both x and y directions from the top of each sphere/node with which label text is to be drawn |
FONT_STYLE, FONT_SIZE, FONT_FAMILY | Font attributes for the test label above each node |
MAGIC_NUMBER | The cotagent of an angle (which should be half of the total angle of view). i.e. if an angle of view of 90 degrees is desired, then set MAGIC_NUMBER to cotan(45degrees) |
LEVEL_SPAN | The distance between nodes in each level in the tree. (This is displacement along the y-axis). |
LEAF_SPAN | The distance between each bottom most leaf in the displayed tree.
All leaves have equal (x axis) spacing between them that is defined by this constant. |
Z_SPAN | The distance of displacement between nodes on the Z axis. |