Mozilla

• Roadmap
• Projects
• Coding
  • Module Owners
  • Hacking
  • Get the Source
  • Build It
• Testing
  • Releases
  • Nightly Builds
  • Report A Problem
• Tools
  • Bugzilla
  • Tinderbox
  • Bonsai
  • LXR
• FAQs

Reporter: A Tool to Collect and Compile Test Results

Newsgroup: mozilla.dev.tech.crypto
Technical contact: Bob Relyea
Yell at the manager: Bob Lord

Overview

Reporteris a productivity tool that automates the web indexing of regression results generated by the regress tool. The tool takes a specfile (defined below) as input, which defines the Component/Test Suite/Platform topology of the report directory structure. The tool then parses all the individual regress summary files found at the lowest level of the reporting hierarcy and updates all the intermediate index.html files. The reporter tool is expected to automate the routine maintenance of updating web content to form an orderly and predictable testing results web, that can be reindexed automatically.

Figure 1: Architecture Overview

 

Command Line Arguments

 

reporter  specfile=<filename> [rptdir=<dirname>] [debug]  Filename is either an absolute path or relative path to an existing filename that contains a specification (as defined below). Dirname is an optional parameter that overrides the dirname line in the general section of the specification file (see below). The optional debug argument will turn on stdout debug information defined below -- soft errors. Note: The arguments above are order independent.

 

Error Reporting

The following table defines hard error will will prevent reporter from completing its task.
 

Error Code	Description
NO_ERROR (0)	Program ran to completion and completed its indexing job.
ERR_NO_RPTDIR (1)	Program could not open the directory rptdir (as defined in [General] section or passed in on command line).
ERR_NO_SPECFILE (2)	Program could not open the specfile passed in on the command line.
ERR_NO_GENERAL (3)	Could not locate a [General] section in the specfile.
ERR_NO_RPTDIR (4)	Could not locate a rptdir= line in the [General] section.   Only if rptdir not passed in on the command line.
ERR_NO_COMPONENT (5)	Specification file does not contain a [Component] section.
ERR_NO_TESTSUITE (6)	Specification file does not contain a [TestSuite] section.
ERR_NO_PLATFORM (7)	Specification file does not contain a [Platform] section.
ERR_IDX_NOWRITE (8)	Failed to write (or over-write) one of the index.html files.
ERR_USAGE (9)	Provided an invalid or undefined argument. See usage.
ERR_BADSECTIONS (10)	One of the required fields in one of the sections was missing.

The following table defines messages which are generated on standard output (if and only if debug parameter is supplied) to log "soft" errors -- errors that can be worked around:
 

Error Message - reported to standard output	Description
Fatal error: One of the required fields (field) in <section name> was not present.	One of the fields required in provide section was missing, see specification., corresponds to the hard error ERR_BADSECTIONS.
reporter:  Malformed summary file <filename> : could not locate <field> field.	The summary file, identified by <filename> did not conform to the specifiction (see below).
reporter: Could not open directory <dirname> defined in section <section name>.	Attempted to open a directory which did not exist.  The dirname variable as composed as follows:  For section = [Component-name], dirname = <rptdir>/<component dirname> For section = [TestSuite-name], dirname = <rptdir>/<component dirname>/<testsuite dirname> For section = [Platform-name], dirname = <rptdir>/<component dirname>/<testsuite dirname>/<platform dirname>
Fatal error: error code = <number> : message	Additionally, all hard errors will be reported, as defined in the table above, to standard output., use for all ERR_ codes defined above that don't have a more specific message defined in this table.
reporter: Usage: reporter  specfile=<filename> [rptdir=<dirname>] [debug]	Corresponds to ERR_USAGE defined above.

 

Format of Summary Specification Files

As a regression suite completes (driven by the regress tool), it writes out an HTML output file, and a regression summary specification file.  Both of these files are utilized in the creation of the higher level summary index files.

The following defines the contents of the reporter summary specification file, as generated by regress:
 

Specification File Element	Description of Element
[Status]	Is the only block in this specification file
mut=description	Defines a description for the module under test
mutversion=versionstring	Defines a version string for the collection of tests. It is assumed that each test will execute against this higher level versionstring version, and that the results will be reported against this versionstring.
platform=<string>	Defines the platform the tests were run under (e.g., HP UX 1.0, Solaris 2.4.1, etc.)
pass=<number>	Number of tests that passed.
fail=<number>	Number of tests that failed due to non-zero return codes, timeouts, OR any other abnormality when running the test.
knownFail=<number>	Number of tests that were known to fail because a bug number was specified in its regress specfile.TD>
malformed=<number>	Number of tests with bad headers (see regress specfile specifications).

 

Format of Reporter Input Specification File

The following defines the contents of the reporter input specification file:
 
 

Section / Field names	Description
[General] rptdir=<directory> [index=filename]	The rptdir entry defines the directory location of an existing directory whose directory topology exists as defined by the [Component-name], [TestSuite-name], and [Platform-name] sections.  The rptdir directory must exist, if not the command will exit with ERR_NO_RPTDIR.  Only a single [General] section should be provided in a specification file. If the index entry is present, the filename is the name of the computed index (e.g., index.html, or index.htm).
[Component-name] desc=<description> version=<version> dirname=<directory name>	[Component-name] is a high level component, such as a complete library.  For example, NSPR 2.0 or NETLIB, or the Security Library.  Multiple [Component-name] sections can be present in a specification file. desc is a string that describes the component (e.g., Security Library). version defines the version of the library that the filed reports should apply to (this is an expected version string) dirname defines the directory name (no / or .. allowed -- just the directory name) which is expected to be present under the dirname supplied in the [General] section or passed in on the command line.
[TestSuite-name] desc=description  component=name dirname=<directory name>	[TestSuite-name] defines an API test suite (a sub-component) of the higher level component.  For example, in the Security Library, APIs such as PKCS#5, PKCS#11, certificates/keys will each have their own indepent test suites (all driven by a different regression specification). Multiple [TestSuite-name] sections can be present, however component must refer to a provided component name section. desc is a string that describes the suite (e.g., PKCS#5).  component defines what component name this test suite is contained in. dirname defines the directory name (no / or .. allowed -- just the directory name) which is expected to be present under the <dirname>/<component name> directory.
[Platform-name] desc=description  dirname=<directory name> testsuite-name=yes|no	[Platform-name] defines what platforms each test suite is currently available on.  It is expected that this mapping is updated as suites are ported and executed against target platforms. desc is a string that describes the platform (e.g., Solaris 2.4.1).  dirname defines the directory name (no / or .. allowed -- just the directory name) which is expected to be present under the <dirname>/<component name> directory. testsuite-name is defined to be either yes or no.  yes means that the TestSuite

 

Pseudo-Code of Reporter Program Flow

Data structures:

enum DirNodeType{ DIRROOT, COMPONENT, TSUITE, PLATFORM, FILE} ;  typedef struct _StatusNode {     char *mut;  /* module under test */     char *mutversion;  /* module under test version */     char *platform;  /* platform string */     int stats[4] ;  /*         stats[0] = pass;  number of tests passed         stats[1] = fail;  number of tests failed         stats[2] = knownfail;   number of tests known to fail         stats[3] = malformed;   number of malformed tests  */ } StatusNode; typedef struct _DirectoryNode {     DirNodeType dntype;      char *name;    /* section name */       char *desc;     /* desc= within a section */      char *dirname;  /* dirname= within a section */      char *mostrecent;  /* only used on dntype = PLATFORM */      StatusNode *statnode;  /* The status for the current level */      DirectoryNode *dnlist[];   /* list of pointers */ } DirectoryNode;

Program logic:

/* 1.0 Process specification file */ Parse command line arguments, on error generate ERR_USAGE, otherwise set debug, rptdir, specfile variables. Open specfile, if not then  ERR_NO_SPECFILE. if index entry in [General Section] set index to filename, otherwise index = index.html. if no [General Section] then ERR_NO_GENERAL. if rptdir=NULL and no rptdir in [General Section] then ERR_NO_RPTDIR. /* 1.1 Add all component names to DirectoryNode tree */ Create DirectoryNode,  set as  root node, set as type DIRROOT. Iterate each section name that starts with Component-name {     Construct DirectoryNode new, add to root node, set as dntype = COMPONENT.     Add name,desc,version, and dirname to new node. } /* 1.2 Add all test suite names to DirectoryNode tree */ Iterate each section name that starts with TestSuite-name {     Locate component where component-name in section matches.     If no match, then print warning message     otherwise {         Construct new DirectoryNode.         Set node desc,dirname values, set as dntype = TESTSUITE         Add new node to component node dnlist.     } } /* 1.3 Add all test suite names to DirectoryNode tree */ Iterate each section name that starts with Platform-name {        Locate testsuite where testsuite-name in section  matches.     If no match, then print warning message     otherwise {         Construct new DirectoryNode.         Set node desc,dirname values, set dntype = PLATFORM.         Add new node to test suite node dnlist.     } } /* 2.0 Open each of the status nodes, and augment tree with HTML file objects */ AccumulatedPath = rptdir Select each node n  starting from top of tree {    Concatenate dirname to AccumulatedPath.      if (n.dntype == PLATFORM) {  /* only look at leaf nodes */          Open directory pointed to by AccumulatedPath, and process each file {               If file ends in  ".htm" or file ends in ".html" {  /* file is a report */                   Is there a corresponding report file? If so {                       Open status file and parse contents into a StatusNode (sn)                        If status file not malformed {                           Construct a new DirectoryNode.                           Set statnode = sn                           Set dirname as file;   /* this should be named something more abstract than dirname */                           Set node dntype = FILE.                           Add node to current node n.dnlist.                       } else print error message                   }                   else print error message.               }         }     }     Remove  dirname from AccumulatedPath } /* 2.1 Select most recent report at the PLATFORM level and store in tree */ For each node that is a PLATFORM node,     Iterate through the dnlist of FILE nodes and select the most recent filename (based on string prior to ".").       Set node.most.recent = filename. /* 2.2  Calculate suite results */ For each node that is a TESTSUITE node,     Construct new StatusNode, all values zero, set node.statnode.     For each PLATFORM node in dnlist,  for each stat, node.stats[i] += platform node stats values. /* 2.3 Calculate component results */ For each node that is a COMPONENT node,     Construct new StatusNode, all values zero,  set node.statnode.     For each PLATFORM node in dnlist,  for each stat, node.stats[i] += platform node stats values. /* 3.0 Write out all index files at ROOT, COMPONENT, SUITE, and PLATFORM level */ for each node in tree do {    switch (node.dntype)       ROOT:   Generate HTML report from dnlist as per HTML formatting.      COMPONENT:  Generate HTML report from dnlist as per HTML formatting.       SUITE:   Generate HTML report from dnlist as per HTML formatting.       PLATFORM:   Generate HTML report from dnlist as per HTML formatting. }   return NO_ERROR.

 
 

Format of Generated index.htm files

 

Level	HTML Format
ROOT	<HTML> <HEAD></TITLE></TITLE></HEAD> <BODY> <TABLE> For each component in ROOT { <TR><TD><A HREF="<filename>/<index>">desc</A></TD></TR> } </TABLE> </BODY> </HTML>
COMPONENT	<HTML> <HEAD></TITLE></TITLE></HEAD> <BODY> <TABLE> For each suite in component node { <TR><TD><A HREF="<filename>/<index>">desc</A></TD></TR> } </TABLE> </BODY> </HTML>
SUITE	<HTML> <HEAD></TITLE></TITLE></HEAD> <BODY> <TABLE> <TR><TD></TD></TR> </TABLE> </BODY> </HTML>
PLATFORM	<HTML> <HEAD></TITLE></TITLE></HEAD> <BODY> <TABLE> <TR><TD></TD></TR> </TABLE> </BODY> </HTML>

 

Sample Reporter Input Specification File The following defines a sample input specification file to reporter (which is highly domain specific):
 

[General] dirname=/u/zigbert/saw/projects/hardcore/prj-ttools/testing/rptroot  [Component-seclib]  desc=Security Library version=2.0 dirname=security-library [TestSuite-pkcs5] desc=Public Key Cryptography Standards # 5 component=seclib dirname=pkcs5 [TestSuite-crypto] desc=Cryptographic Functionality (bulk ciphers, RSA, DSA) component=seclib dirname=libcrypto [Platform=hp10_2]  desc=HP UX 10.2 dirname=hpux10_2 testsuite-pkcs5=yes testsuite-crypto=yes [Platform=sol2_4] desc=Solaris 2.4.1 dirname=sol2_4_1 testsuite-pkcs5=yes

 

---