You are currently viewing a snapshot of www.mozilla.org taken on April 21, 2008. Most of this content is highly out of date (some pages haven't been updated since the project began in 1998) and exists for historical purposes only. If there are any pages on this archive site that you think should be added back to www.mozilla.org, please file a bug.



Reporter Tool Specification

Newsgroup: mozilla.dev.tech.crypto
Technical contact: Bob Relyea
Yell at the manager: Bob Lord


(Updated 7/1/98)

Contents

  • Overview
  • Command Line Interface
  • Error Reporting
  • Summary File Format
  • Reporter Specification File
  • Pseudo-Code,  Program Flow
  • Format of Generated Index.htm
  • Sample Reporter Specfile
  • Notes on VRML display

  • Overview

    Reporter is a productivity tool that automates the web indexing of regression results generated by the regress tool. The tool takes a specfile (defined below) as input, which defines the Component/Test Suite/Platform topology of the report directory structure.  The tool then parses all the individual regress summary files found at the lowest level of the reporting hierarcy and updates all the intermediate index.html files.   The reporter tool is expected to automate the routine maintenance of updating web content to form an orderly and predictable testing results web, that can be reindexed automatically.
     
     
    Figure 1: Architecture Overview

     
    Command Line Arguments
     
    reporter  specfile=<filename> [rptdir=<dirname>] [debug] [novrml] [createdirs]
  • Filename is either an absolute path or relative path to an existing filename that contains a specification (as defined below).
  • Dirname is an optional parameter that overrides the dirname line in the general section of the specification file (see below).
  • The optional debug argument will turn on stdout debug information defined below -- soft errors.
  • The optional novrml argument will turn OFF the VRML display option (which creates .wrl files at the root and at the testsuite levels.) 
  • By default, reporter will generate VRML representation of the reporter tree.  Here is an example. [Note: you may view the legend by selecting View: Legend from the view menu of the cosmo player interface]
  • The optional createdirs will use the reporter specfile and create the corresponding directory tree.   This is useful for users who are creating the directory tree for the first time, or have added new component, testsuites or platforms to the specfile. (Reporter checks for existence of directories before creating them, so old directories or files are kept intact).

  • Note: The arguments above are order independent.
     

    Error Reporting

    The following table defines hard error will will prevent reporter from completing its task.
     
    Error Code Description
    NO_ERROR (0) Program ran to completion and completed its indexing job.
    ERR_NO_RPTDIR (1) Program could not open the directory rptdir (as defined in [General] section or passed in on command line). 
    ERR_NO_SPECFILE (2) Program could not open the specfile passed in on the command line.
    ERR_NO_GENERAL (3) Could not locate a [General] section in the specfile.
    ERR_NO_RPTDIR (4) Could not locate a rptdir= line in the [General] section.   Only if rptdir not passed in on the command line.
    ERR_NO_COMPONENT (5) Specification file does not contain a [Component] section. 
    ERR_NO_TESTSUITE (6) Specification file does not contain a [TestSuite] section.
    ERR_NO_PLATFORM (7) Specification file does not contain a [Platform] section.
    ERR_IDX_NOWRITE (8) Failed to write (or over-write) one of the index.html files. 
    ERR_USAGE (9) Provided an invalid or undefined argument. See usage.
    ERR_BADSECTIONS (10) One of the required fields in one of the sections was missing.
    The following table defines messages which are generated on standard output (if and only if debug parameter is supplied) to log "soft" errors -- errors that can be worked around:
     
    Error Message - reported to standard output Description
    Fatal error: One of the required fields (field) in <section name> was not present. One of the fields required in provide section was missing, see specification., corresponds to the hard error ERR_BADSECTIONS.
    reporter:  Malformed summary file <filename> : could not locate <field> field. The summary file, identified by <filename> did not conform to the specifiction (see below).
    reporter: Could not open directory <dirname> defined in section <section name>. Attempted to open a directory which did not exist.  The dirname variable as composed as follows: 
  • For section = [Component-name], dirname = <rptdir>/<component dirname>
  • For section = [TestSuite-name], dirname = <rptdir>/<component dirname>/<testsuite dirname>
  • For section = [Platform-name], dirname = <rptdir>/<component dirname>/<testsuite dirname>/<platform dirname>
  • Fatal error: error code = <number> : message Additionally, all hard errors will be reported, as defined in the table above, to standard output., use for all ERR_ codes defined above that don't have a more specific message defined in this table.
    reporter: Usage: reporter  specfile=<filename> [rptdir=<dirname>] [debug]  Corresponds to ERR_USAGE defined above.
     

    Format of Summary Specification files

    As a regression suite completes (driven by the regress tool), it writes out an HTML output file, and a regression summary specification file.  Both of these files are utilized in the creation of the higher level summary index files.

    The following defines the contents  of the reporter summary specification file, as generated by regress:
     
    Specification File Element Description of Element
    [Status] Is the only block in this specification file
    mut=description Defines a description for the module under test (e.g., ns/security/lib/crypto module could be described as "Security Library: Cryptographic Module (lib/crypto)".
    mutversion=versionstring Defines a version string for the collection of tests. It is assumed that each test will execute against this higher level versionstring version, and that the results will be reported against this versionstring.
    platform=<string> Defines the platform the tests were run under (e.g., HP UX 1.0, Solaris 2.4.1, etc.)
    pass=<number> Number of tests that passed.
    fail=<number> Number of tests that failed due to non-zero return codes, timeouts, OR any other abnormality when running the test.
    knownFail=<number> Number of tests that were known to fail because a bug number was specified in its regress specfile. 
    malformed=<number> Number of tests with bad headers (see regress specfile specifications).
     

    Format of Reporter Input Specification File

    The following defines the contents of the reporter input specification file:
     
    Section / Field names Description
    [General] 
    rptdir=<directory> 
    [index=filename] 
    [vrml=[TRUE|FALSE]]
  • The rptdir entry defines the directory location of an existing directory whose directory topology exists as defined by the [Component-name], [TestSuite-name], and [Platform-name] sections.  The rptdir directory must exist, if not the command will exit with ERR_NO_RPTDIR.  Only a single [General] section should be provided in a specification file.
  • If the index entry is present, the filename is the name of the computed index (e.g., index.html, or index.htm).
  • If vrml is set to FALSE, then reporter will NOT generate VRML representations of  the root and each test-suite.  By default vrml is set to TRUE.   Also by specifying novrml at the command line will override vrml=TRUE in the specfile.
  •  [Component-name] 
    desc=<description> 
    version=<version> 
    dirname=<directory name>
  • [Component-name] is a high level component, such as a complete library.  For example, NSPR 2.0 or NETLIB, or the Security Library.  Multiple [Component-name] sections can be present in a specification file.
  • desc is a string that describes the component (e.g., Security Library).
  • version defines the version of the library that the filed reports should apply to (this is an expected version string)
  • dirname defines the directory name (no / or .. allowed -- just the directory name) which is expected to be present under the dirname supplied in the [General] section or passed in on the command line.
  • [TestSuite-name] 
    desc=description 
    component=name 
    dirname=<directory name> 
     
  • [TestSuite-name] defines an API test suite (a sub-component) of the higher level component.  For example, in the Security Library, APIs such as PKCS#5, PKCS#11, certificates/keys will each have their own indepent test suites (all driven by a different regression specification).  Multiple [TestSuite-name] sections can be present, however component must refer to a provided component name section.
  • desc is a string that describes the suite (e.g., PKCS#5). 
  • component defines what component name this test suite is contained in.
  • dirname defines the directory name (no / or .. allowed -- just the directory name) which is expected to be present under the <dirname>/<component name> directory.
  • [Platform-name] 
    desc=description 
    dirname=<directory name> 
    testsuite-name=yes|no
  • [Platform-name] defines what platforms each test suite is currently available on.  It is expected that this mapping is updated as suites are ported and executed against target platforms.
  • desc is a string that describes the platform (e.g., Solaris 2.4.1). 
  • dirname defines the directory name (no / or .. allowed -- just the directory name) which is expected to be present under the <dirname>/<component name> directory.
  • testsuite-name is defined to be either yes or no.  Yes means that the TestSuite directory will contain a directory that holds test results from that name platform
  •  

    Pseudo-code of reporter program flow

    Data structures:
     
    enum DirNodeType{ DIRROOT, COMPONENT, TSUITE, PLATFORM, FILE} ;  

    typedef struct _StatusNode {  
        char *mut;  /* module under test */  
        char *mutversion;  /* module under test version */  
        char *platform;  /* platform string */  
        int stats[4] ;  /*  
           stats[0] = pass;  number of tests passed  
           stats[1] = fail;  number of tests failed  
           stats[2] = knownfail;   number of tests known to fail  
           stats[3] = malformed;   number of malformed tests  */  
    } StatusNode;  

    typedef struct _DirectoryNode {  
        DirNodeType dntype 
         char *name;    /* section name */  
         char *desc;     /* desc= within a section */  
         char *dirname;  /* dirname= within a section */  
         char *mostrecent;  /* only used on dntype = PLATFORM */  
         char *vrmlFileName; /* for nodes that has a .wrl representation Root, and   
                                             each testsuite, this field indicates the name of the wrl file*/   
         StatusNode *statnode;  /* The status for the current level */  
         DirectoryNode *dnlist[];   /* list of pointers */  
    } DirectoryNode; 

    Program logic
     
    /* 1.0 Process specification file */  
    Parse command line arguments, on error generate ERR_USAGE, otherwise set debug, rptdir, specfile variables.  
    Open specfile, if not then  ERR_NO_SPECFILE.  
    if index entry in [General Section] set index to filename, otherwise index = index.html.  
    if no [General Section] then ERR_NO_GENERAL.  
    if rptdir=NULL and no rptdir in [General Section] then ERR_NO_RPTDIR.  

    /* 1.1 Add all component names to DirectoryNode tree */  
    Create DirectoryNode,  set as  root node, set as type DIRROOT.  
    Iterate each section name that starts with Component-name {  
        Construct DirectoryNode new, add to root node, set as dntype = COMPONENT.  
        Add name,desc,version, and dirname to new node.  
     

    /* 1.2 Add all test suite names to DirectoryNode tree */  
    Iterate each section name that starts with TestSuite-name {  
        Locate component where component-name in section matches.  
        If no match, then print warning message  
        otherwise {  
            Construct new DirectoryNode.  
            Set node desc,dirname values, set as dntype = TESTSUITE  
            Add new node to component node dnlist 
        }  
     

    /* 1.3 Add all test suite names to DirectoryNode tree */  
    Iterate each section name that starts with Platform-name {  
        Locate testsuite where testsuite-name in section  matches.  
        If no match, then print warning message  
        otherwise {  
            Construct new DirectoryNode.  
            Set node desc,dirname values, set dntype = PLATFORM.  
            Add new node to test suite node dnlist 
        }  
     

    /* 2.0 Open each of the status nodes, and augment tree with HTML file objects */  
    AccumulatedPath = rptdir  
    Select each node n  starting from top of tree {  
       Concatenate dirname to AccumulatedPath 
         if (n.dntype == PLATFORM) {  /* only look at leaf nodes */  
             Open directory pointed to by AccumulatedPath, and process each file  
                  If file ends in  ".htm" or file ends in ".html" {  /* file is a report */  
                      Is there a corresponding report file? If so {  
                          Open status file and parse contents into a StatusNode (sn 
                           If status file not malformed {  
                              Construct a new DirectoryNode.  
                              Set statnode = sn  
                              Set dirname as file;   /* this should be named something more abstract than dirname */  
                              Set node dntype = FILE.  
                              Add node to current node n.dnlist 
                          } else print error message  
                      }  
                      else print error message.  
                  }  
            }  
        }  
        Remove  dirname from AccumulatedPath  
     

    /* 2.1 Select most recent report at the PLATFORM level and store in tree */  
    For each node that is a PLATFORM node,  
       Iterate through the dnlist of FILE nodes and select the most recent filename (based on string prior to ".").  
          Set node.most.recent = filename 

    /* 2.2  Calculate suite results */  
    For each node that is a TESTSUITE node,  
        Construct new StatusNode, all values zero, set node.statnode.  
        For each PLATFORM node in dnlist,  for each stat, node.stats[i] += platform node stats values.  

    /* 2.3 Calculate component results */  
    For each node that is a COMPONENT node,  
        Construct new StatusNode, all values zero,  set node.statnode.  
        For each PLATFORM node in dnlist,  for each stat, node.stats[i] += platform node stats values.  

    /* TAT.6 create VRML display */  
    if (vrml option is on)  
        Create a CoordNode tree having 1 to 1 correspondence with the Directory node tree (but up to a certain tree height [currently set to 2]).   Each CoordNode will store the coordinate and color information to paint the VRML representation.  
      
        Calculate the 3-D coordinates and color each Node will have (where a node is a test-suite, or platform, or component etc.)   Color code: RED if more than 50% of tests (under that node) fail or malform, ORANGE if less than 50% of tests fail or malform, GREEN if all the test pass or are known failures, and YELLOW if the total number of tests ran is zero.  
      
       Traverse the CoordNode tree and create the VRML code (a sphere with label) for each node (also generate the appropriate links).    

       By setting the tree height limit to 2 (i.e the tallest CoordNode tree created will have at most 3 levels), only the root and each test suite node will be represented by a .wrl file.    The name of the .wrl file at the root is "Root.wrl" and the at each Testsuite is <testsuitename>.wrl.  

    /* 3.0 Write out all index files at ROOT, COMPONENT, SUITE, and PLATFORM level */  
    for each node in tree do {  
       switch (node.dntype 
          ROOT:   Generate HTML report from dnlist as per HTML formatting.  
         COMPONENT:  Generate HTML report from dnlist as per HTML formatting.  
          SUITE:   Generate HTML report from dnlist as per HTML formatting.  
          PLATFORM:   Generate HTML report from dnlist as per HTML formatting.  
     
      
    return NO_ERROR.

     
     

    Format of generated index.htm files
     
    Level HTML Format
    ROOT <HTML>  
    <HEAD></TITLE></TITLE></HEAD>  
    <BODY>  
    <TABLE>  
    For each component in ROOT {  
    <TR><TD><A HREF="<filename>/<index>">desc</A></TD></TR>  
     
    </TABLE>  
    </BODY>  
    </HTML>
    COMPONENT <HTML>  
    <HEAD></TITLE></TITLE></HEAD>  
    <BODY>  
    <TABLE>  
    For each suite in component node {  
    <TR><TD><A HREF="<filename>/<index>">desc</A></TD></TR>  
     
    </TABLE>  
    </BODY>  
    </HTML>
    SUITE <HTML>  
    <HEAD></TITLE></TITLE></HEAD>  
    <BODY>  
    <TABLE>  
    <TR><TD></TD></TR>  
    </TABLE>  
    </BODY>  
    </HTML>
    PLATFORM <HTML>  
    <HEAD></TITLE></TITLE></HEAD>  
    <BODY>  
    <TABLE>  
    <TR><TD></TD></TR>  
    </TABLE>  
    </BODY>  
    </HTML>
     

    Sample Reporter input specification file

    The following defines a sample input specification file to reporter (which is highly domain specific):
     
    [General]  
    dirname=/u/jhines/saw/projects/hardcore/prj-ttools/testing/rptroot  

    [Component-seclib]  
    desc=Security Library  
    version=2.0  
    dirname=security-library  

    [TestSuite-pkcs5]  
    desc=Public Key Cryptography Standards # 5  
    component=seclib  
    dirname=pkcs5  

    [TestSuite-crypto]  
    desc=Cryptographic Functionality (bulk ciphers, RSA, DSA)  
    component=seclib  
    dirname=libcrypto  

    [Platform=hp10_2]  
    desc=HP UX 10.2  
    dirname=hpux10_2  
    testsuite-pkcs5=yes  
    testsuite-crypto=yes  

    [Platform=sol2_4]  
    desc=Solaris 2.4.1  
    dirname=sol2_4_1  
    testsuite-pkcs5=yes

     

    Notes on the VRML display:
     
  • When viewing with Cosmo player, use the 'seek' button to focus on a desired node.
  • To view the legend, select View: Legend from the Views menu
  • At the bottom most level (regress generated results), all nodes should appear in a line along the z plane (along the line of sight).  The nodes are sorted alphanumerically in increasing order from back to front.  The effect this acheives is that when dates are the labels for these nodes, the most current node will appear closest to the viewer. Hence, the user can easily pick out the most current results.
  • Nodes which have have failure underneath it are closer to the viewer than nodes that are all PASS (i.e. red and orange nodes appear closer, where as green nodes appear farther.)
  • All the code necessary to implement VRML display is packaged in the files:
        genvrml.h  genvrml.c
    genvrml.h contains c preprocessor CONSTANTS that may be changed to alter the VRML display.  Below is a list of constants and their purpose:
     
    Constant Function
    LINE_SHININESS defines the brightness with which the lines interconnecting nodes appear.  Currently set to 1.0 (the maximum).   Value can range from 0.0 to 1.0
    X_ORIGIN, Y_ORIGIN, Z_ORIGIN defines the default origin.
    NODE_RADIUS defines the radius of each sphere/node
    CAPTION_X_OFFSET, CAPTION_Y_OFFSET defines the offset in both x and y directions from the top of each sphere/node with which label text is to be drawn
    FONT_STYLE, FONT_SIZE, FONT_FAMILY Font attributes for the test label above each node
    MAGIC_NUMBER The cotagent of an angle (which should be half of the total angle of view).  i.e. if an angle of view of 90 degrees is desired, then set MAGIC_NUMBER to cotan(45degrees)
    LEVEL_SPAN The distance between nodes in each level in the tree. (This is displacement along the y-axis).
    LEAF_SPAN The distance between each bottom most leaf in the displayed tree. 
    All leaves have equal (x axis) spacing between them that is defined by this constant.
    Z_SPAN The distance of displacement between nodes on the Z axis.