Replacer and Regress Modifications
Newsgroup: mozilla.dev.tech.cryptoTechnical contact: Bob Relyea
Yell at the manager: Bob Lord
(Draft - Last updated: 11/20/1997)
1. Overview
In order to have ability to use single source and single executable to test wide range of cases some modifications are required to current versions of replacer and regress. Simple way to achieve this flexibility is to inject all tested data in a form of following structures. data_type data_table[] = {
data-1,
... ,
data-n
}
These tables can be constructed and injected by replacer from the data collected from its template file. This way we can use single executable to test all required cases.
Regress has to run each test with test ID as a parameter, which will let test executable to extract enough information to select appropriate data. Test ID encoding and decoding can be done many ways. A simply method is to use multiplication and division. Let do simple example to illustrate this method. Lets assume that we are testing n variables. Each of them has known number of values M1, M2, M3, ... ,Mn. and selected variables are V1, V2, ..., Vn.
testId = (…((Vn*Mn-1 + Vn-1)*Mn-2 + Vn-2)*Mn-3
+ …)*M1 + V1;
V1 = testId % M1;
V2 = (testId / M1) % M2;
V3 = ((testId / M1) / M2) % M3;
... ;
Vn = (…((testId / M1) / M2)…) % Mn;
This means, that replacer has to replace $[VARIABLEn] with
corresponding data_table[(…((testId/M1)/M2)…)%Mn] instead of specific
data as before.
testId will be retrieved from test’s first argument.
testId = atoi(argv[1]);
1. Replacer Modifications
- General information block in replacer template file has to specify singleSource = TRUE to indicate new way of testing or singleSource can be pass as a replacer argument.
- Template file specification has to introduce two new “Meta-Variables” $[DATA-TO-TEST] and $[SET-TEST-ID]. $[DATA-TO-TEST] will let replacer to inject all tested data. $[SET-TEST-ID] will let replacer to set testId.
- Each tested variable has to have specified type in its information block contained by replacer template file. This can be define as follow: TYPE = data_type
- Regress template file produce by replacer has to have a way to indicate test IDs to be tested.
2. Regress Modifications
- Regress template file has to contain test IDs generated by replacer. This can be implemented by added statement testId = testIdValue for each test information block
- Current format for regress template file is very inefficient for big number tests, because it stores number of information per each test.
Situation can be improved
- by moving “program” from test information block to general information block since program name is the same for tests.
- by removing “testname” from test information block and adding all necessary components to build above name to general information block
- by letting regress elect random values for the low importance variable we can eliminate necessity to pass test IDs through regress template file.
3. Advantages
- Space savings
- Time savings
- Debugging
Instead of producing significant number of source files and executables, we can operate on single source and executable.
There is no need to waste time to produce huge number of sources, compile them and link them.
There is no need to retrieve or regenerate appropriate sources and executables to start debugging
4. Disadvantages
- You have to be aware of possible syntax conflict between injected data tables and source template variables.
-
Numbers of combinations is limited by test ID representation, which is
currently implemented as “int”.
This means, that for example on 32-bit platforms numbers of combinations is limited to 2,147,483,647.
5. Porting test suites
- add “singleSource = TRUE” in general section of the replacer template file
- add $[DATA-TO-TEST] and $[SET-TEST-ID] in the replacer C template file
- makefile won’t be modified, because there is no need for it
- use exit codes from the range of 1 to 255, 256 reserved to indicate crash