JavaScript Test Framework
Stategies for Embedding QA
Stategies for Embedding QA
Purpose
The purpose of this document is to present some ideas about how a JavaScript framework will work. This framework is intended for use by both developers and QA engineers. It includes individual test scripts which could be executed standalone or in automated test suites.
Design
At its core, the design of the framework is relatively simple. The main ingredient is the JavaScript test scripts. Each script is created to test a specific interface. NgDriver can be used to run a suite of scripts for automated testing. Alternatively, these scripts can be run standalone in the browser.
If there is a topic requiring multiple scripts, like Accessibility, then a test library containing general methods available to all the topical test scripts will be written. This test library (i.e. Accessibility.js) will be included in the individual scripts. At a lower level, there will be a test library containing methods available to ANY script. This library should be called something like testUtilities.js, since it will include helper or utility functions that any script could use (i.e. object creation, try/catch test procedures, posting results, request handling, etc.)
A more problematic aspect of the design, if it were to be included in the framework, will require some type of inclusion of existing C++ test cases in TestEmbed. If this is pursued, there must be direct framework support for these test cases. QA would need considerable help from development to implement this feature. One approach would require using XPConnect to allow the C++ interfaces to talk to JS. This would essentially make everything look like an XPCOM object, permitting the C++ tests to be understood in the JS framework. While TestEmbed's modular style will make it easier to facilitate such an approach, it still would require some work to isolate test objects and test code. An alternative of this method is to rewrite these test cases in JavaScript.
Regardless of whether we utilize the existing C++ code, we still need to create a web browser window to control the chrome. This will be a key part of the framework and will make it possible to manipulate content windows. Isolated test scripts will need to use an nsIWebBrowser object. Other objects like nsIWebNavigation can be QI'd off of nsIWebBrowser. One possibility is to build a component, either in C++ or JS, that creates a QA web browser. We would need to implement chrome interfaces that would use web browser to chrome upcalls. This would involve hooking into JS, invoking JS handlers. (TBD. speak with a developer about this).
Strategies
Embedding QA believes that the soundest approach is to pursue three basic strategies:
The purpose of this document is to present some ideas about how a JavaScript framework will work. This framework is intended for use by both developers and QA engineers. It includes individual test scripts which could be executed standalone or in automated test suites.
Design
At its core, the design of the framework is relatively simple. The main ingredient is the JavaScript test scripts. Each script is created to test a specific interface. NgDriver can be used to run a suite of scripts for automated testing. Alternatively, these scripts can be run standalone in the browser.
If there is a topic requiring multiple scripts, like Accessibility, then a test library containing general methods available to all the topical test scripts will be written. This test library (i.e. Accessibility.js) will be included in the individual scripts. At a lower level, there will be a test library containing methods available to ANY script. This library should be called something like testUtilities.js, since it will include helper or utility functions that any script could use (i.e. object creation, try/catch test procedures, posting results, request handling, etc.)
A more problematic aspect of the design, if it were to be included in the framework, will require some type of inclusion of existing C++ test cases in TestEmbed. If this is pursued, there must be direct framework support for these test cases. QA would need considerable help from development to implement this feature. One approach would require using XPConnect to allow the C++ interfaces to talk to JS. This would essentially make everything look like an XPCOM object, permitting the C++ tests to be understood in the JS framework. While TestEmbed's modular style will make it easier to facilitate such an approach, it still would require some work to isolate test objects and test code. An alternative of this method is to rewrite these test cases in JavaScript.
Regardless of whether we utilize the existing C++ code, we still need to create a web browser window to control the chrome. This will be a key part of the framework and will make it possible to manipulate content windows. Isolated test scripts will need to use an nsIWebBrowser object. Other objects like nsIWebNavigation can be QI'd off of nsIWebBrowser. One possibility is to build a component, either in C++ or JS, that creates a QA web browser. We would need to implement chrome interfaces that would use web browser to chrome upcalls. This would involve hooking into JS, invoking JS handlers. (TBD. speak with a developer about this).
Strategies
Embedding QA believes that the soundest approach is to pursue three basic strategies:
- Understand and improve NgDriver automated approach. The technology is already in place. QA needs to understand and convey to script writers how this works. NgDriver runs individual html files which contain inline JavaScript. These files are used to test specific Embedding interfaces or different permutations of API methods. A developer or QA engineer can write a single script and place it in the NgDriver folder. The user can select All or any of the scripts to run for automation.
- Development of test libraries. QA can develop a two-tier set of libraries which can be used by individual script writers. As described above, there will be a general utility library at the lowest level. One possibility is to include utility methods which could generate html templates customized for Embedding script writing. At the secondary level, Embedding area libraries (i.e. interfaces that group together into areas like accessibility, DOM, listeners) will be able to use this general library.
- Rewrite C++ test cases. QA believes that this will be much
simpler and more expedient than attempting to run C++ test cases in
a JS environment. While it's fairly straightforward to connect from JS
to C++, it's more difficult the other direction. The solution would be
to write individual .idl files for each test interface, encapsulating our
test cases, and wrap them around XPCOM objects. The XPCOM interfaces would
wrap around the C++ cases to generate JS tests.
It will be more practical to rewrite and port existing C++ test cases into JS. Of course there are some existing problems. Some interfaces like nsIClipboardCommands, nsIDOMWindow, and nsISelection are difficult to implement in JS.
Another advantage of having everything written in JS is that it will be easier to use the JS Debugger. It's true that even with C++ code, we could still traverse into deeper code with the debugger and peek into XPCOM objects, but it will be more problematic.