You are currently viewing a snapshot of www.mozilla.org taken on April 21, 2008. Most of this content is highly out of date (some pages haven't been updated since the project began in 1998) and exists for historical purposes only. If there are any pages on this archive site that you think should be added back to www.mozilla.org, please file a bug.

Skip to main content

The NSS Build System:
History and Future Directions

Newsgroup: mozilla.dev.tech.crypto
Main technical contact: Bob Relyea
Manager: Wan-Teh Chang

Contents

Background

Network Security Services (NSS) started out as the base security code used by the original Netscape Navigator products. In the beginning, this code was packaged and built as part of the monolithic client build system. The NSS build system was disentangled from the client build system starting with Communicator. Since then, it has gone through a series of modifications in response to the changing needs of NSS customers, whose products now include the following: Each of these products includes legacy NSS-dependent code and has specialized build requirements. We must continue to support them while working toward the goal of an independent NSS build solution that integrates smoothly with a modular Mozilla build system.

Beyond the goal of merely building without error on all platforms, the NSS team has the goal of building efficient, high-performance crypto code on all platforms. NSS and its current build system contain much embedded knowledge of how to get optimal performance out of specific compilers on specific platforms, and of pitfalls (including compiler deficiencies) to avoid on specific platforms. Code that merely builds correctly on these platforms may be far from optimal for some NSS customers.

This document describes the history of the NSS build system, iPlanet and Mozilla requirements for a new build system, and some proposals for how to meet those requirements. Please send feedback on these proposals to mozilla.dev.tech.crypto.

For more background information on NSS, see the NSS Project Page.

Definitions

This document uses the following terms related to the Netscape, Mozilla, and NSS build systems:

autoconf: A build system framework that is popular in the open source community.

config: The original build system for Netscape Navigator.

coreconf: The build system, based on config, that was developed and is still used for NSS.

svbld: The internal iPlanet wide automated build and release system.

History

1997:
    The browser team frequently "froze" builds based on the original monolithic build system while browser problems were being resolved. When this happened, the NSS team was blocked and could not address the needs of its other customers at the time, the Netscape servers.

    Therefore, as part of the modularity work on Communicator, Netscape engineers disentangled three components from the monolithic Communicator build system to be built as separate binary components. These components represented the basic code that has been shared by Netscape clients and servers since the beginning:

    • NSS provides the core security APIs.
    • Netscape Portable Runtime (NSPR) provides a platform-neutral API for system level and libc like functions
    • DBM provides low-level data-base management services.
    Once this was accomplished, the NSS team was able to meet its other customer needs even when the browser team had to freeze development for the browser.

    To faciliate this change, we created a hybrid version of the client and server build systems that made it possible to build, release, and import binary NSS releases. At this point two important things happened:

    • The client and server teams could not agree on a binary object-naming convention.
    • The client team never implemented a binary import scheme. Instead, client builds were accomplished by pulling the various components from a stable NSS release tag (representing the source used to produce the latest QA'd binary release).
    Within a year, the needs of the various teams releasing binary components fractured this build system into at least two versions: one used by NSS and DBM and one used by NSPR. Although these versions diverged, they remained close enough to move components easily from one build system to another.

    These closely related build systems had three major advantages over the client build system:

    • A single set of Makefiles worked on all platforms (with the exception of Macintosh).
    • The system could easily build separate components by importing binary versions of dependent components. It was no longer necessary to build all the dependencies prior to building one's one product. This greatly improved product development time.
    • The system ran with minimal external tool dependencies: ns/tools for Windows, standard Unix commands for Unix plus either a native compiler/linker or gcc.
1998:
    Netscape announced the historic decision to develop all its browser products as open source. Because of patent restrictions on proprietory code and US export rules, we could not ship the source to our security code at that time. Instead, we patched the client security calls with "hieroglyphs" to obscure the security links in Communicator, which allowed us to publish the rest of the source code.

    The original code contained the Netscape client's config build system (on which coreconf was originally based). Over the next couple of years, Mozilla developers converted the Unix build code to the autoconf system (used by Linux and other open source developers). It started as wrappers, and slowly moved to complete autoconf usage. At this time NSPR was integrated with a dual build system, because NSPR still needed to build and maintain binary releases as a component of the iPlanet server product releases. A separate copy of DBM was converted to the autoconf build system.

    The NSS team went on to solve the problem of how we could produce a security module that would

    • work with a varienty of clients, each with its own version of NSPR (or no version of NSPR)
    • provide the same UI for each client
    • allow multiple applications to share a common set of crytographic "tokens" and certificates at the same time.
    The result was Personal Security Manager (PSM).

    At the same time, the Mozilla team made (or acquired) enhancements to their build system to support the following capabilities:

    • All the object files could be installed in a parallel tree (a feature that the Windows version of Communicator had, but other platforms did not).
    • The ability to build cross plaform.
    • Parallel building.
1999:
    In the wake of signals that the US Government was considering easing the export regulations governing crypto software, the NSS team began the process of sanitizing the NSS code and examining the impediments to building a fully open source version of NSS. One issue we examined was the possibility of converting to the autoconf build system. We attempted to build DBM as a component without pulling the monolithic Mozilla build system. The proposal was to give DBM its own autoconf system, which could be called separately from the monolithic autoconf system.

    The Mozilla team rejected this suggestion. No alternative was available and neither the Mozilla team nor the NSS team had the resources to address the problem, which was not a high priority for either team. Coupled with the difficulties the NSPR team was having with the dual build system, this situation led the NSS team to maintain the coreconf build system for NSS.

    Because of the RSA patent, NSS used components built from proprietary licensed source code that could not be publicly disclosed. For this reason, and also because of export controls, the NSS team built PSM binaries internally and distributed them only as a complete executable program.

    Another important event in 1999 was the acquisition of Netscape by AOL. Soon after, the NSS engineering team, along with the other Netscape server teams, became part of iPlanet E-Commerce Solutions.

2000:
    The RSA patent expired in September 2000, and the NSS team immediately developed replacements for all the proprietary code on which NSS previously depended. As a result of these changes, all NSS source could start being posted to Mozilla. By December the PSM team had created a set of Makefiles that could be attached to the mozilla build system.
2001:
    Changes in export regulations allowed Netscape and Sun to ship NSS in the form of shared libraries, so the NSS team modified NSS to ship as shared libraries in addition to static libraries. In the process we added .def files to control what symbols are exported from our shared libraries.

    To understand the situation today (April 2001), it's important to recognize that, unlike other Mozilla components, PSM, NSS, and NSPR each operate under their own release schedules and product requirement documents (PRDs). Because of this, tree locking and checkin rules will always differ between the NSS build system and the Mozilla build system.

    In the future, as embedding becomes more important, more Mozilla components will start branching off on their own schedules as well. In the long term, iPlanet and other customer requirements for a separate NSS build system converge with Mozilla requirements for separate components. The only difference between NSS and other components in this respect is that NSS is a more mature component with a broader customer base.

    At the same time, the Mozilla team may well have its own requirement to be able to tweak the NSS branch of the Mozilla build system without worrying about NSS release schedule or NSS team approval. Neither of these requirements have ever been stated, but it would be foolish not to recognize that they exist. These are the same requirements that led to the initial bifurcation of coreconf in 1997.

Special Considerations

In addition to meeting the different needs of various customers and engineering teams, the current NSS build system takes account of various specialized customer needs. Future work on the build system must take these considerations into account.

The main consideration involves performance. Most products built with autoconf have code that is conditionally compiled based on the presence or absence of some symbol in a system header file, such as a #define or a certain function declaration. The objective of those measures is to produce code that correctly compiles on all platforms, not using symbols or calling functions that do not exist on all systems.

Some people have wondered why NSS uses information about the particular system for which it is being built as a basis for deciding what exactly to build. The answer is that such information is vital to obtaining maximum performance in NSS. Information that is system-dependent and cannot be gleaned from header files includes the following:

  • whether loop unrolling improves speed on that system or not
  • whether byte writes are signficantly slower than word writes on that platform.
  • whether byte reads are slower than word reads, shifting and masking
  • whether the system has a FPU that is capable of doing 64-bit integer arithmetic
  • whether using the FPU is faster than using the integer ALU for multiplication.
  • compiler flaws--for example, does the compiler's preprocessor truncate all values to 32 bits for evaluating #if expressions?
Some autoconf build systems include large and elaborate sets of programs that attempt to determine the answers to those questions at build times, e.g. running benchmarks, attempting to compile test sources and noting compiler failures, etc. The amount of time required to implement such systems is enormous.

On many platforms, assembler code is necessary to get optimal performance in time-critical functions. Such code is dependent on the type of the target system and also the tools used to build. For example, gnu assembler syntax is totally different from any system's native assembler. Such assembler code is either present or not in the product's sources, and it is built, or not, based on whether it is appropriate to the system and tools being used. Such files will continue to be built based on information about the system for which they are being built.

iPlanet Requirements for the NSS Build System

The iPlanet NSS engineering team's customers and build requirements are different from those of the Mozilla team. Unlike other Mozilla components, PSM, NSS, and NSPR operate under their own release schedules and PRDs. Because of this, tree locking and checkin rules will always differ from those for Mozilla. The NSS team needs to be able to make adjustments to its build system, whether based on coreconf or autoconf, without gates or controls by the Mozilla team. This means NSS can never be part of a monolithic build system.

The iPlanet engineering team has the following general requirements for the new NSS build system, in order of priority:

  1. Build NSS without pulling parts of Mozilla that iPlanet products do not need or use and that are not being pulled in present NSS builds for iPlanet.
  2. Preserve and where possible improve the current level of performance, which is important for many iPlanet server products.
  3. Build NSS with minimal external tools dependencies. The bigger these dependencies, the bigger pressure on our release engineering team. When these dependencies vary greatly from the dependencies of other components built by our release team, iPlanet management starts asking nasty cost/value questions.
  4. Build all NSS platforms (except Macintosh, see note below) with a single build system and a single set of Makefiles.
  5. Build from imported binary components. Production, QA'd versions of NSS should be supplied by building with production, QA'ed, binary versions of NSPR and DBM.
  6. Build as a component reliably on all platforms (except Macintosh, see below).
  7. Build from a single checked out tree on all NSS platforms simultaneously (requirement of our svbld system).
NOTE: "All NSS platforms" in the list above does not include Macintosh. The only component that needs NSS on Macintosh is PSM, so the PSM build process also builds its own version of NSS for Macintosh.

Mozilla Requirements for the NSS Build System

We believe that Mozilla has the following requirements for the new NSS build system, in order of priority:
  1. NSS should build reliably when invoked from the Mozilla build system.
  2. NSS should build from a read-only tree.
  3. NSS should build cross-platform.
  4. NSS should not interfere with the -j flag (that is, NSS should not force serialization of a parallel build).
  5. NSS should be built by some standard autoconf invocation.

Proposals

We are considering two proposals for a new NSS build system. Please send comments to mozilla.dev.tech.crypto.

Proposal One:

    The NSS engineering team integrates the Mozilla build requirements into coreconf. Mozilla Requirements 1 through 3 are relatively easy to do in coreconf. It takes one day to get cross-plaform code working (there is only one command in the coreconf build that needs to be native). coreconf is already OBJDIR driven. It would take a couple of days to allow it to build its objects into a separate object tree.

    Mozilla Requirement 4 requires more understanding about what parallelized builds mean.

    Mozilla Requirement 5 is actually the hardest, although it seems to be the lowest priority.

    In the meantime, the Mozilla team is working at its own pace toward a componentized version of autoconf that can be used to produce a version of NSS as simply and reliably as the current NSS 3.2 Build Instructions.

    This proposal meets all the listed requirements without using autoconf.

Proposal Two:
    The NSS engineering team continues using its own build system, and the Mozilla team develops a parallel build system that is decoupled from NSS (preferably in a separate tree or tag). Developers can overlay the parallel build system on an existing NSS tree to produce an autoconf tree that the mozilla team may use to build NSS from. This build system is maintained by the Mozilla community.

    The C code for this special tree would still be pulled from a stable, tested version of NSS and not modified. This suggests a tree where autoconf is overlayed on top of an existing NSS system. There is an example of this already in security/dbm, where the security Makefiles pull in the mozilla DBM source but use the coreconf build system to build DBM.

    The disadvantage of this proposal is we put off the work componentizing the Mozilla build system until necko, gecko, or someone else needs it (which is not necessarily a bad resource trade-off).